Mailman 3 - Distutils-SIG

Allow to debug extension modules
by Ivan Pozdeev 09 Jan '18

09 Jan '18

In the https://mail.python.org/pipermail/python-ideas/2018-January/048558.html thread, it was discovered that, stumpingly enough, it seems not possible to debug extension modules! * For a release Python, it's not even possible to build an extension without optimizations * For a debug Python, it's possible to build but not possible to install it, or any other package that depends on it Moreover, the feedback says that it makes no sense to build an extension with _DEBUG against a release Python and vice versa. But distutils' build_ext logic does (or at least trying to do since that breaks) just this, as it looks at the --debug option to `build' instead of checking the type of the running Python. Am I missing something crucial here, or do I need to fix distutils now to be able to debug a problem in my extension module? -- Regards, Ivan

1 1

Add processor generation, Windows version (and possibly other things) to wheel platform tag
by Ivan Pozdeev 09 Jan '18

09 Jan '18

This is a follow-up to this thread (the 2nd link is continuation): https://mail.python.org/pipermail/distutils-sig/2017-October/031750.html https://mail.python.org/pipermail/distutils-sig/2017-November/031759.html After discovering https://www.python.org/dev/peps/pep-0425/ , I see that what I was asking for is add more data to the wheel's platform tag. Specifically: * Processor generation if it's higher than what Python officially supports * Windows version if it's higher than what Python officially supports (like it's already done for MacOS) In fact, references to other libraries that the wheel expects on the system that Nick Coghlan proposed in https://www.python.org/dev/peps/pep-0459/ <https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension> can also be implemented in the platform tag. They belong there with the current infrastructure because: * Anything external to a Python installation that is expected on the system, combined, _is_ the platform. * A wheel's name has to be unique and descriptive (it needs to include everything required for pip to pick the best download). Sure, this can make the name very long, but the more to describe, the longer the description. <https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension> -- Regards, Ivan

1 0

Should abi3 tag be usable on Windows?
by Cosimo Lupo 07 Jan '18

07 Jan '18

Hello, CFFI has recently added support for Py_LIMITED_API extension modules built for CPython 3. The wheel module since version 0.30.0 also supports passing —py-limited-api cp3X to bdist_wheel command to allow the generated .whl to be installed on all CPython versions equal or greater than the one specified. Yesterday I was trying to apply this on a cffi-built extension module, and it worked for Linux and macOS but failed for Windows: https://github.com/hynek/argon2_cffi/pull/32 The AssertionError from wheel.pep425tags complains that a tag with abi3 would be unsupported for the target platform. Alex Gronholm commented > imp.get_suffixes() does not seem to contain any ABI3 suffixes, but I'm not sure if this is even applicable on Windows. Incidentally, I noticed one specific package, PyQt5, that distributes both abi3-tagged wheels for Mac and manylinux and Windows wheels for a range of cp35.cp36.cp37 but with abi tag set as “none”, and they do seem to work. So, can one make such py_limited_api wheels work on Windows with the current state of the tooling, and if so how? Thank you in advance Cosimo Lupo

1 2

Immutability of Release Metadata in Warehouse
by Donald Stufft 25 Dec '17

25 Dec '17

For those who are not aware, legacy PyPI would allow you to run ``twine register`` on a release that had already been created in order to modify the metadata that PyPI had recorded for that release (keyed by version number). This wasn’t a super widely used feature, but it’s primary use case was when folks would mistakenly release a project that had a broken description that wouldn’t correctly render on PyPI. With the move from the legacy PyPI code base to Warehouse for handling uploads, this feature had been somewhat inadvertently lost. This issue was raised in https://github.com/pypa/warehouse/issues/2170 <https://github.com/pypa/warehouse/issues/2170>. This email is provide notice that after thinking about this issue for awhile, we’re *not* going to restore the ability to modify the metadata associated with a release after it has been uploaded.The eventual goal is that it should be ideally possible to treat the files that are on PyPI as the source of truth for all metadata, and that the data stored in the database is simply an optimization for accessing and presenting that data on PyPI. Obviously if we allow modifications to the metadata as stored in the PyPI database, that would allow this metadata to “drift” from what is actually stored in the files, which would prevent that goal from being realized. It is true that even with this change, there is not a guarantee that the metadata in the database does not match what is in the file(s) that have been uploaded to PyPI, even going into the future. Thus the decision to not restore this feature is not the only step on the way to being able to assert this guarantee, but it is *one* of them. The most common reason for wanting to modify any metadata after the fact is to fix typos etc that made it into the description prior to publication. It is our opinion that the best way to handle these is to either cut a new release (it can be a post release if that’s all that has changed) or to validate the description field prior to publication (which can currently be done using readme_renderer or restview). Longer term we also plan to introduce the ability to “stage” releases so that releases and files can be uploaded to a temporary location within Warehouse, and visible with a special URL to allow people to manually validate the actual bits that were uploaded before pushing a “finalize” or “publish” button that would flip it from being a mutable, hidden release to an immutable, publicly viewable release. If folks have other use cases where they’ve used the ability to modify release metadata after it had been released that they feel is an important enough use case that it needs to be supported in some way, please let us know either in this thread or by opening an issue on https://github.com/pypa/warehouse <https://github.com/pypa/warehouse> so we can figure out if it’s something we want to support, if one of the other mechanisms we’re planning on adding will support it, or if there is some new mechanism we can add that can support it better.

4 6

How do add Numpy package on Python
by Jeremiah Lethoba 23 Dec '17

23 Dec '17

Hello I installed Python on a MacBook but it does not have numpy package. How do i add it ? -- This message and attachments are subject to a disclaimer. Please refer to http://www.it.up.ac.za/documentation/governance/disclaimer/ for full details.

1 0

Re: [Distutils] Immutability of Release Metadata in Warehouse
by Matthias Bussonnier 20 Dec '17

20 Dec '17

One case I could see is the use of the requires_python metadata. It was not included in the recent release of Django 2.0 (which is py 3 only) and making a new release will be useless as pip on py2 will still see Django 2.0.0 as Py 2 compatible download it and crash. I'm going to assume with time more packages will be affected even with minor version of Python. -- M On Dec 19, 2017 21:17, "Sumana Harihareswara" <sh(a)changeset.nyc> wrote: For those who want to track the relevant GitHub issues: * supporting Markdown https://github.com/pypa/warehouse/issues/869 * staged releases https://github.com/pypa/warehouse/issues/726 * advising packagers to run `python setup.py check -r -s` https://github.com/pypa/python-packaging-user-guide/issues/210 -- Sumana Harihareswara Changeset Consulting https://changeset.nyc _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

4 4

Warehouse update: highest priority tasks
by Sumana Harihareswara 20 Dec '17

20 Dec '17

Cross-posting from pypa-dev https://groups.google.com/forum/#!topic/pypa-dev/2hulwNQFhRk . I'm realizing that pypa-dev might be a better place to send updates like this than distutils-sig and would appreciate feedback on that question (probably off-list is better). -Sumana Harihareswara -------- Forwarded Message -------- Subject: Warehouse update: highest priority tasks Date: Tue, 19 Dec 2017 18:59:01 -0500 From: Sumana Harihareswara <sh(a)changeset.nyc> To: pypa-dev(a)googlegroups.com Today the Warehouse project team met to discuss what to deliver first, in the Maintainer MVP milestone[0] (full notes on the Python wiki[1]). The point of this first milestone is to give package maintainers a solid chance to try out Warehouse and report critical bugs early. We'll include several features to let users change and update your accounts & project files: * Enable pages to delete, hide, unhide a project/release/file[2] * Enable users to modify their own account[3] * UI for adding maintainers[4] * Implement "reset lost password" feature[5] Due to changes in hosting documentation on pythonhosted, we will implement a button for users to delete or redirect the documentation they uploaded.[6] We'll make and maintain a roadmap and publicize it on pypi.org, then move it to our GitHub repo when work probably slows down after the launch.[7] We've slotted in some additional issues, and we made some decisions about release metadata immutability[8] and a couple other issues you’ll hear more about on distutils-sig in the next couple days. So now that we've made this first pass and are fairly confident what’s left between us and this first milestone, we're figuring out how long it'll be till we can deliver that. Laura Hampton and I will keep you posted. And we welcome help in GitHub[9] and feedback on GitHub issues or here on the list! -Sumana Harihareswara [0] https://github.com/pypa/warehouse/milestone/8 [1] https://wiki.python.org/psf/PackagingWG/2017-12-19-Warehouse [2] https://github.com/pypa/warehouse/issues/424 [3] https://github.com/pypa/warehouse/issues/423 [4] https://github.com/pypa/warehouse/issues/956 [5] https://github.com/pypa/warehouse/issues/1228 [6] https://github.com/pypa/warehouse/issues/582 [7] https://github.com/pypa/warehouse/issues/1322 [8] https://mail.python.org/pipermail/distutils-sig/2017-December/031826.html [9] https://github.com/pypa/warehouse/issues?q=is%3Aissue+is%3Aopen+label%3A%22… P.S. Thanks to Laura Hampton for help with prep, meeting-running, and followup. -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

(no subject)
by Suchitrarani Ojha 20 Dec '17

20 Dec '17

Sir, I have recently downloaded python 3.6 version and I am very new to python.I tried to open excel files in python using openpyxl but its shows"Module not found error".So I tried to install it using pip install openpyxl and it was showing Syntax error:'Invalid syntax'.Can you help me in resolving the issue. Regards suchitra

2 1

Policy for deleting packages, releases, and artifacts
by Jimmy Jia 19 Dec '17

19 Dec '17

Hi everyone, This is moved over from https://github.com/pypa/packaging-problems/issues/112. Currently, PyPI has no limitations around deleting packages, releases, or artifacts. This can be problematic for users, as user builds can break in an unsolicited manner if a dependency is removed from PyPI. In the Node ecosystem, a similar lack of limitations there caused significant problems about a year and a half ago, when a widely-used package was deleted following a dispute: http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm, http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy. At the time, the scope of the impact was characterized as "breaking the world". To resolve this, npm adopted a policy where package deletions (there's no distinction between a release and an artifact there) could only be done for the first 24 hours after a release was published. Deletions after the 24 hour mark require contacting npm support, and are contingent on the absence of dependents for the deleted release. Of course, npm is a venture-backed for-profit enterprise that has a paid support team – that's not the case here, so the "support" half of the solution above doesn't make sense here. However, as a starting point, it might still be a good idea to restrict package deletion after that 24-hour window. At least, in the examples given in the links above, the deletion of old packages is strictly a nice-to-have for the package maintainers, balanced against potentially breaking impacts for users. Thanks in advance for any feedback. Thanks, Jimmy Jia

1 1

Re: [Distutils] PEP 345, 566, 508 version specifiers and OR clauses
by Nick Coghlan 16 Dec '17

16 Dec '17

On 14 Dec. 2017 1:55 pm, "Donald Stufft" <donald(a)stufft.io> wrote: Overall I’m +1, not sure if it would be a new PEP or an amendment to the current PEPs but I think it’s a good idea. It would be a new PEP bumping the metadata version to 1.4 (that will be less tedious than it used to be though, since it will only have to cover the enhancement, not duplicate all the old field descriptions). Somwehat related though, we should probably have a summary table in PyPUG mapping metadata versions to minimum required tooling versions & their respective release dates. That way folks can make informed decisions about their installability choices before deciding which version to use in their published metadata. Cheers, Nick. _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

1 0

Or in version spec...
by Chris Barker - NOAA Federal 14 Dec '17

14 Dec '17

Sorry to lose the thread — lousy iPhone mail app... Conda supports or in its meta.yaml format: https://conda.io/docs/user-guide/tasks/build-packages/package-spec.html#bui… Maybe look to that for prior art? And it would be mildly less confusing to have consistency between the systems. -Chris Sent from my iPhone

2 1

PEP 345, 566, 508 version specifiers and OR clauses
by Barry Warsaw 14 Dec '17

14 Dec '17

I'm about to release a new version of importlib_resources, so I want to get my flit.ini's require-python clause right. We support Python 2.7, and 3.4 and beyond. This makes me sad: requires-python = '>=2.7,!=3.0,!=3.1,!=3.2,!=3.3' Of course, I'd like to write this like: requires-python = '(>=2.7 and <3) or >= 3.4' I understand that OR clauses aren't supported under any syntax currently, but as PEPs 566 and 508 are still open/active, wouldn't it be reasonable to support something like this explicitly? It seems like wanting to support 2.7 and some versions of Python 3 (but not all) is a fairly common need. Cheers, -Barry

5 7

RFC: PEP 566 - Metadata for Python Software Packages 1.3
by Dustin Ingram 11 Dec '17

11 Dec '17

Hi all, I'd appreciate your feedback on the following Metadata 1.3 PEP. The goal here is not to provide a full specification for all fields as in previous PEPs, but to: * Motivate and describe the addition of the new fields or changes to existing fields; * Point to the Core Metadata Specifications reference document as the canonical source for the specification. Previous discussions: * https://mail.python.org/pipermail/distutils-sig/2017-September/031465.html * https://github.com/python/peps/issues/388 Thanks, D. PEP: 566 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Dustin Ingram <di(a)di.codes> Discussions-To: distutils-sig <distutils-sig at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 1-Dec-2017 Python-Version: 3.x Post-History: Replaces: 345 Abstract ======== This PEP describes the changes between versions 1.2 and 1.3 of the core metadata specification for Python packages. Version 1.2 is specified in PEP 345. It also changes to the canonical source for field specifications to the `Core Metadata Specification`_ reference document, which includes specifics of the field names, and their semantics and usage. Fields ====== The canonical source for the names and semantics of each of the supported metadata fields is the `Core Metadata Specification`_ document. Fields marked with "(Multiple use)" may be specified multiple times in a single PKG-INFO file. Other fields may only occur once in a PKG-INFO file. Fields marked with "(optional)" are not required to appear in a valid PKG-INFO file; all other fields must be present. New in Version 1.3 ------------------ Description-Content-Type (optional) ::::::::::::::::::::::::::::::::::: A string stating the markup syntax (if any) used in the distribution's description, so that tools can intelligently render the description. Historically, tools like PyPI assume that a package's description is formatted in `reStructuredText (reST) <http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html>`_, and fall back on plain text if the description is not valid reST. The introduction of this field allows PyPI to support additional types of markup syntax, and not need to make this assumption. The full specification for this field is defined in the `Core Metadata Specification`_. Provides-Extra (optional, multiple use) ::::::::::::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be a valid Python identifier. May be used to make a dependency conditional on whether the optional feature has been requested. This introduction of this field allows packge installation tools (such as ``pip``) to determine which extras are provided by a given package, and so that package publication tools (such as ``twine``) can check for issues with environment markers which use extras. The full specification for this field is defined in the `Core Metadata Specification`_. Changed in Version 1.3 ---------------------- Name :::: The specification for the format of this field is now identical to the distribution name specification defined in PEP 508. Version Specifiers ================== Version numbering requirements and the semantics for specifying comparisons between versions are defined in PEP 440. Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. The environment marker format used to declare such a condition is defined in the environment markers section of PEP 508. JSON-compatible Metadata ======================== It may be necessary to store metadata in a data structure which does not allow for multiple repeated keys, such as JSON. The canonical method to transform metadata fields into such a data structure is as follows: #. The original key-value format should be read with ``email.parser.HeaderParser``; #. All transformed keys should be reduced to lower case, but otherwise should retain all other characters; #. The transformed value for any field marked with "(Multiple-use") should be a single list containing all the original values for the given key; #. The ``Keywords`` field should be converted to a list by splitting the original value on whitespace characters; #. The result should be stored as a string-keyed dictionary. Summary of Differences From PEP 345 =================================== * Metadata-Version is now 1.3. * Fields are now specified via the `Core Metadata Specification`_. * Added two new fields: ``Description-Content-Type`` and ``Provides-Extra`` * Acceptable values for the ``Name`` field are now specified as per PEP 508. * Added canonical method of transformation into JSON-compatible data structure. References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. .. _`Core Metadata Specification`: https://packaging.python.org/specifications/core-metadata/ Copyright ========= This document has been placed in the public domain. Acknowledgements ================ Thanks to Nick Colgan for contributing to this PEP. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 80 End:

8 13

Multiple package authors
by Barry Warsaw 07 Dec '17

07 Dec '17

I think I implicitly knew this, but as I've just released a package (to be announced soon) that actually has multiple authors, I found out first hand that PyPI rejects uploads where the author-email field isn't a completely valid email address, and that there is no support for multiple author emails. As it turns out, you can kludge this into your pyproject.toml or setup.py file. flit for example separates multiple emails with a newline, but you could also separate them with commas. You don't notice the problem until PyPI rejects the upload (with a 400 IIRC). I filed this issue with flit: https://github.com/takluyver/flit/issues/153 It looks like Thomas agrees that at least flit will eventually validate its fields so you error early. It was a bit of a PITA to do my upload because I didn't notice the problem until after I'd tagged the repo. Multiple package authors doesn't seem like that fringe of a use case; are there any plans, documents, PEPs, musings, grumbles about supporting multiple package authors explicitly? Cheers, -Barry

2 2

Outstanding questions for PEP 541: Package Index Name Retention
by Sumana Harihareswara 07 Dec '17

07 Dec '17

While getting up to speed on Warehouse I saw how many package transfer requests are waiting for PEP 541[0] to be accepted, so I thought it might be helpful to round up what I see as the outstanding questions. 1) Usage criteria for abandoned projects. > The tricky part there is that "being used" is a tough concept to > define. Over what time period? What amount of downloading counts as > "used"?-- Chris Rose[1] Perhaps a month? suggests Matthias Bussonnier.[2] The ensuing discussion includes thoughts on locking old versions of the project[3]; as I see it, that's a potential feature request for Warehouse, but not something to build into this PEP. (Similarly, Nick Timkovich's idea of "salting" hot-button names on PyPI so it isn't possible to register projects like "android"[4] is a feature idea I like but I think this PEP does not need to wait for it.) 2) A few copyedits from Chris Barker.[5] 3) "How would I, for example, start the process of flagging a project as abandoned?" -- Nick Timkovich[6] It seems to me that the PEP's wording in "Removal of an abandoned project" says we'll allow transfer of abandoned projects but will not remove them only for being abandoned. However, the PEP currently doesn't include a "where to file a ticket" line in that section (as it does in "Invalid projects"). Also, if there's some other reason we should be willing to remove abandoned projects even without a transfer, e.g., the project has a critical security flaw, we should say that somewhere in this PEP or in a different policy document. There is one item Łukasz mentioned this in the "I decided to not otherwise touch on:" list regarding the revision on January 14th[7] that I think deserves one more chance for discussion. :) A few people brought up making the reachability criteria and instructions crisper. > Regarding reachability: contact attempts should also include the > relevant project's issue tracker if attempts at private contact have > failed. > > This step is important as it allows a project's *user* community to > respond, even if the person that actually pushes the buttons to > upload new releases to PyPI is out of contact for some reason. -- Nick Coghlan[8] Nick Timkovich also suggested some specific instructions[9] that would help set expectations to, among other things, reassure maintainers about the length of offline vacations they can take without worrying about having their packages usurped. :) I'm totally fine with folks saying to me: no, these are all addressed, or not important enough to slow adoption of this PEP. In which case, yay, I hope Donald can accept it and we can start processing the backlog. [0] https://www.python.org/dev/peps/pep-0541/ [1] https://mail.python.org/pipermail/distutils-sig/2017-January/030017.html [2] https://mail.python.org/pipermail/distutils-sig/2017-January/030020.html [3] https://mail.python.org/pipermail/distutils-sig/2017-January/030034.html [4] https://mail.python.org/pipermail/distutils-sig/2017-January/030006.html [5] https://mail.python.org/pipermail/distutils-sig/2017-September/031517.html [6] https://mail.python.org/pipermail/distutils-sig/2017-January/030005.html [7] https://mail.python.org/pipermail/distutils-sig/2017-January/030009.html [8] https://mail.python.org/pipermail/distutils-sig/2017-January/030008.html [9] https://mail.python.org/pipermail/distutils-sig/2017-January/030005.html -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

3 2

Update on Warehouse kickoff and milestones
by Sumana Harihareswara 07 Dec '17

07 Dec '17

As Donald and Nick mentioned last month,[0] several folks are -- thanks to Mozilla Open Source Support -- working on deploying Warehouse and making PyPI more sustainable. I'm the main project manager on this effort and am putting notes on wiki.python.org as we go.[1] Ernest W. Durbin III gave the closing talk at North Bay Python a few days ago: "Running Vintage Software: PyPI's Aging Codebase."[2] As he mentioned (video[3]), he's one of the people who'll be working on this project, funded by MOSS, to get Warehouse across the finish line! The full cast is: * Donald Stufft, backend * Dustin Ingram, backend * Ernest W. Durbin III, backend * Laura Hampton, project management, testing, documentation * Nicole Harris, frontend and design * Sumana Harihareswara, project management, product management, testing, documentation None of us are working 40 hours/week on this but the six people I listed above are dedicating committed time to the work. And we're grateful for help from Mark Mangoba, Eric Holscher, and of course several other folks who are involved in the PSF, PyPA, or MOSS. We had a kickoff call on Monday[4] and today a few of us spoke in #pypa-dev and fleshed out our milestones[5] a bit. This may still change, but right now, the sequence is: 1. Maintainer Minimum Viable Product (for package owners) -- ask some maintainers to use, test, and report back 2. End User MVP -- ask some Python users who aren't package owners to use Warehouse, test, and report back 3. Publicize beta more broadly (invite more testing, redirect some `pip install` traffic) 4. Launch/redirect `pip` and browsers to Warehouse 5. Shut down legacy PyPI 6. Cool feature ideas that are not on critical path We'll be putting issues about infrastructure work (e.g., Kubernetes) in https://github.com/python/pypi-infra . We've started filing, updating, and arranging issues but that isn't yet complete enough for me to even start making schedule estimates. I aim to have a clearer picture next week. (I won't spam this list with updates when there's nothing new to say, but at least as we start I figure you might want news a few times a month to learn how things are shaping up; please let me know privately if my estimation there is way off.) [0] https://mail.python.org/pipermail/distutils-sig/2017-November/031782.html [1] https://wiki.python.org/psf/PackagingWG [2] https://2017.northbaypython.org/schedule/presentation/5/ [3] https://www.youtube.com/watch?v=xyPd4WOR_ng [4] https://wiki.python.org/psf/PackagingWG/2017-12-04-Warehouse [5] https://github.com/pypa/warehouse/milestones -- Sumana Harihareswara Changeset Consulting https://changeset.nyc

1 0

PyPi/Warehouse & User Agent Strings
by Simon Kennedy 06 Dec '17

06 Dec '17

A few quick questions about pip/pypi and user agent strings. Does PyPi/Warehouse do anything with the user agent string or does it just go into the BigQuery system? What elements of the user agent string are used? Is the pip source code the reference document or is there a separate specification? --- Simon

2 1

Adding local options to commands
by Simon Kennedy 06 Dec '17

06 Dec '17

Is there a recognized way to add options to distutils commands that are only for my local environment? For example I stick all my build artifacts in a directory, that is not in the source directory, and that will not be on the machine of someone who checks out my code. So I don't want to add it into setup.cfg where it will get checked in. For example I set bdist-base = ../../build/astral/master/tmp/bdist that other people won't want it to be written to. Currently I've created a setup-dev.cfg file and added code to my local distutils to also look in this file for options.

1 0

Re: [Distutils] RFC: PEP 566 - Metadata for Python Software Packages 1.3
by Nathaniel Smith 06 Dec '17

06 Dec '17

On Dec 5, 2017 08:42, "Dustin Ingram" <dustin.ingram(a)gmail.com> wrote: Provides-Extra (optional, multiple use) ::::::::::::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be a valid Python identifier. May be used to make a dependency conditional on whether the optional feature has been requested. This introduction of this field allows packge installation tools (such as ``pip``) to determine which extras are provided by a given package, and so that package publication tools (such as ``twine``) can check for issues with environment markers which use extras. I haven't followed this so sorry if this is an annoying comment, but having read this description I still don't really understand what Provides-Extra is doing. Don't packages already include extra information? What problem motivated this? -n

3 2

Caching entry points - performance
by Thomas Kluyver 30 Nov '17

30 Nov '17

To pick up the caching discussion again, I've started to experiment with a couple of different caching techniques. The headline results: a cold-start scan of entry points goes from about 4.5 seconds with no caching, to 0.45 seconds with caching. There's only a small difference (for me) between having a cache file per sys.path directory and a single per-user cache file. I've posted lots more details here: https://github.com/takluyver/entrypoints/issues/16 I keep my sys.path quite short, and the difference between the two approaches presumably increases the more sys.path entries you have. If anyone has time to test the different techniques, I'd be interested to know what your results look like. I've put (Linux) instructions on the same issue. If you'd like to try with a different format or caching mechanism, feel free to add some scripts and see how they compare. I haven't yet tried to integrate either technique into the entrypoints module. Thomas

2 1

Funding Warehouse Development/Deployment
by Donald Stufft 29 Nov '17

29 Nov '17

As I’m sure most or everyone on this list is aware, the work to replace PyPI with Warehouse has been underway for quite awhile. I’m happy to say that we’ve now got funding from a MOSS grant that should finally help us get this across the finish line and deployed for “real”. You can read more about this at https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html <https://pyfound.blogspot.com/2017/11/the-psf-awarded-moss-grant-pypi.html> but the tl;dr is basically we’ve been awarded 170k with the explicit goal of that money being to launch Warehouse and shutdown legacy PyPI.

2 1

buildout & extra-path
by Alessandro Dentella 28 Nov '17

28 Nov '17

Hi, I realized that extra-path is not always honored in buildout. I have a Django application that uses django-recipe with extra-path. I also want a pure ipython program that share the same sys.path. but if I write: [ipython] recipe = zc.recipe.egg eggs = ${buildout:eggs} extra-paths = ${buildout:directory}/apps I get a script that doesn't have the extra-path. I explicitly have to add: interpreter = ipython-dj is this the desired behaviour? sandro -- Sandro Dentella *:-) http://trepalchi.it Il portale degli artisti http://www.reteisi.org Soluzioni libere per le scuole http://sqlkit.argolinux.org SQLkit home page - PyGTK/python/sqlalchemy

2 1

pythonhosted.org doc upload no longer works
by Giampaolo Rodola' 26 Nov '17

26 Nov '17

I know it was deprecated long ago in favor of readthedocs but I kept postponing it and my doc is still hosted on https://pythonhosted.org/psutil/ . I just took a look at: https://pypi.python.org/pypi?:action=pkg_edit&name=psutil ...and the form to upload the doc in zip format is gone. The only available option is the "destroy documentation" button. I would like to at least upload a static HTML page redirecting to the new doc which I will soon move on readthedocs. Is that possible? Thanks in advance. -- Giampaolo - http://grodola.blogspot.com

6 18

PEP 517 reference consumer
by Thomas Kluyver 16 Nov '17

16 Nov '17

I have found a bit more time to work on my PEP 517 consumer API: https://github.com/takluyver/pep517 It provides two main interfaces: - 'pep517.wrappers' is a lower-level interface to call the backend hooks in a subprocess. - 'pep517.envbuild' is a higher-level interface which creates a temporary environment, uses pip to install build dependencies, and builds a wheel or an sdist. I hope that some frontend tools will be able to use these interfaces directly, but I also intend it as a reference consumer which backend authors can use to test their implementation. As such, I'd like to propose moving the project into the control of the Python Packagaing Authority - i.e. transferring the Github repo and giving other people upload rights on PyPI. In other news, Flit 0.12 is out, providing a PEP 517 backend. Though come to think of it, I may have forgotten to use the tar pax format. Thomas

3 9

PyPI Limits of Wheel Size
by Phil Thompson 13 Nov '17

13 Nov '17

I am hitting a limit on the size of a wheel when uploading to PyPI - the wheel is about 100M. When this has happened in the past I have contacted Donald who has been kind enough to increase the limit (I believe it is a per-project limit). Currently where should I make this request? Thanks, Phil

2 1

Fwd: Re: [Python-ideas] Add processor generation to wheel metadata
by Ivan Pozdeev 07 Nov '17

07 Nov '17

Forwarding here as suggested. -------- Forwarded Message -------- Subject: Re: [Python-ideas] Add processor generation to wheel metadata Date: Mon, 30 Oct 2017 23:02:08 -0700 From: Nathaniel Smith <njs(a)pobox.com> To: Ivan Pozdeev <vano(a)mail.mipt.ru> CC: python-ideas(a)python.org <python-ideas(a)python.org> On Mon, Oct 30, 2017 at 5:45 AM, Ivan Pozdeev via Python-ideas <python-ideas(a)python.org> wrote: > Generally, packages are compiled for the same processor generation as the > corresponding Python. > But not always -- e.g. NumPy opted for SSE2 even for Py2 to work around some > compiler bug > (https://github.com/numpy/numpy/issues/6428). > I was bitten by that at an old machine once and found out that there is no > way for `pip' to have checked for that. > Besides, performance-oriented packages like the one mentioned could probably > benefit from newer instructions. You should probably resend this to distutils-sig instead of python-ideas -- that's where discussions about python packaging happen. (Python-ideas is more for discussions about the language itself.) -n -- Nathaniel J. Smith -- https://vorpus.org

3 5

Where to get help changing ownership of Pypi package?
by Stuart Axon 05 Nov '17

05 Nov '17

Hi, I realise this is the wrong place, but does anyone know who could help resolve ownership of the Pypi package PyCairo https://sourceforge.net/p/pypi/support-requests/728/ There are also a bunch of similar tickets, I guess nobody checks these any more ? :( S++

3 2

Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
by Nathaniel Smith 31 Oct '17

31 Oct '17

On Oct 27, 2017 11:49, "Alex Domoradov" <alex.hha(a)gmail.com> wrote: RUN pip install --upgrade pip Try upgrading setuptools here too. -n

6 7

What's the use case of testpypi?
by Toshio Kuratomi 31 Oct '17

31 Oct '17

When we locked down pypi to prevent uploading an sdist to overwrite a previous I remember that some people wanted a brief window to check for brown paper bag issues and be able to upload a new tarball in that window if needed. IIRC, those people were told to use testpypi for that sort of thing. Upload potential tarball to testpypi. If it works, go ahead and rerun to upload to the real pypi. If it doesn't, fix it and try again. This past week I decided to try out that workflow for a project that I'm managing the releases for and ran into a snag. testpypi has the same behaviour as production pypi wherein you can only upload a given sdist once and afterwards it's no longer allowed. For the use case above this is problematic. It essentially changes the idea of "test your release on testpypi before making a release" into "You can have to chances to get it right if you use testpypi" which, although better than uploading directly to pypi, still leaves a lot of room for error (let's face it: I know that if I'm stuck releasing late at night due to time constraints and make a mistake, chances are better than normal that my fix won't be the perfection that my ordinary code is and could have other showstopper bugs that I'd want my testing to catch as well ;-) Is this something that we could change for testpypi? It could be implemented in many ways: straight overwrite, being able to destroy a version so that it seems to never have existed, or being able to destroy and recreate a package so that it has no uploaded sdists recorded. On the other side of the usefulness of enabling the testing use case above, such a change would be a difference between testpypi and production pypi meaning that it would no longer be testing exactly the same functionality as will be deployed in production. I'm not sure if that's a more important consideration or not. I figured that unless I asked, I would never know the answer :-) Thanks, -Toshio

6 8

[proposal] shared distribution installations
by RonnyPfannschmidt 31 Oct '17

31 Oct '17

Hi everyone, since a while now various details of installing python packages in virtualenvs caused me grief a) typically each tox folder in a project is massive, and has a lot of duplicate files, recreating them, managing and iterating them takes quite a while b) for nicely separated deployments, each virtualenv for an application takes a few hundred megabytes - that quickly can saturate disk space even if a reasonable amount was reserved c) installation and recreation of virtualenvs with the same set of packages takes quite a while (even with pip caches this is slow, and there is no good reason to avoid making it completely instantaneous) in order to elevate those issues i would like to propose a new installation layout, where instead of storing each distribution in every python all distributions would share a storage, and each individual environment would only have references to the packages that where "installed/activated" for them this would massively reduce time required to create the contents of the environments and also the space required since blindly expanding sys.path would lead to similar performance issues as where seen with setuptools/buildout multi-version installs, this mechanism would also need a element on sys.meta_path that handles inexpensive dispatch to the toplevels and metadata files of each packages (off hand i would assume linear walking of hundreds of entries simply isn't that effective) however there would be need for some experimentation to see what tradeoff is sensible there I hope this mail will spark enough discussion to enable the creation of a PEP and a prototype. Best, Ronny

8 8

Wheel 1.0 roadmap
by Alex Grönholm 30 Oct '17

30 Oct '17

I am planning for a 1.0.0 release of the "wheel" library. I would like to start using semver from this point onwards, which in the case of wheel means that its command line interface should be well defined and remain backwards compatible. As part of this effort, I've rewritten the documentation (currently in the "docs-update" branch on Github) to conform to the PyPA guidelines. Wheel also had some generated API documentation on ReadTheDocs, but as discussed privately with Daniel Holth and Nick Coghlan, wheel should not have a public API going forward so I've deleted that documentation. I've also taken a hard look at wheel's features and would like to remove those which I consider to be either useless or harmful. I've added these tasks as issues on Github. All the issues that I'd like to get resolved by 1.0.0 have been tagged with the proper milestone marker here: https://github.com/pypa/wheel/milestone/1 Feedback is very welcome! ps. Daniel, if you're reading this, would you mind giving the new docs a once-over? Also, if you can suggest where to put the "story" page, I'll link it back to the main index file.

5 4

ANN: distlib 0.2.6 released on PyPI
by Vinay Sajip 28 Oct '17

28 Oct '17

I've just released version 0.2.6 of distlib on PyPI [1]. For newcomers,distlib is a library of packaging functionality which is intended to beusable as the basis for third-party packaging tools. The main changes in this release are as follows: * Fixed #99: Updated to handle a case where sys.getfilesystemencoding() returns None. * Fixed #97: Eliminated a crash in EggInfoDistribution.list_distinfo_files() which was caused by trying to open a non-existent file. * Fixed #96: SimpleScrapingLocator no longer fails prematurely when scraping links due to invalid versions. * Improved error messages issued when interpreting markers. * Improved the shebangs written into installed scripts when the interpreter path is very long or contains spaces (to cater for a limitation in shebang line parsing on Linux). * Updated launcher binaries. A more detailed change log is available at [2]. Please try it out, and if you find any problems or have any suggestions forimprovements, please give some feedback using the issue tracker! [3] Regards, Vinay Sajip [1] https://pypi.python.org/pypi/distlib/0.2.6[2] https://goo.gl/M3kQzR[3] https://bitbucket.org/pypa/distlib/issues/new

1 0

Disabling non HTTPS access to APIs on PyPI
by Donald Stufft 28 Oct '17

28 Oct '17

Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/`` <http://pypi.python.org/simple/%60%60>, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening. Today I’ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning. The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.

5 4

Entry points: specifying and caching
by Thomas Kluyver 27 Oct '17

27 Oct '17

We're increasingly using entry points in Jupyter to help integrate third-party components. This brings up a couple of things that I'd like to do: 1. Specification As far as I know, there's no document describing the details of entry points; it's a de-facto standard established by setuptools. It seems to work quite well, but it's worth writing down what is unofficially standardised. I would like to see a document on https://packaging.python.org/specifications/ saying: - Where build tools should put entry points in wheels - Where entry points live in installed distributions - The file format (including allowed characters, case sensitivity...) I guess I'm volunteering to write this, although if someone else wants to, don't let me stop you. ;-) I'd also be happy to hear that I'm wrong, that this specification already exists somewhere. If it does, can we add a link from https://packaging.python.org/specifications/ ? 2. Caching "There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors" I know that caching is going to make things more complex, but at present a scan of available entry points requires a stat() for every installed package, plus open()+read()+parse for every installed package that provides entry points. This doesn't scale well, especially on spinning hard drives. By eliminating a call to pygments which caused an entry points scan, we cut the cold-start time of IPython almost in half on one HDD system (11s -> 6s; PR 10859). As packaging improves, the trend is to break functionality into more, smaller packages, which is only going to make this worse (though I hope we never end up with a left-pad package ;-). Caching could allow entry points to be used in places where the current performance penalty is too much. I envisage a cache working something like this: - Each directory on sys.path can have a cache file, e.g. 'entry-points.json' - I suggest JSON because Python can parse it efficiently, and it's not intended to be directly edited by humans. Other options? SQLite? Does someone want to do performance comparisons? - There is a command to scan all packages in a directory and build the cache file - After an install tool (e.g. pip) has added/removed packages from a directory, it should call that command to rebuild the cache. - A second command goes through all directories on sys.path and rebuilds their cache files - this lets the user rebuild caches if something has gone wrong. - Applications looking for entry points can choose from a range of behaviours depending on how important accuracy and performance are. E.g. ignore all caches, only use caches, use caches for directories where they exist, or try caches first and then scan packages if a key is missing. In the best case, when the caches exist and you trust them, loading them would cost one set of filesystem operations per sys.path entry, rather than per package. Thanks, Thomas

12 85

Disabling non HTTPS access to APIs on PyPI
by Alex Domoradov 27 Oct '17

27 Oct '17

> This will probably (unfortunately) break some things for some people, > which is worrying. But it is the right thing to do and good advice in > general. Did I miss something, but I can't install some package. I use docker and get the following $ cat Dockerfile FROM ubuntu:16.04 RUN apt-get update RUN apt-get install -y python-pip RUN pip install --upgrade pip $ docker run -it --rm pip bash root@a633a43c5df6:/# pip -V pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7) root@a633a43c5df6:/# pip install google-appengine Collecting google-appengine Downloading google-appengine-1.5.1.tar.gz (897kB) 100% |################################| 901kB 745kB/s Complete output from command python setup.py egg_info: Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-hd5GWR/google-appengine/setup.py", line 3, in <module> ez_setup.use_setuptools() File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line 145, in use_setuptools return _do_download(version, download_base, to_dir, download_delay) File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line 124, in _do_download to_dir, download_delay) File "/usr/local/lib/python2.7/dist-packages/ez_setup.py", line 193, in download_setuptools src = urlopen(url) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 435, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 548, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 473, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) * urllib2.HTTPError: HTTP Error 403: SSL is required* ---------------------------------------- Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-hd5GWR/google-appengine/ Do I need to make some additional upgrades? I just thought that pip is the only one tool that I have to upgrade

1 0

Fwd: [Python-Dev] PEP 561: Distributing and Packaging Type Information
by Nick Coghlan 27 Oct '17

27 Oct '17

Hi folks, Forwarding this PEP from python-dev, since there's a small piece of it that falls into distutils-sig's area of responsibility: claiming the "Typing" key in the distribution metadata to allow projects to state whether or not they natively provide type annotations. This seems like a reasonable addition to me, and it doesn't have any specific implications for packaging tools beyond the existing convention of "ignore-but-preserve any metadata keys you don't understand". Cheers, Nick. ---------- Forwarded message ---------- From: Ethan Smith <ethan(a)ethanhs.me> Date: 27 October 2017 at 08:42 Subject: [Python-Dev] PEP 561: Distributing and Packaging Type Information To: python-dev(a)python.org Hello all, I have completed an implementation for PEP 561, and believe it is time to share the PEP and implementation with python-dev Python-ideas threads: * PEP 561: Distributing and Packaging Type Information * PEP 561 v2 - Packaging Static Type Information * PEP 561: Distributing Type Information V3 The live version is here: https://www.python.org/dev/peps/pep-0561/ As always, duplicated below. Ethan Smith --------------------------------------------------- PEP: 561 Title: Distributing and Packaging Type Information Author: Ethan Smith <ethan(a)ethanhs.me> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 09-Sep-2017 Python-Version: 3.7 Post-History: Abstract ======== PEP 484 introduced type hinting to Python, with goals of making typing gradual and easy to adopt. Currently, typing information must be distributed manually. This PEP provides a standardized means to package and distribute type information and an ordering for type checkers to resolve modules and collect this information for type checking using existing packaging architecture. Rationale ========= Currently, package authors wish to distribute code that has inline type information. However, there is no standard method to distribute packages with inline type annotations or syntax that can simultaneously be used at runtime and in type checking. Additionally, if one wished to ship typing information privately the only method would be via setting ``MYPYPATH`` or the equivalent to manually point to stubs. If the package can be released publicly, it can be added to typeshed [1]_. However, this does not scale and becomes a burden on the maintainers of typeshed. Additionally, it ties bugfixes to releases of the tool using typeshed. PEP 484 has a brief section on distributing typing information. In this section [2]_ the PEP recommends using ``shared/typehints/pythonX.Y/`` for shipping stub files. However, manually adding a path to stub files for each third party library does not scale. The simplest approach people have taken is to add ``site-packages`` to their ``MYPYPATH``, but this causes type checkers to fail on packages that are highly dynamic (e.g. sqlalchemy and Django). Specification ============= There are several motivations and methods of supporting typing in a package. This PEP recognizes three (3) types of packages that may be created: 1. The package maintainer would like to add type information inline. 2. The package maintainer would like to add type information via stubs. 3. A third party would like to share stub files for a package, but the maintainer does not want to include them in the source of the package. This PEP aims to support these scenarios and make them simple to add to packaging and deployment. The two major parts of this specification are the packaging specifications and the resolution order for resolving module type information. The packaging spec is based on and extends PEP 345 metadata. The type checking spec is meant to replace the ``shared/typehints/pythonX.Y/`` spec of PEP 484 [2]_. New third party stub libraries are encouraged to distribute stubs via the third party packaging proposed in this PEP in place of being added to typeshed. Typeshed will remain in use, but if maintainers are found, third party stubs in typeshed are encouraged to be split into their own package. Packaging Type Information -------------------------- In order to make packaging and distributing type information as simple and easy as possible, the distribution of type information, and typed Python code is done through existing packaging frameworks. This PEP adds a new item to the ``*.distinfo/METADATA`` file to contain metadata about a package's support for typing. The new item is optional, but must have a name of ``Typed`` and have a value of either ``inline`` or ``stubs``, if present. Metadata Examples:: Typed: inline Typed: stubs Stub Only Packages '''''''''''''''''' For package maintainers wishing to ship stub files containing all of their type information, it is prefered that the ``*.pyi`` stubs are alongside the corresponding ``*.py`` files. However, the stubs may be put in a sub-folder of the Python sources, with the same name the ``*.py`` files are in. For example, the ``flyingcircus`` package would have its stubs in the folder ``flyingcircus/flyingcircus/``. This path is chosen so that if stubs are not found in ``flyingcircus/`` the type checker may treat the subdirectory as a normal package. The normal resolution order of checking ``*.pyi`` before ``*.py`` will be maintained. Third Party Stub Packages ''''''''''''''''''''''''' Third parties seeking to distribute stub files are encouraged to contact the maintainer of the package about distribution alongside the package. If the maintainer does not wish to maintain or package stub files or type information inline, then a "third party stub package" should be created. The structure is similar, but slightly different from that of stub only packages. If the stubs are for the library ``flyingcircus`` then the package should be named ``flyingcircus-stubs`` and the stub files should be put in a sub-directory named ``flyingcircus``. This allows the stubs to be checked as if they were in a regular package. In addition, the third party stub package should indicate which version(s) of the runtime package are supported by indicating the runtime package's version(s) through the normal dependency data. For example, if there was a stub package ``flyingcircus-stubs``, it can indicate the versions of the runtime ``flyingcircus`` package supported through ``install_requires`` in distutils based tools, or the equivalent in other packaging tools. Type Checker Module Resolution Order ------------------------------------ The following is the order that type checkers supporting this PEP should resolve modules containing type information: 1. User code - the files the type checker is running on. 2. Stubs or Python source manually put in the beginning of the path. Type checkers should provide this to allow the user complete control of which stubs to use, and patch broken stubs/inline types from packages. 3. Third party stub packages - these packages can supersede the installed untyped packages. They can be found at ``pkg-stubs`` for package ``pkg``, however it is encouraged to check the package's metadata using packaging query APIs such as ``pkg_resources`` to assure that the package is meant for type checking, and is compatible with the installed version. 4. Inline packages - finally, if there is nothing overriding the installed package, and it opts into type checking. 5. Typeshed (if used) - Provides the stdlib types and several third party libraries Type checkers that check a different Python version than the version they run on must find the type information in the ``site-packages``/``dist-packages`` of that Python version. This can be queried e.g. ``pythonX.Y -c 'import site; print(site.getsitepackages())'``. It is also recommended that the type checker allow for the user to point to a particular Python binary, in case it is not in the path. To check if a package has opted into type checking, type checkers are recommended to use the ``pkg_resources`` module to query the package metadata. If the ``typed`` package metadata has ``None`` as its value, the package has not opted into type checking, and the type checker should skip that package. Implementation ============== A CPython branch with a modified distutils supporting the ``typed`` setup keyword lives here: [impl]_. In addition, a sample package with inline types is available [typed_pkg]_, as well as a sample package [pkg_checker]_ which reads the metadata of installed packages and reports on their status as either not typed, inline typed, or a stub package. Acknowledgements ================ This PEP would not have been possible without the ideas, feedback, and support of Ivan Levkivskyi, Jelle Zijlstra, Nick Coghlan, Daniel F Moisset, and Guido van Rossum. Version History =============== * 2017-10-26 * Added implementation references. * Added acknowledgements and version history. * 2017-10-06 * Rewritten to use .distinfo/METADATA over a distutils specific command. * Clarify versioning of third party stub packages. * 2017-09-11 * Added information about current solutions and typeshed. * Clarify rationale. References ========== .. [1] Typeshed (https://github.com/python/typeshed) .. [2] PEP 484, Storing and Distributing Stub Files (https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-fil…) .. [impl] CPython sample implementation (https://github.com/ethanhs/cpython/tree/typeddist) .. [typed_pkg] Sample typed package (https://github.com/ethanhs/sample-typed-package) .. [pkg_checker] Sample package checker (https://github.com/ethanhs/check_typedpkg) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ ncoghlan%40gmail.com -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Distutils] The software update framework
by Thomas Güttler 26 Oct '17

26 Oct '17

1 0

How do native namespaces work? (And do they play well with setuptools?)
by Thomas Nyberg 25 Oct '17

25 Oct '17

Hello, I previously posted this to python-list as well as reddit: https://mail.python.org/pipermail/python-list/2017-October/726935.html https://www.reddit.com/r/learnpython/comments/78emwr/how_to_native_namespac… So far (as pointed out in the reddit link I posted) it seems like maybe I can't use native namespaces in conjunction with setuptools and must instead use pip. Is this correct? Here is my original message which contains all the details: I'm trying to understand native namespaces. I'm currently using python 3.5 as packaged in debian 9. I've been following the instructions here: https://packaging.python.org/guides/packaging-namespace-packages/#native-na… Those instructions link to the following example: https://github.com/pypa/sample-namespace-packages/tree/master/native I presume I'm doing something silly, but I can't get that to work. I've tried creating a virtual environment as: -------------------------------------------------- $ python3 -m venv venv $ source venv/bin/activate -------------------------------------------------- _Note_: The virtual environment is active, but I'm stripping the (venv) off the prompt below to shorten the lines. And then I've gone into those repo folders and run their setup.py files (e.g. for namespace a): -------------------------------------------------- $ cd sample-namespace-packages/native/pkg_a/ $ python3 setup.py install -------------------------------------------------- Then if I try to run their sample verification file, things fail: -------------------------------------------------- $ cat sample-namespace-packages/verify_packages.py from example_pkg import a from example_pkg import b print(a.name) print(a.__path__) print(b.name) print(b.__path__) $ python3 sample-namespace-packages/verify_packages.py Traceback (most recent call last): File "sample-namespace-packages/verify_packages.py", line 1, in <module> from example_pkg import a ImportError: No module named 'example_pkg' -------------------------------------------------- Things seem to be installing: -------------------------------------------------- $ ls venv/lib/python3.5/site-packages/ easy-install.pth pkg_resources easy_install.py pkg_resources-0.0.0.dist-info example_pkg_a-1-py3.5.egg __pycache__ example_pkg_b-1-py3.5.egg setuptools pip setuptools-32.3.1.dist-info pip-9.0.1.dist-info -------------------------------------------------- Am I missing something totally obvious here? Does anyone here see what I'm doing wrong? Thanks for any help! Cheers, Thomas

2 3

The software update framework
by Thomas Güttler 24 Oct '17

24 Oct '17

2 1

Re PEP 508
by Michael Laing 23 Oct '17

23 Oct '17

I have a correction and a couple of possible improvements to the parsley grammar. Is there a convenient way to submit them? Thanks, ml

3 2

Documentation link on PyPI.org
by Ronald Oussoren 20 Oct '17

20 Oct '17

Hi, Is it possible to create a documentation in the section with “Project Links” on the page describing a release on PyPI.org <http://pypi.org/>? That is, can I add some metadata to an sdist or wheel that would add an entry to that list, and if so how? I’m using setuptools to create the distribution artefacts. The reason I ask is that I currently use an index.html on pythonhosted.org that redirects to the real documentation because that adds a link to the documentation when looking at the release on pypi.python.org. That doesn’t work pypi.org. Ronald

3 6

Claim/remove pypi package named sc0ns
by Bill Deegan 10 Oct '17

10 Oct '17

Greetings, There's a package named sc0ns which is an ancient packaging of a SCons release: https://pypi.python.org/pypi/sc0ns vs. https://pypi.python.org/pypi/SCons Is it possible to have the sc0ns package removed/hidden? And/or is there a way to contact that package owner to request same? Thanks, Bill Deegan SCons Project Co-Manager

2 1

Claiming Ownership of Existing Package
by xoviat 09 Oct '17

09 Oct '17

All: I would like to claim ownership of the "flint" package on pypi so that I can upload a new package with the same name. The existing project page states that it is abandoned.

1 0

PIP not recognized in python shell
by Milind Rangnekar 04 Oct '17

04 Oct '17

Hello, >From window's 10 command prompt, I can install pip and other packages: C:\>pip Usage: pip <command> [options] Commands: install Install packages. uninstall Uninstall packages. freeze Output installed packages in requirements format. list List installed packages. show Show information about installed packages. search Search PyPI for packages. wheel Build wheels from your requirements. zip DEPRECATED. Zip individual packages. unzip DEPRECATED. Unzip individual packages. help Show help for commands. General Options: -h, --help Show help. --isolated Run pip in an isolated mode, ignoring environment variables and user configuration. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show version and exit. -q, --quiet Give less output. --log <path> Path to a verbose appending log. --proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port. --retries <retries> Maximum number of retries each connection should attempt (default 5 times). --timeout <sec> Set the socket timeout (default 15 seconds). --exists-action <action> Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup. --trusted-host <hostname> Mark this host as trusted, even though it does not have valid or any HTTPS. --cert <path> Path to alternate CA bundle. --client-cert <path> Path to SSL client certificate, a single file containing the private key and the certificate in PEM format. --cache-dir <dir> Store the cache data in <dir>. --no-cache-dir Disable the cache. --disable-pip-version-check Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index. C:\>python -m pip install beautifulsoup4 You are using pip version 6.0.8, however version 9.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting beautifulsoup4 Downloading beautifulsoup4-4.6.0-py3-none-any.whl (86kB) 100% |################################| 90kB 1.5MB/s Installing collected packages: beautifulsoup4 Successfully installed beautifulsoup4-4.6.0 C:\> However, PIP is not recognized in Python 3.4.3 Shell. >>> pip install SyntaxError: invalid syntax >>> pip Traceback (most recent call last): File "<pyshell#11>", line 1, in <module> pip NameError: name 'pip' is not defined >>> python -m pip install beautifulsoup4 SyntaxError: invalid syntax Please advise. Thanks

3 2

Migrating interoperability specs to packaging.python.org
by Nick Coghlan 04 Oct '17

04 Oct '17

Some time ago, I started the process [1] of adjusting how distutils-sig uses the PEP process so that the reference specifications will live on packaging.python.org, and we use the PEP process to manage *changes* to those specifications, rather than serving as the specifications themselves (that is, adopting a process closer to the URL-centric way the Python language reference is managed, rather than using the RFCstyle PEP-number-centric model the way we do now). I never actually finished that work, and as a result, it's currently thoroughly unclear [2] that Description-Content-Type and Provides-Extra are defined at https://packaging.python.org/specifications/#core-metadata rather than in a PEP. I'm currently at the CPython core development sprint in San Francisco, and I'm thinking that finalising that migration [3] and updating the affected PEPs accordingly (most notably, PEP 345) is likely to be a good use of my time. However, I'm also wondering if it may still be worthwhile writing a metadata 1.3 PEP that does the following things: 1. Explicitly notes the addition of the two new fields 2. Describes the process change for packaging interoperability specifications 3. Defines a canonical transformation between the human-readable key:value format and a more automation friendly JSON format That PEP would then essentially be the first one to use the new process: it would supersede PEP 345 as the latest metadata specification, but it would *also* redirect readers to the relevant URL on packaging.python.org as the canonical source of the specification, rather than being the reference documentation in its own right. Cheers, Nick. [1] https://github.com/pypa/pypa.io/issues/11#issuecomment-173833061 [2] https://github.com/python/peps/issues/388 P.S. Daniel, if you're currently thinking "I proposed defining an incremental metadata 1.3 tweak years ago!", aye, you did. My subsequent additions to PEP 426 were a classic case of second-system syndrome: https://en.wikipedia.org/wiki/Second-system_effect (which we suspected long ago, hence that PEP's original deferral) Fortunately, the disciplining effect of working with a primarily volunteer contributor base has prevented my over-engineered change-for-change's-sake-rather-than-to-solve-actual-user-problems version from becoming reality ;) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 6

Reproducible builds (Sdist)
by Matthias Bussonnier 02 Oct '17

02 Oct '17

Hello there, I'm going to ask questions about Reproducible Builds, a previous thread have been started in March[1], but does not cover some of the questions I have. In particular I'm interested in the reproducible build of an _sdist_. That is to say the process of going from a given commit to the corresponding TGZ file. It is my understanding that setting SOURCE_DATE_EPOCH (SDE for short) should allow a reproducible building of an Sdist; And by reproducible I mean that the tgz itself is the same byte for byte; (the unpacked-content being the same is a weaker form I'm less interested in). Is this assumption correct? In particular I cannot seem to be able to do that without unpacking and repacking the tgz myself; because the copy_tree-taring and the gziping by default embed the current timestamp of when these functions were ran. Am I missing something ? Second; is there a convention to store the SDE value ? I don't seem to be able to find one. It is nice to have reproducible build; but if it's a pain for reproducers to find the SDE value that highly decrease the value of SDE build. Also congrats for pep 517 and thanks for everyone who participated; Thanks -- Matthias 1: https://mail.python.org/pipermail/distutils-sig/2017-March/030284.html

4 5

Extracting distutils into setuptools
by xoviat 02 Oct '17

02 Oct '17

This was a comment by @zooba (Steve Dower): > (FWIW, I think it makes *much* more sense for setuptools to fix this by simply forking all of distutils and never looking back. But since we don't live in that world yet, it went into distutils.) And here is my response: > Since you mention it, I agree with that proposal. But currently we have core developers contributing to distutils and @jaraco contributing to setuptools. @jaraco is quite competent, but I doubt that he would be able to maintain an independent fork of distutils by himself. > In short, I think your proposal is a good one, but how can we allocate manpower? (issue31595 on bugs.python.org) So what do others think of this? My sense of things is that people are open to the idea, but there isn't a plan to make it happen.

11 37

PEP 517 - is it ready?
by Thomas Kluyver 29 Sep '17

29 Sep '17

Discussion on PEP 517 seems to have settled down, and I believe that Nick said he was about ready to accept it. Is everyone involved satisfied with the current state? Or is there anything else you think should be considered before accepting it? https://www.python.org/dev/peps/pep-0517/

5 7

New wheel release (v0.30.0)
by Alex Grönholm 26 Sep '17

26 Sep '17

I have now made my first wheel release as the new maintainer. This release contains bug fixes and documentation updates. I decided to make a release now to get a fix out for some long standing issues. A proper overhaul of the documentation is coming later. The project has also been migrated from Bitbucket to Github <https://github.com/pypa/wheel>.

3 2

how to install
by Minesh Parikh 25 Sep '17

25 Sep '17

how to install setup.py file on my website

2 1

Questions about Python 3.6.2 installation
by Carlos Baptista 15 Sep '17

15 Sep '17

Greetings, I'm currently using Linux Mint 18.2 "Sonya" (Cinnamon-64) and Python there're two versions installed by default on my system: *Python 2.7.12* and *Python 3.5.2* (picture bellow). [image: Imagem inline 2] I would like to install the latest Python version (3.6.2). As far as I know, I can't install it over Python 2. However, I would like to know if I can install the 3.6.2 over 3.5.2. If that's possible, how can I do that? Thanks in advance.

2 1

Accepting PEP 541?
by Lukasz Langa 13 Sep '17

13 Sep '17

I gathered and responded to feedback on this early this year. I think the PEP is ready to be accepted. https://www.python.org/dev/peps/pep-0541/ <https://www.python.org/dev/peps/pep-0541/> - Ł

3 2

Install psutil without gcc?
by Thomas Güttler 11 Sep '17

11 Sep '17

It would be great, if I could install psutil without the need for a gcc on linux. The maintainer (Giampaolo Rodola) is interested, but the roadmap is not clear yet. He said: > It's something which needs to be investigated. A quick Google searched brought me here: > https://github.com/joerick/cibuildwheel > ...so it appears it's definitively possible. Is this the current state of the art, or is there a better way? Regards, Thomas Güttler Related issue: https://github.com/giampaolo/psutil/issues/824 -- Thomas Guettler http://www.thomas-guettler.de/ I am looking for feedback: https://github.com/guettli/programming-guidelines

3 5

(no subject)
by Umar Rasul 09 Sep '17

09 Sep '17

How can I install pandas on Python 2.7 Sent from Windows Mail

1 0

(no subject)
by Natasha Kelkar 08 Sep '17

08 Sep '17

Respected sir/madam I am trying to install jpredapi in python 2.1 using command python -m pip install jpreadapi but getting an error Can you please help me in this? -- Natasha Kelkar Institute of Bioinformatics and Biotechnology(iBB), Savitribai Phule Pune University.

2 1

supporting editable installs by PEP-517/518 by shim wheels - proposing a common implementation
by RonnyPfannschmidt 07 Sep '17

07 Sep '17

Hi Everyone, i'd like to propose a common tooling for installing packages in editable mode which is based on generating an actual wheel, which includes shim files for the python packages allows to sanely instal/uninstall editable packages a while back i implemented a prototype in gumby-elf which serves as my example. i'd like to start a discussion on this, and hopefully ensure that tools like flit and setuptools adapt a similar mechanism, ideally sharing a implementation once this discussion is finished i'd also like to implement it for setuptools -- Ronny

3 4

Re: [Distutils] PEP 517 again
by Chris Barker 05 Sep '17

05 Sep '17

On Mon, Sep 4, 2017 at 6:00 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> Do the Linux distros use pip to build their packages? > > > > Not that I am aware of. > > Fedora's build macros for Python projects currently rely on running > setup.py directly, but we've been considering switching to pip instead > since 2013 or so. PEP 517 is likely to provide the impetus to switch > from "maybe we should do that" to "we need to do that, at least if > setup.py is missing, and potentially always, so we get more consistent > installation metadata" > which is exactly why I tried to do it in conda. In a post PEP 517 world, that may be a good way to go. I'm looking forward to it. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

string types for paths in PEP 517
by Nathaniel Smith 05 Sep '17

05 Sep '17

Hi all, Quick question about an arcane topic: currently, PEP 517 says that paths are always represented as unicode strings. For example, when the frontend calls build_wheel, it has to create a temporary dir to hold the output wheel, and it passes this in as an absolute path represented as a unicode string. In Python 3 I think this is totally fine, because the surrogate-escape system means that all paths can be represented as unicode strings, even on systems like Linux where you can have paths that are invalid according to Python's idea of the filesystem encoding. In Python 2, if I understand correctly (and I'm not super confident that I do), then there is no surrogate-escape, and it's possible to have paths that can't be represented as a unicode object. For example, if someone's home directory is /home/stéfan in UTF-8 but Python thinks that the locale is C, and a frontend tries to make a tmpdir in $HOME/.local/tmp/ and pass it to a backend then... everything blows up, I guess? So I guess this is a question for those unfortunate souls who understand these details better than me (hi Nick!): is this actually a problem, and is there anything we can/should do differently? -n -- Nathaniel J. Smith -- https://vorpus.org

6 6

Re: [Distutils] PEP 517 again
by Chris Barker 05 Sep '17

05 Sep '17

> conda-build doesn't produce or consume wheels. It works by creating a > clean > > conda environment and running a shell script to install the python > package > > into that environment To be really clear -- conda build doesn't directly use ANY build/install system. All files produced by the install > > script are then put into a tar file, which is conda's version of a wheel. > right -- conda-build simply runs the script it is told to run (by the recipe), and then looks and sees what files were installed. In theory, one could build a conda package directly out of a binary wheel -- after all, as mentioned, they are actually pretty similar, but most folks would probably just use pip inside their conda-build recipe. In fact, that is already being done -- most python packages are built with "setup.py install", but some are built by using pip to install a wheel. But I didn't think that the goal of PEP 517 was to ensconce pip as the only way to install a package in Python. So if a build system can install (like setuptools currently does), then conda recipes may well call it directly. I don't think this really has any bearing on the the PEP, though -- As I understand it, the goal of the PEP is to formalize the interface between build systems and package managers -- and that can only be a good thing for other package managers. NOTE: For the most part, folks want to use a pip-compatible install in conda-build, because users tend to also directly use pip to install stuff in conda environments e.g. if there is not a conda package available. If a conda package were built with, say, plain distutils, then pip won't find the meta data it's looking for, and might falsely see an unresolved dependency. Personally, I've always avoided pip installing packages in conda -- preferring to make conda packages for all the deps I need, but conda-pip compatibility has come a long way, so it does work fine for many users. But it would be nice if packages didn't NEED to be pip installed -- but I think that's an issue with the wheel spec (that may well be solved). -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

11 45

PEP 541
by nassim daoud 03 Sep '17

03 Sep '17

Hello, There is a friend of mine wondering, if abandoned PyPI projects can be claimed. And if so what the procedure in order to do that. My best regards, Daoud Nassim

2 1

why do we still put version numbers into dist-info folders
by RonnyPfannschmidt 01 Sep '17

01 Sep '17

Hi everyone, a while now i have wondered - why put version numbers into the dist info folders not only makes it lookup more expensive (need to search for a distribution->folder map) its also not serving a purpose, as the rest of the package is unversioned at the root of site-packages. in addition all those versions in folder names, that serve no purpose, and create duplication/conflicts now creep into other peps as well, please lets take a note at setuptools - for editable installs it wont put a version in to the egg info folder and imho thats the correct way to deal with data thats not actually versioned at that level where its being stored. -- Ronny

3 3

Re: [Distutils] PEP 517 again
by Chris Barker 31 Aug '17

31 Aug '17

On Thu, Aug 31, 2017 at 8:57 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > > As to using pip to build wheels -- there is good reason to do that > > now, but in s post PEP 517 world, one would call the build system > > directly to build a wheel-- after all, all pip should be doing is > > calling the build system anyway. > > I disagree - "pip wheel" will still be useful, for example to obtain a > wheel from PyPI either by downloading or download & build. Also just > to have a unified interface that works regardless of the project > backend - if a project switches from setuptools to flit, for example, > it would be good if deployment and test scripts didn't have to change. > Isn't that why we have PEP 517? I unified interface to build tools? So I still expect pip wheel to be useful in a post-PEP 517 world. Maybe so -- but all pip should be doing is passing off the job to the back-end. Again, the package manager, well, manages the packages. It shouldn't be concerned with how to build them. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

3 2

Re: [Distutils] PEP 517 again
by Chris Barker 31 Aug '17

31 Aug '17

On Thu, Aug 31, 2017 at 9:04 AM, Donald Stufft <donald(a)stufft.io> wrote: > I'm a bit confused -- are we talking about the backwards compatible > path to the future -- or the end-game? > > > Pip purposely overrides what setuptools tags the wheel with in order to > make it extremely specific to the Python it is currently being run on. Now, > a big part of why it needs to do this, is because a lot of setup.py were > not written with wheel in mind, so they produce broken wheels by default. > right -- just so we're clear that this is hack to maintain backward compatibility. Though that seems like a hack that setuptools should have, not pip. > One such example is a setup.py that produces a pure Python wheel for PyPy, > and a C-extension wheel for Python. If someone installs that project on > PyPy first and they only have a sdist available on PyPI, then pip will > cache a pure Python wheel for that project, and by default (without this > bit to force it) that wheel would be acceptable for CPython too and would > have been used instead of building a wheel with a compiled extension for > CPython. > Frankly, I'm not sure that pip should be sharing a cache between PyPy and cPython anyway.. (I've had other issues with pip's fairly aggressive caching in the past...) But if it does, then pip should always look to see if there is a more "specific" version of a package available anyway. But I see your point -- if there is a cached pure-python wheel, pip can go look on PyPi for a cpython wheel, but if it isn't there, and there is only an sdist, it would have to download and build it to find out if there is a cPython version. Which of course, would then happen (and be wasted effort) essentially all the time, as most packages are pure python. But the proper solution to this is for packages that have both a pure python and cpython version available to put the wheels up on pypi. After all, this is the unusual case. In fact, I see PEP 517 as a step toward more wheels, and fewer sdists for distribution. -CHB > You could argue that the new crop of build backends should handle this > better— and hopefully you’re correct— but I think that the two desires are > at odd with one another. Most build backends are going to want to, by > default, tag the wheel with as general of a tag as they think will work > given the information available to them, whereas with pip’s internal wheel > cache, we generally want that wheel to be as specific to the current > version of Python as possible. > > Another possible solution is to stop using the wheel tagging format to > encode this information for pip’s internal wheel cache and to encode it > into the path, so that instead of having PyPy and CPython share a cache > directory, they each have their own. That isn’t an unreasonable mechanism > for doing that and which one gets used is an implementation detail of pip. > > — > Donald Stufft > > > > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 1

Re: [Distutils] PEP 517 again
by xoviat 29 Aug '17

29 Aug '17

Oh, BDFL, will you settle whether we should raise an exception or return Notimplemented? On Aug 25, 2017 8:28 PM, "Greg Ewing" <greg.ewing(a)canterbury.ac.nz> wrote: xoviat wrote: > I mean how is opening a file different than attempting to build an sdist? > 1. Opening a file is a very common operation. 2. Most file opens are expected to succeed, and if one doesn't, the appropriate place to deal with that is almost never at the site of the open call, but somewhere further up. In contrast, there's probably only about one place in any given frontend where the backend's build_sdist method gets invoked, and it's unlikely the ability to let a not-implemented exception bubble up from that point would be of great use. -- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org https://mail.python.org/mailman/listinfo/distutils-sig

12 57

PEP 517 - a plan of action
by Paul Moore 29 Aug '17

29 Aug '17

I feel that the discussion on PEP 517 is nearing an end. The outstanding questions are pretty detail-level, and there's little progress being made on them because the situations involved are mostly theoretical in the absence of actual implementations. So I propose the following plan of action: 1. Accept PEP 517 (at least provisionally). 2. Thomas to complete and publish his library of helper functions for PEP 517 consumers (optional - but I think he said he had such a thing in progress). 3. Build PEP 517 interfaces for flit and setuptools. 4. Implement PEP 517 support in pip. 5. Review what we learned from the above. 6. If needed, revise the PEP in view of actual implementation experience, and finalise it. Further frontends and backends can wait for the results of this process if they are concerned about tracking a provisional PEP. Hopefully, it shouldn't be too long a process (I'd like to see PEP 517 in the next release of pip, if possible). Notes ----- 1. On any remaining questions for the PEP, let's just go for the simplest solution for now. Let the PEP authors decide what that is, with Nick as BDFL arbitrating if there's disagreement. 2. In my view, the PEP 517 support in pip should fully embrace the principle in the PEP that it's the backend's job to ensure that sdist->wheel and build_wheel produce equivalent results (when both are possible), and should therefore *only* use build_wheel, at least until experience demonstrates that we have to protect users against badly written backends. 3. My suspicion is that the setuptools backend will pretty clearly demonstrate whether it's hard or easy to conform to the requirements in the PEP, which can inform pip's stance on a "pure build_wheel" approach. 4. If pip goes for direct wheel builds, this does mean that the build_sdist and get_requires_for_build_sdist hooks won't be exercised in any actual code during this process. IMO, that's OK, as those hooks cover important (if currently theoretical) cases for validating the "publish my project" workflow, and tools that want to address those cases (such as tox) are likely to need the sdist hooks later. If this seems like a reasonable plan, I'd suggest that people add links to issues or PRs for PEP 517 implementations in flit, setuptools and pip to this thread - as I'm sure all the participants here will want to review the concrete implementations. Please understand - I'm not trying to steamroller the process here, or force particular decisions. It's just that I know I'm burned out on the discussions, to the extent that I'm skimming mails and largely ignoring many of the hypothetical arguments, and I'm sure others are too. So we need to move to a point where we can discuss actual, in-development code rather than possibilities, and I believe the above plan gets us to that point as fast as possible. Paul

4 6

PBR/distutils2 with pep517 Support Was: Conditionless setup.py
by Thomas Güttler 28 Aug '17

28 Aug '17

Am 25.08.2017 um 15:00 schrieb Paul Moore: > > One thought - are the PBR and/or distutils2 teams looking at providing > PEP 517 support? Assuming they are, have they had a change to review > the PEP to ensure it suits their needs? And if they aren't, what is it > about the PEP that makes them unwilling to do so? Hi Paul, I think this is a very good question! I started a new thread to make this more visible. Are there PBR/distutils2 hackers on this list here? If yes: Do you support this pep? If no: where can you meet PBR/distutils2 hackers? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

4 4

Re: [Distutils] PEP 517 again
by xoviat 26 Aug '17

26 Aug '17

Should probably take Guido out of this email chain now. And does everyone agree with what Nathaniel said on sys.path? On Aug 26, 2017 11:00 AM, "Daniel Holth" <dholth(a)gmail.com> wrote: I'm bored with that On Sat, Aug 26, 2017, 11:59 xoviat <xoviat(a)gmail.com> wrote: > Is everyone on board with that? > > On Aug 26, 2017 4:29 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote: > >> On 26 August 2017 at 03:17, Guido van Rossum <guido(a)python.org> wrote: >> > In pretty much any other context, if you have an operation that returns >> an >> > regular value or an error value, the error value should be None. >> (Exceptions >> > include e.g. returning a non-negative int or -1 for errors, or True for >> > success and False for errors.) >> >> So, given that build_sdist returns the path of the newly built sdist, >> the correct way to signal "I didn't manage to build a sdist" would be >> to return None. >> >> Now that it's put this way, it seems glaringly obvious to me that this >> is the correct thing to do. >> Paul >> > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG(a)python.org > https://mail.python.org/mailman/listinfo/distutils-sig >

1 0

Conditionless setup.py
by Thomas Güttler 26 Aug '17

26 Aug '17

The setup.py of numpy looks big and complicated: https://github.com/numpy/numpy/blob/master/setup.py I guess there are even more complicated setup.py files. Do you think it will be possible in the future to make setup.py conditionless (no "if", no "else", no function calls)? I mean that the setuptools/distutils/whatever libraries will handle all needed conditions? This would mean setup.py would be a pure data structure which could be encoded in json or yaml. What do you think? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

9 13

PEP 517 again
by Thomas Kluyver 26 Aug '17

26 Aug '17

Nathaniel seems to be busy with other things at the moment, so I hope he won't mind me passing on this list of things he'd like to resolve with the draft PEP. I'll quote his comments and put my responses inline. > - needs some sort of NotImplemented(Error) handling specified We've gone round return-value vs exception a few times, but I don't think that debate is particularly heated, so it probably just needs someone to say "this is what we're doing". I can be that someone if needs be. But does anyone feel strongly about it? > - The out-of-tree build example (that makes an sdist and then builds > it) seems like exactly the kind of implementation that Donald says he > doesn't want to see? OTOH I think Nick said he wants to see a more > elaborated specification of out-of-tree build strategies with this > specifically as one of them. > - Nick has suggested that the to-be-defined NotImplemented(Error) > handling can be used by build_wheel to indicate that it can't do > out-of-tree builds, so maybe the frontend should try an in-tree build > instead. I don't know how to convert this into a real spec, though -- > like in general, if I'm a frontend and I call `hook(*args, **kwargs)` > and it says "can't do that", then how do I know what the problem is > and what I should try instead? > - What happens if prepare_build_metadata returns NotImplemented / > raises NotImplementedError? > - I don't understand how out-of-tree builds and prepare_build_metadata > are supposed to interact. > - Also, AFAICT the out-of-tree build feature has no motivation > anymore. The only mentioned use case is to support incremental build > features in automatic build pipelines like Travis, but there are a > number of these build pipelines, and lots of existing build systems > that support out-of-tree builds, and AFAICT no-one actually uses them > together like this. (I think it's because most build systems' > out-of-tree build features are designed on the assumption that there's > a human running the show who will deal with any edge cases.) I believe the out-of-tree build option Nathaniel is referring to is the build_directory parameter to build_wheel(). His proposed APIs remove that parameter, and elsewhere in his email he describes that no-one seems to want it: everyone thinks someone else wants it. By my understanding, the reasons for including the build_directory parameter are: 1. Without an explicit build directory, the developers of pip are concerned that build backends will modify the source tree and cause issues which users report as bugs to pip. 2. A frontend-controlled build directory could be used for caching intermediate build artifacts. This was a secondary argument after we had the idea, and we've never really fleshed out how we expect it to work. There's also a concern that if the first round of frontends always use an empty tempdir, backends will end up relying on that. I think this second argument is a weak one unless we spend the time to figure out the details. Do other people see the situation in a similar way? How might we move forwards on this? > - needs some sort of conclusion to the srcdir-on-sys.path issue. While Nathaniel is in the minority regarding srcdir-on-sys.path, he argues that most of us are assuming a default position without really thinking through it, which is certainly true of me. I don't feel strongly about this topic, but I do want to come to a conclusion. As Nathaniel does feel strongly about it, does anyone object to either: A. Leaving the source dir off sys.path when loading the hooks, and making a special backend which exposes hooks from the CWD. B. Adding another key in the TOML file to control whether hooks can be loaded from the source dir. I can live with either, but I prefer A, because it means fewer options to describe in the spec. Thomas

8 45

Fwd: Re: PEP 517 again
by xoviat 25 Aug '17

25 Aug '17

---------- Forwarded message ---------- From: "xoviat" <xoviat(a)gmail.com> Date: Aug 24, 2017 8:39 AM Subject: Re: [Distutils] PEP 517 again To: "Thomas Kluyver" <thomas(a)kluyver.me.uk> Cc: That's actually the general argument against exceptions and why golang doesn't have them. I have not seen notimplemented used in the wild ever though. Also, regarding the build_dir argument, it won't be used in pip. The pip devs may have said that but it's too much work to support now and most of them seem to have a shortage of time. Pip will be copying to a temporary directory for a long time. On Aug 24, 2017 8:28 AM, "Thomas Kluyver" <thomas(a)kluyver.me.uk> wrote: > On Thu, Aug 24, 2017, at 02:21 PM, xoviat wrote: > > I mean is this golang or Python? In Python, you raise notimplementederror. > > > But there's a NotImplemented singleton in Python as well. The argument for > using a return value is that the hook code has to deliberately return that, > whereas a NotImplementedError could bubble up from some internal call, in > which case it should really be registered as an error. >

8 51

Fwd: Fwd: Re: PEP 517 again
by xoviat 24 Aug '17

24 Aug '17

---------- Forwarded message ---------- From: xoviat <xoviat(a)gmail.com> Date: 2017-08-24 14:26 GMT-05:00 Subject: Re: [Distutils] Fwd: Re: PEP 517 again To: Thomas Kluyver <thomas(a)kluyver.me.uk> And also, are Nick and Paul okay with this? 2017-08-24 14:24 GMT-05:00 xoviat <xoviat(a)gmail.com>: > So in summary: > - NotImplementedError > - Remove build_dir and specify that no modifications can be made to the > source directory > > On the third issue: > > I understand where Nathaniel is coming from in regards to sys.path. But > there should at least be an option for this because of for example, numpy: > numpy builds itself with it's own build system that is defined in the > source directory. If you remove '' from sys.path, then numpy cannot build > itself because there is no other numpy.distutils package. > > 2017-08-24 14:00 GMT-05:00 Thomas Kluyver <thomas(a)kluyver.me.uk>: > >> On Thu, Aug 24, 2017, at 07:30 PM, Leonardo Rochael Almeida wrote: >> >> It looks like a lot of trouble for a feature that is designed to solve a >> problem for a very thin intersection of the Venn diagram of people who: >> >> a. wants to quickly iterate while hacking a package >> >> b. doesn't want to bother learning how to trigger the incremental build >> system of that package or how to install it in editable mode. >> >> For better or worse, editable installs exist and don't need to be defined >> by PEP 517 just to handle this case. I'm not convinced it's worth the >> additional complexity of the spec just for that. >> >> >> This is a good point. Given that developers can use the build tool >> directly for development purposes, rather than going through the PEP 517 >> interface, we don't need to prioritise that use case for the spec. >> >> So for the time being, I'd prefer to say that the build_wheel() hook >> should not create or modify files in the source directory. Backends may >> keep their own caches elsewhere to speed up repeated builds, but any more >> provisions for developer use cases should be left for a future PEP. And >> developers can invoke enscons/flit/... directly in the meantime. >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG(a)python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> >

1 0

Mirroring PyPI JSON Locally
by Cooper Ry Lees 24 Aug '17

24 Aug '17

Hi all, First time emailer, so please be kind. Also, if this is not the right mailing list for PyPA talk, I apologize. Please point me in the right direction if so (Brett Canon pointed me here). The main reason I have emailed here is I believe it may be PEP time to standardize the JSON metadata that PyPI makes available, like what was done for the `'simple API` described in PEP503. I've been doing a bit of work on `bandersnatch` (I didn't name it), which is a PEP 381 mirroring package and wanted to enhance it to also mirror the handy JSON metadata PyPI generates and makes available @ https://pypi.python.org/pypi/PKG_NAME/json. I've done a PR on bandersnatch as a POC that mirrors both the PyPI directory structure (URL/pypi/PKG_NAME/json) and created a standardizable URL/json/PKG_NAME that the former symlinks to (to be served by NGINX / some other proxy). I'm also contemplating naming the directory 'metadata' rather than JSON so if some new hotness / we want to change the format down the line we're not stuck with json as the dirname. This PR can be found here: https://bitbucket.org/pypa/bandersnatch/pull-requests/33/save-json- metadata-to-mirror My main use case is to write a very simple async 'verifier' tool that will crawl all the JSON files and then ensure the packages directory on each of my internal mirrors (I have a mirror per region / datacenter) have all the files they should. I sync centrally (to save resource on the PyPI infrastructure) and then rsync out all the diffs to each region / datacenter, and under some failure scenarios I could miss a file or many. So I feel using JSON pulled down from the authoritative source will allow an async job to verify the MD5 of all the package files on each mirror. What are peoples thoughts here? Is it worth a PEP similar to PEP503 going forward? Can people enhance / share some thoughts on this idea. Thanks, Cooper Ry Lees me(a)cooperlees.com <me(a)copperlees.com> https://cooperlees.com/

6 13

PEP 517: Bootstrapping setuptools
by xoviat 22 Aug '17

22 Aug '17

Previously, the attempt to move setuptools off of vendored dependencies failed because it was not done correctly: install_requires was set to the vendored dependencies but not setup_requires, which would have been required to correctly specify dependencies. However, setup_requires would have introduced circular dependencies that are impossible to correctly resolve, so that would have simply shifted an impossible problem onto pip. The principle issue then, is that setuptools requires setuptools to build itself. And although, this problem was previously not worth solving because of 'setup.py install', it is now not a difficult problem to solve if we rethink slightly what should be included in the setuptools respository: - Rather than re-generating egg_info on each setup.py invocation, we can cache dist-info in the setuptools respository. - We can implement a simple entry_points script that updates dist-info based on the contents of setup.py, which is only runnable of course after setuptools is installed - We can implement the reference build backend for setuptools: simply copy the files and dist-info into a new compliant zip archive Viola! Setuptools is no longer required to build setuptools, and the installation process is fully compliant with Python PEPs.

8 16

[issue166] Install path changes when data_files is present
by Salvo Tomaselli 21 Aug '17

21 Aug '17

New submission from Salvo Tomaselli: I have a python thing that I'm installing in a custom path, using setup.py install --root=/tmp/turi13 --install-purelib=custom/path However, if the setup.py file contains a data_files field, then, unless i also specify --install-data, the --install-purelib parameter gets ignored. I think an error should be generated and the entire thing should fail instead. Same for all of those --install-* parameters. ---------- messages: 791 nosy: ltworf priority: bug status: unread title: Install path changes when data_files is present _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue166> _______________________________________________

1 0

Transfering modules
by Ian Hartley 21 Aug '17

21 Aug '17

Hello, I have a question that I haven't been able to find online or in the python documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment. I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter. Since I was not able to transfer packages to the newer version, I tried to install my module to the working python 3.6.1 version, but similarly, could not find any documentation about how to do this. I also tried to completely uninstall the python 3.6.2 build, but after running sudo-remove commands, and then testing by booting python3, the build running is still 3.6.2, even after removing the framework etc. I am sincerely confused by this and it has unfortunately brought my experiment to a standstill. If you could offer any illumination on this, I would be extremely grateful. I truly thank you in advance for any help you can provide. Best regards, Ian Hartley

5 11

Claiming packages as per PEP 541?
by Samuel Carlsson 21 Aug '17

21 Aug '17

I've seem people claiming packages on the Support Requests <https://sourceforge.net/p/pypi/support-requests/?source=navbar> page on source forge, I've also done so myself. However, most of these request seems to go unnoticed. The PEP 541 does not seem to specify any way actually performing claims to transfer ownership of packages. In short: What is the procedure to claim a project on PyPI? /samuel

1 0

PEP 517: Build frontend responsibilities
by 12345 67890 19 Aug '17

19 Aug '17

All: After reviewing the PEP, I personally feel that it relies too much on the backend Doing The Right Thing. As it currently stands, it is my understanding that the build backend is called in the source directory and is then responsible for handling the entire process to build a compliant wheel. In a PEP 517 world, the build backends may be poorly written initially and may not fully comply with the PEP’s goal to produce the same wheel that would be produced from an sdist. Current process: - Frontend calls backend in source directory build-wheel Proposed process: - Frontend copies source tree to temporary directory - Frontend invokes build-sdist to build an sdist - Frontend extracts sdist to new temporary directory - Frontend reloads backend from sdist directory and invokes build-wheel The proposed process is more computationally intensive, but moves compliance logic out of the build backend. In addition, we can make some modifications based on this proposed process: - Remove parameter for build_dir in build_wheel since it is always the current directory - Optimization: add new parameter to build_sdist called copy_extra, that when set to false, only includes files in sdist needed to build a wheel. - Specify that build_wheel should fail if PKG-INFO is not present so that it is always invoked in an sdist I realized that I may have pushed to have this completed earlier, but I agree with Donald that we need to enforce a process rather than hoping for the best. Regards, xoviat

6 21

status check on PEP 517
by 12345 67890 15 Aug '17

15 Aug '17

Nick: Do you have any update on when the PEP will be completed? Regards, xoviat

4 3

Distuils MSVC link failure between libraries
by P 15 Aug '17

15 Aug '17

Hi there, I'm trying to create multiple C++ extensions that have dependencies between them. This works fine on Linux with gcc, but I get link failures on Windows with MSVC due to this <https://github.com/python/cpython/blob/master/Lib/distutils/_msvccompiler.p…> "*IMPLIB*" is an optional argument, when not specified, the generation of exp/lib files default to the same filename and directory as the dll. If precompiled extensions are to be published with MSVC and expected to be linked against (maybe this is not recommended?) the lib files are required so it seems a bit peculiar to me to not have them with the dll as is the default behavior. Proof of concept <https://github.com/0xz/distutils_msvc/blob/master/setup.py> Thanks Philip

2 1

pip-installing from a VCS url with a commit hash
by Chris Jerdonek 11 Aug '17

11 Aug '17

I was interested in working on the following issue but thought it would be worth asking this list about first: https://github.com/pypa/pip/issues/4507 If pip-installing a dependency from a VCS url of the following form: git+https://myvcs.com/some_dependency@sometag#egg=SomeDependency and "sometag" is the hash of a commit, should pip interpret it as a commit hash, or should it first check for a branch or tag of that name? One argument in favor of interpreting it as a commit hash is greater determinism. If someone later adds a branch or tag with that name, the dependency will continue to resolve to the original commit. (However, someone could still remove the original commit before adding a branch or tag with that name.) Another argument is that, without this behavior, once a branch or tag with that name is added, there will no longer be a way to refer to the original commit. Either way, I think the log message can be improved. Every time I see that log message, I think to myself, "of course it's a commit, it's a 40-character hexadecimal string!" --Chris

1 0

new PyPI: a rant from a package maintainer
by Lucas Boppre Niehues 07 Aug '17

07 Aug '17

I just had a dreadful day dealing with PyPI and the migrations that are happening. I'm posting this rant here in hopes of reaching the people who can fix it (apologies if I'm in the wrong list), or at least offer an explanation of the events that happened. Disclaimers: - Any and all feedback is welcome, but please be charitable. - I'm angry *at* things, I hope not to offend anybody. - I'm posting this rant here and not reddit or hn to ensure the discussion stays between the interested parties and not the whole internet. For formatting reasons the text is here: https://gist.github.com/boppreh/ac7522b3a4ac46b4f6010eecddc57f21 Thank you for your attention

11 16

PEP 517 Status
by 12345 67890 06 Aug '17

06 Aug '17

Hello all: I have been attempting to resolve the horrible easy_install behavior where compiled wheels published on pypi are not used to fulfill setup requirements. To address this issue, I have submitted pull requests to both setuptools and pip that would allow a proprietary communication method that can later be easily adjusted to be fully PEP 517 compliant, saving work while resolving a specific issue. However, everyone appears to be ignoring these pull requests, in my estimation, because PEP 517 is not fully completed. That’s fine, but partial PEP 517 compliance in one area is better than nothing at all, which is currently the default scenario. Regards, xoviat

2 2

status check on PEP 517
by Nathaniel Smith 05 Aug '17

05 Aug '17

Hi all, So looking over the big threads from the last week, I *think* I have a reasonable sense of what different people think about the different trade-offs here... but I'm not sure. And I'm guessing I'm not the only one feeling uncertain here. So in this post I'm not going to argue for anything at all! It's purely for information gathering, to try and stop, take stock, and figure out where everyone actually is at this point. To make this concrete: I'm *pretty* sure (?) that at this point all the basic elements in my "simplified" rewrite are things that we now have consensus are needed in some form, so maybe we can use that as a kind of "minimal core" reference point: https://github.com/njsmith/peps/commits/517-refactor-streamline/pep-0517.txt (Again, I'm NOT arguing for this right now, I just figure there's been enough stuff flying by that it's useful to have a concrete reference point.) And I'd really appreciate hearing answers to these questions, from at least Daniel, Donald, Nick, Paul, Thomas, and of course anyone else who wants to speak up: - Is there anything missing from this "minimal core" that *you or your projects* need? - Is there anything missing that you think will cause problems for *others*? - Is there anything in there that you feel adds unnecessary complexity? - Is there anything missing that you think is definitely something we'll want eventually? Is it something where writing a small dedicated followup PEP would produce a materially worse outcome than including it from the start? Thanks! -n PS: just for the record, the link above is almost identical to the version I sent before; main change is some TODO notes added at the end. Diff view: https://github.com/njsmith/peps/compare/64b839a39101bc96bef883f09185e2c6c77… -- Nathaniel J. Smith -- https://vorpus.org

10 42

Re: [Distutils] status check on PEP 517
by xoviat＠gmail.com 05 Aug '17

05 Aug '17

Folks, Can we get these issues sorted before pip 10 so that the pip committers will accept my PR addressing the horrible easy_install behavior? Regards, xoviat

1 0

setup.cfg - to interpolate or not to interpolate
by Jason R. Coombs 28 Jul '17

28 Jul '17

In Setuptools #889 (https://github.com/pypa/setuptools/issues/889) and Python 20754 (http://bugs.python.org/issue20754), setuptools and distutils are proposing a backward-incompatible change - to remove ConfigParser interpolation when parsing setup.cfg (and maybe other config) files. I’d previously thought that it was an unintended regression from Python 2 to introduce interpolation, but on deeper investigation, interpolation has been present all along. While it’s conceivable that no projects are relying on the interpolation behavior, I’d be surprised if none have at least worked around the interpolation behavior by escaping percent signs. So I’m left with the sinking feeling there’s no easy transition from the status quo to the desired state (no interpolation). I’d like feedback, in either ticket, about whether this change is worth the trouble and if so if there are any ways to mitigate the risks of introducing such a change. If someone can assist on this effort, that would also be greatly appreciated. I’ll likely abandon the effort and close the tickets as won’t fix unless we can identify someone to champion the effort.

4 5

how to build a python extension with a shared object in osx10.11
by bin 27 Jul '17

27 Jul '17

I have installed libfoo.dylib into /usr/local/lib And my python extension's setup.py: from distutils.core import setup, Extension module1 = Extension('bar', libraries=['libfoo2.dylib', ], language="c++", extra_compile_args=['-v', ], sources=['main.cpp']) setup(name='PackageName', version='1.0', description='This is a demo package', ext_modules=[module1]) but it build failed for can not founding the libfoo2.dylib. Here is complie message: "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/lipo" -create -output build/temp.macosx-10.11-intel-2.7/main.o /var/folders/45/gz7ljrz94mdct3nvm9l5ftnr0000gp/T/main-684613.o /var/folders/45/gz7ljrz94mdct3nvm9l5ftnr0000gp/T/main-94fc36.o c++ -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -Wl,-F. build/temp.macosx-10.11-intel-2.7/main.o -L/usr/local/lib -llibfoo2.dylib -o build/lib.macosx-10.11-intel-2.7/bar.so ld: library not found for -llibfoo2.dylib clang: error: linker command failed with exit code 1 (use -v to see invocation) error: command 'c++' failed with exit status 1

2 1

PyPi’s predictable download url
by Aleks Bunin 27 Jul '17

27 Jul '17

Hello, I’m writing installation instructions for a package and have a question: what is the correct predictable URL for the package? I found couple. Which one of these guaranteed won’t change again in the future? - https://pypi.python.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.… (used to work in the past, but doesn’t seem to work now, copied from virtualenv’s docs <https://virtualenv.pypa.io/en/latest/installation/>) - https://pypi.io/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz - https://pypi.org/packages/source/v/virtualenv/virtualenv-15.1.0.tar.gz - https://files.pythonhosted.org/packages/source/v/virtualenv/virtualenv-15.1… Is there a predictable URL, which will return latest version of the package? Thank you. — Regards, *Aleks* *Bunin* -- *DISCLAIMER* This communication is intended only for the personal use of the recipient(s) named above and may contain information and/or attachments that are privileged, confidential, proprietary and protected from disclosure. If you are not the intended recipient of this message, you are hereby notified that any dissemination, distribution or copying of this message, and any attachments thereto, is strictly prohibited. If you receive this message in error, please promptly delete it without further distribution and reply to the sender that you have received the message in error. This communication or any attachment thereto is not intended, and should not be construed, as a investment advice or and does not constitute an offer or commitment, a solicitation of an offer, or any advice or recommendation, to enter into or conclude any transaction. This communication or any attachment thereto is not intended, and should not be construed, as legal, tax, public accounting or auditing advice or opinions. You should consult your legal counsel, accountants and/or tax advisors prior to making any decisions or taking any action concerning the matters in this communication. Attachments hereto may have additional important disclosures and disclaimers, which you should read.

10 18

Manual upload of RPM files no longer works
by Anthony Tuininga 26 Jul '17

26 Jul '17

All, I just tried to upload some RPM files using the "Files" section in PyPI. I get the following response: Gone (This API has been deprecated and removed from legacy PyPI in favor of using the APIs available in the new PyPI.org implementation of PyPI (located at https://pypi.org/). For more information about migrating your use of this API to PyPI.org, please see https://packaging.python.org/guides/migrating-to-pypi-org/#uploading. For more information about the sunsetting of this API, please see https://mail.python.org/pipermail/distutils-sig/2017-June/030766.html) I have used twine for wheels and source distributions without any difficulty. But it refuses to support RPM files: ValueError: Unknown distribution format: 'cx_Oracle-6.0rc2-py26-1.x86_64.rpm' Is there any way to get these uploaded now? I can't use bdist_rpm upload and I'd like to verify the files *before* they are uploaded, not afterwards! Thoughts? Anthony

3 2

Installing modules
by HARISHYAM KUMAR 26 Jul '17

26 Jul '17

No any program run in python 2.7 Please suggest me how to down load modules to run required programs Sent from Mail for Windows 10

1 0

Maintenance of the "wheel" package
by Alex Grönholm 25 Jul '17

25 Jul '17

Looking at the commit history, the maintenance of "wheel" has more or less stalled since 2016. I would like to volunteer myself as a co-maintainer of this project, if allowed. Here's my list of things I would like to do with it: * Move the project from Bitbucket to the PyPA organization on Github * Add a setup() keyword (and documentation for it) for license_file (currently only usable via the deprecated [metadata] section in setup.cfg) * Refactor the test suite to make better use of py.test * Clean up code style for PEP 8 compliance

2 2

Getting dependecies of package from PyPiJSON
by Krzysiek Płachno 21 Jul '17

21 Jul '17

Hi! I'm contributing to XWiki open source project http://www.xwiki.org/. XWiki platform is written in Java and apart from many wonderful features allows for scripting in python on its pages. My current task is to enable installing packages form PyPi repository so that they can be later used in scripts. The installment is done in custom way making python packages available for Java classloader. To make downloaded package working it's needed to install also dependencies. Is it possible to get dependencies information directly from PyPiJSON API? (e.g. by adding some request parameter or header in GET request) Or is it possible to this data in any other way (apart from downloading package)? Thanks for help in advance! Best, Krzysztof

7 12

requirements for variable python
by Robin Becker 19 Jul '17

19 Jul '17

Is there a way to build a universal wheel which needs an additional module when python 2.7 is being used. Currently the run requirements for the module are python-dateutil jsonschema but in addition jsonschema needs functools32 for python 2.7. Is there a way to get the extra requirement used when python 2.7 is the installer? Or does this package not qualify for universalness. -- Robin Becker

2 2

Pip not working
by Michael Afolabi 19 Jul '17

19 Jul '17

Hello Python, I have been trying to install pip onto my windows laptop but even though I have python 3.6. It still will not run. Every time I type pip it returns by saying that it is not recognized as an internal or external command. Also when i type in python -m pip install, it returns with python.exe: No module named pip. I was wondering if you can provide me a solution to this problem Thanking you, Mike

2 2

Re: [Distutils] Notes on pip's requirements for PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

On Jul 17, 2017 8:48 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote: On 17 July 2017 at 14:45, Nathaniel Smith <njs(a)pobox.com> wrote: > On Mon, Jul 17, 2017 at 6:42 AM, Daniel Holth <dholth(a)gmail.com> wrote: >> A minor detail for pip strategy option #2 is that sdists do not have to have >> PKG-INFO. > > ...should they be required to have PKG-INFO? This seems like a > relevant question regardless of whether pip goes with "strategy option > #2" :-). A note of caution here - this way lies (at least some level) of defining an official sdist format. I'm not against that (specifically, I'd be happy if we did that, just formalising what setuptools currently does in some usable form, and leave any sort of "sdist 2.0" debate for when we've all got our breath back after this discussion) but I'll note that we went round this loop very early on here (which resulted in the PEP being written in terms of "source trees") so we need to be careful not to go back over old ground. There's already a short paragraph that gives some minimal requirements on sdists; we needed it for build_sdist. Now that I look again though, it turns out that it already mandates the presence of PKG-INFO. -n

2 1

Notes on pip's requirements for PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

Hi all, I happened to talk to Donald on IRC last night, and said I'd write up some notes on that discussion. (He's also seen these notes, but is partially-offline dealing with jury duty.) [Edit: but apparently still replying to email on the list, so some of this repeats things he's posted since I sent it to him to look at, but oh well :-). That's life in the fast-paced world of F/OSS development I guess.] One thing we tried to get a handle on is what, exactly, are constraints that PEP 517 is trying to deal with. As far as pip goes, these are some of the situations that Donald's worried about handling gracefully when it comes to 'pip install .': a) The source tree is on read-only media b) People do an editable install in a source tree, and then follow it up with a regular install ('pip install -e . && pip install .'). Apparently this can cause problems because of the residual metadata from the editable install confusing setuptools. c) Someone does 'pip install .' on a complex package, then discovers that while the build succeeded, it's missing some optional feature because some necessary system library was missing, so they install that library and re-try the 'pip install .'. Will the build backend notice that the library has appeared, or does it only do environment sniffing the first time it's run in a given tree? d) Mounting the same source tree into multiple different docker containers: 'docker run ubuntu -v .:/io pip install /io && docker run alpine -v .:/io install /io'. Will the build backend notice that these two systems have incompatible C toolchains and C ABIs, so you can't share .o files between them? In principle a build system could notice this by detecting that the compiler and system headers have changed, but in traditional build systems it's common to intentionally *not* include these in your build cache key, because you don't want to rebuild the world after every apt upgrade. (For example, gcc provides the -M switch to generate make rules to trigger rebuilds on header changes, and the -MM switch to do the same but ignoring changes in system headers; lots of projects choose to use -MM. This is related to the thing where incremental builds are traditionally a bit sloppy and intended for experts only.) e) Build systems that have different sources of truth for what goes into an sdist versus what goes into a wheel, and thus can easily end up in a situation where a direct VCS->wheel build gives a different result than a VCS->sdist->wheel build. The most prominent offender here is distutils/setuptools with its MANIFEST.in mess, but Donald is nervous that other systems might also reproduce this mistake. f) And finally Donald feels "it's just more hygenic to have ``pip install .`` not modify ., similiarly to how [he] would be upset if ``pip install foo-1.0.tar.gz`` modified foo-1.0.tar.gz in some way". Of course, no system can avoid every problem; the overall goal here is harm reduction and minimizing spurious bugs filed on pip, not perfection. None of these cases arise for 'pip install name' or 'pip install sdist.tar.gz'; it's really only 'pip install .' on an user-provided source tree. ---------- For reference, here's my analysis of how these particular desiderata relate to some possible approaches: - Pip's current system for handling builds from existing source trees (copytree + setup.py bdist_wheel -or- setup.py install) handles (a), (c), (d), (f), but not (b) or (e), unless someone has previously done an in-place build in the tree, in which case it handles (a) and (f) but not (b), (c), (d), or (e). Which is kind of unfortunate, since (a) and (f) are probably the least important items. - When sdist->unpack->wheel is possible, it automatically handles all of these cases. - If a build backend *only* does in-place builds (like meson) and *does not support* in-place or editable installs, then having an out-of-place build_wheel hook automatically takes care of everything except (e). - Otherwise, an out-of-place build_wheel hook acts handles (a), (c), (d), (f) but not (b) or (e), unless someone has previously done an in-place build in the tree, in which case it handles (a) and (f) but not (b), (c), (d), or (e) (assuming that the build system can get confused between artifacts left behind by in-place builds when doing an out-of-place build, which appears to be a common feature of existing systems). Notice that this means that as far as this score card is concerned, out-of-place build_wheel is identical to pip's current copytree + in-place build_wheel. The key insight here is that an out-of-tree build is nicer for the *next* person to use this source tree, but when pip runs what it cares about is whether the *last* build was in-tree or out-of-tree, and that's not something that it has any control over. - If you have a system that supports in-place builds but not build_sdist, then copytree + a "clean build" hook would cover everything except (e). Notice that, for the same reason as above, it doesn't matter whether the clean build is in-tree or out-of-tree, just that it has the ability to avoid any interference from junk left inside the tree by any previous build. ---------- Anyway. Back to things we discussed. As far as we understand, the way that flit comes into this is that flit cannot generate an sdist: - from a VCS directory when the VCS tools are unavailable - from an unpacked sdist (Thomas, is that right?) So on that basis, we considered a few different strategies that pip might take for handling 'pip install .': Pip Strategy Option 1: First make sure that the setuptools backend build_wheel hook takes responsibility for doing the sdist->unpack->wheel dance when running from a VCS checkout. And then, now that we don't have to worry about setuptools messing things up, make pip's strategy be to just do a plain in-place or out-of-place build_wheel. As you can probably guess from the list above, Donald didn't feel comfortable with this, at least to start with -- he thinks that it might be possible for pip to transition to this in the future if it turns out that new backends are generally of high quality, but until we have more experience, he's nervous. This makes sense to me too -- I think we can plausibly mandate that build backends take care of (e) (MANIFEST.in problems -- that's basically the idea of making setuptools's build backend responsible for doing the sdist->unpac->wheel dance), and (a) (read-only source tree) is a rare edge case that can reasonably be handled by a special case in the frontend, but all the other items actually are regular occurences in modern build systems. Really he wants pip to go via sdist 99% of the time, because that's the case that Just Works, and then have whatever fallback is necessary for the 1%. In some sense it doesn't even matter that much what the fallback is, because if this rare edge case is a bit slow or a bit more error prone, well... it's a rare edge case. Even for flit, 99% of the time people will just be installing wheels, not downloading and manually unpacking sdists, so as long as it *works* then it doesn't have to be perfect. Anyway, this leads to the last two possible strategies we discussed: Pip Strategy Option 2: Check to see if a PKG-INFO file is present. If so, do an in-place build_wheel; otherwise, do build_sdist->unpack->build_wheel. The key thing here is that because it's pip checking for the PKG-INFO instead of querying the build backend, then every source tree has to either *be* an sdist, or be able *produce* an sdist; there's no fallback for non-sdist trees that can't produce sdists. (At least if you want to support 'pip install .') Pip Strategy Option 3: Ask the build backend whether it can do a build_sdist. If so, do build_sdist->unpack->build_wheel; otherwise, do copytree->in-place build_wheel, or out-of-place build_wheel, or whatever, it doesn't matter that much to pip. As far as Donald is concerned, either of these options would be fine. For "option 2" (having pip check for PKG-INFO), the main potential flaw we noted is that while it's OK with flit not being able to build an sdist from an sdist, it will error out in the case where you try to do 'pip install .' on a flit VCS checkout and flit can't find the VCS tools. Donald is fine with that, on the grounds that oh well, if you're working with a VCS checkout and need VCS tools to get consistent result then printing an error message telling people to install them is a fine outcome. (In particular, he observes that flit *can* produce sdist-inconsistent results in this case: if there's a .py file present in the source but not added to the VCS, and you do build_wheel, then he thinks that the .py file will be installed iff the VCS tools are unavailable, and would never be included in any sdist. Thomas, is this right?) However, he thinks that Thomas objected to raising an error in this case, and wants to allow flit to work. (Thomas, is that right?) So possibly that's an argument for preferring "option 3" (ask the backend if it can do an sdist). Another point about option 2 that we didn't discuss but that I'll mention now is that it does kinda enforce that all build backends support building sdists in general, which might be good or bad depending on what you think about that. ----------- A few more general points that Donald made: - He doesn't care much about in-place vs out-of-place in general, we can provide either or both as far as he's concerned. It doesn't matter to pip. - But, if we do support both in-place and out-of-place, then he strongly feels that there should *not* be any semantic difference between them "because it just adds another "path" and possible inconsistency", plus if a build backend is able to do something smart it should just do it all the time. So I think this is a point of disagreement between Donald and Nick. - If pip is going to go with "option 3" (i.e., attempting to do build_sdist->unpack->build_wheel, unless the build backend says it can't), then he strongly feels that this fallback should be triggered by some *explicit* signal from the backend saying that build_sdist is not supported; in general he wants build_sdist raising an error to be an immediate failure, rather than triggering the fallback path. Basically what he's saying is that he wants something like my NotImplemented proposal or equivalent. Regarding this last point: I want to point out this would also mean that when building via pip, the fallback could *only* be triggered by backends that explicitly ask for it, so they're the only ones who could possibly end up exposing their users to any weirdness in the fallback path, and it's up to them to figure out how to deal with it. In particular, it means that setuptools and flit are both handled fine regardless of what we decide about in-place vs. out-of-place, because setuptools will always go via the sdist path and thus be built from a pristine directory, and flit doesn't care about in-place vs. out-of-place anyway. Basically, as long as pip has some kind of build_sdist and build_wheel and build_sdist can signal "not implemented" in a clean way, then pip is happy. My analysis above suggests that beyond this, there's *one* possible extension that pip might want: In the future, it might turn out that there is a build backend that is like flit in that it can't always generate an sdist, but is like setuptools in that it can produce broken results when run in a unclean directory. If that happens, and if we decide that it's important for such a backend to play nicely with pip *and* to support incremental builds *and* to do both of these through the PEP 517 hook interface, then it might be useful to have something like a "make_clean_then_build_wheel" hook for pip to call. My feeling is we can probably defer this for now though? This is like 3 edge cases on top of each other and we don't even know if such a backend will ever exist. ---------- So none of the above actually has much to do with the in-place vs out-of-place debate; basically the conclusion is that pip doesn't care. But we still have to pick something. Donald and I both have some preference for only having one way to do it; given that the semantics are supposed to be identical, why bother making everyone implement both? So: should we start with in-place (and possibly add out-of-place as an option later for specific use cases like build pipelines), or should we start by mandating out-of-place? Neither of us have a particularly strong opinion on this. I guess I have mild preference for starting with in-place, because: - Out-of-place is more complicated to implement than in-place, and it's going to be difficult to explain to build system authors why we're forcing them to do this extra work for some obscure cases they might not care about and that we haven't articulated well. - It might seem like only supporting out-of-place builds would simplify pip's life, because the problems above that are caused by in-place build detritus would go away. But this isn't really true, because even if we don't *expose* in-place build functionality through the PEP 517 hooks, then most build systems are still going to implement this, and that means we still need to be prepared to handle the case where the user has done an in-place build, which is the tricky one. Plus, editable installs intrinsically leave detritus in the source tree. - For me, one of the major goals of PEP 517 is to make life easier for *really* gnarly projects -- and here I'm not thinking of like numpy or scipy, they have it easy; I'm thinking of the folks who have, like, a dozen C different libraries with build systems from the 80s vendored inside their source tree. I'm nervous that forcing these folks with complicated embedded multi-package builds to support out-of-tree builds would be a significant burden and present an obstacle to uptake. Out-of-place builds are definitely a more advanced feature and the direction good build systems move in, but I don't want to leave anyone behind. - We have zero users for this functionality right now, which is usually not a good situation for trying to write a spec. - I don't think it will be a big deal to add out-of-place support later if we want to. But really it's all that other stuff that's important to sort out first. -- Nathaniel J. Smith -- https://vorpus.org

4 4

A possible refactor/streamlining of PEP 517
by Nathaniel Smith 18 Jul '17

18 Jul '17

Hi all, I just attempted an experimental refactor/streamlining of PEP 517, to match what I think it should look like :-). I haven't submitted it as a PR to the PEPs repository yet since I don't know if others will agree with the changes, but I've pasted the full text below, or you can see the text online at: https://github.com/njsmith/peps/blob/517-refactor-streamline/pep-0517.txt and the diff at: https://github.com/python/peps/compare/master...njsmith:517-refactor-stream… Briefly, the changes are: - Rearrange text into (hopefully) better signposted sections with better organization - Clarify a number of details that have come up in discussion (e.g., be more explicit that the hooks are run with the process working directory set to the source tree, and why) - Drop prepare_wheel_metadata and prepare_wheel_build_files (for now); add detailed rationale for why we might want to add them back later. - Add an "extensions" hook namespace to allow prototyping of future extensions. - Rename get_build_*_requires -> get_requires_for_build_* to make the naming parallelism more obvious - Add the option to declare an operation unsupported by returning NotImplemented - Instead of defining a default value for get_requires_for_build_*, make it mandatory for get_require_for_build_* and build_* to appear together; this seems simpler now that we have multiple high-level operations defined in the same PEP, and also simplifies the definition of the NotImplemented semantics. - Update title to better match the scope we ended up with - Add a TODO to decide how to handle backends that don't want to have multiple hooks called from the same process, including some discussion of the options. --------------------- PEP: 517 Title: Supporting non-setup.py-based build backends in pyproject.toml Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com>, Thomas Kluyver <thomas(a)kluyver.me.uk> BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com> Discussions-To: <distutils-sig(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015, 1 July 2017 ========== Abstract ========== While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like `DRY <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_-compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they're often quirky, fragile, and expensive to maintain, and yet (c) it's very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. ========================== Reversion to Draft Status ========================== While this PEP was provisionally accepted for implementation in `pip` and other tools, some additional concerns were subsequently raised around adequately supporting out of tree builds. It has been reverted to Draft status while those concerns are being resolved. ======================= Terminology and goals ======================= A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.tar.gz``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as *sdists* for short. A *build frontend* is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree's *build backend*. In a command like ``pip wheel some-directory/``, pip is acting as a build frontend. An *integration frontend* is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements. This may require locating, building, and installing a combination of wheels and sdists. In a command like ``pip install lxml==2.4.0``, pip is acting as an integration frontend. ============== Source trees ============== There is an existing, legacy source tree format involving ``setup.py``. We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. We'll refer to it as the ``setup.py``\-style. Here we define a new style of source tree based around the ``pyproject.toml`` file defined in PEP 518, extending the ``[build-system]`` table in that file with one additional key, ``build-backend``. Here's an example of how it would look:: [build-system] # Defined by PEP 518: requires = ["flit"] # Defined by this PEP: build-backend = "flit.api:main" ``build-backend`` is a string naming a Python object that will be used to perform the build (see below for details). This is formatted following the same ``module:object`` syntax as a ``setuptools`` entry point. For instance, if the string is ``"flit.api:main"`` as in the example above, this object would be looked up by executing the equivalent of:: import flit.api backend = flit.api.main It's also legal to leave out the ``:object`` part, e.g. :: build-backend = "flit.api" which acts like:: import flit.api backend = flit.api Formally, the string should satisfy this grammar:: identifier = (letter | '_') (letter | '_' | digit)* module_path = identifier ('.' identifier)* object_path = identifier ('.' identifier)* entry_point = module_path (':' object_path)? And we import ``module_path`` and then lookup ``module_path.object_path`` (or just ``module_path`` if ``object_path`` is missing). If the ``pyproject.toml`` file is absent, or the ``build-backend`` key is missing, the source tree is not using this specification, and tools should fall back to running ``setup.py``. Where the ``build-backend`` key exists, it takes precedence over ``setup.py``, and source trees need not include ``setup.py`` at all. Projects may still wish to include a ``setup.py`` for compatibility with tools that do not use this spec. ========================= Build backend interface ========================= The build backend object is expected to have callable attributes called "hooks", which the build frontend can use to perform various actions. The two high-level actions defined by this spec are creation of an sdist (analogous to the legacy ``setup.py sdist`` command) and building of a wheel (analogous to the legacy ``setup.py bdist_wheel`` command). We additionally define a namespace for tool-specific hooks, which may be useful for prototyping future extensions to this specification. General rules for all hooks --------------------------- Finding the source tree ~~~~~~~~~~~~~~~~~~~~~~~ All hooks are run with the process working directory set to the root of the source tree (i.e., the directory containing ``pyproject.toml``). To find the source tree, hooks should call ``os.getpwd()`` or equivalent. Rationale: the process working directory has to be set to something, and if we were to leave it up to the build frontend to pick, then packages developers would accidentally write code that assumes a particular answer here (example: ``long_desc = open("README.rst").read()``), and this code would break when used with other build frontends. So it's important that we standardize a value for all build frontends to use consistently. And this is the obvious thing to specify it as, especially because it's compatible with popular and long-standing conventions like calling ``open("README.rst").read()``. Then, given that we've decided to standardize on working directory = source directory, it makes sense to say that this is the *only* way that this information is passed, because providing a second redundant way (example: as an explicit argument to hooks) would only increase the possiblity of error without any benefit. Lifecycle ~~~~~~~~~ XX TODO: do we want to require frontends to use a new process for every hook call, or do we want to require backends to support multiple calls from the same process? Apparently scons and setuptools both can get cranky if you try to invoke them twice from the same process, so *someone* will be spawning extra processes here; the question is where to put that responsibility. The basic trade-off is that making it the backend's responsibility has better best-case performance if both the frontend and backend are able to re-use a single host process; but, if common frontends end up using new processes for each hook call for other reasons, then in practice either backends will end up spawning unnecessary extra processes, or else will end up with poorly tested paths when multiple hooks are run in the same process. Given that ``get_build_*_requires`` → ``build_*`` in general requires changing the Python environment, it doesn't necessarily make sense to run these in the same process anyway. However, there's an important special case where it does: when ``get_build_*_requires`` returns ``[]``. And this is probably the overwhelmingly most common case. Does it even matter? Windows is notoriously slow at spawning subprocesses. As a quick test, I tried measuring the time to spawn CPython 3.6 + import a package on a Windows 10 VM running on my laptop. ``python3.6 -c "import flit"`` was about 300 ms per call; ``python3.6 -c "import setuptools"`` was about 600 ms per call. We could also potentially get fancy and have a flag to let the frontend and backend negotiate this (e.g. ``process_reuse_safe`` as an opt-in flag). This could also be added later as an extension, as long as we initially default to requiring separate processes for each hook. Calling conventions ~~~~~~~~~~~~~~~~~~~ Hooks MAY be called with positional or keyword arguments, so backends implementing them MUST be careful to make sure that their signatures – including argument names – exactly match those specified here. Output ~~~~~~ Hooks MAY print arbitrary informational text on stdout and stderr. They MUST NOT read from stdin, and the build frontend MAY close stdin before invoking the hooks. The build frontend may capture stdout and/or stderr from the backend. If the backend detects that an output stream is not a terminal/console (e.g. ``not sys.stdout.isatty()``), it SHOULD ensure that any output it writes to that stream is UTF-8 encoded. The build frontend MUST NOT fail if captured output is not valid UTF-8, but it MAY not preserve all the information in that case (e.g. it may decode using the *replace* error handler in Python). If the output stream is a terminal, the build backend is responsible for presenting its output accurately, as for any program running in a terminal. If a hook raises any exception, or causes the process to terminate, then this indicates that the operation has failed. User-specified configuration ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ All hooks take a standard ``config_settings`` argument. This argument is an arbitrary dictionary provided as an "escape hatch" for users to pass ad-hoc configuration into individual package builds. Build backends MAY assign any semantics they like to this dictionary. Build frontends SHOULD provide some mechanism for users to specify arbitrary string-key/string-value pairs to be placed in this dictionary. For example, they might support some syntax like ``--package-config CC=gcc``. Build frontends MAY also provide arbitrary other mechanisms for users to place entries in this dictionary. For example, ``pip`` might choose to map a mix of modern and legacy command line arguments like:: pip install \ --package-config CC=gcc \ --global-option="--some-global-option" \ --build-option="--build-option1" \ --build-option="--build-option2" into a ``config_settings`` dictionary like:: { "CC": "gcc", "--global-option": ["--some-global-option"], "--build-option": ["--build-option1", "--build-option2"], } Of course, it's up to users to make sure that they pass options which make sense for the particular build backend and package that they are building. Hook execution environment ~~~~~~~~~~~~~~~~~~~~~~~~~~ One of the responsibilities of a build frontend is to set up the Python environment in which the build backend will run. We do not require that any particular "virtual environment" mechanism be used; a build frontend might use virtualenv, or venv, or no special mechanism at all. But whatever mechanism is used MUST meet the following criteria: - All requirements specified by the project's build-requirements must be available for import from Python. In particular, the distributions specified in the ``pyproject.toml`` key ``build-system.requires`` must be made available to all hooks. Some hooks have additional requirements documented below. - This must remain true even for new Python subprocesses spawned by the build environment, e.g. code like:: import sys, subprocess subprocess.check_call([sys.executable, ...]) must spawn a Python process which has access to all the project's build-requirements. For example, this is necessary to support build backends that want to run legacy ``setup.py`` scripts in a subprocess. - All command-line scripts provided by the build-required packages must be present in the build environment's PATH. For example, if a project declares a build-requirement on `flit <https://flit.readthedocs.org/en/latest/>`__, then the following must work as a mechanism for running the flit command-line tool:: import subprocess subprocess.check_call(["flit", ...]) A build backend MUST be prepared to function in any environment which meets the above criteria. In particular, it MUST NOT assume that it has access to any packages except those that are present in the stdlib, or that are explicitly declared as build-requirements. Building an sdist ----------------- Building an sdist involves three phases: 1. The frontend calls the backend's ``get_requires_for_build_sdist`` hook to query for any extra requirements that are needed for the sdist build. 2. The frontend obtains those requirements. For example, it might download them from PyPI and install them into some kind of virtual environment. 3. The frontend calls the backend's ``build_sdist`` hook to create the sdist. If either hook is missing, or returns the built-in constant ``NotImplemented``. (Note that this is the object ``NotImplemented``, *not* the string ``"NotImplemented"``), then this indicates that this backend does not support building an sdist from this source tree. For example, some build backends might only support building sdists from a VCS checkout, and not from an unpacked sdist. If this occurs then the frontend should respond in whatever way it feels is appropriate. For example, it might display an error to the user. get_requires_for_build_sdist ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: def get_requires_for_build_sdist(config_settings): ... Computes any additional requirements needed for ``build_sdist``. Returns: a list of strings containing PEP 508 dependency specifications, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml``. Example:: def get_requires_for_build_sdist(config_settings): return ["cython"] Or if there are no additional requirements beyond those specified in ``pyproject.toml``:: def get_requires_for_build_sdist(config_settings): return [] build_sdist ~~~~~~~~~~~ :: def build_sdist(sdist_directory, config_settings): ... Builds a ``.tar.gz`` source distribution and places it in the specified ``sdist_directory``. Returns: The basename (not the full path) of the new ``.tar.gz`` file, as a unicode string, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml`` and by the return value of ``get_requires_for_build_sdist``. Notes: A .tar.gz source distribution (sdist) is named like ``{name}-{version}.tar.gz`` (for example: ``foo-1.0.tar.gz``), and contains a single top-level directory called ``{name}-{version}`` (for example: ``foo-1.0``), which contains the source files of the package. This directory must also contain the ``pyproject.toml`` from the build directory, and a PKG-INFO file containing metadata in the format described in `PEP 345 <https://www.python.org/dev/peps/pep-0345/>`_. Although historically zip files have also been used as sdists, this hook should produce a gzipped tarball. This is already the more common format for sdists, and having a consistent format makes for simpler tooling, so build backends MUST generate ``.tar.gz`` sdists. The generated tarball should use the modern POSIX.1-2001 pax tar format, which specifies UTF-8 based file names. This is not yet the default for the tarfile module shipped with Python 3.6, so backends using the tarfile module need to explicitly pass ``format=tarfile.PAX_FORMAT``. Building a wheel ---------------- The interface for building a wheel is exactly analogous to that for building an sdist: the same three phases, the same interpretation of ``NotImplemented``, etc., except of course that at the end it produces a wheel instead of an sdist. get_requires_for_build_wheel ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :: def get_requires_for_build_wheel(config_settings): ... Computes any additional requirements needed for ``build_wheel``. Returns: a list of strings containing PEP 508 dependency specifications, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml``. Example:: def get_requires_for_build_wheel(config_settings): return ["wheel >= 0.25", "setuptools"] build_wheel ~~~~~~~~~~~ :: def build_wheel(wheel_directory, config_settings): ... Builds a ``.whl`` binary distribution, and places it in the specified ``wheel_directory``. Returns: the basename (not the full path) of the new ``.whl``, as a unicode string, or ``NotImplemented``. Execution environment: everything specified by the ``build-system.requires`` key in ``pyproject.toml`` and by the return value of ``get_requires_for_build_wheel``. Note: If you unpack an sdist named ``{name}-{version}.tar.gz``, and then build a wheel from it, then the resulting wheel MUST be named ``{name}-{version}-{compat-info}.whl``. Extensions ---------- Particular frontends and backends MAY coordinate to define additional hooks beyond those described here, but they MUST NOT claim top-level attributes on the build backend object to do so; these attributes are reserved for future PEPs. Backends MAY provide a ``extensions`` dict, and the semantics of the object at ``BACKEND.extensions["XX"]`` can be defined by the project that owns the name ``XX`` on PyPI. For example, the pip project could choose to define extension hooks like:: BACKEND.extensions["pip"].get_wheel_metadata or:: BACKEND.extensions["pip"]["prepare_build_files"] ===================================================== Recommendations for build frontends (non-normative) ===================================================== A build frontend MAY use any mechanism for setting up a build environment that meets the above criteria. For example, simply installing all build-requirements into the global environment would be sufficient to build any compliant package -- but this would be sub-optimal for a number of reasons. This section contains non-normative advice to frontend implementors. A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. This has two benefits: - It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment -- which is a problem when the user requests ``pip install package1 package2``. Or if the user already has pbr==1.8.1 installed in their global environment, and a package build-requires pbr==1.7.2, then downgrading the user's version would be rather rude. - It acts as a kind of public health measure to maximize the number of packages that actually do declare accurate build-dependencies. We can write all the strongly worded admonitions to package authors we want, but if build frontends don't enforce isolation by default, then we'll inevitably end up with lots of packages on PyPI that build fine on the original author's machine and nowhere else, which is a headache that no-one needs. However, there will also be situations where build-requirements are problematic in various ways. For example, a package author might accidentally leave off some crucial requirement despite our best efforts; or, a package might declare a build-requirement on ``foo >= 1.0`` which worked great when 1.0 was the latest version, but now 1.1 is out and it has a showstopper bug; or, the user might decide to build a package against numpy==1.7 -- overriding the package's preferred numpy==1.8 -- to guarantee that the resulting build will be compatible at the C ABI level with an older version of numpy (even if this means the resulting build is unsupported upstream). Therefore, build frontends SHOULD provide some mechanism for users to override the above defaults. For example, a build frontend could have a ``--build-with-system-site-packages`` option that causes the ``--system-site-packages`` option to be passed to virtualenv-or-equivalent when creating build environments, or a ``--build-requirements-override=my-requirements.txt`` option that overrides the project's normal build-requirements. The general principle here is that we want to enforce hygiene on package *authors*, while still allowing *end-users* to open up the hood and apply duct tape when necessary. =================================== Comparison to competing proposals =================================== The primary difference between this and competing proposals (in particular, PEP 516) is that our build backend is defined via a Python hook-based interface rather than a command-line based interface. We do *not* expect that this will, by itself, intrinsically reduce the complexity calling into the backend, because build frontends will in any case want to run hooks inside a child -- this is important to isolate the build frontend itself from the backend code and to better control the build backends execution environment. So under both proposals, there will need to be some code in ``pip`` to spawn a subprocess and talk to some kind of command-line/IPC interface, and there will need to be some code in the subprocess that knows how to parse these command line arguments and call the actual build backend implementation. So this diagram applies to all proposals equally:: +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | implementation | +-----------+ +---------------+ +----------------+ The key difference between the two approaches is how these interface boundaries map onto project structure:: .-= This PEP =-. +-----------+ +---------------+ | +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | |______________________________________| | Owned by pip, updated in lockstep | | | PEP-defined interface boundary Changes here require distutils-sig .-= Alternative =-. +-----------+ | +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | |____________________________________________| | Owned by build backend, updated in lockstep | PEP-defined interface boundary Changes here require distutils-sig By moving the PEP-defined interface boundary into Python code, we gain three key advantages. **First**, because there will likely be only a small number of build frontends (``pip``, and... maybe a few others?), while there will likely be a long tail of custom build backends (since these are chosen separately by each package to match their particular build requirements), the actual diagrams probably look more like:: .-= This PEP =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn-> | child cmdline | -Python+> | backend | | (pip) | | interface | | | implementation | +-----------+ +---------------+ | +----------------+ | | +----------------+ +> | backend | | | implementation | | +----------------+ : : .-= Alternative =-. +-----------+ +---------------+ +----------------+ | frontend | -spawn+> | child cmdline | -Python-> | backend | | (pip) | | | interface | | implementation | +-----------+ | +---------------+ +----------------+ | | +---------------+ +----------------+ +> | child cmdline | -Python-> | backend | | | interface | | implementation | | +---------------+ +----------------+ : : That is, this PEP leads to less total code in the overall ecosystem. And in particular, it reduces the barrier to entry of making a new build system. For example, this is a complete, working build backend:: # mypackage_custom_build_backend.py import os.path import pathlib def get_requires_for_build_wheel(config_settings): return ["wheel"] def build_wheel(wheel_directory, config_settings): from wheel.archive import archive_wheelfile filename = "mypackage-0.1-py2.py3-none-any" path = os.path.join(wheel_directory, filename) archive_wheelfile(path, "src/") return filename def _exclude_hidden_and_special_files(archive_entry): """Tarfile filter to exclude hidden and special files from the archive""" if entry.isfile() or entry.isdir(): if not os.path.basename(archive_entry.name).startswith("."): return archive_entry return None def get_requires_for_build_sdist(config_settings): return [] def build_sdist(sdist_dir, config_settings): sdist_subdir = "mypackage-0.1" sdist_path = pathlib.Path(sdist_dir) / (sdist_subdir + ".tar.gz") sdist = tarfile.open(sdist_path, "w:gz", format=tarfile.PAX_FORMAT) # Tar up the whole directory, minus hidden and special files sdist.add(os.getcwd(), arcname=sdist_subdir, filter=_exclude_hidden_and_special_files) return sdist_subdir + ".tar.gz" Of course, this is a *terrible* build backend: it requires the user to have manually set up the wheel metadata in ``src/mypackage-0.1.dist-info/``; when the version number changes it must be manually updated in multiple places... but it works, and more features could be added incrementally. Much experience suggests that large successful projects often originate as quick hacks (e.g., Linux -- "just a hobby, won't be big and professional"; `IPython/Jupyter <https://en.wikipedia.org/wiki/IPython#Grants_and_awards>`_ -- `a grad student's ``$PYTHONSTARTUP`` file <http://blog.fperez.org/2012/01/ipython-notebook-historical.html>`_), so if our goal is to encourage the growth of a vibrant ecosystem of good build tools, it's important to minimize the barrier to entry. **Second**, because Python provides a simpler yet richer structure for describing interfaces, we remove unnecessary complexity from the specification -- and specifications are the worst place for complexity, because changing specifications requires painful consensus-building across many stakeholders. In the command-line interface approach, we have to come up with ad hoc ways to map multiple different kinds of inputs into a single linear command line (e.g. how do we avoid collisions between user-specified configuration arguments and PEP-defined arguments? how do we specify optional arguments? when working with a Python interface these questions have simple, obvious answers). When spawning and managing subprocesses, there are many fiddly details that must be gotten right, subtle cross-platform differences, and some of the most obvious approaches -- e.g., using stdout to return data for the ``build_requires`` operation -- can create unexpected pitfalls (e.g., what happens when computing the build requirements requires spawning some child processes, and these children occasionally print an error message to stdout? obviously a careful build backend author can avoid this problem, but the most obvious way of defining a Python interface removes this possibility entirely, because the hook return value is clearly demarcated). In general, the need to isolate build backends into their own process means that we can't remove IPC complexity entirely -- but by placing both sides of the IPC channel under the control of a single project, we make it much cheaper to fix bugs in the IPC interface than if fixing bugs requires coordinated agreement and coordinated changes across the ecosystem. **Third**, and most crucially, the Python hook approach gives us much more powerful options for evolving this specification in the future. For concreteness, imagine that next year we add a new ``build_wheel2`` hook, which replaces the current ``build_wheel2`` hook with something that adds new features (for example, the ability to build multiple wheels from the same source tree). In order to manage the transition, we want it to be possible for build frontends to transparently use ``build_wheel2`` when available and fall back onto ``build_wheel`` otherwise; and we want it to be possible for build backends to define both methods, for compatibility with both old and new build frontends. Furthermore, our mechanism should also fulfill two more goals: (a) If new versions of e.g. ``pip`` and ``flit`` are both updated to support the new interface, then this should be sufficient for it to be used; in particular, it should *not* be necessary for every project that *uses* ``flit`` to update its individual ``pyproject.toml`` file. (b) We do not want to have to spawn extra processes just to perform this negotiation, because process spawns can easily become a bottleneck when deploying large multi-package stacks on some platforms (Windows). In the interface described here, all of these goals are easy to achieve. Because ``pip`` controls the code that runs inside the child process, it can easily write it to do something like:: command, backend, args = parse_command_line_args(...) if command == "build_wheel": if hasattr(backend, "build_wheel2"): backend.build_wheel2(...) elif hasattr(backend, "build_wheel"): backend.build_wheel(...) else: # error handling In the alternative where the public interface boundary is placed at the subprocess call, this is not possible -- either we need to spawn an extra process just to query what interfaces are supported (as was included in an earlier draft of PEP 516, an alternative to this), or else we give up on autonegotiation entirely (as in the current version of that PEP), meaning that any changes in the interface will require N individual packages to update their ``pyproject.toml`` files before any change can go live, and that any changes will necessarily be restricted to new releases. ==================== Evolutionary notes ==================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static ``pyproject.toml`` that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into calls to ``setup.py``. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pyproject.toml``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to stop doing that. ================================ Rejected and deferred features ================================ A number of potential extra features were discussed beyond the above. For the most part the decision was made that it was better to defer trying to implement these until we had more experience with the basic interface, and to provide a minimal extension interface (the ``extensions`` dictionary) that will allow us to prototype these features before standardizing them. Specifically: * Editable installs: This PEP originally specified another hook, ``install_editable``, to do an editable install (as with ``pip install -e``). It was removed due to the complexity of the topic, but may be specified in a later PEP. Briefly, the questions to be answered include: what reasonable ways existing of implementing an 'editable install'? Should the backend or the frontend pick how to make an editable install? And if the frontend does, what does it need from the backend to do so. * Getting wheel metadata from a source tree without building a wheel: it's believed that when pip adds a backtracking constraint solver for package dependencies, it may be useful to add a hook to query a source tree to get metadata about the wheel that it *would* generate, if it were asked to build a wheel. Specifically, the kind of situation where it's anticipated that this might come up is: 1. Package A depends on B and C==1.0 2. B is only available as an sdist 3. We fetch the sdist for the latest version of B, build it into a wheel, and then discover that it depends on C==1.5, which means that it isn't compatible with this version of A. 4. We fetch the sdist for the latest-but-one version of B, build it into a wheel, and then discover that it depends on C==1.4, which means that it isn't compatible with this version of A. 5. We fetch the sdist for the latest-but-two version of B... The idea would be that we could reduce (but not eliminate) the cost of steps 3, 4, 5, ... if there were a way to query a build backend to find out the requirements without actually building a wheel, which is a potentially expensive operation. Of course, these repeated fetches are expensive no matter what we do, so the ideal solution would be to provide wheels for B, so that none of this needs to be done at all. And for many packages (for example, pure Python packages), building a wheel is nearly as cheap as fetching the metadata. And building a wheel also has the advantage of giving us something we can store in the wheel cache for next time. But perhaps this is still a good idea for packages that are particularly slow to build (for example, complex packages like scipy or qt). It was eventually decided to defer this for now, since it adds non-trivial complexity for build backends (the metadata fetching phase and the wheel building phase run at different times, yet have to produce consistent results), and until pip's backtracking resolver is actually implemented, we're only guessing at the value of this optimization and the exact semantics it will require. * A specialized hook for copying a source tree into a new source tree: in certain cases, like when installing directly from a local VCS checkout, pip prefers to copy the source tree to a temporary directory before building it. This provides some protection against build systems that can give incorrect results when repeatedly building in the same tree. Historically, pip has accomplished this copy using a simple ``shutil.copytree``, but this causes various problems, like copying large git checkouts or intermediate artifacts from previous in-place builds. In the future, therefore, pip might move to a multi-step process like: 1. Create an sdist from the VCS checkout 2. Unpack this sdist into a temporary directory. 3. Build a wheel from the unpacked sdist. 4. Install the wheel. Even better, this provides some guarantee that going from VCS checkout → sdist → wheel will produce identical results to going directly from VCS checkout → wheel. However, this runs into a potential problem: what if this particular combination of source tree + build backend can't actually build an sdist? (For example, `flit <http://flit.readthedocs.org/>`__ may have this limitation for certain trees unpacked from sdists.) Therefore, we considered adding an optional hook like ``prepare_temporary_tree_for_build_wheel`` that would copy the required source files into a specified temporary directory. But: * Such a hook would add non-trivial complexity to this spec: it requires us to promote the idea of an "out of tree build" to a first class concept, and specify which kinds of trees are required to support which operations, etc. * A major motivation for doing the build-sdist-unpack-sdist dance in the first place is that we don't trust the backend code to produce the same result when building from a VCS checkout as when building from an sdist, but if we don't trust the backend then it seems odd to add a special hook that puts the backend in charge of doing the dance. * If sdist creation is unsupported, then pip can fall back on a ``shutil.copytree`` strategy in just a few lines of code * And in fact, for the one known case where this might be a problem (unpacked sdist using flit), ``shutil.copytree`` is essentially optimal * Though in fact for flit, this is still a pointless expense – doing an in-place build is perfectly safe and even more efficient. * Plus projects using flit always have wheels, so this will essentially never even come up in the first place * And pip hasn't even implemented the sdist optimization for legacy ``setup.py``\-based projects yet, so we have no operational experience to refer to and it might turn out there are some unknown-unknowns that we'll want to take into account before standardizing an optimization for it here. And since this would be an optional hook anyway, it's just as easy to add later once the exact parameters are better understood. * There was some discussion of extending these hooks to allow a single source tree to produce multiple wheels. But this is a complex enough topic that it clearly requires its own PEP. * We also discussed making the wheel and sdist hooks build unpacked directories containing the same contents as their respective archives. In some cases this could avoid the need to pack and unpack an archive, but this seems like premature optimisation. It's advantageous for tools to work with archives as the canonical interchange formats (especially for wheels, where the archive format is already standardised). Close control of archive creation is important for reproducible builds. And it's not clear that tasks requiring an unpacked distribution will be more common than those requiring an archive. =========== Copyright =========== This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -n -- Nathaniel J. Smith -- https://vorpus.org

14 161

Including soabi tag in --user path?
by Nathaniel Smith 16 Jul '17

16 Jul '17

Hi all, A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory. In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like ~/.local/lib/python3.5/site-packages Wouldn't it make more sense for it to be something like ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").) Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7? -n -- Nathaniel J. Smith -- https://vorpus.org

4 11

PEP 420 on python2
by Asmodehn Shade 15 Jul '17

15 Jul '17

Hi everyone, I m very new around here, I just wanted to let you know that while writing a custom importer for py2 and py3, I ended up backporting some python3 libimport code to python2. The code is over there : https://github.com/asmodehn/filefinder2 This way I now support importing packages without __init__.py in python2. It was simpler for my usecase than using pkg_resources or pkgutil. So if anyone is interested in that, let me know, while I still have my head around this. Cheers ! -- AlexV

2 3

Upload documentation to be hosted at ‘pythonhosted.org’
by Ben Finney 14 Jul '17

14 Jul '17

Howdy all, Is ‘pythonhosted.org’ still a recommended service for hosting documentation for Python distributions? It certainly seems so, from the page at <URL:https://pythonhosted.org/>: This site hosts documentation uploaded by authors of packages in the Python Package Index. To upload documentation, go to your package edit page (http://pypi.python.org/pypi?%3Aaction=pkg_edit&name=yourpackage), and fill out the form at the bottom of the page. Should that change for the fact that ‘pypi.python.org’ is in the process of deprecation for uploads? -- \ “If nature has made any one thing less susceptible than all | `\ others of exclusive property, it is the action of the thinking | _o__) power called an idea” —Thomas Jefferson, 1813-08-13 | Ben Finney

2 2

how sdist generation works in enscons
by Daniel Holth 06 Jul '17

06 Jul '17

Here's how sdist generation works in enscons. enscons, a build tool that exists to prototype new Python packaging features, is just a set of tools for SCons that makes it easier to generate wheels and sdists. If targets with certain names exist (sdist, bdist_wheel) then the provide setup.py shim and new PEP 517 wrapper will interoperate with pip. For enscons itself, the sdist build rule looks like sdist = env.SDist(source=FindSourceFiles() + ['PKG-INFO', 'setup.py', 'README.rst', 'CHANGES'])env.Alias('sdist', sdist) Enscons would be able to build its own sdist outside a repository, and it would include the listed files plus everything that is used to build the other targets (the wheel). A different package runs 'hg manifest' in a subprocess to feed to env.SDist(), a natural thing to do in this kind of build system. That one would not be able to build another sdist from an unpacked sdist. Enscons itself doesn't have a way to know whether sdist generation will succeed other than trying to run 'SCons sdist' against the user provided build script.

2 1

Finishing up PEP 517
by Thomas Kluyver 29 Jun '17

29 Jun '17

Yes, it's PEP 517 again! Here's the current text: https://www.python.org/dev/peps/pep-0517/ We currently say: > All other hooks [except get_build_requires] are executed in an environment which contains both the bootstrap requirements specified in the pyproject.toml hook and those specified by the get_build_requires hook. It's not clear to me whether this should be required for the build_sdist and prepare_build_files hooks, nor whether there are any adverse consequences of specifying it like this anyway. Thoughts? I'm going to start putting together a wrapper to call the PEP517 hooks in a subprocess, which could be used by frontends. Thomas

8 48

PEP 518 build dependencies reinstalled unconditionally?
by Pauli Virtanen 29 Jun '17

29 Jun '17

Hi, PEP 518 introduces build requirements that pip installs before calling setup.py. As it currently appears to be, pip installs them to a temporary environment used during the build, with "pip install --ignore-installed ...", so that packages given in pyproject.toml are reinstalled temporarily even if they are already available in the system. Better behavior could be to reinstall build-requirements only if the requirement is not already satisfied by system packages. Pip issue: https://github.com/pypa/pip/issues/4582 Numpy is a common build requirement for several packages providing C extensions that deal with numpy arrays. The current preinstallation behavior causes the following problems: 1) Numpy C extension ABI is backward compatible, but not forward compatible. As a consequence, C extensions built using a newer version of Numpy than installed on the system may simply segfault. (If there's a way to mark the version requirement in the built wheel, the end result is incompatible with the system numpy version.) This can easily happen if unsuspecting people put 'requires=numpy' to pyproject.toml. 2) It wastes time and resources unnecessarily: numpy rebuild can be 10 min per pop, plus downloading stuff from internet. -- Pauli Virtanen

1 0

Provide separate development and documentation URLs in PyPI metadata?
by Brett Cannon 28 Jun '17

28 Jun '17

When you go to PyPI.org for a project you will find a link to the "homepage". Now for some projects that's their development site, e.g. GitHub URL. For others it's their documentation site, e.g. Read the Docs. And not all projects link to both from their PyPI page (e.g. yesterday I noticed flit didn't directly link to its doc site, although Thomas fixed this when I pointed it out). So my question/idea is if it would make sense to have separate, explicit development and documentation URLs in the PyPI metadata? For me that would make a project's PyPI page a true homepage as I would know that no matter what I could find where it's developed or where the docs are by going to PyPI. This compares to now where either I gamble and go to PyPI in hopes the developer provided the link there or hope I craft the right search on Google (which based on my search yesterday for [Sphinx makefile] shows I don't always succeed at). Anyway, just an idea I had based on my flit experience yesterday plus a tweet sent to me. (And if PyPI already supports this somehow then Thomas should brace for the feature request from me 😉.)

6 12

Fatal error "Python.h"
by Rick 26 Jun '17

26 Jun '17

Hi Can anyone help? I’m pretty new to Python and Raspberry Pi. When running PIP, I seem to get fatal error: Python.h: No such file or directory <hash>include “Python.h” compilation terminated I get this when attempting to download both Cython and scikit-image. So I’m assuming it is some absent file or path reference on the Raspberry Pi installation I have, rather than in the distribution package? What is this file “Python.h” that it is trying to include?

4 3

Python 3.6
by Ken R. Lewis 25 Jun '17

25 Jun '17

Hello! I am new to Python as I just downloaded it today. I have a script I need to run that has "requests" in it. How do I go about getting this set up? Also, I am on a Windows 7 machine. Here is the code: #Need to install requests package for python #easy_install requests import requests thanks, Ken Ken Lewis Trinity Health I.T. Programmer Lead (701) 858-6423 Cell (701)833-8234 Fax (701) 858-6407 Ken.Lewis(a)trinityhealth.org<mailto:Ken.Lewis@trinityhealth.org> ________________________________ Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is privileged, confidential, or otherwise exempt from disclosure under applicable law. Any unauthorized review, copy, use, disclosure or distribution is prohibited. If you have received this communication in error, please delete it from your system without copying or forwarding it and notify the sender of the error by reply e-mail.

2 1

Can u provide metadata about the PyPI
by Priya Lakshmi 24 Jun '17

24 Jun '17

Respected Python expert I just want a help. Can u provide us the metadata available about the modules in PyPI giant database. For example, we want the number of packages available, the number of downloads for each package, etc.... So we can think of doing some good maintenance work. thank you. G Priyalakshmi Assistant Professor (Selection Grade) Department of Applied Mathematics and Computational Sciences PSG College of Technology Coimbatore

2 1

Re: [Distutils] PEP 517: editable installs
by Nathaniel Smith 17 Jun '17

17 Jun '17

On Jun 16, 2017 11:59 AM, "Daniel Holth" <dholth(a)gmail.com> wrote: I noticed PEP 517 uses a prefix for editable installs. Which part of the pep are you looking at? All I see about editable installs is the note saying that they're deferred to a future pep. -n

3 4

PEP 517: Python 3 finder?
by Daniel Holth 17 Jun '17

17 Jun '17

Build systems should be able to run under a different version of Python than the one that is running 'pip install'. Does PEP 517 have anything to say about that? Then a flit back end could have a small amount of Python 2.7 compatible interface code and create the wheel with Python 3 anyway.

6 6

PEP 517: editable installs
by Daniel Holth 16 Jun '17

16 Jun '17

I noticed PEP 517 uses a prefix for editable installs. What enscons needs for development install, which is cribbed from setuptools, is to perform a build-in-place and write to a directory that accepts Python .pth files, essentially adding the build-in-place directory to PYTHONPATH. Enscons uses the 'purelib' directory for editable installs. It adds the source directory of the development package to the path and writes an .egg-link to the 'purelib' directory. pip knows how to uninstall these. The code is in enscons.setup.

1 0

PEP 517: Open questions around artifact export directories
by Nick Coghlan 16 Jun '17

16 Jun '17

Hi folks, Thanks for the recent discussions around the PEP 517 build hooks PEP - I think they've clarified and improved the PEP in several areas and are setting us up well for an interoperability API that is both flexible and straightforward for frontends and backends to support. However, I think they've also raised some new still open questions, which I'd like to discuss specifically in this thread. 1. Should backends create artifacts directly, or simply export their contents, and let frontends assemble them into redistributable archives? 2. Are backends responsible in any way for naming artifacts, or should that responsibility lie entirely with frontends? For the first question, in the latest PEP draft, we have: - the sdist hook is `export_sdist`, with the expected output being a filesystem directory - the metadata hook is `get_wheel_metadata`, with the expected output being the contents of the metadata directory - the wheel hook is `build_wheel`, with the expected output being an already assembled binary wheel archive - the build preparation hook is `prepare_build_files`, with the expected output being the subset of files specfically needed to assemble a full wheel file with `build_wheel` For the second question: - the sdist hook is currently unclear on whether the backend should write the sdist contents directly into the frontend supplied directory, or into a suitably named subdirectory - the metadata hook is explicit that the backend should write the metadata directly into the given directory - the wheel hook is explicit that the backend should create already assembled wheel archives in the given directory - the build preparation hook is explicit that the backend should write the metadata directly into the given directory When considered as a unified specification this seems confusingly inconsistent to me, although understandable given that we started out covering only the well specified wheel format, and eventually expanded it to cover the substantially less well specified notions of sdist archives and temporary build directories. While I'm open to being persuaded otherwise, my current thinking is that these concerns could be suitably addressed by making the following amendments: 1. Move all artifact naming responsibilities to the frontend. All backend hooks would just write files in the defined format into the directory supplied by the frontend (and those directories would have no defined naming scheme - they're just a staging area for the backend to pass content to the frontend) 2. Change the names of the artifact hooks to be "export_sdist_content" and "export_wheel_content" to emphasise that backends are responsible for generating the artifact *contents*, not the artifacts themselves 3. Change the optional metadata preparation hook to be "prepare_wheel_metadata" for consistency with "prepare_build_files". This gives us a consistent naming scheme where the "export_*" hooks are required, but each has an optional "prepare_*" counterpart that backends may implement as a more efficient alternative when the full artifact contents aren't (or may not be) needed. Assuming we add support for building multiple wheels from a single source archive in the future, we'd do this through some kind of declarative API based mechanism like adding a new optional "wheel_name" parameter to `prepare_wheel_metadata` and `export_wheel_content` (to tell the backend which wheel the frontend is interested in), along with an additional field in [build-system] to tell frontends which secondary wheels are defined by that project. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 55