Mailman 3 November 2015 - Distutils-SIG

PyPi not allowing duplicate filenames
by Chris Barker April 20, 2020

April 20, 2020

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 24

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs April 17, 2020

April 17, 2020

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Surviving a Compromise of PyPI - PEP 458 and 480
by Vladimir Diaz Feb. 12, 2020

Feb. 12, 2020

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

7 53

Upgrading dependencies
by Charlie Moad April 29, 2018

April 29, 2018

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby April 29, 2018

April 29, 2018

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet April 29, 2018

April 29, 2018

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood April 29, 2018

April 29, 2018

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee April 29, 2018

April 29, 2018

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee April 29, 2018

April 29, 2018

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann April 29, 2018

April 29, 2018

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward April 29, 2018

April 29, 2018

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

Versioned trove classifiers for Django
by James Bennett April 6, 2017

April 6, 2017

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

namespace_package
by KP Sept. 12, 2016

Sept. 12, 2016

I'm not sure where the issue is, but when I specify a namespace_package in the setup.py file, I can indeed have multiple packages with the same base (foo.bar, foo.blah, etc...). The files all install in to the same directory. It drops the foo/__init__.py that would be doing the extend_path, and instead adds a ".pth" file that is a bit over my head. The problem is that it does not seem to traverse the entire sys.path to find multiple foo packages. If I do not specify namespace_packages and instead just use the pkgutil.extend_path, then this seems to allow the packages to be in multiple places in the sys.path. Is there something additional for the namespace_package that i need to specify in order for all of the sys.path to be checked? I'm using 18.5 setuptools....but I am not sure if this somehow ties in to wheel/pip, since I'm using that for the actual install.

4 19

Missing IPv6 support on pypi.python.org
by Baptiste Jonglez July 13, 2016

July 13, 2016

Hi, pypi.python.org is currently not reachable over IPv6. I know this issue was brought up before [1,2]. This is a real issue for us, because our backend servers are IPv6-only (clients never need to talk to backend servers, they go through IPv4-enabled HTTP frontends). So, deploying packages from pypi on the IPv6-only servers is currently a pain. What is the roadmap to add IPv6 support? It seems that Fastly has already deployed IPv6 [3]. Thanks, Baptiste [1] https://mail.python.org/pipermail/distutils-sig/2014-June/024465.html [2] https://bitbucket.org/pypa/pypi/issues/90/missing-ipv6-connectivity [3] http://bgp.he.net/AS54113#_prefixes6

4 6

build system abstraction PEP, take #2
by Robert Collins Dec. 10, 2015

Dec. 10, 2015

This is the update I promised...full prose, since enough has changed that a diff would be noise. :PEP: XX :Title: Build system abstraction for pip/conda etc :Version: $Revision$ :Last-Modified: $Date$ :Author: Robert Collins <rbtcollins(a)hp.com>, Nathaniel Smith <njs(a)pobox.com> :BDFL-Delegate: Donald Stufft <donald(a)stufft.io> :Discussions-To: distutils-sig <distutils-sig(a)python.org> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 26-Oct-2015 :Post-History: XX :Requires: The new dependency-specification PEP. Abstract ======== This PEP specifies a programmatic interface for pip [#pip]_ and other distribution or installation tools to use when working with Python source trees (both the developer tree - e.g. the git tree - and source distributions). The programmatic interface allows decoupling of pip from its current hard dependency on setuptools [#setuptools]_ able for two key reasons: 1. It enables new build systems that may be much easier to use without requiring them to even appear to be setuptools. 2. It facilitates setuptools itself changing its user interface without breaking pip, giving looser coupling. The interface needed to permit pip to install build systems also enables pip to install build time requirements for packages which is an important step in getting pip to full feature parity with the installation components of easy-install. As PEP-426 [#pep426]_ is draft, we cannot utilise the metadata format it defined. However PEP-427 wheels are in wide use and fairly well specified, so we have adopted the METADATA format from that for specifying distribution dependencies. However something was needed for communicating bootstrap requirements and build requirements - but a thin JSON schema is sufficient when overlaid over the new dependency specification PEP. Motivation ========== There is significant pent-up frustration in the Python packaging ecosystem around the current lock-in between build system and pip. Breaking that lock-in is better for pip, for setuptools, and for other build systems like flit [#flit]_. Specification ============= Overview -------- Build tools will be located by reading a file ``pypa.json`` from the root directory of the source tree. That file describes how to get the build tool and the name of the command to run to invoke the tool. All tools will be expected to conform to a single command line interface modelled on pip's existing use of the setuptools setup.py interface. pypa.json --------- The file ``pypa.json`` acts as neutron configuration file for pip and other tools that want to build source trees to consult for configuration. The absence of a ``pypa.json`` file in a Python source tree implies a setuptools or setuptools compatible build system. The JSON has the following schema. Extra keys are ignored. schema The version of the schema. This PEP defines version "1". Defaults to "1" when absent. All tools reading the file must error on an unrecognised schema version. bootstrap_requires Optional list of dependency specifications [#dependencyspec] that must be installed before running the build tool. For instance, if using flit, then the requirements might be:: bootstrap_requires: ["flit"] build_command A mandatory Python format string [#strformat]_ describing the command to run. For instance, if using flit then the build command might be:: build_command: ["flit"] If using a command which is a runnable module fred:: {PYTHON} -m fred Process interface ----------------- The command to run is defined by a simple Python format string [#strformat]_. This permits build systems with dedicated scripts and those that are invoked using "python -m somemodule". Processes will be run with the current working directory set to the root of the source tree. When run, processes should not read from stdin - while pip currently runs build systems with stdin connected to it's own stdin, stdout and stderr are redirected and no communication with the user is possible. As usual with processes, a non-zero exit status indicates an error. Available variables ------------------- PYTHON The Python interpreter in use. This is important to enable calling things which are just Python entry points. ${PYTHON} -m foo Subcommands ----------- There are a number of separate subcommands that build systems must support. The examples below use a build_command of ``flit`` for illustrative purposes. build_requires Query build requirements. Build requirements are returned as a JSON document with one key ``build_requires`` consisting of a list of dependency specifications [#dependencyspec]_. Additional keys must be ignored. The build_requires command is the only command run without setting up a build environment. Example command:: flit build_requires metadata Query project metadata. The metadata and only the metadata should be output on stdout. pip would run metadata just once to determine what other packages need to be downloaded and installed. The metadata is output as a wheel METADATA file per PEP-427 [#pep427]_. Note that the metadata generated by the metadata command, and the metadata present in a generated wheel must be identical. Example command:: flit metadata wheel -d OUTPUT_DIR Command to run to build a wheel of the project. OUTPUT_DIR will point to an existing directory where the wheel should be output. Stdout and stderr have no semantic meaning. Only one file should be output - if more are output then pip would pick an arbitrary one to consume. Example command:: flit wheel -d /tmp/pip-build_1234 develop Command to do an in-place 'development' installation of the project. Stdout and stderr have no semantic meaning. Not all build systems will be able to perform develop installs. If a build system cannot do develop installs, then it should error when run. Note that doing so will cause use operations like ``pip install -e foo`` to fail. The build environment --------------------- Except for the build_requires command, all commands are run within a build environment. No specific implementation is required, but a build environment must achieve the following requirements. 1. All dependencies specified by the project's build_requires must be available for import from within ``$PYTHON``. 1. All command-line scripts provided by the build-required packages must be present in ``$PATH``. A corollary of this is that build systems cannot assume access to any Python package that is not declared as a build_requires or in the Python standard library. Hermetic builds --------------- This specification does not prescribe whether builds should be hermetic or not. Existing build tools like setuptools will use installed versions of build time requirements (e.g. setuptools_scm) and only install other versions on version conflicts or missing dependencies. However its likely that better consistency can be created by always isolation builds and using only the specified dependencies. However there are nuanced problems there - such as how can users force the avoidance of a bad version of a build requirement which meets some packages dependencies. Future PEPs may tackle this problem, but it is not currently in scope - it does not affect the metadata required to coordinate between build systems and things that need to do builds, and thus is not PEP material. Upgrades -------- 'pypa.json' is versioned to permit future changes without requiring compatibility. The sequence for upgrading either of schemas in a new PEP will be: 1. Issue new PEP defining an updated schema. If the schema is not entirely backward compatible then a new version number must be defined. 2. Consumers (e.g. pip) implement support for the new schema version. 3. Package authors opt into the new schema when they are happy to introduce a dependency on the version of 'pip' (and potentially other consumers) that introduced support for the new schema version. The *same* process will take place for the initial deployment of this PEP:- the propogation of the capability to use this PEP without a `setuptools shim`_ will be largely gated by the adoption rate of the first version of pip that supports it. Static metadata in sdists ------------------------- This PEP does not tackle the current inability to trust static metadata in sdists. That is a separate problem to identifying and consuming the build system that is in use in a source tree, whether it came from an sdist or not. Handling of compiler options ---------------------------- Handling of different compiler options is out of scope for this specification. pip currently handles compiler options by appending user supplied strings to the command line it runs when running setuptools. This approach is sufficient to work with the build system interface defined in this PEP, with the exception that globally specified options will stop working globally as different build systems evolve. That problem can be solved in pip (or conda or other installers) without affecting interoperability. In the long term, wheels should be able to express the difference between wheels built with one compiler or options vs another, and that is PEP material. Examples ======== An example 'pypa.json' for using flit:: {"bootstrap_requires": ["flit"], "build_command": ["flit"]} When 'pip' reads this it would prepare an environment with flit in it before trying to use flit. Because flit doesn't have setup-requires support today, `flit build_requires` would just output a constant string:: {"build_requires": []} `flit metadata` would interrogate `flit.ini` and marshal the metadata into a wheel METADATA file and output that on stdout. `flit wheel` would need to accept a `-d` parameter that tells it where to output the wheel (pip needs this). Backwards Compatibility ======================= Older pips will remain unable to handle alternative build systems. This is no worse than the status quo - and individual build system projects can decide whether to include a shim ``setup.py`` or not. All existing build systems that can product wheels and do develop installs should be able to run under this abstraction and will only need a specific adapter for them constructed and published on PyPI. In the absence of a ``pypa.json`` file, tools like pip should assume a setuptools build system and use setuptools commands directly. Network effects --------------- Projects that adopt build systems that are not setuptools compatible - that is that they have no setup.py, or the setup.py doesn't accept commands that existing tools try to use - will not be installable by those existing tools. Where those projects are used by other projects, this effect will cascade. In particular, because pip does not handle setup-requires today, any project (A) that adopts a setuptools-incompatible build system and is consumed as a setup-requirement by a second project (B) which has not itself transitioned to having a pypa.json will make B uninstallable by any version of pip. This is because setup.py in B will trigger easy-install when 'setup.py egg_info' is run by pip, and that will try and fail to install A. As such we recommend that tools which are currently used as setup-requires either ensure that they keep a `setuptools shim`_ or find their consumers and get them all to upgrade to the use of a `pypa.json` in advance of moving themselves. Pragmatically that is impossible, so the advice is to keep a setuptools shim indefinitely - both for projects like pbr, setuptools_scm and also projects like numpy. setuptools shim --------------- It would be possible to write a generic setuptools shim that looks like ``setup.py`` and under the hood uses ``pypa.json`` to drive the builds. This is not needed for pip to use the system, but would allow package authors to use the new features while still retaining compatibility with older pip versions. Rationale ========= This PEP started with a long mailing list thread on distutils-sig [#thread]_. Subsequent to that a online meeting was held to debug all the positions folk had. Minutes from that were posted to the list [#minutes]_. This specification is a translation of the consensus reached there into PEP form, along with some arbitrary choices on the minor remaining questions. The basic heuristic for the design has to been to focus on introducing an abstraction without requiring development not strictly tied to the abstraction. Where the gap is small to improvements, or the cost of using the existing interface is very high, then we've taken on having the improvement as a dependency, but otherwise defered such to future iterations. We chose wheel METADATA files rather than defining a new specification, because pip can already handle wheel .dist-info directories which encode all the necessary data in a METADATA file. PEP-426 can't be used as it's still draft, and defining a new metadata format, while we should do that, is a separate problem. Using a directory on disk would not add any value to the interface (pip has to do that today due to limitations in the setuptools CLI). The use of 'develop' as a command is because there is no PEP specifying the interoperability of things that do what 'setuptools develop' does - so we'll need to define that before pip can take on the responsibility for doing the 'develop' step. Once thats done we can issue a successor PEP to this one. The use of a command line API rather than a Python API is a little contentious. Fundamentally anything can be made to work, and the pip maintainers have spoken strongly in favour of retaining a process based interface - something that is mature and robust in pip today. The choice of JSON as a file format is a compromise between several constraints. Firstly there is no stdlib YAML interpreter, nor one for any of the other low-friction structured file formats. Secondly, INIParser is a poor format for a number of reasons, primarily that it has very minimal structure - but pip's maintainers are not fond of it. JSON is in the stdlib, has sufficient structure to permit embedding anything we want in future without requiring embedded DSL's. Donald suggested using ``setup.cfg`` and the existing setuptools command line rather than inventing something new. While that would permit interoperability with less visible changes, it requires nearly as much engineering on the pip side - looking for the new key in setup.cfg, implementing the non-installed environments to run the build in. And the desire from other build system authors not to confuse their users by delivering something that looks like but behaves quite differently to setuptools seems like a bigger issue than pip learning how to invoke a custom build tool. The metadata and wheel commands are required to have consistent metadata to avoid a race condition that could otherwise happen where pip reads the metadata, acts on it, and then the resulting wheel has incompatible requirements. That race is exploited today by packages using PEP-426 environment markers, to work with older pip versions that do not support environment markers. That exploit is not needed with this PEP, because either the setuptools shim is in use (with older pip versions), or an environment marker ready pip is in use. The setuptools shim can take care of exploiting the difference older pip versions require. References ========== .. [#pip] pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/) .. [#setuptools] setuptools, the defacto Python package build system (https://pythonhosted.org/setuptools/) .. [#flit] flit, a simple way to put packages in PyPI (http://flit.readthedocs.org/en/latest/) .. [#pypi] PyPI, the Python Package Index (https://pypi.python.org/) .. [#shellvars] Shellvars, an implementation of shell variable rules for Python. (https://github.com/testing-cabal/shellvars) .. [#pep426] PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/) .. [#pep427] PEP-427, Python distribution metadata. (https://www.python.org/dev/peps/pep-0427/) .. [#thread] The kick-off thread. (https://mail.python.org/pipermail/distutils-sig/2015-October/026925.html) .. [#minutes] The minutes. (https://mail.python.org/pipermail/distutils-sig/2015-October/027214.html) .. [#strformat] The Python string formatting syntax. (https://docs.python.org/3.1/library/string.html#format-string-syntax) .. [#dependencyspec] Dependency specification language PEP. (https://github.com/pypa/interoperability-peps/pull/56) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

8 18

Installing packages using pip
by Ines Barata Dec. 9, 2015

Dec. 9, 2015

Hello, I want to install the following version of pygame in my windows10.pro:install pygame-1.9.2a0-cp33-none-win_amd64.whl, but the file is in *.whl* format which I dont have any program to open it I checked the information in https://docs.python.org/3/installing/ to understand how to use pip. But didn't make it... My result is the following, using *python3.5.0 IDLE*: >>> pip *install* pygame-1.9.2a0-cp34-none-win32.whl SyntaxError: invalid syntax >>> pip *pygame*-1.9.2a0-cp34-none-win32.whl SyntaxError: invalid syntax >>> python -m *pip* install pygame-1.9.2a0-cp33-none-win_amd64.whl SyntaxError: invalid syntax >>> python3.*5*.0 -m pip install pygame-1.9.2a0-cp33-none-win_amd64.whl SyntaxError: invalid syntax >>> Can you help me solve my problem?

15 30

distutils.command.build_clib: How to add additional compiler flags for cl.exe?
by Kim Walisch Dec. 3, 2015

Dec. 3, 2015

Hi, For distutils.command.build_clib the commonly used code below does not work for adding additional compiler flags (it works using distutils.command.build_ext): extra_compile_args = '-fopenmp' On Unix-like system I found a workaround which allows to specify additional compiler flags for distutils.command.build_clib: cflags = distutils.sysconfig.get_config_var('CFLAGS') distutils.sysconfig._config_vars['CFLAGS'] = cflags + " -fopenmp" Unfortunately this does not work with Microsoft's C/C++ compiler cl.exe. Does anybody know how I can add additional compiler flags for cl.exe and distutils.command.build_clib? Thanks and best regards!

2 1

FINAL DRAFT: Dependency specifier PEP
by Robert Collins Nov. 27, 2015

Nov. 27, 2015

:PEP: XX :Title: Dependency specification for Python Software Packages :Version: $Revision$ :Last-Modified: $Date$ :Author: Robert Collins <rbtcollins(a)hp.com> :BDFL-Delegate: Donald Stufft <donald(a)stufft.io> :Discussions-To: distutils-sig <distutils-sig(a)python.org> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 11-Nov-2015 :Post-History: XX Abstract ======== This PEP specifies the language used to describe dependencies for packages. It draws a border at the edge of describing a single dependency - the different sorts of dependencies and when they should be installed is a higher level problem. The intent is provide a building block for higher layer specifications. The job of a dependency is to enable tools like pip [#pip]_ to find the right package to install. Sometimes this is very loose - just specifying a name, and sometimes very specific - referring to a specific file to install. Sometimes dependencies are only relevant in one platform, or only some versions are acceptable, so the language permits describing all these cases. The language defined is a compact line based format which is already in widespread use in pip requirements files, though we do not specify the command line option handling that those files permit. There is one caveat - the URL reference form, specified in PEP-440 [#pep440]_ is not actually implemented in pip, but since PEP-440 is accepted, we use that format rather than pip's current native format. Motivation ========== Any specification in the Python packaging ecosystem that needs to consume lists of dependencies needs to build on an approved PEP for such, but PEP-426 [#pep426]_ is mostly aspirational - and there are already existing implementations of the dependency specification which we can instead adopt. The existing implementations are battle proven and user friendly, so adopting them is arguably much better than approving an aspirational, unconsumed, format. Specification ============= Examples -------- All features of the language shown with a name based lookup:: requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7.10" A minimal URL based lookup:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346… Concepts -------- A dependency specification always specifies a distribution name. It may include extras, which expand the dependencies of the named distribution to enable optional features. The version installed can be controlled using version limits, or giving the URL to a specific artifact to install. Finally the dependency can be made conditional using environment markers. Grammar ------- We first cover the grammar briefly and then drill into the semantics of each section later. A distribution specification is written in ASCII text. We use a parsley [#parsley]_ grammar to provide a precise grammar. It is expected that the specification will be embedded into a larger system which offers framing such as comments, multiple line support via continuations, or other such features. The full grammar including annotations to build a useful parse tree is included at the end of the PEP. Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note: URI is defined in std-66 [#std66]_:: version_cmp = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '===' version = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+ version_one = version_cmp version wsp* version_many = version_one (wsp* ',' version_one)* versionspec = ( '(' version_many ')' ) | version_many urlspec = '@' wsp* <URI_reference> Environment markers allow making a specification only take effect in some environments:: marker_op = version_cmp | 'in' | 'not' wsp+ 'in' python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' | '-' | '_' | '*') dquote = '"' squote = '\\'' python_str = (squote (python_str_c | dquote)* squote | dquote (python_str_c | squote)* dquote) env_var = ('python_version' | 'python_full_version' | 'os_name' | 'sys_platform' | 'platform_release' | 'platform_system' | 'platform_version' | 'platform_machine' | 'python_implementation' | 'implementation_name' | 'implementation_version' | 'extra' # ONLY when defined by a containing layer ) marker_var = env_var | python_str marker_expr = ('(' wsp* marker wsp* ')' | (marker_var wsp* marker_op wsp* marker_var)) marker = wsp* marker_expr ( wsp* ('and' | 'or') wsp* marker_expr)* quoted_marker = ';' wsp* marker Optional components of a distribution may be specified using the extras field:: identifier = letterOrDigit ( letterOrDigit | (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )* name = identifier extras_list = identifier (wsp* ',' wsp* identifier)* extras = '[' wsp* extras_list? wsp* ']' Giving us a rule for name based requirements:: name_req = name wsp* extras? wsp* versionspec? wsp* quoted_marker? And a rule for direct reference specifications:: url_req = name wsp* extras? wsp* urlspec wsp+ quoted_marker? Leading to the unified rule that can specify a dependency.:: specification = wsp* ( url_req | name_req ) wsp* Whitespace ---------- Non line-breaking whitespace is mostly optional with no semantic meaning. The sole exception is detecting the end of a URL requirement. Names ----- Python distribution names are currently defined in PEP-345 [#pep345]_. Names act as the primary identifier for distributions. They are present in all dependency specifications, and are sufficient to be a specification on their own. However, PyPI places strict restrictions on names - they must match a case insensitive regex or they won't be accepted. Accordingly in this PEP we limit the acceptable values for identifiers to that regex. A full redefinition of name may take place in a future metadata PEP:: ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ Extras ------ An extra is an optional part of a distribution. Distributions can specify as many extras as they wish, and each extra results in the declaration of additional dependencies of the distribution **when** the extra is used in a dependency specification. For instance:: requests[security] Extras union in the dependencies they define with the dependencies of the distribution they are attached to. The example above would result in requests being installed, and requests own dependencies, and also any dependencies that are listed in the "security" extra of requests. If multiple extras are listed, all the dependencies are unioned together. Versions -------- See PEP-440 [#pep440]_ for more detail on both version numbers and version comparisons. Version specifications limit the versions of a distribution that can be used. They only apply to distributions looked up by name, rather than via a URL. Version comparison are also used in the markers feature. The optional brackets around a version are present for compatibility with PEP-345 [#pep345]_ but should not be generated, only accepted. Environment Markers ------------------- Environment markers allow a dependency specification to provide a rule that describes when the dependency should be used. For instance, consider a package that needs argparse. In Python 2.7 argparse is always present. On older Python versions it has to be installed as a dependency. This can be expressed as so:: argparse;python_version<"2.7" A marker expression evalutes to either True or False. When it evaluates to False, the dependency specification should be ignored. The marker language is a subset of Python itself, chosen for the ability to safely evaluate it without running arbitrary code that could become a security vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This PEP fixes some issues that were observed in the design described in PEP-426 [#pep426]_. Comparisons in marker expressions are typed by the comparison operator. The <marker_op> operators that are not in <version_cmp> perform the same as they do for strings in Python. The <version_cmp> operators use the PEP-440 [#pep440]_ version comparison rules when those are defined (that is when both sides have a valid version specifier). If there is no defined PEP-440 behaviour and the operator exists in Python, then the operator falls back to the Python behaviour. Otherwise an error should be raised. e.g. the following will result in errors:: "dog" ~= "fred" python_version ~= "surprise" User supplied constants are always encoded as strings with either ``'`` or ``"`` quote marks. Note that backslash escapes are not defined, but existing implementations do support them. They are not included in this specification because they add complexity and there is no observable need for them today. Similarly we do not define non-ASCII character support: all the runtime variables we are referencing are expected to be ASCII-only. The variables in the marker grammar such as "os_name" resolve to values looked up in the Python runtime. With the exception of "extra" all values are defined on all Python versions today - it is an error in the implementation of markers if a value is not defined. Unknown variables must raise an error rather than resulting in a comparison that evaluates to True or False. Variables whose value cannot be calculated on a given Python implementation should evaluate to ``0`` for versions, and an empty string for all other variables. The "extra" variable is special. It is used by wheels to signal which specifications apply to a given extra in the wheel ``METADATA`` file, but since the ``METADATA`` file is based on a draft version of PEP-426, there is no current specification for this. Regardless, outside of a context where this special handling is taking place, the "extra" variable should result in an error like all other unknown variables. .. list-table:: :header-rows: 1 * - Marker - Python equivalent - Sample values * - ``os_name`` - ``os.name`` - ``posix``, ``java`` * - ``sys_platform`` - ``sys.platform`` - ``linux``, ``linux2``, ``darwin``, ``java1.8.0_51`` (note that "linux" is from Python3 and "linux2" from Python2) * - ``platform_machine`` - ``platform.machine()`` - ``x86_64`` * - ``python_implementation`` - ``platform.python_implementation()`` - ``CPython``, ``Jython`` * - ``platform_release`` - ``platform.release()`` - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51`` * - ``platform_system`` - ``platform.system()`` - ``Linux``, ``Windows``, ``Java`` * - ``platform_version`` - ``platform.version()`` - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014`` ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation`` ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015; root:xnu-2782.40.9~2/RELEASE_X86_64`` * - ``python_version`` - ``platform.python_version()[:3]`` - ``3.4``, ``2.7`` * - ``python_full_version`` - ``platform.python_version()`` - ``3.4.0``, ``3.5.0b1`` * - ``implementation_name`` - ``sys.implementation.name`` - ``cpython`` * - ``implementation_version`` - see definition below - ``3.4.0``, ``3.5.0b1`` * - ``extra`` - An error except when defined by the context interpreting the specification. - ``test`` The ``implementation_version`` marker variable is derived from ``sys.implementation.version``:: def format_full_version(info): version = '{0.major}.{0.minor}.{0.micro}'.format(info) kind = info.releaselevel if kind != 'final': version += kind[0] + str(info.serial) return version if hasattr(sys, 'implementation'): implementation_version = format_full_version(sys.implementation.version) else: implementation_version = "0" Backwards Compatibility ======================= Most of this PEP is already widely deployed and thus offers no compatibiltiy concerns. There are however a few points where the PEP differs from the deployed base. Firstly, PEP-440 direct references haven't actually been deployed in the wild, but they were designed to be compatibly added, and there are no known obstacles to adding them to pip or other tools that consume the existing dependency metadata in distributions - particularly since they won't be permitted to be present in PyPI uploaded distributions anyway. Secondly, PEP-426 markers which have had some reasonable deployment, particularly in wheels and pip, will handle version comparisons with ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is less than "2.7.9". This backward incompatibility is deliberate. We are also defining new operators - "~=" and "===", and new variables - ``platform_release``, ``platform_system``, ``implementation_name``, and ``implementation_version`` which are not present in older marker implementations. The variables will error on those implementations. Users of both features will need to make a judgement as to when support has become sufficiently widespread in the ecosystem that using them will not cause compatibility issues. Thirdly, PEP-345 required brackets around version specifiers. In order to accept PEP-345 dependency specifications, brackets are accepted, but they should not be generated. Rationale ========= In order to move forward with any new PEPs that depend on environment markers, we needed a specification that included them in their modern form. This PEP brings together all the currently unspecified components into a specified form. The requirement specifier was adopted from the EBNF in the setuptools pkg_resources documentation, since we wish to avoid depending on a defacto, vs PEP specified, standard. Complete Grammar ================ The complete parsley grammar:: wsp = ' ' | '\t' version_cmp = wsp* <'<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='> version = wsp* <( letterOrDigit | '-' | '_' | '.' | '*' | '+' | '!' )+> version_one = version_cmp:op version:v wsp* -> (op, v) version_many = version_one:v1 (wsp* ',' version_one)*:v2 -> [v1] + v2 versionspec = ('(' version_many:v ')' ->v) | version_many urlspec = '@' wsp* <URI_reference> marker_op = version_cmp | 'in' | 'not' wsp+ 'in' python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' | '-' | '_' | '*' | '#') dquote = '"' squote = '\\'' python_str = (squote <(python_str_c | dquote)*>:s squote | dquote <(python_str_c | squote)*>:s dquote) -> s env_var = ('python_version' | 'python_full_version' | 'os_name' | 'sys_platform' | 'platform_release' | 'platform_system' | 'platform_version' | 'platform_machine' | 'python_implementation' | 'implementation_name' | 'implementation_version' | 'extra' # ONLY when defined by a containing layer ):varname -> lookup(varname) marker_var = env_var | python_str marker_expr = (("(" wsp* marker:m wsp* ")" -> m) | ((marker_var:l wsp* marker_op:o wsp* marker_var:r)) -> (l, o, r)) marker = (wsp* marker_expr:m ( wsp* ("and" | "or"):o wsp* marker_expr:r -> (o, r))*:ms -> (m, ms)) quoted_marker = ';' wsp* marker identifier = <letterOrDigit ( letterOrDigit | (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )*> name = identifier extras_list = identifier:i (wsp* ',' wsp* identifier)*:ids -> [i] + ids extras = '[' wsp* extras_list?:e wsp* ']' -> e name_req = (name:n wsp* extras?:e wsp* versionspec?:v wsp* quoted_marker?:m -> (n, e or [], v or [], m)) url_req = (name:n wsp* extras?:e wsp* urlspec:v wsp+ quoted_marker?:m -> (n, e or [], v, m)) specification = wsp* ( url_req | name_req ):s wsp* -> s # The result is a tuple - name, list-of-extras, # list-of-version-constraints-or-a-url, marker-ast or None URI_reference = <URI | relative_ref> URI = scheme ':' hier_part ('?' query )? ( '#' fragment)? hier_part = ('//' authority path_abempty) | path_absolute | path_rootless | path_empty absolute_URI = scheme ':' hier_part ( '?' query )? relative_ref = relative_part ( '?' query )? ( '#' fragment )? relative_part = '//' authority path_abempty | path_absolute | path_noscheme | path_empty scheme = letter ( letter | digit | '+' | '-' | '.')* authority = ( userinfo '@' )? host ( ':' port )? userinfo = ( unreserved | pct_encoded | sub_delims | ':')* host = IP_literal | IPv4address | reg_name port = digit* IP_literal = '[' ( IPv6address | IPvFuture) ']' IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+ IPv6address = ( ( h16 ':'){6} ls32 | '::' ( h16 ':'){5} ls32 | ( h16 )? '::' ( h16 ':'){4} ls32 | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32 | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32 | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32 | ( ( h16 ':'){0,4} h16 )? '::' ls32 | ( ( h16 ':'){0,5} h16 )? '::' h16 | ( ( h16 ':'){0,6} h16 )? '::' ) h16 = hexdig{1,4} ls32 = ( h16 ':' h16) | IPv4address IPv4address = dec_octet '.' dec_octet '.' dec_octet '.' Dec_octet nz = ~'0' digit dec_octet = ( digit # 0-9 | nz digit # 10-99 | '1' digit{2} # 100-199 | '2' ('0' | '1' | '2' | '3' | '4') digit # 200-249 | '25' ('0' | '1' | '2' | '3' | '4' | '5') )# %250-255 reg_name = ( unreserved | pct_encoded | sub_delims)* path = ( path_abempty # begins with '/' or is empty | path_absolute # begins with '/' but not '//' | path_noscheme # begins with a non-colon segment | path_rootless # begins with a segment | path_empty ) # zero characters path_abempty = ( '/' segment)* path_absolute = '/' ( segment_nz ( '/' segment)* )? path_noscheme = segment_nz_nc ( '/' segment)* path_rootless = segment_nz ( '/' segment)* path_empty = pchar{0} segment = pchar* segment_nz = pchar+ segment_nz_nc = ( unreserved | pct_encoded | sub_delims | '@')+ # non-zero-length segment without any colon ':' pchar = unreserved | pct_encoded | sub_delims | ':' | '@' query = ( pchar | '/' | '?')* fragment = ( pchar | '/' | '?')* pct_encoded = '%' hexdig unreserved = letter | digit | '-' | '.' | '_' | '~' reserved = gen_delims | sub_delims gen_delims = ':' | '/' | '?' | '#' | '(' | ')?' | '@' sub_delims = '!' | '$' | '&' | '\\'' | '(' | ')' | '*' | '+' | ',' | ';' | '=' hexdig = digit | 'a' | 'A' | 'b' | 'B' | 'c' | 'C' | 'd' | 'D' | 'e' | 'E' | 'f' | 'F' A test program - if the grammar is in a string ``grammar``:: import os import sys import platform from parsley import makeGrammar grammar = """ wsp ... """ tests = [ "A", "aa", "name", "name>=3", "name>=3,<2", "name [fred,bar] @ http://foo.com ; python_version=='2.7'", "name[quux, strange];python_version<'2.7' and platform_version=='2'", "name; os_name=='dud' and (os_name=='odd' or os_name=='fred')", "name; os_name=='dud' and os_name=='odd' or os_name=='fred'", ] def format_full_version(info): version = '{0.major}.{0.minor}.{0.micro}'.format(info) kind = info.releaselevel if kind != 'final': version += kind[0] + str(info.serial) return version if hasattr(sys, 'implementation'): implementation_version = format_full_version(sys.implementation.version) implementation_name = sys.implementation.name else: implementation_version = '0' implementation_name = '' bindings = { 'implementation_name': implementation_name, 'implementation_version': implementation_version, 'os_name': os.name, 'platform_machine': platform.machine(), 'platform_release': platform.release(), 'platform_system': platform.system(), 'platform_version': platform.version(), 'python_full_version': platform.python_version(), 'python_implementation': platform.python_implementation(), 'python_version': platform.python_version()[:3], 'sys_platform': sys.platform, } compiled = makeGrammar(grammar, {'lookup': bindings.__getitem__}) for test in tests: parsed = compiled(test).specification() print(parsed) References ========== .. [#pip] pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/) .. [#pep345] PEP-345, Python distribution metadata version 1.2. (https://www.python.org/dev/peps/pep-0345/) .. [#pep426] PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/) .. [#pep440] PEP-440, Python distribution metadata. (https://www.python.org/dev/peps/pep-0440/) .. [#std66] The URL specification. (https://tools.ietf.org/html/rfc3986) .. [#parsley] The parsley PEG library. (https://pypi.python.org/pypi/parsley/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

11 50

night build just started failing with TypeError: must be type, not classobj on Python 2.7
by Chris Withers Nov. 24, 2015

Nov. 24, 2015

Hi All, Nightly builds of one of my packages (https://travis-ci.org/Mortar/mortar_rdb) just began failing with: TypeError: must be type, not classobj ...but only on Python 2. The full traceback is here: https://travis-ci.org/Mortar/mortar_rdb/jobs/93030519#L256 I haven't pushed any changed to that project in a few weeks/days. Does anyone know what might have changed to cause this problem? I couldn't see any recent releases of setuptools or pip, so wondering what I'm missing... cheers, Chris

3 4

night build just started failing with TypeError: must be type, not classobj on Python 2.7
by Chris Withers Nov. 24, 2015

Nov. 24, 2015

Hi All, Nightly builds of one of my packages (https://travis-ci.org/Mortar/mortar_rdb) just began failing with: TypeError: must be type, not classobj ...but only on Python 2. The full traceback is here: https://travis-ci.org/Mortar/mortar_rdb/jobs/93030519#L256 I haven't pushed any changed to that project in a few weeks/days. Does anyone know what might have changed to cause this problem? I couldn't see any recent releases of setuptools or pip, so wondering what I'm missing... cheers, Chris

1 0

New Design Landed in Warehouse
by Donald Stufft Nov. 23, 2015

Nov. 23, 2015

As many of you may know, I’ve been working on Warehouse which is designed to replace the PyPI code base with something modern and maintainable. That is progressing but we’ve hit a particular milestone that I’m really excited by. With the help of the awesome nlhkabu [1][2] we’ve gotten a (still WIP) design that has now landed in the repository and has been deployed to warehouse.python.org and warehouse-staging.python.org. Check it out and see what you think. If you find any issues feel free to open them up on https://github.com/pypa/warehouse. [1] https://twitter.com/nlhkabu/ [2] https://github.com/nlhkabu/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

11 14

Does PEP 0440 allow omitting the separator for development releases?
by Christopher Wilson Nov. 23, 2015

Nov. 23, 2015

(I posted this to http://stackoverflow.com/questions/33849399/does-pep-0440-allow-omitting-th… <http://stackoverflow.com/questions/33849399/does-pep-0440-allow-omitting-th…> and was recommended to send an inquiry to this mailer) PEP 0440 <https://www.python.org/dev/peps/pep-0440/> includes the following two statements which on first glance seem to be contradictory: Development releases allow a . , - , or a _ separator as well as omitting the separator all together. The normal form of this is with the . separator. This allows versions such as 1.2-dev2 or 1.2dev2 which normalize to 1.2.dev2 . and Note that devN and postN MUST always be preceded by a dot, even when used immediately following a numeric version (e.g. 1.0.dev456 , 1.0.post1 ). Is the 2nd statement true for developers of Python distributions? Or is it stating that the version (once normalized) must always be preceded by a dot? Thanks, Chris

2 1

Re: [Distutils] New Design Landed in Warehouse
by Donald Stufft Nov. 21, 2015

Nov. 21, 2015

> On Nov 21, 2015, at 8:32 AM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > > On Sat, 21 Nov 2015 08:29:43 -0500 > Donald Stufft <donald(a)stufft.io> wrote: >> >> We have an issue for reducing the size of the header at https://github.com/pypa/warehouse/issues/793. >> >> We also have a number of users (of varying skill levels I believe) lined up to do user testing of the new design with. Nicole is handling that but I believe she is planning to start that the beginning of December. If you’re interested in participating with that contacting Nicole to see if she needs any more volunteers for it would be a good idea. > > Great to hear about! I'm ok with participating but I'm not sure where > to contact Nicole (also, how much time is expected from participants?). Ah oops, here’s the Call to Action issue we had https://github.com/pypa/warehouse/issues/717. > >> Feedback is useful, though the best form of it would be to open or comment on issues on the repository so we can track when we’ve either fixed it, or can at least acknowledge it and say that we believe the current system is better for more users. I want to stress that the design is not “finished”, it’s very much a WIP still. Consider this sort of like an alpha or something. > > Thank you. I assumed that some kind of feedback was expected. Apologies > if that was too early. Oh not at all. I just wanted to make sure it was clear that it was still a construction zone of sorts. I think we’ve had some folks (not you or anything) think the design was “done” (in so much that anything with software is ever “done”) and just wanted to call it out explicitly. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

Re: [Distutils] New Design Landed in Warehouse
by Donald Stufft Nov. 21, 2015

Nov. 21, 2015

> On Nov 21, 2015, at 3:11 AM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > > On Fri, 20 Nov 2015 23:51:31 -0800 > Noah Kantrowitz <noah(a)coderanger.net> wrote: >>> >>> Useful content starts only 2/3 down the first page. The large "pip >>> install six" snippet probably doesn't deserve being that proeminent >>> (or being there at all), and is ironically redundant with the "how do >>> I install this?" link just below. >> >> I think you have a highly specialized view of what is "useful content" compared to the average user. > > "Useful content" is the page body, i.e. the project's description, as > opposed to navigation and other layout elements. > > As for "highly specialized", we won't know without a user study. What > does the average user look for on a PyPI project page? My hypothesis is > that they mainly want to learn about the package, i.e. read its > description (perhaps also the list of supported platforms and the > changelog). It makes sense to prioritize the content the user is > looking for. > We have an issue for reducing the size of the header at https://github.com/pypa/warehouse/issues/793. We also have a number of users (of varying skill levels I believe) lined up to do user testing of the new design with. Nicole is handling that but I believe she is planning to start that the beginning of December. If you’re interested in participating with that contacting Nicole to see if she needs any more volunteers for it would be a good idea. I’d also like to stress that prior to, well Yesterday really, this design had not been paired with live data at all and had just lived in a static site. So part of this all is getting it integrated with the site and seeing how it feels with it and what works and what doesn’t. Feedback is useful, though the best form of it would be to open or comment on issues on the repository so we can track when we’ve either fixed it, or can at least acknowledge it and say that we believe the current system is better for more users. I want to stress that the design is not “finished”, it’s very much a WIP still. Consider this sort of like an alpha or something. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

devpi-{server,web}-2.5.0 perf/bug fix releases
by holger krekel Nov. 19, 2015

Nov. 19, 2015

We just released devpi-{server,web}-2.5.0 to pypi, see changelogs below for more details. While it's not required to do an export/import cycle for this release it's recommended especially if you are running with replicas. Docs for the private pypi packaging server at: http://doc.devpi.net Thanks to Florian Schulze, Jason R. Coombs and all issue reporters. For your information, we are now starting work for devpi-server-3.0 which will introduce further speedups, internal code simplifications and new features (like mirroring from arbitrary pypi-servers). cheers, holger krekel server-2.5.0 (2015-11-19) ------------------------- - fix a regression of 2.3.0 which would cause many write-transactions for mirrored simple-page entries that didn't change. Previous to the fix, accesses to mirrored simple pages will result in a new write-transaction every 30 minutes if the page is accessed which is likely on a somewhat busy site. If you running with replicas it is recommended to do an an export/import cycle to remove all the unneccessary writes that were produced since devpi-server-2.3.0. They delay the setup of new replicas considerably. - add info about pypi_whitelist on simple page when root/pypi is blocked for a project. - replica simple-page serving will not unneccessarily wait for new simple-page entries to arrive at the replication side if the master does not return any changes in the initial simple-page request. Previously a replica would wait for the replication-thread to catch up even if no links changed. - fix setup.py to work on py34 and with LANG="C" environments. Thanks Jason R. Coombs. - fix issue284: allow users who are listed in acl_upload to delete packages web-2.5.0 (2015-11-19) ---------------------- - fix issue288: classifiers rendering wrong with read only data views - index.pt, project.pt, version.pt: added info about pypi_whitelist. This requires devpi-server > 2.4.0 to work. - fix issue286: indexing of most data failed due to new read only views

1 0

Packaging shared objects with setuptools
by Mario Pezzoni Nov. 17, 2015

Nov. 17, 2015

Hello, I am wrapping a c++ library with cython. I compile the pyxs and the c++ code with cmake using the cython-cmake-example. If I import the package from the build directory it works flawlessly, if I install through "python setup.py install" into a virtual environment it breaks because the shared objects are not installed (but all the pure .py modules are installed). How do I tell setuptools to install the shared objects? Mario

2 1

Re: [Distutils] Link to lib missing in PEP: FINAL DRAFT: Dependency specifier PEP
by Thomas Güttler Nov. 17, 2015

Nov. 17, 2015

Am 17.11.2015 um 15:13 schrieb Ronny Pfannschmidt: > That is no acceptable way to answer in particular since the original mail continued with places where to put the link I was asking for a link from the PEP to the implementation. I received a "no". I replyed with my opinion. What is not acceptable? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

1 0

Re: [Distutils] New PyPUG Tutorials
by Marcus Smith Nov. 17, 2015

Nov. 17, 2015

> > > To have the most success, the writers will certainly need feedback from > subject matter experts, so the process will include 2 stages where we > specifically ask for feedback from PyPA-Dev and Distutils-Sig: 1) To > validate the initial proposal that covers the scope of the changes, and 2) > to review the actual PRs to PyPUG for accuracy, when it's time for merging. > I'll post again with more details as those stages occur. > So, I'm back to post the initial proposal as mentioned above. The proposal was put together by Daniel Beck (http://www.danieldbeck.com), one of the writers Nicole (from the Warehouse team) was able to rally with her call for volunteers [1]. It includes a *tentative* new outline, but keep in mind that however it turns out, we will maintain redirects for the current critical links, like the tool recommendations page, and the two current tutorials. As work proceeds, it will take place as PRs to the develop branch of the PyPUG (https://github.com/pypa/python-packaging-user-guide/), so people can follow along and help in the technical review process. When the final product is deemed worthy and ready, there will be a final call for review before the merge of develop back into master. Both Daniel and Nicole are now subscribed to distutils-sig, and can respond to any questions, concerns, or feedback. Here's the proposal, as quoted from an email from Daniel: ----------------------------------------- Scope --------- The guide is meant to answer this question: "How do Python, packages, and packaging tools fit together?" The guide's not to meant to replace tools' docs; rather, it's to show when and how these tools and other artifacts work as an ecosystem. Audience ------------- I get the impression that these are the major audience groups we want to help with the guide: - Python users who are at an 'advanced beginner' level (e.g., know how to import from the standard library, but want to learn to install a third-party package for the first time - Python users with intermediate experience who want to make and distribute a package for the first time - Python users with more experience who need a refresher or want to understand what the "one way to do it" is right now - Python users with lots of experience who want to take on some advanced or specialized packaging task Outline ---------- I sketched out a gross outline for the guide. This is not complete, sometimes speculative, and certainly subject to change, but I wanted to give you an overall picture of how I imagine the guide could look when we've had some time to work on this. Again, because the existing content is so good, I don't expect much, if anything, will be deleted—so just because it's not listed here doesn't mean anything. Rather, I expect that stuff will be easier to find. :-) 1. About this guide 2. Getting help 3. Quick references (i.e., five minute tutorials) a. Installing a package b. Making a package c. Publish a package on PyPI 4. Intro to Packaging* a. Who is this for? b. Prereqs (making sure Python, etc. are working) c. What's a package? d. Where do packages come from? e. Installing a package for the first time f. Using a virtualenv g. Making a package for the first time h. Publishing your first package 5. The Packaging Ecosystem (concept docs) a. Overview of tools b. Developing packages c. Distributing packages d. Indices and caches 6. Practices and patterns (task docs) a. Making packages b. Publishing packages c. Using packages 7. History and legacy of packaging a. that whole distutils->setuptools->distribute->setuptools saga b. history of PyPA c. eggs vs wheels d. easy_install vs pip e. cheeseshop vs warehouse 8. Contributing to this guide a. Style guide * This is the only part that I think functions as a continuous unit meant to be read from start to end. I'm imagining something that would work as, for example, a lesson plan for a PyLadies workshop. For the rest of the guide, I want to work with the expectation that every page is a landing page. Timeline ------------ Here's my current perspective on how quickly we can move on these things: - In the next few days: contacting volunteers to let them know that things are happening soon (Nicole) - In the 7 or 8 days: decide on workflow questions (Nicole and Marcus), amend project roles (Nicole and Marcus), and ask for distutils-sig approval (Marcus) - In the next two weeks: start opening issues and inviting volunteers to start writing - By the end of November: most, if not all, existing content could be reorganized into a new outline - By mid-December: have publishable versions of the new tutorials - By the end of January: potentially finished? [1] https://github.com/pypa/warehouse/issues/729

2 2

Re: [Distutils] build system abstraction PEP, take #2
by Ronny Pfannschmidt Nov. 17, 2015

Nov. 17, 2015

I forgot to reply to the list, so I re-added it To further explain, in my setuptools replacement experiment I implement the develop command by generating a a wheel Using a local version tag `+develop`, Instead of the actual code, for each top-level it contains a shim python file that will load the real code from the editable location However it will use pip to install that wheel and create scripts/exe files In particular windows support for a full develop command is orders of magnitude more easy if one let's pip make the scripts(exe files) I think it's very helpful for tool authors if pip has ways for them to avoid platform pitfalls like exe files Am 17. November 2015 08:15:41 MEZ, schrieb Robert Collins <robertc(a)robertcollins.net>: >On 17 November 2015 at 19:16, Ronny Pfannschmidt ><opensource(a)ronnypfannschmidt.de> wrote: >> The develop command should be able to generate a wheel with a local >version >> that pip can install >> That way tool authors can completely avoid writing the target folders >and >> its possible to always run completely unprivileged builds > >Thats an interesting idea, but wholly different to what 'develop' does >and is used for. My goal here is to only commit to new things where >the new thing is all of: > - a small amount of work > - meets existing use cases > - able to be put into pip/setuptools/wheel/etc now (rather than after >a bunch of other things are done) > >I completely agree with the idea that develop as it stands has a >number of limitations, but there is no generic implementation >available for pip to reuse; there is no feature in pip that would do >what you want either; nor even a proof of concept. I think as such its >premature to PEPify it. > >What could be done is to take the abstract build system work (which >*is* within reach for pip) and create a implementation with a new >schema version and whatever semantics you want around this new >developish thing, and write an incremental PEP (so you don't need any >of the PEP I've put up, just inherit from it and define your new thing >- and then we can kick the tires on it and see how its going to work >etc. > >Note that what you describe 'generate a wheel with a local version' is >actually exactly what the 'wheel' command does as far as I can tell, >but I'm sure thats not what you have in mind - so please do expand on >the differences, in >paranoid-folk-make-sure-folk-cannot-misunderstand-style ;) > >-Rob > > >-- >Robert Collins <rbtcollins(a)hp.com> >Distinguished Technologist >HP Converged Cloud -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

1 0

New PEP : dependency specification
by Robert Collins Nov. 16, 2015

Nov. 16, 2015

Since we ended up with a hard dependency on this for the bootstrap thing (regardless of 'smaller step' or not) - I've broken this out of PEP 426, made it an encoding of the current status quo rather than an aspirational change. Since it has a dependency on markers, I had to choose whether to block on James' marker PEP, contribute to that, or include it. I think on balance it makes sense to have it in one document since the markers bit is actually quite shallow, so I've done that (after discussing with James). This could perhaps replace PEP 496 then, or be given a new number. Donald has graciously agreed to be a BDFL-delegate for it. The PR for it is https://github.com/pypa/interoperability-peps/pull/56 Full text follows. :PEP: XX :Title: Dependency specification for Python Software Packages :Version: $Revision$ :Last-Modified: $Date$ :Author: Robert Collins <rbtcollins(a)hp.com> BDFL-Delegate: Donald Stufft <donald(a)stufft.io> :Discussions-To: distutils-sig <distutils-sig(a)python.org> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 11-Nov-2015 :Post-History: XX Abstract ======== This PEP specifies the language used to describe dependencies for packages. It draws a border at the edge of describing a single dependency - the different sorts of dependencies and when they should be installed is a higher level problem. The intent is provide a building block for higher layer specifications. The job of a dependency is to enable tools like pip [#pip]_ to find the right package to install. Sometimes this is very loose - just specifying a name, and sometimes very specific - referring to a specific file to install. Sometimes dependencies are only relevant in one platform, or only some versions are acceptable, so the language permits describing all these cases. The language defined is a compact line based format which is already in widespread use in pip requirements files, though we do not specify the command line option handling that those files permit. There is one caveat - the URL reference form, specified in PEP-440 [#pep440]_ is not actually implemented in pip, but since PEP-440 is accepted, we use that format rather than pip's current native format. Motivation ========== Any specification in the Python packaging ecosystem that needs to consume lists of dependencies needs to build on an approved PEP for such, but PEP-426 [#pep426]_ is mostly aspirational - and there are already existing implementations of the dependency specification which we can instead adopt. The existing implementations are battle proven and user friendly, so adopting them is arguably much better than approving an aspirational, unconsumed, format. Specification ============= Examples -------- All features of the language shown with a name based lookup:: requests \ [security] >= 2.8.1, == 2.8.* \ ; python_version < "2.7.10" \ # Fix HTTPS on older Python versions. A minimal URL based lookup:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346… Concepts -------- A dependency specification always specifies a distribution name. It may include extras, which expand the dependencies of the named distribution to enable optional features. The version installed can be controlled using version limits, or giving the URL to a specific artifact to install. Finally the dependency can be made conditional using environment markers. Grammar ------- We first cover the grammar briefly and the drill into the semantics of each section later. A distribution specification is written in ASCII text. We use ABNF [#abnf]_ to provide a precise grammar. Specifications may have comments starting with a '#' and running to the end of the line:: comment = "#" *(WSP / VCHAR) Specifications may be spread across multiple lines if desired using continuations - a single backslash followed by a new line ('\\\\n'):: CSP = 1*(WSP / ("\\" LF)) Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note: URI is defined in std-66 [#std66]_:: version-cmp = "<" / "<=" / "!=" / "==" / ">=" / ">" / "~=" / "===" version = 1*( DIGIT / ALPHA / "-" / "_" / "." / "*" ) versionspec = version-cmp version *(',' version-cmp version) urlspec = "@" URI Environment markers allow making a specification only take effect in some environments:: marker-op = version-cmp / "in" / "not in" python-str-c = (WSP / ALPHA / DIGIT / "(" / ")" / "." / "{" / "}" / "-" / "_" / "*" python-str = "'" *(python-str-c / DQUOTE) "'" python-str =/ DQUOTE *(python-str-c / "'") DQUOTE marker-var = python-str / "python_version" / "python_full_version" / "os_name"" / "sys_platform" / "platform_release" / "platform_version" / "platform_machine" / "platform_python_implementation" / "implementation_name" / "implementation_version" / "platform_dist_name" "platform_dist_version" / "platform_dist_id" marker-expr = "(" marker ")" / (marker-var [marker-op marker-var]) marker = marker-expr *( ("and" / "or") marker-expr) name-marker = ";" *CSP marker url-marker = ";" 1*CSP marker Optional components of a distribution may be specified using the extras field:: identifier = 1*( DIGIT / ALPHA / "-" / "_" ) name = identifier extras = "[" identifier *("," identifier) "]" Giving us a rule for name based requirements:: name_req = name [CSP extras] [CSP versionspec] [CSP name-marker] And a rule for direct reference specifications:: url_req = name [CSP extras] urlspec [CSP url-marker] Leading to the unified rule that can specify a dependency:: specification = (name_req / location_req) [CSP comment] Whitespace ---------- Non line-breaking whitespace is optional and has no semantic meaning. A line break indicates the end of a specification. Specifications can be continued across multiple lines using a continuation. Comments -------- A specification can have a comment added to it by starting the comment with a "#". After a "#" the rest of the line can contain any text whatsoever. Continuations within a comment are ignored. Names ----- Python distribution names are currently defined in PEP-345 [#pep345]_. Names act as the primary identifier for distributions. They are present in all dependency specifications, and are sufficient to be a specification on their own. Extras ------ An extra is an optional part of a distribution. Distributions can specify as many extras as they wish, and each extra results in the declaration of additional dependencies of the distribution **when** the extra is used in a dependency specification. For instance:: requests[security] Extras union in the dependencies they define with the dependencies of the distribution they are attached to. The example above would result in requests being installed, and requests own dependencies, and also any dependencies that are listed in the "security" extra of requests. If multiple extras are listed, all the dependencies are unioned together. Versions -------- See PEP-440 [#pep440]_ for more detail on both version numbers and version comparisons. Version specifications limit the versions of a distribution that can be used. They only apply to distributions looked up by name, rather than via a URL. Version comparison are also used in the markers feature. Environment Markers ------------------- Environment markers allow a dependency specification to provide a rule that describes when the dependency should be used. For instance, consider a package that needs argparse. In Python 2.7 argparse is always present. On older Python versions it has to be installed as a dependency. This can be expressed as so:: argparse;python_version<"2.7" A marker expression evalutes to either True or False. When it evaluates to False, the dependency specification should be ignored. The marker language is a subset of Python itself, chosen for the ability to safely evaluate it without running arbitrary code that could become a security vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This PEP fixes some issues that were observed in the described in PEP-426 [#pep426]_. Comparisons in marker expressions are typed by the comparison operator. The <marker-op> operators that are not in <version-cmp> perform the same as they do for strings in Python. The <version-cmp> operators use the PEP-440 [#pep440]_ version comparison rules if both sides are valid versions. If either side is not a valid version, then the comparsion falls back to the same behaviour as in for string in Python if the operator exists in Python. For those operators which are not defined in Python, the result should be False. The variables in the marker grammar such as "os_name" resolve to values looked up in the Python runtime. If a particular value is not available (such as ``sys.implementation.name`` in versions of Python prior to 3.3, or ``platform.dist()`` on non-Linux systems), the default value will be used. .. list-table:: :header-rows: 1 * - Marker - Python equivalent - Sample values - Default if unavailable * - ``os_name`` - ``os.name`` - ``posix``, ``java`` - "" * - ``sys_platform`` - ``sys.platform`` - ``linux``, ``darwin``, ``java1.8.0_51`` - "" * - ``platform_release`` - ``platform.release()`` - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51`` - "" * - ``platform_machine`` - ``platform.machine()`` - ``x86_64`` - "" * - ``platform_python_implementation`` - ``platform.python_implementation()`` - ``CPython``, ``Jython`` - "" * - ``implementation_name`` - ``sys.implementation.name`` - ``cpython`` - "" * - ``platform_version`` - ``platform.version()`` - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014`` ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation`` ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015; root:xnu-2782.40.9~2/RELEASE_X86_64`` - "" * - ``platform_dist_name`` - ``platform.dist()[0]`` - ``Ubuntu`` - "" * - ``platform_dist_version`` - ``platform.dist()[1]`` - ``14.04`` - "" * - ``platform_dist_id`` - ``platform.dist()[2]`` - ``trusty`` - "" * - ``python_version`` - ``platform.python_version()[:3]`` - ``3.4``, ``2.7`` - "0" * - ``python_full_version`` - see definition below - ``3.4.0``, ``3.5.0b1`` - "0" * - ``implementation_version`` - see definition below - ``3.4.0``, ``3.5.0b1`` - "0" The ``python_full_version`` and ``implementation_version`` marker variables are derived from ``sys.version_info`` and ``sys.implementation.version`` respectively, in accordance with the following algorithm:: def format_full_version(info): version = '{0.major}.{0.minor}.{0.micro}'.format(info) kind = info.releaselevel if kind != 'final': version += kind[0] + str(info.serial) return version python_full_version = format_full_version(sys.version_info) implementation_version = format_full_version(sys.implementation.version) ``python_full_version`` will typically correspond to ``sys.version.split()[0]``. If a particular version number value is not available (such as ``sys.implementation.version`` in versions of Python prior to 3.3) the corresponding marker variable returned by setuptools will be set to ``0`` Backwards Compatibility ======================= Most of this PEP is already widely deployed and thus offers no compatibiltiy concerns. There are however two key points where the PEP differs from the deployed base. Firstly, PEP-440 direct references haven't actually been deployed in the wild, but they were designed to be compatibly added, and there are no known obstacles to adding them to pip or other tools that consume the existing dependency metadata in distributions. Secondly, PEP-426 markers which have had some reasonable deployment, particularly in wheels and pip, will handle version comparisons with ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is less than "2.7.9". This backward incompatibility is deliberate. We are also defining new operators - "~=" and "===", and new variables - the ``platform_dist_*`` variables which are not present in older marker implementations. The variables will fall back to "" on those implementations, permitting reasonably graceful upgrade. The new version comparisons will cause errors, so adoption may require waiting some time for deployment to be widespread. Rationale ========= In order to move forward with any new PEPs that depend on environment markers, we needed a specification that included them. The requirement specifier EBNF is lifted from setuptools pkg_resources documentation, since we can't sensible depend on a defacto standard. References ========== .. [#pip] pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/) .. [#pep345] PEP-345, Python distribution metadata version 1.2. (https://www.python.org/dev/peps/pep-0345/) .. [#pep426] PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/) .. [#pep440] PEP-440, Python distribution metadata. (https://www.python.org/dev/peps/pep-0440/) .. [#abnf] ABNF specification. (https://tools.ietf.org/html/rfc5234) .. [#std66] The URL specification. (https://tools.ietf.org/html/rfc3986) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

5 25

Current PyPI storage requirements?
by Nick Coghlan Nov. 16, 2015

Nov. 16, 2015

This isn't an urgent question, but rather a "if the stats are readily available, I'm curious as to the answer" one: what are PyPi's current storage requirements? The warehouse.python.org front page indicates how many objects there are, but not the amount of space they take up. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Version Spec compatible with PEP 0440 as well as SemVer
by Jamiel Almeida Nov. 15, 2015

Nov. 15, 2015

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 # Version Spec compatible with PEP 0440 as well as SemVer ## Short Version/TL;DR I want to know if the following versions would abide by SemVer AND PEP 0440 as I would like both to be valid for my projects (order matters). The idea is to create a formal specification based on the Semantic Versioning Specification that is fully compatible with both Semantic Versioning and PEP 0440. As such, I'd like to know what you think. ### Examples: 0.0.0 # Before the first public release, local iterations and commits to the repository when a public API is still not present. 0.1.0 # First public release, since major = 0, public API is expected to be unstable 0.1.1 # First patch applied to the first public release 0.2.0 1.0.0-a0 # Alpha pre-release of version 1 (first stable release) 1.0.0-a1 1.0.0-b0 1.0.0-rc0 # First release candidate 1.0.0-rc1 1.0.0 1.0.1 1.0.2 ## Long version ### Intention I got very interested in versioning and format of public version identifiers a couple of days ago, and was studying different formats used by projects. I really liked the two "big" propositions I found, which are Semantic Versioning from semver.org (or "SemVer" from now on) and PEP 0440 (or "the PEP" from now on). I'm aware of the existence of forks to SemVer that "fix" it to be compatible with the PEP, e.g. OpenStack Foundation PBR's "Linux/Python Compatible Semantic Versioning 3.0.0" [1]. I'm also aware of discussions in mailing lists involving the "fixing" the PEP, e.g. "module version number support for semver.org" in python-ideas [2]. I have no intention to fix either, rather have a version specification for the projects I lead or my personal projects that is compatible with both. And by compatible with both I hope to have a simple spec that still provides the determinism in downloads/upgrades introduced by the PEP, even if that means forgoing "versatility" of some suffixes and identifiers, for simplicity. As well as providing a meaning and rationale for each element of a version-number/release (the "sem" in SemVer) Keywords are "determinism", "simplicity"; and hopefully not "bikeshedding", nor "trolling", nor "flame-baiting". ### Compatible bits I quote the section of the PEP that talks about compatibility with SemVer [3]: > The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") > aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 > specification) are fully compatible with the version scheme defined in > this PEP, and abiding by these aspects is encouraged. > > Semantic versions containing a hyphen (pre-releases - clause 10) or a > plus sign (builds - clause 11) are not compatible with this PEP and > are not permitted in the public version field. > > One possible mechanism to translate such semantic versioning based > source labels to compatible public versions is to use the .devN suffix > to specify the appropriate version order. So I know the first four versions in my example above are compatible with both, as well as the last. The question arises when reading the above quote after having studied the prescriptions of the PEP in sections "Pre-releases" [4] and "Pre-release separators" [5] (under "Normalization") ### Pre-releases For the PEP, pre-releases are allowed to be separated by a dash from the release component described as "major.minor.patch" in Semantic Versioning, and as "major.minor.micro" in the PEP. So that part of clause 10 of version 2.0.0-rc-1 [6] (quoted below, as this was the one used when writing the PEP) on the Semantic Versioning format isn't the problem, where the problem lies is that SemVer allows that pre-release identifier to be multiple "sections" and to allow many more combinations of characters. > 10. A pre-release version MAY be denoted by appending a dash and a > series of dot separated identifiers immediately following the > patch version. Identifiers MUST be comprised of only ASCII > alphanumerics and dash [0-9A-Za-z-]. Pre-release versions satisfy > but have a lower precedence than the associated normal version. > Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92. Question #1: : If restricted to being one of "a", "b", or "rc" followed by a numeric component from 0-9, and also restricted to only one section, as shown on the examples I presented, would this be still incompatible with the PEP after being normalized? Question #2: : In the above-quoted version section of the PEP, pre-release versions aren't permitted to be part of the "public version field", what does this mean? I see that there are versions of software distributed using PyPI that have pre-release identifiers, e.g. Sphinx==1.3b3 [7] Semantically, a version "1.3b3" is the same as "1.3.0-b3" (an identifier compliant with SemVer, and the PEP after being normalized to "1.3.0.b3") But when saying "public version field", does the PEP prescribes that I "SHOULD NOT" use this identifier when publishing or what? Or was the intention that it not be part of the "final release" identifier (since it is after all a pre-release identifier). When I wrote the above examples, the spirit was to abide by SemVer (all clauses) while still being compatible with PEP 0440. The limitation of the numeric component to be [0-9] is to avoid sorting problems caused by a subtle difference in how these are sorted in SemVer vs The PEP. Since my examples use letters and numbers for this section identifier, SemVer prescribes lexical ordering by ASCII sort order, while the PEP compares same pre-release stage versions in numerical order. I.E. in SemVer "a1" < "a11" < "a2" as identifier components are expected to be separated by dots if they have different meaning. ### Aside. There was an update in this clause of SemVer from version 2.0.0-rc-1 as used in the PEP to version 2.0.0 (current), this update specified that the sections separated by dots must not be empty, must not include leading zeroes, and a small blurb on the meaning of this identifier. There was also a renumbering due to the deleting of a section before. These changes are irrelevant for this discussion. See [8] It's also important to note that I consider changes to documentation between minor releases a patch (SemVer) or (micro) in the PEP, and this number is updated when the code is "published" (using the definition in PEP 0426 [9]). This mechanism could fit into PEP 0440's concept of "post-release" but I'm choosing to forgo this concept to be able to stick to SemVer and not alter sorting order by using '-post#', '-r#', or '-rev#', which would force me to use '-c#' instead of '-rc#'. But it is possible and I might consider it in the future. Code pushed to the repository or merged from a pull-request doesn't fit into this definition of publication (the one used in PEP 0426). I consider part of this definition, however, the tagging a release in the version control repository, uploading to index servers, and in doing that providing the software to what PEP 0426 defines as "software integrators". As such, if after releasing a sample version "3.1.2", I were to push any number of commits and/or merge pull requests, this code or otherwise file changes are not considered to be published under the definition above, nor is it part of 3.1.2. Code is considered published and part of a version, when, as part of the development cycle; the version number on the repository is updated. This update would entail updating the version number in the code, and updating the changelog in the repository to the new version number and with all changes that have been added since last version was published". The change in version number and update to changelog should happen in the same commit, and this commit should not contain any other changes. This commit is then tagged with the version id, prepended with the character 'v'; and this new version is then to be uploaded to package indexes and sent to integrators. As such, and with the limitation of pre-release staged versions to only 10 (0-9) it is advised that the documentation be complete before pre-releasing in alpha, and only bugfixes be submitted (these may also be in the documentation) during a pre-release stage. The code and documentation is to be considered "frozen" for those pre-releases, and as such, changes should not be taken lightly and version bumps not be done trivially. End aside. ### Local versions / Build metadata In relation to what I call "+identifier"s, both the PEP and SemVer allow for "Local version identifier" or "build metadata" as they respectively refer to the section after the "public version identifier" in the PEP, and "normal version" in SemVer. This section is, on both documents, allowed to be included after a separator, specifically, the character "+" (plus sign). There are two differences, however. The first difference is syntactical, SemVer allows for letter, digit, and '-' (hyphen) characters in subsections separated by '.' (dot) in clause 10 of version 2.0.0 of the specification [8] (please don't confuse the clause with the previous #10, that was for the rc1 of the specification); while the PEP doesn't allow '-' (hyphen) on the segments as it calls them. The second difference is a semantic one, with an added "kicker". In SemVer, this section has no restrictions and is considered part of the public identifier. In contrast, it is prescribed in the PEP that this section of the identifier SHOULD NOT be used when publishing, and MAY be used to denote local versions built from source, and SHOULD be used by downstream projects when releasing a version of the upstream project (see section of the PEP about local version identifiers [10]). Also, please note that the PEP has specific meanings for "MAY" "SHOULD", and "SHOULD NOT". The "kicker" I mention, is that in SemVer two "build metadata" identifiers are not considered when determining precedence, while in the PEP they do. In more human terms, according to the PEP, my project should not use the "+identifier" for it's own releases, downstream projects can when they release a new version of my project that is compatible as defined in both SemVer and the PEP. And as long as those downstream projects don't use '-' (hyphen) these versions released by them would still abide by both SemVer and the PEP. [1]: http://docs.openstack.org/developer/pbr/semver.html [2]: http://grokbase.com/t/python/python-ideas/144e5x67tq/module-version-number-… [3]: https://www.python.org/dev/peps/pep-0440/#semantic-versioning [4]: https://www.python.org/dev/peps/pep-0440/#pre-releases [5]: https://www.python.org/dev/peps/pep-0440/#pre-release-separators [6]: http://semver.org/spec/v2.0.0-rc.1.html [7]: https://pypi.python.org/pypi/Sphinx/1.3b3 [8]: http://semver.org/spec/v2.0.0.html [9]: https://www.python.org/dev/peps/pep-0426/ [10]: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers - -- Jamiel Almeida <jamiel.almeida(a)gmail.com> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.29 (Darwin) iQIcBAEBCgAGBQJWSAAOAAoJEHY8IXipLI+0uUwP/RLnCk62QAjBpmMsC/O5lk3c FqL5fD5ZaDjX5paxy3aoKU83UY5D5KTWiMeBbpZ2ae6nf00UP5hKEjoi3eioQn6Q I8ELE1vHXP2X9s+iJRzzbO6zMVtYQ/OHWtJq/YaGeOQdtN8mfl4e6esyMK6Tldl6 TstKLEPMQXL5zxgXPzcudvYW3wj7MXa1Pn9xcY06LgO5ERtLM4JCk2PJMzbZ8R8r gS8+B5UbVLuu9aE5n9ZjDbP1A07jKSSmwFdoxatvG176Vp7ha4xl80Kc0liYl9ke L1Zn3whpCBn7C5amO2RzmklniOgs56MCURo4wyqe+4V1egl9yio298YvrAvDRT4d TEftliEsErsCQjDpPutkYYgynARktU/xQHRZUw7cXNeqNfTuzA7ZaDuFgO3X6fDf A7Bd3BbMq9h3FsMTGq7Tb2F6cF8W4wVASTZTb3endRWN9X+khVr6Bkvhl9qr5cWh XG3xXBhdAku9vUaqxEXVyUlKoFytt3UwFQpJgvgB/eKffVCcXa+e3RSsNaN5p5dO yxCuqblHHjF9JHfEEybG3IWZzL3eyBgMPEGLC9LtHCrGwKXh0m+oQU8UOYrSDld7 RfcdoYzbrZ0k+rdiKPXiOx59vB3FWKKfZAeGempE6efhYlYAKsVpQ31S1sxQZH9w 6VBZmyFa+fGGYVfgToQs =VeL1 -----END PGP SIGNATURE-----

1 0

The future of invoking pip
by Donald Stufft Nov. 13, 2015

Nov. 13, 2015

There is currently a semi related set of problems that I'd really like to figure out an answer too so we can begin to work on a migration path and close these out. This is dealing with a fairly fundamental aspect of pip so I'm bringing it up here to try and get wider discussion than the issue tracker or pypa-dev list. Currently pip installs a number of commands like ``pip``, ``pipX`` and ``pipX.Y`` where the X and X.Y corresponds to the version of Python that pip is installed into. Pip installs into whatever Python is currently executing it so this gives some ability to control which version of Python you're installing into (``pip2.7`` for Python 2.7 etc). However, this has a few problems: * It really only works if you only have one version of Python installed for each X.Y series, as soon as you have multiple versions in one series (including alternative implementations besides CPython) you run into a problem of which Python 2.7 is pip2.7 for and how do you invoke it for the *other* Python 2.7's. Something like pip2.7.8 is really ugly, and still doesn't solve the problem when you have multiple 2.7.8 installs and pip-pypy2-4.0 is even uglier. * The above gets *really* confusing when ``pipX`` or ``pip`` do not agree with what ``pythonX`` and ``python`` point to. * Having pip need to be installed into each Python means you end up with a bunch of independent pip installations which all need to be independently updated. We've made this better by having recent pips warn you if you're not on the latest version, but it's not unusual for me personally to have 30+ different installations of pip on my personal desktop. That isn't great. * Having ``pip`` on Windows requires us to create a "script wrapper" which is just a shim .exe that just executes a python script. This is due to the fact that Windows special cases .exe in places. This causes problems because you can't actually do ``pip install --upgrade pip`` on Windows because you'll get an error trying to update the script wrapper, you need to do ``py -m pip install --upgrade pip`` so the .exe file isn't currently open. One possible solution to the above problems is to try and move away from using ``pip``, ``pipX`` and ``pipX.Y`` and instead push people (and possibly deprecate ``pip`` &etc) towards using ``python -m pip`` instead. This makes it unambigious which python you're modifying and completely removes all of the confusion around that. However this too has problems: * There is a lot of documentation out there in many projects that tell people to use ``pip install ...``, the long tail on getting people moved to this will be very long. * It's more to type, 10 more characters on *nix and 6 more characters on Windows which makes it more akward and annoying to use. This is particularly annoying inside of a virtual environment where there isn't really any ambiguity when one is activated. * It still has the annoyance around having multiple pip installs all over the place and needing to manage those. * We still support Python 2.6 which doesn't support executing a package only modules via ``-m``. So we'll break Python 2.6 unless people do ``python -m pip.__main__`` or we move pip/* to _pip/* and make a pip.py which will break things for people using pip as a library (which isn't currently supported). Another possible option is to modify pip so that instead of installing into site-packages we instead create an "executable zip file" which is a simple zip file that has all of pip inside of it and a top level __main__.py. Python can directly execute this as if it were installed. We would no longer have any command except for ``pip`` (which is really this executable zip file). This script would default to installing into ``python``, and you can direct it to install into a different python using a (new) flag of ``-p`` or ``--python`` where you can specify either a full path or a command that we'll search $PATH for. * We'll break the usage of ``python -m pip``. * We'll break the usage of ``pipX`` and ``pipX.Y``. * We'll break the usage of anyone using pip as a library (but this is not actually supported). * We still can't do ``pip install --upgrade pip`` by default, and since there's no ability to do ``py -m pip install --upgrade pip`` we'll need to figure something out (perhaps skip updating the .exe? I don't know). The first option is the status quo and thus doesn't represent any *new* breakage but has all of the problems we have currently. The second option uses the ``-m`` operater but has problems with 2.6 and it's not as nice to type but it removes all the ambiguity. The third option introduces the most breakage, but it removes all the amiguity *and* it removes the need to manage a number of pip installations. A side conversation here is that pip currently bundles a number of dependencies inside of itself because it can't really have any dependencies. This works fine but it's a bit of an annoyance to maintain. The larger problem is that a number of downstreams don't like this and have attempted to route around it to varying degrees of success. One benefit of the third option is that we can remove the need to directly copy the bundled libraries into the pip source code and we can install just bundle it inside the built zip file. Thoughts? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

25 83

"option --single-version-externally-managed not recognized" when pip install mysql-connector-python-rf
by Chris Withers Nov. 13, 2015

Nov. 13, 2015

Hi All, My Travis nightly builds of a package started failing this morning with the following: https://travis-ci.org/Mortar/mortar_rdb/jobs/90919221 pip install mysql-connector-python-rf You are using pip version 6.0.7, however version 7.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting mysql-connector-python-rf Downloading mysql-connector-python-rf-2.1.3.tar.gz (271kB) 100% |################################| 274kB 2.1MB/s Installing collected packages: mysql-connector-python-rf Running setup.py install for mysql-connector-python-rf usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: option --single-version-externally-managed not recognized Complete output from command /home/travis/virtualenv/python3.4.2/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-_f5a57lj/mysql-connector-python-rf/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-jd_zku67-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/python3.4.2/include/site/python3.4: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: option --single-version-externally-managed not recognized mysql-connector-python-rf hasn't changed in ages, and I haven't pushed any commits to mortar_rdb in some time. Has pip or setuptools changed in the last 24 hrs? This something that might have changed on TravisCI? cheers, Chris

2 2

How to exclude a directory from a module that has tests?
by Nagy László Zsolt Nov. 13, 2015

Nov. 13, 2015

Hello, My source distribution contains a single module. It is **not** a package. I have used this: setup(name='yubistorm', version=__version__, description='Asnychronous two factor authentication client for YubiCloud with Tornado', long_description="""Provides a simple module that can be used from a Tornado server to authenticate users """ + """ with YubiCloud (https://www.yubico.com/products/services-software/yubicloud/)""", author='László Zsolt Nagy', author_email='nagylzs(a)gmail.com', license="LGPL v3", py_modules=['yubistorm'], requires=['tornado (>=4.3)'], url="https://bitbucket.org/nagylzs/yubistorm", classifiers=[ 'Topic :: Security', 'Topic :: Internet :: WWW/HTTP', "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: CPython", ], ) The problem is that there is a "test" directory and it is added to the source distribution. I want to exclude that. I know that there is an "exclude" parameter for find_packages(). But this is not a package. This is a single module. How do I exclude a directory then? Thanks, Laszlo

2 1

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
by Nathaniel Smith Nov. 13, 2015

Nov. 13, 2015

[Adding distutils-sig to the CC as a heads-up. The context is that numpy is looking at deprecating the use of 'python setup.py install' and enforcing the use of 'pip install .' instead, and running into some issues that will probably need to be addressed if 'pip install .' is going to become the standard interface to work with source trees.] On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: [...] > Hmm, after some more testing I'm going to have to bring up a few concerns > myself: > > 1. ``pip install .`` still has a clear bug; it starts by copying everything > (including .git/ !) to a tempdir with shutil, which is very slow. And the > fix for that will go via ``setup.py sdist``, which is still slow. Ugh. If 'pip (install/wheel) .' is supposed to become the standard way to build things, then it should probably build in-place by default. Working in a temp dir makes perfect sense for 'pip install <requirement>' or 'pip install <url>', but if the user supplies an actual named on-disk directory then presumably the user is expecting this directory to be used, and to be able to take advantage of incremental rebuilds etc., no? > 2. ``pip install .`` silences build output, which may make sense for some > usecases, but for numpy it just sits there for minutes with no output after > printing "Running setup.py install for numpy". Users will think it hangs and > Ctrl-C it. https://github.com/pypa/pip/issues/2732 I tend to agree with the commentary there that for end users this is different but no worse than the current situation where we spit out pages of "errors" that don't mean anything :-). I posted a suggestion on that bug that might help with the apparent hanging problem. > 3. ``pip install .`` refuses to upgrade an already installed development > version. For released versions that makes sense, but if I'm in a git tree > then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal > to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1). Ugh, this is clearly just a bug -- `pip install .` should always unconditionally install, IMO. (Did you file a bug yet?) At least the workaround is just 'pip uninstall numpy; pip install .', which is still better the running 'setup.py install' and having it blithely overwrite some files and not others. The first and last issue seem like ones that will mostly only affect developers, who should mostly have the ability to deal with these weird issues (or just use setup.py install --force if that's what they prefer)? This still seems like a reasonable trade-off to me if it also has the effect of reducing the number of weird broken installs among our thousands-of-times-larger userbase. -n -- Nathaniel J. Smith -- http://vorpus.org

6 44

Serverside Dependency Resolution and Virtualenv Build Server
by Thomas Güttler Nov. 12, 2015

Nov. 12, 2015

I wrote down a tought about Serverside Dependency Resolution and Virtualenv Build Server What do you think? Latest version: https://github.com/guettli/virtualenv-build-server virtualenv-build-server ####################### Rough roadmap how a server to build virtualenvs for the python programming language could be implemented. Highlevel goal -------------- Make creating new virtual envionments for the Python programming language easy and fast. Input: fuzzy requirements like this: django>=1.8, requests=>2.7 Output: virtualenv with packages installed. Two APIs ------------ #. Resolve fuzzy requirements to a fixed set of packages with exactly pinned versions. #. Read fixed set of packages. Build virtualenv according to given platform. Steps ----- Steps: #. Client sends list of fuzzy requirements to server: * I need: django>=1.8, requests=>2.7, ... #. Server solves the fuzzy requirements to a fixed set of requirememnts: django==1.8.2, requests==2.8.1, ... #. Client reads the fixed set of requirements. #. Optional: Client sends fixed set of requirements to the server. Telling him the plattform * My platform: sys.version==2.7.6 and sys.platform=linux2 #. Server builds a virtualenv according to the fixed set of requirements. #. Server sends the environment to the client #. Client unpacks the data and has a usable virtualenv Benefits -------- Speed: * There is only one round-trip from client to server. If the dependencies get resolved on the client the client would need to download the available version information. * Caching: If the server gets input parameters (fuzzy requirements and platform information) which he has seen before, he can return the cached result from the previous request. Possible Implementations ------------------------ APIs ==== Both APIs could be implementated by a webservice/Rest interface passing json or yaml. Serverside ========== Implementation Strategie "PostgreSQL" ..................................... Since the API is de-coupled from the internals the implementation could be exchanged without the need for changes at the client side. I suggest using the PostgreSQL und resolving the dependcy graph using SQL (WITH RECURSIVE). The package and version data gets stored in PostgreSQL via ORM (Django or SQLAlchemy). The version numbers need to be normalized to ascii to allow fast comparision. Related: https://www.python.org/dev/peps/pep-0440/ Implementation Strategie "Node.js" .................................. I like python, but I am not married with it. Why not use a different tools that is already working? Maybe the node package manager: https://www.npmjs.com/ Questions --------- Are virtualenv relocatable? AFAIK they are not. General Thoughts ---------------- * Ignore Updates. Focus on creating new virtualenvs. The server can do caching and that's why I prefer creating virtualenvs which never get updated. They get created and removed (immutable). I won't implement it -------------------- This idea is in the public domain. If you are young and brave or old and wise: Go ahead, try to implement it. Please communicate early and often. Ask on mailing-lists or me for feedback. Good luck :-) I love feedback --------------- Please tell me want you like or dislike: * typos and spelling stuff (I am not a native speaker) * alternative implementation strategies. * existing software which does this (even if implemented in a different programming language). * ... -- http://www.thomas-guettler.de/

4 8

command line versus python API for build system abstraction (was Re: build system abstraction PEP)
by Nathaniel Smith Nov. 11, 2015

Nov. 11, 2015

On Sun, Nov 8, 2015 at 5:28 PM, Robert Collins <robertc(a)robertcollins.net> wrote: > +The use of a command line API rather than a Python API is a little > +contentious. Fundamentally anything can be made to work, and Robert wants to > +pick something thats sufficiently lowest common denominator that > +implementation is straight forward on all sides. Picking a CLI for that makes > +sense because all build systems will need a CLI for end users to use anyway. I agree that this is not terribly important, and anything can be made to work. Having pondered it all for a few more weeks though I think that the "entrypoints-style" interface actually is unambiguously better, so let me see about making that case. What's at stake? ---------------------- Option 1, as in Robert's PEP: The build configuration file contains a string like "flit --dump-build-description" (or whatever), which names a command to run, and then a protocol for running this command to get information on the actual build system interface. Build operations are performed by executing these commands as subprocesses. Option 2, my preference: The build configuration file contains a string like "flit:build_system_api" (or whatever) which names a Python object accessed like import flit flit.build_system_api (This is the same syntax used for naming entry points.) Which would then have attributes and methods describing the actual build system interface. Build operations are performed by calling these methods. Why does it matter? ---------------------------- First, to be clear: I think that no matter which choice we make here, the final actual execution path is going to end up looking very similar. Because even if we go with the entry-point-style Python hooks, the build frontends like pip will still want to spawn a child to do the actual calls -- this is important for isolating pip from the build backend and the build backend from pip, it's important because the build backend needs to execute in a different environment than pip itself, etc. So no matter what, we're going to have some subprocess calls and some IPC. The difference is that in the subprocess approach, the IPC machinery is all written into the spec, and build frontends like pip implement one half while build backends implement the other half. In the Python API approach, the spec just specifies the Python calling conventions, and both halves of the IPC code live are implemented inside each build backend. Concretely, the way I imagine this would work is that pip would set up the build environment, and then it would run build-environment/bin/python path/to/pip-worker-script.py <args> where pip-worker-script.py is distributed as part of pip. (In simple cases it could simply be a file inside pip's package directory; if we want to support execution from pip-inside-a-zip-file then we need a bit of code to unpack it to a tempfile before executing it. Creating a tempfile is not a huge additional burden given that by the time we call build hooks we will have already created a whole temporary python environment...) In the subprocess approach, we have to write a ton of text describing all the intricacies of IPC. We have to specify how the command line gets split (or is it passed to the shell?), and specify a JSON-based protocol, and what happens to stdin/stdout/stderr, and etc. etc. In the Python API approach, we still have to do all the work of figuring these things out, but they would live inside pip's code, instead of in a PEP. The actual PEP text would be much smaller. It's not clear which approach leads to smaller code overall. If there are F frontends and B backends, then in the subprocess approach we collectively have to write F+B pieces of IPC code, and in the Python API approach we collectively have to write 2*F pieces of IPC code. So on this metric the Python API is a win if F < B, which would happen if e.g. everyone ends up using pip for their frontend but with lots of different backends, which seems plausible? But who knows. But now suppose that there's some bug in that complicated IPC protocol (which I would rate as about a 99.3% likelihood in our first attempt, because cross-platform compatible cross-process IPC is super annoying and fiddly). In the subprocess approach, fixing this means that we need to (a) write a PEP, and then (b) fix F+B pieces of code simultaneously on some flag day, and possibly test F*B combinations for correct interoperation. In the Python API approach, fixing this means patching whichever frontend has the bug, no PEPs or flag days necessary. In addition, the ability to evolve the two halves of the IPC channel together allows for better efficiency. For example, in Robert's current PEP there's some machinery added that hopes to let pip cache the result of the "--dump-build-description" call. This is needed because in the subprocess approach, the minimum number of subprocess calls you need to do something is two: one to ask what command to call, and a second to actually execute the command. In the python API approach, you can just go ahead and spawn a subprocess that knows what method it wants to call, and it can locate that method and then call it in a single shot, thus avoiding the need for an error-prone caching scheme. And the flexibility also helps in the face of future changes, too. Like, suppose that we start out with a do_build hook, and then later add a do_build2 hook that takes an extra argument or something, and pip wants to call do_build2 if it exists, and fall back on do_build otherwise. In the subprocess approach, you have to get the build description, check which hooks are provided, and then once you've decided which one you want to call you can spawn a second subprocess to do that. In the python API approach, pip can move this fallback logic directly into its hook-calling worker. (If it wants to.) So it still avoids the extra subprocess call. Finally, I think that it probably is nicer for pip to bite the bullet and take on more of the complexity budget here in order to make things simpler for build backends, because pip is already a highly complex project that undergoes lots of scrutiny from experts, which is almost certainly not going to be as true for all build backends. And the Python API approach is dead simple to explain and implement for the build backend side. I understand that the pip devs who are reading this might disagree, which is why I also wrote down the (IMO) much more compelling arguments above :-). But hey, still worth mentioning... -n -- Nathaniel J. Smith -- http://vorpus.org

8 26

devpi-server-2.4.0 and friends: speedup, fixes, profiling
by holger krekel Nov. 11, 2015

Nov. 11, 2015

We just pushed devpi-{server,web,client,common} release files out to pypi. Most notably, the private pypi package server allows much faster installs due to much improved simple-page serving speed. See the changelog below for a host of other changes and fixes as well as for compatibility warnings. Docs about the devpi system are to be found here: http://doc.devpi.net Many thanks to my co-maintainer Florian Schulze and particularly to Stephan Erb and Chad Wagner for their contributions. cheers, holger -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu devpi-server 2.4.0 (2015-11-11) ------------------------------- - NOTE: devpi-server-2.4 is compatible to data from devpi-server-2.3 but not the other way round. Once you run devpi-server-2.4 you can not go back. It's always a good idea to make a backup before trying a new version :) - NOTE: if you use "--logger-cfg" with .yaml files you will need to install pyyaml yourself as devpi-server-2.4 dropped it as a direct dependency as it does not install for win32/python3.5 and is not needed for devpi-server operations except for logging configuration. Specifying a *.json file always works. - add timeout to replica requests - fix issue275: improve error message when a serverdir exists but has no version - improve testing mechanics and name normalization related to storing doczips - refine keyfs to provide lazy deep readonly-views for dict/set/list/tuple types by default. This introduces safety because users (including plugins) of keyfs-values can only write/modify a value by explicitly getting it with readonly=False (thereby deep copying it) and setting it with the transaction. It also allows to avoid unnecessary copy-operations when just reading values. - fix issue283: pypi cache didn't work for replicas. - performance improvements for simple pages with lots of releases. this also changed the db layout of the caching from pypi.python.org mirrors but will seamlessly work on older data, see NOTE at top. - add "--profile-requests=NUM" option which turns on per-request profiling and will print out after NUM requests are executed and then restart profiling. - fix tests for pypy. We officially support pypy now. devpi-client-2.3.2 (2015-11-11) ------------------------------- - fix git submodules for devpi upload. ``.git`` is a file not a folder for submodules. Before this fix the repository which contains the submodule was found instead, which caused a failure, because the files aren't tracked there. - new option "devpi upload --setupdir-only" which will only vcs-export the directory containing setup.py. You can also set "setupdirs-only = 1" in the "[devpi:upload]" section of setup.cfg for the same effect. Thanks Chad Wagner for the PR. devpi-web 2.4.2 (2015-11-11) ---------------------------- - log exceptions during search index updates. - adapted tests/code to work with devpi-server-2.4 devpi-common 2.0.8 (2015-11-11) ------------------------------- - fix URL.joinpath to not add double slashes

1 0

BDFL Delegates for distutils-sig PEPs
by Nick Coghlan Nov. 10, 2015

Nov. 10, 2015

We've got to a point where the original standing delegations to myself and Richard Jones to act as BDFL-Delegates for metadata interoperability and pypi.python.org related aren't scaling adequately, so given Paul's recent delegation for PEP 470, and Donald handling PEP 503 directly, it seems like an opportune time to put something in writing about that. For PyPA/distutils-sig specific PEPs, we've effectively adopted the following approach to assigning BDFL-Delegates in resolving PEPs 470 and 503: ================================= Whenever a new PEP is put forward on distutils-sig, any PyPA core reviewer that believes they are suitably experienced to make the final decision on that PEP may offer to serve as the BDFL's delegate (or "PEP czar") for that PEP. If their self-nomination is accepted by the other PyPA core reviewer, the lead PyPI maintainer and the lead CPython representative on distutils-sig, then they will have the authority to approve (or reject) that PEP. ================================= And translating the nominated roles to the folks currently filling them: "lead PyPI maintainer" = Donald Stufft; "lead CPython representative on distutils-sig" = me. "PyPA core reviewer" isn't a term we've previously used, but I'm aiming to capture "has approval rights for pull requests to one or more of the PyPA maintained source code or documentation repos". Some further details for the benefit of folks not aware of the relevant history: * a couple of years ago, we amended PEP 1 to give the "Discussions-To" header some additional force for PEPs which don't directly affect CPython: """PEP review and resolution may also occur on a list other than python-dev (for example, distutils-sig for packaging related PEPs that don't immediately affect the standard library). In this case, the "Discussions-To" heading in the PEP will identify the appropriate alternative list where discussion, review and pronouncement on the PEP will occur.""" * we *didn't* update the section about assignment of BDFL-Delegates. Instead, I received a general delegation for packaging metadata interoperability PEPs, and Richard Jones received one for pypi.python.org related PEPs * Richard subsequently passed the latter delegation on to Donald, since Donald had taken over as the lead maintainer for PyPI The section in PEP 1 for CPython BDFL-Delegates reads as follows: ================================= However, whenever a new PEP is put forward, any core developer that believes they are suitably experienced to make the final decision on that PEP may offer to serve as the BDFL's delegate (or "PEP czar") for that PEP. If their self-nomination is accepted by the other core developers and the BDFL, then they will have the authority to approve (or reject) that PEP. ================================= This process can be appropriately described as "volunteering to be blamed" - if a PEP from a less experienced contributor subsequently proves to be a mistake, that's on the BDFL-Delegate for saying "Yes", not on the PEP author for proposing it. Mostly though, it's so there's someone to have the final say on the fiddly little details that go into getting from a general concept to an actual implementation, without getting mired down in design-by-committee on every incidental detail. As PEP authors, we'll also often ask someone else specifically to volunteer as BDFL-Delegate, because we trust their judgement in relation to the topic at hand (e.g. I asked Martin von Löwis to be BDFL-Delegate for the original ensurepip PEP because I knew he was skeptical of the idea, so a design that passed muster with him was likely to have suitably addressed the ongoing maintainability concerns. Guido did something similar when he asked Mark Shannon to be BDFL-Delegate for PEP 484's gradual typing). Regards, Nick. P.S. It's becoming clear to me that I should probably write a companion PEP to PEP 1 that specifically describes distutils-sig's usage of the PEP process (and how that differs from the normal CPython processes), but hopefully this post provides sufficient clarification for now. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 12

Proposed language for how build environments work in the new build system interface
by Nathaniel Smith Nov. 10, 2015

Nov. 10, 2015

Hi all, Following the strategy of trying to break out the different controversial parts of the new build system interface, here's some proposed text defining the environment that a build frontend like pip provides to a project-specific build backend. Robert's PEP currently disclaims all of this as out-of-scope, but I think it's good to get something down, since in practice we'll have to figure something out before any implementations can exist. And I think the text below pretty much hits the right points. What might be controversial about this nonetheless is that I'm not sure that pip *can* reasonably implement all the requirements as written without adding a dependency on virtualenv (at least for older pythons -- obviously this is no big deal for new pythons since venv is now part of the stdlib). I think the requirements are correct, so... Donald, what do you think? -n ---- The build environment --------------------- One of the responsibilities of a build frontend is to set up the environment in which the build backend will run. We do not require that any particular "virtual environment" mechanism be used; a build frontend might use virtualenv, or venv, or no special mechanism at all. But whatever mechanism is used MUST meet the following criteria: - All requirements specified by the project's build-requirements must be available for import from Python. - This must remain true even for new Python subprocesses spawned by the build environment, e.g. code like:: import sys, subprocess subprocess.check_call([sys.executable, ...]) must spawn a Python process which has access to all the project's build-requirements. This is necessary e.g. for build backends that want to run legacy ``setup.py`` scripts in a subprocess. [TBD: the exact wording here will probably need some tweaking depending on whether we end up using an entrypoint-like mechanism for specifying build backend hooks (in which case we can assume that hooks automatically have access to sys.executable), or a subprocess-based mechanism (in which case we'll need some other way to communicate the path to the python interpreter to the build backend, e.g. a PYTHON= envvar). But the basic requirement is pretty much the same either way.] - All command-line scripts provided by the build-required packages must be present in the build environment's PATH. For example, if a project declares a build-requirement on `flit <https://flit.readthedocs.org/en/latest/>`_, then the following must work as a mechanism for running the flit command-line tool:: import subprocess subprocess.check_call(["flit", ...]) A build backend MUST be prepared to function in any environment which meets the above criteria. In particular, it MUST NOT assume that it has access to any packages except those that are present in the stdlib, or that are explicitly declared as build-requirements. Recommendations for build frontends (non-normative) ................................................... A build frontend MAY use any mechanism for setting up a build environment that meets the above criteria. For example, simply installing all build-requirements into the global environment would be sufficient to build any compliant package -- but this would be sub-optimal for a number of reasons. This section contains non-normative advice to frontend implementors. A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies. This has two benefits: - It allows for a single installation run to build multiple packages that have contradictory build-requirements. E.g. if package1 build-requires pbr==1.8.1, and package2 build-requires pbr==1.7.2, then these cannot both be installed simultaneously into the global environment -- which is a problem when the user requests ``pip install package1 package2``. Or if the user already has pbr==1.8.1 installed in their global environment, and a package build-requires pbr==1.7.2, then downgrading the user's version would be rather rude. - It acts as a kind of public health measure to maximize the number of packages that actually do declare accurate build-dependencies. We can write all the strongly worded admonitions to package authors we want, but if build frontends don't enforce isolation by default, then we'll inevitably end up with lots of packages on PyPI that build fine on the original author's machine and nowhere else, which is a headache that no-one needs. However, there will also be situations where build-requirements are problematic in various ways. For example, a package author might accidentally leave off some crucial requirement despite our best efforts; or, a package might declare a build-requirement on `foo >= 1.0` which worked great when 1.0 was the latest version, but now 1.1 is out and it has a showstopper bug; or, the user might decide to build a package against numpy==1.7 -- overriding the package's preferred numpy==1.8 -- to guarantee that the resulting build will be compatible at the C ABI level with an older version of numpy (even if this means the resulting build is unsupported upstream). Therefore, build frontends SHOULD provide some mechanism for users to override the above defaults. For example, a build frontend could have a ``--build-with-system-site-packages`` option that causes the ``--system-site-packages`` option to be passed to virtualenv-or-equivalent when creating build environments, or a ``--build-requirements-override=my-requirements.txt`` option that overrides the project's normal build-requirements. The general principle here is that we want to enforce hygiene on package *authors*, while still allowing *end-users* to open up the hood and apply duct tape when necessary. -- Nathaniel J. Smith -- http://vorpus.org

2 1

PyPA Roadmap
by Marcus Smith Nov. 10, 2015

Nov. 10, 2015

Based on discussions in another thread [1], I've posted a PR to pypa.io for a "PyPA Roadmap" PR: https://github.com/pypa/pypa.io/pull/7 built version: http://pypaio.readthedocs.org/en/roadmap/roadmap/ To be clear, I'm not trying to dictate anything here, but rather just trying to mirror what I think is going on for the sake of new (or old) people, who don't have a full picture of the major todo items. I'm asking for help to make this as accurate as possible and to keep it accurate as our plans change. thanks, Marcus [1] https://mail.python.org/pipermail/distutils-sig/2015-October/027346.html , although it seems a number of emails in this thread never made it to the archive due to the python mail server failure.

5 9

Update to my skeletal PEP for a new build system interface
by Nathaniel Smith Nov. 9, 2015

Nov. 9, 2015

Hi all, Here's a quick update to my draft PEP for a new build system interface, last seen here: https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html There isn't terribly much here, and Robert and I should really figure out how to reconcile what we have, but since I was rearranging some stuff anyway and prepping possible new sections, I figured I'd at least post this. I addressed all the previous comments so hopefully it is boring and non-controversial :-). Changes: - It wasn't clear that the sdist metadata stuff was really helping, so I took it out for now. So it doesn't get lost, I split it out as a standalone deferred-status PEP to the pypa repository: https://github.com/pypa/interoperability-peps/pull/57 - Rewrote stuff to deal with Paul's comments - Added new terminology: "build frontend" for something like pip, and "build backend" for the project specific hooks that pip calls. Seems helpful. -n ---- PEP: ?? Title: A build-system independent format for source trees Version: $Revision$ Last-Modified: $Date$ Author: Nathaniel J. Smith <njs(a)pobox.com> Status: Draft Type: Standards-Track Content-Type: text/x-rst Created: 30-Sep-2015 Post-History: 1 Oct 2015, 25 Oct 2015 Discussions-To: <distutils-sig(a)python.org> Abstract ======== While ``distutils`` / ``setuptools`` have taken us a long way, they suffer from three serious problems: (a) they're missing important features like usable build-time dependency declaration, autoconfiguration, and even basic ergonomic niceties like `DRY <https://en.wikipedia.org/wiki/Don%27t_repeat_yourself>`_-compliant version number management, and (b) extending them is difficult, so while there do exist various solutions to the above problems, they're often quirky, fragile, and expensive to maintain, and yet (c) it's very difficult to use anything else, because distutils/setuptools provide the standard interface for installing packages expected by both users and installation tools like ``pip``. Previous efforts (e.g. distutils2 or setuptools itself) have attempted to solve problems (a) and/or (b). This proposal aims to solve (c). The goal of this PEP is get distutils-sig out of the business of being a gatekeeper for Python build systems. If you want to use distutils, great; if you want to use something else, then that should be easy to do using standardized methods. The difficulty of interfacing with distutils means that there aren't many such systems right now, but to give a sense of what we're thinking about see `flit <https://github.com/takluyver/flit>`_ or `bento <https://cournape.github.io/Bento/>`_. Fortunately, wheels have now solved many of the hard problems here -- e.g. it's no longer necessary that a build system also know about every possible installation configuration -- so pretty much all we really need from a build system is that it have some way to spit out standard-compliant wheels and sdists. We therefore propose a new, relatively minimal interface for installation tools like ``pip`` to interact with package source trees and source distributions. Terminology and goals ===================== A *source tree* is something like a VCS checkout. We need a standard interface for installing from this format, to support usages like ``pip install some-directory/``. A *source distribution* is a static snapshot representing a particular release of some source code, like ``lxml-3.4.4.zip``. Source distributions serve many purposes: they form an archival record of releases, they provide a stupid-simple de facto standard for tools that want to ingest and process large corpora of code, possibly written in many languages (e.g. code search), they act as the input to downstream packaging systems like Debian/Fedora/Conda/..., and so forth. In the Python ecosystem they additionally have a particularly important role to play, because packaging tools like ``pip`` are able to use source distributions to fulfill binary dependencies, e.g. if there is a distribution ``foo.whl`` which declares a dependency on ``bar``, then we need to support the case where ``pip install bar`` or ``pip install foo`` automatically locates the sdist for ``bar``, downloads it, builds it, and installs the resulting package. Source distributions are also known as *sdists* for short. A *build frontend* is a tool that users might run that takes arbitrary source trees or source distributions and builds wheels from them. The actual building is done by each source tree's *build backend*. In a command like ``pip wheel some-directory/``, pip is acting as a build frontend. An *integration frontend* is a tool that users might run that takes a set of package requirements (e.g. a requirements.txt file) and attempts to update a working environment to satisfy those requirements. This may require locating, building, and installing a combination of wheels and sdists. In a command like ``pip install lxml==2.4.0``, pip is acting as an integration frontend. Source trees ============ We retroactively declare the legacy source tree format involving ``setup.py`` to be "version 0". We don't try to specify it further; its de facto specification is encoded in the source code and documentation of ``distutils``, ``setuptools``, ``pip``, and other tools. A "version 1" (or greater) source tree is any directory which contains a file named ``pypackage.cfg``, which will -- in some manner whose details are TBD -- describe the package's build dependencies and how to invoke the project-specific build backend. This mechanism: - Will allow for both static and dynamic specification of build dependencies - Will have some degree of isolation of different builds from each other, so that it will be possible for a single run of pip to install one package that build-depends on ``foo == 1.1`` and another package that build-depends on ``foo == 1.2``. - Will leave the actual installation of the package in the hands of a specialized installation tool like pip (i.e. individual package build systems will not need to know about things like --user versus --global or make decisions about when and how to modify .pth files) [TBD: the exact set of operations to be supported and their detailed semantics] [TBD: should builds be performed in a fully isolated environment, or should they get access to packages that are already installed in the target install environment? The former simplifies a number of things, but Robert was skeptical it would be possible.] [TBD: the form of the communication channel between an installation tool like ``pip`` and the build system, over which these operations are requested] [TBD: the syntactic details of the configuration file format itself. We can change the name too if we want, I just think it's useful to have a single name to refer to it for now, and this is the last and least interesting thing to figure out.] Source distributions ==================== For now, we continue with the legacy sdist format which is mostly undefined, but basically comes down to: a file named {NAME}-{PACKAGE}.{EXT}, which unpacks into a buildable source tree. Traditionally these have always contained "version 0" source trees; we now allow them to also contain version 1+ source trees. Integration frontends require that an sdist named {NAME}-{PACKAGE}.{EXT} will generate a wheel named {NAME}-{PACKAGE}-{COMPAT-INFO}.whl. [TBD: whether we want to adopt a new sdist format along with this -- my read of the room is that it's sounding like people are leaning towards deferring that for a separate round of standardization, but we'll see what we think once some of the important details above have been hammered out] Evolutionary notes ================== A goal here is to make it as simple as possible to convert old-style sdists to new-style sdists. (E.g., this is one motivation for supporting dynamic build requirements.) The ideal would be that there would be a single static pypackage.cfg that could be dropped into any "version 0" VCS checkout to convert it to the new shiny. This is probably not 100% possible, but we can get close, and it's important to keep track of how close we are... hence this section. A rough plan would be: Create a build system package (``setuptools_pypackage`` or whatever) that knows how to speak whatever hook language we come up with, and convert them into calls to ``setup.py``. This will probably require some sort of hooking or monkeypatching to setuptools to provide a way to extract the ``setup_requires=`` argument when needed, and to provide a new version of the sdist command that generates the new-style format. This all seems doable and sufficient for a large proportion of packages (though obviously we'll want to prototype such a system before we finalize anything here). (Alternatively, these changes could be made to setuptools itself rather than going into a separate package.) But there remain two obstacles that mean we probably won't be able to automatically upgrade packages to the new format: 1) There currently exist packages which insist on particular packages being available in their environment before setup.py is executed. This means that if we decide to execute build scripts in an isolated virtualenv-like environment, then projects will need to check whether they do this, and if so then when upgrading to the new system they will have to start explicitly declaring these dependencies (either via ``setup_requires=`` or via static declaration in ``pypackage.cfg``). 2) There currently exist packages which do not declare consistent metadata (e.g. ``egg_info`` and ``bdist_wheel`` might get different ``install_requires=``). When upgrading to the new system, projects will have to evaluate whether this applies to them, and if so they will need to stop doing that. Copyright ========= This document has been placed in the public domain. -- Nathaniel J. Smith -- http://vorpus.org

4 5

free idea: whim file
by Daniel Holth Nov. 9, 2015

Nov. 9, 2015

Sometimes it might be desirable to do wheel-like installs without actually creating an archive. Instead, a whim (wheel internal manifest) file could communicate the idea of wheel, just a bunch of files in categories, without the zip file. The format would be no more than a mapping of category names 'purelib', 'platlib', 'headers', 'scripts', 'data', to a list of tuples with the file's path on the disk and its path relative to the category. { "category" : [ ('path on disk', 'path relative to category'), ... ] } The dist-info directory could be segregated into its own category 'metadata' with each target path as "distname-1.0.dist-info/FILE" i.e. its full path relative to the root of a wheel file. An installer could consume whim directly bypassing zip. A wheel archiver could consume a whim file and produce the archive with correct MANIFEST.

1 0

build system abstraction PEP
by Robert Collins Nov. 9, 2015

Nov. 9, 2015

Since Nathaniel seems busy, I've taken the liberty of drafting a narrow PEP based on the conversations that arose from the prior discussion. It (naturally) has my unique flavor, but builds on the work Nathaniel had put together, so I've put his name as a co-author even though he hasn't seen a word of it until now :) - all errors and mistakes are therefore mine... Current draft text in rendered form at: https://gist.github.com/rbtcollins/666c12aec869237f7cf7 I've run it past Donald and he has a number of concerns - I think we'll need to discuss them here, and possibly in another hangout, to get a path forward. Cheers, Rob -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

14 40

Platform tags for OS X binary wheels
by Robert McGibbon Nov. 7, 2015

Nov. 7, 2015

Hi, I just tried to run `pip install numpy` on my OS X 10.10.3 box, and it proceeds to download and compile the tarball from PyPI from source (very slow). I see, however, that pre-compiled OS X wheel files are available on PyPI for OS X 10.6 and later. Checking the code, it looks like pip is picking up the platform tag through `distutils.util.get_platform()`, which returns 'macosx-10.5-x86_64' on this machine. At root, I think this comes from the MACOSX_DEPLOYMENT_TARGET=10.5 entry in the Makefile at `python3.5/config-3.5m/Makefile`. I know that this value is used by distutils compiling python extension modules -- presumably so that they can be distributed to any target machine with OS X >=10.5 -- so that's good. But is this the right thing for pip to be using when checking whether a binary wheel is compatible? I see it mentioned <https://www.python.org/dev/peps/pep-0425/#id13> in PEP 425, so perhaps this was already hashed out on the list. Best, Robert

5 12

Why github and bitbucket?
by Thomas Güttler Nov. 6, 2015

Nov. 6, 2015

From: http://python-packaging-user-guide.readthedocs.org/en/latest/glossary/ > Python Packaging Authority (PyPA) > PyPA is a working group that maintains many of the relevant projects in Python packaging. They maintain a site at https://www.pypa.io, host projects on github and bitbucket, and discuss issues on the pypa-dev mailing list. Why are there pypa on github and bitbucket? Is one the master and the other the mirror? Or does one host part A and the other hosts part B? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

8 15

Re: [Distutils] New PEP : dependency specification
by Marcus Smith Nov. 6, 2015

Nov. 6, 2015

> > > So both the abstract build system PEP and donalds setup.py interface > depend on having a bootstrap dependency list written into a file in > the source tree. your build PEP said stuff like this "Additional data *may* be included, but the ``build_requires`` and ``metadata_version`` keys must be present" that leads me to think you need more than just this specification of a single dependency.

2 1

Which Build Distribution Formats do exist?
by Thomas Güttler Nov. 5, 2015

Nov. 5, 2015

>From http://python-packaging-user-guide.readthedocs.org/en/latest/glossary/ > Egg > A Built Distribution format introduced by setuptools, which is being replaced by Wheel. Which other Built Distribution formats do exist beside egg and wheel? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

4 5

The Update Framework, integrate into PyPI?
by Thomas Güttler Nov. 5, 2015

Nov. 5, 2015

I read the RoadMap (Thank you Marcus Smith) and came across this: > An effort to integrate PyPI with the “The Update Framework” (TUF). This is specified in PEP458 I see a trend to immutable systems everywhere. Updates are a pain. Building new systems is easier. With current hardware and good software it is easier to build new systems instead of updating existing systems. It is like from pets to cattle: - pets: you give them names and care for them (do updates) - cattle: you give them numbers and if they get ill you get rid of them. Maybe I am missing something. But why is there an effort to create "The Update Framework”, and why integrate it with pypi? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

6 6

[issue164] dfswferfergerge
by andrew31137 Nov. 3, 2015

Nov. 3, 2015

New submission from andrew31137: hello googlsx ---------- files: 1.html messages: 757 nosy: andrew31137 priority: wish status: unread title: dfswferfergerge Added file: http://bugs.python.org/setuptools/file178/1.html _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue164> _______________________________________________

1 0

Time for a setuptools_lite??
by Chris Barker Nov. 3, 2015

Nov. 3, 2015

This is kind-of related to the othe thread: "Remove distutils, was: ..." But more specific, so I thought I'd start a new one. Here are my thoughts: We had distutils -- there was a lot it didn't do that the "Masses" needed, so setuptools was born. It proved to be useful to a lot of people, and grew a large userbase.... But a lot was "wrong" with setuptools -- most prominently (in my mind anyway) that it put too many kind-sorta orthogonal stuff into one package: building, installing, distributing, managing version, managing dependencies, managing non-python resources, (and others??). And we didn't like how easy-install installed things :-) So distribute, and pip, and wheel, and now a new backward compatible setuptools was born. But we still have a bunch of should be orthogonal stuff tangled up together. In particular, I find that I often find easy-install getting invoked when I don't want ot to, and I get those darn eggs scattered all over the place, and easy_install.pth, and ???? I think if I am really careful about what I invoke when, this could be avoided, but the reality is that I've been dealing with this for years, and am trying desperately to do things the "right, modern" way, and I still get ugliness. I seriously doubt that I am alone. So -- here's my thought: I think we have it pretty well mapped out what functionality belongs where: one system for building packages (setuptools?) one system for installing packages and managing dependencies (pip) one system (really standard) for metadata and distributing packages (wheel) [I'm just whipping this out off the top of my head, I'm sure we'd need to more clearly define what belongs where] So why not have a setuptools-lite that only does the building? We need to keep the full over-burdened setuptools around, because lot sof folks are using those features. But for those of us that are doing something fairly new, and don't want to use stuff that setuptools "shouldn't" be doing, I'd love to have a setuptools-lite that only did what I really need, and was guaranteed NOT to accidentally introduce easy_install, etc... This seems to me to be a way to go forward -- as it is we'll have people using setuptools features that they "shouldn't" forever, and never be able to move to a cleaner system. Or maybe a flag: import setuptools setuptools.use_only_modern() That would make the dependencies easier -- i.e. pip depends on some of setuptools being there -- hard to say that it needs either setuptools OR setuptools_lite. Maybe I'm missing something here, but if the goal is for there to be one way to do things, let's have a tool chain that only does things one way..... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 42

A smaller step towards de-specializing setuptools/distutils
by Donald Stufft Nov. 2, 2015

Nov. 2, 2015

Hello! So I've been reading the various threads (well trying to, there's a lot going on in all of them) and I'm trying to reconile them all in my head and sorting out how things are going to look if we add them. I'm worried about adding more complication to an already underspecified format and making it a lot harder for tooling to work with these things. I'm wondering if there isn't a smaller step we can take towards allowing better interopability with diffrent build systems while also making the situation clearer when someone is still using distutils or setuptools. For the sake of argument, let's say we do something like this: Standardize a key in setup.cfg that can be used for setup_requires. This will be a static list of items which need to be installed in order to execute the setup.py. Unlike the current key (which is inside of setup.py) you don't have to deal with problems of trying to defer the import of the items or dealing with setuptools at all since your setup.py will not be invoked until after those items are installed. For people who want to continue to use setuptools, this file is trivial to parse with the standard library, so they can actually parse it and stick it into a setup_requires as well which will have the benefit that the tool calling the setup.py can handle those dependencies but still have it working on older versions of those tools without requiring duplication. This can also help work to make setuptools less special. Right now you basically have to just manually install setuptools prior to installing anything that uses it (or at all for something like pip when installing from sdist). With this we could pretty easily make it so that the rule is something like "If there is a setup.cfg:setup_requires, then assume that everything that is needed to execute the setup.py is listed there, else assume an implicit 'setuptools' requirement". This means that end users won't need to have setuptools installed at all, because we'll treat it (mostly) like just another build tool. This would also (for the first time) make it possible for things to depend on a specific version of setuptools instead of having to support every version of setuptools ever. I think that could solve the problem of "bootstrapping" the requirements to execute a setup.cfg and extract that from being implicitly handled by setuptools. That still doesn't handle the problem of making it possible to actually invoke the now installed build system. I think that what we should do is figure out the minimal interface we need from a ``setup.py`` based on what already exists. This would be a much smaller API surface than what exists today and wouldn't (ideally) include anything that is an implementation detail of setuptools. We would also need to standard on what flags an arguments the various commands of setup.py MUST accept. This would of course not require a setup.py to implement _only_ that particular interface so additional commands that setuptools already provide can stay, they just won't be required. Off the top of my head, I think we'd probably want to have the ``bdist_wheel``, ``sdist``, and ``build`` commands. We would also need something like ``egg_info``, but I think we could probably diverge a bit from what setuptools does now and make it ``dist_info`` so it's backed by the same standard that Wheel is already. I think that these four commands can support everything we actually _need_ to support. This isn't everything we actually use today, but since this would be opt-in, we can actually be more restrictive about what build/installation tools would call and what level of fallback we need to support for them. The way it would work when what we have available is a sdist is something like this: We download a sdist and we notice that there is a setup.cfg:setup_requires. This toggles a stricter mode of operation where we no longer attempt to do as many fallbacks or hacks to try and work around various broken shims with setuptools. We read this key and install those items needed to execute the setup.py. Once we do that, then pip would invoke ``setup.py bdist_wheel`` and build a wheel from that [1]. Once we have a wheel built, we'll feed that data back into the resolver [2] and use the runtime dependency information from within that wheel to continue resolving the dependencies. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is not being installed in editable. In this case we'll call ``setup.py sdist`` first, then feed that into the above. OR We have an "arbitrary directory" (VCS, local FS, whatever) on disk that is being installed as an editable. In this case, we'll call ``setup.py build --inplace`` first, then do something to ensure that the inplace directory is on sys.path. This is currently undefined because I don't know exactly what we'd need to do to make this work, but I think it should be possible and will be more consistent. We'll probably also need something like ``setup.py egg_info`` or ``setup.py dist_info``, but balancing backwards compat with removing setuptools specifics is something we'd need to figure out. So why do I think something like this would be better? * It uses interfaces that are already there (for the most part) which means that it's easier for people to adapt their current tooling to it, and to do it in a way that will work with existing legacy packages. * It defines an interface that build systems must adhere too, instead of defining an interface that lets you query for how you actually interact with the build system (sort of a plugin system). This removes indirection and in my experience, interfaces are generally less problematic then "plugins". * It doesn't drastically change the mechanics of how the underlying thing works so it's a much smaller iteration on existing concepts rather than throwing everything out and starting from scratch. * It moves us towards an overall installation "path" that I think will be benefical for us and will help to reduce the combinatorial explosion of ways that a set of dependencies can be installed. * It will be easier to integrate setuptools into this new system (since it largely already implements it) which means that it's easier for us to move projects to upgrade piece meal to this new system since it would only require dropping a ``setup.cfg`` with a ``setup_requires``. This also means we could adjust setuptools (like adding a dist_info command) and have the implicit setup.cfg:setup_requires be 'setuptools>=somever' instead of just 'setuptools'. What this doesn't solve: * This essentially doesn't solve any of the dynamic vs static metadata issues with the legacy sdist format, but that's OK because it's just a small layering (a new setup.cfg feature) of a new feature combined with just standardizing what already exists. Drastically altering the sdist format to try and shoe-horn static metadata in there is probably not something that is going to work well in practice (and needs a more thought out, newer format). * Dynamically specified _build_ requirements (or really, build requirements at all other than via setup_requires). You can sort of kludge dynamically specified build requirements by making a sort of meta-package that you put in your static setup_requires that generats dynamic runtime requirements. I think keeping this step simple though and work on enabling breaking the dependency on setuptools/distutils in this step and then waiting to see if that is "enough" or if we need to layer on additional features (like dynamic or otherwise seperately declared build requirements). [1] This isn't related to the Wheel cache. In this stricter mode we would only ever build a wheel and install from that. If caching is on then we'll save that wheel and reuse it next time, if caching is not on then we'll just throw that wheel away at the end of the run. [2] Pretending for a minute that we have a real resolver. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

10 29

Brian Goetz - Stewardship: the Sobering Parts
by Donald Stufft Nov. 2, 2015

Nov. 2, 2015

This was shown to me today, and I thought I’d share it here since it pretty much applies 100% to what we’re doing on this list too (other than the fact the examples are from Java) - https://www.youtube.com/watch?v=2y5Pv4yN0b0 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

4 3

Removing the aspirational aspects of PEP 426
by Nick Coghlan Nov. 1, 2015

Nov. 1, 2015

Caught up on distutils-sig this morning - I don't have any additional comments on Robert's and Nathaniel's draft PEPs except to say "I really like the look of where they're going, and look forward to reading the next iterations on them that take into account the feedback already received" :) The idea of treating extras as subdistributions with their own wheel metadata also sounds promising (and analogous to building multiple binary RPMs from a single source RPM). >From an sdist metadata perspective, though, I think the right thing to do is to descope PEP 426 to just the stuff we *need* for the build system improvements, and defer everything else (e.g. JSON-LD, SPDX, richer dependency semantics, etc) to a future metadata 3.0 proposal (or potentially metadata extensions, or 2.x format updates). The one major enhancement I think would be worth keeping is the metadata extension system (including mandatory extension support), since that provides a way to experiment with ideas that we may later standardise in 3.0. I'm not sure when I'd have time to work on that myself though, so I'm definitely open to expressions of interest in taking a hatchet to the PEP. Nathaniel, Robert, perhaps you'd be interested in that as part of the build interface standardisation? Regards, Nick.

3 4