Mailman 3 March 2017 - Distutils-SIG

PyPi not allowing duplicate filenames
by Chris Barker 20 Apr '20

20 Apr '20

Hi all, PyPi does not allow duplicate file names -- this makes lots of sense, because you really don't want people to go to PyPi one day and grab a file, and then go there another day, grab a file with exactly the same name, and have it be a different file. However.... We are all too human, and make mistakes when doing a release. All to often someone pushed a broken file up to PyPi, often realizes it pretty quickly -- before anyone has a chance to even download it (or only the dev team as, for testing...). In fact, I was in a sprint last summer, and we decided to push our package up to PyPi -- granted, we were all careless amateurish noobs, but we ended up making I think 4! minor version bumps because we had done something stupid in the sdist. Also -- the latest numpy release got caught in this, too: """ * We ran into a problem with pipy not allowing reuse of filenames and a resulting proliferation of *.*.*.postN releases. Not only were the names getting out of hand, some packages were unable to work with the postN suffix. """ So -- I propose that PyPi allow projects to replace existing files if they REALLY REALLY want to. You should have to jump through all sorts of hoops, and make it really clear that it is a BAD IDEA in the general case, but it'd be good to have it be possible. After all -- PyPi does not take on responsibility for anything else about what's in those packages, and Python itself is all about "we're all consenting adults here" I suppose we could even put in some heuristics about how long the file as been there, how many times it's been downloaded, etc. Just a thought.....I really hate systems that don't let me roll back mistakes, even when I discover them almost immediately... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

13 24

ImportError running 'test' on namespace package - other package in namespace not found
by Jason R. Coombs 17 Apr '20

17 Apr '20

I've got two projects: mynamespace.myprojectA and mynamespace.myprojectB myprojectB depends on myprojectA. I'm using setuptools 0.6c8 to manage both projects. Both projects are registered using 'setup develop'. Both projects are accessible from an interactive interpreter: PS C:\Users\me\projects> python Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import mynamespace.myprojectA >>> import mynamespace.myprojectB >>> from mynamespace.myprojectA import mymoduleZ However, when I run 'setup test' in myprojectB, the tests fail with File ".mymoduleZ.py", line NNN, in [some context] from mynamespace.myprojectA.mymoduleZ import MyClassC ImportError: No module named myprojectA.mymoduleZ In setup.py, the test_suite is nose.collector. I searched and couldn't find anyone else with this problem. Is this a supported configuration? Is there something I can do to make tests work with interdependent projects with the same root namespace? If there's not something obvious I should be doing differently, I'm happy to put together a minimal test case that reproduces the problem. Any suggestions are appreciated. Sincerely, Jason R. Coombs

3 6

Surviving a Compromise of PyPI - PEP 458 and 480
by Vladimir Diaz 12 Feb '20

12 Feb '20

Hello everyone, I am a research programmer at the NYU School of Engineering. My colleagues (Trishank Kuppusamy and Justin Cappos) and I are requesting community feedback on our proposal, "Surviving a Compromise of PyPI." The two-stage proposal can be reviewed online at: PEP 458 http://legacy.python.org/dev/peps/pep-0458/ PEP 480 http://legacy.python.org/dev/peps/pep-0480/ Summary of the Proposal: "Surviving a Compromise of PyPI" proposes how the Python Package Index (PyPI) can be amended to better protect end users from altered or malicious packages, and to minimize the extent of PyPI compromises against affected users. The proposed integration allows package managers such as pip to be more secure against various types of security attacks on PyPI and defend end users from attackers responding to package requests. Specifically, these PEPs describe how PyPI processes should be adapted to generate and incorporate repository metadata, which are signed text files that describe the packages and metadata available on PyPI. Package managers request (along with the packages) the metadata on PyPI to verify the authenticity of packages before they are installed. The changes to PyPI and tools will be minimal by leveraging a library, The Update Framework <https://github.com/theupdateframework/tuf>, that generates and transparently validates the relevant metadata. The first stage of the proposal (PEP 458 <http://legacy.python.org/dev/peps/pep-0458/>) uses a basic security model that supports verification of PyPI packages signed with cryptographic keys stored on PyPI, requires no action from developers and end users, and protects against malicious CDNs and public mirrors. To support continuous delivery of uploaded packages, PyPI administrators sign for uploaded packages with an online key stored on PyPI infrastructure. This level of security prevents packages from being accidentally or deliberately tampered with by a mirror or a CDN because the mirror or CDN will not have any of the keys required to sign for projects. The second stage of the proposal (PEP 480 <http://legacy.python.org/dev/peps/pep-0480/>) is an extension to the basic security model (discussed in PEP 458) that supports end-to-end verification of signed packages. End-to-end signing allows both PyPI and developers to sign for the packages that are downloaded by end users. If the PyPI infrastructure were to be compromised, attackers would be unable to serve malicious versions of these packages without access to the project's developer key. As in PEP 458, no additional action is required by end users. However, PyPI administrators will need to periodically (perhaps every few months) sign metadata with an offline key. PEP 480 also proposes an easy-to-use key management solution for developers, how to interface with a potential build farm on PyPI infrastructure, and discusses the security benefits of end-to-end signing. The second stage of the proposal simultaneously supports real-time project registration and developer signatures, and when configured to maximize security on PyPI, less than 1% of end users will be at risk even if an attacker controls PyPI and goes undetected for a month. We thank Nick Coghlan and Donald Stufft for their valuable contributions, and Giovanni Bajo and Anatoly Techtonik for their feedback. Thanks, PEP 458 & 480 authors.

7 53

Upgrading dependencies
by Charlie Moad 29 Apr '18

29 Apr '18

Is it currently possible to upgrade dependencies as well when upgrading a packages? If not, this would be a really nice feature to add to easy_install. Maybe a call like: easy_install --upgrade --upgrade-deps Package Obviously it makes sense to leave this off by default. Thanks, Charlie

2 1

Changes to --find-links and --upgrade in SVN
by Phillip J. Eby 29 Apr '18

29 Apr '18

Elvelind Grandin reported a problem with the "develop" command that turned out to be a flaw in its --find-links support. Specifically, it wasn't ever processing the links. :) In the process of fixing it, I wound up cleaning up an annoying (to me, at least) quirk of the previous workings of --find-links. It used to be that find-links would always be processed first, no matter what, even if you were doing a completely local operation. This would've been especially annoying if it carried over into "develop", so I made some changes. Now, if an item passed to --find-links is local (a filename or file: URL), or a direct link to an egg or other distribution, it is indexed immediately. Remote URLs are now only retrieved if a dependency can't be resolved locally, or if you use the -U or --upgrade options (this goes for "develop" too). Note that this is a behavior change for easy_install, which was effectively treating --find-links as though you'd specified --upgrade in certain cases. So, if you're used to getting upgrades downloaded as a result of using --find-links, please note that this will no longer happen. EasyInstall will now *only* go online if a dependency can't be resolved locally, if -U or --upgrade is used, or if you provided suitable direct URLs via an argument or --find-links, or via a link in a local .html file.

2 1

Problems with svn in windows
by Sharky On PTNet 29 Apr '18

29 Apr '18

Hi, I´m trying to do "easy_install setuptools==dev" in a windows box but have an error with svn command. """... 'svn' is not recognized as an internal or external command, ...""" I have installed TortoiseSVN but it seems that don't have command line, is all with the Explorer. Can you tell me what client I may use in windows? Thanks for all, Helio Pereira Sharky @ PTNet

3 2

local development scenarios
by tgreenwood 29 Apr '18

29 Apr '18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I'm new to distutils/and the egg system. I have not researched the archives as much as I should, so please excuse me if this is familiar territory... Q: Is it possible for dependency checks to work for non pipy deployed eggs? - A.egg - B.egg (setup_requires and install_requires A.egg) - email A and B eggs to friend - tell friend to : easy_install B - but B.egg does not install A...in fact, in my test case, it fails b/c it cannot find A in pipy - in this example, what happens if friend does not have net connection on target machine? how can they install from the eggs alone and get the dep checking to work? Q: How does my friend run the unittests on the egg that they have just installed? As I am developing, I frequently run the unittests and after every bdist_egg build, I also run the unit tests that way, too. How can the consumer do the same? Q: One of my apps is a command line app, say foo.py is a command line app that takes arg1, arg2,... How can the user run this app when it is encased in an egg? Q: How do I define my own pipy? While I like the idea of deploying there for the overall community, I don't want to send my half baked code into the wild. But I'd like to try the general functionality of having a deploy site for my beta customers. How can I set this up? Thanks in advance. - -Todd -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDl9Hfz6uXX4lQc/URAnY4AJ47DFHL/cpEltmqoUyGqzhsK5FjRACgt01v akc3Wh3IOXVv2jiS8TszhuE= =Rv/x -----END PGP SIGNATURE-----

5 12

Choice of dependencies?
by John J Lee 29 Apr '18

29 Apr '18

Is it possible for a package to depend on one of several packages, with the user having the option to pick? For example, my package P might depend on package A, plus either package B or package C. I know I could (ab)use extras_require, but in my case this is a real dependency rather than an optional extra. The other option I have is to just pick one of the dependencies (B or C in my example above), and thereby force people to install it even though it's not strictly required. Neither option seems particularly attractive. John

3 6

setuptools doc patch
by John J Lee 29 Apr '18

29 Apr '18

The comma removed by the patch below implies that users working from a Subversion checkout should not use EasyInstall to periodically fetch the latest version. That's not what you intended to say, I assume? http://peak.telecommunity.com/DevCenter/setuptools You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to +are working from a Subversion checkout rather than using EasyInstall to periodically fetch the latest version. Hmm, probably much better than the patch above: -You should also inform your users of the need to run this command, if they -are working from a Subversion checkout, rather than using EasyInstall to -periodically fetch the latest version. +If your users are working from a Subversion checkout rather than using +EasyInstall, you should also inform them of the need to run this command +to periodically fetch the latest version. John

2 2

Re: [Distutils] distutils install probs on suse 9.0
by Mats Wichmann 29 Apr '18

29 Apr '18

At 05:43 AM 5/18/2004, David Handy wrote: >On Tue, 18 May 2004, Floris van der Tak wrote: > >> my attempt to install the distutils on suse 9.0 (which provides python >> 2.3) fails: > >Why are you installing distutils on a system that already has Python 2.3 >installed? Python 2.3 normally comes with distutils as part of the Python >standard library. > >Did suse distribute a version of Python that had distutils removed? It should come from python-devel

2 1

Sample setup.py for Numeric Python
by Greg Ward 29 Apr '18

29 Apr '18

Hi again -- [cc'd to Paul Dubois: you said you weren't following the distutils sig anymore, but this directly concerns NumPy and I'd like to get your input!] here's that sample setup.py for NumPy. See below for discussion (and questions!). ------------------------------------------------------------------------ #!/usr/bin/env python # Setup script example for building the Numeric extension to Python. # This does sucessfully compile all the .dlls. Nothing happens # with the .py files currently. # Move this file to the Numerical directory of the LLNL numpy # distribution and run as: # python numpysetup.py --verbose build_ext # # created 1999/08 Perry Stoll __rcsid__ = "$Id: numpysetup.py,v 1.1 1999/09/12 20:42:48 gward Exp $" from distutils.core import setup setup (name = "numerical", version = "0.01", description = "Numerical Extension to Python", url = "http://www.python.org/sigs/matrix-sig/", ext_modules = [ ( '_numpy', { 'sources' : [ 'Src/_numpymodule.c', 'Src/arrayobject.c', 'Src/ufuncobject.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/numpy.def' } ), ( 'multiarray', { 'sources' : ['Src/multiarraymodule.c'], 'include_dirs' : ['./Include'], 'def_file': 'Src/multiarray.def' } ), ( 'umath', { 'sources': ['Src/umathmodule.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/umath.def' } ), ( 'fftpack', { 'sources': ['Src/fftpackmodule.c', 'Src/fftpack.c'], 'include_dirs' : ['./Include'], 'def_file' : 'Src/fftpack.def' } ), ( 'lapack_lite', { 'sources' : [ 'Src/lapack_litemodule.c', 'Src/dlapack_lite.c', 'Src/zlapack_lite.c', 'Src/blas_lite.c', 'Src/f2c_lite.c' ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/lapack_lite.def' } ), ( 'ranlib', { 'sources': ['Src/ranlibmodule.c', 'Src/ranlib.c', 'Src/com.c', 'Src/linpack.c', ], 'include_dirs' : ['./Include'], 'def_file' : 'Src/ranlib.def' } ), ] ) ------------------------------------------------------------------------ First, what d'you think? Too clunky and verbose? Too much information for each extension? I kind of think so, but I'm not sure how to reduce it elegantly. Right now, the internal data structures needed to compile a module are pretty obviously exposed: is this a good thing? Or should there be some more compact form for setup.py that will be expanded later into the full glory we see above? I've already made one small step towards reducing the amount of cruft by factoring 'include_dirs' out and supplying it directly as a parameter to 'setup()'. (But that needs code not in the CVS archive yet, so I've left the sample setup.py the same for now.) The next thing I'd like to do is get that damn "def_file" out of there. To support it in MSVCCompiler, there's already an ugly hack that unnecessarily affects both the UnixCCompiler and CCompiler classes, and I want to get rid of that. (I refer to passing the 'build_info' dictionary into the compiler classes, if you're familiar with the code -- that dictionary is part of the Distutils extension-building system, and should not propagate into the more general compiler classes.) But I don't want to give these weird "def file" things standing on the order of source files, object files, libraries, etc., because they seem to me to be a bizarre artifact of one particular compiler, rather than something present in a wide range of C/C++ compilers. Based on the NumPy model, it seems like there's a not-too-kludgy way to handle this problem. Namely: if building extension "foo": if file "foo.def" found in same directory as "foo.c" add "/def:foo.def" to MSVC command line this will of course require some platform-specific code in the build_ext command class, but I figured that was coming eventually, so why put it off? ;-) To make this hack work with NumPy, one change would be necessary: rename Src/numpy.def to Src/_numpy.def to match Src/_numpy.c, which implements the _numpy module. Would this be too much to ask of NumPy? (Paul?) What about other module distributions that support MSVC++ and thus ship with "def" files? Could they be made to accomodate this scheme? Thanks for your feedback -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

4 5

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Source of confusion
by Thomas Güttler 10 Apr '17

10 Apr '17

Quoting https://www.pypa.io/en/latest/ {{{ They host projects on github and bitbucket, and discuss issues on the pypa-dev and distutils-sig mailing lists. }}} I don't know where to go to. ... Choices .... too many choices ..... ... giving me the feeling of uncertainty. I am feeling fear. I am stupid and missing a guiding hand which gives me simple straight forward step by step instruction. Please help me. Regards, Thomas Güttler -- I am looking for feedback for my personal programming guidelines: https://github.com/guettli/programming-guidelines

9 20

Versioned trove classifiers for Django
by James Bennett 06 Apr '17

06 Apr '17

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

Wheel files for PPC64le
by Leonardo Bianconi 30 Mar '17

30 Mar '17

Hi all! I have been discussed the creation of a PEP, that describes how to create wheel files for the PPC64le architecture on wheel-builders (https://mail.python.org/pipermail/wheel-builders/) since January (https://mail.python.org/pipermail/wheel-builders/2017-January/000245.html). As all discussion from that list are done, the next step is it be reviewed here, and then create a draft on github, see it bellow: Abstract ======== This PEP proposes the extension of PEP 513 [1]_, which means extending the work done for platform tag ``manylinux1_``, initially created for x86_64 and i686 systems, to work on PowerPC 64 little endian (ppc64le), making wheel files available for this architecture. The platform tag, of this architecture for Python package built distributions, is called ``manylinux3_{ppc64le}``. Rationale ========= Currently on PowerPC 64 little endian, the ``pip install`` process downloads the module source code and builds it on the fly, to after that, install it. This process may cause a divergence on the presence of optional libraries it uses. One example of that is numpy, which optionally can use the OpenBlas [2]_; or BLAS [3]_; or neither of them. For each situation the performance of the module is affected and, badly enough, an end user is not able to know what is causing that. Building wheel files for the new architecture considers all work done on PEP 513 [1]_ with some changes proposed to handle the parameters for another architectures. The ``manylinux3`` policy ========================= Based on PEP 513 [1]_, the policy follows the same rules and library dependencies, but with the following versions for backward compatibility and base Operational System: * Backward compatibility versions: GLIBC <= 2.17 CXXABI <= 1.3.7 GLIBCXX <= 3.4.9 GCC <= 4.8.5 * Base Operational System: The stock O.S. release need to be the CentOS 7 [4]_, as it is the first CentOS release available for PowerPC64 little endian. The tag version for ppc64le architecture starts with 3 (``manylinux3_``), as it is supposed to be the version to match the CentOS 7 [4]_ in the future of the tag for x86_64/i686 architecture. There is the possibility of both tags diverge until it reaches the version 3, then a new PEP may be create to converge both to the manylinux baseline. Compilation of Compliant Wheels =============================== As compiling wheel files that meet the ``manylinux3`` standard for PowerPC64 little endian requires a specific Linux distro and version, the following tool is provided: Docker Image (Will be implemented when CentOS be available on Docker) ------------ The Docker Image is based on CentOS 7 [4]_, which is the first PowerPC 64 little endian CentOS release. The Image contains all necessary tools in the requested version to build wheel files (gcc, g++ and gfortran 4.8.5). Machine Image ------------- A full machine image containing all necessary software is provided for developers until CentOS be available on Docker for ppc64le. Cloud Service ------------- There are Cloud Services that provide ppc64le virtual machines for development. These machines can be used for the development of the wheel files, as CentOS 7 [4]_ an option for O.S.. All steps to obtain a machine on it is available for developers. Auditwheel ---------- This tool is an already provided item from PEP 513 [1]_, but needs to support the new architecture, so we propose the following changes: 1. Change the JSON file to handle more than one architecture, adding the compatible libraries and versions list for it. 2. Add a new filed in the JSON object to handle a list of architecture that the object is compatible. 3. When reading the JSON file, only consider the objects with the correspondent machine architecture. Platform detection for Installers ================================= The platform detection is almost the same as described in PEP 513 [1]_, but with the following proposed change: 1. Add the platform ppc64le in the platform list as a compatible one: [``linux-x86_64``, ``linux-i686``, ``linux-ppc64le``] 2. Add an if to switch architecture and consider the correct version of the GLIBC on ``return have_compatible_glibc(2, 5)``. References ========== .. [1] PEP 513 -- A Platform Tag for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0513/) .. [2] OpenBLAS -- An optimized BLAS library (http://www.openblas.net/) .. [3] BLAS -- Basic Linear Algebra Subprograms (http://www.netlib.org/blas/) .. [4] CentOS 7 Release Notes (https://wiki.centos.org/Manuals/ReleaseNotes/CentOS7) .. [5] CentOS 5.11 Release Notes (https://wiki.centos.org/Manuals/ReleaseNotes/CentOS5.11) Thanks, Leonardo Bianconi.

3 4

obtaining project name and packages
by Chris Jerdonek 29 Mar '17

29 Mar '17

Hi, this seems like a simple question, but I haven't been able to find the answer online: What is the current recommended way to get (1) the name of a project, and (2) the names of the top-level packages installed by a project (not counting the project's dependencies). You have access to / can run the project's setup.py, and you're also allowed to assume that the project is installed. For example, for (1) I know you can do-- $ python setup.py --name But I'm not sure if accessing setup.py is no longer recommended (as opposed to going through a tool like pip). Thanks a lot, --Chris

3 3

package ownership transfer: kobo
by Ken Dreyer 27 Mar '17

27 Mar '17

Hi folks, Would someone please give me access to upload new versions of kobo to PyPI? I've filed the transfer request [1] a while back, and several users are asking for new versions. [2] - Ken [1] https://sourceforge.net/p/pypi/support-requests/632/ [2] https://github.com/release-engineering/kobo/issues/24

2 1

GSoC 2017 - Request for Comments on Proposal
by Pradyun Gedam 25 Mar '17

25 Mar '17

Hello Everyone! I had previously sent a mail on this list, stating that I would like to work on pip's dependency resolution for my GSoC 2017. I now have drafted a proposal for the same; with help from my mentors - Donald Stufft and Justin Cappos. I'd also take this opportunity to thank them for agreeing to be my mentors for this GSoC. I would like to request for comments on my proposal for GSoC - it is hosted at https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb. Please find trailing a MarkDown version of the proposal. Thanks, Pradyun Gedam ----- # Adding Proper Dependency Resolution to pip - **Name:** Pradyun S. Gedam - **Email:** [pradyunsg(a)gmail.com][mailto-email] - **Github:** [pradyunsg][github-profile] - **University:** [VIT University, Vellore, India][vit-homepage] - **Course:** Bachelor of Technology in Computer Science and Engineering - **Course Term:** 2016/17 - 2019/20 (4 Year) - **Timezone:** IST (GMT +5:30) - **GSoC Blog RSS Feed URL:** < https://pradyunsg.github.io/gsoc-2017/feed.xml> [github-profile]: http://github.com/pradyunsg/ [vit-homepage]: http://vit.ac.in/ [mailto-email]: mailto:pradyunsg@gmail.com ## About Me I was introduced to Python about five years ago, through "Core Python Programming" by Weasley J Chun. After the initial struggle with Python 2/3, the ball was set rolling and hasn't stopped since. I have fiddled around with Game Programming (PyGame), Computer Vision (OpenCV), Data Analytics (Pandas, SciPy, NumPy), transcompilers (Py2C) and more. As a high school student, I did internship at Enthought in 2013 and 2014. The two summers that I spent at Enthought were a great learning experience and I thoroughly enjoyed the environment there. Other than Python, I have also used C, C++, Web Technologies (JavaScript, HTML, CSS) and preprocessors (Pug, TypeScript, LESS/SASS/SCSS), Java and Bash/Zsh for some other projects. Curious to understand how pip works, I began looking into pip's source code. I started contributing to pip in May 2016. I am now fairly familiar with the design of pip and have a fair understanding of how it works, due to the various contributions I have made to pip in the past year. ## Mentors - Donald Stufft (@dstufft on GitHub) - Justin Cappos (@JustinCappos on GitHub) Communication with the mentors will be done mostly on issues and pull requests on pip's GitHub repository. If at any point in time, a real time discussion is to be done with the mentors, it would be done over IRC or Skype. Email can also be used if needed. ## Proposal This project is regarding improving dependency resolution performed within pip by implementing a dependency resolver within it. ### Abstract Currently, pip does not resolve dependencies correctly when there are conflicting requirements. The lack of dependency resolution has caused hard-to-debug bugs/failures due to the installation of incompatible packages. The lack of a dependency resolver is also a blocker for various other features - adding an upgrade-all functionality to pip and properly determining build-time dependencies for packages are two such features. ### Deliverables At the end of this project, pip will have the ability to: - detect requirement conflicts - resolve conflicting dependency requirements where possible ### Implementation The implementation language will be Python. The code will maintain the compatibility requirements of pip - the same source code will support the multiple Python implementations and version, including but not limited to, CPython 2.7, CPython 3.3+, PyPy 2, PyPy3. New Tests for the functionality introduced will be added to pip's current test suite. User documentation would not be a major part of this project. The only changes would be to mention pip can now resolve dependencies properly. There are certain sections that might need updating. #### Overview The project will be composed of the following stages: 1. Refactor the dependency resolution logic of pip into a separate module. 1. Implement dependency information caching in pip. 1. Implement a backtracking dependency resolver to resolve the dependencies. Every stage depends on the previous ones being completed. This step-wise approach would make incremental improvements so that there is a constant feedback on the work being done as well as making it easy to change course without losing existing work; if needed for some unforeseen reason. #### Discussion There is a class in pip - `RequirementSet`, which is currently a God class. It is responsible for the following: 1. Holding a list of things to be installed 1. Downloading Files 1. Dependency Resolution 1. Unpacking downloaded files (preparation for installation) 1. Uninstalling packages 1. Installing packages This is clearly a bad situation since this is most of the heavy lifting involved in pip. These responsibilities need to be separated and moved out into their independent modules/classes, to allow for simplification of `RequirementSet` while providing a clean platform for the remaining work. This is the most tricky portion of this project, given the complexity of `RequirementSet` as it stands today. There are two kinds of distributions that may be used to install a package - wheels (binary) and sdists (source). When installing a package, pip builds a wheel from an sdist and then proceeds to install the wheel. The difference between the two formats of distribution relevant to this project is - wheels store the information about dependencies within them statically; sdists do not. Determining the dependencies of a wheel distribution is merely the matter of fetching the information from a METADATA file within the `.whl` file. The dependency information for an sdist, on the other hand, can only be determined after running its `setup.py` file on the target system. This means that dependencies of an sdist depend on how its `setup.py` behaves which can change due to variations on target systems or could even contain through random choices. Computation of an sdist's dependencies on the target system is a time-consuming task since it potentially involves fetching a package from PyPI and executing it's setup.py to get the dependency information. In order to improve performance, once an sdist's dependencies are computed, they would be stored in a cache so that during dependency resolution, the dependencies of an sdist are not computed every time they are needed. Further, pip caches wheels it downloads or builds meaning that any installed package or downloaded wheel's dependency information would available statically, without the need to go through the sdist dependency cache. Like the wheel cache, sdist-dependency-cache will be a file system based cache. The sdist-dependency-cache would only be used if the corresponding sdist is being used. Since sdist dependencies are non-deterministic, the cached dependency information is potentially incorrect - in certain corner cases such as using random choices in setup.py files. Such uses are not seen as important enough to cater to, compared the benefits of having a cache. Further, this is already the case with the information in the wheel cache. SAT solver based resolution is not feasible for pip since a SAT solver needs the entire set of packages and their dependencies to compute a solution, which cannot be generated due to the aforementioned non-deterministic behaviour of setup.py file. Computing dependency information for all of PyPI on a target system for installing a package is simply not feasible. The most reasonable approach is using a backtracking solver. Such a solver can be incrementally provided information about the dependencies of a package and would only need dependency information about packages in the dependency graph of the current system. There is a need to keep a cache of visited packages during dependency resolution. A certain package-version combination may be reached via multiple paths and it is an inefficient use of computation time to re-compute that whether it is indeed going to satisfy the requirements or not. By storing information about which package-version combinations have been visited and do (or do not) satisfy the constraints, it is possible to speedup the resolution. Consider the following example: ``` A-1.0 (depends: B) A-2.0 (depends: B and E-1.0) B-1.0 (depends: C and D) C-1.0 (depends: D) D-1.0 D-1.1 (depends: E-2.0) E-1.0 E-2.0 ``` If an installation of A is required, either A-2.0 or D-1.1 should not be installed because they have a conflicting requirement - E. While there are multiple possible solutions to this situation, the "most obvious" one us to use the D-1.0, instead of not installing A-2.0. Further, as multiple packages depend on D, the algorithm would "reach it" multiple times. By maintaining a cache for the visited packages, it is possible to achieve a speedup in such a scenario. #### Details Pull requests would be made on a regular basis during the project to ensure that the feedback loop is quick. This also reduces the possibilities of conflicts due to unrelated changes in pip. All the code will be tested within pip's existing testing infrastructure. It has everything needed to write tests related to all the changes to be made. Every PR made to pip as a part of this project will contain new tests or modifications to existing ones as needed. ##### Stage 1 Initially, some abstractions will be introduced to the pip codebase to improve the reuse of certain common patterns within the codebase. This includes cleaner temporary directory management through a `TempDirectory` class. `RequirementSet.prepare_files` and `RequirementSet._prepare_file` are downloading, unpacking packages as well as doing what pip does as dependency resolution. Taking these functions apart neatly is going to be a tricky task. The following is a listing of the final modules that will be responsible for the various tasks that are currently being performed by `RequirementSet`: - `pip.resolve` - Dependency Resolution - `pip.download` - Downloading Files & Unpacking downloaded files - `pip.req.req_set` - Holding a list of things to be installed / uninstalled - `pip.operations.install` - Installing Packages (preparation for installation) - `pip.operations.uninstall` - Uninstalling Packages To be able to proceed to the next step, only the dependency resolution related code needs to be refactored into a separate module. Other portions of `RequirementSet` are not required to be refactored but the same will be tackled as optional deliverables. In other words, only `pip.resolve` needs to be completed to be able to proceed to the next stage in this project. This is needed since in Stage 3, the resolver would be written in `pip.resolve`, independent of the rest of the codebase. ##### Stage 2 A new module `pip.cache` will be created. Within this module, all the caching will be handled. Thus, the code for the current wheel cache would be moved. The new code for a dependency cache for sdists would also be written here. The new cache would hold all of an sdist's egg-info. The information will be stored on the file-system in a sub directory structure much like that of the wheel cache, in a directory structure based on hash of the sdist file holding the egg-info at the end. This will be done in a class named `EggInfoCache`. `EggInfoCache` cache will be used only if a corresponding wheel for an sdist is not available. Installing an sdist results in the creation of a wheel which contains the dependency information, which would be used over the information available in the `EggInfoCache`. To be able to proceed to the next step, it is required that `EggInfoCache` is implemented. ##### Stage 3 The module `pip.resolve` will be modified and a class named `BacktrackingResolver` will be added to it. The class does what you expect it to do - it would resolve dependencies with recursive backtracking. As described above, there will be some global state stored about the packages that have been explored. Other than the maintenance of global state, in the form of the cache, the rest of the algorithm will essentially follow the same structure as any backtracking algorithm. The project would be complete when the aforementioned resolver is implemented. #### Existing Work There is existing work directly related to dependency resolution in pip, done by multiple individuals. - Robert Collins (un-merged closed pull request on pypa/pip) This has an incomplete backtracking dependency resolver and dependency caching. - Sebastien Awwad (branch on a separate fork) This was used for the depresolve project, to investigate the state of Dependency Resolution in PyPI/pip ecosystem. - `pip-compile` (separate project) This has a backtracking dependency resolver implemented to overcome pip's inablity to resolve dependencies. Their work would be used for reference, where appropriate, during the course of the project. Further, there are many package managers which implement dependency resolution, which would also be looked into. ### Tentative Timeline - Community Bonding Period: **5 May - 29 May** - Clean up and finalize my existing pull requests. - Read existing literature regarding dependency resolution. - Inspect existing implementations of dependency resolvers. GOAL: Be ready for the work coming up. - Week 1: **30 May - 5 June** - Introduce abstractions across pip's codebase to make refactoring `RequirementSet` easier. - Begin breaking down `RequirementSet.prepare_file`. - Week 2: **6 June - 12 June** - Continue working on the refactor of `RequirementSet`. - Week 3: **13 June - 19 June** - Continue working on the refactor of `RequirementSet`. - Finish and polish `pip.resolve`. GOAL: `pip.resolve` module will be ready, using the current resolution strategy. - Week 4: **20 June - 26 June** - Finish and polish all work on the refactor of `RequirementSet`. - Week 5: **27 June - 3 July** - Move code for `WheelCache` into a new `pip.cache` module. - Write tests for `pip.cache.EggInfoCache`, based on `WheelCache`. - Begin implementation of `pip.cache.EggInfoCache`. - Week 6: **4 July - 10 July** - Finish and polish `pip.cache.EggInfoCache`. GOAL: A cache for storing dependency information of sdists would be ready to add to pip. - Week 7: **10 July - 16 July** - Create a comprehensive set of tests for the dependency resolver. (in `tests/unit/test_resolve.py`) - Begin implementation on the backtracking algorithm. GOAL: A comprehensive test bed is ready for testing the dependency resolver. - Week 8: **17 July - 24 July** - Complete a rough implementation of the backtracking algorithm GOAL: An implementation of a dependency resolver to begin running tests on and work on improving. - Week 9: **25 July - 31 July** - Fixing bugs in dependency resolver - Week 10: **1 August - 6 August** - Finish and polish work on dependency resolver GOAL: A ready-to-merge PR adding a backtracking dependency resolver for pip. - Week 11: **6 August - 13 August** Buffer Week. - Week 12: **13 August - 21 August** Buffer Week. Finalization of project for evaluation. If the deliverable is achieved ahead of schedule, the remaining time will be utilized to resolve open issues on pip's repository in the order of priority as determined under the guidance of the mentors. #### Other Commitments I expect to not be able to put in 40 hour/week for at most 3 weeks throughout the working period, due to the schedule of my university. I will have semester-end examinations - from 10th May 2017 to 24th May 2017 - during the Community Bonding Period. My university will re-open for my third semester on 12 July 2017. I expect mid-semester examinations to be held in my University around 20th August 2017. During these times, I would not be able to put in full 40 hour weeks due to the academic workload. I might take a 3-4 day break during this period, regarding which I would be informing my mentor around a week in advance. I will be completely free from 30th May 2017 to 11 July 2017. ### Future Work There seems to be some interest in being able to reuse the above dependency resolution algorithm in other packaging related tools, specifically from the buildout project. I intend to eventually move the dependency resolution code that would come out of this project into a separate library to allow for reuse by installer projects - pip, buildout and other tools. ## Previous Contributions to pip (As on 12th March 2017) ### Issues Authored: - #3785 - Prefering wheel-based installation over source-based installation (Open) - #3786 - Make install command upgrade packages by default (Closed) - #3787 - Check if pip broke the dependency graph and warn the user (Open) - #3807 - Tests fail since change on PyPI (Closed) - #3809 - Switch to TOML for configuration files (Open) - #3871 - Provide a way to perform non-eager upgrades (Closed) - #4198 - Travis CI - pypy broken dues to dependency change in pycrypto (Closed) - #4282 - What's the release schedule? (Closed) Participated: - #59 - Add "upgrade" and "upgrade-all" commands (Open) - #988 - Pip needs a dependency resolver (Open) - #1056 - pip install -U does not remember whether a package was installed with --user (Open) - #1511 - ssl certificate hostname mismatch errors presented badly (Open) - #1668 - Default to --user (Open) - #1736 - Create a command to make it easy to access the configuration file (Open) - #1737 - Don't tell the user what they meant, just do what they meant (Open) - #2313 - Automated the Creation and Upload of Release Artifacts (Open) - #2732 - pip install hangs with interactive setup.py setups (Open) - #3549 - pip -U pip fails (Open) - #3580 - Update requests/urllib3 (Closed) - #3610 - pip install from package from github, with github dependencies (Open) - #3788 - `pip` version suggested is older than the version which is installed (Closed) - #3789 - Error installing Mayavi in Mac OS X (Closed) - #3798 - On using python -m pip install -upgrade pip.. its throwing an error like the one below (Closed) - #3811 - no matching distribution found for install (Closed) - #3814 - pip could not find a version that satisfies the requirement oslo.context (Closed) - #3876 - support git refs in @ syntax (Open) - #4021 - Will you accept PRs with pep484 type hints? (Open) - #4087 - pip list produces error (Closed) - #4149 - Exception thrown when binary is already linked to /usr/local/bin (Open) - #4160 - Pip does not seem to be handling deep requirements correctly (Open) - #4162 - Let --find-links be context aware to support github, gitlab, etc. links (Open) - #4170 - pip list |head raises BrokenPipeError (Open) - #4182 - pip install should install packages in order to avoid ABI incompatibilities in compiled (Open) - #4186 - IOError: [Errno 13] Permission denied: '/usr/local/bin/pip' (Open) - #4206 - Where on Windows 10 is pip.conf or pip.ini located? (Closed) - #4221 - Feature request: Check if user has permissions before downloading files (Closed) - #4229 - "pip uninstall" is too noisy (Open) #### Pull Requests Authored: - #3806 - Change install command's default behaviour to upgrade packages by default (Closed, superseded by #3972) - #3808 - Fix Tests (Merged) - #3818 - Improve UX and tests of check command (Merged) - #3972 - Add an upgrade-strategy option (Merged) - #3974 - [minor] An aesthetic change to wheel command source (Merged) - #4192 - Move out all the config code to a separate module (Merged) - #4193 - Add the ability to autocorrect a user's command (Open) - #4199 - Fix Tests for Travis CI (Merged) - #4200 - Reduce the API exposed by the configuration class (Merged) - #4232 - Update documentation to mention upgrade-strategy (Merged) - #4233 - Nicer permissions error message (Open) - #4240 - [WIP] Add a configuration command (Open) Participated: - #2716 - Issue #988: new resolver for pip (Closed) - #2975 - Different mock dependencies based on Python version (Merged) - #3744 - Add a "Upgrade all local installed packages" (Open) - #3750 - Add a `pip check` command. (Merged) - #3751 - tox.ini: Add "cover" target (Open) - #3794 - Use the canonicalize_name function for finding .dist-info (Merged) - #4142 - Optionally load C dependencies based on platform (Open) - #4144 - Install build dependencies as specified in PEP 518 (Open) - #4150 - Clarify default for --upgrade-strategy is eager (Merged) - #4194 - Allow passing a --no-progress-bar to the install script to surpress progress bar (Merged) - #4201 - Register req_to_install for cleanup sooner (Merged) - #4202 - Switch to 3.6.0 final as our latest 3.x release (Merged) - #4211 - improve message when installing requirements file (#4127) (Merged) - #4241 - Python 3.6 is supported (Merged) ## References 1. [pypa/pip#988](https://github.com/pypa/pip/issues/988) Tracking issue for adding a proper dependency resolver in pip. Contains links to various useful resources. 1. [pypa/pip#2716](https://github.com/pypa/pip/issues/2716) Prior work by Robert Collins for adding a proper dependency resolver in pip. 1. [Python Dependency Resolution]( https://docs.google.com/document/d/1x_VrNtXCup75qA3glDd2fQOB2TakldwjKZ6pXaA… ) A writeup by Sebastian Awwad on the current state of dependency resolution in pip and PyPI in general. 1. [PSF Application Template]( https://wiki.python.org/moin/SummerOfCode/ApplicationTemplate2017) For guidance on how to write the application and what information is needed. 1. [Stork: Secure Package Management for VM Environments]( http://isis.poly.edu/~jcappos/papers/cappos_stork_dissertation_08.pdf) A Paper by Justin Cappos about Stork, used for reference regarding Backtracking Resolution

3 4

GSoC 2017 - RFC on Proposal
by Pradyun Gedam 25 Mar '17

25 Mar '17

Hello Everyone! I had previously sent a mail on this list, stating that I would like to work on pip's dependency resolution for my GSoC 2017. I now have drafted a proposal for the same; with help from my mentors - Donald Stufft and Justin Cappos. I'd also take this opportunity to thank them for agreeing to be my mentors for this GSoC. I would like to request for comments on my proposal for GSoC - it is hosted at https://gist.github.com/pradyunsg/5cf4a35b81f08b6432f280aba6f511eb. Please find trailing a MarkDown version of the proposal. Thanks, Pradyun Gedam ----- # Adding Proper Dependency Resolution to pip - **Name:** Pradyun S. Gedam - **Email:** [pradyunsg(a)gmail.com][mailto-email] - **Github:** [pradyunsg][github-profile] - **University:** [VIT University, Vellore, India][vit-homepage] - **Course:** Bachelor of Technology in Computer Science and Engineering - **Course Term:** 2016/17 - 2019/20 (4 Year) - **Timezone:** IST (GMT +5:30) - **GSoC Blog RSS Feed URL:** < https://pradyunsg.github.io/gsoc-2017/feed.xml> [github-profile]: http://github.com/pradyunsg/ [vit-homepage]: http://vit.ac.in/ [mailto-email]: mailto:pradyunsg@gmail.com ## About Me I was introduced to Python about five years ago, through "Core Python Programming" by Weasley J Chun. After the initial struggle with Python 2/3, the ball was set rolling and hasn't stopped since. I have fiddled around with Game Programming (PyGame), Computer Vision (OpenCV), Data Analytics (Pandas, SciPy, NumPy), transcompilers (Py2C) and more. As a high school student, I did internship at Enthought in 2013 and 2014. The two summers that I spent at Enthought were a great learning experience and I thoroughly enjoyed the environment there. Other than Python, I have also used C, C++, Web Technologies (JavaScript, HTML, CSS) and preprocessors (Pug, TypeScript, LESS/SASS/SCSS), Java and Bash/Zsh for some other projects. Curious to understand how pip works, I began looking into pip's source code. I started contributing to pip in May 2016. I am now fairly familiar with the design of pip and have a fair understanding of how it works, due to the various contributions I have made to pip in the past year. ## Mentors - Donald Stufft (@dstufft on GitHub) - Justin Cappos (@JustinCappos on GitHub) Communication with the mentors will be done mostly on issues and pull requests on pip's GitHub repository. If at any point in time, a real time discussion is to be done with the mentors, it would be done over IRC or Skype. Email can also be used if needed. ## Proposal This project is regarding improving dependency resolution performed within pip by implementing a dependency resolver within it. ### Abstract Currently, pip does not resolve dependencies correctly when there are conflicting requirements. The lack of dependency resolution has caused hard-to-debug bugs/failures due to the installation of incompatible packages. The lack of a dependency resolver is also a blocker for various other features - adding an upgrade-all functionality to pip and properly determining build-time dependencies for packages are two such features. ### Deliverables At the end of this project, pip will have the ability to: - detect requirement conflicts - resolve conflicting dependency requirements where possible ### Implementation The implementation language will be Python. The code will maintain the compatibility requirements of pip - the same source code will support the multiple Python implementations and version, including but not limited to, CPython 2.7, CPython 3.3+, PyPy 2, PyPy3. New Tests for the functionality introduced will be added to pip's current test suite. User documentation would not be a major part of this project. The only changes would be to mention pip can now resolve dependencies properly. There are certain sections that might need updating. #### Overview The project will be composed of the following stages: 1. Refactor the dependency resolution logic of pip into a separate module. 1. Implement dependency information caching in pip. 1. Implement a backtracking dependency resolver to resolve the dependencies. Every stage depends on the previous ones being completed. This step-wise approach would make incremental improvements so that there is a constant feedback on the work being done as well as making it easy to change course without losing existing work; if needed for some unforeseen reason. #### Discussion There is a class in pip - `RequirementSet`, which is currently a God class. It is responsible for the following: 1. Holding a list of things to be installed 1. Downloading Files 1. Dependency Resolution 1. Unpacking downloaded files (preparation for installation) 1. Uninstalling packages 1. Installing packages This is clearly a bad situation since this is most of the heavy lifting involved in pip. These responsibilities need to be separated and moved out into their independent modules/classes, to allow for simplification of `RequirementSet` while providing a clean platform for the remaining work. This is the most tricky portion of this project, given the complexity of `RequirementSet` as it stands today. There are two kinds of distributions that may be used to install a package - wheels (binary) and sdists (source). When installing a package, pip builds a wheel from an sdist and then proceeds to install the wheel. The difference between the two formats of distribution relevant to this project is - wheels store the information about dependencies within them statically; sdists do not. Determining the dependencies of a wheel distribution is merely the matter of fetching the information from a METADATA file within the `.whl` file. The dependency information for an sdist, on the other hand, can only be determined after running its `setup.py` file on the target system. This means that dependencies of an sdist depend on how its `setup.py` behaves which can change due to variations on target systems or could even contain through random choices. Computation of an sdist's dependencies on the target system is a time-consuming task since it potentially involves fetching a package from PyPI and executing it's setup.py to get the dependency information. In order to improve performance, once an sdist's dependencies are computed, they would be stored in a cache so that during dependency resolution, the dependencies of an sdist are not computed every time they are needed. Further, pip caches wheels it downloads or builds meaning that any installed package or downloaded wheel's dependency information would available statically, without the need to go through the sdist dependency cache. Like the wheel cache, sdist-dependency-cache will be a file system based cache. The sdist-dependency-cache would only be used if the corresponding sdist is being used. Since sdist dependencies are non-deterministic, the cached dependency information is potentially incorrect - in certain corner cases such as using random choices in setup.py files. Such uses are not seen as important enough to cater to, compared the benefits of having a cache. Further, this is already the case with the information in the wheel cache. SAT solver based resolution is not feasible for pip since a SAT solver needs the entire set of packages and their dependencies to compute a solution, which cannot be generated due to the aforementioned non-deterministic behaviour of setup.py file. Computing dependency information for all of PyPI on a target system for installing a package is simply not feasible. The most reasonable approach is using a backtracking solver. Such a solver can be incrementally provided information about the dependencies of a package and would only need dependency information about packages in the dependency graph of the current system. There is a need to keep a cache of visited packages during dependency resolution. A certain package-version combination may be reached via multiple paths and it is an inefficient use of computation time to re-compute that whether it is indeed going to satisfy the requirements or not. By storing information about which package-version combinations have been visited and do (or do not) satisfy the constraints, it is possible to speedup the resolution. Consider the following example: ``` A-1.0 (depends: B) A-2.0 (depends: B and E-1.0) B-1.0 (depends: C and D) C-1.0 (depends: D) D-1.0 D-1.1 (depends: E-2.0) E-1.0 E-2.0 ``` If an installation of A is required, either A-2.0 or D-1.1 should not be installed because they have a conflicting requirement - E. While there are multiple possible solutions to this situation, the "most obvious" one us to use the D-1.0, instead of not installing A-2.0. Further, as multiple packages depend on D, the algorithm would "reach it" multiple times. By maintaining a cache for the visited packages, it is possible to achieve a speedup in such a scenario. #### Details Pull requests would be made on a regular basis during the project to ensure that the feedback loop is quick. This also reduces the possibilities of conflicts due to unrelated changes in pip. All the code will be tested within pip's existing testing infrastructure. It has everything needed to write tests related to all the changes to be made. Every PR made to pip as a part of this project will contain new tests or modifications to existing ones as needed. ##### Stage 1 Initially, some abstractions will be introduced to the pip codebase to improve the reuse of certain common patterns within the codebase. This includes cleaner temporary directory management through a `TempDirectory` class. `RequirementSet.prepare_files` and `RequirementSet._prepare_file` are downloading, unpacking packages as well as doing what pip does as dependency resolution. Taking these functions apart neatly is going to be a tricky task. The following is a listing of the final modules that will be responsible for the various tasks that are currently being performed by `RequirementSet`: - `pip.resolve` - Dependency Resolution - `pip.download` - Downloading Files & Unpacking downloaded files - `pip.req.req_set` - Holding a list of things to be installed / uninstalled - `pip.operations.install` - Installing Packages (preparation for installation) - `pip.operations.uninstall` - Uninstalling Packages To be able to proceed to the next step, only the dependency resolution related code needs to be refactored into a separate module. Other portions of `RequirementSet` are not required to be refactored but the same will be tackled as optional deliverables. In other words, only `pip.resolve` needs to be completed to be able to proceed to the next stage in this project. This is needed since in Stage 3, the resolver would be written in `pip.resolve`, independent of the rest of the codebase. ##### Stage 2 A new module `pip.cache` will be created. Within this module, all the caching will be handled. Thus, the code for the current wheel cache would be moved. The new code for a dependency cache for sdists would also be written here. The new cache would hold all of an sdist's egg-info. The information will be stored on the file-system in a sub directory structure much like that of the wheel cache, in a directory structure based on hash of the sdist file holding the egg-info at the end. This will be done in a class named `EggInfoCache`. `EggInfoCache` cache will be used only if a corresponding wheel for an sdist is not available. Installing an sdist results in the creation of a wheel which contains the dependency information, which would be used over the information available in the `EggInfoCache`. To be able to proceed to the next step, it is required that `EggInfoCache` is implemented. ##### Stage 3 The module `pip.resolve` will be modified and a class named `BacktrackingResolver` will be added to it. The class does what you expect it to do - it would resolve dependencies with recursive backtracking. As described above, there will be some global state stored about the packages that have been explored. Other than the maintenance of global state, in the form of the cache, the rest of the algorithm will essentially follow the same structure as any backtracking algorithm. The project would be complete when the aforementioned resolver is implemented. #### Existing Work There is existing work directly related to dependency resolution in pip, done by multiple individuals. - Robert Collins (un-merged closed pull request on pypa/pip) This has an incomplete backtracking dependency resolver and dependency caching. - Sebastien Awwad (branch on a separate fork) This was used for the depresolve project, to investigate the state of Dependency Resolution in PyPI/pip ecosystem. - `pip-compile` (separate project) This has a backtracking dependency resolver implemented to overcome pip's inablity to resolve dependencies. Their work would be used for reference, where appropriate, during the course of the project. Further, there are many package managers which implement dependency resolution, which would also be looked into. ### Tentative Timeline - Community Bonding Period: **5 May - 29 May** - Clean up and finalize my existing pull requests. - Read existing literature regarding dependency resolution. - Inspect existing implementations of dependency resolvers. GOAL: Be ready for the work coming up. - Week 1: **30 May - 5 June** - Introduce abstractions across pip's codebase to make refactoring `RequirementSet` easier. - Begin breaking down `RequirementSet.prepare_file`. - Week 2: **6 June - 12 June** - Continue working on the refactor of `RequirementSet`. - Week 3: **13 June - 19 June** - Continue working on the refactor of `RequirementSet`. - Finish and polish `pip.resolve`. GOAL: `pip.resolve` module will be ready, using the current resolution strategy. - Week 4: **20 June - 26 June** - Finish and polish all work on the refactor of `RequirementSet`. - Week 5: **27 June - 3 July** - Move code for `WheelCache` into a new `pip.cache` module. - Write tests for `pip.cache.EggInfoCache`, based on `WheelCache`. - Begin implementation of `pip.cache.EggInfoCache`. - Week 6: **4 July - 10 July** - Finish and polish `pip.cache.EggInfoCache`. GOAL: A cache for storing dependency information of sdists would be ready to add to pip. - Week 7: **10 July - 16 July** - Create a comprehensive set of tests for the dependency resolver. (in `tests/unit/test_resolve.py`) - Begin implementation on the backtracking algorithm. GOAL: A comprehensive test bed is ready for testing the dependency resolver. - Week 8: **17 July - 24 July** - Complete a rough implementation of the backtracking algorithm GOAL: An implementation of a dependency resolver to begin running tests on and work on improving. - Week 9: **25 July - 31 July** - Fixing bugs in dependency resolver - Week 10: **1 August - 6 August** - Finish and polish work on dependency resolver GOAL: A ready-to-merge PR adding a backtracking dependency resolver for pip. - Week 11: **6 August - 13 August** Buffer Week. - Week 12: **13 August - 21 August** Buffer Week. Finalization of project for evaluation. If the deliverable is achieved ahead of schedule, the remaining time will be utilized to resolve open issues on pip's repository in the order of priority as determined under the guidance of the mentors. #### Other Commitments I expect to not be able to put in 40 hour/week for at most 3 weeks throughout the working period, due to the schedule of my university. I will have semester-end examinations - from 10th May 2017 to 24th May 2017 - during the Community Bonding Period. My university will re-open for my third semester on 12 July 2017. I expect mid-semester examinations to be held in my University around 20th August 2017. During these times, I would not be able to put in full 40 hour weeks due to the academic workload. I might take a 3-4 day break during this period, regarding which I would be informing my mentor around a week in advance. I will be completely free from 30th May 2017 to 11 July 2017. ### Future Work There seems to be some interest in being able to reuse the above dependency resolution algorithm in other packaging related tools, specifically from the buildout project. I intend to eventually move the dependency resolution code that would come out of this project into a separate library to allow for reuse by installer projects - pip, buildout and other tools. ## Previous Contributions to pip (As on 12th March 2017) ### Issues Authored: - #3785 - Prefering wheel-based installation over source-based installation (Open) - #3786 - Make install command upgrade packages by default (Closed) - #3787 - Check if pip broke the dependency graph and warn the user (Open) - #3807 - Tests fail since change on PyPI (Closed) - #3809 - Switch to TOML for configuration files (Open) - #3871 - Provide a way to perform non-eager upgrades (Closed) - #4198 - Travis CI - pypy broken dues to dependency change in pycrypto (Closed) - #4282 - What's the release schedule? (Closed) Participated: - #59 - Add "upgrade" and "upgrade-all" commands (Open) - #988 - Pip needs a dependency resolver (Open) - #1056 - pip install -U does not remember whether a package was installed with --user (Open) - #1511 - ssl certificate hostname mismatch errors presented badly (Open) - #1668 - Default to --user (Open) - #1736 - Create a command to make it easy to access the configuration file (Open) - #1737 - Don't tell the user what they meant, just do what they meant (Open) - #2313 - Automated the Creation and Upload of Release Artifacts (Open) - #2732 - pip install hangs with interactive setup.py setups (Open) - #3549 - pip -U pip fails (Open) - #3580 - Update requests/urllib3 (Closed) - #3610 - pip install from package from github, with github dependencies (Open) - #3788 - `pip` version suggested is older than the version which is installed (Closed) - #3789 - Error installing Mayavi in Mac OS X (Closed) - #3798 - On using python -m pip install -upgrade pip.. its throwing an error like the one below (Closed) - #3811 - no matching distribution found for install (Closed) - #3814 - pip could not find a version that satisfies the requirement oslo.context (Closed) - #3876 - support git refs in @ syntax (Open) - #4021 - Will you accept PRs with pep484 type hints? (Open) - #4087 - pip list produces error (Closed) - #4149 - Exception thrown when binary is already linked to /usr/local/bin (Open) - #4160 - Pip does not seem to be handling deep requirements correctly (Open) - #4162 - Let --find-links be context aware to support github, gitlab, etc. links (Open) - #4170 - pip list |head raises BrokenPipeError (Open) - #4182 - pip install should install packages in order to avoid ABI incompatibilities in compiled (Open) - #4186 - IOError: [Errno 13] Permission denied: '/usr/local/bin/pip' (Open) - #4206 - Where on Windows 10 is pip.conf or pip.ini located? (Closed) - #4221 - Feature request: Check if user has permissions before downloading files (Closed) - #4229 - "pip uninstall" is too noisy (Open) #### Pull Requests Authored: - #3806 - Change install command's default behaviour to upgrade packages by default (Closed, superseded by #3972) - #3808 - Fix Tests (Merged) - #3818 - Improve UX and tests of check command (Merged) - #3972 - Add an upgrade-strategy option (Merged) - #3974 - [minor] An aesthetic change to wheel command source (Merged) - #4192 - Move out all the config code to a separate module (Merged) - #4193 - Add the ability to autocorrect a user's command (Open) - #4199 - Fix Tests for Travis CI (Merged) - #4200 - Reduce the API exposed by the configuration class (Merged) - #4232 - Update documentation to mention upgrade-strategy (Merged) - #4233 - Nicer permissions error message (Open) - #4240 - [WIP] Add a configuration command (Open) Participated: - #2716 - Issue #988: new resolver for pip (Closed) - #2975 - Different mock dependencies based on Python version (Merged) - #3744 - Add a "Upgrade all local installed packages" (Open) - #3750 - Add a `pip check` command. (Merged) - #3751 - tox.ini: Add "cover" target (Open) - #3794 - Use the canonicalize_name function for finding .dist-info (Merged) - #4142 - Optionally load C dependencies based on platform (Open) - #4144 - Install build dependencies as specified in PEP 518 (Open) - #4150 - Clarify default for --upgrade-strategy is eager (Merged) - #4194 - Allow passing a --no-progress-bar to the install script to surpress progress bar (Merged) - #4201 - Register req_to_install for cleanup sooner (Merged) - #4202 - Switch to 3.6.0 final as our latest 3.x release (Merged) - #4211 - improve message when installing requirements file (#4127) (Merged) - #4241 - Python 3.6 is supported (Merged) ## References 1. [pypa/pip#988](https://github.com/pypa/pip/issues/988) Tracking issue for adding a proper dependency resolver in pip. Contains links to various useful resources. 1. [pypa/pip#2716](https://github.com/pypa/pip/issues/2716) Prior work by Robert Collins for adding a proper dependency resolver in pip. 1. [Python Dependency Resolution]( https://docs.google.com/document/d/1x_VrNtXCup75qA3glDd2fQOB2TakldwjKZ6pXaA… ) A writeup by Sebastian Awwad on the current state of dependency resolution in pip and PyPI in general. 1. [PSF Application Template]( https://wiki.python.org/moin/SummerOfCode/ApplicationTemplate2017) For guidance on how to write the application and what information is needed. 1. [Stork: Secure Package Management for VM Environments]( http://isis.poly.edu/~jcappos/papers/cappos_stork_dissertation_08.pdf) A Paper by Justin Cappos about Stork, used for reference regarding Backtracking Resolution

2 3

Canonical Repo URL: Make "pip install -e foo" work
by Thomas Güttler 25 Mar '17

25 Mar '17

AFAIK it is impossible to do this: pip install -e foo You need to use the repo URL up to now: pip install -e git+https://example.com/repos/foo#egg=foo AFAIK the fast/short implementation of "pip install -e foo" does not work, since pip can't access metadata of package foo without downloading the whole package. Or am I wrong - is this possible? But how cares for useless downloaded bytes? I don't care. It should be possible to download the whole package "foo", then look at the metadata which is provided by it. Take the canonical repo url, and then get the source from the repo. AFAIK there is no official way to define a "Canonical Repo URL" up to now. If I want to provide it for my custom packages. How could I do this? Regards, Thomas Güttler -- Thomas Guettler http://www.thomas-guettler.de/

4 16

Pip different installation behavior when using --no-cache-dir
by Laurent Mazuel 24 Mar '17

24 Mar '17

Hello, I was talking with Brett Cannon at work today about why “python setup.py install” is called when `--no-cache-dir` is specified instead of python setup.py bdist_wheel in the normal case. He didn't know the answer and suggested I ask here to see if it's on purpose or just an oversight of when caching was introduced. Basically we have a situation where we tweak a little the bdist_wheel step in our setup.py to take advantage of the “flat” install of pip, and allow easyinstall to work perfectly from the sdist at the same time. Pip install of the sdist file works like a charm, since as expected pip call bdist_wheel first (and apply our changes): > pip install .\azure-common-1.1.4.zip Processing d:\vsprojects\azure-sdk-for-python-official\wheelhouse\azure-common-1.1.4.zip Building wheels for collected packages: azure-common Running setup.py bdist_wheel for azure-common ... done Stored in directory: C:\Users\lmazuel\AppData\Local\pip\Cache\wheels\4c\3a\10\5e2ef6db79d3785728205a4b5b8348eb41a474ec99505cd865 Successfully built azure-common Installing collected packages: azure-common Successfully installed azure-common-1.1.4 However, the same call with –no-cache-dir bypasses the bdist_wheel step: > pip install --no-cache-dir .\azure-common-1.1.4.zip Processing d:\vsprojects\azure-sdk-for-python-official\wheelhouse\azure-common-1.1.4.zip Installing collected packages: azure-common Running setup.py install for azure-common ... done Successfully installed azure-common-1.1.4 It seems to me that pip should always call bdist_wheel, since in theory the wheel building step can be changed to fit the platform (in my understanding of sdist VS egg VS wheel). And so even –no-cache-dir should call bdist_wheel, even if the wheel is not cached at the end. Pip install of the sdist is not our most common situation since our wheels are universals, but I’m still interesting to improve my pip knowledge 😊 What do you think? Thanks! Laurent

1 0

reproducible builds
by Robin Becker 22 Mar '17

22 Mar '17

An issue has been raised for reportlab to support a specific environment variable namely SOURCE_DATE_EPOCH. The intent is that we should get our time from this variable rather than time.localtime(time.time()) so that produced documents are more invariant. First off is this a reasonable request? The variable is defined by debian here https://reproducible-builds.org/specs/source-date-epoch/ What happens if other distros decide not to use this environment variable? Do I really want distro specific code in the package? In addition we already have our own mechanism for making the produced documents invariant although it might require an extension to support externally specified date & time as in the debian variable. In short where does the distro responsibility and package maintainers boundary need to be? -- Robin Becker

10 18

[proposal] version-free + lookup-friendly dist-info location
by Ronny Pfannschmidt 18 Mar '17

18 Mar '17

Hi everyone, while looking over the recent peps i noticed that we keep a few inherent inefficiencies in where to find dist-info folders because they include version numbers, to get a distribution we have to search for it which is no longer really sensible as we no longer have multi-version installation in any upcoming standard. in order to address that i'd like to propose to switch from "{distribution}-{version}.dist-info/" to "{distribution}.dist-info/" given that it has been used since quite a while i would prefer a quick feedback loop from the ML before thinking about writing a PEP. -- Ronny

7 15

Best practice to build binary wheels on Github+Travis and upload to PyPI
by Lele Gaifax 16 Mar '17

16 Mar '17

Hi all, I'd like to learn how to configure a project I keep on Github so that at release time it will trigger a build of binary wheels for different versions of Python 3 and eventually uploading them to PyPI. At first I tried to follow the Travis deploy instruction[1], but while that works for source distribution it cannot be used to deploy binary wheels because AFAICT Travis does not build “manylinux1”-marked wheels. I then found the manylinux-demo project[2] that uses Docker and contains a a script able to build the wheels for every available version of Python. OTOH, it does not tackle to PyPI upload step. I will try to distill a custom recipe for my own needs looking at how other packages implemented this goal, but I wonder if there is already some documentation that could help me understanding better how to intersect the above steps. Thanks in advance for any hint, ciao, lele. [1] https://docs.travis-ci.com/user/deployment/pypi/ [2] https://github.com/pypa/python-manylinux-demo -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

2 2

Re: [Distutils] GnuPG signatures on PyPI: why so few?
by Ian Cordasco 15 Mar '17

15 Mar '17

On Mar 11, 2017 6:47 PM, "Ben Finney" <bignose(a)debian.org> wrote: Howdy all, What prospects are there for PyPI to have GnuPG-signed packages by default? Debian's UScan has the ability to find, download, and verify the GnuPG signature for a package source release. Lintian will remind the maintainer if a Debian source package is not taking advantage of this. However, this only works if upstream releases are actually accompanied by a valid GnuPG signature each time. The PyPI infrastructure supports this; why isn't it more widely encouraged? This thread from 2016 has a possible answer: while you can use GPG as is to verify that yes, "Donald Stufft" signed a particular package, you cannot use it to determine if "Donald Stufft" is *allowed* to sign for that package, a valid signature from me on the requests project should be just as invalid as an invalid signature from anyone on the requests project. The only namespacing provided by GPG itself is "trusted key" vs "not trusted key". […] I am aware of a single tool anywhere that actively supports verifying the signatures that people upload to PyPI, and that is Debian's uscan program. […] All in all, I think that there is not a whole lot of point to having this feature in PyPI, it is predicated a bunch of invalid assumptions (as detailed above) and I do not believe end users are actually even using the keys that are being uploaded. […] Thus, I would like to remove this feature from PyPI […]. <URL:https://mail.python.org/pipermail/distutils-sig/2016- May/028933.html> The thread has some discussion, and Barry Warsaw makes the case for Debian's use for signed releases. The last (?) post in the thread has a kind of interim conclusion: My main concern when implementing this is how to communicate it to users […]. [A move that gives the impression] "we're getting rid of this thing that only kinda works now in favor of something amazing that doesn't exist yet" is just not a popular move. <URL:https://mail.python.org/pipermail/distutils-sig/2016- May/028946.html> In response to polite requests for signed releases, some upstream I've only ever seen condescending requests in the past but perhaps we have different definitions of "polite" or perhaps things have genuinely changed. maintainers are now pointing to that thread and closing bug reports as “won't fix”. You may have noticed in that thread that there are plans for better mechanisms. Mechanisms that don't add significantly more burden to maintainers of the software we know and love who do this for free and with their spare time. What prospect is there in the Python community to get signed upstream releases become the obvious norm? Not every package on PyPI is redistributed via Linux packagers. Why then should someone publishing their tiny little first package have to go through the hassle of creating a GPG key? As a maintainer of Twine, I will never force someone to have learned how to install GPG on their platform, create a key that package maintainers won't belittle them for, and maintain the key's security in order to upload something to PyPI. Further GPG depends on trust. Do you mean to imply that Debian trusts PyPI packages with a signature more than those without? Even if the key used to sign it has never been signed by another person? What about keys signed by people you've never met? Someone can manufacture their own web of trust if they want to. Why is GPG seen as done kind of magic authenticity bullet? If you can find a tool that is easy to install on Linux, Windows, and Mac, which solves the problems above by virtue of having very good defaults, and is accessible to anyone with less than a few hours to waste on it... Then maybe I would collaborate to make it a requirement.

7 9

Re: [Distutils] GnuPG signatures on PyPI: why so few?
by Nick Coghlan 15 Mar '17

15 Mar '17

On 14 March 2017 at 03:46, Steve Dower <steve.dower(a)python.org> wrote: > Another drive-by contribution: what if twine printed the hashes for > anything it uploads with a message basically saying "here are the things > you should publish somewhere for this release so people can check the > validity of your packages after they download them"? > > I suspect many publishers have never considered this is something they > could or should do. Some very basic prompting could easily lead to it > becoming part of the normal workflow. > Huh, and with most PyPI publishers using public version control systems, their source control repo itself could even serve as "a trusted channel that they control and the PyPI service can't influence". For example, the artifact hashes could be written out by default to: .released_artifacts/<version>/<artifact_name>.sha256 And if twine sees the hash file exists before it starts the upload, it could complain that the given artifact had already been published even before PyPI complains about it. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 6

PEP 426 moved back to Draft status
by Nick Coghlan 15 Mar '17

15 Mar '17

Hi folks, After a few years of dormancy, I've finally moved the metadata 2.0 specification back to Draft status: https://github.com/python/peps/commit/8ae8b612d4ea8b3bf5d8a7b795ae8aec48bbb… Based on our last round of discussion, I've culled a lot of the complexity around dependency declarations, cutting it back to just 4 pre-declared extras (dev, doc, build, test), and some reserved extras that can be used to say "don't install this, even though you normally would" (self, runtime). I've also deleted a lot of the text related to thing that we now don't need to worry about until the first few standard metadata extensions are being defined. I think the biggest thing it needs right now is a major editing pass from someone that isn't me to help figure out which explanatory sections can be culled completely, while still having the specification itself make sense. >From a technical point of view, the main "different from today" piece that we have left is the Provide & Obsoleted-By fields, and I'm seriously wondering if it might make sense to just delete those entirely for now, and reconsider them later as a potential metadata extension. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 10

FYI - "Trending" on Warehouse
by Donald Stufft 13 Mar '17

13 Mar '17

Just an FYI, I’ve replaced the long stagnation “top downloads” on the Warehouse / pypi.org <http://pypi.org/> homepage with “Trending” projects. Since “trending” can mean a lot of different things as far as how it’s computed, here’s how I’m currently doing it [1]: Using a look back over the last 30 days of downloads I compute a “zscore” for each project for yesterday (effectively, how many standard deviations away from from the mean yesterday was for that project in total downloads). The trending projects is then the top 5 projects in terms of zscore for yesterday (recomputed every day at ~3am UTC). Because it’s a lot easier for a project with an average of 5 downloads to jump to 100 than it is for a project with 50000 downloads jump to 1000000 I have tried to exclude any projects with very few downloads from this, so in order to qualify to be trending a project must receive at least 5,000 downloads in a month. If you happen to be some sort of sciencey person and you know of a better way to query what is effectively a table with a row for every download for every project to determine which ones are trending, feel free to open an issue or create a PR or something. I don’t really know what I’m doing here :) Anyways, that’s all! [1] https://github.com/pypa/warehouse/blob/a36435b9865000cdaae97b948af48c33f7d8… <https://github.com/pypa/warehouse/blob/a36435b9865000cdaae97b948af48c33f7d8…> — Donald Stufft

1 0

Python 3.5
by Adh 11 Mar '17

11 Mar '17

Hello, I do not arrive to install python 3.5 in the terminal, I use Ubuntu. Wich command do I have to tape to install it? Thank you for your answer

2 1

Critical: PR for packaging.specifiers not found issue
by Andrey Galkin 10 Mar '17

10 Mar '17

Can someone please take a look at https://github.com/pypa/setuptools/pull/990 ? Previously, the issue was reported by another user and then rejected: https://github.com/pypa/setuptools/issues/967 The problem is reproducible on both Python 2.7.13 and 3.5.3 shipped in Debian Stretch. Yes, it's not yet visible in other OSes including Ubuntu with previous patch versions of 2.7 & 3.5. I believe it's related to this change: bpo-27419: Standard __import__() no longer look up “__import__” in globals or builtins for importing submodules or “from import”. Fixed handling an error of non-string package name. https://bugs.python.org/issue27419 The packaging module does not export specifiers in __init__.py. It can be easily triggered with "pip -e source_dir". I can confirm the issue gets vanished once PR with one-liner is applied to latest setuptools located in virtualenv. Error output: Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 1, in <module> File "/vagrant/setup.py", line 67, in <module> setup(**config) File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/home/vagrant/.virtualenv-2.7/local/lib/python2.7/site-packages/setuptools/dist.py", line 320, in __init__ _Distribution.__init__(self, attrs) File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__ self.finalize_options() File "/home/vagrant/.virtualenv-2.7/local/lib/python2.7/site-packages/setuptools/dist.py", line 387, in finalize_options ep.load()(self, ep.name, value) File "/home/vagrant/.virtualenv-2.7/local/lib/python2.7/site-packages/setuptools/dist.py", line 166, in check_specifier except packaging.specifiers.InvalidSpecifier as error: AttributeError: 'module' object has no attribute 'specifiers' Complete output from command python setup.py egg_info: Traceback (most recent call last): File "/home/vagrant/.virtualenv-3.5/lib/python3.5/site-packages/setuptools/dist.py", line 165, in check_specifier packaging.specifiers.SpecifierSet(value) AttributeError: module 'packaging' has no attribute 'specifiers' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 1, in <module> File "/vagrant/setup.py", line 67, in <module> setup(**config) File "/usr/lib/python3.5/distutils/core.py", line 108, in setup _setup_distribution = dist = klass(attrs) File "/home/vagrant/.virtualenv-3.5/lib/python3.5/site-packages/setuptools/dist.py", line 320, in __init__ _Distribution.__init__(self, attrs) File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__ self.finalize_options() File "/home/vagrant/.virtualenv-3.5/lib/python3.5/site-packages/setuptools/dist.py", line 387, in finalize_options ep.load()(self, ep.name, value) File "/home/vagrant/.virtualenv-3.5/lib/python3.5/site-packages/setuptools/dist.py", line 166, in check_specifier except packaging.specifiers.InvalidSpecifier as error: AttributeError: module 'packaging' has no attribute 'specifiers'

1 0

Data on requirement files on GitHub
by Jannis Gebauer 09 Mar '17

09 Mar '17

Hi, I ran a couple of queries against GitHubs public big query dataset [0] last week. I’m interested in requirement files in particular, so I ran a query extracting all available requirement files. Since queries against this dataset are rather expensive ($7 on all repos), I thought I’d share the raw data here [1]. The data contains the repo name, the requirements file path and the contents of the file. Every line represents a JSON blob, read it with: with open('data.json') as f: for line in f.readlines(): data = json.loads(line) Maybe that’s of interest to some of you. If you have any ideas on what to do with the data, please let me know. — Jannis Gebauer [0]: https://cloud.google.com/bigquery/public-data/github <https://cloud.google.com/bigquery/public-data/github> [1]: https://github.com/jayfk/requirements-dataset <https://github.com/jayfk/requirements-dataset>

4 4

custom setup.py link arguments order
by Gabriel Ganne 08 Mar '17

08 Mar '17

Hi, I'm currently writing a python C module which has a chained dependency: - mymodule requires libb - libb requires liba To that effect, within setup.py, I link against both liba and libb libraries=['a', 'b'], Also, as I'm working on Ubuntu, I want to add -Wl,--no-as-needed to make sure that the symbols not immediately needed will still be stripped. extra_link_args=['-Wl,--no-as-needed'], However, it seems that the extra_link_args are systematically appended at the end of the link line, but for this to work, the '-Wl,--no-as-needed' argument need to be *before* the link against my two libraries. How can I choose the order of my link arguments that I pass to gcc using setup.py ? Best regards, -- Gabriel Ganne -- Gabriel Ganne

1 0

install_requires setup.py install vs pip install
by Tim Golden 07 Mar '17

07 Mar '17

I have a setup.py which looks like this: from setuptools import setup setup( name='install_requires', py_modules = ["install_requires"], install_requires=['PyQt5'], ) For the purposes of the discussion, there is an install_requires.py in the same directory. I have created and activated a standard Python 3.5 venv on Windows: py -3.5 -mvenv .venv .venv\scripts\activate.bat python -mpip install --upgrade pip (I don't believe the Python version or the venv matter here, but including them for reproducibility). If I pip install the module, the PyQt5 install dependency is found and installed: (.venv) C:\work-in-progress\install_requires>pip install . Processing c:\work-in-progress\install_requires Collecting PyQt5 (from install-requires==0.0.0) Using cached PyQt5-5.8.1-5.8.0-cp35.cp36.cp37-none-win_amd64.whl Collecting sip==4.19 (from PyQt5->install-requires==0.0.0) Using cached sip-4.19-cp35-none-win_amd64.whl Installing collected packages: sip, PyQt5, install-requires Running setup.py install for install-requires ... done Successfully installed PyQt5-5.8.1 install-requires-0.0.0 sip-4.19 If, instead, I setup.py install the module, I get the following messages: Processing dependencies for install-requires==0.0.0 Searching for PyQt5 Reading https://pypi.python.org/simple/PyQt5/ No local packages or download links found for PyQt5 error: Could not find suitable distribution for Requirement.parse('PyQt5') However, if I substitute instead "requests" or "simplejson" (both well-known packages) then setup.py install succeeds. My cursory inspection of https://pypi.python.org/simple/pyqt5/ doesn't reveal anything obviously different except for the complexity of the filenames. I've searched around, including in the archives of this group, but can't find that this is a known issue. If I had to guess from the evidence, it would be that pip ships a more sophisticated parser of complex wheel filenames than setuptools. Can anyone advise, please? TJG

2 2

Deprecating download counts in API?
by Donald Stufft 06 Mar '17

06 Mar '17

So, as most folks are aware PyPI has long had a cumulative download count available in it’s API. This has been on and off again broken for a *long* time and arguably the numbers in there have been “wrong” even when it was working because we had no way to reproduce them from scratch (and thus whenever a bug occurred we’d flat out lose data or add incorrect data with no way to correct it). In the meantime, we’ve gotten a much better source of querying for download information available inside of Google’s BigQuery database [1][2]. Not only is this able to be recreated “from scratch” so we can, if needed, fix massive data bugs but it provides MUCH more information than the previous downloads and a very powerful query language to go along with it. Unless there is some sort of massive outcry, I plan to deprecate and ultimately remove the download counts available in the PyPI API, instead preferring people to start using the BigQuery data instead. This more or less reflects the current state of things, since it has been on and off broken (typically broken) for something like a year now. [1] https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html <https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html> [2] https://langui.sh/2016/12/09/data-driven-decisions/ <https://langui.sh/2016/12/09/data-driven-decisions/> — Donald Stufft

2 4

Announcing new Buildout documentation
by Jim Fulton 06 Mar '17

06 Mar '17

The old horrible doctest-based buildout documentation has finally been replaced: http://docs.buildout.org Jim -- Jim Fulton http://jimfulton.info

2 1

help required
by Dominic Lund 06 Mar '17

06 Mar '17

I have a third party python module I have just downloaded It is in a zip file in my download directory How do I use pip to install it? Or can I 'install' it manually?

2 1

GSoC 2017 - Plan of Action for dependency resolver
by Pradyun Gedam 04 Mar '17

04 Mar '17

Hello Everyone! Google released the list of accepted organizations for GSoC 2017 and PSF is one of them. I guess this would a good time for me to seek feedback on the approach I'm planning to take for my potential GSoC project. I hope this mailing list is the right place to do so. --- Here's my current plan of action along with reasoning for the choices made: A separate PR will be made for each of these stages. Every stage does depend on the previous ones being completed. 1. Refactor all dependency resolution responsibility in pip into a new, separate module. This would allow any future changes/improvements in the dependency resolution to be added without major changes in the rest of the code-base. As of today, the RequirementSet class within pip seems to be doing a lot of work and dependency resolution is a responsibility that doesn't need to given to it, especially when it's avoidable. 2. Implement dependency information caching. This would allow the resolver to not cause the re-computation of the dependencies of a package, if they have already been computed, speeding up the resolution. 3. Implement a backtracking resolver. A backtracking solver would be appropriate given that we don't have a way to pre-compute the dependencies for *all* the packages or statically determine the dependencies - a SAT solver would not be feasible. 4. (if time permits) Move any dependency resolution code out into a separate library. This would make it possible for other projects (like buildout or a future pip replacement) to reuse the dependency resolver. By making each of the stages separate PRs, incremental improvements would be made so that even if I leave this project midway, there will be some work merged already if someone comes back to this problem later. That said, I don't intend to leave this project midway. I do intend to reuse some of the work done by Robert Collins in PR #2716 on pip's GitHub repository. Stages 2 and 3 are separate because I see them as distinctly different tasks which touch very different portions of the code-base. There's is strong coupling between them though. I'm looking forward to the feedback. :) Regards, Pradyun

7 14

How to change the linking command in a setuptools building process?
by Xavier Olive 01 Mar '17

01 Mar '17

I maintain a Cython binding to some OCaml code (through their respective C interface). For past versions, I managed to cheat and distribute a wheel file for Windows through cross-compilation. Now, I finally managed a clean and native way to produce the library for Windows 64 bits. For the 32 bits cross-compiled version, I had a specific target in my setup.py with the proper commands to execute. Back on Windows, I would like to stick to a setuptoolsic way of doing, but the thing is I need to replace the regular linking command link.exe with a different tool (resp. flexlink.exe, shipped with OCaml on Windows) Don't panic: flexlink.exe just builds some assembler shit before compiling and linking with the regular link.exe. It is the proper way to link OCaml executables and shared libraries under Windows. For MacOS and Linux, the traditional Extension pattern works like a charm as follows (mlobject is produced by OCaml a bit earlier in the file after some timestamp checks, asmrunlib is the full path to the equivalent of python36.dll for OCaml) : extensions = [ Extension("foo", ["foo.pyx", "interface_c.c"], language="c", include_dirs=INCLUDE, extra_compile_args=compileargs, extra_link_args=[mlobject, asmrunlib, ] ) ] Let's say I limit myself to Python>=3.5, I guess (by comparison with too big projects like NumPy) I would need to start by extending distutils._msvccompiler.MSVCCompiler and replace the self.linker = _find_exe("link.exe", paths) with something based on flexlink.exe. The problem is that I have no idea how they manage the plumbing work that comes next (connecting this extended compiler and making it look like the regular msvc to the setup process). I suppose it is not thoroughly documented anywhere and that if they were able to do more than that in NumPy, I should be able to reach my goal somehow. My setup.py is still reasonably basic and a solution that keeps the whole building/packaging process in one single file would be great! Xavier (Question first asked here https://stackoverflow.com/q/42519377/1595335 before being advised this mailing-list)

1 0

win amd_x64 Python 2.7.8 --> 2.7.13 woes
by Robin Becker 01 Mar '17

01 Mar '17

I find my extensions compiled for windows amd_x64 with python 2.7.8 no longer work after I updated python to 2.7.13. Is that expected? I had assumed that the cpy27 wheels that I make would work with any python 2.7, but this makes me doubt that. To get the reportlab tests to complete I rebuilt all my extensions and installed a newer version of pillow. In addition to that the uninstallation of the amd64 python 2.7.8 has also uninstalled the x86 version of python 2.7.8 :( -- Robin Becker

1 1