Mailman 3 August 2015 - Distutils-SIG

Can now compile simple extensions
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

Versioned trove classifiers for Django
by James Bennett April 6, 2017

April 6, 2017

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

Working toward Linux wheel support
by Nate Coraor Sept. 21, 2015

Sept. 21, 2015

Hi all, I've recently been working on adding SOABI support for Python 2.x and other pieces needed to get wheels w/ C extensions for Linux working. Here's the work for wheels: https://bitbucket.org/pypa/wheel/pull-request/54/ Based on that, I've added support for those wheels to pip here: https://github.com/natefoo/pip/tree/linux-wheels As mentioned in the wheels PR, there are some questions and decisions made that I need guidance on: - On Linux, the distro name/version (as determined by platform.linux_distribution()) will be appended to the platform string, e.g. linux_x86_64_ubuntu_14_04. This is going to be necessary to make a reasonable attempt at wheel compatibility in PyPI. But this may violate PEP 425. - By default, wheels will be built using the most specific platform information. In practice, I build our wheels[1] using Debian Squeeze in Docker and therefore they should work on most currently "supported" Linuxes, but allowing such wheels to PyPI could still be dangerous because forward compatibility is not always guaranteed (e.g. if a SO version/name changes, or a C lib API method changes in a non-backward compatible way but the SO version/name does not change). That said, I'd be happy to make a much more generalized version of our docker-build[2] system that'd allow package authors to easily/rapidly build distro/version-specific wheels for many of the popular Linux distros. We can assume that a wheel built on a vanilla install of e.g. Ubuntu 14.04 will work on any other installation of 14.04 (this is what the distro vendors promise, anyway). - I attempt to set the SOABI if the SOABI config var is unset, this is for Python 2, but will also be done even on Python 3. Maybe that is the wrong decision (or maybe SOABI is guaranteed to be set on Python 3). - Do any other implementations define SOABI? PyPy does not, I did not test others. What should we do with these? Because the project I work for[3] relies heavily on large number of packages, some of which have complicated build-time dependencies, we have always provided them as eggs and monkeypatched platform support back in to pkg_resources. Now that the PyPA has settled on wheels as the preferred binary packaging format, I am pretty heavily motivated to do the work to work out all the issues with this implementation. Thanks, --nate [1] https://wheels.galaxyproject.org/ [2] https://github.com/galaxyproject/docker-build/ [3] https://galaxyproject.org/

22 110

Reviving PEP 470 - Removing External Hosting support on PyPI
by Donald Stufft Sept. 4, 2015

Sept. 4, 2015

While developing Warehouse, one of the things I wanted to get done was a final ruling on PEP 470. With that in mind I’d like to bring it back up for discussion and hopefully ultimately a ruling. Their are two major differences in this version of PEP 470, and I’d like to point them out explicitly. Removal of the “External Repository Discover” feature. I’ve been thinking about this for awhile, and I finally removed it. I’ve always been uncomfortable with this feature and I finally realized why it was. Essentially, the major use case for not hosting things on PyPI that I think PyPI can reasonably be expected to accommodate is people who cannot publish their software to the US for various reasons. At the time I came up with the solution I did, It was an attempt to placate the folks who were against PEP 470 while assuming very few people would ever actually use it, essentially a junk feature to push the PEP through. I think that the feature itself is a bad feature and I think it presents a poor experience for people who want to use it, so I’ve removed it from the PEP and instead focused the PEP on explicitly recommending that all installers should implement the ability to specify multiple repositories and deprecating and removing the ability for finding anything but files hosted by the repository itself on /simple/. I recognize this is a regression for anyone who *does* have concerns with uploading their projects to a server hosted in the US. If there is someone that has this concern, and is also willing to put in the effort and legwork required, I will happily collaborate with them to design a solution that both follows whatever legal requirements they might have, as well as provides a good experience for people using PyPI and pip. I have some rough ideas on what this could look like, but I think it’s really a separate discussion since I believe externally hosted files like we were is an overall bad experience for people and is largely a historic accident from how PyPI and Python packaging has evolved. I don’t want to derail this thread or PEP exploring these ideas (some of which I don’t even know if they would satisfy the requirements since it’s all dealing with legal jurisdictions other than my own), but i wanted to make explicit that someone who knows the legalities and is willing to put in the work can reach out to me. The other major difference is that I’ve shortened the time schedule from 6 months to 3 months. Given that authors are either going to upload their projects to PyPI or not and there is no longer a need to setup an external index I think a shorter time schedule is fine, especially since they will be given a script they can run that will spider their projects for any installable files and upload them to PyPI for them in a quick one shot deal that would require very little effort for them. Everything else in the PEP is basically the same except for rewordings. I do need a BDFL Delegate for this PEP, Richard does not have the time to do it and the other logical candidate for a PyPI centric PEP is myself, but I don’t feel it’s appropriate to BDFL Delegate my own PEP. You can see the PEP online at https://www.python.org/dev/peps/pep-0470/ (make sure it’s updated and you see the one that has Aug 26 2015 in it’s Post History). The PEP has also been inlined below. ----------------- Abstract ======== This PEP proposes the deprecation and removal of support for hosting files externally to PyPI as well as the deprecation and removal of the functionality added by PEP 438, particularly rel information to classify different types of links and the meta-tag to indicate API version. Rationale ========= Historically PyPI did not have any method of hosting files nor any method of automatically retrieving installables, it was instead focused on providing a central registry of names, to prevent naming collisions, and as a means of discovery for finding projects to use. In the course of time setuptools began to scrape these human facing pages, as well as pages linked from those pages, looking for things it could automatically download and install. Eventually this became the "Simple" API which used a similar URL structure however it eliminated any of the extraneous links and information to make the API more efficient. Additionally PyPI grew the ability for a project to upload release files directly to PyPI enabling PyPI to act as a repository in addition to an index. This gives PyPI two equally important roles that it plays in the Python ecosystem, that of index to enable easy discovery of Python projects and central repository to enable easy hosting, download, and installation of Python projects. Due to the history behind PyPI and the very organic growth it has experienced the lines between these two roles are blurry, and this blurring has caused confusion for the end users of both of these roles and this has in turn caused ire between people attempting to use PyPI in different capacities, most often when end users want to use PyPI as a repository but the author wants to use PyPI solely as an index. This confusion comes down to end users of projects not realizing if a project is hosted on PyPI or if it relies on an external service. This often manifests itself when the external service is down but PyPI is not. People will see that PyPI works, and other projects works, but this one specific one does not. They often times do not realize who they need to contact in order to get this fixed or what their remediation steps are. PEP 438 attempted to solve this issue by allowing projects to explicitly declare if they were using the repository features or not, and if they were not, it had the installers classify the links it found as either "internal", "verifiable external" or "unverifiable external". PEP 438 was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with the final transition implemented in pip 1.5 (released on Jan 2, 2014). PEP 438 was successful in bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed ups for a lot of people, however it did so by introducing a new point of confusion and pain for both the end users and the authors. By moving to using explicit multiple repositories we can make the lines between these two roles much more explicit and remove the "hidden" surprises caused by the current implementation of handling people who do not want to use PyPI as a repository. Key User Experience Expectations -------------------------------- #. Easily allow external hosting to "just work" when appropriately configured at the system, user or virtual environment level. #. Eliminate any and all references to the confusing "verifiable external" and "unverifiable external" distinction from the user experience (both when installing and when releasing packages). #. The repository aspects of PyPI should become *just* the default package hosting location (i.e. the only one that is treated as opt-out rather than opt-in by most client tools in their default configuration). Aside from that aspect, hosting on PyPI should not otherwise provide an enhanced user experience over hosting your own package repository. #. Do all of the above while providing default behaviour that is secure against most attackers below the nation state adversary level. Why Additional Repositories? ---------------------------- The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisfy the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familiar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects that wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. This could also be used for projects wishing to host files which cannot be uploaded to PyPI, such as multi-gigabyte data files or, currently at least, Linux Wheels. Why Not PEP 438 or Similar? --------------------------- While the additional search location support has existed in pip and setuptools for quite some time support for PEP 438 has only existed in pip since the 1.4 version, and still has yet to be implemented in setuptools. The design of PEP 438 did mean that users still benefited for projects which did not require external files even with older installers, however for projects which *did* require external files, users are still silently being given either potentially unreliable or, even worse, unsafe files to download. This system is also unique to Python as it arises out of the history of PyPI, this means that it is almost certain that this concept will be foreign to most, if not all users, until they encounter it while attempting to use the Python toolchain. Additionally, the classification system proposed by PEP 438 has, in practice, turned out to be extremely confusing to end users, so much so that it is a position of this PEP that the situation as it stands is completely untenable. The common pattern for a user with this system is to attempt to install a project possibly get an error message (or maybe not if the project ever uploaded something to PyPI but later switched without removing old files), see that the error message suggests ``--allow-external``, they reissue the command adding that flag most likely getting another error message, see that this time the error message suggests also adding ``--allow-unverified``, and again issue the command a third time, this time finally getting the thing they wish to install. This UX failure exists for several reasons. #. If pip can locate files at all for a project on the Simple API it will simply use that instead of attempting to locate more. This is generally the right thing to do as attempting to locate more would erase a large part of the benefit of PEP 438. This means that if a project *ever* uploaded a file that matches what the user has requested for install that will be used regardless of how old it is. #. PEP 438 makes an implicit assumption that most projects would either upload themselves to PyPI or would update themselves to directly linking to release files. While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so. More concerning however, is the fact that very few projects have opted to directly and safely link to files and instead they still simply link to pages which must be scraped in order to find the actual files, thus rendering the safe variant (``--allow-external``) largely useless. #. Even if an author wishes to directly link to their files, doing so safely is non-obvious. It requires the inclusion of a MD5 hash (for historical reasons) in the hash of the URL. If they do not include this then their files will be considered "unverified". #. PEP 438 takes a security centric view and disallows any form of a global opt in for unverified projects. While this is generally a good thing, it creates extremely verbose and repetitive command invocations such as:: $ pip install --allow-external myproject --allow-unverified myproject myproject $ pip install --allow-all-external --allow-unverified myproject myproject Multiple Repository/Index Support ================================= Installers SHOULD implement or continue to offer, the ability to point the installer at multiple URL locations. The exact mechanisms for a user to indicate they wish to use an additional location is left up to each individual implementation. Additionally the mechanism discovering an installation candidate when multiple repositories are being used is also up to each individual implementation, however once configured an implementation should not discourage, warn, or otherwise cast a negative light upon the use of a repository simply because it is not the default repository. Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Installers SHOULD also implement some mechanism for removing or otherwise disabling use of the default repository. The exact specifics of how that is achieved is up to each individual implementation. Installers SHOULD also implement some mechanism for whitelisting and blacklisting which projects a user wishes to install from a particular repository. The exact specifics of how that is achieved is up to each individual implementation. Deprecation and Removal of Link Spidering ========================================= A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP 438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will be defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set permanently to ``pypi-only``. At the same time, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 3 months from the time of that email that all external links will be removed from the installer APIs. This email **MUST** include instructions for converting their projects to be hosted on PyPI and **MUST** include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email **MUST** also include instructions for setting up their own index page. This email must also contain a link to the Terms of Service for PyPI as many users may have signed up a long time ago and may not recall what those terms are. Finally this email must also contain a list of the links registered with PyPI where we were able to detect an installable file was located. Two months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of three. Finally a month later all projects will be switched to the ``pypi-only`` mode and PyPI will be modified to remove the externally linked files functionality. Summary of Changes ================== Repository side --------------- #. Deprecate and remove the hosting modes as defined by PEP 438. #. Restrict simple API to only list the files that are contained within the repository. Client side ----------- #. Implement multiple repository support. #. Implement some mechanism for removing/disabling the default repository. #. Deprecate / Remove PEP 438 Impact ====== To determine impact, we've looked at all projects using a method of searching PyPI which is similar to what pip and setuptools use and searched for all files available on PyPI, safely linked from PyPI, unsafely linked from PyPI, and finally unsafely available outside of PyPI. When the same file was found in multiple locations it was deduplicated and only counted it in one location based on the following preferences: PyPI > Safely Off PyPI > Unsafely Off PyPI. This gives us the broadest possible definition of impact, it means that any single file for this project may no longer be visible by default, however that file could be years old, or it could be a binary file while there is a sdist available on PyPI. This means that the *real* impact will likely be much smaller, but in an attempt not to miscount we take the broadest possible definition. At the time of this writing there are 65,232 projects hosted on PyPI and of those, 59 of them rely on external files that are safely hosted outside of PyPI and 931 of them rely on external files which are unsafely hosted outside of PyPI. This shows us that 1.5% of projects will be affected in some way by this change while 98.5% will continue to function as they always have. In addition, only 5% of the projects affected are using the features provided by PEP 438 to safely host outside of PyPI while 95% of them are exposing their users to Remote Code Execution via a Man In The Middle attack. Data Sovereignty ================ In the discussions around previous versions of this PEP, one of the key use cases for wanting to host files externally to PyPI was due to data sovereignty requirements for people living in jurisdictions outside of the USA, where PyPI is currently hosted. The author of this PEP is not blind to these concerns and realizes that this PEP represents a regression for the people that have these concerns, however the current situation is presenting an extremely poor user experience and the feature is only being used by a small percentage of projects. In addition, the data sovereignty problems requires familarity with the laws outside of the home jurisdiction of the author of this PEP, who is also the principal developer and operator of PyPI. For these reasons, a solution for the problem of data sovereignty has been deferred and is considered outside of the scope for this PEP. If someone for whom the issue of data sovereignty matters to them wishes to put forth the effort, then at that time a system can be designed, implemented, and ultimately deployed and operated that would satisfy both the needs of non US users that cannot upload their projects to a system on US soil and the quality of user experience that is attempted to be created on PyPI. Rejected Proposals ================== Allow easier discovery of externally hosted indexes --------------------------------------------------- A previous version of this PEP included a new feature added to both PyPI and installers that would allow project authors to enter into PyPI a list of URLs that would instruct installers to ignore any files uploaded to PyPI and instead return an error telling the end user about these extra URLs that they can add to their installer to make the installation work. This idea is rejected because it provides a similar painful end user experience where people will first attempt to install something, get an error, then have to re-run the installation with the correct options. Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. * Continue on the suggested path of PEP 438 and remove the option to unsafely host externally but continue to allow the option to safely host externally. These proposals are rejected because: * The classification system introduced in PEP 438 in an entirely unique concept to PyPI which is not generically applicable even in the context of Python packaging. Adding additional concepts comes at a cost. * The classification system itself is non-obvious to explain and to pre-determine what classification of link a project will require entails inspecting the project's ``/simple/<project>/`` page, and possibly any URLs linked from that page. * The ability to host externally while still being linked for automatic discovery is mostly a historic relic which causes a fair amount of pain and complexity for little reward. * The installer's ability to optimize or clean up the user interface is limited due to the nature of the implicit link scraping which would need to be done. This extends to the ``--allow-*`` options as well as the inability to determine if a link is expected to fail or not. * The mechanism paints a very broad brush when enabling an option, while PEP 438 attempts to limit this with per package options. However a project that has existed for an extended period of time may often times have several different URLs listed in their simple index. It is not unusual for at least one of these to no longer be under control of the project. While an unregistered domain will sit there relatively harmless most of the time, pip will continue to attempt to install from it on every discovery phase. This means that an attacker simply needs to look at projects which rely on unsafe external URLs and register expired domains to attack users. Implement this PEP, but Do Not Remove the Existing Links -------------------------------------------------------- This is essentially the backwards compatible version of this PEP. It attempts to allow people using older clients, or clients which do not implement this PEP to continue on as if nothing had changed. This proposal is rejected because the vast bulk of those scenarios are unsafe uses of the deprecated features. It is the opinion of this PEP that silently allowing unsafe actions to take place on behalf of end users is simply not an acceptable solution. Copyright ========= This document has been placed in the public domain. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

13 50

Bogus search result
by Antoine Pitrou Aug. 31, 2015

Aug. 31, 2015

Hello, I don't if that's just me, but I go to https://pypi.python.org/pypi and type "llvmlite" into the search box, clicking the "search" button leads me to the page for the Numba package. It should either lead me to the page for the *llvmlite* package or show me a list of search results. Regards Antoine.

2 1

PyPI search RPC not working as expected
by Vinay Sajip Aug. 28, 2015

Aug. 28, 2015

The PyPI XML-RPC API seems not to be working as expected. The following simple script: from pprint import pprint import sys try: from xmlrpclib import ServerProxy except ImportError: from xmlrpc.client import ServerProxy rpc_proxy = ServerProxy('https://pypi.python.org/pypi') if len(sys.argv) < 2: pkg = 'sarge' else: pkg = sys.argv[1] pprint(rpc_proxy.search({'name': pkg})) Returns an empty list when passed the command line argument "tatterdema" (it should match my project "tatterdemalion"), whereas it does return a non-empty list when passed "sarg" (matching my project "sarge"), or when passed "jobswo" (matching my project "jobsworth"). My project "ragamuffin" also fails to show up if passed to the script. Can anyone shed any light on this? Have there been any changes to how the search is supposed to work? Regards, Vinay Sajip

1 0

403 response when registering a new package in PyPI via setup.py
by Nicholas H.Tollervey Aug. 27, 2015

Aug. 27, 2015

Hi, I get a 403 when I try to register a package on PyPI under the name "training" using setup.py (and via the form too). A cursory search tells me that the name is free. However, a buddy tells me that perhaps the name isn't free - but the associated package has no releases yet. If this is the case it sounds like the presumed existing "training" module is in a bad/wrong state. Would it be possible to give PyPI a kick with a proverbial spanner so I can, in fact, register my package and/or remove the existing erroneous entry under the name "training". Put simply, what steps can I take to fix this? Thanks for your help, Nicholas.

1 0