Mailman 3 October 2014 - Distutils-SIG

devpi-server-2.1.1: replication/url fixes
by holger krekel 15 Oct '14

15 Oct '14

Hi all, the caching and private pypi server, devpi-server-2.1.1, is out and fixes some bugs, see changelog below. It is fully backward compatible, no export/import cycle required. For more info, see http://doc.devpi.net. best, holger krekel, merlinux GmbH 2.1.1 ---------------- - fix replication issue reported by a customer: if a replica lags behind a master and a file was created and then deleted meanwhile, the replica could get stuck with a FileReplicationError. We now let the master report a 410 GONE code so that the replica knows it can safely proceed because the file was deleted later anyways. - generate "systemd" configuration example when "--gen-config" is issued. Thanks Pavel Sedlak. - fix issue109: fix relative URLs in simple index pages and 404 errors on uploading toxresults and downloading files when serving under an outside URL with a sub path. Thanks to Joe Holloway for detailed infos. - drop limitation on maximum documentation size. Body size is now only controlled by frontends such as nginx. Thanks Stephan Erb. - use newer version of virtualenv for jenkins trigger. Thank brunsgaard.

1 0

setuptools script installation in Mac OS
by Robin Becker 15 Oct '14

15 Oct '14

I have a setuptools based setup.py that contains this > entry_points = dict( > console_scripts = [ > 'rml2pdf=rlextra.rml2pdf.rml2pdf:main', > 'pageCatcher=rlextra.pageCatcher.pageCatcher:scriptInterp', > 'pdfexplorer=rlextra.pageCatcher.pdfexplorer:test', > ], > gui_scripts = [ > 'diagra=rlextra.graphics.guiedit.guiedit:mainApp', > ], > ), this appears to work OK and creates all 4 scripts in Windows, Linux & Mac OS, however, when uninstalling or upgrading I get an error for Mac OS when instead of trying to remove a pageCatcher script it tries instead to remove pagecatcher. ie > > Proceed (y/n)? y > Exception: > Traceback (most recent call last): > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main > status = self.run(options, args) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/commands/uninstall.py", line 59, in run > requirement_set.uninstall(auto_confirm=options.yes) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 1024, in uninstall > req.uninstall(auto_confirm=auto_confirm) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 594, in uninstall > paths_to_remove.remove(auto_confirm) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 1781, in remove > renames(path, new_path) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/util.py", line 295, in renames > shutil.move(old, new) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 301, in move > copy2(src, real_dst) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 130, in copy2 > copyfile(src, dst) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile > with open(src, 'rb') as fsrc: > IOError: [Errno 2] No such file or directory: '/Users/rptlab/tmp/ttt/bin/pagecatcher' > > Storing debug log for failure in /Users/rptlab/.pip/pip.log this is with pip 1.5.2 & python 2.7.6 in Darwin 10.8.0 and other later Macs. Is this just a case issue? -- Robin Becker

2 1

cdecimal licensing/hosting (was: some questions about PEP470)
by Ian Cordasco 14 Oct '14

14 Oct '14

On Sun, Oct 12, 2014 at 1:44 PM, Alex Gaynor <alex.gaynor(a)gmail.com> wrote: > Stefan Krah <stefankrah <at> freenet.de> writes: > >> >> >> > (for example right now bytereef.org is down, so >> > we’d not discover any files there). >> >> Indeed. It was up reliably since 2005, down for maintenance on >> September 23rd (before ShellShock ...). Then I discovered that >> someone had put up m3-cdecimal on PyPI (presumably abusing PyPI >> as their private repo --- there are several m3-* packages now). >> >> This triggered some reflection on whether I would make a significant >> effort in the future to keep things running smoothly for an open source >> community where authors are largely viewed as expendable. > > I don't know what it means for "authors to be largely viewed as expendable", > but half the point of hosting things on PyPI is that you *don't* need to do any > work at all as an author for reliable delivery of your package. > >> >> Subsequently the downtime (again, the first one since 2005) was picked >> up for propagandistic purposes on Twitter and Reddit. > > Ok, but you seem to be doing the other side's propaganda. Every single person > I've spoken to agrees that this just underscores the need to encourage packages > to be on PyPI. > >> >> Last year I would have felt an obligation to minimize the downtime >> to an hour at most. I no longer feel any such obligations and I'll >> do it when I have time. >> > > Ok. The PyPI administrators still feel an obligation to their users, so I'll > prefer packages under their care. > >> Stefan Krah >> > > Cheers, > Alex Perhaps Stefan's referring to my tweets about the inability to reach bytereef but those weren't propaganda tweets. Those were tweets born out of utter frustration. Further, I'm rather shocked that you've decided to allow the site to remain unreachable because someone did what your license allowed them to do (redistribute the software while retaining the required information: copyright, license, etc). If you think that makes you expendable, you're half right. Users can redistribute your software, that's the nature of the license you chose to use. You're wrong because you, the author, are still very valuable to those very users who may encounter a bug in the future. I don't see how intentionally keeping your site unreachable does anything but hurt your users (unless of course you want them to redistribute it themselves or switch to Python 3.4). Does this mean that companies using devpi to keep an internal index that also have copies of cdecimal are somehow violating your rights? They're doing exactly what your license allows them to do. Or is it just that some group has decided to redistribute it directly through PyPI? I'm thoroughly confused here.

2 1

PEP 470 - Once More, with Feeling
by Donald Stufft 13 Oct '14

13 Oct '14

Alright, here's yet another go at PEP 470. See it online at www.python.org/dev/peps/pep-0470/ or reproduced in full down below. The diff between this version and the last is available at https://hg.python.org/peps/rev/2855fa903e89. Important Notes: * Continue to use a ``<meta>`` link instead of an href to prevent older installers from silently picking up insecure hosting URLs. * Reduce the overall impact by dropping the special case for PIL and instead scan all projects for URLs which add installable files and move them to the new external repository feature. * Reduce the overall impact by explicitly stating that PyPI will add the location of any external repository in the UI for people using installers which have not implemented the discovery feature. * Explicitly call out the key user experience requirements of a solution to the general problem. * Simplify the ``<meta>`` tag a bit, and also add explicit repository vs find-links types as well as include an example of the ``find-links`` type. * Allow a project to both host files on PyPI and register external repositories, these can be used for things which cannot be hosted on PyPI such as data files or Linux Wheels while still using PyPI as the repository for "regular" situations. * Mandate that the discovery mechanism must exist in a released pip prior to starting the deprecation process (with the exception of ``pypi-only`` for new projects) and call this out using an important admonition. * Explicitly call out the fact that 99.5% of the users of the deprecated features are doing so unsafely. * Explicitly reject the idea of preserving the existing links indefinentely. * Removed all examples which used the ``--extra-index-url`` feature of pip to remove the distraction of the discussion of how that currently works and in what scenarios it's safe or unsafe. Compatability: I've thought it over and gone back and forth on it to myself and others. I cannot justify an attempt to preserve backwards compatability when that backwards compatability is almost entirely unsafe to begin with. What I have done is remove the special case of PIL and essentially apply that to all projects. This should mean that all projects should have the correct metadata immediately without any need for interaction by the authors of said projects. I've also explicitly included adding the new metadata to the PyPI web UI to improve the discoverability for users of installers which don't have the discovery features. PEP: Abstract ======== This PEP proposes a mechanism for project authors to register with PyPI an external repository where their project's downloads can be located. This information can than be included as part of the simple API so that installers can use it to tell users where the item they are attempting to install is located and what they need to do to enable this additional repository. In addition to adding discovery information to make explicit multiple repositories easy to use, this PEP also deprecates and removes the implicit multiple repository support which currently functions through directly or indirectly linking off site via the simple API. Finally this PEP also proposes deprecating and removing the functionality added by PEP 438, particularly the additional rel information and the meta tag to indicate the API version. This PEP *does* not propose mandating that all authors upload their projects to PyPI in order to exist in the index nor does it propose any change to the human facing elements of PyPI. Rationale ========= Historically PyPI did not have any method of hosting files nor any method of automatically retrieving installables, it was instead focused on providing a central registry of names, to prevent naming collisions, and as a means of discovery for finding projects to use. In the course of time setuptools began to scrape these human facing pages, as well as pages linked from those pages, looking for things it could automatically download and install. Eventually this became the "Simple" API which used a similar URL structure however it eliminated any of the extraneous links and information to make the API more efficient. Additionally PyPI grew the ability for a project to upload release files directly to PyPI enabling PyPI to act as a repository in addition to an index. This gives PyPI two equally important roles that it plays in the Python ecosystem, that of index to enable easy discovery of Python projects and central repository to enable easy hosting, download, and installation of Python projects. Due to the history behind PyPI and the very organic growth it has experienced the lines between these two roles are blurry, and this blurring has caused confusion for the end users of both of these roles and this has in turn caused ire between people attempting to use PyPI in different capacities, most often when end users want to use PyPI as a repository but the author wants to use PyPI solely as an index. This confusion comes down to end users of projects not realizing if a project is hosted on PyPI or if it relies on an external service. This often manifests itself when the external service is down but PyPI is not. People will see that PyPI works, and other projects works, but this one specific one does not. They often times do not realize who they need to contact in order to get this fixed or what their remediation steps are. By moving to using explicit multiple repositories we can make the lines between these two roles much more explicit and remove the "hidden" surprises caused by the current implementation of handling people who do not want to use PyPI as a repository. However simply moving to explicit multiple repositories is a regression in discoverability, and for that reason this PEP adds an extension to the current simple API which will enable easy discovery of the specific repository that a project can be found in. PEP 438 attempted to solve this issue by allowing projects to explicitly declare if they were using the repository features or not, and if they were not, it had the installers classify the links it found as either "internal", "verifiable external" or "unverifiable external". PEP 438 was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with the final transition implemented in pip 1.5 (released on Jan 2, 2014). PEP 438 was successful in bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed ups for a lot of people, however it did so by introducing a new point of confusion and pain for both the end users and the authors. Key User Experience Expectations -------------------------------- #. Easily allow external hosting to "just work" when appropriately configured at the system, user or virtual environment level. #. Easily allow package authors to tell PyPI "my releases are hosted <here>" and have that advertised in such a way that tools can clearly communicate it to users, without silently introducing unexpected dependencies on third party services. #. Eliminate any and all references to the confusing "verifiable external" and "unverifiable external" distinction from the user experience (both when installing and when releasing packages). #. The repository aspects of PyPI should become *just* the default package hosting location (i.e. the only one that is treated as opt-out rather than opt-in by most client tools in their default configuration). Aside from that aspect, hosting on PyPI should not otherwise provide an enhanced user experience over hosting your own package repository. #. Do all of the above while providing default behaviour that is secure against most attackers below the nation state adversary level. Why Additional Repositories? ---------------------------- The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisfy the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familiar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. This could also be used for projects wishing to host files which cannot be uploaded to PyPI, such as multi-gigabyte data files or, currently at least, Linux Wheels. Why Not PEP 438 or Similar? --------------------------- While the additional search location support has existed in pip and setuptools for quite some time support for PEP 438 has only existed in pip since the 1.4 version, and still has yet to be implemented in setuptools. The design of PEP 438 did mean that users still benefited for projects which did not require external files even with older installers, however for projects which *did* require external files, users are still silently being given either potentially unreliable or, even worse, unsafe files to download. This system is also unique to Python as it arises out of the history of PyPI, this means that it is almost certain that this concept will be foreign to most, if not all users, until they encounter it while attempting to use the Python toolchain. Additionally, the classification system proposed by PEP 438 has, in practice, turned out to be extremely confusing to end users, so much so that it is a position of this PEP that the situation as it stands is completely untenable. The common pattern for a user with this system is to attempt to install a project possibly get an error message (or maybe not if the project ever uploaded something to PyPI but later switched without removing old files), see that the error message suggests ``--allow-external``, they reissue the command adding that flag most likely getting another error message, see that this time the error message suggests also adding ``--allow-unverified``, and again issue the command a third time, this time finally getting the thing they wish to install. This UX failure exists for several reasons. #. If pip can locate files at all for a project on the Simple API it will simply use that instead of attempting to locate more. This is generally the right thing to do as attempting to locate more would erase a large part of the benefit of PEP 438. This means that if a project *ever* uploaded a file that matches what the user has requested for install that will be used regardless of how old it is. #. PEP 438 makes an implicit assumption that most projects would either upload themselves to PyPI or would update themselves to directly linking to release files. While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so. More concerning however, is the fact that very few projects have opted to directly and safely link to files and instead they still simply link to pages which must be scraped in order to find the actual files, thus rendering the safe variant (``--allow-external``) largely useless. #. Even if an author wishes to directly link to their files, doing so safely is non-obvious. It requires the inclusion of a MD5 hash (for historical reasons) in the hash of the URL. If they do not include this then their files will be considered "unverified". #. PEP 438 takes a security centric view and disallows any form of a global opt in for unverified projects. While this is generally a good thing, it creates extremely verbose and repetitive command invocations such as:: $ pip install --allow-external myproject --allow-unverified myproject myproject $ pip install --allow-all-external --allow-unverified myproject myproject Multiple Repository/Index Support ================================= Installers SHOULD implement or continue to offer, the ability to point the installer at multiple URL locations. The exact mechanisms for a user to indicate they wish to use an additional location is left up to each individual implementation. Additionally the mechanism discovering an installation candidate when multiple repositories are being used is also up to each individual implementation, however once configured an implementation should not discourage, warn, or otherwise cast a negative light upon the use of a repository simply because it is not the default repository. Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Installers SHOULD also implement some mechanism for removing or otherwise disabling use of the default repository. The exact specifics of how that is achieved is up to each individual implementation. Installers SHOULD also implement some mechanism for whitelisting and blacklisting which projects a user wishes to install from a particular repository. The exact specifics of how that is achieved is up to each individual implementation. External Index Discovery ======================== One of the problems with using an additional index is one of discovery. Users will not generally be aware that an additional index is required at all much less where that index can be found. Projects can attempt to convey this information using their description on the PyPI page however that excludes people who discover their project organically through ``pip search``. To support projects that wish to externally host their files and to enable users to easily discover what additional indexes are required, PyPI will gain the ability for projects to register external index URLs along with an associated comment for each. These URLs will be made available on the simple page however they will not be linked or provided in a form that older installers will automatically search them. This ability will take the form of a ``<meta>`` tag. The name of this tag must be set to ``repository`` or ``find-link`` and the content will be a link to the location of the repository. An optional data-description attribute will convey any comments or description that the author has provided. An example would look something like:: <meta name="repository" content="https://index.example.com/" data-description="Primary Repository"> <meta name="repository" content="https://index.example.com/Ubuntu-14.04/" data-description="Wheels built for Ubuntu 14.04"> <meta name="find-link" content="https://links.example.com/find-links/" data-description="A flat index for find links"> When an installer fetches the simple page for a project, if it finds this additional meta-data then it should use this data to tell the user how to add one or more of the additional URLs to search in. This message should include any comments that the project has included to enable them to communicate to the user and provide hints as to which URL they might want (e.g. if some are only useful or compatible with certain platforms or situations). When the installer has implemented the auto discovery mechanisms they should also deprecate any of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal at the end of the deprecation period proposed by the PEP. In addition to the API for programtic access to the registered external repositories, PyPI will also prevent these URLs in the UI so that users with an installer that does not implement the discovery mechanism can still easily discover what repository the project is using to host itself. This feature **MUST** be added to PyPI and be contained in a released version of pip prior to starting the deprecation and removal process for the implicit offsite hosting functionality. Deprecation and Removal of Link Spidering ========================================= .. important:: The deprecation specified in this section **MUST** not start to until after the discovery mechanisms have been implemented and released in pip. The only exception to this is the addition of the ``pypi-only`` mode and defaulting new projects to it without abilility to switch to a different mode. A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP 438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will be defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. ``pypi-only`` projects will still be able to register external index URLs as described above - the "pypi-only" refers only to the download links that are published directly on PyPI. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set to ``pypi-only``. After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. This email **MUST** include instructions for converting their projects to be hosted on PyPI and **MUST** include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email **MUST** also include instructions for setting up their own index page and registering that with PyPI, including the fact that they can use pythonhosted.org as a host for an index page without requiring them to host any additional infrastructure or purchase a TLS certificate. This email must also contain a link to the Terms of Service for PyPI as many users may have signed up a long time ago and may not recall what those terms are. Finally this email must also contain a list of the links registered with PyPI where we were able to detect an installable file was located. Five months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of six. Finally a month later all projects will be switched to the ``pypi-only`` mode and PyPI will be modified to remove the externally linked files functionality, when switching these projects to the ``pypi-only`` mode we will move any links which are able to be used for discovering other projects automatically to as an external repository. Summary of Changes ================== Repository side --------------- #. Implement simple API changes to allow the addition of an external repository. #. *(Optional, Mandatory on PyPI)* Deprecate and remove the hosting modes as defined by PEP 438. #. *(Optional, Mandatory on PyPI)* Restrict simple API to only list the files that are contained within the repository and the external repository metadata. Client side ----------- #. Implement multiple repository support. #. Implement some mechanism for removing/disabling the default repository. #. Implement the discovery mechanism. #. *(Optional)* Deprecate / Remove PEP 438 Impact ====== The large impact of this PEP will be that for users of older installation clients they will not get a discovery mechanism built into the install command. This will require them to browse to the PyPI web UI and discover the repository there. Since any URLs required to instal a project will be automatically migrated to the new format, the biggest change to users will be requiring a new option to install these projects. Looking at the numbers the actual impact should be quite low, with it affecting just 3.8% of projects which host any files only externally or 2.2% which have their latest version hosted only externally. 6674 unique IP addresses have accessed the Simple API for these 3.8% of projects in a single day (2014-09-30). Of those, 99.5% of them installed something which could not be verified, and thus they were open to a Remote Code Execution via a Man-In-The-Middle attack, while 7.9% installed something which could be verified and only 0.4% only installed things which could be verified. This means that 99.5% users of these features, both new and old, are doing something unsafe, and for anything using an older copy of pip or using setuptools at all they are silently unsafe. Projects Which Rely on Externally Hosted files ---------------------------------------------- This is determined by crawling the simple index and looking for installable files using a similar detection method as pip and setuptools use. The "latest" version is determined using ``pkg_resources.parse_version`` sort order and it is used to show whether or not the latest version is hosted externally or only old versions are. ============ ======= ================ =================== ======= \ PyPI External (old) External (latest) Total ============ ======= ================ =================== ======= **Safe** 43313 16 39 43368 **Unsafe** 0 756 1092 1848 **Total** 43313 772 1131 45216 ============ ======= ================ =================== ======= Top Externally Hosted Projects by Requests ------------------------------------------ This is determined by looking at the number of requests the ``/simple/<project>/`` page had gotten in a single day. The total number of requests during that day was 10,623,831. ============================== ======== Project Requests ============================== ======== PIL 63869 Pygame 2681 mysql-connector-python 1562 pyodbc 724 elementtree 635 salesforce-python-toolkit 316 wxPython 295 PyXML 251 RBTools 235 python-graph-core 123 cElementTree 121 ============================== ======== Top Externally Hosted Projects by Unique IPs -------------------------------------------- This is determined by looking at the IP addresses of requests the ``/simple/<project>/`` page had gotten in a single day. The total number of unique IP addresses during that day was 124,604. ============================== ========== Project Unique IPs ============================== ========== PIL 4553 mysql-connector-python 462 Pygame 202 pyodbc 181 elementtree 166 wxPython 126 RBTools 114 PyXML 87 salesforce-python-toolkit 76 pyDes 76 ============================== ========== Rejected Proposals ================== Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. * Continue on the suggested path of PEP 438 and remove the option to unsafely host externally but continue to allow the option to safely host externally. These proposals are rejected because: * The classification system introduced in PEP 438 in an entirely unique concept to PyPI which is not generically applicable even in the context of Python packaging. Adding additional concepts comes at a cost. * The classification system itself is non-obvious to explain and to pre-determine what classification of link a project will require entails inspecting the project's ``/simple/<project>/`` page, and possibly any URLs linked from that page. * The ability to host externally while still being linked for automatic discovery is mostly a historic relic which causes a fair amount of pain and complexity for little reward. * The installer's ability to optimize or clean up the user interface is limited due to the nature of the implicit link scraping which would need to be done. This extends to the ``--allow-*`` options as well as the inability to determine if a link is expected to fail or not. * The mechanism paints a very broad brush when enabling an option, while PEP 438 attempts to limit this with per package options. However a project that has existed for an extended period of time may often times have several different URLs listed in their simple index. It is not unusual for at least one of these to no longer be under control of the project. While an unregistered domain will sit there relatively harmless most of the time, pip will continue to attempt to install from it on every discovery phase. This means that an attacker simply needs to look at projects which rely on unsafe external URLs and register expired domains to attack users. Implement this PEP, but Do Not Remove the Existing Links -------------------------------------------------------- This is essentially the backwards compatible version of this PEP. It attempts to allow people using older clients, or clients which do not implement this PEP to continue on as if nothing had changed. This proposal is rejected because the vast bulk of those scenarios are unsafe uses of the deprecated features. It is the opinion of this PEP that silently allowing unsafe actions to take place on behalf of end users is simply not an acceptable solution. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 1

Having a "less complete" configuration for a project
by Paul Moore 12 Oct '14

12 Oct '14

I'm building a project that allows users to create files and directory structures from static data. The core of the project will be a command line API that includes a number of features that depend on 3rd party libraries. But I can see the core of the project being useful for smaller tasks, for example generating test data. So what I'd like is to have the "standard" installation "pip install myproj" build the full application, but have a way to specify (for example in a tox config) that only the core library, without the larger dependencies, be installed. Doing this the other way round is easy using extras: pip install myproj -> installs myproj and core dependencies pip install myproj[cmdline] -> installs the above, plus jinja2, click, ... But the problem with this approach is that "normal users" need to know to specify the extra. Is it possible to switch this round somehow, so that I have an "extra" that *removes* some of the dependencies? (I could have 2 projects, a core one and a cmdline one that depends on the first and adds the command-line API. I don't really like that option, though, not least because it means maintaining 2 projects rather than one, so I'm exploring whether there are other ways of achieving the objective). Any thoughts on how I could do this? Paul

2 4

PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting
by Donald Stufft 08 Oct '14

08 Oct '14

Here’s round 4 of PEP 470, I believe I’ve addressed the comments from the previous thread. I've also tried to clarify the text as well as the motivations better. You can view this online at: https://www.python.org/dev/peps/pep-0470/ --------------------- PEP: 470 Title: Using Multi Repository Support for External to PyPI Package File Hosting Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft <donald(a)stufft.io>, BDFL-Delegate: Richard Jones <richard(a)python.org> Discussions-To: distutils-sig(a)python.org Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2014 Post-History: 14-May-2014, 05-Jun-2014, 03-Oct-2014 Replaces: 438 Abstract ======== This PEP proposes a mechanism for project authors to register with PyPI an external repository where their project's downloads can be located. This information can than be included as part of the simple API so that installers can use it to tell users where the item they are attempting to install is located and what they need to do to enable this additional repository. In addition to adding discovery information to make explicit multiple repositories easy to use, this PEP also deprecates and removes the implicit multiple repository support which currently functions through directly or indirectly linking offsite via the simple API. Finally this PEP also proposes deprecating and removing the functionality added by PEP 438, particularly the additional rel information and the meta tag to indicate the API version. This PEP *does* not propose mandating that all authors upload their projects to PyPI in order to exist in the index nor does it propose any change to the human facing elements of PyPI. Rationale ========= Historically PyPI did not have any method of hosting files nor any method of automatically retrieving installables, it was instead focused on providing a central registry of names, to prevent naming collisions, and as a means of discovery for finding projects to use. In the course of time setuptools began to scrape these human facing pages, as well as pages linked from those pages, looking for things it could automatically download and install. Eventually this became the "Simple" API which used a similar URL structure however it eliminated any of the extraneous links and information to make the API more efficient. Additionally PyPI grew the ability for a project to upload release files directly to PyPI enabling PyPI to act as a repository in addition to an index. This gives PyPI two equally important roles that it plays in the Python ecosystem, that of index to enable easy discovery of Python projects and central repository to enable easy hosting, download, and installation of Python projects. Due to the history behind PyPI and the very organic growth it has experienced the lines between these two roles are blurry, and this blurriness has caused confusion for the end users of both of these roles and this has in turn caused ire between people attempting to use PyPI in different capacities, most often when end users want to use PyPI as a repository but the author wants to use PyPI soley as an index. By moving to using explict multiple repositories we can make the lines between these two roles much more explicit and remove the "hidden" surprises caused by the current implementation of handling people who do not want to use PyPI as a repository. However simply moving to explicit multiple repositories is a regression in discoverablity, and for that reason this PEP adds an extension to the current simple API which will enable easy discovery of the specific repository that a project can be found in. PEP 438 attempted to solve this issue by allowing projects to explicitly declare if they were using the repository features or not, and if they were not, it had the installers classify the links it found as either "internal", "verifiable external" or "unverifiable external". PEP 438 was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with the final transition implemented in pip 1.5 (released on Jan 2, 2014). PEP 438 was successful in bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed ups for a lot of people, however it did so by introducing a new point of confusion and pain for both the end users and the authors. Why Additional Repositories? ---------------------------- The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisify the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. Setting up an external repository is very simple, it can be achieved with nothing more than a filesystem, some files to host, and any web server capable of serving files and generating an automated index of directories (commonly called "autoindex"). This can be as simple as: :: $ mkdir -p /var/www/index.example.com/ $ mkdir -p /var/www/index.example.com/myproject/ $ mv ~/myproject-1.0.tar.gz /var/www/index.example.com/myproject/ $ twistd -n web --path /var/www/index.example.com/ Using this additional location within pip is also simple and can be included on a per invocation, per shell, or per user basis. The pip 6.0 will also include the ability to configure this on a per virtual environment or per machine basis as well. This can be as simple as: :: $ # As a CLI argument $ pip install --extra-index-url https://index.example.com/ myproject $ # As an environment variable $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject $ # With a configuration file $ echo "[global]\nextra-index-url = https://pypi.example.com/" > ~/.pip/pip.conf $ pip install myproject Why Not PEP 438 or Similar? --------------------------- While the additional search location support has existed in pip and setuptools for quite some time support for PEP 438 has only existed in pip since the 1.4 version, and still has yet to be implemented in setuptools. The design of PEP 438 did mean that users still benefited for projects which did not require external files even with older installers, however for projects which *did* require external files, users are still silently being given either potentionally unreliable or, even worse, unsafe files to download. This system is also unique to Python as it arises out of the history of PyPI, this means that it is almost certain that this concept will be foreign to most, if not all users, until they encounter it while attempting to use the Python toolchain. Additionally, the classification system proposed by PEP 438 has, in practice, turned out to be extremely confusing to end users, so much so that it is a position of this PEP that the situation as it stands is completely untenable. The common pattern for a user with this system is to attempt to install a project possibly get an error message (or maybe not if the project ever uploaded something to PyPI but later switched without removing old files), see that the error message suggests ``--allow-external``, they reissue the command adding that flag most likely getting another error message, see that this time the error message suggests also adding ``--allow-unverified``, and again issue the command a third time, this time finally getting the thing they wish to install. This UX failure exists for several reasons. 1. If pip can locate files at all for a project on the Simple API it will simply use that instead of attempting to locate more. This is generally the right thing to do as attempting to locate more would erase a large part of the benefit of PEP 438. This means that if a project *ever* uploaded a file that matches what the user has requested for install that will be used regardless of how old it is. 2. PEP 438 makes an implicit assumption that most projects would either upload themselves to PyPI or would update themselves to directly linking to release files. While a large number of projects *did* ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so. More concerning however, is the fact that very few projects have opted to directly and safely link to files and instead they still simply link to pages which must be scraped in order to find the actual files, thus rendering the safe variant (``--allow-external``) largely useless. 3. Even if an author wishes to directly link to their files, doing so safely is non-obvious. It requires the inclusion of a MD5 hash (for historical reasons) in the hash of the URL. If they do not include this then their files will be considered "unverified". 4. PEP 438 takes a security centric view and disallows any form of a global opt in for unverified projects. While this is generally a good thing, it creates extremely verbose and repetive command invocations such as: :: $ pip install --allow-external myproject --allow-unverified myproject myproject $ pip install --allow-all-external --allow-unverified myproject myproject Multiple Repository/Index Support ================================= Installers SHOULD implement or continue to offer, the ability to point the installer at multiple URL locations. The exact mechanisms for a user to indicate they wish to use an additional location is left up to each indidivdual implementation. Additionally the mechanism discovering an installation candidate when multiple repositories are being used is also up to each individual implementation, however once configured an implementation should not discourage, warn, or otherwise cast a negative light upon the use of a repository simply because it is not the default repository. Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Installers SHOULD also implement some mechanism for removing or otherwise disabling use of the default repository. The exact specifics of how that is achieved is up to each indidivdual implementation. End users wishing to limit what files they pull from which repository can simply use `devpi <http://doc.devpi.net/latest/>`_ to whitelist projects from PyPI or another repository. External Index Discovery ======================== One of the problems with using an additional index is one of discovery. Users will not generally be aware that an additional index is required at all much less where that index can be found. Projects can attempt to convey this information using their description on the PyPI page however that excludes people who discover their project organically through ``pip search``. To support projects that wish to externally host their files and to enable users to easily discover what additional indexes are required, PyPI will gain the ability for projects to register external index URLs along with an associated comment for each. These URLs will be made available on the simple page however they will not be linked or provided in a form that older installers will automatically search them. This ability will take the form of a ``<meta>`` tag. The name of this tag must be set to ``external-repository`` and the content will be a link to the location of the external repository. An optional data-description attribute will convey any comments or description that the author has provided. An example would look something like: :: <meta name="external-repository" content="https://index.example.com/" data-description="Primary Repository"> <meta name="external-repository" content="https://index.example.com/Ubuntu-14.04/" data-description="Wheels built for Ubuntu 14.04"> When an external repository is added to a project, new uploads will no longer be permitted to that project. However any existing files will simply be hidden from the simple API and the web interface until all of the external repositories are removed, in which case they will be visible again. PyPI MUST warn authors if adding an external repository will hide files and that warning must persist on any of the project management pages for that particular project. When an installer fetches the simple page for a project, if it finds this additional meta-data and it cannot find any files for that project in it's configured URLs then it should use this data to tell the user how to add one or more of the additional URLs to search in. This message should include any comments that the project has included to enable them to communicate to the user and provide hints as to which URL they might want (e.g. if some are only useful or compatible with certain platforms or situations). When the installer has implemented the auto discovery mechanisms they should also deprecate any of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal at the end of the deprecation period proposed by the PEP. This feature *must* be added to PyPI prior to starting the deprecation and removal process for the implicit offsite hosting functionality. Deprecation and Removal of Link Spidering ========================================= A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP 438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will be defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. ``pypi-only`` projects will still be able to register external index URLs as described above - the "pypi-only" refers only to the download links that are published directly on PyPI. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set to ``pypi-only``. After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. This email *must* include instructions for converting their projects to be hosted on PyPI and *must* include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email *must also* include instructions for setting up their own index page and registering that with PyPI, including the fact that they can use pythonhosted.org as a host for an index page without requiring them to host any additional infrastructure or purchase a TLS certificate. This email must also contain a link to the Terms of Service for PyPI as many users may have signed up a long time ago and may not recall what those terms are. Five months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of six. Finally a month later all projects will be switched to the ``pypi-only`` mode and PyPI will be modified to remove the externally linked files functionality. At this point in time any installers should finally remove any of the deprecated PEP 438 functionality such as ``--allow-external`` and ``--allow-unverified`` in pip. Impact ====== The largest impact of this is going to be projects where the maintainers are no longer maintaining the project, for one reason or another. For these projects it's unlikely that a maintainer will arrive to set the external index metadata which would allow the auto discovery mechanism to find it. Looking at the numbers factoring out PIL (which has been special cased below) the actual impact should be quite low, with it affecting just 3.8% of projects which host any files only externally or 2.2% which have their latest version hosted only externally. 6674 unique IP addresses have accessed the Simple API for these 3.8% of projects in a single day (2014-09-30). Of those, 99.5% of them installed something which could not be verified, and thus they were open to a Remote Code Execution via a Man-In-The-Middle attack, while 7.9% installed something which could be verified and only 0.4% only installed things which could be verified. Projects Which Rely on Externally Hosted files ---------------------------------------------- This is determined by crawling the simple index and looking for installable files using a similar detection method as pip and setuptools use. The "latest" version is determined using ``pkg_resources.parse_version`` sort order and it is used to show whether or not the latest version is hosted externally or only old versions are. ============ ======= ================ =================== ======= \ PyPI External (old) External (latest) Total ============ ======= ================ =================== ======= **Safe** 43313 16 39 43368 **Unsafe** 0 756 1092 1848 **Total** 43313 772 1131 45216 ============ ======= ================ =================== ======= Top Externally Hosted Projects by Requests ------------------------------------------ This is determined by looking at the number of requests the ``/simple/<project>/`` page had gotten in a single day. The total number of requests during that day was 10,623,831. ============================== ======== Project Requests ============================== ======== PIL 63869 Pygame 2681 mysql-connector-python 1562 pyodbc 724 elementtree 635 salesforce-python-toolkit 316 wxPython 295 PyXML 251 RBTools 235 python-graph-core 123 cElementTree 121 ============================== ======== Top Externally Hosted Projects by Unique IPs -------------------------------------------- This is determined by looking at the IP addresses of requests the ``/simple/<project>/`` page had gotten in a single day. The total number of unique IP addresses during that day was 124,604. ============================== ========== Project Unique IPs ============================== ========== PIL 4553 mysql-connector-python 462 Pygame 202 pyodbc 181 elementtree 166 wxPython 126 RBTools 114 PyXML 87 salesforce-python-toolkit 76 pyDes 76 ============================== ========== PIL --- It's obvious from the numbers above that the vast bulk of the impact come from the PIL project. On 2014-05-17 an email was sent to the contact for PIL inquiring whether or not they would be willing to upload to PyPI. A response has not been received as of yet (2014-10-03) nor has any change in the hosting happened. Due to the popularity of PIL this PEP also proposes that during the deprecation period that PyPI Administrators will set the PIL download URL as the external index for that project. Allowing the users of PIL to take advantage of the auto discovery mechanisms although the project has seemingly become unmaintained. Rejected Proposals ================== Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. * Continue on the suggested path of PEP 438 and remove the option to unsafely host externally but continue to allow the option to safely host externally. These proposals are rejected because: * The classification system introduced in PEP 438 in an entirely unique concept to PyPI which is not generically applicable even in the context of Python packaging. Adding additional concepts comes at a cost. * The classification system itself is non-obvious to explain and to pre-determine what classification of link a project will require entails inspecting the project's ``/simple/<project>/`` page, and possibly any URLs linked from that page. * The ability to host externally while still being linked for automatic discovery is mostly a historic relic which causes a fair amount of pain and complexity for little reward. * The installer's ability to optimize or clean up the user interface is limited due to the nature of the implicit link scraping which would need to be done. This extends to the ``--allow-*`` options as well as the inability to determine if a link is expected to fail or not. * The mechanism paints a very broad brush when enabling an option, while PEP 438 attempts to limit this with per package options. However a project that has existed for an extended period of time may often times have several different URLs listed in their simple index. It is not unsusual for at least one of these to no longer be under control of the project. While an unregistered domain will sit there relatively harmless most of the time, pip will continue to attempt to install from it on every discovery phase. This means that an attacker simply needs to look at projects which rely on unsafe external URLs and register expired domains to attack users. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

9 63

Create formal process for claiming 'abandoned' packages
by Daniel Greenfeld 08 Oct '14

08 Oct '14

In order to claim a package as being abandoned it should undergo a formal process that includes: * Placement on a PUBLIC list of packages under review for a grace period to be determined by this discussion * Formal attempts via email and social media (twitter, github, et al) to contact the maintainer. * Investigation of the claimant for the rights to the package. The parties attempting to claim a package may not be the best representatives of the community behind that package, or the Python community in general. Why? * Non-reply does not equal consent. * Access to a commonly (or uncommonly) used package poses security and reliability issues. Why: Scenario 1: I could claim ownership of the redis package, providing a certain-to-fail email for the maintainers of PyPI to investigate? Right now the process leads me to think I would succeed in gaining access. If successful, I would gain complete access to a package used by hundreds of projects for persistence storage. Scenario 2: I could claim ownership of the redis package, while Andy McCurdy (maintainer) was on vacation for two weeks, or sabbatical for six weeks. Again, I would gain access because under the current system non-reply equals consent. Reference: In ticket #407 (https://sourceforge.net/p/pypi/support-requests/407/) someone who does not appear to be vetted managed to gain control of the (arguably) abandoned but still extremely popular django-registration on PyPI. They run one of several HUNDRED forks of django-registration, one that is arguably not the most commonly used. My concern is that as django-registration is the leading package for handling system registration for Python's most popular web framework, handing it over without a full investigation of not just the current maintainer but also the candidate maintainer is risky. Regards, Daniel Greenfeld pydanny(a)gmail.com

15 28

setup.py develop issues
by Cody Piersall 07 Oct '14

07 Oct '14

I am having issues with using setup.py develop on my computer at work. My understanding of the command is that it should make my project import-able without really installing it, so that I can continue to change my files without needing to reinstall the project. Here is a minimum example of the problem. I create a new project (named project) with only a single package, and create the minimum setup.py file that I thought would work. mkdir project cd project mkdir project echo. > project\__init__.py echo.from setuptools import setup>setup.py echo.setup(>>setup.py echo. name='project',>>setup.py echo. packages=['project'],>>setup.py echo.^)>>setup.py cd .. After running these commands (as a batch script), I did cd project python setup.py develop but the project was not importable. I can import it when I run `python setup.py install` though. Here is the output of running python setup.py develop: C:\Users\CWP\Development\project>python setup.py develop running develop running egg_info creating project.egg-info writing top-level names to project.egg-info\top_level.txt writing project.egg-info\PKG-INFO writing dependency_links to project.egg-info\dependency_links.txt writing manifest file 'project.egg-info\SOURCES.txt' reading manifest file 'project.egg-info\SOURCES.txt' writing manifest file 'project.egg-info\SOURCES.txt' running build_ext Creating c:\python34\lib\site-packages\project.egg-link (link to .) Adding project 0.0.0 to easy-install.pth file Installed c:\users\cwp\development\project Processing dependencies for project==0.0.0 Finished processing dependencies for project==0.0.0 I'm using Python 3.4.0 and setuptools 6.0.2. I'm not sure if I'm using setuptools wrong, if my understanding of setuptools is wrong, or if this is a bug in setuptools under Windows. Cody

2 2

pypi simple page for setuptools returning 11034 links
by Joost Molenaar 06 Oct '14

06 Oct '14

Hi all, When I run `pip list --outdated` on my raspberry pi, pip seems to hang on setuptools. Investigating a bit, I found that currently, there are 11034 links being returned [1], weighing 1379 KiB. Many of these are links to issue pages on github and bitbucket, also, many of these links are duplicated 20-25 times. Is this normal/expected? Is it a bug, if so, should I file it somewhere? Regards, Joost Molenaar [1] https://pypi.python.org/simple/setuptools/

4 4

advice re: packaging tasks
by Chris Jerdonek 03 Oct '14

03 Oct '14

Hi, I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack? 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? Thanks, --Chris

4 6