Mailman 3 May 2016 - Distutils-SIG

PyPi upload fails with TypeError
by Luís de Sousa 02 Jun '16

02 Jun '16

Dear all, I am trying to upload a new package to PyPi with this command: $ python3 setup.py sdist upload The process seems to go on well until the running upload bit, at which point I get this exception: Traceback (most recent call last): File "setup.py", line 33, in <module> "Topic :: Scientific/Engineering :: GIS", File "/usr/lib/python3.4/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/distutils/command/upload.py", line 65, in run self.upload_file(command, pyversion, filename) File "/usr/lib/python3.4/distutils/command/upload.py", line 139, in upload_file user_pass = (self.username + ":" + self.password).encode('ascii') TypeError: Can't convert 'NoneType' object to str implicitly I verified that Topic :: Scientific/Engineering :: GIS is a valid topic. What else am I missing? Thank you, Luís Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland.

5 13

PIL install fails: that was to be expected?
by Reinout van Rees 31 May '16

31 May '16

Hi, A colleague is trying to run the buildout for an *old* project that has an *old* dependency on "PIL". So not the "Pillow" replacement, but "PIL". Install fails, becauce https://pypi.python.org/simple/pil/ is empty. What I suspect is that this is because of the everything-should-be-hosted-on-pypi-itself change? Yes, it should be Pillow and yes I'm in complete agreement that everything should be on pypi and yes my colleague can fix this with a quick branch. But I just wanted to make sure that nothing major is broken right now. (And to make this potential problem google-able :-) ) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout(a)vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"

3 3

PEP for specifying build dependencies
by Brett Cannon 26 May '16

26 May '16

Donald, Nathaniel, and I have finished our proposed PEP for specifying a projects' build dependencies. The PEP is being kept at https://github.com/brettcannon/build-deps-pep, so if you find spelling mistakes and grammatical errors please feel free to send a PR to fix them. The only open issue in the PEP at the moment is the bikeshedding topic of what to name the sub-section containing the requirements: `[package.build]` or `[package.build-system]` (we couldn't reach consensus among the three of us on this). Otherwise the three of us are rather happy with the way the PEP has turned out and look forward to this being the first step towards allowing projects to customize their build process better! ----- PEP: NNN Title: Specifying build dependencies for Python Software Packages Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org>, Nathaniel Smith <njs(a)pobox.com>, Donald Stufft <donald(a)stufft.io> BDFL-Delegate: Nick Coghlan Discussions-To: distutils-sig <distutils-sig at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: NN-Mmm-2016 Post-History: NN-Mmm-2016 Abstract ======== This PEP specifies how Python software packages should specify their build dependencies (i.e. what dependencies are required to go from source checkout to built wheel). As part of this specification, a new configuration file is introduced for software packages to use to specify their build dependencies (with the expectation that the same configuration file will be used for future configuration details). Rationale ========= When Python first developed its tooling for building distributions of software for projects, distutils [#distutils]_ was the chosen solution. As time went on, setuptools [#setuptools]_ gained popularity to add some features on top of distutils. Both used the concept of a ``setup.py`` file that project maintainers executed to build distributions of their software (as well as users to install said distribution). Using an executable file to specify build requirements under distutils isn't an issue as distutils is part of Python's standard library. Having the build tool as part of Python means that a ``setup.py`` has no external dependency that a project maintainer needs to worry about to build a distribution of their project. There was no need to specify any dependency information as the only dependency is Python. But when a project chooses to use setuptools, the use of an executable file like ``setup.py`` becomes an issue. You can't execute a ``setup.py`` file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the ``setup.py`` file where that information is stored. It's a catch-22 of a file not being runnable without knowing its own contents which can't be known programmatically unless you run the file. Setuptools tried to solve this with a ``setup_requires`` argument to its ``setup()`` function [#setup_args]_. This solution has a number of issues, such as: * No tooling (besides setuptools itself) can access this information without executing the ``setup.py``, but ``setup.py`` can't be executed without having these items installed. * While setuptools itself will install anything listed in this, they won't be installed until *during* the execution of the ``setup()`` function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the ``setup()`` function. * This cannot include ``setuptools`` itself nor can it include a replacement to ``setuptools``, which means that projects such as ``numpy.distutils`` are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one. * The items listed in ``setup_requires`` get implicily installed whenever you execute the ``setup.py`` but one of the common ways that the ``setup.py`` is executed is via another tool, such as ``pip``, who is already managing dependencies. This means that a command like ``pip install spam`` might end up having both pip and setuptools downloading and installing packages and end users needing to configure *both* tools (and for ``setuptools`` without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently. This has cumulated in a situation where use of ``setup_requires`` is rare, where projects tend to either simply copy and paste snippets between ``setup.py`` files or they eschew it all together in favor of simply documenting elsewhere what they expect the user to have manually installed prior to attempting to build or install their project. All of this has led pip [#pip]_ to simply assume that setuptools is necessary when executing a ``setup.py`` file. The problem with this, though, is it doesn't scale if another project began to gain traction in the commnity as setuptools has. It also prevents other projects from gaining traction due to the friction required to use it with a project when pip can't infer the fact that something other than setuptools is required. This PEP attempts to rectify the situation by specifying a way to list the build dependencies of a project in a declarative fashion in a specific file. This allows a project to list what build dependencies it has to go from e.g. source checkout to wheel, while not falling into the catch-22 trap that a ``setup.py`` has where tooling can't infer what a project needs to build itself. Implementing this PEP will allow projects to specify what they depend on upfront so that tools like pip can make sure that they are installed in order to build the project (the actual driving of the build process is not within the scope of this PEP). Specification ============= The build dependencies will be stored in a file named ``pyproject.toml`` that is written in the TOML format [#toml]_. This format was chosen as it is human-usable (unlike JSON [#json]_), it is flexible enough (unlike configparser [#configparser]_), stems from a standard (also unlike configparser [#configparser]_), and it is not overly complex (unlike YAML [#yaml]_). The TOML format is already in use by the Rust community as part of their Cargo package manager [#cargo]_ and in private email stated they have been quite happy with their choice of TOML. A more thorough discussion as to why various alternatives were not chosen can be read in the `Other file formats`_ section. A top-level ``[package]`` table will represent details specific to the package that the project contains. A ``semantics-version`` key within the ``[package]`` table will represent the semantic version that the ``[package]`` table represents. It will always be set to an integer and will default to a value of ``1`` if unspecified. The version will only be updated when the semantics or absence of a key or sub-table in the ``[package]`` table cannot be interpreted in a backwards-compatible fashion (e.g. the version does not need to change if a new table is added to the semantics of the file, but if a pre-existing field changes its meaning or the behavior when a field is absent changes then the semantic version will need to change). Changes to the meaning of the versions are expected to occur through a PEP. Projects are expected to check the value of the ``semantics-version`` field in their code appropriately. The expectation is tools will do something along the lines of:: if data["package"].get("semantics-version", 1) != 1: raise Exception # Whatever exception is appropriate for the tool. There will be a ``[package.build-system]`` sub-table in the configuration file to store build-related data (although the exact name of the sub-table is an `open issue <#name-of-the-build-related-sub-table>`__ as ``[package.build]`` is another possibility). Initially only one key of the table will be valid: ``requires``. That key will have a value of a list of strings representing the PEP 508 dependencies required to build the project as a wheel [#wheel]_ (currently that means what dependencies are required to execute a ``setup.py`` file to generate a wheel). For the vast majority of Python projects that rely upon setuptools, the ``pyproject.toml`` file will be:: [package.build-system] requires = ['setuptools', 'wheel'] # PEP 508 specifications. Or, the equivalent but more verbose:: [package] semantics-version = 1 # Optional; defaults to 1. # Indentation is optional in TOML and has no semantic meaning. [package.build-system] requires = ['setuptools', 'wheel'] # PEP 508 specifications. Because the use of setuptools and wheel are so expansive in the community at the moment, build tools are expected to use the example configuration file above as their default semantics when a ``pyproject.toml`` file is not present. All other top-level keys and tables are reserved for future use by other PEPs except for the ``[tool]`` table. Within that table, tools can have users specify configuration data as long as they use a sub-table within ``[tool]``, e.g. the `flit <https://pypi.python.org/pypi/flit>`_ tool might store its configuration in ``[tool.flit]``. We need some mechanism to allocate names within the ``tool.*`` namespace, to make sure that different projects don't attempt to use the same sub-table and collide. Our rule is that a project can use the subtable ``tool.$NAME`` if, and only if, they own the entry for ``$NAME`` in the Cheeseshop/PyPI. Open Issues =========== Name of the build-related sub-table ----------------------------------- The authors of this PEP couldn't decide between the names ``[package.build]`` and ``[package.build-system]``, and so it is an open issue over which one to go with. Rejected Ideas ============== Other semantic version key names -------------------------------- Names other than ``semantics-version`` were considered to represent the version of semantics that the configuration file was written for. Both ``configuration-version`` and ``metadata-version`` were both considered, but were rejected due to how people may confuse the key as representing a version of the files contents instead of the version of semantics that the file is interpreted under. A flatter namespace ------------------- An earlier draft of this PEP lacked the ``[package]`` table and had all of its contained values one level higher. In the end it was decided it would be better to scope package-related details to its own table for more clear scoping and easier expansion of this file for future use. Other file formats ------------------ Several other file formats were put forward for consideration, all rejected for various reasons. Key requirements were that the format be editable by human beings and have an implementation that can be vendored easily by projects. This outright exluded certain formats like XML which are not friendly towards human beings and were never seriously discussed. JSON '''' The JSON format [#json]_ was initially considered but quickly rejected. While great as a human-readable, string-based data exchange format, the syntax does not lend itself to easy editing by a human being (e.g. the syntax is more verbose than necessary while not allowing for comments). An example JSON file for the proposed data would be:: { "build": { "requires": [ "setuptools", "wheel>=0.27" ] } } YAML '''' The YAML format [#yaml]_ was designed to be a superset of JSON [#json]_ while being easier to work with by hand. There are three main issues with YAML. One is that the specification is large: 86 pages if printed on letter-sized paper. That leaves the possibility that someone may use a feature of YAML that works with one parser but not another. It has been suggested to standardize on a subset, but that basically means creating a new standard specific to this file which is not tractable long-term. Two is that YAML itself is not safe by default. The specification allows for the arbitrary execution of code which is best avoided when dealing with configuration data. It is of course possible to avoid this behavior -- for example, PyYAML provides a ``safe_load`` operation -- but if any tool carelessly uses ``load`` instead then they open themselves up to arbitrary code execution. While this PEP is focused on the building of projects which inherently involves code execution, other configuration data such as project name and version number may end up in the same file someday where arbitrary code execution is not desired. And finally, the most popular Python implemenation of YAML is PyYAML [#pyyaml]_ which is a large project of a few thousand lines of code and an optional C extension module. While in and of itself this isn't necessary an issue, this becomes more of a problem for projects like pip where they would most likely need to vendor PyYAML as a dependency so as to be fully self-contained (otherwise you end up with your install tool needing an install tool to work). A proof-of-concept re-working of PyYAML has been done to see how easy it would be to potentially vendor a simpler version of the library which shows it is a possibility. An example YAML file is:: build: requires: - setuptools - wheel>=0.27 configparser '''''''''''' An INI-style configuration file based on what configparser [#configparser]_ accepts was considered. Unfortunately there is no specification of what configparser accepts, leading to support skew between versions. For instance, what ConfigParser in Python 2.7 accepts is not the same as what configparser in Python 3 accepts. While one could standardize on what Python 3 accepts and simply vendor the backport of the configparser module, that does mean this PEP would have to codify that the backport of configparser must be used by all project wishes to consume the metadata specified by this PEP. This is overly restrictive and could lead to confusion if someone is not aware of that a specific version of configparser is expected. An example INI file is:: [build] requires = setuptools wheel>=0.27 Python literals ''''''''''''''' Someone proposed using Python literals as the configuration format. All Python programmers would be used to the format, there would implicitly be no third-party dependency to read the configuration data, and it can be safe if something like ``ast.literal_eval()`` [#ast_literal_eval]_. The problem is that to user Python literals you either end up with something no better than JSON, or you end up with something like what Bazel [#bazel]_ uses. In the former the issues are the same as JSON. In the latter, you end up with people consistently asking for more flexibility as users have a hard time ignoring the desire to use some feature of Python that they think they need (one of the co-authors has direct experience with this from the internal usage of Bazel at Google). There is no example format as one was never put forward for consideration. Other file names ---------------- Several other file names were considered and rejected (although this is very much a bikeshedding topic, and so the decision comes down to mostly taste). pysettings.toml Most reasonable alternative. pypa.toml While it makes sense to reference the PyPA [#pypa]_, it is a somewhat niche term. It's better to have the file name make sense without having domain-specific knowledge. pybuild.toml From the restrictive perspective of this PEP this filename makes sense, but if any non-build metadata ever gets added to the file then the name ceases to make sense. pip.toml Too tool-specific. meta.toml Too generic; project may want to have its own metadata file. setup.toml While keeping with traditional thanks to ``setup.py``, it does not necessarily match what the file may contain in the future (.e.g is knowing the name of a project inerhently part of its setup?). pymeta.toml Not obvious to newcomers to programming and/or Python. pypackage.toml & pypackaging.toml Name conflation of what a "package" is (project versus namespace). pydevelop.toml The file may contain details not specific to development. pysource.toml Not directly related to source code. pytools.toml Misleading as the file is (currently) aimed at project management. dstufft.toml Too person-specific. ;) References ========== .. [#distutils] distutils (https://docs.python.org/3/library/distutils.html#module-distutils) .. [#setuptools] setuptools (https://pypi.python.org/pypi/setuptools) .. [#setup_args] setuptools: New and Changed setup() Keywords (http://pythonhosted.org/setuptools/setuptools.html#new-and-changed-setup-ke…) .. [#pip] pip (https://pypi.python.org/pypi/pip) .. [#wheel] wheel (https://pypi.python.org/pypi/wheel) .. [#toml] TOML (https://github.com/toml-lang/toml) .. [#json] JSON (http://json.org/) .. [#yaml] YAML (http://yaml.org/) .. [#configparser] configparser (https://docs.python.org/3/library/configparser.html#module-configparser) .. [#pyyaml] PyYAML (https://pypi.python.org/pypi/PyYAML) .. [#pypa] PyPA (https://www.pypa.io) .. [#bazel] Bazel (http://bazel.io/) .. [#ast_literal_eval] ``ast.literal_eval()`` (https://docs.python.org/3/library/ast.html#ast.literal_eval) .. [#cargo] Cargo, Rust's package manager (http://doc.crates.io/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

15 52

Distutils improvements regarding header installation and building C extension modules
by Sylvain Corlay 25 May '16

25 May '16

Hello everyone, This is my first post here so, apologies if I am breaking any rules. Lately, I have been filing a few bug reports and patches to distutils on bugs.python.org that all concern the installation and build of C++ extensions. *1) The distutils.ccompiler has_flag method.* (http://bugs.python.org/issue26689) When building C++ extension modules that require a certain compiler flag (such as enabling C++11 features), you may want to check if the compiler has such a flag available. I proposed a patch adding a `has_flag` method to ccompiler. It is an equivalent to cmake' s CHECK_CXX_COMPILER_FLAG. The implementation is similar to the one of has_function which by the way has a pending patch by minrk in issue (http://bugs.python.org/issue25544). *2) On the need for something like pip.locations.distutils_scheme in distutils* (http://bugs.python.org/issue26955) When installing a python package that has a directive for the install_headers distutils command, these headers are usually installed under the main python include directory, which can be retrieved with sysconfig.get_path('include') or directly referred to as the 'include' string when setting the include directories of an extension module. However, on some systems like OS X, headers for extension modules are not located in under the python include directory /usr/local/Cellar/pythonX/X.Y.Z/Frameworks/Python.framework/Versions/X.Y/include/pythonX.Y but in /usr/local/include/pythonX.Y. Is there a generic way to find the location where headers are installed in a python install? pip.locations has a distutils_scheme function which seems to be returning the right thing, but it seems to be a bit overkill to require pip. On the other side, no path returned by sysconfig corresponds to `/usr/local/include/pythonX.Y`. *3) On allowing `install_headers` command to follow a specific directory structure* (http://bugs.python.org/issue27123) (instead of making a flat copy of the specified list of headers) Unlike wheel's data files, which can be specified as a list of tuples including the target directories [(target_directory, [list of files for target directory])] the `headers` setup keyword argument only let's you specify a list of files that will be copied over to a sub-directory of sysconfig.get_path('include'). It would be useful to enable the same feature for headers as we have for data files. If this is a feature that you would also like to see, I can propose a patch to the install_headers command. Thanks, Syllvain

2 1

Publicly Queryable Statistics
by Donald Stufft 22 May '16

22 May '16

Hey, One thing I’ve been working on as part of Warehouse, is a subproject that I call “Linehaul”. This is essentially a little statistics daemon that will take specially formatted syslog messages coming off of Fastly and shove them inside of a BigQuery database. I’m happy to report that I’ve just finished the production deployment of this and we’re now sending every download event that hits Fastly into BigQuery. First off, I’d like to thank Felipe Hoffa, Will Curran, and Preston Holmes over at Google for helping me get credits for this sorted out so that we can actually get this going! They’ve been a big help. So onto what this means. Basically, BigQuery gives us the ability to relatively quickly (typically < 60s) query very large datasets using something that is very similar to SQL. Unlike typical time series databases we don’t have to know ahead of time what we want to query on, we can just insert data into rows in a table (and our tables are sharded by days) and then using the SQLlike query language, you can do any sort of query you like. On top of all of this, BigQuery gives me the ability to share the dataset publicly with anyone who is logged into a Google account, which means that *anyone* can query this data and look for any sort of interesting information they can find in it. The cost of any queries you run will be associated with your own account (but the first 1TB of data a month that you query is free I believe, nor are you charged for queries that error out or return cached results). Anyways, you can query these BigQuery tables whose names match the pattern `the-psf:pypi.downloadsYYYYMMDD` and you can see whatever data you want. We’ve only just started recorded the data in this spot so right now there isn’t a whole lot of data available (but it’s constantly streaming in). Once the 22nd rolls over I’m going to delete what data we have available for the 21st, and then start backfilling historical data starting with the 21st and going backwards. You should be able to run queries in a Web UI by navigating to https://bigquery.cloud.google.com/dataset/the-psf:pypi (you might have to accept a Terms of Service). The table schema looks like: https://s.caremad.io/lPpTF6rxWZ/ but it should also be visible on the big query page. Some example queries you might want to run are located at https://gist.github.com/alex/4f100a9592b05e9b4d63 (but note, that’s currently using the *old* not publicly available table name, you’ll need to replace [long-stack-762:pypi.downloads] with [the-psf.pypi:downloads]). If you want to write your own queries, you should be able to find the syntax here: https://cloud.google.com/bigquery/query-reference Anyways, new data should constantly be streaming in, and I should be able to backfill data all the way to Jan of 2014 or so. Hopefully this is useful to folks, and if you find any interesting queries or numbers, please share them! — Donald Stufft

3 5

build system requirements PEP, 3rd draft
by Brett Cannon 16 May '16

16 May '16

Biggest changes since the initial draft: 1. No more semantics-version 2. No more [package] table 3. Settled on [build-system] as the table name 4. The "requires" key is required if [build-system] is defined 5. Changed the title and clarified that this is all about the minimum requirements for the build system to execute (some added support for things like dynamic dependencies for producing build artifacts is for another PEP) 6. Added a JSON Schema for the resulting data from the table because Nick likes his specs :) ---------- PEP: NNN Title: Specifying Minimum Build System Requirements for Python Projects Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon <brett(a)python.org>, Nathaniel Smith <njs(a)pobox.com>, Donald Stufft <donald(a)stufft.io> BDFL-Delegate: Nick Coghlan Discussions-To: distutils-sig <distutils-sig at python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 10-May-2016 Post-History: 10-May-2016, 11-May-2016, 13-May-2016 Abstract ======== This PEP specifies how Python software packages should specify what dependencies they have in order to execute their chosen build system. As part of this specification, a new configuration file is introduced for software packages to use to specify their build dependencies (with the expectation that the same configuration file will be used for future configuration details). Rationale ========= When Python first developed its tooling for building distributions of software for projects, distutils [#distutils]_ was the chosen solution. As time went on, setuptools [#setuptools]_ gained popularity to add some features on top of distutils. Both used the concept of a ``setup.py`` file that project maintainers executed to build distributions of their software (as well as users to install said distribution). Using an executable file to specify build requirements under distutils isn't an issue as distutils is part of Python's standard library. Having the build tool as part of Python means that a ``setup.py`` has no external dependency that a project maintainer needs to worry about to build a distribution of their project. There was no need to specify any dependency information as the only dependency is Python. But when a project chooses to use setuptools, the use of an executable file like ``setup.py`` becomes an issue. You can't execute a ``setup.py`` file without knowing its dependencies, but currently there is no standard way to know what those dependencies are in an automated fashion without executing the ``setup.py`` file where that information is stored. It's a catch-22 of a file not being runnable without knowing its own contents which can't be known programmatically unless you run the file. Setuptools tried to solve this with a ``setup_requires`` argument to its ``setup()`` function [#setup_args]_. This solution has a number of issues, such as: * No tooling (besides setuptools itself) can access this information without executing the ``setup.py``, but ``setup.py`` can't be executed without having these items installed. * While setuptools itself will install anything listed in this, they won't be installed until *during* the execution of the ``setup()`` function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the ``setup()`` function. * This cannot include ``setuptools`` itself nor can it include a replacement to ``setuptools``, which means that projects such as ``numpy.distutils`` are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one. * The items listed in ``setup_requires`` get implicily installed whenever you execute the ``setup.py`` but one of the common ways that the ``setup.py`` is executed is via another tool, such as ``pip``, who is already managing dependencies. This means that a command like ``pip install spam`` might end up having both pip and setuptools downloading and installing packages and end users needing to configure *both* tools (and for ``setuptools`` without being in control of the invocation) to change settings like which repository it installs from. It also means that users need to be aware of the discovery rules for both tools, as one may support different package formats or determine the latest version differently. This has cumulated in a situation where use of ``setup_requires`` is rare, where projects tend to either simply copy and paste snippets between ``setup.py`` files or they eschew it all together in favor of simply documenting elsewhere what they expect the user to have manually installed prior to attempting to build or install their project. All of this has led pip [#pip]_ to simply assume that setuptools is necessary when executing a ``setup.py`` file. The problem with this, though, is it doesn't scale if another project began to gain traction in the commnity as setuptools has. It also prevents other projects from gaining traction due to the friction required to use it with a project when pip can't infer the fact that something other than setuptools is required. This PEP attempts to rectify the situation by specifying a way to list the minimal dependencies of the build system of a project in a declarative fashion in a specific file. This allows a project to list what build dependencies it has to go from e.g. source checkout to wheel, while not falling into the catch-22 trap that a ``setup.py`` has where tooling can't infer what a project needs to build itself. Implementing this PEP will allow projects to specify what build system they depend on upfront so that tools like pip can make sure that they are installed in order to run the build system to build the project. To provide more context and motivation for this PEP, think of the (rough) steps required to produce a built artifact for a project: 1. The source checkout of the project. 2. Installation of the build system. 3. Execute the build system. This PEP covers step #2. It is fully expected that a future PEP will cover step #3, including how to have the build system dynamically specify more dependencies that the build system requires to perform its job. The purpose of this PEP though, is to specify the minimal set of requirements for the build system to simply begin execution. Specification ============= The build system dependencies will be stored in a file named ``pyproject.toml`` that is written in the TOML format [#toml]_. This format was chosen as it is human-usable (unlike JSON [#json]_), it is flexible enough (unlike configparser [#configparser]_), stems from a standard (also unlike configparser [#configparser]_), and it is not overly complex (unlike YAML [#yaml]_). The TOML format is already in use by the Rust community as part of their Cargo package manager [#cargo]_ and in private email stated they have been quite happy with their choice of TOML. A more thorough discussion as to why various alternatives were not chosen can be read in the `Other file formats`_ section. There will be a ``[build-system]`` table in the configuration file to store build-related data. Initially only one key of the table will be valid and mandatory: ``requires``. That key will have a value of a list of strings representing the PEP 508 dependencies required to execute the build system (currently that means what dependencies are required to execute a ``setup.py`` file). To provide a type-specific representation of the resulting data from the TOML file for illustrative purposes only, the following JSON Schema [#jsonschema]_ would match the data format:: { "$schema": "http://json-schema.org/schema#", "type": "object", "additionalProperties": false, "properties": { "build-system": { "type": "object", "additionalProperties": false, "properties": { "requires": { "type": "array", "items": { "type": "string" } } }, "required": ["requires"] }, "tool": { "type": "object" } } } For the vast majority of Python projects that rely upon setuptools, the ``pyproject.toml`` file will be:: [build-system] # Minimum requirements for the build system to execute. requires = ["setuptools", "wheel"] # PEP 508 specifications. Because the use of setuptools and wheel are so expansive in the community at the moment, build tools are expected to use the example configuration file above as their default semantics when a ``pyproject.toml`` file is not present. All other top-level keys and tables are reserved for future use by other PEPs except for the ``[tool]`` table. Within that table, tools can have users specify configuration data as long as they use a sub-table within ``[tool]``, e.g. the `flit <https://pypi.python.org/pypi/flit>`_ tool would store its configuration in ``[tool.flit]``. We need some mechanism to allocate names within the ``tool.*`` namespace, to make sure that different projects don't attempt to use the same sub-table and collide. Our rule is that a project can use the subtable ``tool.$NAME`` if, and only if, they own the entry for ``$NAME`` in the Cheeseshop/PyPI. Rejected Ideas ============== A semantic version key ---------------------- For future-proofing the structure of the configuration file, a ``semantics-version`` key was initially proposed. Defaulting to ``1``, the idea was that if any semantics changes to previously defined keys or tables occurred which were not backwards-compatible, then the ``semantics-version`` would be incremented to a new number. In the end, though, it was decided that this was a premature optimization. The expectation is that changes to what is pre-defined semantically in the configuration file will be rather conservative. And in the instances where a backwards-incompatible change would have occurred, different names can be used for the new semantics to avoid breaking older tools. A more nested namespace ----------------------- An earlier draft of this PEP had a top-level ``[package]`` table. The idea was to impose some scoping for a semantics versioning scheme (see `A semantic version key`_ for why that idea was rejected). With the need for scoping removed, the point of having a top-level table became superfluous. Other table names ----------------- Another name proposed for the ``[build-system]`` table was ``[build]``. The alternative name is shorter, but doesn't convey as much of the intention of what information is store in the table. After a vote on the distutils-sig mailing list, the current name won out. Other file formats ------------------ Several other file formats were put forward for consideration, all rejected for various reasons. Key requirements were that the format be editable by human beings and have an implementation that can be vendored easily by projects. This outright exluded certain formats like XML which are not friendly towards human beings and were never seriously discussed. JSON '''' The JSON format [#json]_ was initially considered but quickly rejected. While great as a human-readable, string-based data exchange format, the syntax does not lend itself to easy editing by a human being (e.g. the syntax is more verbose than necessary while not allowing for comments). An example JSON file for the proposed data would be:: { "build": { "requires": [ "setuptools", "wheel>=0.27" ] } } YAML '''' The YAML format [#yaml]_ was designed to be a superset of JSON [#json]_ while being easier to work with by hand. There are three main issues with YAML. One is that the specification is large: 86 pages if printed on letter-sized paper. That leaves the possibility that someone may use a feature of YAML that works with one parser but not another. It has been suggested to standardize on a subset, but that basically means creating a new standard specific to this file which is not tractable long-term. Two is that YAML itself is not safe by default. The specification allows for the arbitrary execution of code which is best avoided when dealing with configuration data. It is of course possible to avoid this behavior -- for example, PyYAML provides a ``safe_load`` operation -- but if any tool carelessly uses ``load`` instead then they open themselves up to arbitrary code execution. While this PEP is focused on the building of projects which inherently involves code execution, other configuration data such as project name and version number may end up in the same file someday where arbitrary code execution is not desired. And finally, the most popular Python implemenation of YAML is PyYAML [#pyyaml]_ which is a large project of a few thousand lines of code and an optional C extension module. While in and of itself this isn't necessarily an issue, this becomes more of a problem for projects like pip where they would most likely need to vendor PyYAML as a dependency so as to be fully self-contained (otherwise you end up with your install tool needing an install tool to work). A proof-of-concept re-working of PyYAML has been done to see how easy it would be to potentially vendor a simpler version of the library which shows it is a possibility. An example YAML file is:: build: requires: - setuptools - wheel>=0.27 configparser '''''''''''' An INI-style configuration file based on what configparser [#configparser]_ accepts was considered. Unfortunately there is no specification of what configparser accepts, leading to support skew between versions. For instance, what ConfigParser in Python 2.7 accepts is not the same as what configparser in Python 3 accepts. While one could standardize on what Python 3 accepts and simply vendor the backport of the configparser module, that does mean this PEP would have to codify that the backport of configparser must be used by all project wishes to consume the metadata specified by this PEP. This is overly restrictive and could lead to confusion if someone is not aware of that a specific version of configparser is expected. An example INI file is:: [build] requires = setuptools wheel>=0.27 Python literals ''''''''''''''' Someone proposed using Python literals as the configuration format. All Python programmers would be used to the format, there would implicitly be no third-party dependency to read the configuration data, and it can be safe if something like ``ast.literal_eval()`` [#ast_literal_eval]_. The problem is that to user Python literals you either end up with something no better than JSON, or you end up with something like what Bazel [#bazel]_ uses. In the former the issues are the same as JSON. In the latter, you end up with people consistently asking for more flexibility as users have a hard time ignoring the desire to use some feature of Python that they think they need (one of the co-authors has direct experience with this from the internal usage of Bazel at Google). There is no example format as one was never put forward for consideration. Other file names ---------------- Several other file names were considered and rejected (although this is very much a bikeshedding topic, and so the decision comes down to mostly taste). pysettings.toml Most reasonable alternative. pypa.toml While it makes sense to reference the PyPA [#pypa]_, it is a somewhat niche term. It's better to have the file name make sense without having domain-specific knowledge. pybuild.toml From the restrictive perspective of this PEP this filename makes sense, but if any non-build metadata ever gets added to the file then the name ceases to make sense. pip.toml Too tool-specific. meta.toml Too generic; project may want to have its own metadata file. setup.toml While keeping with traditional thanks to ``setup.py``, it does not necessarily match what the file may contain in the future (.e.g is knowing the name of a project inerhently part of its setup?). pymeta.toml Not obvious to newcomers to programming and/or Python. pypackage.toml & pypackaging.toml Name conflation of what a "package" is (project versus namespace). pydevelop.toml The file may contain details not specific to development. pysource.toml Not directly related to source code. pytools.toml Misleading as the file is (currently) aimed at project management. dstufft.toml Too person-specific. ;) References ========== .. [#distutils] distutils (https://docs.python.org/3/library/distutils.html#module-distutils) .. [#setuptools] setuptools (https://pypi.python.org/pypi/setuptools) .. [#setup_args] setuptools: New and Changed setup() Keywords ( http://pythonhosted.org/setuptools/setuptools.html#new-and-changed-setup-ke… ) .. [#pip] pip (https://pypi.python.org/pypi/pip) .. [#wheel] wheel (https://pypi.python.org/pypi/wheel) .. [#toml] TOML (https://github.com/toml-lang/toml) .. [#json] JSON (http://json.org/) .. [#yaml] YAML (http://yaml.org/) .. [#configparser] configparser (https://docs.python.org/3/library/configparser.html#module-configparser) .. [#pyyaml] PyYAML (https://pypi.python.org/pypi/PyYAML) .. [#pypa] PyPA (https://www.pypa.io) .. [#bazel] Bazel (http://bazel.io/) .. [#ast_literal_eval] ``ast.literal_eval()`` (https://docs.python.org/3/library/ast.html#ast.literal_eval) .. [#cargo] Cargo, Rust's package manager (http://doc.crates.io/) .. [#jsonschema] JSON Schema (http://json-schema.org/) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

4 5

windows SSL: CERTIFICATE_VERIFY_FAILED
by Adam GROSZER 16 May '16

16 May '16

Hi, I have here an old windows server 2003R2 (winbot.zope.org). Recently updated python 2.7.0 to 2.7.11. Now running into SSL: CERTIFICATE_VERIFY_FAILED with buildout bootstrap. I installed DigiCertHighAssuranceEVRootCA.crt to "Certificates (Local Computer)\Trusted Root Certification Authorities\Certificates. As detailed at: http://www.databasemart.com/howto/SQLoverssl/How_To_Install_Trusted_Root_Ce… But still get: C:\buildslave\zope.testing\build>c:\Python27_32\python.exe bootstrap.py Traceback (most recent call last): File "bootstrap.py", line 92, in <module> exec(urlopen('https://bootstrap.pypa.io/ez_setup.py').read(), ez) File "c:\Python27_32\lib\urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "c:\Python27_32\lib\urllib2.py", line 431, in open response = self._open(req, data) File "c:\Python27_32\lib\urllib2.py", line 449, in _open '_open', req) File "c:\Python27_32\lib\urllib2.py", line 409, in _call_chain result = func(*args) File "c:\Python27_32\lib\urllib2.py", line 1240, in https_open context=self._context) File "c:\Python27_32\lib\urllib2.py", line 1197, in do_open raise URLError(err) urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)> -- Best regards, Adam GROSZER -- Quote of the day: The Church has many critics but few rivals. - Anonymous

1 1

comparison of configuration languages
by Nathaniel Smith 16 May '16

16 May '16

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned: https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f -n -- Nathaniel J. Smith -- https://vorpus.org

26 97

Download Counts Temporarily Disabled
by Donald Stufft 16 May '16

16 May '16

Hey, Just an FYI, I've disabled download counts on PyPI for the time being. The statistics stack is broken and needs engineering effort to fix it back up to deal with changes to PyPI. It was suggested that hiding the counts would help prevent user confusion when they see things like "downloaded 0 times" making people believe that a library has no users, even if it is a significantly downloaded library. I'm unlikely to get around to fixing the current stack since, as part of Warehouse, I'm working on a *new* statistics stack which is much better. The data collection and storage parts of that stack are already done and I just need to get querying done (made more difficult by the fact that the new system queries can take 10+ seconds to complete, but can be queried on any dimension) and a tool to process the historical data and put it into the new storage engine. Anyways, this is just to let folks know that this isn't a permanent loss of the feature and we won't lose any data. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

1 0

devpi-server-4.0: fixing the pip-8.1.2 issue / pep 503 compliance
by holger krekel 13 May '16

13 May '16

devpi-server-4.0: fixing the pip-8.1.2 problem / PEP503 compliance ============================================================================ We've made available critically important releases of the devpi private packaging available. If you are not using "devpi" yet then you can may just read http://doc.devpi.net and forget about the rest of this announcement. This is for the many who experienced the "pip doesn't install packages anymore with devpi" problem. First of all, you may temporarily pin "pip" to avoid the problem on the client side: pip install pip==8.1.1 This is obviously a crutch but gives you some time to perform the export/import cycle required for hosting private packages via devpi-server-4.0 and being compatible with pip-8.1.2. If you are using devpi-server as a pure pypi.python.org cache you don't need to perform export/import and can just delete your server directory ($HOME/.devpi/server by default) before you install and start up devpi-server-4.0. If you are hosting private packages on devpi you will need to perform an export/import cycle of your server state in order to run devpi-server-4.0. The "4.0" in this case only signals this export/import need -- no other big changes are coming with 4.0. At the end of this announcement we explain some details of why we needed to go for a 4.0 and not just a micro bugfix release. To export from devpi-server-3.X -------------------------------- upgrade to the new devpi-server-3.1.2 before you export, like this: pip install "devpi-server<4.0" Now stop your server and run: devpi-server --export EXPORTDIR --serverdir SERVERDIR where EXPORTDIR should be a fresh new directory and SERVERDIR should be the server state directory ($HOME/.devpi/server by default). To export from devpi-server-2.X -------------------------------- Upgrade to the latest devpi-server-2.X release: pip install "devpi-server<3.0" devpi-common>=2.0.10 Here we force the devpi-common dependency to not accidentally be "devpi-common==2.0.9" which could lead to problems. Now stop your server and run: devpi-server --export EXPORTDIR --serverdir SERVERDIR where EXPORTDIR should be a fresh new directory and SERVERDIR should be the server state directory ($HOME/.devpi/server by default). to import state into devpi-server-4.0 ---------------------------------------- Upgrade to the latest devpi-server-4.X release: pip install "devpi-server<5.0" devpi-web If you don't use "devpi-web" you can leave it out from the pip command. Check you have the right version: devpi-server --version Now import from your previously created EXPORTDIR: devpi-server --serverdir SERVERDIR_NEW --import EXPORTDIR This will take a while if you have many indexes or lots of documentation -- devpi-web will create a search index over all of it during import. You are now good to go -- pip works again! devpi-client also has a 2.6.3 -------------------------------- We also published a minor bugfix "devpi-client-2.6.3" release which should work with both devpi-server-2.6 and devpi-server-4.0 as we are generally trying to keep devpi-client forward/backward compatible. You only need to install devpi-client-2.6.3 if you also install devpi-server into the same virtual environment. Otherwise using devpi-client-2.6.2 with both devpi-server-2.6 and devpi-server-4.0 probably works fine as well. background on the pip/devpi bug for the curious ----------------------------------------------- Besides devpi, also artifactory and other private index servers have experienced failures with pip-8.1.2. The change from 8.1.1 was that pip now asks for PEP503-normalized names when requesting the simple page from an index. Previously "-" and "." would be allowed but with the new normalization "." is substituted with "-". Now "pip install zope.interface" triggers a request to "+simple/zope-interface" and devpi in turns asks pypi.python.org/simple/zope-interface and gets an answer with lots of "zope.interface-*.tar.gz" release links. But those are not matched because without PEP503 "zope.interface" and "zope-interface" are different things. Moreover, pypi.python.org used to redirect to the "true" name but does not do this anymore which contributed to the eventual problem. We decided to go for 4.0 because since 3.0 we base database keys on normalized project names -- and this normalization is used in like 20-30 code places across the devpi system and plugins. Trying to be clever and avoid the export/import and trick "pip-8.1.2" into working looked like a can of worms. Now with devpi-server-4.0 we are using proper PEP503 specified normalization so should be safe. best, holger and florian P.S.: we offer support contracts btw and thank in particular Dolby Laboratories, YouGov Inc and BlueYonder GmbH who funded a lot of the last year's devpi work and now agreed to be named in public - and no, we didn't get around to make a flashy web site yet. For now, just mail holger at merlinux to discuss support and training options.

1 0