Mailman 3 November 2015 - Distutils-SIG

Re: [Distutils] Link to lib missing in PEP: FINAL DRAFT: Dependency specifier PEP
by Thomas Güttler 18 Nov '15

18 Nov '15

Am 17.11.2015 um 15:13 schrieb Ronny Pfannschmidt: > That is no acceptable way to answer in particular since the original mail continued with places where to put the link I was asking for a link from the PEP to the implementation. I received a "no". I replyed with my opinion. What is not acceptable? Regards, Thomas Güttler -- http://www.thomas-guettler.de/

1 0

Re: [Distutils] New PyPUG Tutorials
by Marcus Smith 17 Nov '15

17 Nov '15

> > > To have the most success, the writers will certainly need feedback from > subject matter experts, so the process will include 2 stages where we > specifically ask for feedback from PyPA-Dev and Distutils-Sig: 1) To > validate the initial proposal that covers the scope of the changes, and 2) > to review the actual PRs to PyPUG for accuracy, when it's time for merging. > I'll post again with more details as those stages occur. > So, I'm back to post the initial proposal as mentioned above. The proposal was put together by Daniel Beck (http://www.danieldbeck.com), one of the writers Nicole (from the Warehouse team) was able to rally with her call for volunteers [1]. It includes a *tentative* new outline, but keep in mind that however it turns out, we will maintain redirects for the current critical links, like the tool recommendations page, and the two current tutorials. As work proceeds, it will take place as PRs to the develop branch of the PyPUG (https://github.com/pypa/python-packaging-user-guide/), so people can follow along and help in the technical review process. When the final product is deemed worthy and ready, there will be a final call for review before the merge of develop back into master. Both Daniel and Nicole are now subscribed to distutils-sig, and can respond to any questions, concerns, or feedback. Here's the proposal, as quoted from an email from Daniel: ----------------------------------------- Scope --------- The guide is meant to answer this question: "How do Python, packages, and packaging tools fit together?" The guide's not to meant to replace tools' docs; rather, it's to show when and how these tools and other artifacts work as an ecosystem. Audience ------------- I get the impression that these are the major audience groups we want to help with the guide: - Python users who are at an 'advanced beginner' level (e.g., know how to import from the standard library, but want to learn to install a third-party package for the first time - Python users with intermediate experience who want to make and distribute a package for the first time - Python users with more experience who need a refresher or want to understand what the "one way to do it" is right now - Python users with lots of experience who want to take on some advanced or specialized packaging task Outline ---------- I sketched out a gross outline for the guide. This is not complete, sometimes speculative, and certainly subject to change, but I wanted to give you an overall picture of how I imagine the guide could look when we've had some time to work on this. Again, because the existing content is so good, I don't expect much, if anything, will be deleted—so just because it's not listed here doesn't mean anything. Rather, I expect that stuff will be easier to find. :-) 1. About this guide 2. Getting help 3. Quick references (i.e., five minute tutorials) a. Installing a package b. Making a package c. Publish a package on PyPI 4. Intro to Packaging* a. Who is this for? b. Prereqs (making sure Python, etc. are working) c. What's a package? d. Where do packages come from? e. Installing a package for the first time f. Using a virtualenv g. Making a package for the first time h. Publishing your first package 5. The Packaging Ecosystem (concept docs) a. Overview of tools b. Developing packages c. Distributing packages d. Indices and caches 6. Practices and patterns (task docs) a. Making packages b. Publishing packages c. Using packages 7. History and legacy of packaging a. that whole distutils->setuptools->distribute->setuptools saga b. history of PyPA c. eggs vs wheels d. easy_install vs pip e. cheeseshop vs warehouse 8. Contributing to this guide a. Style guide * This is the only part that I think functions as a continuous unit meant to be read from start to end. I'm imagining something that would work as, for example, a lesson plan for a PyLadies workshop. For the rest of the guide, I want to work with the expectation that every page is a landing page. Timeline ------------ Here's my current perspective on how quickly we can move on these things: - In the next few days: contacting volunteers to let them know that things are happening soon (Nicole) - In the 7 or 8 days: decide on workflow questions (Nicole and Marcus), amend project roles (Nicole and Marcus), and ask for distutils-sig approval (Marcus) - In the next two weeks: start opening issues and inviting volunteers to start writing - By the end of November: most, if not all, existing content could be reorganized into a new outline - By mid-December: have publishable versions of the new tutorials - By the end of January: potentially finished? [1] https://github.com/pypa/warehouse/issues/729

2 2

Re: [Distutils] build system abstraction PEP, take #2
by Ronny Pfannschmidt 17 Nov '15

17 Nov '15

I forgot to reply to the list, so I re-added it To further explain, in my setuptools replacement experiment I implement the develop command by generating a a wheel Using a local version tag `+develop`, Instead of the actual code, for each top-level it contains a shim python file that will load the real code from the editable location However it will use pip to install that wheel and create scripts/exe files In particular windows support for a full develop command is orders of magnitude more easy if one let's pip make the scripts(exe files) I think it's very helpful for tool authors if pip has ways for them to avoid platform pitfalls like exe files Am 17. November 2015 08:15:41 MEZ, schrieb Robert Collins <robertc(a)robertcollins.net>: >On 17 November 2015 at 19:16, Ronny Pfannschmidt ><opensource(a)ronnypfannschmidt.de> wrote: >> The develop command should be able to generate a wheel with a local >version >> that pip can install >> That way tool authors can completely avoid writing the target folders >and >> its possible to always run completely unprivileged builds > >Thats an interesting idea, but wholly different to what 'develop' does >and is used for. My goal here is to only commit to new things where >the new thing is all of: > - a small amount of work > - meets existing use cases > - able to be put into pip/setuptools/wheel/etc now (rather than after >a bunch of other things are done) > >I completely agree with the idea that develop as it stands has a >number of limitations, but there is no generic implementation >available for pip to reuse; there is no feature in pip that would do >what you want either; nor even a proof of concept. I think as such its >premature to PEPify it. > >What could be done is to take the abstract build system work (which >*is* within reach for pip) and create a implementation with a new >schema version and whatever semantics you want around this new >developish thing, and write an incremental PEP (so you don't need any >of the PEP I've put up, just inherit from it and define your new thing >- and then we can kick the tires on it and see how its going to work >etc. > >Note that what you describe 'generate a wheel with a local version' is >actually exactly what the 'wheel' command does as far as I can tell, >but I'm sure thats not what you have in mind - so please do expand on >the differences, in >paranoid-folk-make-sure-folk-cannot-misunderstand-style ;) > >-Rob > > >-- >Robert Collins <rbtcollins(a)hp.com> >Distinguished Technologist >HP Converged Cloud -- Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.

1 0

New PEP : dependency specification
by Robert Collins 16 Nov '15

16 Nov '15

Since we ended up with a hard dependency on this for the bootstrap thing (regardless of 'smaller step' or not) - I've broken this out of PEP 426, made it an encoding of the current status quo rather than an aspirational change. Since it has a dependency on markers, I had to choose whether to block on James' marker PEP, contribute to that, or include it. I think on balance it makes sense to have it in one document since the markers bit is actually quite shallow, so I've done that (after discussing with James). This could perhaps replace PEP 496 then, or be given a new number. Donald has graciously agreed to be a BDFL-delegate for it. The PR for it is https://github.com/pypa/interoperability-peps/pull/56 Full text follows. :PEP: XX :Title: Dependency specification for Python Software Packages :Version: $Revision$ :Last-Modified: $Date$ :Author: Robert Collins <rbtcollins(a)hp.com> BDFL-Delegate: Donald Stufft <donald(a)stufft.io> :Discussions-To: distutils-sig <distutils-sig(a)python.org> :Status: Draft :Type: Standards Track :Content-Type: text/x-rst :Created: 11-Nov-2015 :Post-History: XX Abstract ======== This PEP specifies the language used to describe dependencies for packages. It draws a border at the edge of describing a single dependency - the different sorts of dependencies and when they should be installed is a higher level problem. The intent is provide a building block for higher layer specifications. The job of a dependency is to enable tools like pip [#pip]_ to find the right package to install. Sometimes this is very loose - just specifying a name, and sometimes very specific - referring to a specific file to install. Sometimes dependencies are only relevant in one platform, or only some versions are acceptable, so the language permits describing all these cases. The language defined is a compact line based format which is already in widespread use in pip requirements files, though we do not specify the command line option handling that those files permit. There is one caveat - the URL reference form, specified in PEP-440 [#pep440]_ is not actually implemented in pip, but since PEP-440 is accepted, we use that format rather than pip's current native format. Motivation ========== Any specification in the Python packaging ecosystem that needs to consume lists of dependencies needs to build on an approved PEP for such, but PEP-426 [#pep426]_ is mostly aspirational - and there are already existing implementations of the dependency specification which we can instead adopt. The existing implementations are battle proven and user friendly, so adopting them is arguably much better than approving an aspirational, unconsumed, format. Specification ============= Examples -------- All features of the language shown with a name based lookup:: requests \ [security] >= 2.8.1, == 2.8.* \ ; python_version < "2.7.10" \ # Fix HTTPS on older Python versions. A minimal URL based lookup:: pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346… Concepts -------- A dependency specification always specifies a distribution name. It may include extras, which expand the dependencies of the named distribution to enable optional features. The version installed can be controlled using version limits, or giving the URL to a specific artifact to install. Finally the dependency can be made conditional using environment markers. Grammar ------- We first cover the grammar briefly and the drill into the semantics of each section later. A distribution specification is written in ASCII text. We use ABNF [#abnf]_ to provide a precise grammar. Specifications may have comments starting with a '#' and running to the end of the line:: comment = "#" *(WSP / VCHAR) Specifications may be spread across multiple lines if desired using continuations - a single backslash followed by a new line ('\\\\n'):: CSP = 1*(WSP / ("\\" LF)) Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note: URI is defined in std-66 [#std66]_:: version-cmp = "<" / "<=" / "!=" / "==" / ">=" / ">" / "~=" / "===" version = 1*( DIGIT / ALPHA / "-" / "_" / "." / "*" ) versionspec = version-cmp version *(',' version-cmp version) urlspec = "@" URI Environment markers allow making a specification only take effect in some environments:: marker-op = version-cmp / "in" / "not in" python-str-c = (WSP / ALPHA / DIGIT / "(" / ")" / "." / "{" / "}" / "-" / "_" / "*" python-str = "'" *(python-str-c / DQUOTE) "'" python-str =/ DQUOTE *(python-str-c / "'") DQUOTE marker-var = python-str / "python_version" / "python_full_version" / "os_name"" / "sys_platform" / "platform_release" / "platform_version" / "platform_machine" / "platform_python_implementation" / "implementation_name" / "implementation_version" / "platform_dist_name" "platform_dist_version" / "platform_dist_id" marker-expr = "(" marker ")" / (marker-var [marker-op marker-var]) marker = marker-expr *( ("and" / "or") marker-expr) name-marker = ";" *CSP marker url-marker = ";" 1*CSP marker Optional components of a distribution may be specified using the extras field:: identifier = 1*( DIGIT / ALPHA / "-" / "_" ) name = identifier extras = "[" identifier *("," identifier) "]" Giving us a rule for name based requirements:: name_req = name [CSP extras] [CSP versionspec] [CSP name-marker] And a rule for direct reference specifications:: url_req = name [CSP extras] urlspec [CSP url-marker] Leading to the unified rule that can specify a dependency:: specification = (name_req / location_req) [CSP comment] Whitespace ---------- Non line-breaking whitespace is optional and has no semantic meaning. A line break indicates the end of a specification. Specifications can be continued across multiple lines using a continuation. Comments -------- A specification can have a comment added to it by starting the comment with a "#". After a "#" the rest of the line can contain any text whatsoever. Continuations within a comment are ignored. Names ----- Python distribution names are currently defined in PEP-345 [#pep345]_. Names act as the primary identifier for distributions. They are present in all dependency specifications, and are sufficient to be a specification on their own. Extras ------ An extra is an optional part of a distribution. Distributions can specify as many extras as they wish, and each extra results in the declaration of additional dependencies of the distribution **when** the extra is used in a dependency specification. For instance:: requests[security] Extras union in the dependencies they define with the dependencies of the distribution they are attached to. The example above would result in requests being installed, and requests own dependencies, and also any dependencies that are listed in the "security" extra of requests. If multiple extras are listed, all the dependencies are unioned together. Versions -------- See PEP-440 [#pep440]_ for more detail on both version numbers and version comparisons. Version specifications limit the versions of a distribution that can be used. They only apply to distributions looked up by name, rather than via a URL. Version comparison are also used in the markers feature. Environment Markers ------------------- Environment markers allow a dependency specification to provide a rule that describes when the dependency should be used. For instance, consider a package that needs argparse. In Python 2.7 argparse is always present. On older Python versions it has to be installed as a dependency. This can be expressed as so:: argparse;python_version<"2.7" A marker expression evalutes to either True or False. When it evaluates to False, the dependency specification should be ignored. The marker language is a subset of Python itself, chosen for the ability to safely evaluate it without running arbitrary code that could become a security vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This PEP fixes some issues that were observed in the described in PEP-426 [#pep426]_. Comparisons in marker expressions are typed by the comparison operator. The <marker-op> operators that are not in <version-cmp> perform the same as they do for strings in Python. The <version-cmp> operators use the PEP-440 [#pep440]_ version comparison rules if both sides are valid versions. If either side is not a valid version, then the comparsion falls back to the same behaviour as in for string in Python if the operator exists in Python. For those operators which are not defined in Python, the result should be False. The variables in the marker grammar such as "os_name" resolve to values looked up in the Python runtime. If a particular value is not available (such as ``sys.implementation.name`` in versions of Python prior to 3.3, or ``platform.dist()`` on non-Linux systems), the default value will be used. .. list-table:: :header-rows: 1 * - Marker - Python equivalent - Sample values - Default if unavailable * - ``os_name`` - ``os.name`` - ``posix``, ``java`` - "" * - ``sys_platform`` - ``sys.platform`` - ``linux``, ``darwin``, ``java1.8.0_51`` - "" * - ``platform_release`` - ``platform.release()`` - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51`` - "" * - ``platform_machine`` - ``platform.machine()`` - ``x86_64`` - "" * - ``platform_python_implementation`` - ``platform.python_implementation()`` - ``CPython``, ``Jython`` - "" * - ``implementation_name`` - ``sys.implementation.name`` - ``cpython`` - "" * - ``platform_version`` - ``platform.version()`` - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014`` ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation`` ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015; root:xnu-2782.40.9~2/RELEASE_X86_64`` - "" * - ``platform_dist_name`` - ``platform.dist()[0]`` - ``Ubuntu`` - "" * - ``platform_dist_version`` - ``platform.dist()[1]`` - ``14.04`` - "" * - ``platform_dist_id`` - ``platform.dist()[2]`` - ``trusty`` - "" * - ``python_version`` - ``platform.python_version()[:3]`` - ``3.4``, ``2.7`` - "0" * - ``python_full_version`` - see definition below - ``3.4.0``, ``3.5.0b1`` - "0" * - ``implementation_version`` - see definition below - ``3.4.0``, ``3.5.0b1`` - "0" The ``python_full_version`` and ``implementation_version`` marker variables are derived from ``sys.version_info`` and ``sys.implementation.version`` respectively, in accordance with the following algorithm:: def format_full_version(info): version = '{0.major}.{0.minor}.{0.micro}'.format(info) kind = info.releaselevel if kind != 'final': version += kind[0] + str(info.serial) return version python_full_version = format_full_version(sys.version_info) implementation_version = format_full_version(sys.implementation.version) ``python_full_version`` will typically correspond to ``sys.version.split()[0]``. If a particular version number value is not available (such as ``sys.implementation.version`` in versions of Python prior to 3.3) the corresponding marker variable returned by setuptools will be set to ``0`` Backwards Compatibility ======================= Most of this PEP is already widely deployed and thus offers no compatibiltiy concerns. There are however two key points where the PEP differs from the deployed base. Firstly, PEP-440 direct references haven't actually been deployed in the wild, but they were designed to be compatibly added, and there are no known obstacles to adding them to pip or other tools that consume the existing dependency metadata in distributions. Secondly, PEP-426 markers which have had some reasonable deployment, particularly in wheels and pip, will handle version comparisons with ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is less than "2.7.9". This backward incompatibility is deliberate. We are also defining new operators - "~=" and "===", and new variables - the ``platform_dist_*`` variables which are not present in older marker implementations. The variables will fall back to "" on those implementations, permitting reasonably graceful upgrade. The new version comparisons will cause errors, so adoption may require waiting some time for deployment to be widespread. Rationale ========= In order to move forward with any new PEPs that depend on environment markers, we needed a specification that included them. The requirement specifier EBNF is lifted from setuptools pkg_resources documentation, since we can't sensible depend on a defacto standard. References ========== .. [#pip] pip, the recommended installer for Python packages (http://pip.readthedocs.org/en/stable/) .. [#pep345] PEP-345, Python distribution metadata version 1.2. (https://www.python.org/dev/peps/pep-0345/) .. [#pep426] PEP-426, Python distribution metadata. (https://www.python.org/dev/peps/pep-0426/) .. [#pep440] PEP-440, Python distribution metadata. (https://www.python.org/dev/peps/pep-0440/) .. [#abnf] ABNF specification. (https://tools.ietf.org/html/rfc5234) .. [#std66] The URL specification. (https://tools.ietf.org/html/rfc3986) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

5 25

Current PyPI storage requirements?
by Nick Coghlan 16 Nov '15

16 Nov '15

This isn't an urgent question, but rather a "if the stats are readily available, I'm curious as to the answer" one: what are PyPi's current storage requirements? The warehouse.python.org front page indicates how many objects there are, but not the amount of space they take up. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 2

Version Spec compatible with PEP 0440 as well as SemVer
by Jamiel Almeida 15 Nov '15

15 Nov '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 # Version Spec compatible with PEP 0440 as well as SemVer ## Short Version/TL;DR I want to know if the following versions would abide by SemVer AND PEP 0440 as I would like both to be valid for my projects (order matters). The idea is to create a formal specification based on the Semantic Versioning Specification that is fully compatible with both Semantic Versioning and PEP 0440. As such, I'd like to know what you think. ### Examples: 0.0.0 # Before the first public release, local iterations and commits to the repository when a public API is still not present. 0.1.0 # First public release, since major = 0, public API is expected to be unstable 0.1.1 # First patch applied to the first public release 0.2.0 1.0.0-a0 # Alpha pre-release of version 1 (first stable release) 1.0.0-a1 1.0.0-b0 1.0.0-rc0 # First release candidate 1.0.0-rc1 1.0.0 1.0.1 1.0.2 ## Long version ### Intention I got very interested in versioning and format of public version identifiers a couple of days ago, and was studying different formats used by projects. I really liked the two "big" propositions I found, which are Semantic Versioning from semver.org (or "SemVer" from now on) and PEP 0440 (or "the PEP" from now on). I'm aware of the existence of forks to SemVer that "fix" it to be compatible with the PEP, e.g. OpenStack Foundation PBR's "Linux/Python Compatible Semantic Versioning 3.0.0" [1]. I'm also aware of discussions in mailing lists involving the "fixing" the PEP, e.g. "module version number support for semver.org" in python-ideas [2]. I have no intention to fix either, rather have a version specification for the projects I lead or my personal projects that is compatible with both. And by compatible with both I hope to have a simple spec that still provides the determinism in downloads/upgrades introduced by the PEP, even if that means forgoing "versatility" of some suffixes and identifiers, for simplicity. As well as providing a meaning and rationale for each element of a version-number/release (the "sem" in SemVer) Keywords are "determinism", "simplicity"; and hopefully not "bikeshedding", nor "trolling", nor "flame-baiting". ### Compatible bits I quote the section of the PEP that talks about compatibility with SemVer [3]: > The "Major.Minor.Patch" (described in this PEP as "major.minor.micro") > aspects of semantic versioning (clauses 1-9 in the 2.0.0-rc-1 > specification) are fully compatible with the version scheme defined in > this PEP, and abiding by these aspects is encouraged. > > Semantic versions containing a hyphen (pre-releases - clause 10) or a > plus sign (builds - clause 11) are not compatible with this PEP and > are not permitted in the public version field. > > One possible mechanism to translate such semantic versioning based > source labels to compatible public versions is to use the .devN suffix > to specify the appropriate version order. So I know the first four versions in my example above are compatible with both, as well as the last. The question arises when reading the above quote after having studied the prescriptions of the PEP in sections "Pre-releases" [4] and "Pre-release separators" [5] (under "Normalization") ### Pre-releases For the PEP, pre-releases are allowed to be separated by a dash from the release component described as "major.minor.patch" in Semantic Versioning, and as "major.minor.micro" in the PEP. So that part of clause 10 of version 2.0.0-rc-1 [6] (quoted below, as this was the one used when writing the PEP) on the Semantic Versioning format isn't the problem, where the problem lies is that SemVer allows that pre-release identifier to be multiple "sections" and to allow many more combinations of characters. > 10. A pre-release version MAY be denoted by appending a dash and a > series of dot separated identifiers immediately following the > patch version. Identifiers MUST be comprised of only ASCII > alphanumerics and dash [0-9A-Za-z-]. Pre-release versions satisfy > but have a lower precedence than the associated normal version. > Examples: 1.0.0-alpha, 1.0.0-alpha.1, 1.0.0-0.3.7, 1.0.0-x.7.z.92. Question #1: : If restricted to being one of "a", "b", or "rc" followed by a numeric component from 0-9, and also restricted to only one section, as shown on the examples I presented, would this be still incompatible with the PEP after being normalized? Question #2: : In the above-quoted version section of the PEP, pre-release versions aren't permitted to be part of the "public version field", what does this mean? I see that there are versions of software distributed using PyPI that have pre-release identifiers, e.g. Sphinx==1.3b3 [7] Semantically, a version "1.3b3" is the same as "1.3.0-b3" (an identifier compliant with SemVer, and the PEP after being normalized to "1.3.0.b3") But when saying "public version field", does the PEP prescribes that I "SHOULD NOT" use this identifier when publishing or what? Or was the intention that it not be part of the "final release" identifier (since it is after all a pre-release identifier). When I wrote the above examples, the spirit was to abide by SemVer (all clauses) while still being compatible with PEP 0440. The limitation of the numeric component to be [0-9] is to avoid sorting problems caused by a subtle difference in how these are sorted in SemVer vs The PEP. Since my examples use letters and numbers for this section identifier, SemVer prescribes lexical ordering by ASCII sort order, while the PEP compares same pre-release stage versions in numerical order. I.E. in SemVer "a1" < "a11" < "a2" as identifier components are expected to be separated by dots if they have different meaning. ### Aside. There was an update in this clause of SemVer from version 2.0.0-rc-1 as used in the PEP to version 2.0.0 (current), this update specified that the sections separated by dots must not be empty, must not include leading zeroes, and a small blurb on the meaning of this identifier. There was also a renumbering due to the deleting of a section before. These changes are irrelevant for this discussion. See [8] It's also important to note that I consider changes to documentation between minor releases a patch (SemVer) or (micro) in the PEP, and this number is updated when the code is "published" (using the definition in PEP 0426 [9]). This mechanism could fit into PEP 0440's concept of "post-release" but I'm choosing to forgo this concept to be able to stick to SemVer and not alter sorting order by using '-post#', '-r#', or '-rev#', which would force me to use '-c#' instead of '-rc#'. But it is possible and I might consider it in the future. Code pushed to the repository or merged from a pull-request doesn't fit into this definition of publication (the one used in PEP 0426). I consider part of this definition, however, the tagging a release in the version control repository, uploading to index servers, and in doing that providing the software to what PEP 0426 defines as "software integrators". As such, if after releasing a sample version "3.1.2", I were to push any number of commits and/or merge pull requests, this code or otherwise file changes are not considered to be published under the definition above, nor is it part of 3.1.2. Code is considered published and part of a version, when, as part of the development cycle; the version number on the repository is updated. This update would entail updating the version number in the code, and updating the changelog in the repository to the new version number and with all changes that have been added since last version was published". The change in version number and update to changelog should happen in the same commit, and this commit should not contain any other changes. This commit is then tagged with the version id, prepended with the character 'v'; and this new version is then to be uploaded to package indexes and sent to integrators. As such, and with the limitation of pre-release staged versions to only 10 (0-9) it is advised that the documentation be complete before pre-releasing in alpha, and only bugfixes be submitted (these may also be in the documentation) during a pre-release stage. The code and documentation is to be considered "frozen" for those pre-releases, and as such, changes should not be taken lightly and version bumps not be done trivially. End aside. ### Local versions / Build metadata In relation to what I call "+identifier"s, both the PEP and SemVer allow for "Local version identifier" or "build metadata" as they respectively refer to the section after the "public version identifier" in the PEP, and "normal version" in SemVer. This section is, on both documents, allowed to be included after a separator, specifically, the character "+" (plus sign). There are two differences, however. The first difference is syntactical, SemVer allows for letter, digit, and '-' (hyphen) characters in subsections separated by '.' (dot) in clause 10 of version 2.0.0 of the specification [8] (please don't confuse the clause with the previous #10, that was for the rc1 of the specification); while the PEP doesn't allow '-' (hyphen) on the segments as it calls them. The second difference is a semantic one, with an added "kicker". In SemVer, this section has no restrictions and is considered part of the public identifier. In contrast, it is prescribed in the PEP that this section of the identifier SHOULD NOT be used when publishing, and MAY be used to denote local versions built from source, and SHOULD be used by downstream projects when releasing a version of the upstream project (see section of the PEP about local version identifiers [10]). Also, please note that the PEP has specific meanings for "MAY" "SHOULD", and "SHOULD NOT". The "kicker" I mention, is that in SemVer two "build metadata" identifiers are not considered when determining precedence, while in the PEP they do. In more human terms, according to the PEP, my project should not use the "+identifier" for it's own releases, downstream projects can when they release a new version of my project that is compatible as defined in both SemVer and the PEP. And as long as those downstream projects don't use '-' (hyphen) these versions released by them would still abide by both SemVer and the PEP. [1]: http://docs.openstack.org/developer/pbr/semver.html [2]: http://grokbase.com/t/python/python-ideas/144e5x67tq/module-version-number-… [3]: https://www.python.org/dev/peps/pep-0440/#semantic-versioning [4]: https://www.python.org/dev/peps/pep-0440/#pre-releases [5]: https://www.python.org/dev/peps/pep-0440/#pre-release-separators [6]: http://semver.org/spec/v2.0.0-rc.1.html [7]: https://pypi.python.org/pypi/Sphinx/1.3b3 [8]: http://semver.org/spec/v2.0.0.html [9]: https://www.python.org/dev/peps/pep-0426/ [10]: https://www.python.org/dev/peps/pep-0440/#local-version-identifiers - -- Jamiel Almeida <jamiel.almeida(a)gmail.com> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.29 (Darwin) iQIcBAEBCgAGBQJWSAAOAAoJEHY8IXipLI+0uUwP/RLnCk62QAjBpmMsC/O5lk3c FqL5fD5ZaDjX5paxy3aoKU83UY5D5KTWiMeBbpZ2ae6nf00UP5hKEjoi3eioQn6Q I8ELE1vHXP2X9s+iJRzzbO6zMVtYQ/OHWtJq/YaGeOQdtN8mfl4e6esyMK6Tldl6 TstKLEPMQXL5zxgXPzcudvYW3wj7MXa1Pn9xcY06LgO5ERtLM4JCk2PJMzbZ8R8r gS8+B5UbVLuu9aE5n9ZjDbP1A07jKSSmwFdoxatvG176Vp7ha4xl80Kc0liYl9ke L1Zn3whpCBn7C5amO2RzmklniOgs56MCURo4wyqe+4V1egl9yio298YvrAvDRT4d TEftliEsErsCQjDpPutkYYgynARktU/xQHRZUw7cXNeqNfTuzA7ZaDuFgO3X6fDf A7Bd3BbMq9h3FsMTGq7Tb2F6cF8W4wVASTZTb3endRWN9X+khVr6Bkvhl9qr5cWh XG3xXBhdAku9vUaqxEXVyUlKoFytt3UwFQpJgvgB/eKffVCcXa+e3RSsNaN5p5dO yxCuqblHHjF9JHfEEybG3IWZzL3eyBgMPEGLC9LtHCrGwKXh0m+oQU8UOYrSDld7 RfcdoYzbrZ0k+rdiKPXiOx59vB3FWKKfZAeGempE6efhYlYAKsVpQ31S1sxQZH9w 6VBZmyFa+fGGYVfgToQs =VeL1 -----END PGP SIGNATURE-----

1 0

The future of invoking pip
by Donald Stufft 13 Nov '15

13 Nov '15

There is currently a semi related set of problems that I'd really like to figure out an answer too so we can begin to work on a migration path and close these out. This is dealing with a fairly fundamental aspect of pip so I'm bringing it up here to try and get wider discussion than the issue tracker or pypa-dev list. Currently pip installs a number of commands like ``pip``, ``pipX`` and ``pipX.Y`` where the X and X.Y corresponds to the version of Python that pip is installed into. Pip installs into whatever Python is currently executing it so this gives some ability to control which version of Python you're installing into (``pip2.7`` for Python 2.7 etc). However, this has a few problems: * It really only works if you only have one version of Python installed for each X.Y series, as soon as you have multiple versions in one series (including alternative implementations besides CPython) you run into a problem of which Python 2.7 is pip2.7 for and how do you invoke it for the *other* Python 2.7's. Something like pip2.7.8 is really ugly, and still doesn't solve the problem when you have multiple 2.7.8 installs and pip-pypy2-4.0 is even uglier. * The above gets *really* confusing when ``pipX`` or ``pip`` do not agree with what ``pythonX`` and ``python`` point to. * Having pip need to be installed into each Python means you end up with a bunch of independent pip installations which all need to be independently updated. We've made this better by having recent pips warn you if you're not on the latest version, but it's not unusual for me personally to have 30+ different installations of pip on my personal desktop. That isn't great. * Having ``pip`` on Windows requires us to create a "script wrapper" which is just a shim .exe that just executes a python script. This is due to the fact that Windows special cases .exe in places. This causes problems because you can't actually do ``pip install --upgrade pip`` on Windows because you'll get an error trying to update the script wrapper, you need to do ``py -m pip install --upgrade pip`` so the .exe file isn't currently open. One possible solution to the above problems is to try and move away from using ``pip``, ``pipX`` and ``pipX.Y`` and instead push people (and possibly deprecate ``pip`` &etc) towards using ``python -m pip`` instead. This makes it unambigious which python you're modifying and completely removes all of the confusion around that. However this too has problems: * There is a lot of documentation out there in many projects that tell people to use ``pip install ...``, the long tail on getting people moved to this will be very long. * It's more to type, 10 more characters on *nix and 6 more characters on Windows which makes it more akward and annoying to use. This is particularly annoying inside of a virtual environment where there isn't really any ambiguity when one is activated. * It still has the annoyance around having multiple pip installs all over the place and needing to manage those. * We still support Python 2.6 which doesn't support executing a package only modules via ``-m``. So we'll break Python 2.6 unless people do ``python -m pip.__main__`` or we move pip/* to _pip/* and make a pip.py which will break things for people using pip as a library (which isn't currently supported). Another possible option is to modify pip so that instead of installing into site-packages we instead create an "executable zip file" which is a simple zip file that has all of pip inside of it and a top level __main__.py. Python can directly execute this as if it were installed. We would no longer have any command except for ``pip`` (which is really this executable zip file). This script would default to installing into ``python``, and you can direct it to install into a different python using a (new) flag of ``-p`` or ``--python`` where you can specify either a full path or a command that we'll search $PATH for. * We'll break the usage of ``python -m pip``. * We'll break the usage of ``pipX`` and ``pipX.Y``. * We'll break the usage of anyone using pip as a library (but this is not actually supported). * We still can't do ``pip install --upgrade pip`` by default, and since there's no ability to do ``py -m pip install --upgrade pip`` we'll need to figure something out (perhaps skip updating the .exe? I don't know). The first option is the status quo and thus doesn't represent any *new* breakage but has all of the problems we have currently. The second option uses the ``-m`` operater but has problems with 2.6 and it's not as nice to type but it removes all the ambiguity. The third option introduces the most breakage, but it removes all the amiguity *and* it removes the need to manage a number of pip installations. A side conversation here is that pip currently bundles a number of dependencies inside of itself because it can't really have any dependencies. This works fine but it's a bit of an annoyance to maintain. The larger problem is that a number of downstreams don't like this and have attempted to route around it to varying degrees of success. One benefit of the third option is that we can remove the need to directly copy the bundled libraries into the pip source code and we can install just bundle it inside the built zip file. Thoughts? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

25 83

"option --single-version-externally-managed not recognized" when pip install mysql-connector-python-rf
by Chris Withers 13 Nov '15

13 Nov '15

Hi All, My Travis nightly builds of a package started failing this morning with the following: https://travis-ci.org/Mortar/mortar_rdb/jobs/90919221 pip install mysql-connector-python-rf You are using pip version 6.0.7, however version 7.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Collecting mysql-connector-python-rf Downloading mysql-connector-python-rf-2.1.3.tar.gz (271kB) 100% |################################| 274kB 2.1MB/s Installing collected packages: mysql-connector-python-rf Running setup.py install for mysql-connector-python-rf usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: option --single-version-externally-managed not recognized Complete output from command /home/travis/virtualenv/python3.4.2/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-_f5a57lj/mysql-connector-python-rf/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-jd_zku67-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/travis/virtualenv/python3.4.2/include/site/python3.4: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: option --single-version-externally-managed not recognized mysql-connector-python-rf hasn't changed in ages, and I haven't pushed any commits to mortar_rdb in some time. Has pip or setuptools changed in the last 24 hrs? This something that might have changed on TravisCI? cheers, Chris

2 2

How to exclude a directory from a module that has tests?
by Nagy László Zsolt 13 Nov '15

13 Nov '15

Hello, My source distribution contains a single module. It is **not** a package. I have used this: setup(name='yubistorm', version=__version__, description='Asnychronous two factor authentication client for YubiCloud with Tornado', long_description="""Provides a simple module that can be used from a Tornado server to authenticate users """ + """ with YubiCloud (https://www.yubico.com/products/services-software/yubicloud/)""", author='László Zsolt Nagy', author_email='nagylzs(a)gmail.com', license="LGPL v3", py_modules=['yubistorm'], requires=['tornado (>=4.3)'], url="https://bitbucket.org/nagylzs/yubistorm", classifiers=[ 'Topic :: Security', 'Topic :: Internet :: WWW/HTTP', "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: CPython", ], ) The problem is that there is a "test" directory and it is added to the source distribution. I want to exclude that. I know that there is an "exclude" parameter for find_packages(). But this is not a package. This is a single module. How do I exclude a directory then? Thanks, Laszlo

2 1

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
by Nathaniel Smith 13 Nov '15

13 Nov '15

[Adding distutils-sig to the CC as a heads-up. The context is that numpy is looking at deprecating the use of 'python setup.py install' and enforcing the use of 'pip install .' instead, and running into some issues that will probably need to be addressed if 'pip install .' is going to become the standard interface to work with source trees.] On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: [...] > Hmm, after some more testing I'm going to have to bring up a few concerns > myself: > > 1. ``pip install .`` still has a clear bug; it starts by copying everything > (including .git/ !) to a tempdir with shutil, which is very slow. And the > fix for that will go via ``setup.py sdist``, which is still slow. Ugh. If 'pip (install/wheel) .' is supposed to become the standard way to build things, then it should probably build in-place by default. Working in a temp dir makes perfect sense for 'pip install <requirement>' or 'pip install <url>', but if the user supplies an actual named on-disk directory then presumably the user is expecting this directory to be used, and to be able to take advantage of incremental rebuilds etc., no? > 2. ``pip install .`` silences build output, which may make sense for some > usecases, but for numpy it just sits there for minutes with no output after > printing "Running setup.py install for numpy". Users will think it hangs and > Ctrl-C it. https://github.com/pypa/pip/issues/2732 I tend to agree with the commentary there that for end users this is different but no worse than the current situation where we spit out pages of "errors" that don't mean anything :-). I posted a suggestion on that bug that might help with the apparent hanging problem. > 3. ``pip install .`` refuses to upgrade an already installed development > version. For released versions that makes sense, but if I'm in a git tree > then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal > to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1). Ugh, this is clearly just a bug -- `pip install .` should always unconditionally install, IMO. (Did you file a bug yet?) At least the workaround is just 'pip uninstall numpy; pip install .', which is still better the running 'setup.py install' and having it blithely overwrite some files and not others. The first and last issue seem like ones that will mostly only affect developers, who should mostly have the ability to deal with these weird issues (or just use setup.py install --force if that's what they prefer)? This still seems like a reasonable trade-off to me if it also has the effect of reducing the number of weird broken installs among our thousands-of-times-larger userbase. -n -- Nathaniel J. Smith -- http://vorpus.org

6 44