[Distutils] FINAL DRAFT: Dependency specifier PEP
Donald Stufft
donald at stufft.io
Sun Nov 22 19:45:50 EST 2015
Okay. I’ve read over this, implemented enough of it, and I think it’s gone through enough nit picking. I’m going to go ahead and accept this PEP. It’s largely just standardizing what we are already doing so it’s pretty low impact other than fixing up a few issues and giving implementations something they can point at for what the standard behavior is.
So congratulations to everyone working on this :)
I’ll get this into the PEPs repo and get it pushed.
> On Nov 16, 2015, at 3:46 PM, Robert Collins <robertc at robertcollins.net> wrote:
>
> :PEP: XX
> :Title: Dependency specification for Python Software Packages
> :Version: $Revision$
> :Last-Modified: $Date$
> :Author: Robert Collins <rbtcollins at hp.com>
> :BDFL-Delegate: Donald Stufft <donald at stufft.io>
> :Discussions-To: distutils-sig <distutils-sig at python.org>
> :Status: Draft
> :Type: Standards Track
> :Content-Type: text/x-rst
> :Created: 11-Nov-2015
> :Post-History: XX
>
>
> Abstract
> ========
>
> This PEP specifies the language used to describe dependencies for packages.
> It draws a border at the edge of describing a single dependency - the
> different sorts of dependencies and when they should be installed is a higher
> level problem. The intent is provide a building block for higher layer
> specifications.
>
> The job of a dependency is to enable tools like pip [#pip]_ to find the right
> package to install. Sometimes this is very loose - just specifying a name, and
> sometimes very specific - referring to a specific file to install. Sometimes
> dependencies are only relevant in one platform, or only some versions are
> acceptable, so the language permits describing all these cases.
>
> The language defined is a compact line based format which is already in
> widespread use in pip requirements files, though we do not specify the command
> line option handling that those files permit. There is one caveat - the
> URL reference form, specified in PEP-440 [#pep440]_ is not actually
> implemented in pip, but since PEP-440 is accepted, we use that format rather
> than pip's current native format.
>
> Motivation
> ==========
>
> Any specification in the Python packaging ecosystem that needs to consume
> lists of dependencies needs to build on an approved PEP for such, but
> PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
> implementations of the dependency specification which we can instead adopt.
> The existing implementations are battle proven and user friendly, so adopting
> them is arguably much better than approving an aspirational, unconsumed, format.
>
> Specification
> =============
>
> Examples
> --------
>
> All features of the language shown with a name based lookup::
>
> requests [security,tests] >= 2.8.1, == 2.8.* ; python_version < "2.7.10"
>
> A minimal URL based lookup::
>
> pip @ https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
>
> Concepts
> --------
>
> A dependency specification always specifies a distribution name. It may
> include extras, which expand the dependencies of the named distribution to
> enable optional features. The version installed can be controlled using
> version limits, or giving the URL to a specific artifact to install. Finally
> the dependency can be made conditional using environment markers.
>
> Grammar
> -------
>
> We first cover the grammar briefly and then drill into the semantics of each
> section later.
>
> A distribution specification is written in ASCII text. We use a parsley
> [#parsley]_ grammar to provide a precise grammar. It is expected that the
> specification will be embedded into a larger system which offers framing such
> as comments, multiple line support via continuations, or other such features.
>
> The full grammar including annotations to build a useful parse tree is
> included at the end of the PEP.
>
> Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note:
> URI is defined in std-66 [#std66]_::
>
> version_cmp = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='
> version = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
> version_one = version_cmp version wsp*
> version_many = version_one (wsp* ',' version_one)*
> versionspec = ( '(' version_many ')' ) | version_many
> urlspec = '@' wsp* <URI_reference>
>
> Environment markers allow making a specification only take effect in some
> environments::
>
> marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
> '-' | '_' | '*')
> dquote = '"'
> squote = '\\''
> python_str = (squote (python_str_c | dquote)* squote |
> dquote (python_str_c | squote)* dquote)
> env_var = ('python_version' | 'python_full_version' |
> 'os_name' | 'sys_platform' | 'platform_release' |
> 'platform_system' | 'platform_version' |
> 'platform_machine' | 'python_implementation' |
> 'implementation_name' | 'implementation_version' |
> 'extra' # ONLY when defined by a containing layer
> )
> marker_var = env_var | python_str
> marker_expr = ('(' wsp* marker wsp* ')'
> | (marker_var wsp* marker_op wsp* marker_var))
> marker = wsp* marker_expr ( wsp* ('and' | 'or') wsp* marker_expr)*
> quoted_marker = ';' wsp* marker
>
> Optional components of a distribution may be specified using the extras
> field::
>
> identifier = letterOrDigit (
> letterOrDigit |
> (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )*
> name = identifier
> extras_list = identifier (wsp* ',' wsp* identifier)*
> extras = '[' wsp* extras_list? wsp* ']'
>
> Giving us a rule for name based requirements::
>
> name_req = name wsp* extras? wsp* versionspec? wsp* quoted_marker?
>
> And a rule for direct reference specifications::
>
> url_req = name wsp* extras? wsp* urlspec wsp+ quoted_marker?
>
> Leading to the unified rule that can specify a dependency.::
>
> specification = wsp* ( url_req | name_req ) wsp*
>
> Whitespace
> ----------
>
> Non line-breaking whitespace is mostly optional with no semantic meaning. The
> sole exception is detecting the end of a URL requirement.
>
> Names
> -----
>
> Python distribution names are currently defined in PEP-345 [#pep345]_. Names
> act as the primary identifier for distributions. They are present in all
> dependency specifications, and are sufficient to be a specification on their
> own. However, PyPI places strict restrictions on names - they must match a
> case insensitive regex or they won't be accepted. Accordingly in this PEP we
> limit the acceptable values for identifiers to that regex. A full redefinition
> of name may take place in a future metadata PEP::
>
> ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$
>
> Extras
> ------
>
> An extra is an optional part of a distribution. Distributions can specify as
> many extras as they wish, and each extra results in the declaration of
> additional dependencies of the distribution **when** the extra is used in a
> dependency specification. For instance::
>
> requests[security]
>
> Extras union in the dependencies they define with the dependencies of the
> distribution they are attached to. The example above would result in requests
> being installed, and requests own dependencies, and also any dependencies that
> are listed in the "security" extra of requests.
>
> If multiple extras are listed, all the dependencies are unioned together.
>
> Versions
> --------
>
> See PEP-440 [#pep440]_ for more detail on both version numbers and version
> comparisons. Version specifications limit the versions of a distribution that
> can be used. They only apply to distributions looked up by name, rather than
> via a URL. Version comparison are also used in the markers feature. The
> optional brackets around a version are present for compatibility with PEP-345
> [#pep345]_ but should not be generated, only accepted.
>
> Environment Markers
> -------------------
>
> Environment markers allow a dependency specification to provide a rule that
> describes when the dependency should be used. For instance, consider a package
> that needs argparse. In Python 2.7 argparse is always present. On older Python
> versions it has to be installed as a dependency. This can be expressed as so::
>
> argparse;python_version<"2.7"
>
> A marker expression evalutes to either True or False. When it evaluates to
> False, the dependency specification should be ignored.
>
> The marker language is a subset of Python itself, chosen for the ability to
> safely evaluate it without running arbitrary code that could become a security
> vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This PEP
> fixes some issues that were observed in the design described in PEP-426
> [#pep426]_.
>
> Comparisons in marker expressions are typed by the comparison operator. The
> <marker_op> operators that are not in <version_cmp> perform the same as they
> do for strings in Python. The <version_cmp> operators use the PEP-440
> [#pep440]_ version comparison rules when those are defined (that is when both
> sides have a valid version specifier). If there is no defined PEP-440
> behaviour and the operator exists in Python, then the operator falls back to
> the Python behaviour. Otherwise an error should be raised. e.g. the following
> will result in errors::
>
> "dog" ~= "fred"
> python_version ~= "surprise"
>
> User supplied constants are always encoded as strings with either ``'`` or
> ``"`` quote marks. Note that backslash escapes are not defined, but existing
> implementations do support them. They are not included in this
> specification because they add complexity and there is no observable need for
> them today. Similarly we do not define non-ASCII character support: all the
> runtime variables we are referencing are expected to be ASCII-only.
>
> The variables in the marker grammar such as "os_name" resolve to values looked
> up in the Python runtime. With the exception of "extra" all values are defined
> on all Python versions today - it is an error in the implementation of markers
> if a value is not defined.
>
> Unknown variables must raise an error rather than resulting in a comparison
> that evaluates to True or False.
>
> Variables whose value cannot be calculated on a given Python implementation
> should evaluate to ``0`` for versions, and an empty string for all other
> variables.
>
> The "extra" variable is special. It is used by wheels to signal which
> specifications apply to a given extra in the wheel ``METADATA`` file, but
> since the ``METADATA`` file is based on a draft version of PEP-426, there is
> no current specification for this. Regardless, outside of a context where this
> special handling is taking place, the "extra" variable should result in an
> error like all other unknown variables.
>
> .. list-table::
> :header-rows: 1
>
> * - Marker
> - Python equivalent
> - Sample values
> * - ``os_name``
> - ``os.name``
> - ``posix``, ``java``
> * - ``sys_platform``
> - ``sys.platform``
> - ``linux``, ``linux2``, ``darwin``, ``java1.8.0_51`` (note that "linux"
> is from Python3 and "linux2" from Python2)
> * - ``platform_machine``
> - ``platform.machine()``
> - ``x86_64``
> * - ``python_implementation``
> - ``platform.python_implementation()``
> - ``CPython``, ``Jython``
> * - ``platform_release``
> - ``platform.release()``
> - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51``
> * - ``platform_system``
> - ``platform.system()``
> - ``Linux``, ``Windows``, ``Java``
> * - ``platform_version``
> - ``platform.version()``
> - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014``
> ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation``
> ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015;
> root:xnu-2782.40.9~2/RELEASE_X86_64``
> * - ``python_version``
> - ``platform.python_version()[:3]``
> - ``3.4``, ``2.7``
> * - ``python_full_version``
> - ``platform.python_version()``
> - ``3.4.0``, ``3.5.0b1``
> * - ``implementation_name``
> - ``sys.implementation.name``
> - ``cpython``
> * - ``implementation_version``
> - see definition below
> - ``3.4.0``, ``3.5.0b1``
> * - ``extra``
> - An error except when defined by the context interpreting the
> specification.
> - ``test``
>
> The ``implementation_version`` marker variable is derived from
> ``sys.implementation.version``::
>
> def format_full_version(info):
> version = '{0.major}.{0.minor}.{0.micro}'.format(info)
> kind = info.releaselevel
> if kind != 'final':
> version += kind[0] + str(info.serial)
> return version
>
> if hasattr(sys, 'implementation'):
> implementation_version = format_full_version(sys.implementation.version)
> else:
> implementation_version = "0"
>
> Backwards Compatibility
> =======================
>
> Most of this PEP is already widely deployed and thus offers no compatibiltiy
> concerns.
>
> There are however a few points where the PEP differs from the deployed base.
>
> Firstly, PEP-440 direct references haven't actually been deployed in the wild,
> but they were designed to be compatibly added, and there are no known
> obstacles to adding them to pip or other tools that consume the existing
> dependency metadata in distributions - particularly since they won't be
> permitted to be present in PyPI uploaded distributions anyway.
>
> Secondly, PEP-426 markers which have had some reasonable deployment,
> particularly in wheels and pip, will handle version comparisons with
> ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is less
> than "2.7.9". This backward incompatibility is deliberate. We are also
> defining new operators - "~=" and "===", and new variables -
> ``platform_release``, ``platform_system``, ``implementation_name``, and
> ``implementation_version`` which are not present in older marker
> implementations. The variables will error on those implementations. Users of
> both features will need to make a judgement as to when support has become
> sufficiently widespread in the ecosystem that using them will not cause
> compatibility issues.
>
> Thirdly, PEP-345 required brackets around version specifiers. In order to
> accept PEP-345 dependency specifications, brackets are accepted, but they
> should not be generated.
>
> Rationale
> =========
>
> In order to move forward with any new PEPs that depend on environment markers,
> we needed a specification that included them in their modern form. This PEP
> brings together all the currently unspecified components into a specified
> form.
>
> The requirement specifier was adopted from the EBNF in the setuptools
> pkg_resources documentation, since we wish to avoid depending on a defacto, vs
> PEP specified, standard.
>
> Complete Grammar
> ================
>
> The complete parsley grammar::
>
> wsp = ' ' | '\t'
> version_cmp = wsp* <'<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' | '==='>
> version = wsp* <( letterOrDigit | '-' | '_' | '.' | '*' |
> '+' | '!' )+>
> version_one = version_cmp:op version:v wsp* -> (op, v)
> version_many = version_one:v1 (wsp* ',' version_one)*:v2 -> [v1] + v2
> versionspec = ('(' version_many:v ')' ->v) | version_many
> urlspec = '@' wsp* <URI_reference>
> marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
> '-' | '_' | '*' | '#')
> dquote = '"'
> squote = '\\''
> python_str = (squote <(python_str_c | dquote)*>:s squote |
> dquote <(python_str_c | squote)*>:s dquote) -> s
> env_var = ('python_version' | 'python_full_version' |
> 'os_name' | 'sys_platform' | 'platform_release' |
> 'platform_system' | 'platform_version' |
> 'platform_machine' | 'python_implementation' |
> 'implementation_name' | 'implementation_version' |
> 'extra' # ONLY when defined by a containing layer
> ):varname -> lookup(varname)
> marker_var = env_var | python_str
> marker_expr = (("(" wsp* marker:m wsp* ")" -> m)
> | ((marker_var:l wsp* marker_op:o wsp* marker_var:r))
> -> (l, o, r))
> marker = (wsp* marker_expr:m ( wsp* ("and" | "or"):o wsp*
> marker_expr:r -> (o, r))*:ms -> (m, ms))
> quoted_marker = ';' wsp* marker
> identifier = <letterOrDigit (
> letterOrDigit |
> (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )*>
> name = identifier
> extras_list = identifier:i (wsp* ',' wsp* identifier)*:ids -> [i] + ids
> extras = '[' wsp* extras_list?:e wsp* ']' -> e
> name_req = (name:n wsp* extras?:e wsp* versionspec?:v wsp*
> quoted_marker?:m
> -> (n, e or [], v or [], m))
> url_req = (name:n wsp* extras?:e wsp* urlspec:v wsp+ quoted_marker?:m
> -> (n, e or [], v, m))
> specification = wsp* ( url_req | name_req ):s wsp* -> s
> # The result is a tuple - name, list-of-extras,
> # list-of-version-constraints-or-a-url, marker-ast or None
>
>
> URI_reference = <URI | relative_ref>
> URI = scheme ':' hier_part ('?' query )? ( '#' fragment)?
> hier_part = ('//' authority path_abempty) | path_absolute |
> path_rootless | path_empty
> absolute_URI = scheme ':' hier_part ( '?' query )?
> relative_ref = relative_part ( '?' query )? ( '#' fragment )?
> relative_part = '//' authority path_abempty | path_absolute |
> path_noscheme | path_empty
> scheme = letter ( letter | digit | '+' | '-' | '.')*
> authority = ( userinfo '@' )? host ( ':' port )?
> userinfo = ( unreserved | pct_encoded | sub_delims | ':')*
> host = IP_literal | IPv4address | reg_name
> port = digit*
> IP_literal = '[' ( IPv6address | IPvFuture) ']'
> IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+
> IPv6address = (
> ( h16 ':'){6} ls32
> | '::' ( h16 ':'){5} ls32
> | ( h16 )? '::' ( h16 ':'){4} ls32
> | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32
> | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32
> | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32
> | ( ( h16 ':'){0,4} h16 )? '::' ls32
> | ( ( h16 ':'){0,5} h16 )? '::' h16
> | ( ( h16 ':'){0,6} h16 )? '::' )
> h16 = hexdig{1,4}
> ls32 = ( h16 ':' h16) | IPv4address
> IPv4address = dec_octet '.' dec_octet '.' dec_octet '.' Dec_octet
> nz = ~'0' digit
> dec_octet = (
> digit # 0-9
> | nz digit # 10-99
> | '1' digit{2} # 100-199
> | '2' ('0' | '1' | '2' | '3' | '4') digit # 200-249
> | '25' ('0' | '1' | '2' | '3' | '4' | '5') )# %250-255
> reg_name = ( unreserved | pct_encoded | sub_delims)*
> path = (
> path_abempty # begins with '/' or is empty
> | path_absolute # begins with '/' but not '//'
> | path_noscheme # begins with a non-colon segment
> | path_rootless # begins with a segment
> | path_empty ) # zero characters
> path_abempty = ( '/' segment)*
> path_absolute = '/' ( segment_nz ( '/' segment)* )?
> path_noscheme = segment_nz_nc ( '/' segment)*
> path_rootless = segment_nz ( '/' segment)*
> path_empty = pchar{0}
> segment = pchar*
> segment_nz = pchar+
> segment_nz_nc = ( unreserved | pct_encoded | sub_delims | '@')+
> # non-zero-length segment without any colon ':'
> pchar = unreserved | pct_encoded | sub_delims | ':' | '@'
> query = ( pchar | '/' | '?')*
> fragment = ( pchar | '/' | '?')*
> pct_encoded = '%' hexdig
> unreserved = letter | digit | '-' | '.' | '_' | '~'
> reserved = gen_delims | sub_delims
> gen_delims = ':' | '/' | '?' | '#' | '(' | ')?' | '@'
> sub_delims = '!' | '$' | '&' | '\\'' | '(' | ')' | '*' | '+' |
> ',' | ';' | '='
> hexdig = digit | 'a' | 'A' | 'b' | 'B' | 'c' | 'C' | 'd' |
> 'D' | 'e' | 'E' | 'f' | 'F'
>
> A test program - if the grammar is in a string ``grammar``::
>
> import os
> import sys
> import platform
>
> from parsley import makeGrammar
>
> grammar = """
> wsp ...
> """
> tests = [
> "A",
> "aa",
> "name",
> "name>=3",
> "name>=3,<2",
> "name [fred,bar] @ http://foo.com ; python_version=='2.7'",
> "name[quux, strange];python_version<'2.7' and platform_version=='2'",
> "name; os_name=='dud' and (os_name=='odd' or os_name=='fred')",
> "name; os_name=='dud' and os_name=='odd' or os_name=='fred'",
> ]
>
> def format_full_version(info):
> version = '{0.major}.{0.minor}.{0.micro}'.format(info)
> kind = info.releaselevel
> if kind != 'final':
> version += kind[0] + str(info.serial)
> return version
>
> if hasattr(sys, 'implementation'):
> implementation_version = format_full_version(sys.implementation.version)
> implementation_name = sys.implementation.name
> else:
> implementation_version = '0'
> implementation_name = ''
> bindings = {
> 'implementation_name': implementation_name,
> 'implementation_version': implementation_version,
> 'os_name': os.name,
> 'platform_machine': platform.machine(),
> 'platform_release': platform.release(),
> 'platform_system': platform.system(),
> 'platform_version': platform.version(),
> 'python_full_version': platform.python_version(),
> 'python_implementation': platform.python_implementation(),
> 'python_version': platform.python_version()[:3],
> 'sys_platform': sys.platform,
> }
>
> compiled = makeGrammar(grammar, {'lookup': bindings.__getitem__})
> for test in tests:
> parsed = compiled(test).specification()
> print(parsed)
>
> References
> ==========
>
> .. [#pip] pip, the recommended installer for Python packages
> (http://pip.readthedocs.org/en/stable/)
>
> .. [#pep345] PEP-345, Python distribution metadata version 1.2.
> (https://www.python.org/dev/peps/pep-0345/)
>
> .. [#pep426] PEP-426, Python distribution metadata.
> (https://www.python.org/dev/peps/pep-0426/)
>
> .. [#pep440] PEP-440, Python distribution metadata.
> (https://www.python.org/dev/peps/pep-0440/)
>
> .. [#std66] The URL specification.
> (https://tools.ietf.org/html/rfc3986)
>
> .. [#parsley] The parsley PEG library.
> (https://pypi.python.org/pypi/parsley/)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
>
> ..
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
> End:
>
>
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151122/801b4bef/attachment-0001.sig>
More information about the Distutils-SIG
mailing list