[Distutils] FINAL DRAFT: Dependency specifier PEP
Marcus Smith
qwcode at gmail.com
Wed Nov 18 11:44:35 EST 2015
as it is, this PEP defers the concept of a "Direct Reference URL" to PEP440.
but then PEP440 partially defers to PEP426's "source_url" concept, when it
says "a direct URL reference may be a valid source_url entry"
do we expect PEP440 to be updated to fully own what a "Direct Reference
URL" can be?, since referring to PEP426 is now a dead-end path (and
partially replaced by this PEP)
On Mon, Nov 16, 2015 at 12:46 PM, Robert Collins <robertc at robertcollins.net>
wrote:
> :PEP: XX
> :Title: Dependency specification for Python Software Packages
> :Version: $Revision$
> :Last-Modified: $Date$
> :Author: Robert Collins <rbtcollins at hp.com>
> :BDFL-Delegate: Donald Stufft <donald at stufft.io>
> :Discussions-To: distutils-sig <distutils-sig at python.org>
> :Status: Draft
> :Type: Standards Track
> :Content-Type: text/x-rst
> :Created: 11-Nov-2015
> :Post-History: XX
>
>
> Abstract
> ========
>
> This PEP specifies the language used to describe dependencies for packages.
> It draws a border at the edge of describing a single dependency - the
> different sorts of dependencies and when they should be installed is a
> higher
> level problem. The intent is provide a building block for higher layer
> specifications.
>
> The job of a dependency is to enable tools like pip [#pip]_ to find the
> right
> package to install. Sometimes this is very loose - just specifying a name,
> and
> sometimes very specific - referring to a specific file to install.
> Sometimes
> dependencies are only relevant in one platform, or only some versions are
> acceptable, so the language permits describing all these cases.
>
> The language defined is a compact line based format which is already in
> widespread use in pip requirements files, though we do not specify the
> command
> line option handling that those files permit. There is one caveat - the
> URL reference form, specified in PEP-440 [#pep440]_ is not actually
> implemented in pip, but since PEP-440 is accepted, we use that format
> rather
> than pip's current native format.
>
> Motivation
> ==========
>
> Any specification in the Python packaging ecosystem that needs to consume
> lists of dependencies needs to build on an approved PEP for such, but
> PEP-426 [#pep426]_ is mostly aspirational - and there are already existing
> implementations of the dependency specification which we can instead adopt.
> The existing implementations are battle proven and user friendly, so
> adopting
> them is arguably much better than approving an aspirational, unconsumed,
> format.
>
> Specification
> =============
>
> Examples
> --------
>
> All features of the language shown with a name based lookup::
>
> requests [security,tests] >= 2.8.1, == 2.8.* ; python_version <
> "2.7.10"
>
> A minimal URL based lookup::
>
> pip @
> https://github.com/pypa/pip/archive/1.3.1.zip#sha1=da9234ee9982d4bbb3c72346a6de940a148ea686
>
> Concepts
> --------
>
> A dependency specification always specifies a distribution name. It may
> include extras, which expand the dependencies of the named distribution to
> enable optional features. The version installed can be controlled using
> version limits, or giving the URL to a specific artifact to install.
> Finally
> the dependency can be made conditional using environment markers.
>
> Grammar
> -------
>
> We first cover the grammar briefly and then drill into the semantics of
> each
> section later.
>
> A distribution specification is written in ASCII text. We use a parsley
> [#parsley]_ grammar to provide a precise grammar. It is expected that the
> specification will be embedded into a larger system which offers framing
> such
> as comments, multiple line support via continuations, or other such
> features.
>
> The full grammar including annotations to build a useful parse tree is
> included at the end of the PEP.
>
> Versions may be specified according to the PEP-440 [#pep440]_ rules. (Note:
> URI is defined in std-66 [#std66]_::
>
> version_cmp = wsp* '<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' |
> '==='
> version = wsp* ( letterOrDigit | '-' | '_' | '.' | '*' )+
> version_one = version_cmp version wsp*
> version_many = version_one (wsp* ',' version_one)*
> versionspec = ( '(' version_many ')' ) | version_many
> urlspec = '@' wsp* <URI_reference>
>
> Environment markers allow making a specification only take effect in some
> environments::
>
> marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
> '-' | '_' | '*')
> dquote = '"'
> squote = '\\''
> python_str = (squote (python_str_c | dquote)* squote |
> dquote (python_str_c | squote)* dquote)
> env_var = ('python_version' | 'python_full_version' |
> 'os_name' | 'sys_platform' | 'platform_release' |
> 'platform_system' | 'platform_version' |
> 'platform_machine' | 'python_implementation' |
> 'implementation_name' | 'implementation_version' |
> 'extra' # ONLY when defined by a containing layer
> )
> marker_var = env_var | python_str
> marker_expr = ('(' wsp* marker wsp* ')'
> | (marker_var wsp* marker_op wsp* marker_var))
> marker = wsp* marker_expr ( wsp* ('and' | 'or') wsp*
> marker_expr)*
> quoted_marker = ';' wsp* marker
>
> Optional components of a distribution may be specified using the extras
> field::
>
> identifier = letterOrDigit (
> letterOrDigit |
> (( letterOrDigit | '-' | '_' | '.')* letterOrDigit ) )*
> name = identifier
> extras_list = identifier (wsp* ',' wsp* identifier)*
> extras = '[' wsp* extras_list? wsp* ']'
>
> Giving us a rule for name based requirements::
>
> name_req = name wsp* extras? wsp* versionspec? wsp* quoted_marker?
>
> And a rule for direct reference specifications::
>
> url_req = name wsp* extras? wsp* urlspec wsp+ quoted_marker?
>
> Leading to the unified rule that can specify a dependency.::
>
> specification = wsp* ( url_req | name_req ) wsp*
>
> Whitespace
> ----------
>
> Non line-breaking whitespace is mostly optional with no semantic meaning.
> The
> sole exception is detecting the end of a URL requirement.
>
> Names
> -----
>
> Python distribution names are currently defined in PEP-345 [#pep345]_.
> Names
> act as the primary identifier for distributions. They are present in all
> dependency specifications, and are sufficient to be a specification on
> their
> own. However, PyPI places strict restrictions on names - they must match a
> case insensitive regex or they won't be accepted. Accordingly in this PEP
> we
> limit the acceptable values for identifiers to that regex. A full
> redefinition
> of name may take place in a future metadata PEP::
>
> ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$
>
> Extras
> ------
>
> An extra is an optional part of a distribution. Distributions can specify
> as
> many extras as they wish, and each extra results in the declaration of
> additional dependencies of the distribution **when** the extra is used in a
> dependency specification. For instance::
>
> requests[security]
>
> Extras union in the dependencies they define with the dependencies of the
> distribution they are attached to. The example above would result in
> requests
> being installed, and requests own dependencies, and also any dependencies
> that
> are listed in the "security" extra of requests.
>
> If multiple extras are listed, all the dependencies are unioned together.
>
> Versions
> --------
>
> See PEP-440 [#pep440]_ for more detail on both version numbers and version
> comparisons. Version specifications limit the versions of a distribution
> that
> can be used. They only apply to distributions looked up by name, rather
> than
> via a URL. Version comparison are also used in the markers feature. The
> optional brackets around a version are present for compatibility with
> PEP-345
> [#pep345]_ but should not be generated, only accepted.
>
> Environment Markers
> -------------------
>
> Environment markers allow a dependency specification to provide a rule that
> describes when the dependency should be used. For instance, consider a
> package
> that needs argparse. In Python 2.7 argparse is always present. On older
> Python
> versions it has to be installed as a dependency. This can be expressed as
> so::
>
> argparse;python_version<"2.7"
>
> A marker expression evalutes to either True or False. When it evaluates to
> False, the dependency specification should be ignored.
>
> The marker language is a subset of Python itself, chosen for the ability to
> safely evaluate it without running arbitrary code that could become a
> security
> vulnerability. Markers were first standardised in PEP-345 [#pep345]_. This
> PEP
> fixes some issues that were observed in the design described in PEP-426
> [#pep426]_.
>
> Comparisons in marker expressions are typed by the comparison operator.
> The
> <marker_op> operators that are not in <version_cmp> perform the same as
> they
> do for strings in Python. The <version_cmp> operators use the PEP-440
> [#pep440]_ version comparison rules when those are defined (that is when
> both
> sides have a valid version specifier). If there is no defined PEP-440
> behaviour and the operator exists in Python, then the operator falls back
> to
> the Python behaviour. Otherwise an error should be raised. e.g. the
> following
> will result in errors::
>
> "dog" ~= "fred"
> python_version ~= "surprise"
>
> User supplied constants are always encoded as strings with either ``'`` or
> ``"`` quote marks. Note that backslash escapes are not defined, but
> existing
> implementations do support them. They are not included in this
> specification because they add complexity and there is no observable need
> for
> them today. Similarly we do not define non-ASCII character support: all the
> runtime variables we are referencing are expected to be ASCII-only.
>
> The variables in the marker grammar such as "os_name" resolve to values
> looked
> up in the Python runtime. With the exception of "extra" all values are
> defined
> on all Python versions today - it is an error in the implementation of
> markers
> if a value is not defined.
>
> Unknown variables must raise an error rather than resulting in a comparison
> that evaluates to True or False.
>
> Variables whose value cannot be calculated on a given Python implementation
> should evaluate to ``0`` for versions, and an empty string for all other
> variables.
>
> The "extra" variable is special. It is used by wheels to signal which
> specifications apply to a given extra in the wheel ``METADATA`` file, but
> since the ``METADATA`` file is based on a draft version of PEP-426, there
> is
> no current specification for this. Regardless, outside of a context where
> this
> special handling is taking place, the "extra" variable should result in an
> error like all other unknown variables.
>
> .. list-table::
> :header-rows: 1
>
> * - Marker
> - Python equivalent
> - Sample values
> * - ``os_name``
> - ``os.name``
> - ``posix``, ``java``
> * - ``sys_platform``
> - ``sys.platform``
> - ``linux``, ``linux2``, ``darwin``, ``java1.8.0_51`` (note that
> "linux"
> is from Python3 and "linux2" from Python2)
> * - ``platform_machine``
> - ``platform.machine()``
> - ``x86_64``
> * - ``python_implementation``
> - ``platform.python_implementation()``
> - ``CPython``, ``Jython``
> * - ``platform_release``
> - ``platform.release()``
> - ``3.14.1-x86_64-linode39``, ``14.5.0``, ``1.8.0_51``
> * - ``platform_system``
> - ``platform.system()``
> - ``Linux``, ``Windows``, ``Java``
> * - ``platform_version``
> - ``platform.version()``
> - ``#1 SMP Fri Apr 25 13:07:35 EDT 2014``
> ``Java HotSpot(TM) 64-Bit Server VM, 25.51-b03, Oracle Corporation``
> ``Darwin Kernel Version 14.5.0: Wed Jul 29 02:18:53 PDT 2015;
> root:xnu-2782.40.9~2/RELEASE_X86_64``
> * - ``python_version``
> - ``platform.python_version()[:3]``
> - ``3.4``, ``2.7``
> * - ``python_full_version``
> - ``platform.python_version()``
> - ``3.4.0``, ``3.5.0b1``
> * - ``implementation_name``
> - ``sys.implementation.name``
> - ``cpython``
> * - ``implementation_version``
> - see definition below
> - ``3.4.0``, ``3.5.0b1``
> * - ``extra``
> - An error except when defined by the context interpreting the
> specification.
> - ``test``
>
> The ``implementation_version`` marker variable is derived from
> ``sys.implementation.version``::
>
> def format_full_version(info):
> version = '{0.major}.{0.minor}.{0.micro}'.format(info)
> kind = info.releaselevel
> if kind != 'final':
> version += kind[0] + str(info.serial)
> return version
>
> if hasattr(sys, 'implementation'):
> implementation_version =
> format_full_version(sys.implementation.version)
> else:
> implementation_version = "0"
>
> Backwards Compatibility
> =======================
>
> Most of this PEP is already widely deployed and thus offers no
> compatibiltiy
> concerns.
>
> There are however a few points where the PEP differs from the deployed
> base.
>
> Firstly, PEP-440 direct references haven't actually been deployed in the
> wild,
> but they were designed to be compatibly added, and there are no known
> obstacles to adding them to pip or other tools that consume the existing
> dependency metadata in distributions - particularly since they won't be
> permitted to be present in PyPI uploaded distributions anyway.
>
> Secondly, PEP-426 markers which have had some reasonable deployment,
> particularly in wheels and pip, will handle version comparisons with
> ``python_version`` "2.7.10" differently. Specifically in 426 "2.7.10" is
> less
> than "2.7.9". This backward incompatibility is deliberate. We are also
> defining new operators - "~=" and "===", and new variables -
> ``platform_release``, ``platform_system``, ``implementation_name``, and
> ``implementation_version`` which are not present in older marker
> implementations. The variables will error on those implementations. Users
> of
> both features will need to make a judgement as to when support has become
> sufficiently widespread in the ecosystem that using them will not cause
> compatibility issues.
>
> Thirdly, PEP-345 required brackets around version specifiers. In order to
> accept PEP-345 dependency specifications, brackets are accepted, but they
> should not be generated.
>
> Rationale
> =========
>
> In order to move forward with any new PEPs that depend on environment
> markers,
> we needed a specification that included them in their modern form. This PEP
> brings together all the currently unspecified components into a specified
> form.
>
> The requirement specifier was adopted from the EBNF in the setuptools
> pkg_resources documentation, since we wish to avoid depending on a
> defacto, vs
> PEP specified, standard.
>
> Complete Grammar
> ================
>
> The complete parsley grammar::
>
> wsp = ' ' | '\t'
> version_cmp = wsp* <'<' | '<=' | '!=' | '==' | '>=' | '>' | '~=' |
> '==='>
> version = wsp* <( letterOrDigit | '-' | '_' | '.' | '*' |
> '+' | '!' )+>
> version_one = version_cmp:op version:v wsp* -> (op, v)
> version_many = version_one:v1 (wsp* ',' version_one)*:v2 -> [v1] + v2
> versionspec = ('(' version_many:v ')' ->v) | version_many
> urlspec = '@' wsp* <URI_reference>
> marker_op = version_cmp | 'in' | 'not' wsp+ 'in'
> python_str_c = (wsp | letter | digit | '(' | ')' | '.' | '{' | '}' |
> '-' | '_' | '*' | '#')
> dquote = '"'
> squote = '\\''
> python_str = (squote <(python_str_c | dquote)*>:s squote |
> dquote <(python_str_c | squote)*>:s dquote) -> s
> env_var = ('python_version' | 'python_full_version' |
> 'os_name' | 'sys_platform' | 'platform_release' |
> 'platform_system' | 'platform_version' |
> 'platform_machine' | 'python_implementation' |
> 'implementation_name' | 'implementation_version' |
> 'extra' # ONLY when defined by a containing layer
> ):varname -> lookup(varname)
> marker_var = env_var | python_str
> marker_expr = (("(" wsp* marker:m wsp* ")" -> m)
> | ((marker_var:l wsp* marker_op:o wsp*
> marker_var:r))
> -> (l, o, r))
> marker = (wsp* marker_expr:m ( wsp* ("and" | "or"):o wsp*
> marker_expr:r -> (o, r))*:ms -> (m, ms))
> quoted_marker = ';' wsp* marker
> identifier = <letterOrDigit (
> letterOrDigit |
> (( letterOrDigit | '-' | '_' | '.')* letterOrDigit )
> )*>
> name = identifier
> extras_list = identifier:i (wsp* ',' wsp* identifier)*:ids -> [i] +
> ids
> extras = '[' wsp* extras_list?:e wsp* ']' -> e
> name_req = (name:n wsp* extras?:e wsp* versionspec?:v wsp*
> quoted_marker?:m
> -> (n, e or [], v or [], m))
> url_req = (name:n wsp* extras?:e wsp* urlspec:v wsp+
> quoted_marker?:m
> -> (n, e or [], v, m))
> specification = wsp* ( url_req | name_req ):s wsp* -> s
> # The result is a tuple - name, list-of-extras,
> # list-of-version-constraints-or-a-url, marker-ast or None
>
>
> URI_reference = <URI | relative_ref>
> URI = scheme ':' hier_part ('?' query )? ( '#' fragment)?
> hier_part = ('//' authority path_abempty) | path_absolute |
> path_rootless | path_empty
> absolute_URI = scheme ':' hier_part ( '?' query )?
> relative_ref = relative_part ( '?' query )? ( '#' fragment )?
> relative_part = '//' authority path_abempty | path_absolute |
> path_noscheme | path_empty
> scheme = letter ( letter | digit | '+' | '-' | '.')*
> authority = ( userinfo '@' )? host ( ':' port )?
> userinfo = ( unreserved | pct_encoded | sub_delims | ':')*
> host = IP_literal | IPv4address | reg_name
> port = digit*
> IP_literal = '[' ( IPv6address | IPvFuture) ']'
> IPvFuture = 'v' hexdig+ '.' ( unreserved | sub_delims | ':')+
> IPv6address = (
> ( h16 ':'){6} ls32
> | '::' ( h16 ':'){5} ls32
> | ( h16 )? '::' ( h16 ':'){4} ls32
> | ( ( h16 ':')? h16 )? '::' ( h16 ':'){3} ls32
> | ( ( h16 ':'){0,2} h16 )? '::' ( h16 ':'){2} ls32
> | ( ( h16 ':'){0,3} h16 )? '::' h16 ':' ls32
> | ( ( h16 ':'){0,4} h16 )? '::' ls32
> | ( ( h16 ':'){0,5} h16 )? '::' h16
> | ( ( h16 ':'){0,6} h16 )? '::' )
> h16 = hexdig{1,4}
> ls32 = ( h16 ':' h16) | IPv4address
> IPv4address = dec_octet '.' dec_octet '.' dec_octet '.' Dec_octet
> nz = ~'0' digit
> dec_octet = (
> digit # 0-9
> | nz digit # 10-99
> | '1' digit{2} # 100-199
> | '2' ('0' | '1' | '2' | '3' | '4') digit # 200-249
> | '25' ('0' | '1' | '2' | '3' | '4' | '5') )#
> %250-255
> reg_name = ( unreserved | pct_encoded | sub_delims)*
> path = (
> path_abempty # begins with '/' or is empty
> | path_absolute # begins with '/' but not '//'
> | path_noscheme # begins with a non-colon segment
> | path_rootless # begins with a segment
> | path_empty ) # zero characters
> path_abempty = ( '/' segment)*
> path_absolute = '/' ( segment_nz ( '/' segment)* )?
> path_noscheme = segment_nz_nc ( '/' segment)*
> path_rootless = segment_nz ( '/' segment)*
> path_empty = pchar{0}
> segment = pchar*
> segment_nz = pchar+
> segment_nz_nc = ( unreserved | pct_encoded | sub_delims | '@')+
> # non-zero-length segment without any colon ':'
> pchar = unreserved | pct_encoded | sub_delims | ':' | '@'
> query = ( pchar | '/' | '?')*
> fragment = ( pchar | '/' | '?')*
> pct_encoded = '%' hexdig
> unreserved = letter | digit | '-' | '.' | '_' | '~'
> reserved = gen_delims | sub_delims
> gen_delims = ':' | '/' | '?' | '#' | '(' | ')?' | '@'
> sub_delims = '!' | '$' | '&' | '\\'' | '(' | ')' | '*' | '+' |
> ',' | ';' | '='
> hexdig = digit | 'a' | 'A' | 'b' | 'B' | 'c' | 'C' | 'd' |
> 'D' | 'e' | 'E' | 'f' | 'F'
>
> A test program - if the grammar is in a string ``grammar``::
>
> import os
> import sys
> import platform
>
> from parsley import makeGrammar
>
> grammar = """
> wsp ...
> """
> tests = [
> "A",
> "aa",
> "name",
> "name>=3",
> "name>=3,<2",
> "name [fred,bar] @ http://foo.com ; python_version=='2.7'",
> "name[quux, strange];python_version<'2.7' and
> platform_version=='2'",
> "name; os_name=='dud' and (os_name=='odd' or os_name=='fred')",
> "name; os_name=='dud' and os_name=='odd' or os_name=='fred'",
> ]
>
> def format_full_version(info):
> version = '{0.major}.{0.minor}.{0.micro}'.format(info)
> kind = info.releaselevel
> if kind != 'final':
> version += kind[0] + str(info.serial)
> return version
>
> if hasattr(sys, 'implementation'):
> implementation_version =
> format_full_version(sys.implementation.version)
> implementation_name = sys.implementation.name
> else:
> implementation_version = '0'
> implementation_name = ''
> bindings = {
> 'implementation_name': implementation_name,
> 'implementation_version': implementation_version,
> 'os_name': os.name,
> 'platform_machine': platform.machine(),
> 'platform_release': platform.release(),
> 'platform_system': platform.system(),
> 'platform_version': platform.version(),
> 'python_full_version': platform.python_version(),
> 'python_implementation': platform.python_implementation(),
> 'python_version': platform.python_version()[:3],
> 'sys_platform': sys.platform,
> }
>
> compiled = makeGrammar(grammar, {'lookup': bindings.__getitem__})
> for test in tests:
> parsed = compiled(test).specification()
> print(parsed)
>
> References
> ==========
>
> .. [#pip] pip, the recommended installer for Python packages
> (http://pip.readthedocs.org/en/stable/)
>
> .. [#pep345] PEP-345, Python distribution metadata version 1.2.
> (https://www.python.org/dev/peps/pep-0345/)
>
> .. [#pep426] PEP-426, Python distribution metadata.
> (https://www.python.org/dev/peps/pep-0426/)
>
> .. [#pep440] PEP-440, Python distribution metadata.
> (https://www.python.org/dev/peps/pep-0440/)
>
> .. [#std66] The URL specification.
> (https://tools.ietf.org/html/rfc3986)
>
> .. [#parsley] The parsley PEG library.
> (https://pypi.python.org/pypi/parsley/)
>
> Copyright
> =========
>
> This document has been placed in the public domain.
>
>
>
> ..
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
> End:
>
>
> --
> Robert Collins <rbtcollins at hp.com>
> Distinguished Technologist
> HP Converged Cloud
> _______________________________________________
> Distutils-SIG maillist - Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151118/c0f429d7/attachment-0001.html>
More information about the Distutils-SIG
mailing list