[Python-checkins] peps: Apply PEP 502 changes from Mike Miller
chris.angelico
python-checkins at python.org
Tue Sep 15 03:37:33 CEST 2015
https://hg.python.org/peps/rev/1bee90a93fa2
changeset: 6060:1bee90a93fa2
user: Chris Angelico <rosuav at gmail.com>
date: Tue Sep 15 11:37:22 2015 +1000
summary:
Apply PEP 502 changes from Mike Miller
files:
pep-0502.txt | 453 ++++++++++++++------------------------
1 files changed, 163 insertions(+), 290 deletions(-)
diff --git a/pep-0502.txt b/pep-0502.txt
--- a/pep-0502.txt
+++ b/pep-0502.txt
@@ -1,44 +1,46 @@
PEP: 502
-Title: String Interpolation Redux
+Title: String Interpolation - Extended Discussion
Version: $Revision$
Last-Modified: $Date$
Author: Mike G. Miller
Status: Draft
-Type: Standards Track
+Type: Informational
Content-Type: text/x-rst
Created: 10-Aug-2015
Python-Version: 3.6
-Note: Open issues below are stated with a question mark (?),
-and are therefore searchable.
-
Abstract
========
-This proposal describes a new string interpolation feature for Python,
-called an *expression-string*,
-that is both concise and powerful,
-improves readability in most cases,
-yet does not conflict with existing code.
+PEP 498: *Literal String Interpolation*, which proposed "formatted strings" was
+accepted September 9th, 2015.
+Additional background and rationale given during its design phase is detailed
+below.
-To achieve this end,
-a new string prefix is introduced,
-which expands at compile-time into an equivalent expression-string object,
-with requested variables from its context passed as keyword arguments.
+To recap that PEP,
+a string prefix was introduced that marks the string as a template to be
+rendered.
+These formatted strings may contain one or more expressions
+built on `the existing syntax`_ of ``str.format()``.
+The formatted string expands at compile-time into a conventional string format
+operation,
+with the given expressions from its text extracted and passed instead as
+positional arguments.
+
At runtime,
-the new object uses these passed values to render a string to given
-specifications, building on `the existing syntax`_ of ``str.format()``::
+the resulting expressions are evaluated to render a string to given
+specifications::
>>> location = 'World'
- >>> e'Hello, {location} !' # new prefix: e''
- 'Hello, World !' # interpolated result
+ >>> f'Hello, {location} !' # new prefix: f''
+ 'Hello, World !' # interpolated result
+
+Format-strings may be thought of as merely syntactic sugar to simplify traditional
+calls to ``str.format()``.
.. _the existing syntax: https://docs.python.org/3/library/string.html#format-string-syntax
-This PEP does not recommend to remove or deprecate any of the existing string
-formatting mechanisms.
-
Motivation
==========
@@ -50,12 +52,16 @@
with similar use cases,
the amount of code necessary to build similar strings is substantially higher,
while at times offering lower readability due to verbosity, dense syntax,
-or identifier duplication. [1]_
+or identifier duplication.
+
+These difficulties are described at moderate length in the original
+`post to python-ideas`_
+that started the snowball (that became PEP 498) rolling. [1]_
Furthermore, replacement of the print statement with the more consistent print
function of Python 3 (PEP 3105) has added one additional minor burden,
an additional set of parentheses to type and read.
-Combined with the verbosity of current formatting solutions,
+Combined with the verbosity of current string formatting solutions,
this puts an otherwise simple language at an unfortunate disadvantage to its
peers::
@@ -66,7 +72,7 @@
# Python 3, str.format with named parameters
print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(**locals()))
- # Python 3, variation B, worst case
+ # Python 3, worst case
print('Hello, user: {user}, id: {id}, on host: {hostname}'.format(user=user,
id=id,
hostname=
@@ -74,7 +80,7 @@
In Python, the formatting and printing of a string with multiple variables in a
single line of code of standard width is noticeably harder and more verbose,
-indentation often exacerbating the issue.
+with indentation exacerbating the issue.
For use cases such as smaller projects, systems programming,
shell script replacements, and even one-liners,
@@ -82,36 +88,17 @@
this verbosity has likely lead a significant number of developers and
administrators to choose other languages over the years.
+.. _post to python-ideas: https://mail.python.org/pipermail/python-ideas/2015-July/034659.html
+
Rationale
=========
-Naming
-------
-
-The term expression-string was chosen because other applicable terms,
-such as format-string and template are already well used in the Python standard
-library.
-
-The string prefix itself, ``e''`` was chosen to demonstrate that the
-specification enables expressions,
-is not limited to ``str.format()`` syntax,
-and also does not lend itself to `the shorthand term`_ "f-string".
-It is also slightly easier to type than other choices such as ``_''`` and
-``i''``,
-while perhaps `less odd-looking`_ to C-developers.
-``printf('')`` vs. ``print(f'')``.
-
-.. _the shorthand term: reference_needed
-.. _less odd-looking: https://mail.python.org/pipermail/python-dev/2015-August/141147.html
-
-
-
Goals
-------------
-The design goals of expression-strings are as follows:
+The design goals of format strings are as follows:
#. Eliminate need to pass variables manually.
#. Eliminate repetition of identifiers and redundant parentheses.
@@ -133,40 +120,44 @@
characters to enclose strings.
It is not reasonable to choose one of them now to enable interpolation,
while leaving the other for uninterpolated strings.
-"Backtick" characters (`````) are also `constrained by history`_ as a shortcut
-for ``repr()``.
+Other characters,
+such as the "Backtick" (or grave accent `````) are also
+`constrained by history`_
+as a shortcut for ``repr()``.
This leaves a few remaining options for the design of such a feature:
* An operator, as in printf-style string formatting via ``%``.
* A class, such as ``string.Template()``.
-* A function, such as ``str.format()``.
-* New syntax
+* A method or function, such as ``str.format()``.
+* New syntax, or
* A new string prefix marker, such as the well-known ``r''`` or ``u''``.
-The first three options above currently work well.
+The first three options above are mature.
Each has specific use cases and drawbacks,
yet also suffer from the verbosity and visual noise mentioned previously.
-All are discussed in the next section.
+All options are discussed in the next sections.
.. _constrained by history: https://mail.python.org/pipermail/python-ideas/2007-January/000054.html
+
Background
-------------
-This proposal builds on several existing techniques and proposals and what
+Formatted strings build on several existing techniques and proposals and what
we've collectively learned from them.
+In keeping with the design goals of readability and error-prevention,
+the following examples therefore use named,
+not positional arguments.
-The following examples focus on the design goals of readability and
-error-prevention using named parameters.
Let's assume we have the following dictionary,
and would like to print out its items as an informative string for end users::
>>> params = {'user': 'nobody', 'id': 9, 'hostname': 'darkstar'}
-Printf-style formatting
-'''''''''''''''''''''''
+Printf-style formatting, via operator
+'''''''''''''''''''''''''''''''''''''
This `venerable technique`_ continues to have its uses,
such as with byte-based protocols,
@@ -178,7 +169,7 @@
In this form, considering the prerequisite dictionary creation,
the technique is verbose, a tad noisy,
-and relatively readable.
+yet relatively readable.
Additional issues are that an operator can only take one argument besides the
original string,
meaning multiple parameters must be passed in a tuple or dictionary.
@@ -190,8 +181,8 @@
.. _venerable technique: https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
-string.Template
-'''''''''''''''
+string.Template Class
+'''''''''''''''''''''
The ``string.Template`` `class from`_ PEP 292
(Simpler String Substitutions)
@@ -202,7 +193,7 @@
Template('Hello, user: $user, id: ${id}, on host: $hostname').substitute(params)
-Also verbose, however the string itself is readable.
+While also verbose, the string itself is readable.
Though functionality is limited,
it meets its requirements well.
It isn't powerful enough for many cases,
@@ -232,8 +223,8 @@
It was superseded by the following proposal.
-str.format()
-''''''''''''
+str.format() Method
+'''''''''''''''''''
The ``str.format()`` `syntax of`_ PEP 3101 is the most recent and modern of the
existing options.
@@ -253,36 +244,32 @@
host=hostname)
'Hello, user: nobody, id: 9, on host: darkstar'
+The verbosity of the method-based approach is illustrated here.
+
.. _syntax of: https://docs.python.org/3/library/string.html#format-string-syntax
PEP 498 -- Literal String Formatting
''''''''''''''''''''''''''''''''''''
-PEP 498 discusses and delves partially into implementation details of
-expression-strings,
-which it calls f-strings,
-the idea and syntax
-(with exception of the prefix letter)
-of which is identical to that discussed here.
-The resulting compile-time transformation however
-returns a string joined from parts at runtime,
-rather than an object.
+PEP 498 defines and discusses format strings,
+as also described in the `Abstract`_ above.
-It also, somewhat controversially to those first exposed to it,
-introduces the idea that these strings shall be augmented with support for
-arbitrary expressions,
-which is discussed further in the following sections.
-
+It also, somewhat controversially to those first exposed,
+introduces the idea that format-strings shall be augmented with support for
+arbitrary expressions.
+This is discussed further in the
+Restricting Syntax section under
+`Rejected Ideas`_.
PEP 501 -- Translation ready string interpolation
'''''''''''''''''''''''''''''''''''''''''''''''''
The complimentary PEP 501 brings internationalization into the discussion as a
-first-class concern, with its proposal of i-strings,
+first-class concern, with its proposal of the i-prefix,
``string.Template`` syntax integration compatible with ES6 (Javascript),
deferred rendering,
-and a similar object return value.
+and an object return value.
Implementations in Other Languages
@@ -374,7 +361,8 @@
Designers of `Template strings`_ faced the same issue as Python where single
and double quotes were taken.
Unlike Python however, "backticks" were not.
-They were chosen as part of the ECMAScript 2015 (ES6) standard::
+Despite `their issues`_,
+they were chosen as part of the ECMAScript 2015 (ES6) standard::
console.log(`Fifteen is ${a + b} and\nnot ${2 * a + b}.`);
@@ -391,8 +379,10 @@
* User implemented prefixes supported.
* Arbitrary expressions are supported.
+.. _their issues: https://mail.python.org/pipermail/python-ideas/2007-January/000054.html
.. _Template strings: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/template_strings
+
C#, Version 6
'''''''''''''
@@ -428,13 +418,14 @@
Additional examples
'''''''''''''''''''
-A number of additional examples may be `found at Wikipedia`_.
+A number of additional examples of string interpolation may be
+`found at Wikipedia`_.
+
+Now that background and history have been covered,
+let's continue on for a solution.
.. _found at Wikipedia: https://en.wikipedia.org/wiki/String_interpolation#Examples
-Now that background and imlementation history have been covered,
-let's continue on for a solution.
-
New Syntax
----------
@@ -442,178 +433,47 @@
This should be an option of last resort,
as every new syntax feature has a cost in terms of real-estate in a brain it
inhabits.
-There is one alternative left on our list of possibilities,
+There is however one alternative left on our list of possibilities,
which follows.
New String Prefix
-----------------
-Given the history of string formatting in Python,
-backwards-compatibility,
+Given the history of string formatting in Python and backwards-compatibility,
implementations in other languages,
-and the avoidance of new syntax unless necessary,
+avoidance of new syntax unless necessary,
an acceptable design is reached through elimination
rather than unique insight.
-Therefore, we choose to explicitly mark interpolated string literals with a
-string prefix.
+Therefore, marking interpolated string literals with a string prefix is chosen.
-We also choose an expression syntax that reuses and builds on the strongest of
+We also choose an expression syntax that reuses and builds on the strongest of
the existing choices,
-``str.format()`` to avoid further duplication.
-
-
-Specification
-=============
-
-String literals with the prefix of ``e`` shall be converted at compile-time to
-the construction of an ``estr`` (perhaps ``types.ExpressionString``?) object.
-Strings and values are parsed from the literal and passed as tuples to the
-constructor::
+``str.format()`` to avoid further duplication of functionality::
>>> location = 'World'
- >>> e'Hello, {location} !'
+ >>> f'Hello, {location} !' # new prefix: f''
+ 'Hello, World !' # interpolated result
- # becomes
- # estr('Hello, {location} !', # template
- ('Hello, ', ' !'), # string fragments
- ('location',), # expressions
- ('World',), # values
- )
+PEP 498 -- Literal String Formatting, delves into the mechanics and
+implementation of this design.
-The object interpolates its result immediately at run-time::
- 'Hello, World !'
-
-
-ExpressionString Objects
-------------------------
-
-The ExpressionString object supports both immediate and deferred rendering of
-its given template and parameters.
-It does this by immediately rendering its inputs to its internal string and
-``.rendered`` string member (still necessary?),
-useful in the majority of use cases.
-To allow for deferred rendering and caller-specified escaping,
-all inputs are saved for later inspection,
-with convenience methods available.
-
-Notes:
-
-* Inputs are saved to the object as ``.template`` and ``.context`` members
- for later use.
-* No explicit ``str(estr)`` call is necessary to render the result,
- though doing so might be desired to free resources if significant.
-* Additional or deferred rendering is available through the ``.render()``
- method, which allows template and context to be overriden for flexibility.
-* Manual escaping of potentially dangerous input is available through the
- ``.escape(escape_function)`` method,
- the rules of which may therefore be specified by the caller.
- The given function should both accept and return a single modified string.
-
-* A sample Python implementation can `found at Bitbucket`_:
-
-.. _found at Bitbucket: https://bitbucket.org/mixmastamyk/docs/src/default/pep/estring_demo.py
-
-
-Inherits From ``str`` Type
-'''''''''''''''''''''''''''
-
-Inheriting from the ``str`` class is one of the techniques available to improve
-compatibility with code expecting a string object,
-as it will pass an ``isinstance(obj, str)`` test.
-ExpressionString implements this and also renders its result into the "raw"
-string of its string superclass,
-providing compatibility with a majority of code.
-
-
-Interpolation Syntax
---------------------
-
-The strongest of the existing string formatting syntaxes is chosen,
-``str.format()`` as a base to build on. [10]_ [11]_
-
-..
-
-* Additionally, single arbitrary expressions shall also be supported inside
- braces as an extension::
-
- >>> e'My age is {age + 1} years.'
-
- See below for section on safety.
-
-* Triple quoted strings with multiple lines shall be supported::
-
- >>> e'''Hello,
- {location} !'''
- 'Hello,\n World !'
-
-* Adjacent implicit concatenation shall be supported;
- interpolation does not `not bleed into`_ other strings::
-
- >>> 'Hello {1, 2, 3} ' e'{location} !'
- 'Hello {1, 2, 3} World !'
-
-* Additional implementation details,
- for example expression and error-handling,
- are specified in the compatible PEP 498.
-
-.. _not bleed into: https://mail.python.org/pipermail/python-ideas/2015-July/034763.html
-
-
-Composition with Other Prefixes
--------------------------------
-
-* Expression-strings apply to unicode objects only,
- therefore ``u''`` is never needed.
- Should it be prevented?
-
-* Bytes objects are not included here and do not compose with e'' as they
- do not support ``__format__()``.
-
-* Complimentary to raw strings,
- backslash codes shall not be converted in the expression-string,
- when combined with ``r''`` as ``re''``.
-
-
-Examples
---------
-
-A more complicated example follows::
-
- n = 5; # t0, t1 = … TODO
- a = e"Sliced {n} onions in {t1-t0:.3f} seconds."
- # returns the equvalent of
- estr("Sliced {n} onions in {t1-t0:.3f} seconds", # template
- ('Sliced ', ' onions in ', ' seconds'), # strings
- ('n', 't1-t0:.3f'), # expressions
- (5, 0.555555) # values
- )
-
-With expressions only::
-
- b = e"Three random numbers: {rand()}, {rand()}, {rand()}."
- # returns the equvalent of
- estr("Three random numbers: {rand():f}, {rand():f}, {rand():}.", # template
- ('Three random numbers: ', ', ', ', ', '.'), # strings
- ('rand():f', 'rand():f', 'rand():f'), # expressions
- (rand(), rand(), rand()) # values
- )
+Additional Topics
+=================
Safety
-----------
In this section we will describe the safety situation and precautions taken
-in support of expression-strings.
+in support of format-strings.
-#. Only string literals shall be considered here,
+#. Only string literals have been considered for format-strings,
not variables to be taken as input or passed around,
making external attacks difficult to accomplish.
- * ``str.format()`` `already handles`_ this use-case.
- * Direct instantiation of the ExpressionString object with non-literal input
- shall not be allowed. (Practicality?)
+ ``str.format()`` and alternatives `already handle`_ this use-case.
#. Neither ``locals()`` nor ``globals()`` are necessary nor used during the
transformation,
@@ -622,37 +482,72 @@
#. To eliminate complexity as well as ``RuntimeError`` (s) due to recursion
depth, recursive interpolation is not supported.
-#. Restricted characters or expression classes?, such as ``=`` for assignment.
-
However,
mistakes or malicious code could be missed inside string literals.
Though that can be said of code in general,
that these expressions are inside strings means they are a bit more likely
to be obscured.
-.. _already handles: https://mail.python.org/pipermail/python-ideas/2015-July/034729.html
+.. _already handle: https://mail.python.org/pipermail/python-ideas/2015-July/034729.html
-Mitigation via tools
+Mitigation via Tools
''''''''''''''''''''
The idea is that tools or linters such as pyflakes, pylint, or Pycharm,
-could check inside strings for constructs that exceed project policy.
-As this is a common task with languages these days,
-tools won't have to implement this feature solely for Python,
+may check inside strings with expressions and mark them up appropriately.
+As this is a common task with programming languages today,
+multi-language tools won't have to implement this feature solely for Python,
significantly shortening time to implementation.
-Additionally the Python interpreter could check(?) and warn with appropriate
-command-line parameters passed.
+Farther in the future,
+strings might also be checked for constructs that exceed the safety policy of
+a project.
+
+
+Style Guide/Precautions
+-----------------------
+
+As arbitrary expressions may accomplish anything a Python expression is
+able to,
+it is highly recommended to avoid constructs inside format-strings that could
+cause side effects.
+
+Further guidelines may be written once usage patterns and true problems are
+known.
+
+
+Reference Implementation(s)
+---------------------------
+
+The `say module on PyPI`_ implements string interpolation as described here
+with the small burden of a callable interface::
+
+ > pip install say
+
+ from say import say
+ nums = list(range(4))
+ say("Nums has {len(nums)} items: {nums}")
+
+A Python implementation of Ruby interpolation `is also available`_.
+It uses the codecs module to do its work::
+
+ > pip install interpy
+
+ # coding: interpy
+ location = 'World'
+ print("Hello #{location}.")
+
+.. _say module on PyPI: https://pypi.python.org/pypi/say/
+.. _is also available: https://github.com/syrusakbary/interpy
Backwards Compatibility
-----------------------
-By using existing syntax and avoiding use of current or historical features,
-expression-strings (and any associated sub-features),
-were designed so as to not interfere with existing code and is not expected
-to cause any issues.
+By using existing syntax and avoiding current or historical features,
+format strings were designed so as to not interfere with existing code and are
+not expected to cause any issues.
Postponed Ideas
@@ -666,20 +561,12 @@
the finer details diverge at almost every point,
making a common solution unlikely: [15]_
-* Use-cases
-* Compile and run-time tasks
-* Interpolation Syntax
+* Use-cases differ
+* Compile vs. run-time tasks
+* Interpolation syntax needs
* Intended audience
* Security policy
-Rather than try to fit a "square peg in a round hole,"
-this PEP attempts to allow internationalization to be supported in the future
-by not preventing it.
-In this proposal,
-expression-string inputs are saved for inspection and re-rendering at a later
-time,
-allowing for their use by an external library of any sort.
-
Rejected Ideas
--------------
@@ -687,17 +574,24 @@
Restricting Syntax to ``str.format()`` Only
'''''''''''''''''''''''''''''''''''''''''''
-This was deemed not enough of a solution to the problem.
+The common `arguments against`_ support of arbitrary expresssions were:
+
+#. `YAGNI`_, "You aren't gonna need it."
+#. The feature is not congruent with historical Python conservatism.
+#. Postpone - can implement in a future version if need is demonstrated.
+
+.. _YAGNI: https://en.wikipedia.org/wiki/You_aren't_gonna_need_it
+.. _arguments against: https://mail.python.org/pipermail/python-ideas/2015-August/034913.html
+
+Support of only ``str.format()`` syntax however,
+was deemed not enough of a solution to the problem.
+Often a simple length or increment of an object, for example,
+is desired before printing.
+
It can be seen in the `Implementations in Other Languages`_ section that the
developer community at large tends to agree.
-
-The common `arguments against`_ arbitrary expresssions were:
-
-#. YAGNI, "You ain't gonna need it."
-#. The change is not congruent with historical Python conservatism.
-#. Postpone - can implement in a future version if need is demonstrated.
-
-.. _arguments against: https://mail.python.org/pipermail/python-ideas/2015-August/034913.html
+String interpolation with arbitrary expresssions is becoming an industry
+standard in modern languages due to its utility.
Additional/Custom String-Prefixes
@@ -720,7 +614,7 @@
expressions could be used safely or not.
The concept was also difficult to describe to others. [12]_
-Always consider expression-string variables to be unescaped,
+Always consider format string variables to be unescaped,
unless the developer has explicitly escaped them.
@@ -735,33 +629,13 @@
which could encourage bad habits. [13]_
-Reference Implementation(s)
-===========================
-
-An expression-string implementation is currently attached to PEP 498,
-under the ``f''`` prefix,
-and may be available in nightly builds.
-
-A Python implementation of Ruby interpolation `is also available`_,
-which is similar to this proposal.
-It uses the codecs module to do its work::
-
- > pip install interpy
-
- # coding: interpy
- location = 'World'
- print("Hello #{location}.")
-
-.. _is also available: https://github.com/syrusakbary/interpy
-
-
Acknowledgements
================
-* Eric V. Smith for providing invaluable implementation work and design
- opinions, helping to focus this PEP.
-* Others on the python-ideas mailing list for rejecting the craziest of ideas,
- also helping to achieve focus.
+* Eric V. Smith for the authoring and implementation of PEP 498.
+* Everyone on the python-ideas mailing list for rejecting the various crazy
+ ideas that came up,
+ helping to keep the final design in focus.
References
@@ -771,7 +645,6 @@
(https://mail.python.org/pipermail/python-ideas/2015-July/034659.html)
-
.. [2] Briefer String Format
(https://mail.python.org/pipermail/python-ideas/2015-July/034669.html)
--
Repository URL: https://hg.python.org/peps
More information about the Python-checkins
mailing list