Mailman 3 January 2017 - Python-ideas

How to respond to trolling (Guido van Rossum)
by Simon Lovell 13 Jan '17

13 Jan '17

I feel I have to respond to this one. More than half of what I suggested could have and should be implemented. In particular the truthiness of non-boolean data and the lack of a reasonable SQL syntax. Several other points have been discussed endlessly on the internet but without a satisfactory (IMO) answer being given. I don't know what is meant by some insults having been thrown in. Calling truthiness of non boolean data "Ugly" is an insult? It is ugly. Yes, I should have double checked the chained assignment before posting and perhaps including some things which weren't changing added negative value. Regarding this comment. 'I use vim, which is very respectable, thank you. You'd like me to use "EditPlus 2" or equivalent', I think you should familiarise yourself with the "map!" function in vi and vim - put it in your .exrc file or .vimrc (vim only). e.g. "map! if if ^M ^Mendif". Regarding the with function, to those not familiar with what I was referring to that is a construct in Delphi and some other languages which works like this: ReallyLongFileDescriptor=open("file") with ReallyLongFileDescriptor: x=readline() // note the lack of "ReallyLongFileDescriptor." print x Delphi is even worse in that you can add more than one prefix in your with statement. Yes, you can put #endif at the end of every "if" statement etc. That requires a checker in the vein of the ccheck of yore to be enforced. These things aren't desirable.

8 8

RFC: PEP 540 version 3 (Add a new UTF-8 mode)
by Victor Stinner 13 Jan '17

13 Jan '17

Hi, I also implemented my PEP 540, you can now test it! Use the latest patch attached to: http://bugs.python.org/issue29240 I made multiple changes since the first version of my PEP: * The UTF-8 Strict mode now only uses strict for inputs and outputs: it keeps surrogateescape for operating system data. Read the "Use the strict error handler for operating system data" alternative for the rationale. * The POSIX locale now enables the UTF-8 mode. See the "Don't modify the encoding of the POSIX locale" alternative for the rationale. * Specify the priority between -X utf8, PYTHONUTF8, PYTHONIOENCODING, etc. The PEP version 3 has a longer rationale with more example. IMHO the "List a directory into stdout" use case is the most representative case of "UNIX should just work" thing and encoding issues: https://www.python.org/dev/peps/pep-0540/#list-a-directory-into-stdout It reads data from the operating system (directory content) and writes it into an output (stdout). It combines two things which are similar but different in subtle ways. I included example with commands and their output to this use case, to have a more "real world" example instead of a long list of theorical things :-) Read the PEP 540 online (HTML): https://www.python.org/dev/peps/pep-0540/ Full text below. Victor PEP: 540 Title: Add a new UTF-8 mode Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-January-2016 Python-Version: 3.7 Abstract ======== Add a new UTF-8 mode, disabled by default, to ignore the locale and force the usage of the UTF-8 encoding. Basically, the UTF-8 mode behaves as Python 2: it "just works" and don't bother users with encodings, but it can produce mojibake. The UTF-8 mode can be configured as strict to prevent mojibake. New ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added to control the UTF-8 mode. The POSIX locale enables the UTF-8 mode. Rationale ========= "It's not a bug, you must fix your locale" is not an acceptable answer ---------------------------------------------------------------------- Since Python 3.0 was released in 2008, the usual answer to users getting Unicode errors is to ask developers to fix their code to handle Unicode properly. Most applications and Python modules were fixed, but users keep reporting Unicode errors regulary: see the long list of issues in the `Links`_ section below. In fact, a second class of bug comes from a locale which is not properly configured. The usual answer to such bug report is: "it is not a bug, you must fix your locale". Technically, the answer is correct, but from a practical point of view, the answer is not acceptable. In many cases, "fixing the issue" is an hard task. Moreover, sometimes, the usage of the POSIX locale is deliberate. A good example of a concrete issue are build systems which create a fresh environment for each build using a chroot, a container, a virtual machine or something else to get reproductible builds. Such setup usually uses the POSIX locale. To get 100% reproductible builds, the POSIX locale is a good choice: see the `Locales section of reproducible-builds.org <https://reproducible-builds.org/docs/locales/>`_. UNIX users don't expect Unicode errors, since the common command lines tools like ``cat``, ``grep`` or ``sed`` never fail with Unicode errors. These users expect that Python 3 "just works" with any locale and don't bother them with encodings. From their point of the view, the bug is not their locale but is obviously Python 3. Since Python 2 handles data as bytes, it's rarer in Python 2 compared to Python 3 to get Unicode errors. It also explains why users also perceive Python 3 as the root cause of their Unicode errors. Some users expect that Python 3 just works with any locale and so don't bother with mojibake, whereas some developers are working hard to prevent mojibake and so expect that Python 3 fails early before creating mojibake. Since different group of users have different expectations, there is no silver bullet which solves all issues at once. Last but not least, backward compatibility should be preserved whenever possible. Locale and operating system data -------------------------------- .. _operating system data: Python uses an encoding called the "filesystem encoding" to decide how to encode and decode data from/to the operating system: * file content * command line arguments: ``sys.argv`` * standard streams: ``sys.stdin``, ``sys.stdout``, ``sys.stderr`` * environment variables: ``os.environ`` * filenames: ``os.listdir(str)`` for example * pipes: ``subprocess.Popen`` using ``subprocess.PIPE`` for example * error messages: ``os.strerror(code)`` for example * user and terminal names: ``os``, ``grp`` and ``pwd`` modules * host name, UNIX socket path: see the ``socket`` module * etc. At startup, Python calls ``setlocale(LC_CTYPE, "")`` to use the user ``LC_CTYPE`` locale and then store the locale encoding as the "filesystem error". It's possible to get this encoding using ``sys.getfilesystemencoding()``. In the whole lifetime of a Python process, the same encoding and error handler are used to encode and decode data from/to the operating system. The ``os.fsdecode()`` and ``os.fsencode()`` functions can be used to decode and encode operating system data. These functions use the filesystem error handler: ``sys.getfilesystemencodeerrors()``. .. note:: In some corner case, the *current* ``LC_CTYPE`` locale must be used instead of ``sys.getfilesystemencoding()``. For example, the ``time`` module uses the *current* ``LC_CTYPE`` locale to decode timezone names. The POSIX locale and its encoding --------------------------------- The following environment variables are used to configure the locale, in this preference order: * ``LC_ALL``, most important variable * ``LC_CTYPE`` * ``LANG`` The POSIX locale,also known as "the C locale", is used: * if the first set variable is set to ``"C"`` * if all these variables are unset, for example when a program is started in an empty environment. The encoding of the POSIX locale must be ASCII or a superset of ASCII. On Linux, the POSIX locale uses the ASCII encoding. On FreeBSD and Solaris, ``nl_langinfo(CODESET)`` announces an alias of the ASCII encoding, whereas ``mbstowcs()`` and ``wcstombs()`` functions use the ISO 8859-1 encoding (Latin1) in practice. The problem is that ``os.fsencode()`` and ``os.fsdecode()`` use ``locale.getpreferredencoding()`` codec. For example, if command line arguments are decoded by ``mbstowcs()`` and encoded back by ``os.fsencode()``, an ``UnicodeEncodeError`` exception is raised instead of retrieving the original byte string. To fix this issue, Python checks since Python 3.4 if ``mbstowcs()`` really uses the ASCII encoding if the the ``LC_CTYPE`` uses the the POSIX locale and ``nl_langinfo(CODESET)`` returns ``"ASCII"`` (or an alias to ASCII). If not (the effective encoding is not ASCII), Python uses its own ASCII codec instead of using ``mbstowcs()`` and ``wcstombs()`` functions for `operating system data`_. See the `POSIX locale (2016 Edition) <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html>`_. POSIX locale used by mistake ---------------------------- In many cases, the POSIX locale is not really expected by users who get it by mistake. Examples: * program started in an empty environment * User forcing LANG=C to get messages in english * LANG=C used for bad reasons, without being aware of the ASCII encoding * SSH shell * Linux installed with no configured locale * chroot environment, Docker image, container, ... with no locale is configured * User locale set to a non-existing locale, typo in the locale name for example C.UTF-8 and C.utf8 locales -------------------------- Some UNIX operating systems provide a variant of the POSIX locale using the UTF-8 encoding: * Fedora 25: ``"C.utf8"`` or ``"C.UTF-8"`` * Debian (eglibc 2.13-1, 2011), Ubuntu: ``"C.UTF-8"`` * HP-UX: ``"C.utf8"`` It was proposed to add a ``C.UTF-8`` locale to the glibc: `glibc C.UTF-8 proposal <https://sourceware.org/glibc/wiki/Proposals/C.UTF-8>`_. It is not planned to add such locale to BSD systems. Popularity of the UTF-8 encoding -------------------------------- Python 3 uses UTF-8 by default for Python source files. On Mac OS X, Windows and Android, Python always use UTF-8 for operating system data. For Windows, see the `PEP 529`_: "Change Windows filesystem encoding to UTF-8". On Linux, UTF-8 became the de facto standard encoding, replacing legacy encodings like ISO 8859-1 or ShiftJIS. For example, using different encodings for filenames and standard streams is likely to create mojibake, so UTF-8 is now used *everywhere*. The UTF-8 encoding is the default encoding of XML and JSON file format. In January 2017, UTF-8 was used in `more than 88% of web pages <https://w3techs.com/technologies/details/en-utf8/all/all>`_ (HTML, Javascript, CSS, etc.). See `utf8everywhere.org <http://utf8everywhere.org/>`_ for more general information on the UTF-8 codec. .. note:: Some applications and operating systems (especially Windows) use Byte Order Markers (BOM) to indicate the used Unicode encoding: UTF-7, UTF-8, UTF-16-LE, etc. BOM are not well supported and rarely used in Python. Old data stored in different encodings and surrogateescape ---------------------------------------------------------- Even if UTF-8 became the de facto standard, there are still systems in the wild which don't use UTF-8. And there are a lot of data stored in different encodings. For example, an old USB key using the ext3 filesystem with filenames encoded to ISO 8859-1. The Linux kernel and the libc don't decode filenames: a filename is used as a raw array of bytes. The common solution to support any filename is to store filenames as bytes and don't try to decode them. When displayed to stdout, mojibake is displayed if the filename and the terminal don't use the same encoding. Python 3 promotes Unicode everywhere including filenames. A solution to support filenames not decodable from the locale encoding was found: the ``surrogateescape`` error handler (`PEP 383`_), store undecodable bytes as surrogate characters. This error handler is used by default for `operating system data`_, by ``os.fsdecode()`` and ``os.fsencode()`` for example (except on Windows which uses the ``strict`` error handler). Standard streams ---------------- Python uses the locale encoding for standard streams: stdin, stdout and stderr. The ``strict`` error handler is used by stdin and stdout to prevent mojibake. The ``backslashreplace`` error handler is used by stderr to avoid Unicode encode error when displaying non-ASCII text. It is especially useful when the POSIX locale is used, because this locale usually uses the ASCII encoding. The problem is that `operating system data`_ like filenames are decoded using the ``surrogateescape`` error handler (`PEP 383`_). Displaying a filename to stdout raises a Unicode encode error if the filename contains an undecoded byte stored as a surrogate character. Python 3.6 now uses ``surrogateescape`` for stdin and stdout if the POSIX locale is used: `issue #19977 <http://bugs.python.org/issue19977>`_. The idea is to passthrough `operating system data`_ even if it means mojibake, because most UNIX applications work like that. Most UNIX applications store filenames as bytes, usually simply because bytes are first-citizen class in the used programming language, whereas Unicode is badly supported. .. note:: The encoding and/or the error handler of standard streams can be overriden with the ``PYTHONIOENCODING`` environment variable. Proposal ======== Changes ------- Add a new UTF-8 mode, disabled by default, to ignore the locale and force the usage of the UTF-8 encoding with the ``surrogateescape`` error handler, instead using the locale encoding (with ``strict`` or ``surrogateescape`` error handler depending on the case). Basically, the UTF-8 mode behaves as Python 2: it "just works" and don't bother users with encodings, but it can produce mojibake. It can be configured as strict to prevent mojibake: the UTF-8 encoding is used with the ``strict`` error handler for inputs and outputs, but the ``surrogateescape`` error handler is still used for `operating system data`_. New ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable are added to control the UTF-8 mode. The UTF-8 mode is enabled by ``-X utf8`` or ``PYTHONUTF8=1``. The UTF-8 is configured as strict by ``-X utf8=strict`` or ``PYTHONUTF8=strict``. Other option values fail with an error. The POSIX locale enables the UTF-8 mode. In this case, the UTF-8 mode can be explicitly disabled by ``-X utf8=0`` or ``PYTHONUTF8=0``. The ``-X utf8`` has the priority over the ``PYTHONUTF8`` environment variable. For example, ``PYTHONUTF8=0 python3 -X utf8`` enables the UTF-8 mode. Encoding and error handler -------------------------- The UTF-8 mode changes the default encoding and error handler used by open(), os.fsdecode(), os.fsencode(), sys.stdin, sys.stdout and sys.stderr: ============================ ======================= ========================== ========================== Function Default UTF-8 or POSIX locale UTF-8 Strict ============================ ======================= ========================== ========================== open() locale/strict **UTF-8/surrogateescape** **UTF-8**/strict os.fsdecode(), os.fsencode() locale/surrogateescape **UTF-8**/surrogateescape **UTF-8**/surrogateescape sys.stdin, sys.stdout locale/strict **UTF-8/surrogateescape** **UTF-8**/strict sys.stderr locale/backslashreplace **UTF-8**/backslashreplace **UTF-8**/backslashreplace ============================ ======================= ========================== ========================== By comparison, Python 3.6 uses: ============================ ======================= ========================== Function Default POSIX locale ============================ ======================= ========================== open() locale/strict locale/strict os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape sys.stdin, sys.stdout locale/strict locale/**surrogateescape** sys.stderr locale/backslashreplace locale/backslashreplace ============================ ======================= ========================== The UTF-8 mode uses the ``surrogateescape`` error handler instead of the strict mode for convenience: the idea is that data not encoded to UTF-8 are passed through "Python" without being modified, as raw bytes. The ``PYTHONIOENCODING`` environment variable has the priority on the UTF-8 mode for standard streams. For example, ``PYTHONIOENCODING=latin1 python3 -X utf8`` uses the Latin1 encoding for stdin, stdout and stderr. Encodings used by ``open()``, highest priority first: * *encoding* and *errors* parameters (if set) * UTF-8 mode * os.device_encoding(fd) * os.getpreferredencoding(False) Rationale --------- The UTF-8 mode is disabled by default to keep hard Unicode errors when encoding or decoding `operating system data`_ failed, and to keep the backward compatibility. The user is responsible to enable explicitly the UTF-8 mode, and so is better prepared for mojibake than if the UTF-8 mode would be enabled *by default*. The UTF-8 mode should be used on systems known to be configured with UTF-8 where most applications speak UTF-8. It prevents Unicode errors if the user overrides a locale *by mistake* or if a Python program is started with no locale configured (and so with the POSIX locale). Most UNIX applications handle `operating system data`_ as bytes, so ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables have a limited impact on how these data are handled by the application. The Python UTF-8 mode should help to make Python more interoperable with the other UNIX applications in the system assuming that *UTF-8* is used everywhere and that users *expect* UTF-8. Ignoring ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables in Python is more convenient, since they are more commonly misconfigured *by mistake* (configured to use an encoding different than UTF-8, whereas the system uses UTF-8), rather than being misconfigured by intent. Expected mojibake and surrogate character issues ------------------------------------------------ The UTF-8 mode only affects code running directly in Python, especially code written in pure Python. The other code, called "external code" here, is not aware of this mode. Examples: * C libraries called by Python modules like OpenSSL * The application code when Python is embedded in an application In the UTF-8 mode, Python uses the ``surrogateescape`` error handler which stores bytes not decodable from UTF-8 as surrogate characters. If the external code uses the locale and the locale encoding is UTF-8, it should work fine. External code using bytes ^^^^^^^^^^^^^^^^^^^^^^^^^ If the external code process data as bytes, surrogate characters are not an issue since they are only used inside Python. Python encodes back surrogate characters to bytes at the edges, before calling external code. The UTF-8 mode can produce mojibake since Python and external code don't both of invalid bytes, but it's a deliberate choice. The UTF-8 mode can be configured as strict to prevent mojibake and be fail early when data is not decodable from UTF-8 or not encodable to UTF-8. External code using text ^^^^^^^^^^^^^^^^^^^^^^^^ If the external code uses text API, for example using the ``wchar_t*`` C type, mojibake should not occur, but the external code can fail on surrogate characters. Use Cases ========= The following use cases were written to help to understand the impact of chosen encodings and error handlers on concrete examples. The "Always work" results were written to prove the benefit of having a UTF-8 mode which works with any data and any locale, compared to the existing old Python versions. The "Mojibake" column shows that ignoring the locale causes a pratical issue: the UTF-8 mode produces mojibake if the terminal doesn't use the UTF-8 encoding. List a directory into stdout ---------------------------- Script listing the content of the current directory into stdout:: import os for name in os.listdir(os.curdir): print(name) Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 **Yes** **Yes** Python 3 No No Python 3.5, POSIX locale **Yes** **Yes** UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= "No" means that the script can fail on decoding or encoding a filename depending on the locale or the filename. To be able to always work, the program must be able to produce mojibake. Mojibake is more user friendly than an error with a truncated or empty output. Example with a directory which contains the file called ``b'xxx\xff'`` (the byte ``0xFF`` is invalid in UTF-8). Default and UTF-8 Strict mode fail on ``print()`` with an encode error:: $ python3.7 ../ls.py Traceback (most recent call last): File "../ls.py", line 5, in <module> print(name) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ... $ python3.7 -X utf8=strict ../ls.py Traceback (most recent call last): File "../ls.py", line 5, in <module> print(name) UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' ... The UTF-8 mode, POSIX locale, Python 2 and the UNIX ``ls`` command work but display mojibake:: $ python3.7 -X utf8 ../ls.py xxx� $ LC_ALL=C /python3.6 ../ls.py xxx� $ python2 ../ls.py xxx� $ ls 'xxx'$'\377' List a directory into a text file --------------------------------- Similar to the previous example, except that the listing is written into a text file:: import os names = os.listdir(os.curdir) with open("/tmp/content.txt", "w") as fp: for name in names: fp.write("%s\n" % name) Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 **Yes** **Yes** Python 3 No No Python 3.5, POSIX locale No No UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= "Yes" involves that mojibake can be produced. "No" means that the script can fail on decoding or encoding a filename depending on the locale or the filename. Typical error:: $ LC_ALL=C python3 test.py Traceback (most recent call last): File "test.py", line 5, in <module> fp.write("%s\n" % name) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128) Display Unicode characters into stdout -------------------------------------- Very basic example used to illustrate a common issue, display the euro sign (U+20AC: €):: print("euro: \u20ac") Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 No No Python 3 No No Python 3.5, POSIX locale No No UTF-8 mode **Yes** **Yes** UTF-8 Strict mode **Yes** **Yes** ======================== ============ ========= The UTF-8 and UTF-8 Strict modes will always encode the euro sign as UTF-8. If the terminal uses a different encoding, we get mojibake. Replace a word in a text ------------------------ The following scripts replaces the word "apple" with "orange". It reads input from stdin and writes the output into stdout:: import sys text = sys.stdin.read() sys.stdout.write(text.replace("apple", "orange")) Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 **Yes** **Yes** Python 3 No No Python 3.5, POSIX locale **Yes** **Yes** UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= Producer-consumer model using pipes ----------------------------------- Let's say that we have a "producer" program which writes data into its stdout and a "consumer" program which reads data from its stdin. On a shell, such programs are run with the command:: producer | consumer The question if these programs will work with any data and any locale. UNIX users don't expect Unicode errors, and so expect that such programs "just works". If the producer only produces ASCII output, no error should occur. Let's say the that producer writes at least one non-ASCII character (at least one byte in the range ``0x80..0xff``). To simplify the problem, let's say that the consumer has no output (don't write result into a file or stdout). A "Bytes producer" is an application which cannot fail with a Unicode error and produces bytes into stdout. Let's say that a "Bytes consumer" does not decode stdin but stores data as bytes: such consumer always work. Common UNIX command line tools like ``cat``, ``grep`` or ``sed`` are in this category. Many Python 2 applications are also in this category. "Python producer" and "Python consumer" are producer and consumer implemented in Python. Bytes producer, Bytes consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It always work, but it is out of the scope of this PEP since it doesn't involve Python. Python producer, Bytes consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python producer:: print("euro: \u20ac") Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 No No Python 3 No No Python 3.5, POSIX locale No No UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= The question here is not if the consumer is able to decode the input, but if Python is able to produce its ouput. So it's similar to the `Display Unicode characters into stdout`_ case. UTF-8 modes work with any locale since the consumer doesn't try to decode its stdin. Bytes producer, Python consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python consumer:: import sys text = sys.stdin.read() result = text.replace("apple", "orange") # ignore the result Result: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 **Yes** **Yes** Python 3 No No Python 3.5, POSIX locale **Yes** **Yes** UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= Python 3 fails on decoding stdin depending on the input and the locale. Python producer, Python consumer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Python producer:: print("euro: \u20ac") Python consumer:: import sys text = sys.stdin.read() result = text.replace("apple", "orange") # ignore the result Result, same Python version used for the producer and the consumer: ======================== ============ ========= Python Always work? Mojibake? ======================== ============ ========= Python 2 No No Python 3 No No Python 3.5, POSIX locale No No UTF-8 mode **Yes** **Yes** UTF-8 Strict mode No No ======================== ============ ========= This case combines a Python producer with a Python consumer, so the result is the subset of `Python producer, Bytes consumer`_ and `Bytes producer, Python consumer`_. Backward Compatibility ====================== The main backward incompatible change is that the UTF-8 encoding is now used by default if the locale is POSIX. Since the UTF-8 encoding is used with the ``surrogateescape`` error handler, encoding errors should not occur and so the change should not break applications. The more likely source of trouble comes from external libraries. Python can decode successfully data from UTF-8, but a library using the locale encoding can fail to encode the decoded text back to bytes. Hopefully, encoding text in a library is a rare operation. Very few libraries expect text, most libraries expect bytes and even manipulate bytes internally. The PEP only changes the default behaviour if the locale is POSIX. For other locales, the *default* behaviour is unchanged. Alternatives ============ Don't modify the encoding of the POSIX locale --------------------------------------------- A first version of the PEP did not change the encoding and error handler used of the POSIX locale. The problem is that adding the ``-X utf8`` command line option or setting the ``PYTHONUTF8`` environment variable is not possible in some cases, or at least not convenient. Moreover, many users simply expect that Python 3 behaves as Python 2: don't bother them with encodings and "just works" in all cases. These users don't worry about mojibake, or even expect mojibake because of complex documents using multiple incompatibles encodings. Always use UTF-8 ---------------- Python already always use the UTF-8 encoding on Mac OS X, Android and Windows. Since UTF-8 became the de facto encoding, it makes sense to always use it on all platforms with any locale. The risk is to introduce mojibake if the locale uses a different encoding, especially for locales other than the POSIX locale. Force UTF-8 for the POSIX locale -------------------------------- An alternative to always using UTF-8 in any case is to only use UTF-8 when the ``LC_CTYPE`` locale is the POSIX locale. The `PEP 538`_ "Coercing the legacy C locale to C.UTF-8" of Nick Coghlan proposes to implement that using the ``C.UTF-8`` locale. Use the strict error handler for operating system data ------------------------------------------------------ Using the ``surrogateescape`` error handler for `operating system data`_ creates surprising surrogate characters. No Python codec (except of ``utf-7``) accept surrogates, and so encoding text coming from the operating system is likely to raise an error error. The problem is that the error comes late, very far from where the data was read. The ``strict`` error handler can be used instead to decode (``os.fsdecode()``) and encode (``os.fsencode()``) operating system data, to raise encoding errors as soon as possible. It helps to find bugs more quickly. The main drawback of this strategy is that it doesn't work in practice. Python 3 is designed on top on Unicode strings. Most functions expect Unicode and produce Unicode. Even if many operating system functions have two flavors, bytes and Unicode, the Unicode flavar is used is most cases. There are good reasons for that: Unicode is more convenient in Python 3 and using Unicode helps to support the full Unicode Character Set (UCS) on Windows (even if Python now uses UTF-8 since Python 3.6, see the `PEP 528`_ and the `PEP 529`_). For example, if ``os.fsdecode()`` uses ``utf8/strict``, ``os.listdir(str)`` fails to list filenames of a directory if a single filename is not decodable from UTF-8. As a consequence, ``shutil.rmtree(str)`` fails to remove a directory. Undecodable filenames, environment variables, etc. are simply too common to make this alternative viable. Links ===== PEPs: * `PEP 538 <https://www.python.org/dev/peps/pep-0538/>`_: "Coercing the legacy C locale to C.UTF-8" * `PEP 529 <https://www.python.org/dev/peps/pep-0529/>`_: "Change Windows filesystem encoding to UTF-8" * `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_: "Change Windows console encoding to UTF-8" * `PEP 383 <https://www.python.org/dev/peps/pep-0383/>`_: "Non-decodable Bytes in System Character Interfaces" Main Python issues: * `Issue #29240: Implementation of the PEP 540: Add a new UTF-8 mode <http://bugs.python.org/issue29240>`_ * `Issue #28180: sys.getfilesystemencoding() should default to utf-8 <http://bugs.python.org/issue28180>`_ * `Issue #19977: Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale <http://bugs.python.org/issue19977>`_ * `Issue #19847: Setting the default filesystem-encoding <http://bugs.python.org/issue19847>`_ * `Issue #8622: Add PYTHONFSENCODING environment variable <https://bugs.python.org/issue8622>`_: added but reverted because of many issues, read the `Inconsistencies if locale and filesystem encodings are different <https://mail.python.org/pipermail/python-dev/2010-October/104509.html>`_ thread on the python-dev mailing list Incomplete list of Python issues related to Unicode errors, especially with the POSIX locale: * 2016-12-22: `LANG=C python3 -c "import os; os.path.exists('\xff')" <http://bugs.python.org/issue29042#msg283821>`_ * 2014-07-20: `issue #22016: Add a new 'surrogatereplace' output only error handler <http://bugs.python.org/issue22016>`_ * 2014-04-27: `Issue #21368: Check for systemd locale on startup if current locale is set to POSIX <http://bugs.python.org/issue21368>`_ -- read manually /etc/locale.conf when the locale is POSIX * 2014-01-21: `Issue #20329: zipfile.extractall fails in Posix shell with utf-8 filename <http://bugs.python.org/issue20329>`_ * 2013-11-30: `Issue #19846: Python 3 raises Unicode errors with the C locale <http://bugs.python.org/issue19846>`_ * 2010-05-04: `Issue #8610: Python3/POSIX: errors if file system encoding is None <http://bugs.python.org/issue8610>`_ * 2013-08-12: `Issue #18713: Clearly document the use of PYTHONIOENCODING to set surrogateescape <http://bugs.python.org/issue18713>`_ * 2013-09-27: `Issue #19100: Use backslashreplace in pprint <http://bugs.python.org/issue19100>`_ * 2012-01-05: `Issue #13717: os.walk() + print fails with UnicodeEncodeError <http://bugs.python.org/issue13717>`_ * 2011-12-20: `Issue #13643: 'ascii' is a bad filesystem default encoding <http://bugs.python.org/issue13643>`_ * 2011-03-16: `issue #11574: TextIOWrapper should use UTF-8 by default for the POSIX locale <http://bugs.python.org/issue11574>`_, thread on python-dev: `Low-Level Encoding Behavior on Python 3 <https://mail.python.org/pipermail/python-dev/2011-March/109361.html>`_ * 2010-04-26: `Issue #8533: regrtest: use backslashreplace error handler for stdout <http://bugs.python.org/issue8533>`_, regrtest fails with Unicode encode error if the locale is POSIX Some issues are real bug in applications which must set explicitly the encoding. Well, it just works in the common case (locale configured correctly), so what? But the program "suddenly" fails when the POSIX locale is used (probably for bad reasons). Such bug is not well understood by users. Example of such issue: * 2013-11-21: `pip: open() uses the locale encoding to parse Python script, instead of the encoding cookie <http://bugs.python.org/issue19685>`_ -- pip must use the encoding cookie to read a Python source code file * 2011-01-21: `IDLE 3.x can crash decoding recent file list <http://bugs.python.org/issue10974>`_ Prior Art ========= Perl has a ``-C`` command line option and a ``PERLUNICODE`` environment varaible to force UTF-8: see `perlrun <http://perldoc.perl.org/perlrun.html>`_. It is possible to configure UTF-8 per standard stream, on input and output streams, etc. Copyright ========= This document has been placed in the public domain.

7 18

PEP 540: Add a new UTF-8 mode
by Victor Stinner 13 Jan '17

13 Jan '17

Hi, Nick Coghlan asked me to review his PEP 538 "Coercing the legacy C locale to C.UTF-8": https://www.python.org/dev/peps/pep-0538/ Nick wants to change the default behaviour. I'm not sure that I'm brave enough to follow this direction, so I proposed my old "-X utf8" command line idea as a new PEP: add a new UTF-8 mode, *disabled by default*. These 2 PEPs are the follow-up of the Windows PEP 529 (Change Windows filesystem encoding to UTF-8) and the issue #19977 (Use "surrogateescape" error handler for sys.stdin and sys.stdout on UNIX for the C locale). The topic (switching to UTF-8 on UNIX) is actively discussed on: http://bugs.python.org/issue28180 Read the PEP online (HTML): https://www.python.org/dev/peps/pep-0540/ Victor PEP: 540 Title: Add a new UTF-8 mode Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 5-January-2016 Python-Version: 3.7 Abstract ======== Add a new UTF-8 mode, opt-in option to use UTF-8 for operating system data instead of the locale encoding. Add ``-X utf8`` command line option and ``PYTHONUTF8`` environment variable. Context ======= Locale and operating system data -------------------------------- Python uses the ``LC_CTYPE`` locale to decide how to encode and decode data from/to the operating system: * file content * command line arguments: ``sys.argv`` * standard streams: ``sys.stdin``, ``sys.stdout``, ``sys.stderr`` * environment variables: ``os.environ`` * filenames: ``os.listdir(str)`` for example * pipes: ``subprocess.Popen`` using ``subprocess.PIPE`` for example * error messages * name of a timezone * user name, terminal name: ``os``, ``grp`` and ``pwd`` modules * host name, UNIX socket path: see the ``socket`` module * etc. At startup, Python calls ``setlocale(LC_CTYPE, "")`` to use the user ``LC_CTYPE`` locale and then store the locale encoding, ``sys.getfilesystemencoding()``. In the whole lifetime of a Python process, the same encoding and error handler are used to encode and decode data from/to the operating system. .. note:: In some corner case, the *current* ``LC_CTYPE`` locale must be used instead of ``sys.getfilesystemencoding()``. For example, the ``time`` module uses the *current* ``LC_CTYPE`` locale to decode timezone names. The POSIX locale and its encoding --------------------------------- The following environment variables are used to configure the locale, in this preference order: * ``LC_ALL``, most important variable * ``LC_CTYPE`` * ``LANG`` The POSIX locale,also known as "the C locale", is used: * if the first set variable is set to ``"C"`` * if all these variables are unset, for example when a program is started in an empty environment. The encoding of the POSIX locale must be ASCII or a superset of ASCII. On Linux, the POSIX locale uses the ASCII encoding. On FreeBSD and Solaris, ``nl_langinfo(CODESET)`` announces an alias of the ASCII encoding, whereas ``mbstowcs()`` and ``wcstombs()`` functions use the ISO 8859-1 encoding (Latin1) in practice. The problem is that ``os.fsencode()`` and ``os.fsdecode()`` use ``locale.getpreferredencoding()`` codec. For example, if command line arguments are decoded by ``mbstowcs()`` and encoded back by ``os.fsencode()``, an ``UnicodeEncodeError`` exception is raised instead of retrieving the original byte string. To fix this issue, Python now checks since Python 3.4 if ``mbstowcs()`` really uses the ASCII encoding if the the ``LC_CTYPE`` uses the the POSIX locale and ``nl_langinfo(CODESET)`` returns ``"ASCII"`` (or an alias to ASCII). If not (the effective encoding is not ASCII), Python uses its own ASCII codec instead of using ``mbstowcs()`` and ``wcstombs()`` functions for operating system data. See the `POSIX locale (2016 Edition) <http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap07.html>`_. C.UTF-8 and C.utf8 locales -------------------------- Some operating systems provide a variant of the POSIX locale using the UTF-8 encoding: * Fedora 25: ``"C.utf8"`` or ``"C.UTF-8"`` * Debian (eglibc 2.13-1, 2011): ``"C.UTF-8"`` * HP-UX: ``"C.utf8"`` It was proposed to add a ``C.UTF-8`` locale to glibc: `glibc C.UTF-8 proposal <https://sourceware.org/glibc/wiki/Proposals/C.UTF-8>`_. Popularity of the UTF-8 encoding -------------------------------- Python 3 uses UTF-8 by default for Python source files. On Mac OS X, Windows and Android, Python always use UTF-8 for operating system data instead of the locale encoding. For Windows, see the `PEP 529: Change Windows filesystem encoding to UTF-8 <https://www.python.org/dev/peps/pep-0529/>`_. On Linux, UTF-8 became the defacto standard encoding by default, replacing legacy encodings like ISO 8859-1 or ShiftJIS. For example, using different encodings for filenames and standard streams is likely to create mojibake, so UTF-8 is now used *everywhere*. The UTF-8 encoding is the default encoding of XML and JSON file format. In January 2017, UTF-8 was used in `more than 88% of web pages <https://w3techs.com/technologies/details/en-utf8/all/all>`_ (HTML, Javascript, CSS, etc.). See `utf8everywhere.org <http://utf8everywhere.org/>`_ for more general information on the UTF-8 codec. .. note:: Some applications and operating systems (especially Windows) use Byte Order Markers (BOM) to indicate the used Unicode encoding: UTF-7, UTF-8, UTF-16-LE, etc. BOM are not well supported and rarely used in Python. Old data stored in different encodings and surrogateescape ---------------------------------------------------------- Even if UTF-8 became the defacto standard, there are still systems in the wild which don't use UTF-8. And there are a lot of data stored in different encodings. For example, an old USB key using the ext3 filesystem with filenames encoded to ISO 8859-1. The Linux kernel and the libc don't decode filenames: a filename is used as a raw array of bytes. The common solution to support any filename is to store filenames as bytes and don't try to decode them. When displayed to stdout, mojibake is displayed if the filename and the terminal don't use the same encoding. Python 3 promotes Unicode everywhere including filenames. A solution to support filenames not decodable from the locale encoding was found: the ``surrogateescape`` error handler (`PEP 393 <https://www.python.org/dev/peps/pep-0393/>`_), store undecodable bytes as surrogate characters. This error handler is used by default for operating system data, by ``os.fsdecode()`` and ``os.fsencode()`` for example (except on Windows which uses the ``strict`` error handler). Standard streams ---------------- Python uses the locale encoding for standard streams: stdin, stdout and stderr. The ``strict`` error handler is used by stdin and stdout to prevent mojibake. The ``backslashreplace`` error handler is used by stderr to avoid Unicode encode error when displaying non-ASCII text. It is especially useful when the POSIX locale is used, because this locale usually uses the ASCII encoding. The problem is that operating system data like filenames are decoded using the ``surrogateescape`` error handler (PEP 393). Displaying a filename to stdout raises an Unicode encode error if the filename contains an undecoded byte stored as a surrogate character. Python 3.6 now uses ``surrogateescape`` for stdin and stdout if the POSIX locale is used: `issue #19977 <http://bugs.python.org/issue19977>`_. The idea is to passthrough operating system data even if it means mojibake, because most UNIX applications work like that. Most UNIX applications store filenames as bytes, usually simply because bytes are first-citizen class in the used programming language, whereas Unicode is badly supported. .. note:: The encoding and/or the error handler of standard streams can be overriden with the ``PYTHONIOENCODING`` environment variable. Proposal ======== Add a new UTF-8 mode, opt-in option to use UTF-8 for operating system data instead of the locale encoding: * Add ``-X utf8`` command line option * Add ``PYTHONUTF8=1`` environment variable Add also a strict UTF-8 mode, enabled by ``-X utf8=strict`` or ``PYTHONUTF8=strict``. The UTF-8 mode changes the default encoding and error handler used by open(), os.fsdecode(), os.fsencode(), sys.stdin, sys.stdout and sys.stderr: ============================ ======================= ======================= ====================== ====================== Function Default, other locales Default, POSIX locale UTF-8 UTF-8 Strict ============================ ======================= ======================= ====================== ====================== open() locale/strict locale/strict UTF-8/surrogateescape UTF-8/strict os.fsdecode(), os.fsencode() locale/surrogateescape locale/surrogateescape UTF-8/surrogateescape UTF-8/strict sys.stdin locale/strict locale/surrogateescape UTF-8/surrogateescape UTF-8/strict sys.stdout locale/strict locale/surrogateescape UTF-8/surrogateescape UTF-8/strict sys.stderr locale/backslashreplace locale/backslashreplace UTF-8/backslashreplace UTF-8/backslashreplace ============================ ======================= ======================= ====================== ====================== The UTF-8 mode is disabled by default to keep hard Unicode errors when encoding or decoding operating system data failed, and to keep the backward compatibility. The user is responsible to enable explicitly the UTF-8 mode, and so is better prepared for mojibake than if the UTF-8 mode would be enabled *by default*. The UTF-8 mode should be used on systems known to be configured with UTF-8 where most applications speak UTF-8. It prevents Unicode errors if the user overrides a locale *by mistake* or if a Python program is started with no locale configured (and so with the POSIX locale). Most UNIX applications handle operating system data as bytes, so ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables have a limited impact on how these data are handled by the application. The Python UTF-8 mode should help to make Python more interoperable with the other UNIX applications in the system assuming that *UTF-8* is used everywhere and that users *expect* UTF-8. Ignoring ``LC_ALL``, ``LC_CTYPE`` and ``LANG`` environment variables in Python is more convenient, since they are more commonly misconfigured *by mistake* (configured to use an encoding different than UTF-8, whereas the system uses UTF-8), rather than being misconfigured by intent. Backward Compatibility ====================== Since the UTF-8 mode is disabled by default, it has no impact on the backward compatibility. The new UTF-8 mode must be enabled explicitly. Alternatives ============ Always use UTF-8 ---------------- Python already always use the UTF-8 encoding on Mac OS X, Android and Windows. Since UTF-8 became the defacto encoding, it makes sense to always use it on all platforms with any locale. The risk is to introduce mojibake if the locale uses a different encoding, especially for locales other than the POSIX locale. Force UTF-8 for the POSIX locale -------------------------------- An alternative to always using UTF-8 in any case is to only use UTF-8 when the ``LC_CTYPE`` locale is the POSIX locale. The `PEP 538: Coercing the legacy C locale to C.UTF-8 <https://www.python.org/dev/peps/pep-0538/>`_ of Nick Coghlan proposes to implement that using the ``C.UTF-8`` locale. Related Work ============ Perl has a ``-C`` command line option and a ``PERLUNICODE`` environment varaible to force UTF-8: see `perlrun <http://perldoc.perl.org/perlrun.html>`_. It is possible to configure UTF-8 per standard stream, on input and output streams, etc. Copyright ========= This document has been placed in the public domain.

14 58

Re: [Python-ideas] Settable defaulting to decimal instead of float
by George Fischhof 12 Jan '17

12 Jan '17

2017-01-12 13:13 GMT+01:00 Stephan Houben <stephanh42(a)gmail.com>: > Something like: > > from __syntax__ import decimal_literal > > which would feed the rest of the file through the "decimal_literal" > transpiler. > (and not influence anything in other files). > > Not sure if you would want to support multiple transpilers per file. > > Note that Racket has something similar with their initial "#lang ..." > directive. > That only allows a single "language". Possibly wisely so. > > Stephan > > > 2017-01-12 12:59 GMT+01:00 אלעזר <elazarg(a)gmail.com>: > >> I think such proposals are special cases of a general theme: a compiler >> pragma, similar to "from __future__", to make Python support >> domain-specific syntax in the current file. Whether it's decimal literals >> or matrix/vector literals etc. >> >> I think it will be nice to make some tool, external to Python, that will >> allow defining such "sibling languages" (transpiled into Python) easily and >> uniformly. >> >> Elazar >> >> בתאריך יום ה׳, 12 בינו' 2017, 13:21, מאת Paul Moore ‏<p.f.moore(a)gmail.com >> >: >> >>> On 12 January 2017 at 10:28, Victor Stinner <victor.stinner(a)gmail.com> >>> wrote: >>> > George requested this feature on the bug tracker: >>> > http://bugs.python.org/issue29223 >>> > >>> > George was asked to start a discusson on this list. I posted the >>> > following comment before closing the issue: >>> > >>> > You are not the first one to propose the idea. >>> >>> OK, but without additional detail (for example, how would the proposed >>> flag work, if the main module imports module A, then would float >>> literals in A be decimal or binary? Both could be what the user wants) >>> it's hard to comment. And as you say, most of this has been discussed >>> before, so I'd like to see references back to the previous discussions >>> in any proposal, with explanations of how the new proposal addresses >>> the objections raised previously. >>> >>> Paul >> >> Most of the time one of my students talks to me about decimal vs > binary, they're thinking that a decimal literal (or converting the > default non-integer literal to be decimal) is a panacea to the "0.1 + > 0.2 != 0.3" problem. Perhaps the real solution is a written-up > explanation of why binary floating point is actually a good thing, and > not just a backward-compatibility requirement? > > ChrisA from __future__ import use_decimal_instead_of_float or any other import would be very good. The most important thing in my point of view is that I do not want to convert every variable every time to decimal. Accuracy is important for me (yes, 0.1 + 0.2 should equal to 0.3 , no more, no less ;-) ) And if it is mentioned, I would like to ask why binary floating point is "better". It is faster, I agree, but why "better"? Actually I create test automation (I am a senior QA eng), the fastest test case runs for about 1-2 minutes. I do not know the exact time difference between binary and decimal arithmetic, but I do not care with this. My test would run some microseconds faster. It does not matter at a minute range. In the tests I calculate with numbers with 4 decimal digits, and I need exact match. ;-) Actually I have a new colleague, they did image analysis, and the calculated much calculations, and they used a library (not python) that is accurate as well. Becasue accuracy was more important for them as well. BR George

3 3

How to respond to trolling
by Guido van Rossum 12 Jan '17

12 Jan '17

Was it really necessary for all the usual folks on this list to engage with the "Python review" threads? I think a much more effective response would have been a resounding silence. -- --Guido van Rossum (python.org/~guido)

11 12

api suggestions for the cProfile module
by Thane Brimhall 11 Jan '17

11 Jan '17

I use cProfile a lot, and would like to suggest three backwards-compatible improvements to the API. 1: When using cProfile on a specific piece of code I often use the enable() and disable() methods. It occurred to me that this would be an obvious place to use a context manager. 2: Enhance the `print_stats` method on Profile to accept more options currently available only through the pstats.Stats class. For example, strip_dirs could be a boolean argument, and limit could accept an int. This would reduce the number of cases you'd need to use the more complex API. 3: I often forget which string keys are available for sorting. It would be nice to add an enum for these so a user could have their linter and IDE check that value pre-runtime. Since it would subclass `str` and `Enum` it would still work with all currently existing code. The current documentation contains the following code: import cProfile, pstats, io pr = cProfile.Profile() pr.enable() # ... do something ... pr.disable() s = io.StringIO() sortby = 'cumulative' ps = pstats.Stats(pr, stream=s).sort_stats(sortby) ps.print_stats() print(s.getvalue()) While the code below doesn't exactly match the functionality above (eg. not using StringIO), I envision the context manager working like this, along with some adjustments on how to get the stats from the profiler: import cProfile, pstats with cProfile.Profile() as pr: # ... do something ... pr.print_stats(sort=pstats.Sort.cumulative, limit=10, strip_dirs=True) As you can see, the code is shorter and somewhat more self-documenting. The best thing about these suggestions is that as far as I can tell they would be backwards-compatible API additions. What do you think? Thank you in advance for your time! /Thane

5 6

Re: [Python-ideas] Python reviewed
by Simon Lovell 11 Jan '17

11 Jan '17

Oh one last thing (I hope), the argument for having the current slice notation by Dijkstra, that it looks messy to have a loop where the contents are never executed or can no longer be executed is ridiculous! That *should* look messy. for range(1,1): means executing once to me. If you had 1 based, two of four of the other idioms would work the same: s[:n] + s[n:] == s // doesn't work. I don't think it should work though len(s[:n]) == n // works len(s[:-n]) == n // rather independent but would still work if language is otherwise unchanged. len(s[n:i]) == i - n // doesn't work. Does it need to? Rgds

2 1

api suggestions for the cProfile module
by Jelle Zijlstra 11 Jan '17

11 Jan '17

2017-01-10 8:57 GMT-08:00 Ethan Furman <ethan(a)stoneleaf.us>: > On 01/10/2017 08:36 AM, Thane Brimhall wrote: > > Does anyone have thoughts on this topic? I assume the silence is because >> this suggestion is too trivial to matter. >> > > Sometimes it's just a matter of timing. :) > > I use cProfile a lot, and would like to suggest three backwards-compatible >> improvements to the API. >> >> 1: When using cProfile on a specific piece of code I often use the >> enable() and disable() methods. It occurred to me that this would >> be an obvious place to use a context manager. >> > > Absolutely. > > 2: Enhance the `print_stats` method on Profile to accept more options >> currently available only through the pstats.Stats class. For example, >> strip_dirs could be a boolean argument, and limit could accept an int. >> This would reduce the number of cases you'd need to use the more complex >> API. >> > > I don't have much experience with cProfile, but this seems reasonable. > > 3: I often forget which string keys are available for sorting. It would >> be nice to add an enum for these so a user could have their linter and >> IDE check that value pre-runtime. Since it would subclass `str` and >> `Enum` it would still work with all currently existing code. >> > > Absolutely! :) > > The current documentation contains the following code: >> >> import cProfile, pstats, io >> pr = cProfile.Profile() >> pr.enable() >> # ... do something ... >> pr.disable() >> s = io.StringIO() >> sortby = 'cumulative' >> ps = pstats.Stats(pr, stream=s).sort_stats(sortby) >> ps.print_stats() >> print(s.getvalue()) >> >> While the code below doesn't exactly match the functionality above (eg. >> not >> using StringIO), I envision the context manager working like this, along >> with some adjustments on how to get the stats from the profiler: >> >> import cProfile, pstats >> with cProfile.Profile() as pr: >> # ... do something ... >> pr.print_stats(sort=pstats.Sort.cumulative, limit=10, >> strip_dirs=True) >> >> As you can see, the code is shorter and somewhat more self-documenting. >> The >> best thing about these suggestions is that as far as I can tell they >> would >> be backwards-compatible API additions. >> > > The `pr.print_stats... line should not be inside the `with` block unless > you want to profile that part as well. > > These suggestions seem fairly uncontroversial. Have you opened an issue > on the issue tracker? > > The fun part of the patch will be the C code, but a Python > proof-of-concept would be useful. > > These changes may not even require C code, since (contrary to its name) cProfile actually is implemented partly in Python. For example, the context manager change could be made simply by adding __enter__ and __exit__ to the cProfile.Profile class. > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

2 1

Re: [Python-ideas] Python Reviewed
by Simon Lovell 10 Jan '17

10 Jan '17

Hi Kyle, I don't see the harm caused from having a do-while construct. Not the most used construct but it is sometimes useful and not having it means you need to either have a first time through marker or a break at the end of a "while True:" loop. I would say that None should also be non-boolean. Otherwise you are saying that things which might be None would be True if not None. Re: SQLAlchemy, this does not resolve the issues satisfactorily to me. Re: half-loop. The idea that you have the same code before entry and at the end of the loop is really ugly and raises the potential for errors. I can remember doing something similarly ugly with this for looping through a cursor but I can't recall why I couldn't just do a .fetchall() and then loop through the results. Maybe I ultimately changed it to do precisely that. Perhaps you have too big a data set to load into memory all at once though and can't do it that way. Anyway, the SQL processing is all too difficult in Python and Java and nearly all modern languages. Re: Conditional expression. You could have: "<result> = if <condition> then <true-value> else <false value". Reading the actual syntax I find requires more concentration than it should. However, this means another keyword*. * General comment: I posted this because Googling didn't give me a satisfactory answer to why Python is the way that it is. I think I see it now. Guido hates keywords. I don't find this particularly logical but it is what it is and it also isn't going to change. That seems also to explain the else keyword at the end of the while loop. Anyway, I think this discussion has reached its natural conclusion here.

4 3

Re: [Python-ideas] Python reviewed
by Simon Lovell 10 Jan '17

10 Jan '17

Thanks for the feedback guys. A few quick comments: Re: Colons. I'm sure I've seen that FAQ before. I may be arrogant but I can't take it seriously. Being "slightly" easier to read is hardly a reason for a mandatory structure. Re: PEP249. I thought I'd detailed quite a bit of what I thought should be possible. Is there a forum for advancing this? Re: do-while - that a is a loop construct that executes once before evaluating the condition. Supported by most languages. Re: Counters starting at zero vs one, Fortran has a neat solution to this for arrays if not strings - allow the programmer to select the starting index. I've seen -1 and 1000, for example. I can't say I'm convinced by Dijkstra's argument but it is somewhat esoteric because it isn't changing. When I've programmed for loops in C, sometimes you want zero based and sometimes one based, while most times you don't really care. To make it readable I would wherever possible write either: for (i=0;i<j;i++) for (i=1;i<=j;i++) // In both cases always executing j times Rgds

7 7