Mailman 3 January 2017 - Python-ideas

Context manager to temporarily set signal handlers
by Thomas Kluyver Jan. 22, 2017

Jan. 22, 2017

Not uncommonly, I want to do something like this in code: import signal # Install my own signal handler prev_hup = signal.signal(signal.SIGHUP, my_handler) prev_term = signal.signal(signal.SIGTERM, my_handler) try: do_something_else() finally: # Restore previous signal handlers signal.signal(signal.SIGHUP, prev_hup) signal.signal(signal.SIGTERM, prev_term) This works if the existing signal handler is a Python function, or the special values SIG_IGN (ignore) or SIG_DFL (default). However, it breaks if code has set a signal handler in C: this is not returned, and there is no way in Python to reinstate a C-level signal handler once we've replaced it from Python. I propose two possible solutions: 1. The high-level approach: a context manager which can temporarily set one or more signal handlers. If this was implemented in C, it could restore C-level as well as Python-level signal handlers. 2. A lower level approach: signal() and getsignal() would gain the ability to return an opaque object which refers to a C-level signal handler. The only use for this would be to pass it back to signal.signal() to set it as a signal handler again. The context manager from (1) could then be implemented in Python. Crosslinking http://bugs.python.org/issue13285 Thomas

2 1

PEP 538 (C locale coercion) now depends on PEP 540 (UTF-8 mode)
by Nick Coghlan Jan. 22, 2017

Jan. 22, 2017

Hi folks, I've pushed an update to PEP 538 that eliminates any fallback to the en_US.UTF-8 locale, and instead relies solely on PEP 540's bypassing of the platform's locale subsystem in cases where the default C locale is specified. This revision means that the two PEPs can now be read as follows: * PEP 540 will make CPython's default behaviour significantly more user friendly when the OS claims that the system API encoding is ASCII, at the cost of making it inconsistent with other C/C++ components in the same process * PEP 538 is then a follow-on proposal to fix that inconsistency by coercing the C/C++ locale settings to better align with CPython's decision to ignore the nominal ASCII encoding I won't repost the whole thing here, but will instead provide relevant links and note a few particular highlights of the changes: * Full PEP: https://www.python.org/dev/peps/pep-0538/ * Diff: https://github.com/python/peps/commit/481573aa2722f515b38e30beaaffc5e1fb9bb… Major changes: * en_US-UTF-8 fallback eliminated in favour of the PYTHONUTF8 feature in PEP 540 * "LC_CTYPE=UTF-8" fallback added with the aim of improving behaviour on *BSD systems * surrogateescape is enabled on standard streams by default (see behavioural examples in the PEP for rationale) * notes that the proposal makes it possible to fix an interaction bug with the GNU readline module on Android I haven't updated the reference implementation yet, as I'd prefer to see PEP 540 resolved and merged before doing that. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

Ideas for improving the struct module
by Elizabeth Myers Jan. 21, 2017

Jan. 21, 2017

Hello, I've noticed a lot of binary protocols require variable length bytestrings (with or without a null terminator), but it is not easy to unpack these in Python without first reading the desired length, or reading bytes until a null terminator is reached. I've noticed the netstruct library (https://github.com/stendec/netstruct) has a format specifier, $, which assumes the previous type to pack/unpack is the string's length. This is an interesting idea in of itself, but doesn't handle the null-terminated string chase. I know $ is similar to pascal strings, but sometimes you need more than 255 characters :p. For null-terminated strings, it may be simpler to have a specifier for those. I propose 0, but this point can be bikeshedded over endlessly if desired ;) (I thought about using n/N but they're :P). It's worth noting that (maybe one of?) Perl's equivalent to the struct module, whose name escapes me atm, has a module which can handle this case. I can't remember if it handled variable length or zero-terminated though; maybe it did both. Perl is more or less my 10th language. :p This pain point is an annoyance imo and would greatly simplify a lot of code if implemented, or something like it. I'd be happy to take a look at implementing it if the idea is received sufficiently warmly. -- Elizabeth

16 41

PEP 538: Coercing the legacy C locale to C.UTF-8
by Nick Coghlan Jan. 18, 2017

Jan. 18, 2017

Hi folks, Many of you would have seen Victor's recent PEP proposing the introduction a new "UTF-8" mode that told Python to use UTF-8 by default in the legacy C locale (similar to the way CPython behaves on Mac OS X, Android and iOS), as well as allowing explicit selection of that mode regardless of the current locale settings. That was prompted by my proposal in PEP 538 to start coercing the legacy C locale to C.UTF-8 (when we have the ability and opportunity to do so), and otherwise at least warn that we don't expect the legacy C locale to work properly. That PEP has now been through its initial round of review on the Python Linux SIG, and updated to address both the feedback received there, as well as some of the points Victor raised in PEP 540. The rendered version is available at https://www.python.org/dev/peps/pep-0538/ and the plain text version is included inline below. Folks that have already read PEP 540 may want to start with the new section that looks at the way the two PEPs are potentially complementary to each other rather than competitive: https://www.python.org/dev/peps/pep-0538/#relationship-with-other-peps In particular, the approach in PEP 540 may be a better last resort alternative than setting "LC_CTYPE=en_US.UTF-8" on platforms that don't provide either C.UTF-8 or C.utf8 (which is what the current draft of PEP 538 proposes) Cheers, Nick. =============================== PEP: 538 Title: Coercing the legacy C locale to C.UTF-8 Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Dec-2016 Python-Version: 3.7 Abstract ======== An ongoing challenge with Python 3 on \*nix systems is the conflict between needing to use the configured locale encoding by default for consistency with other C/C++ components in the same process and those invoked in subprocesses, and the fact that the standard C locale (as defined in POSIX:2001) specifies a default text encoding of ASCII, which is entirely inadequate for the development of networked services and client applications in a multilingual world. This PEP proposes that the way the CPython implementation handles the default C locale be changed such that: * the standalone CPython binary will automatically attempt to coerce the ``C`` locale to ``C.UTF-8`` (preferred), ``C.utf8`` or ``en_US.UTF-8`` unless the new ``PYTHONCOERCECLOCALE`` environment variable is set to ``0`` * if the subsequent runtime initialization process detects that the legacy ``C`` locale remains active (e.g. locale coercion is disabled, or the runtime is embedded in an application other than the main CPython binary), it will emit a warning on stderr that use of the legacy ``C`` locale's default ASCII text encoding may cause various Unicode compatibility issues Explicitly configuring the ``C.UTF-8`` or ``en_US.UTF-8`` locales has already been used successfully for a number of years (including by the PEP author) to get Python 3 running reliably in environments where no locale is otherwise configured (such as Docker containers). With this change, any \*nix platform that does *not* offer at least one of the ``C.UTF-8``, ``C.utf8`` or ``en_US.UTF-8`` locales as part of its standard configuration would only be considered a fully supported platform for CPython 3.7+ deployments when a locale other than the default ``C`` locale is configured explicitly. Redistributors (such as Linux distributions) with a narrower target audience than the upstream CPython development team may also choose to opt in to this behaviour for the Python 3.6.x series by applying the necessary changes as a downstream patch when first introducing Python 3.6.0. Background ========== While the CPython interpreter is starting up, it may need to convert from the ``char *`` format to the ``wchar_t *`` format, or from one of those formats to ``PyUnicodeObject *``, before its own text encoding handling machinery is fully configured. It handles these cases by relying on the operating system to do the conversion and then ensuring that the text encoding name reported by ``sys.getfilesystemencoding()`` matches the encoding used during this early bootstrapping process. On Apple platforms (including both Mac OS X and iOS), this is straightforward, as Apple guarantees that these operations will always use UTF-8 to do the conversion. On Windows, the limitations of the ``mbcs`` format used by default in these conversions proved sufficiently problematic that PEP 528 and PEP 529 were implemented to bypass the operating system supplied interfaces for binary data handling and force the use of UTF-8 instead. On Android, the locale settings are of limited relevance (due to most applications running in the UTF-16-LE based Dalvik environment) and there's limited value in preserving backwards compatibility with other locale aware C/C++ components in the same process (since it's a relatively new target platform for CPython), so CPython bypasses the operating system provided APIs and hardcodes the use of UTF-8 (similar to its behaviour on Apple platforms). On non-Apple and non-Android \*nix systems however, these operations are handled using the C locale system in glibc, which has the following characteristics [4_]: * by default, all processes start in the ``C`` locale, which uses ``ASCII`` for these conversions. This is almost never what anyone doing multilingual text processing actually wants (including CPython and C/C++ GUI frameworks). * calling ``setlocale(LC_ALL, "")`` reconfigures the active locale based on the locale categories configured in the current process environment * if the locale requested by the current environment is unknown, or no specific locale is configured, then the default ``C`` locale will remain active The specific locale category that covers the APIs that CPython depends on is ``LC_CTYPE``, which applies to "classification and conversion of characters, and to multibyte and wide characters" [5_]. Accordingly, CPython includes the following key calls to ``setlocale``: * in the main ``python`` binary, CPython calls ``setlocale(LC_ALL, "")`` to configure the entire C locale subsystem according to the process environment. It does this prior to making any calls into the shared CPython library * in ``Py_Initialize``, CPython calls ``setlocale(LC_CTYPE, "")``, such that the configured locale settings for that category *always* match those set in the environment. It does this unconditionally, and it *doesn't* revert the process state change in ``Py_Finalize`` (This summary of the locale handling omits several technical details related to exactly where and when the text encoding declared as part of the locale settings is used - see PEP 540 for further discussion, as these particular details matter more when decoupling CPython from the declared C locale than they do when overriding the locale with one based on UTF-8) These calls are usually sufficient to provide sensible behaviour, but they can still fail in the following cases: * SSH environment forwarding means that SSH clients will often forward client locale settings to servers that don't have that locale installed. This leads to CPython running in the default ASCII-based C locale * some process environments (such as Linux containers) may not have any explicit locale configured at all. As with unknown locales, this leads to CPython running in the default ASCII-based C locale The simplest way to deal with this problem for currently released versions of CPython is to explicitly set a more sensible locale when launching the application. For example:: LC_ALL=C.UTF-8 LANG=C.UTF-8 python3 ... In the specific case of Docker containers and similar technologies, the appropriate locale setting can be specified directly in the container image definition. Another common failure case is developers specifying ``LANG=C`` in order to see otherwise translated user interface messages in English, rather than the more narrowly scoped ``LC_MESSAGES=C``. Relationship with other PEPs ============================ This PEP shares a common problem statement with PEP 540 (improving Python 3's behaviour in the default C locale), but diverges markedly in the proposed solution: * PEP 540 proposes to entirely decouple CPython's default text encoding from the C locale system in that case, allowing text handling inconsistencies to arise between CPython and other C/C++ components running in the same process and in subprocesses. This approach aims to make CPython behave less like a locale-aware C/C++ application, and more like C/C++ independent language runtimes like the JVM, .NET CLR, Go, Node.js, and Rust * this PEP proposes to instead override the legacy C locale with a more recently defined locale that uses UTF-8 as its default text encoding. This means that the text encoding override will apply not only to CPython, but also to any locale aware extension modules loaded into the current process, as well as to locale aware C/C++ applications invoked in subprocesses that inherit their environment from the parent process. This approach aims to retain CPython's traditional strong support for integration with other components written in C and C++, while actively helping to push forward the adoption and standardisation of the C.UTF-8 locale as a Unicode-aware replacement for the legacy C locale While the two PEPs present alternate proposed behavioural improvements that align with the interests of different parts of the Python user community, they don't actually conflict at a technical level. That means it would be entirely possible to implement both of them, and end up with a situation where redistributors, application integrators, and end users can choose between: * coercing the default ASCII based C locale to a UTF-8 based locale * instructing CPython to ignore the C locale and use UTF-8 instead * doing both of the above (with this option as the default legacy C locale handling) * forcing use of the default ASCII based C locale by setting both PYTHONCOERCECLOCALE=0 and PYTHONUTF8=0 If this approach was taken, then the proposed modifications to PEP 11 would be adjusted to indicate that the only unsupported configurations are those where both the legacy C locale coercion and the C locale text encoding bypass are disabled. Given such a hybrid implementation, it would also be reasonable to drop the ``en_US.UTF-8`` legacy fallback from the list of UTF-8 locales tried as a coercion target and instead rely solely on the C locale text encoding bypass in such cases. Motivation ========== While Linux container technologies like Docker, Kubernetes, and OpenShift are best known for their use in web service development, the related container formats and execution models are also being adopted for Linux command line application development. Technologies like Gnome Flatpak [7_] and Ubunty Snappy [8_] further aim to bring these same techniques to Linux GUI application development. When using Python 3 for application development in these contexts, it isn't uncommon to see text encoding related errors akin to the following:: $ docker run --rm fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")' Unable to decode the command from the command line: UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed $ docker run --rm ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")' Unable to decode the command from the command line: UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed Even though the same command is likely to work fine when run locally:: $ python3 -c 'print("ℙƴ☂ℌøἤ")' ℙƴ☂ℌøἤ The source of the problem can be seen by instead running the ``locale`` command in the three environments:: $ locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG=en_AU.UTF-8 LC_CTYPE="en_AU.UTF-8" LC_ALL= $ docker run --rm fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG= LC_CTYPE="POSIX" LC_ALL= $ docker run --rm ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG= LANGUAGE= LC_CTYPE="POSIX" LC_ALL= In this particular example, we can see that the host system locale is set to "en_AU.UTF-8", so CPython uses UTF-8 as the default text encoding. By contrast, the base Docker images for Fedora and Debian don't have any specific locale set, so they use the POSIX locale by default, which is an alias for the ASCII-based default C locale. The simplest way to get Python 3 (regardless of the exact version) to behave sensibly in Fedora and Debian based containers is to run it in the ``C.UTF-8`` locale that both distros provide:: $ docker run --rm -e LANG=C.UTF-8 fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")' ℙƴ☂ℌøἤ $ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")' ℙƴ☂ℌøἤ $ docker run --rm -e LANG=C.UTF-8 fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG=C.UTF-8 LC_CTYPE="C.UTF-8" LC_ALL= $ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG=C.UTF-8 LANGUAGE= LC_CTYPE="C.UTF-8" LC_ALL= The Alpine Linux based Python images provided by Docker, Inc, already use the C.UTF-8 locale by default:: $ docker run --rm python:3 python3 -c 'print("ℙƴ☂ℌøἤ")' ℙƴ☂ℌøἤ $ docker run --rm python:3 locale | grep -E 'LC_ALL|LC_CTYPE|LANG' LANG=C.UTF-8 LANGUAGE= LC_CTYPE="C.UTF-8" LC_ALL= Similarly, for custom container images (i.e. those adding additional content on top of a base distro image), a more suitable locale can be set in the image definition so everything just works by default. However, it would provide a much nicer and more consistent user experience if CPython were able to just deal with this problem automatically rather than relying on redistributors or end users to handle it through system configuration changes. While the glibc developers are working towards making the C.UTF-8 locale universally available for use by glibc based applications like CPython [6_], this unfortunately doesn't help on platforms that ship older versions of glibc without that feature, and also don't provide C.UTF-8 as an on-disk locale the way Debian and Fedora do. For these platforms, the best widely available fallback option is the ``en_US.UTF-8`` locale, which while still being unfortunately Anglo-centric, is at least significantly less Anglo-centric than the ASCII text encoding assumption in the default C locale. In the specific case of C locale coercion, the Anglo-centrism implied by the use of ``en_US.UTF-8`` can be mitigated by configuring only the ``LC_CTYPE`` locale category, rather than overriding all the locale categories:: $ docker run --rm -e LANG=C.UTF-8 centos/python-35-centos7 python3 -c 'print("ℙƴ☂ℌøἤ")' Unable to decode the command from the command line: UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 7: surrogates not allowed $ docker run --rm -e LC_CTYPE=en_US.UTF-8 centos/python-35-centos7 python3 -c 'print("ℙƴ☂ℌøἤ")' ℙƴ☂ℌøἤ Specification ============= To better handle the cases where CPython would otherwise end up attempting to operate in the ``C`` locale, this PEP proposes that CPython automatically attempt to coerce the legacy ``C`` locale to a UTF-8 based locale when it is run as a standalone command line application. It further proposes to emit a warning on stderr if the legacy ``C`` locale is in effect at the point where the language runtime itself is initialized, in order to warn system and application integrators that they're running CPython in an unsupported configuration. Legacy C locale coercion in the standalone Python interpreter binary -------------------------------------------------------------------- When run as a standalone application, CPython has the opportunity to reconfigure the C locale before any locale dependent operations are executed in the process. This means that it can change the locale settings not only for the CPython runtime, but also for any other C/C++ components running in the current process (e.g. as part of extension modules), as well as in subprocesses that inherit their environment from the current process. After calling ``setlocale(LC_ALL, "")`` to initialize the locale settings in the current process, the main interpreter binary will be updated to include the following call:: const char *ctype_loc = setlocale(LC_CTYPE, NULL); This cryptic invocation is the API that C provides to query the current locale setting without changing it. Given that query, it is possible to check for exactly the ``C`` locale with ``strcmp``:: ctype_loc != NULL && strcmp(ctype_loc, "C") == 0 # true only in the C locale Given this information, CPython can then attempt to coerce the locale to one that uses UTF-8 rather than ASCII as the default encoding. Three such locales will be tried: * ``C.UTF-8`` (available at least in Debian, Ubuntu, and Fedora 25+, and expected to be available by default in a future version of glibc) * ``C.utf8`` (available at least in HP-UX) * ``en_US.UTF-8`` (available at least in RHEL and CentOS) For ``C.UTF-8`` and ``C.utf8``, the coercion will be implemented by actually setting the ``LANG`` and ``LC_ALL`` environment variables to the candidate locale name, such that future calls to ``setlocale()`` will see them, as will other components looking for those settings (such as GUI development frameworks). The last fallback isn't ideal as a coercion target (as it changes more than just the default text encoding), but has the benefit of currently being more widely available than the C.UTF-8 locale. To minimize the chance of side effects, only the ``LC_CTYPE`` environment variable would be set when using this legacy fallback option, with the other locale categories being left alone. Given time, more environments are expected to provide a ``C.UTF-8`` locale by default, so falling all the way back to the ``en_US.UTF-8`` option is expected to become less common. When this locale coercion is activated, the following warning will be printed on stderr, with the warning containing whichever locale was successfully configured:: Python detected LC_CTYPE=C, LC_ALL & LANG set to C.UTF-8 (set PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour). When falling all the way back to the ``en_US.UTF-8`` locale, the message would be slightly different:: Python detected LC_CTYPE=C, LC_CTYPE set to en_US.UTF-8 (set PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour). This locale coercion will mean that the standard Python binary should once again "just work" in the three main failure cases we're aware of (missing locale settings, SSH forwarding of unknown locales, and developers explicitly requesting ``LANG=C``), as long as the target platform provides at least one of the candidate UTF-8 based environments. If ``PYTHONCOERCECLOCALE=0`` is set, or none of the candidate locales is successfully configured, then initialization will continue as usual in the C locale and the Unicode compatibility warning described in the next section will be emitted just as it would for any other application. The interpreter will always check for the ``PYTHONCOERCECLOCALE`` environment variable (even when running under the ``-E`` or ``-I`` switches), as the locale coercion check necessarily takes place before any command line argument processing. Changes to the runtime initialization process --------------------------------------------- By the time that ``Py_Initialize`` is called, arbitrary locale-dependent operations may have taken place in the current process. This means that by the time it is called, it is *too late* to switch to a different locale - doing so would introduce inconsistencies in decoded text, even in the context of the standalone Python interpreter binary. Accordingly, when ``Py_Initialize`` is called and CPython detects that the configured locale is still the default ``C`` locale, the following warning will be issued:: Python runtime initialized with LC_CTYPE=C (a locale with default ASCII encoding), which may cause Unicode compatibility problems. Using C.UTF-8 (if available) as an alternative Unicode-compatible locale is recommended. In this case, no actual change will be made to the locale settings. Instead, the warning informs both system and application integrators that they're running Python 3 in a configuration that we don't expect to work properly. New build-time configuration options ------------------------------------ While both of the above behaviours would be enabled by default, they would also have new associated configuration options and preprocessor definitions for the benefit of redistributors that want to override those default settings. The locale coercion behaviour would be controlled by the flag ``--with[out]-c-locale-coercion``, which would set the ``PY_COERCE_C_LOCALE`` preprocessor definition. The locale warning behaviour would be controlled by the flag ``--with[out]-c-locale-warning``, which would set the ``PY_WARN_ON_C_LOCALE`` preprocessor definition. On platforms where they would have no effect (e.g. Mac OS X, iOS, Android, Windows) these preprocessor variables would always be undefined. Platform Support Changes ======================== A new "Legacy C Locale" section will be added to PEP 11 that states: * as of Python 3.7, the legacy C locale is no longer officially supported, and any Unicode handling issues that occur only in that locale and cannot be reproduced in an appropriately configured non-ASCII locale will be closed as "won't fix" * as of Python 3.7, \*nix platforms are expected to provide at least one of ``C.UTF-8``, ``C.utf8`` or ``en_US.UTF-8`` as an alternative to the legacy ``C`` locale. On platforms which don't yet provide any of these locales, an explicit non-ASCII locale setting will be needed to configure a fully supported environment for running Python 3.7+ Rationale ========= Improving the handling of the C locale -------------------------------------- It has been clear for some time that the C locale's default encoding of ``ASCII`` is entirely the wrong choice for development of modern networked services. Newer languages like Rust and Go have eschewed that default entirely, and instead made it a deployment requirement that systems be configured to use UTF-8 as the text encoding for operating system interfaces. Similarly, Node.js assumes UTF-8 by default (a behaviour inherited from the V8 JavaScript engine) and requires custom build settings to indicate it should use the system locale settings for locale-aware operations. Both the JVM and the .NET CLR use UTF-16-LE as their primary encoding for passing text between applications and the underlying platform. The challenge for CPython has been the fact that in addition to being used for network service development, it is also extensively used as an embedded scripting language in larger applications, and as a desktop application development language, where it is more important to be consistent with other C/C++ components sharing the same process, as well as with the user's desktop locale settings, than it is with the emergent conventions of modern network service development. The core premise of this PEP is that for *all* of these use cases, the default "C" locale is the wrong choice, and furthermore that the following assumptions are valid: * in desktop application use cases, the process locale will *already* be configured appropriately, and if it isn't, then that is an operating system level problem that needs to be reported to and resolved by the operating system provider * in network service development use cases (especially those based on Linux containers), the process locale may not be configured *at all*, and if it isn't, then the expectation is that components will impose their own default encoding the way Rust, Go and Node.js do, rather than trusting the legacy C default encoding of ASCII the way CPython currently does Using "strict" error handling by default ---------------------------------------- By coercing the locale away from the legacy C default and its assumption of ASCII as the preferred text encoding, this PEP also disables the implicit use of the "surrogateescape" error handler on the standard IO streams that was introduced in Python 3.5. This is deliberate, as while UTF-8 as the preferred text encoding is a good working assumption for network service development and for more recent releases of client operating systems, it still isn't a universally valid assumption. In particular, GB 18030 [12_] is a Chinese national text encoding standard that handles all Unicode code points, but is incompatible with both ASCII and UTF-8. Similarly, Shift-JIS [13_] and ISO-2022-JP [14_] remain in widespread use in Japan, and are incompatible with both ASCII and UTF-8. Using strict error handling on the standard streams means that attempting to pass information from a host system using one of these encodings into a container application that is assuming the use of UTF-8 or vice-versa is likely to cause an immediate Unicode encoding or decoding error, rather than potentially causing silent data corruption. Dropping official support for Unicode handling in the legacy C locale --------------------------------------------------------------------- We've been trying to get strict bytes/text separation to work reliably in the legacy C locale for over a decade at this point. Not only haven't we been able to get it to work, neither has anyone else - the only viable alternatives identified have been to pass the bytes along verbatim without eagerly decoding them to text (Python 2.x, Ruby, etc), or else to ignore the nominal C/C++ locale encoding entirely and assume the use of either UTF-8 (PEP 540, Rust, Go, Node.js, etc) or UTF-16-LE (JVM, .NET CLR). While this PEP ensures that developers that need to do so can still opt-in to running their Python code in the legacy C locale, it also makes clear that we *don't* expect Python 3's Unicode handling to be reliable in that configuration, and the recommended alternative is to use a more appropriate locale setting. Providing implicit locale coercion only when running standalone --------------------------------------------------------------- Over the course of Python 3.x development, multiple attempts have been made to improve the handling of incorrect locale settings at the point where the Python interpreter is initialised. The problem that emerged is that this is ultimately *too late* in the interpreter startup process - data such as command line arguments and the contents of environment variables may have already been retrieved from the operating system and processed under the incorrect ASCII text encoding assumption well before ``Py_Initialize`` is called. The problems created by those inconsistencies were then even harder to diagnose and debug than those created by believing the operating system's claim that ASCII was a suitable encoding to use for operating system interfaces. This was the case even for the default CPython binary, let alone larger C/C++ applications that embed CPython as a scripting engine. The approach proposed in this PEP handles that problem by moving the locale coercion as early as possible in the interpreter startup sequence when running standalone: it takes place directly in the C-level ``main()`` function, even before calling in to the `Py_Main()`` library function that implements the features of the CPython interpreter CLI. The ``Py_Initialize`` API then only gains an explicit warning (emitted on ``stderr``) when it detects use of the ``C`` locale, and relies on the embedding application to specify something more reasonable. Querying LC_CTYPE for C locale detection ---------------------------------------- ``LC_CTYPE`` is the actual locale category that CPython relies on to drive the implicit decoding of environment variables, command line arguments, and other text values received from the operating system. As such, it makes sense to check it specifically when attempting to determine whether or not the current locale configuration is likely to cause Unicode handling problems. Setting both LANG & LC_ALL for C.UTF-8 locale coercion ------------------------------------------------------ Python is often used as a glue language, integrating other C/C++ ABI compatible components in the current process, and components written in arbitrary languages in subprocesses. Setting ``LC_ALL`` to ``C.UTF-8`` imposes a locale setting override on all C/C++ components in the current process and in any subprocesses that inherit the current environment. Setting ``LANG`` to ``C.UTF-8`` ensures that even components that only check the ``LANG`` fallback for their locale settings will still use ``C.UTF-8``. Together, these should ensure that when the locale coercion is activated, the switch to the C.UTF-8 locale will be applied consistently across the current process and any subprocesses that inherit the current environment. Allowing restoration of the legacy behaviour -------------------------------------------- The CPython command line interpreter is often used to investigate faults that occur in other applications that embed CPython, and those applications may still be using the C locale even after this PEP is implemented. Providing a simple on/off switch for the locale coercion behaviour makes it much easier to reproduce the behaviour of such applications for debugging purposes, as well as making it easier to reproduce the behaviour of older 3.x runtimes even when running a version with this change applied. Implementation ============== NOTE: The currently posted draft implementation is for a previous iteration of the PEP prior to the incorporation of the feedback noted in [11_]. It was broadly the same in concept (i.e. coercing the legacy C locale to one based on UTF-8), but differs in several details. A draft implementation of the change (including test cases) has been posted to issue 28180 [1_], which is an end user request that ``sys.getfilesystemencoding()`` default to ``utf-8`` rather than ``ascii``. Backporting to earlier Python 3 releases ======================================== Backporting to Python 3.6.0 --------------------------- If this PEP is accepted for Python 3.7, redistributors backporting the change specifically to their initial Python 3.6.0 release will be both allowed and encouraged. However, such backports should only be undertaken either in conjunction with the changes needed to also provide the C.UTF-8 locale by default, or else specifically for platforms where that locale is already consistently available. Backporting to other 3.x releases --------------------------------- While the proposed behavioural change is seen primarily as a bug fix addressing Python 3's current misbehaviour in the default ASCII-based C locale, it still represents a reasonable significant change in the way CPython interacts with the C locale system. As such, while some redistributors may still choose to backport it to even earlier Python 3.x releases based on the needs and interests of their particular user base, this wouldn't be encouraged as a general practice. Acknowledgements ================ The locale coercion approach proposed in this PEP is inspired directly by Armin Ronacher's handling of this problem in the ``click`` command line utility development framework [2_]:: $ LANG=C python3 -c 'import click; cli = click.command()(lambda:None); cli()' Traceback (most recent call last): ... RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps. This system supports the C.UTF-8 locale which is recommended. You might be able to resolve your issue by exporting the following environment variables: export LC_ALL=C.UTF-8 export LANG=C.UTF-8 The change was originally proposed as a downstream patch for Fedora's system Python 3.6 package [3_], and then reformulated as a PEP for Python 3.7 with a section allowing for backports to earlier versions by redistributors. The initial draft was posted to the Python Linux SIG for discussion [10_] and then amended based on both that discussion and Victor Stinner's work in PEP 540 [11_]. The "ℙƴ☂ℌøἤ" string used in the Unicode handling examples throughout this PEP is taken from Ned Batchelder's excellent "Pragmatic Unicode" presentation [9_]. References ========== .. [1] CPython: sys.getfilesystemencoding() should default to utf-8 (http://bugs.python.org/issue28180) .. [2] Locale configuration required for click applications under Python 3 (http://click.pocoo.org/5/python3/#python-3-surrogate-handling) .. [3] Fedora: force C.UTF-8 when Python 3 is run under the C locale (https://bugzilla.redhat.com/show_bug.cgi?id=1404918) .. [4] GNU C: How Programs Set the Locale ( https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html) .. [5] GNU C: Locale Categories (https://www.gnu.org/software/libc/manual/html_node/Locale-Categories.html) .. [6] glibc C.UTF-8 locale proposal (https://sourceware.org/glibc/wiki/Proposals/C.UTF-8) .. [7] GNOME Flatpak (http://flatpak.org/) .. [8] Ubuntu Snappy (https://www.ubuntu.com/desktop/snappy) .. [9] Pragmatic Unicode (http://nedbatchelder.com/text/unipain.html) .. [10] linux-sig discussion of initial PEP draft (https://mail.python.org/pipermail/linux-sig/2017-January/000014.html) .. [11] Feedback notes from linux-sig discussion and PEP 540 (https://github.com/python/peps/issues/171) .. [12] GB 18030 (https://en.wikipedia.org/wiki/GB_18030) .. [13] Shift-JIS (https://en.wikipedia.org/wiki/Shift_JIS) .. [14] ISO-2022 (https://en.wikipedia.org/wiki/ISO/IEC_2022) Copyright ========= This document has been placed in the public domain under the terms of the CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

2 1

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

5 13

Fused multiply-add (FMA)
by Juraj Sukop Jan. 17, 2017

Jan. 17, 2017

Hello! Fused multiply-add (henceforth FMA) is an operation which calculates the product of two numbers and then the sum of the product and a third number with just one floating-point rounding. More concretely: r = x*y + z The value of `r` is the same as if the RHS was calculated with infinite precision and the rounded to a 32-bit single-precision or 64-bit double-precision floating-point number [1]. Even though one FMA CPU instruction might be calculated faster than the two separate instructions for multiply and add, its main advantage comes from the increased precision of numerical computations that involve the accumulation of products. Examples which benefit from using FMA are: dot product [2], compensated arithmetic [3], polynomial evaluation [4], matrix multiplication, Newton's method and many more [5]. C99 includes `fma` function to `math.h` [6] and emulates the calculation if the FMA instruction is not present on the host CPU [7]. PEP 7 states that "Python versions greater than or equal to 3.6 use C89 with several select C99 features" and that "Future C99 features may be added to this list in the future depending on compiler support" [8]. This proposal is then about adding new `fma` function with the following signature to `math` module: math.fma(x, y, z) '''Return a float representing the result of the operation `x*y + z` with single rounding error, as defined by the platform C library. The result is the same as if the operation was carried with infinite precision and rounded to a floating-point number.''' There is a simple module for Python 3 demonstrating the fused multiply-add operation which was build with simple `python3 setup.py build` under Linux [9]. Any feedback is greatly appreciated! Juraj Sukop [1] https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation [2] S. Graillat, P. Langlois, N. Louvet. Accurate dot products with FMA. 2006 [3] S. Graillat, Accurate Floating Point Product and Exponentiation. 2007. [4] S. Graillat, P. Langlois, N. Louvet. Improving the compensated Horner scheme with a Fused Multiply and Add. 2006 [5] J.-M. Muller, N. Brisebarre, F. de Dinechin, C.-P. Jeannerod, V. Lefèvre, G. Melquiond, N. Revol, D. Stehlé, S. Torres. Handbook of Floating-Point Arithmetic. 2010. Chapter 5 [6] ISO/IEC 9899:TC3, "7.12.13.1 The fma functions", Committee Draft - Septermber 7, 2007 [7] https://git.musl-libc.org/cgit/musl/tree/src/math/fma.c [8] https://www.python.org/dev/peps/pep-0007/ [9] https://github.com/sukop/fma

11 16

Python dependancies
by Mathieu TORTUYAUX Jan. 16, 2017

Jan. 16, 2017

Hello everyone, I'm used to work with python and contribute to open-source projects. And now, many projects need to run with dependancies. So I wondering, if it could be a good idea to integrate a sniffer into Python to detecte if project's dependancies are up to date. And each time Python project is run developer will be aware if dependancies are up to date. I think isn't the first time that this idea is submitted. So I am looking forward your feedbacks ! Mathieu Tortuyaux

5 5

[PEP-0541] On issues with reclaiming namespaces in PyPi
by Chris Rose Jan. 15, 2017

Jan. 15, 2017

I want to address one gap in the PEP regarding reclaiming abandoned names: Version reuse. The problem with reusing names is that existing applications or installations that reference the old one, unless they pin the version name precisely. Even in that case, I foresee issues with version collision, especially if the abandoned project was well-versioned in the same model (semver or otherwise) that the new project uses. I'm deeply concerned by the idea of installer code suddenly picking up a new project... with possibly different dependencies on its own, either with old or clashing versions. I recognize it's going to be rare, but these incidents will definitely impact the repeatability of builds depending on PyPi. I think the criteria for reuse of a name must include usage limits; if the package is being downloaded on a steady basis by accounts that can't be shown to belong to known integration systems, reuse should not be allowed. -- Chris R. ====== Not to be taken literally, internally, or seriously. Twitter: http://twitter.com/offby1

2 1

Things that won't change (proposed PEP)
by Steven D'Aprano Jan. 13, 2017

Jan. 13, 2017

I have a proposal for an Informational PEP that lists things which will not change in Python. If accepted, it could be linked to from the signup page for the mailing list, and be the one obvious place to point newcomers to if they propose the same old cliches. Thoughts? * * * * * * * * * * PEP: XXX Title: Things that won't change in Python Version: $Revision$ Last-Modified: $Date$ Author: Steven D'Aprano <steve(a)pearwood.info> Status: Draft Type: Informational Content-Type: text/x-rst Created: 11-Jan-2017 Post-History: 12-Jan-2017 Abstract ======== This PEP documents things which will not change in future versions of Python. Rationale ========= This PEP hopes to reduce the noise on `Python-Ideas <https://mail.python.org/mailman/listinfo/python-ideas>`_ and other mailing lists. If you have a proposal for future Python development, and it requires changing one of the things listed here, it is dead in the water and has **no chance of being accepted**, either because the benefit is too little, the cost of changing the language (including backwards compatibility) is too high, or simply because it goes against the design preferred by the BDFL. Many of these things are already listed in the `FAQs <https://docs.python.org/3/faq/design.html>`_. You should be familiar with both Python and the FAQs before proposing changes to the language. Just because something is not listed here does not necessarily mean that it will be changed. Each proposal will be weighed on its merits, costs compared to benefits. Sometimes the decision will come down to a matter of subjective taste, in which case the BDFL has the final say. Language Direction ================== Python 3 -------- This shouldn't need saying, but Python 3 will not be abandoned. Python 2.8 ---------- There will be `no official Python 2.8 <https://www.python.org/dev/peps/pep-0404/>`_ , although third parties are welcome to fork the language, backport Python 3 features, and maintain the hybrid themselves. Just don't call it "Python 2.8", or any other name which gives the impression that it is maintained by the official Python core developers. Type checking ------------- Duck-typing remains a fundamental part of Python and `type checking <https://www.python.org/dev/peps/pep-0484/#non-goals>`_ will not be mandatory. Even if the Python interpreter someday gains a built-in type checker, it will remain optional. Syntax ====== Braces ------ It doesn't matter whether optional or mandatory, whether spelled ``{ }`` like in C, or ``BEGIN END`` like in Pascal, braces to delimit code blocks are not going to happen. For another perspective on this, try running ``from __future__ import braces`` at the interactive interpreter. (There is a *tiny* loophole here: multi-statement lambda, or Ruby-like code blocks have not been ruled out. Such a feature may require some sort of block delimiter -- but it won't be braces, as they clash with the syntax for dicts and sets.) Colons after statements that introduce a block ---------------------------------------------- Statements which introduce a code block, such as ``class``, ``def``, or ``if``, require a colon. Colons have been found to increase readability. See the `FAQ <https://docs.python.org/3/faq/design.html#why-are-colons-required-for-the-i…>`_ for more detail. End statements -------------- Python does not use ``END`` statements following blocks. Given significant indentation, they are redundant and add noise to the source code. If you really want end markers, use a comment ``# end``. Explicit self ------------- Explicit ``self`` is a feature, not a bug. See the `FAQ <https://docs.python.org/3/faq/design.html#why-must-self-be-used-explicitly-…>`_ for more detail. Print function -------------- The ``print`` statement in Python 1 and 2 was a mistake that Guido long regretted. Now that it has been corrected in Python 3, it will not be reverted back to a statement. See `PEP 3105 <https://www.python.org/dev/peps/pep-3105/>`_ for more details. Significant indentation ----------------------- `Significant indentation <https://docs.python.org/3/faq/design.html#why-does-python-use-indentation-f…>`_ is a core feature of Python. Other syntax ------------ Python will not use ``$`` as syntax. Guido doesn't like it. (But it is okay to use ``$`` in DSLs like template strings and regular expressions.) Built-in Functions And Types ============================ Strings ------- Strings are `immutable <https://docs.python.org/3/faq/design.html#why-are-python-strings-immutable>`_ and represent Unicode code points, not bytes. Bools ----- ``bool`` is a subclass of ``int``, with ``True == 1`` and ``False == 0``. This is mostly for historical reasons, but the benefit of changing it now is too low to be worth breaking backwards compatibility and the enormous disruption it would cause. Built-in functions ------------------ Python is an object-oriented language, but it is not *purely* object-oriented. Not everything needs to be `a method of some object <http://steve-yegge.blogspot.com.au/2006/03/execution-in-kingdom-of-nouns.ht…>`_, and functions have their advantages. See the `FAQ <https://docs.python.org/3/faq/design.html#why-does-python-use-methods-for-s…>`_ for more detail. Other Language Features ======================= The interactive interpreter --------------------------- The default prompt is ``>>> ``. Guido likes it that way. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

22 29

Settable defaulting to decimal instead of float
by George Fischhof Jan. 13, 2017

Jan. 13, 2017

Hi There, Settable defaulting to decimal instead of float It would be good to be able to use decimal automatically instead of float if there is a setting. For example an environment variable or a flag file. Where and when accuracy is more important than speed, the user could set this flag, and calculate with decimal numbers as learned in the school. I think several people would use this function Best regards, George

12 17