Python-Dev
Threads by month
- ----- 2025 -----
- February
- January
- ----- 2024 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
May 2017
- 79 participants
- 47 discussions
Hello, everyone. I wanted to contribute to Python so I began by following
the steps given here: https://docs.python.org/devguide/
WHile executing
./python -m test -j3
I encountered the following error:
383 tests OK.
1 test failed:
test_re
21 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
test_msilib test_ossaudiodev test_smtpnet test_socketserver
test_startfile test_timeout test_tix test_tk test_ttk_guionly
test_urllib2net test_urllibnet …
[View More]test_winconsoleio test_winreg
test_winsound test_xmlrpc_net test_zipfile64
Total duration: 24 min 25 sec
Tests result: FAILURE
How do I resolve this?
Regards,
Pranav.
[View Less]
4
5
ACTIVITY SUMMARY (2017-05-05 - 2017-05-12)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 5958 (+29)
closed 36149 (+40)
total 42107 (+69)
Open issues with patches: 2384
Issues opened (47)
==================
#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273 reopened by haypo
#30283: [2.7] …
[View More]Backport test_regrtest (partially) on Python 2.7
http://bugs.python.org/issue30283 opened by haypo
#30284: Build CPython out of tree with a read-only source tree
http://bugs.python.org/issue30284 opened by haypo
#30287: cpython and Clang Static Analyzer
http://bugs.python.org/issue30287 opened by dilyan.palauzov
#30290: IDLE: add tests for help_about.py
http://bugs.python.org/issue30290 opened by terry.reedy
#30291: Allow windows launcher to specify bit lengths with & without m
http://bugs.python.org/issue30291 opened by Steve Barnes
#30294: ./configure, pydebug and pymalloc
http://bugs.python.org/issue30294 opened by dilyan.palauzov
#30295: msvcrt SetErrorMode not documented
http://bugs.python.org/issue30295 opened by giampaolo.rodola
#30296: Remove unnecessary tuples, lists, sets, and dicts from Lib
http://bugs.python.org/issue30296 opened by jdufresne
#30299: Display the bytecode when compiled a regular expression in deb
http://bugs.python.org/issue30299 opened by serhiy.storchaka
#30300: asyncio.Controller
http://bugs.python.org/issue30300 opened by barry
#30301: multiprocessing: AttributeError: 'SimpleQueue' object has no a
http://bugs.python.org/issue30301 opened by Daniel Moore
#30302: Improve .__repr__ implementation for datetime.timedelta
http://bugs.python.org/issue30302 opened by musically_ut
#30303: IDLE: Add _utest to textview
http://bugs.python.org/issue30303 opened by louielu
#30304: TestCase.assertMultiLineEqual only registered for Unicode stri
http://bugs.python.org/issue30304 opened by martin.panter
#30306: release arguments of contextmanager
http://bugs.python.org/issue30306 opened by Martin.Teichmann
#30310: tkFont.py assumes that all font families are encoded as ascii
http://bugs.python.org/issue30310 opened by culler
#30312: Small correction in set code sample
http://bugs.python.org/issue30312 opened by mcocdawc
#30313: Tests of Python 2.7 VS9.0 buildbots must be run with -uall -rw
http://bugs.python.org/issue30313 opened by haypo
#30314: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 b
http://bugs.python.org/issue30314 opened by haypo
#30315: test_ftplib.TestTLS_FTPClass: "[Errno 54] Connection reset by
http://bugs.python.org/issue30315 opened by haypo
#30316: test_default_timeout() of test_threading.BarrierTests: random
http://bugs.python.org/issue30316 opened by haypo
#30317: test_timeout() of test_multiprocessing_spawn.WithManagerTestBa
http://bugs.python.org/issue30317 opened by haypo
#30318: test_distutils is too verbose on Windows
http://bugs.python.org/issue30318 opened by haypo
#30319: test_invalid_authentication() of test_imaplib: ConnectionRese
http://bugs.python.org/issue30319 opened by haypo
#30323: concurrent.futures.Executor.map() consumes all memory when big
http://bugs.python.org/issue30323 opened by Klamann
#30325: Buildbot: send email notifications to buildbot-status@
http://bugs.python.org/issue30325 opened by haypo
#30328: test_ssl.test_connect_with_context(): ConnectionResetError on
http://bugs.python.org/issue30328 opened by haypo
#30329: test_imaplib.test_login_cram_md5(): OSError: [WinError 10022]
http://bugs.python.org/issue30329 opened by haypo
#30330: test_socket.test_idna(): socket.gaierror: [Errno 11001] getadd
http://bugs.python.org/issue30330 opened by haypo
#30331: TestPOP3_TLSClass: socket.timeout: timed out on AMD64 FreeBSD
http://bugs.python.org/issue30331 opened by haypo
#30333: test_multiprocessing_forkserver: poll() failed on AMD64 FreeBS
http://bugs.python.org/issue30333 opened by haypo
#30335: Document deprecated alias of assertNotRegex
http://bugs.python.org/issue30335 opened by Jim Fasarakis-Hilliard
#30337: Vague wording of pkgutil.walk_packages parameter 'prefix'
http://bugs.python.org/issue30337 opened by smsilb
#30339: test_multiprocessing_main_handling: "RuntimeError: Timed out w
http://bugs.python.org/issue30339 opened by haypo
#30340: Optimize out non-capturing groups
http://bugs.python.org/issue30340 opened by serhiy.storchaka
#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341 opened by xiang.zhang
#30343: Subclassed json.JSONEncoder does not respect default method fo
http://bugs.python.org/issue30343 opened by Xophmeister
#30344: test_multiprocessing.test_notify_all(): AssertionError: 6 != 5
http://bugs.python.org/issue30344 opened by haypo
#30345: test_gdb fails on Python 3.6 when built with LTO+PGO
http://bugs.python.org/issue30345 opened by haypo
#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346 opened by Matt Gilson
#30347: itertools.groupby() can fail a C assert()
http://bugs.python.org/issue30347 opened by arigo
#30348: IDLE: Add fetch completions and get entity unittest
http://bugs.python.org/issue30348 opened by louielu
#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349 opened by serhiy.storchaka
#30350: devguide suggests to use VS 2008 to build Python 2.7, but VS 2
http://bugs.python.org/issue30350 opened by haypo
#30351: regrtest hangs on x86 Windows XP 2.7
http://bugs.python.org/issue30351 opened by haypo
#30352: The 'in' syntax should work with any object that implements __
http://bugs.python.org/issue30352 opened by Edouard KLEIN
Most recent 15 issues with no replies (15)
==========================================
#30351: regrtest hangs on x86 Windows XP 2.7
http://bugs.python.org/issue30351
#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349
#30348: IDLE: Add fetch completions and get entity unittest
http://bugs.python.org/issue30348
#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341
#30339: test_multiprocessing_main_handling: "RuntimeError: Timed out w
http://bugs.python.org/issue30339
#30333: test_multiprocessing_forkserver: poll() failed on AMD64 FreeBS
http://bugs.python.org/issue30333
#30329: test_imaplib.test_login_cram_md5(): OSError: [WinError 10022]
http://bugs.python.org/issue30329
#30328: test_ssl.test_connect_with_context(): ConnectionResetError on
http://bugs.python.org/issue30328
#30318: test_distutils is too verbose on Windows
http://bugs.python.org/issue30318
#30314: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 b
http://bugs.python.org/issue30314
#30304: TestCase.assertMultiLineEqual only registered for Unicode stri
http://bugs.python.org/issue30304
#30303: IDLE: Add _utest to textview
http://bugs.python.org/issue30303
#30301: multiprocessing: AttributeError: 'SimpleQueue' object has no a
http://bugs.python.org/issue30301
#30290: IDLE: add tests for help_about.py
http://bugs.python.org/issue30290
#30282: object returned by tarfile.extractfile has an incorrect value
http://bugs.python.org/issue30282
Most recent 15 issues waiting for review (15)
=============================================
#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349
#30347: itertools.groupby() can fail a C assert()
http://bugs.python.org/issue30347
#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341
#30340: Optimize out non-capturing groups
http://bugs.python.org/issue30340
#30325: Buildbot: send email notifications to buildbot-status@
http://bugs.python.org/issue30325
#30310: tkFont.py assumes that all font families are encoded as ascii
http://bugs.python.org/issue30310
#30306: release arguments of contextmanager
http://bugs.python.org/issue30306
#30299: Display the bytecode when compiled a regular expression in deb
http://bugs.python.org/issue30299
#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262
#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248
#30246: fix several error messages in struct
http://bugs.python.org/issue30246
#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242
#30231: test_imaplib needs a TLS server accepting self-signed certific
http://bugs.python.org/issue30231
#30224: remove outdated checks in struct
http://bugs.python.org/issue30224
#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219
Top 10 most discussed issues (10)
=================================
#30283: [2.7] Backport test_regrtest (partially) on Python 2.7
http://bugs.python.org/issue30283 14 msgs
#15786: IDLE code completion window can hang or misbehave with mouse
http://bugs.python.org/issue15786 10 msgs
#29243: --enable-optimizations makes common build commands always need
http://bugs.python.org/issue29243 10 msgs
#29447: Add/check os.PathLike support for the tempfile module's 'dir'
http://bugs.python.org/issue29447 7 msgs
#30181: Correct the parsing of a test case docstring.
http://bugs.python.org/issue30181 7 msgs
#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219 7 msgs
#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262 7 msgs
#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273 6 msgs
#30291: Allow windows launcher to specify bit lengths with & without m
http://bugs.python.org/issue30291 6 msgs
#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346 6 msgs
Issues closed (38)
==================
#26336: Expose regex bytecode of compiled pattern object
http://bugs.python.org/issue26336 closed by terry.reedy
#26700: Make digest_size a class variable
http://bugs.python.org/issue26700 closed by gregory.p.smith
#28787: Out of tree --with--dtrace builds fail with a traceback
http://bugs.python.org/issue28787 closed by haypo
#29823: mimetypes guesses XSL mimetype when passed an XML file
http://bugs.python.org/issue29823 closed by martin.panter
#29956: math.exp documentation is misleading
http://bugs.python.org/issue29956 closed by belopolsky
#29979: cgi.parse_multipart is not consistent with FieldStorage
http://bugs.python.org/issue29979 closed by orsenthil
#29990: Range checking in GB18030 decoder
http://bugs.python.org/issue29990 closed by xiang.zhang
#30024: Treat `import a.b.c as m` as `m = sys.modules['a.b.c']`
http://bugs.python.org/issue30024 closed by serhiy.storchaka
#30048: If a task is canceled at the right moment, the cancellation is
http://bugs.python.org/issue30048 closed by inada.naoki
#30168: Class Logger is unindented in the documentation.
http://bugs.python.org/issue30168 closed by vinay.sajip
#30218: shutil.unpack_archive doesn't support PathLike
http://bugs.python.org/issue30218 closed by brett.cannon
#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272 closed by berker.peksag
#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275 closed by brett.cannon
#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276 closed by bmwiedemann
#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281 closed by xiang.zhang
#30285: Optimize case-insensitive regular expressions
http://bugs.python.org/issue30285 closed by serhiy.storchaka
#30286: ctypes FreeLibrary fails on Windows 64-bit
http://bugs.python.org/issue30286 closed by giampaolo.rodola
#30288: ssl.wrap_ssl will fail on do_handshake if default parameters a
http://bugs.python.org/issue30288 closed by botter
#30289: make distclean and Misc/python-config.sh
http://bugs.python.org/issue30289 closed by xiang.zhang
#30292: Why is there a concept "unbound method" in <Descriptor HowTo G
http://bugs.python.org/issue30292 closed by martin.panter
#30293: Peephole binops folding can lead to memory and bytecache ballo
http://bugs.python.org/issue30293 closed by rhettinger
#30297: Recursive starmap causes Segmentation fault
http://bugs.python.org/issue30297 closed by rhettinger
#30298: Weak deprecations for inline regular expression modifiers
http://bugs.python.org/issue30298 closed by serhiy.storchaka
#30305: python 2.7.13 join issue
http://bugs.python.org/issue30305 closed by ezio.melotti
#30307: https://docs.python.org/3/tutorial/introduction.html#strings S
http://bugs.python.org/issue30307 closed by serhiy.storchaka
#30308: Add code coverage for argument in random.shuffle
http://bugs.python.org/issue30308 closed by rhettinger
#30311: random.shuffle pointlessly shuffles dicts
http://bugs.python.org/issue30311 closed by rhettinger
#30320: test_eintr.test_sigwaitinfo(): race condition on AMD64 FreeBSD
http://bugs.python.org/issue30320 closed by haypo
#30321: format() function prints fillchar as backslash twice
http://bugs.python.org/issue30321 closed by zach.ware
#30322: PyObject_GetIter does not behave as documented on dict objects
http://bugs.python.org/issue30322 closed by Sam De Meyer
#30324: Error using newline='' when writing to CSV file
http://bugs.python.org/issue30324 closed by martin.panter
#30326: Version may be out of date in travis.yml
http://bugs.python.org/issue30326 closed by Jensen Taylor
#30327: Version may be out of date in travis.yml
http://bugs.python.org/issue30327 closed by brett.cannon
#30332: Errors that aren't actually defined are purple in the IDLE
http://bugs.python.org/issue30332 closed by terry.reedy
#30334: [Windows] support.rmtree() should retry on WindowsError: [Erro
http://bugs.python.org/issue30334 closed by haypo
#30336: Pull Requests need more detail
http://bugs.python.org/issue30336 closed by brett.cannon
#30338: LC_ALL=en_US + io.open() => LookupError: (osx)
http://bugs.python.org/issue30338 closed by haypo
#30342: [2.7] sysconfig.is_python_build() doesn't work if Python is bu
http://bugs.python.org/issue30342 closed by haypo
[View Less]
1
0

May 11, 2017
Hi,
You can now subscribe to this mailing list to get email notifications
when the state of a buildbot changes from success (green)/warning
(orange) to failure (red):
https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/
Only these state changes (green=>red, orange=>red) send an email notification.
See http://bugs.python.org/issue30325 for the technical details on how
these emails are sent.
Thanks to Zachary Ware for his strong support on creating this list
and fixing …
[View More]buildbot issues ;-)
Victor
[View Less]
1
0
A comment on a recent SO answer [1] wondered why my aenum library wasn't mentioned in the docs to help guide people that
needed/wanted more advanced Enum options to it. I responded that Python was not in the habit of mentioning third-party
libraries in the docs.
However, I thought I would double-check here to see if it might be a good idea.
Pros:
- drop-in replacement for the stdlib Enum
- has many advanced features such as
- auto __init__ building
- multi-value members
- …
[View More]duplicate value but non-aliasing members
- etc.
- I'm the primary/only maintainer for both
Cons:
- third-party library
Thoughts?
--
~Ethan~
[1] http://stackoverflow.com/a/43855536/208880
[View Less]
6
6
Not sure where to ask about this, so I'm asking here.
In the on-line docs, at the very bottom of a page, in fine print, is a link: _Find a bug?_ Following that link leads to
a short page with some advice on how to handle it. Under the second heading [1] is this paragraph:
> If you’re short on time, you can also email documentation bug reports
> to docs(a)python.org (behavioral bugs can be sent to python-list(a)python.org).
> ‘docs@’ is a mailing list run by volunteers; your …
[View More]request will be noticed,
> though it may take a while to be processed.
Why is python-list the place to send behavioral bugs to? It's been my experience that folks there will (rightly) ask
the individual to file a bug on the tracker.
--
~Ethan~
[1] https://docs.python.org/3/bugs.html#documentation-bugs
[View Less]
5
5

PEP 484 proposal: don't default to Optional if argument default is None
by Guido van Rossum May 10, 2017
by Guido van Rossum May 10, 2017
May 10, 2017
There's a proposal to change one detail of PEP 484. It currently says:
An optional type is also automatically assumed when the default value is
None, for example::
def handle_employee(e: Employee = None): ...
This is equivalent to::
def handle_employee(e: Optional[Employee] = None) -> None: ...
Now that we've got some experience actually using Optional with mypy
(originally mypy ignored Optional), we're beginning to think that this was
a bad idea. There's more discussion at
https://…
[View More]github.com/python/typing/issues/275 and an implementation of the
change (using a command-line flag) in
https://github.com/python/mypy/pull/3248.
Thoughts? Some function declarations will become a bit more verbose, but we
gain clarity (many users of annotations don't seem to be familiar with this
feature) and consistency (since this rule doesn't apply to variable
declarations and class attribute declarations).
--
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
[View Less]
5
5
Hi,
tl;dr Are you ok to backport my change replacing "make touch" with
"make regen-all"? (commit a5c62a8e)
Since the creation of CPython, generated files were regenerated
depending on file modification time. For development, that's a
convenient feature. But in practice, it caused a long list of pain
points. It caused me many issues in my experience:
* On Solaris, Python failed to regenerated the AST because only the
system Python was Python 2.6 and the script required Python 2.7 or
newer. …
[View More]The "make touch" workaround didn't help, also because of the
old version of the system Python.
* On FreeBSD, generated files require "python" but only "python2.7" or
"python3.6" programs are available. In The build system was enhanced
to try pythonX.Y and then "python".
* "make touch" workaround requires Mercurial, but also a specific
version of Mercurial: more than once, "make touch" failed because my
Mercurial version was too old.
* Since CPython migrated to Git, "make touch" doesn't work anymore.
Sorry, I didn't check why exactly, but I would prefer to not depend on
Git *and* Mercurial.
For all these reasons, it was decided to modify the CPython (UNIX/BSD)
build system to not regenerate generated files based on file
modification time anymore, but require an explicit action: "make
regen-all". I already pushed my change to the master branch:
https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd…
---
commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b
Author: Victor Stinner <victor.stinner(a)gmail.com>
Date: Wed May 3 18:21:48 2017 +0200
bpo-23404: make touch becomes make regen-all (#1405)
Don't rebuild generated files based on file modification time
anymore, the action is now explicit. Replace "make touch"
with "make regen-all".
Changes:
* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch
* Add a new "make regen-all" command to rebuild all generated files
* Add subcommands to only generate specific files:
- regen-ast: Include/Python-ast.h and Python/Python-ast.c
- regen-grammar: Include/graminit.h and Python/graminit.c
- regen-importlib: Python/importlib_external.h and Python/importlib.h
- regen-opcode: Include/opcode.h
- regen-opcode-targets: Python/opcode_targets.h
- regen-typeslots: Objects/typeslots.inc
* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN
* pgen is now only built by by "make regen-grammar"
* Add $(srcdir)/ prefix to paths to source files to handle correctly
compilation outside the source directory
Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make"
default target building Python.
---
See the issue for the full rationale:
http://bugs.python.org/issue23404
My commit fixed the two remaining FreeBSD buildbots which were broken
because of broken "make touch". All FreeBSD buildbots are repaired on
master!
Ok, now the question is: are you ok to backport this change to Python
2.7, 3.5 and 3.6?
I started with a backport to 3.6:
https://github.com/python/cpython/pull/1461
See also "Test somehow that generated files are up to date: run make
regen-all" issue:
http://bugs.python.org/issue30259
Victor
[View Less]
6
9

May 9, 2017
Hi folks,
Late last year I started working on a change to the CPython CLI (*not* the
shared library) to get it to coerce the legacy C locale to something based
on UTF-8 when a suitable locale is available.
After a couple of rounds of iteration on linux-sig and python-ideas, I'm
now bringing it to python-dev as a concrete proposal for Python 3.7.
For most folks, reading the Abstract plus the draft docs updates in the
reference implementation will tell you everything you need to know (if the
C.…
[View More]UTF-8, C.utf8 or UTF-8 locales are available, the CLI will automatically
attempt to coerce the legacy C locale to one of those rather than
persisting with the latter's default assumption of ASCII as the preferred
text encoding).
However, the full PEP goes into a lot more detail on:
* exactly what's broken about CPython's behaviour in the legacy C locale
* why I'm in favour of this particular approach to fixing it (i.e. it
integrates better with other C/C++ components, as well as being amenable to
redistributor backports for 3.6, and environment based configuration for
3.5 and earlier)
* why I think implementing both this change *and* Victor's more
comprehensive "PYTHONUTF8 mode" proposal in PEP 540 will be better than
implementing just one or the other (in some situations, ignoring the
platform locale subsystem entirely really is the right approach, and that's
the aspect PEP 540 tackles, while this PEP tackles the situations where the
C locale behaviour is broken, but you still need to be consistent with the
platform settings).
Cheers,
Nick.
==================================
PEP: 538
Title: Coercing the legacy C locale to a UTF-8 based locale
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 28-Dec-2016
Python-Version: 3.7
Post-History: 03-Jan-2017 (linux-sig),
07-Jan-2017 (python-ideas),
05-Mar-2017 (python-dev)
Abstract
========
An ongoing challenge with Python 3 on \*nix systems is the conflict between
needing to use the configured locale encoding by default for consistency
with
other C/C++ components in the same process and those invoked in
subprocesses,
and the fact that the standard C locale (as defined in POSIX:2001) typically
implies a default text encoding of ASCII, which is entirely inadequate for
the
development of networked services and client applications in a multilingual
world.
PEP 540 proposes a change to CPython's handling of the legacy C locale such
that CPython will assume the use of UTF-8 in such environments, rather than
persisting with the demonstrably problematic assumption of ASCII as an
appropriate encoding for communicating with operating system interfaces.
This is a good approach for cases where network encoding interoperability
is a more important concern than local encoding interoperability.
However, it comes at the cost of making CPython's encoding assumptions
diverge
from those of other C and C++ components in the same process, as well as
those
of components running in subprocesses that share the same environment.
It also requires changes to the internals of how CPython itself works,
rather
than using existing configuration settings that are supported by Python
versions prior to Python 3.7.
Accordingly, this PEP proposes that independently of the UTF-8 mode proposed
in PEP 540, the way the CPython implementation handles the default C locale
be
changed such that:
* unless the new ``PYTHONCOERCECLOCALE`` environment variable is set to
``0``,
the standalone CPython binary will automatically attempt to coerce the
``C``
locale to the first available locale out of ``C.UTF-8``, ``C.utf8``, or
``UTF-8``
* if the locale is successfully coerced, and PEP 540 is not accepted, then
``PYTHONIOENCODING`` (if not otherwise set) will be set to
``utf-8:surrogateescape``.
* if the locale is successfully coerced, and PEP 540 *is* accepted, then
``PYTHONUTF8`` (if not otherwise set) will be set to ``1``
* if the subsequent runtime initialization process detects that the legacy
``C`` locale remains active (e.g. none of ``C.UTF-8``, ``C.utf8`` or
``UTF-8``
are available, locale coercion is disabled, or the runtime is embedded in
an
application other than the main CPython binary), and the ``PYTHONUTF8``
feature defined in PEP 540 is also disabled (or not implemented), it will
emit a warning on stderr that use of the legacy ``C`` locale's default
ASCII
text encoding may cause various Unicode compatibility issues
With this change, any \*nix platform that does *not* offer at least one of
the
``C.UTF-8``, ``C.utf8`` or ``UTF-8`` locales as part of its standard
configuration would only be considered a fully supported platform for
CPython
3.7+ deployments when either the new ``PYTHONUTF8`` mode defined in PEP 540
is
used, or else a suitable locale other than the default ``C`` locale is
configured explicitly (e.g. `en_AU.UTF-8`, ``zh_CN.gb18030``).
Redistributors (such as Linux distributions) with a narrower target audience
than the upstream CPython development team may also choose to opt in to this
locale coercion behaviour for the Python 3.6.x series by applying the
necessary
changes as a downstream patch when first introducing Python 3.6.0.
Background
==========
While the CPython interpreter is starting up, it may need to convert from
the ``char *`` format to the ``wchar_t *`` format, or from one of those
formats
to ``PyUnicodeObject *``, in a way that's consistent with the locale
settings
of the overall system. It handles these cases by relying on the operating
system to do the conversion and then ensuring that the text encoding name
reported by ``sys.getfilesystemencoding()`` matches the encoding used during
this early bootstrapping process.
On Apple platforms (including both Mac OS X and iOS), this is
straightforward,
as Apple guarantees that these operations will always use UTF-8 to do the
conversion.
On Windows, the limitations of the ``mbcs`` format used by default in these
conversions proved sufficiently problematic that PEP 528 and PEP 529 were
implemented to bypass the operating system supplied interfaces for binary
data
handling and force the use of UTF-8 instead.
On Android, many components, including CPython, already assume the use of
UTF-8
as the system encoding, regardless of the locale setting. However, this
isn't
the case for all components, and the discrepancy can cause problems in some
situations (for example, when using the GNU readline module [16_]).
On non-Apple and non-Android \*nix systems, these operations are handled
using
the C locale system in glibc, which has the following characteristics [4_]:
* by default, all processes start in the ``C`` locale, which uses ``ASCII``
for these conversions. This is almost never what anyone doing multilingual
text processing actually wants (including CPython and C/C++ GUI
frameworks).
* calling ``setlocale(LC_ALL, "")`` reconfigures the active locale based on
the locale categories configured in the current process environment
* if the locale requested by the current environment is unknown, or no
specific
locale is configured, then the default ``C`` locale will remain active
The specific locale category that covers the APIs that CPython depends on is
``LC_CTYPE``, which applies to "classification and conversion of characters,
and to multibyte and wide characters" [5_]. Accordingly, CPython includes
the
following key calls to ``setlocale``:
* in the main ``python`` binary, CPython calls ``setlocale(LC_ALL, "")`` to
configure the entire C locale subsystem according to the process
environment.
It does this prior to making any calls into the shared CPython library
* in ``Py_Initialize``, CPython calls ``setlocale(LC_CTYPE, "")``, such that
the configured locale settings for that category *always* match those set
in
the environment. It does this unconditionally, and it *doesn't* revert the
process state change in ``Py_Finalize``
(This summary of the locale handling omits several technical details related
to exactly where and when the text encoding declared as part of the locale
settings is used - see PEP 540 for further discussion, as these particular
details matter more when decoupling CPython from the declared C locale than
they do when overriding the locale with one based on UTF-8)
These calls are usually sufficient to provide sensible behaviour, but they
can
still fail in the following cases:
* SSH environment forwarding means that SSH clients may sometimes forward
client locale settings to servers that don't have that locale installed.
This
leads to CPython running in the default ASCII-based C locale
* some process environments (such as Linux containers) may not have any
explicit locale configured at all. As with unknown locales, this leads to
CPython running in the default ASCII-based C locale
The simplest way to deal with this problem for currently released versions
of
CPython is to explicitly set a more sensible locale when launching the
application. For example::
LC_ALL=C.UTF-8 LANG=C.UTF-8 python3 ...
The ``C.UTF-8`` locale is a full locale definition that uses ``UTF-8`` for
the
``LC_CTYPE`` category, and the same settings as the ``C`` locale for all
other
categories (including ``LC_COLLATE``). It is offered by a number of Linux
distributions (including Debian, Ubuntu, Fedora, Alpine and Android) as an
alternative to the ASCII-based C locale.
Mac OS X and other \*BSD systems have taken a different approach, and
instead
of offering a ``C.UTF-8`` locale, instead offer a partial ``UTF-8`` locale
that
only defines the ``LC_CTYPE`` category. On such systems, the preferred
environmental locale adjustment is to set ``LC_CTYPE=UTF-8`` rather than to
set
``LC_ALL`` or ``LANG``. [17_]
In the specific case of Docker containers and similar technologies, the
appropriate locale setting can be specified directly in the container image
definition.
Another common failure case is developers specifying ``LANG=C`` in order to
see otherwise translated user interface messages in English, rather than the
more narrowly scoped ``LC_MESSAGES=C``.
Relationship with other PEPs
============================
This PEP shares a common problem statement with PEP 540 (improving Python
3's
behaviour in the default C locale), but diverges markedly in the proposed
solution:
* PEP 540 proposes to entirely decouple CPython's default text encoding from
the C locale system in that case, allowing text handling inconsistencies
to
arise between CPython and other C/C++ components running in the same
process
and in subprocesses. This approach aims to make CPython behave less like a
locale-aware C/C++ application, and more like C/C++ independent language
runtimes like the JVM, .NET CLR, Go, Node.js, and Rust
* this PEP proposes to override the legacy C locale with a more recently
defined locale that uses UTF-8 as its default text encoding. This means
that
the text encoding override will apply not only to CPython, but also to any
locale aware extension modules loaded into the current process, as well
as to
locale aware C/C++ applications invoked in subprocesses that inherit their
environment from the parent process. This approach aims to retain
CPython's
traditional strong support for integration with other components written
in C and C++, while actively helping to push forward the adoption and
standardisation of the C.UTF-8 locale as a Unicode-aware replacement for
the legacy C locale in the wider C/C++ ecosystem
After reviewing both PEPs, it became clear that they didn't actually
conflict
at a technical level, and the proposal in PEP 540 offered a superior option
in
cases where no suitable locale was available, as well as offering a better
reference behaviour for platforms where the notion of a "locale encoding"
doesn't make sense (for example, embedded systems running MicroPython rather
than the CPython reference interpreter).
Meanwhile, this PEP offered improved compatibility with other C/C++
components,
and an approach more amenable to being backported to Python 3.6 by
downstream
redistributors.
As a result, this PEP was amended to refer to PEP 540 as a complementary
solution that offered improved behaviour both when locale coercion
triggered,
as well as when none of the standard UTF-8 based locales were available.
The availability of PEP 540 also meant that the ``LC_CTYPE=en_US.UTF-8``
legacy
fallback was removed from the list of UTF-8 locales tried as a coercion
target,
with CPython instead relying solely on the proposed PYTHONUTF8 mode in such
cases.
Motivation
==========
While Linux container technologies like Docker, Kubernetes, and OpenShift
are
best known for their use in web service development, the related container
formats and execution models are also being adopted for Linux command line
application development. Technologies like Gnome Flatpak [7_] and
Ubunty Snappy [8_] further aim to bring these same techniques to Linux GUI
application development.
When using Python 3 for application development in these contexts, it isn't
uncommon to see text encoding related errors akin to the following::
$ docker run --rm fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in
position 7: surrogates not allowed
$ docker run --rm ncoghlan/debian-python python3 -c 'print("ℙƴ☂ℌøἤ")'
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in
position 7: surrogates not allowed
Even though the same command is likely to work fine when run locally::
$ python3 -c 'print("ℙƴ☂ℌøἤ")'
ℙƴ☂ℌøἤ
The source of the problem can be seen by instead running the ``locale``
command
in the three environments::
$ locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
LANG=en_AU.UTF-8
LC_CTYPE="en_AU.UTF-8"
LC_ALL=
$ docker run --rm fedora:25 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
LANG=
LC_CTYPE="POSIX"
LC_ALL=
$ docker run --rm ncoghlan/debian-python locale | grep -E
'LC_ALL|LC_CTYPE|LANG'
LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_ALL=
In this particular example, we can see that the host system locale is set to
"en_AU.UTF-8", so CPython uses UTF-8 as the default text encoding. By
contrast,
the base Docker images for Fedora and Debian don't have any specific locale
set, so they use the POSIX locale by default, which is an alias for the
ASCII-based default C locale.
The simplest way to get Python 3 (regardless of the exact version) to behave
sensibly in Fedora and Debian based containers is to run it in the
``C.UTF-8``
locale that both distros provide::
$ docker run --rm -e LANG=C.UTF-8 fedora:25 python3 -c 'print("ℙƴ☂ℌøἤ")'
ℙƴ☂ℌøἤ
$ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python python3 -c
'print("ℙƴ☂ℌøἤ")'
ℙƴ☂ℌøἤ
$ docker run --rm -e LANG=C.UTF-8 fedora:25 locale | grep -E
'LC_ALL|LC_CTYPE|LANG'
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_ALL=
$ docker run --rm -e LANG=C.UTF-8 ncoghlan/debian-python locale | grep
-E 'LC_ALL|LC_CTYPE|LANG'
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_ALL=
The Alpine Linux based Python images provided by Docker, Inc, already use
the
C.UTF-8 locale by default::
$ docker run --rm python:3 python3 -c 'print("ℙƴ☂ℌøἤ")'
ℙƴ☂ℌøἤ
$ docker run --rm python:3 locale | grep -E 'LC_ALL|LC_CTYPE|LANG'
LANG=C.UTF-8
LANGUAGE=
LC_CTYPE="C.UTF-8"
LC_ALL=
Similarly, for custom container images (i.e. those adding additional
content on
top of a base distro image), a more suitable locale can be set in the image
definition so everything just works by default. However, it would provide a
much
nicer and more consistent user experience if CPython were able to just deal
with this problem automatically rather than relying on redistributors or end
users to handle it through system configuration changes.
While the glibc developers are working towards making the C.UTF-8 locale
universally available for use by glibc based applications like CPython [6_],
this unfortunately doesn't help on platforms that ship older versions of
glibc
without that feature, and also don't provide C.UTF-8 as an on-disk locale
the
way Debian and Fedora do. For these platforms, the mechanism proposed in
PEP 540 at least allows CPython itself to behave sensibly, albeit without
any
mechanism to get other C/C++ components that decode binary streams as text
to
do the same.
Design Principles
=================
The above motivation leads to the following core design principles for the
proposed solution:
* if a locale other than the default C locale is explicitly configured,
we'll
continue to respect it
* if we're changing the locale setting without an explicit config option,
we'll
emit a warning on stderr that we're doing so rather than silently changing
the process configuration. This will alert application and system
integrators
to the change, even if they don't closely follow the PEP process or Python
release announcements. However, to minimize the chance of introducing new
problems for end users, we'll do this *without* using the warnings
system, so
even running with ``-Werror`` won't turn it into a runtime exception
* any changes made will use *existing* configuration options
To minimize the negative impact on systems currently correctly configured to
use GB-18030 or another partially ASCII compatible universal encoding leads
to
an additional design principle:
* if a UTF-8 based Linux container is run on a host that is explicitly
configured to use a non-UTF-8 encoding, and tries to exchange locally
encoded data with that host rather than exchanging explicitly UTF-8
encoded
data, CPython will endeavour to correctly round-trip host provided data
that
is concatenated or split solely at common ASCII compatible code points,
but
may otherwise emit nonsensical results.
Specification
=============
To better handle the cases where CPython would otherwise end up attempting
to operate in the ``C`` locale, this PEP proposes that CPython automatically
attempt to coerce the legacy ``C`` locale to a UTF-8 based locale when it is
run as a standalone command line application.
It further proposes to emit a warning on stderr if the legacy ``C`` locale
is in effect at the point where the language runtime itself is initialized,
and the PEP 540 UTF-8 encoding override is also disabled, in order to warn
system and application integrators that they're running CPython in an
unsupported configuration.
Legacy C locale coercion in the standalone Python interpreter binary
--------------------------------------------------------------------
When run as a standalone application, CPython has the opportunity to
reconfigure the C locale before any locale dependent operations are executed
in the process.
This means that it can change the locale settings not only for the CPython
runtime, but also for any other C/C++ components running in the current
process (e.g. as part of extension modules), as well as in subprocesses that
inherit their environment from the current process.
After calling ``setlocale(LC_ALL, "")`` to initialize the locale settings in
the current process, the main interpreter binary will be updated to include
the following call::
const char *ctype_loc = setlocale(LC_CTYPE, NULL);
This cryptic invocation is the API that C provides to query the current
locale
setting without changing it. Given that query, it is possible to check for
exactly the ``C`` locale with ``strcmp``::
ctype_loc != NULL && strcmp(ctype_loc, "C") == 0 # true only in the C
locale
This call also returns ``"C"`` when either no particular locale is set, or
the
nominal locale is set to an alias for the ``C`` locale (such as ``POSIX``).
Given this information, CPython can then attempt to coerce the locale to one
that uses UTF-8 rather than ASCII as the default encoding.
Three such locales will be tried:
* ``C.UTF-8`` (available at least in Debian, Ubuntu, and Fedora 25+, and
expected to be available by default in a future version of glibc)
* ``C.utf8`` (available at least in HP-UX)
* ``UTF-8`` (available in at least some \*BSD variants)
For ``C.UTF-8`` and ``C.utf8``, the coercion will be implemented by actually
setting the ``LANG`` and ``LC_ALL`` environment variables to the candidate
locale name, such that future calls to ``setlocale()`` will see them, as
will
other components looking for those settings (such as GUI development
frameworks).
For the platforms where it is defined, ``UTF-8`` is a partial locale that
only
defines the ``LC_CTYPE`` category. Accordingly, only the ``LC_CTYPE``
environment variable would be set when using this fallback option.
To adjust automatically to future changes in locale availability, these
checks
will be implemented at runtime on all platforms other than Mac OS X and
Windows,
rather than attempting to determine which locales to try at compile time.
If the locale settings are changed successfully, and the
``PYTHONIOENCODING``
environment variable is currently unset, then it will be forced to
``PYTHONIOENCODING=utf-8:surrogateescape``.
When this locale coercion is activated, the following warning will be
printed on stderr, with the warning containing whichever locale was
successfully configured::
Python detected LC_CTYPE=C, LC_ALL & LANG set to C.UTF-8 (set another
locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion
behaviour).
When falling back to the ``UTF-8`` locale, the message would be slightly
different::
Python detected LC_CTYPE=C, LC_CTYPE set to UTF-8 (set another locale
or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).
In combination with PEP 540, this locale coercion will mean that the
standard
Python binary *and* locale aware C/C++ extensions should once again "just
work"
in the three main failure cases we're aware of (missing locale
settings, SSH forwarding of unknown locales, and developers explicitly
requesting ``LANG=C``), as long as the target platform provides at least one
of the candidate UTF-8 based environments.
If ``PYTHONCOERCECLOCALE=0`` is set, or none of the candidate locales is
successfully configured, then initialization will continue as usual in the C
locale and the Unicode compatibility warning described in the next section
will
be emitted just as it would for any other application.
The interpreter will always check for the ``PYTHONCOERCECLOCALE``
environment
variable (even when running under the ``-E`` or ``-I`` switches), as the
locale
coercion check necessarily takes place before any command line argument
processing.
Changes to the runtime initialization process
---------------------------------------------
By the time that ``Py_Initialize`` is called, arbitrary locale-dependent
operations may have taken place in the current process. This means that
by the time it is called, it is *too late* to switch to a different locale -
doing so would introduce inconsistencies in decoded text, even in the
context
of the standalone Python interpreter binary.
Accordingly, when ``Py_Initialize`` is called and CPython detects that the
configured locale is still the default ``C`` locale *and* the ``PYTHONUTF8``
feature from PEP 540 is disabled, the following warning will
be issued::
Python runtime initialized with LC_CTYPE=C (a locale with default ASCII
encoding), which may cause Unicode compatibility problems. Using C.UTF-8
C.utf8, or UTF-8 (if available) as alternative Unicode-compatible
locales is recommended.
In this case, no actual change will be made to the locale settings.
Instead, the warning informs both system and application integrators that
they're running Python 3 in a configuration that we don't expect to work
properly.
The second sentence providing recommendations would be conditionally
compiled
based on the operating system (e.g. recommending ``LC_CTYPE=UTF-8`` on \*BSD
systems.
New build-time configuration options
------------------------------------
While both of the above behaviours would be enabled by default, they would
also have new associated configuration options and preprocessor definitions
for the benefit of redistributors that want to override those default
settings.
The locale coercion behaviour would be controlled by the flag
``--with[out]-c-locale-coercion``, which would set the
``PY_COERCE_C_LOCALE``
preprocessor definition.
The locale warning behaviour would be controlled by the flag
``--with[out]-c-locale-warning``, which would set the
``PY_WARN_ON_C_LOCALE``
preprocessor definition.
On platforms where they would have no effect (e.g. Mac OS X, iOS, Android,
Windows) these preprocessor variables would always be undefined.
Platform Support Changes
========================
A new "Legacy C Locale" section will be added to PEP 11 that states:
* as of CPython 3.7, the legacy C locale is only supported when operating in
"UTF-8" mode. Any Unicode handling issues that occur only in that locale
and cannot be reproduced in an appropriately configured non-ASCII locale
will
be closed as "won't fix"
* as of CPython 3.7, \*nix platforms are expected to provide at least one of
``C.UTF-8`` (full locale), ``C.utf8`` (full locale) or ``UTF-8`` (
``LC_CTYPE``-only locale) as an alternative to the legacy ``C`` locale.
Any Unicode related integration problems with C/C++ extensions that occur
only in that locale and cannot be reproduced in an appropriately
configured
non-ASCII locale will be closed as "won't fix".
Rationale
=========
Improving the handling of the C locale
--------------------------------------
It has been clear for some time that the C locale's default encoding of
``ASCII`` is entirely the wrong choice for development of modern networked
services. Newer languages like Rust and Go have eschewed that default
entirely,
and instead made it a deployment requirement that systems be configured to
use
UTF-8 as the text encoding for operating system interfaces. Similarly,
Node.js
assumes UTF-8 by default (a behaviour inherited from the V8 JavaScript
engine)
and requires custom build settings to indicate it should use the system
locale settings for locale-aware operations. Both the JVM and the .NET CLR
use UTF-16-LE as their primary encoding for passing text between
applications
and the underlying platform.
The challenge for CPython has been the fact that in addition to being used
for
network service development, it is also extensively used as an embedded
scripting language in larger applications, and as a desktop application
development language, where it is more important to be consistent with other
C/C++ components sharing the same process, as well as with the user's
desktop
locale settings, than it is with the emergent conventions of modern network
service development.
The core premise of this PEP is that for *all* of these use cases, the
assumption of ASCII implied by the default "C" locale is the wrong choice,
and furthermore that the following assumptions are valid:
* in desktop application use cases, the process locale will *already* be
configured appropriately, and if it isn't, then that is an operating
system
or embedding application level problem that needs to be reported to and
resolved by the operating system provider or application developer
* in network service development use cases (especially those based on Linux
containers), the process locale may not be configured *at all*, and if it
isn't, then the expectation is that components will impose their own
default
encoding the way Rust, Go and Node.js do, rather than trusting the legacy
C
default encoding of ASCII the way CPython currently does
Defaulting to "surrogateescape" error handling on the standard IO streams
-------------------------------------------------------------------------
By coercing the locale away from the legacy C default and its assumption of
ASCII as the preferred text encoding, this PEP also disables the implicit
use
of the "surrogateescape" error handler on the standard IO streams that was
introduced in Python 3.5 ([15_]), as well as the automatic use of
``surrogateescape`` when operating in PEP 540's UTF-8 mode.
Rather than introducing yet another configuration option to address that,
this PEP proposes to use the existing ``PYTHONIOENCODING`` setting to ensure
that the ``surrogateescape`` handler is enabled when the interpreter is
required to make assumptions regarding the expected filesystem encoding.
The aim of this behaviour is to attempt to ensure that operating system
provided text values are typically able to be transparently passed through a
Python 3 application even if it is incorrect in assuming that that text has
been encoded as UTF-8.
In particular, GB 18030 [12_] is a Chinese national text encoding standard
that handles all Unicode code points, that is formally incompatible with
both
ASCII and UTF-8, but will nevertheless often tolerate processing as
surrogate
escaped data - the points where GB 18030 reuses ASCII byte values in an
incompatible way are likely to be invalid in UTF-8, and will therefore be
escaped and opaque to string processing operations that split on or search
for
the relevant ASCII code points. Operations that don't involve splitting on
or
searching for particular ASCII or Unicode code point values are almost
certain to work correctly.
Similarly, Shift-JIS [13_] and ISO-2022-JP [14_] remain in widespread use in
Japan, and are incompatible with both ASCII and UTF-8, but will tolerate
text
processing operations that don't involve splitting on or searching for
particular ASCII or Unicode code point values.
As an example, consider two files, one encoded with UTF-8 (the default
encoding
for ``en_AU.UTF-8``), and one encoded with GB-18030 (the default encoding
for
``zh_CN.gb18030``)::
$ python3 -c 'open("utf8.txt", "wb").write("ℙƴ☂ℌøἤ\n".encode("utf-8"))'
$ python3 -c 'open("gb18030.txt",
"wb").write("ℙƴ☂ℌøἤ\n".encode("gb18030"))'
On disk, we can see that these are two very different files::
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "rb").read().strip());
\
print("GB18030:", open("gb18030.txt",
"rb").read().strip())'
UTF-8:
b'\xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4\n'
GB18030:
b'\x816\xbd6\x810\x9d0\x817\xa29\x816\xbc4\x810\x8b3\x816\x8d6\n'
That nevertheless can both be rendered correctly to the terminal as long as
they're decoded prior to printing::
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "r",
encoding="utf-8").read().strip()); \
print("GB18030:", open("gb18030.txt", "r",
encoding="gb18030").read().strip())'
UTF-8: ℙƴ☂ℌøἤ
GB18030: ℙƴ☂ℌøἤ
By contrast, if we just pass along the raw bytes, as ``cat`` and similar
C/C++
utilities will tend to do::
$ LANG=en_AU.UTF-8 cat utf8.txt gb18030.txt
ℙƴ☂ℌøἤ
�6�6�0�0�7�9�6�4�0�3�6�6
Even setting a specifically Chinese locale won't help in getting the
GB-18030 encoded file rendered correctly::
$ LANG=zh_CN.gb18030 cat utf8.txt gb18030.txt
ℙƴ☂ℌøἤ
�6�6�0�0�7�9�6�4�0�3�6�6
The problem is that the *terminal* encoding setting remains UTF-8,
regardless
of the nominal locale. A GB18030 terminal can be emulated using the
``iconv``
utility::
$ cat utf8.txt gb18030.txt | iconv -f GB18030 -t UTF-8
鈩櫰粹槀鈩屆羔激
ℙƴ☂ℌøἤ
This reverses the problem, such that the GB18030 file is rendered correctly,
but the UTF-8 file has been converted to unrelated hanzi characters, rather
than
the expected rendering of "Python" as non-ASCII characters.
With the emulated GB18030 terminal encoding, assuming UTF-8 in Python
results
in *both* files being displayed incorrectly::
$ python3 -c 'print("UTF-8: ", open("utf8.txt", "r",
encoding="utf-8").read().strip()); \
print("GB18030:", open("gb18030.txt", "r",
encoding="gb18030").read().strip())' \
| iconv -f GB18030 -t UTF-8
UTF-8: 鈩櫰粹槀鈩屆羔激
GB18030: 鈩櫰粹槀鈩屆羔激
However, setting the locale correctly means that the emulated GB18030
terminal
now displays both files as originally intended::
$ LANG=zh_CN.gb18030 \
python3 -c 'print("UTF-8: ", open("utf8.txt", "r",
encoding="utf-8").read().strip()); \
print("GB18030:", open("gb18030.txt", "r",
encoding="gb18030").read().strip())' \
| iconv -f GB18030 -t UTF-8
UTF-8: ℙƴ☂ℌøἤ
GB18030: ℙƴ☂ℌøἤ
The rationale for retaining ``surrogateescape`` as the default IO encoding
is
that it will preserve the following helpful behaviour in the C locale::
$ cat gb18030.txt \
| LANG=C python3 -c "import sys; print(sys.stdin.read())" \
| iconv -f GB18030 -t UTF-8
ℙƴ☂ℌøἤ
Rather than reverting to the exception seen when a UTF-8 based locale is
explicitly configured::
$ cat gb18030.txt \
| python3 -c "import sys; print(sys.stdin.read())" \
| iconv -f GB18030 -t UTF-8
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib64/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 0:
invalid start byte
Note: an alternative to setting ``PYTHONIOENCODING`` as the PEP currently
proposes would be to instead *always* default to ``surrogateescape`` on the
standard streams, and require the use of ``PYTHONIOENCODING=:strict`` to
request
text encoding validation during stream processing. Adopting such an approach
would bring Python 3 more into line with typical C/C++ tools that pass along
the raw bytes without checking them for conformance to their nominal
encoding,
and would hence also make the last example display the desired output::
$ cat gb18030.txt \
| PYTHONIOENCODING=:surrogateescape python3 -c "import sys;
print(sys.stdin.read())" \
| iconv -f GB18030 -t UTF-8
ℙƴ☂ℌøἤ
Dropping official support for ASCII based text handling in the legacy C
locale
------------------------------------------------------------------------------
We've been trying to get strict bytes/text separation to work reliably in
the
legacy C locale for over a decade at this point. Not only haven't we been
able
to get it to work, neither has anyone else - the only viable alternatives
identified have been to pass the bytes along verbatim without eagerly
decoding
them to text (C/C++, Python 2.x, Ruby, etc), or else to ignore the nominal
C/C++ locale encoding entirely and assume the use of either UTF-8 (PEP 540,
Rust, Go, Node.js, etc) or UTF-16-LE (JVM, .NET CLR).
While this PEP ensures that developers that need to do so can still opt-in
to
running their Python code in the legacy C locale, it also makes clear that
we
*don't* expect Python 3's Unicode handling to be reliable in that
configuration,
and the recommended alternative is to use a more appropriate locale setting.
Providing implicit locale coercion only when running standalone
---------------------------------------------------------------
Over the course of Python 3.x development, multiple attempts have been made
to improve the handling of incorrect locale settings at the point where the
Python interpreter is initialised. The problem that emerged is that this is
ultimately *too late* in the interpreter startup process - data such as
command
line arguments and the contents of environment variables may have already
been
retrieved from the operating system and processed under the incorrect ASCII
text encoding assumption well before ``Py_Initialize`` is called.
The problems created by those inconsistencies were then even harder to
diagnose
and debug than those created by believing the operating system's claim that
ASCII was a suitable encoding to use for operating system interfaces. This
was
the case even for the default CPython binary, let alone larger C/C++
applications that embed CPython as a scripting engine.
The approach proposed in this PEP handles that problem by moving the locale
coercion as early as possible in the interpreter startup sequence when
running
standalone: it takes place directly in the C-level ``main()`` function, even
before calling in to the `Py_Main()`` library function that implements the
features of the CPython interpreter CLI.
The ``Py_Initialize`` API then only gains an explicit warning (emitted on
``stderr``) when it detects use of the ``C`` locale, and relies on the
embedding application to specify something more reasonable.
Querying LC_CTYPE for C locale detection
----------------------------------------
``LC_CTYPE`` is the actual locale category that CPython relies on to drive
the
implicit decoding of environment variables, command line arguments, and
other
text values received from the operating system.
As such, it makes sense to check it specifically when attempting to
determine
whether or not the current locale configuration is likely to cause Unicode
handling problems.
Setting both LANG & LC_ALL for C.UTF-8 locale coercion
------------------------------------------------------
Python is often used as a glue language, integrating other C/C++ ABI
compatible
components in the current process, and components written in arbitrary
languages in subprocesses.
Setting ``LC_ALL`` to ``C.UTF-8`` imposes a locale setting override on all
C/C++ components in the current process and in any subprocesses that inherit
the current environment. This is important to handle cases where the problem
has arisen from a setting like ``LC_CTYPE=UTF-8`` being provided on a system
where no ``UTF-8`` locale is defined (e.g. when a Mac OS X ssh client is
configured to forward locale settings, and the user logs into a Linux
server).
Setting ``LANG`` to ``C.UTF-8`` ensures that even components that only check
the ``LANG`` fallback for their locale settings will still use ``C.UTF-8``.
Together, these should ensure that when the locale coercion is activated,
the
switch to the C.UTF-8 locale will be applied consistently across the current
process and any subprocesses that inherit the current environment.
Allowing restoration of the legacy behaviour
--------------------------------------------
The CPython command line interpreter is often used to investigate faults
that
occur in other applications that embed CPython, and those applications may
still
be using the C locale even after this PEP is implemented.
Providing a simple on/off switch for the locale coercion behaviour makes it
much easier to reproduce the behaviour of such applications for debugging
purposes, as well as making it easier to reproduce the behaviour of older
3.x
runtimes even when running a version with this change applied.
Implementation
==============
A draft implementation of the change (including test cases and
documentation)
is linked from issue 28180 [1_], which is an end user request that
``sys.getfilesystemencoding()`` default to ``utf-8`` rather than ``ascii``.
This patch is now being maintained as the ``pep538-coerce-c-locale`` feature
branch [18_] in Nick Coghlan's fork of the CPython repository on GitHub.
NOTE: As discussed in [1_], the currently posted draft implementation has
some
known issues on Android.
Backporting to earlier Python 3 releases
========================================
Backporting to Python 3.6.0
---------------------------
If this PEP is accepted for Python 3.7, redistributors backporting the
change
specifically to their initial Python 3.6.0 release will be both allowed and
encouraged. However, such backports should only be undertaken either in
conjunction with the changes needed to also provide a suitable locale by
default, or else specifically for platforms where such a locale is already
consistently available.
Backporting to other 3.x releases
---------------------------------
While the proposed behavioural change is seen primarily as a bug fix
addressing
Python 3's current misbehaviour in the default ASCII-based C locale, it
still
represents a reasonably significant change in the way CPython interacts with
the C locale system. As such, while some redistributors may still choose to
backport it to even earlier Python 3.x releases based on the needs and
interests of their particular user base, this wouldn't be encouraged as a
general practice.
However, configuring Python 3 *environments* (such as base container
images) to use these configuration settings by default is both allowed
and recommended.
Acknowledgements
================
The locale coercion approach proposed in this PEP is inspired directly by
Armin Ronacher's handling of this problem in the ``click`` command line
utility development framework [2_]::
$ LANG=C python3 -c 'import click; cli = click.command()(lambda:None);
cli()'
Traceback (most recent call last):
...
RuntimeError: Click will abort further execution because Python 3 was
configured to use ASCII as encoding for the environment. Either run
this
under Python 2 or consult http://click.pocoo.org/python3/ for mitigation
steps.
This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
The change was originally proposed as a downstream patch for Fedora's
system Python 3.6 package [3_], and then reformulated as a PEP for Python
3.7
with a section allowing for backports to earlier versions by redistributors.
The initial draft was posted to the Python Linux SIG for discussion [10_]
and
then amended based on both that discussion and Victor Stinner's work in
PEP 540 [11_].
The "ℙƴ☂ℌøἤ" string used in the Unicode handling examples throughout this
PEP
is taken from Ned Batchelder's excellent "Pragmatic Unicode" presentation
[9_].
Stephen Turnbull has long provided valuable insight into the text encoding
handling challenges he regularly encounters at the University of Tsukuba
(筑波大学).
References
==========
.. [1] CPython: sys.getfilesystemencoding() should default to utf-8
(http://bugs.python.org/issue28180)
.. [2] Locale configuration required for click applications under Python 3
(http://click.pocoo.org/5/python3/#python-3-surrogate-handling)
.. [3] Fedora: force C.UTF-8 when Python 3 is run under the C locale
(https://bugzilla.redhat.com/show_bug.cgi?id=1404918)
.. [4] GNU C: How Programs Set the Locale
(
https://www.gnu.org/software/libc/manual/html_node/Setting-the-Locale.html)
.. [5] GNU C: Locale Categories
(
https://www.gnu.org/software/libc/manual/html_node/Locale-Categories.html)
.. [6] glibc C.UTF-8 locale proposal
(https://sourceware.org/glibc/wiki/Proposals/C.UTF-8)
.. [7] GNOME Flatpak
(http://flatpak.org/)
.. [8] Ubuntu Snappy
(https://www.ubuntu.com/desktop/snappy)
.. [9] Pragmatic Unicode
(http://nedbatchelder.com/text/unipain.html)
.. [10] linux-sig discussion of initial PEP draft
(https://mail.python.org/pipermail/linux-sig/2017-January/000014.html)
.. [11] Feedback notes from linux-sig discussion and PEP 540
(https://github.com/python/peps/issues/171)
.. [12] GB 18030
(https://en.wikipedia.org/wiki/GB_18030)
.. [13] Shift-JIS
(https://en.wikipedia.org/wiki/Shift_JIS)
.. [14] ISO-2022
(https://en.wikipedia.org/wiki/ISO/IEC_2022)
.. [15] Use "surrogateescape" error handler for sys.stdin and sys.stdout on
UNIX for the C locale
(https://bugs.python.org/issue19977)
.. [16] test_readline.test_nonascii fails on Android
(http://bugs.python.org/issue28997)
.. [17] UTF-8 locale discussion on "locale.getdefaultlocale() fails on Mac
OS X with default language set to English"
(http://bugs.python.org/issue18378#msg215215)
.. [18] GitHub branch diff for ``ncoghlan:pep538-coerce-c-locale``
(
https://github.com/python/cpython/compare/master...ncoghlan:pep538-coerce-c…
)
Copyright
=========
This document has been placed in the public domain under the terms of the
CC0 1.0 license: https://creativecommons.org/publicdomain/zero/1.0/
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
[View Less]
10
37
My allergies have hit me hard so I'm not thinking at full capacity, but did
we ever decide if supporting os.PathLike in the stdlib was viewed as an
enhancement or bugfix? Specifically I'm thinking of
https://bugs.python.org/issue30218 for adding support to
shutil.unpack_archive() and whether it should be backported to 3.6.
9
22
ACTIVITY SUMMARY (2017-04-28 - 2017-05-05)
Python tracker at http://bugs.python.org/
To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.
Issues counts and deltas:
open 5929 (+21)
closed 36109 (+61)
total 42038 (+82)
Open issues with patches: 2384
Issues opened (58)
==================
#29094: Regression in zipfile writing in 2.7.13
http://bugs.python.org/issue29094 reopened by gregory.p.smith
#30166: Import command-line …
[View More]parsing modules only when needed
http://bugs.python.org/issue30166 reopened by terry.reedy
#30202: Update test.test_importlib.test_abc to test find_spec()
http://bugs.python.org/issue30202 opened by brett.cannon
#30203: AttributeError in Popen.communicate()
http://bugs.python.org/issue30203 opened by Luke Campagnola
#30210: No Documentation on tkinter dnd module
http://bugs.python.org/issue30210 opened by anthony-flury
#30211: Bdb: add docstrings
http://bugs.python.org/issue30211 opened by csabella
#30212: test_ssl.py is broken in Centos7
http://bugs.python.org/issue30212 opened by david-cpi
#30213: ZipFile from 'a'ppend-mode file generates invalid zip
http://bugs.python.org/issue30213 opened by BoppreH
#30214: make_zip.py lacks command a few line options.
http://bugs.python.org/issue30214 opened by Decorater
#30216: xdrlib.Unpacker.unpack_string returns bytes (docs say should b
http://bugs.python.org/issue30216 opened by rnash
#30217: Missing entry for the tilde (~) operator in the Index
http://bugs.python.org/issue30217 opened by lebigot
#30218: shutil.unpack_archive doesn't support PathLike
http://bugs.python.org/issue30218 opened by Jelle Zijlstra
#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219 opened by desbma
#30220: Why are the custom messages for ValueError and TypeError suppr
http://bugs.python.org/issue30220 opened by pgacv2
#30222: make_zip.py had a bug when setting up full 64 bit distribution
http://bugs.python.org/issue30222 opened by Decorater
#30223: Add Lib/test/__main__.py in 2.7
http://bugs.python.org/issue30223 opened by serhiy.storchaka
#30224: remove outdated checks in struct
http://bugs.python.org/issue30224 opened by xiang.zhang
#30226: Modernize make_ssl_certs
http://bugs.python.org/issue30226 opened by christian.heimes
#30227: test_site must not write outside the build directory: must not
http://bugs.python.org/issue30227 opened by haypo
#30228: Open a file in text mode requires too many syscalls
http://bugs.python.org/issue30228 opened by haypo
#30229: Closing a BufferedRandom calls lseek() mutliple times
http://bugs.python.org/issue30229 opened by haypo
#30230: Move quick test in PyObject_IsSubClass outside of PyType_Check
http://bugs.python.org/issue30230 opened by Jim Fasarakis-Hilliard
#30231: test_imaplib needs a TLS server accepting self-signed certific
http://bugs.python.org/issue30231 opened by haypo
#30235: Validate shutil supports path-like objects, update docs accord
http://bugs.python.org/issue30235 opened by brett.cannon
#30237: Access violation due to CancelSynchronousIo of console read
http://bugs.python.org/issue30237 opened by eryksun
#30238: 2to3 doesn't detect or fix Exception indexing
http://bugs.python.org/issue30238 opened by dlenski
#30241: Add contextlib.AbstractAsyncContextManager
http://bugs.python.org/issue30241 opened by Jelle Zijlstra
#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242 opened by xiang.zhang
#30244: Emit a ResourceWarning in concurrent.futures executor destruct
http://bugs.python.org/issue30244 opened by haypo
#30245: possible overflow when organize struct.pack_into error message
http://bugs.python.org/issue30245 opened by xiang.zhang
#30246: fix several error messages in struct
http://bugs.python.org/issue30246 opened by xiang.zhang
#30247: Make importlib.machinery class handle os.PathLike path
http://bugs.python.org/issue30247 opened by louielu
#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248 opened by serhiy.storchaka
#30249: improve struct.unpack_from's error message like struct.pack_in
http://bugs.python.org/issue30249 opened by xiang.zhang
#30250: StreamIO truncate behavior of current position
http://bugs.python.org/issue30250 opened by Albert.Zeyer
#30251: Windows Visual Studio solution does not have an install target
http://bugs.python.org/issue30251 opened by steveire
#30252: Configuration system does not work for Windows/Visual Studio
http://bugs.python.org/issue30252 opened by steveire
#30253: Python does not build without WITH_THREADS defined on Windows/
http://bugs.python.org/issue30253 opened by steveire
#30256: Adding a SyncManager Queue proxy to a SyncManager dict or Name
http://bugs.python.org/issue30256 opened by jjdmon
#30258: [2.7] regrtest: handle child process crash
http://bugs.python.org/issue30258 opened by haypo
#30259: Test somehow that generated files are up to date: run make reg
http://bugs.python.org/issue30259 opened by haypo
#30260: sock_dealloc() may call __repr__ when socket class is already
http://bugs.python.org/issue30260 opened by asterite
#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262 opened by palaviv
#30263: regrtest: log the system load?
http://bugs.python.org/issue30263 opened by haypo
#30266: AbstractContextManager should support __method__ = None
http://bugs.python.org/issue30266 opened by Jelle Zijlstra
#30267: Deprecate os.path.commonprefix
http://bugs.python.org/issue30267 opened by Valentin.Lorentz
#30268: Make mimetypes.guess_type accept path-like objects
http://bugs.python.org/issue30268 opened by Valentin.Lorentz
#30270: Remove sqlite3.Cache display method
http://bugs.python.org/issue30270 opened by palaviv
#30271: Make sqlite3 statement cache optional
http://bugs.python.org/issue30271 opened by palaviv
#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272 opened by ollieparanoid2
#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273 opened by haypo
#30274: Make importlib.abc.ExtensionFileLoader.__init__() documentatio
http://bugs.python.org/issue30274 opened by brett.cannon
#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275 opened by Jeff Zhang
#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276 opened by bmwiedemann
#30278: The element div#sidebar on documentation requires the attribut
http://bugs.python.org/issue30278 opened by Kyle Thomas
#30280: Warning -- threading._dangling was modified by test_asyncio on
http://bugs.python.org/issue30280 opened by haypo
#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281 opened by xiang.zhang
#30282: object returned by tarfile.extractfile has an incorrect value
http://bugs.python.org/issue30282 opened by Ondrej Kubecka
Most recent 15 issues with no replies (15)
==========================================
#30282: object returned by tarfile.extractfile has an incorrect value
http://bugs.python.org/issue30282
#30280: Warning -- threading._dangling was modified by test_asyncio on
http://bugs.python.org/issue30280
#30278: The element div#sidebar on documentation requires the attribut
http://bugs.python.org/issue30278
#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275
#30274: Make importlib.abc.ExtensionFileLoader.__init__() documentatio
http://bugs.python.org/issue30274
#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272
#30271: Make sqlite3 statement cache optional
http://bugs.python.org/issue30271
#30266: AbstractContextManager should support __method__ = None
http://bugs.python.org/issue30266
#30256: Adding a SyncManager Queue proxy to a SyncManager dict or Name
http://bugs.python.org/issue30256
#30253: Python does not build without WITH_THREADS defined on Windows/
http://bugs.python.org/issue30253
#30252: Configuration system does not work for Windows/Visual Studio
http://bugs.python.org/issue30252
#30251: Windows Visual Studio solution does not have an install target
http://bugs.python.org/issue30251
#30249: improve struct.unpack_from's error message like struct.pack_in
http://bugs.python.org/issue30249
#30245: possible overflow when organize struct.pack_into error message
http://bugs.python.org/issue30245
#30244: Emit a ResourceWarning in concurrent.futures executor destruct
http://bugs.python.org/issue30244
Most recent 15 issues waiting for review (15)
=============================================
#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281
#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262
#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248
#30246: fix several error messages in struct
http://bugs.python.org/issue30246
#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242
#30224: remove outdated checks in struct
http://bugs.python.org/issue30224
#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219
#30193: Support the buffer protocol in json.loads()
http://bugs.python.org/issue30193
#30192: hashlib module breaks with 64-bit kernel and 32-bit user space
http://bugs.python.org/issue30192
#30176: curses attribute constants list is incomplete
http://bugs.python.org/issue30176
#30162: Add _PyTuple_Empty and make PyTuple_New(0) never failing
http://bugs.python.org/issue30162
#30156: PYTHONDUMPREFS segfaults on exit
http://bugs.python.org/issue30156
#30152: Reduce the number of imports for argparse
http://bugs.python.org/issue30152
#30143: Using collections ABC from collections.abc rather than collect
http://bugs.python.org/issue30143
#30134: BytesWarning is missing from the documents
http://bugs.python.org/issue30134
Top 10 most discussed issues (10)
=================================
#19903: Idle: Use inspect.signature for calltips
http://bugs.python.org/issue19903 23 msgs
#23404: 'make touch' does not work with git clones of the source repos
http://bugs.python.org/issue23404 15 msgs
#30223: Add Lib/test/__main__.py in 2.7
http://bugs.python.org/issue30223 13 msgs
#29094: Regression in zipfile writing in 2.7.13
http://bugs.python.org/issue29094 11 msgs
#26362: Approved API for creating a temporary file path
http://bugs.python.org/issue26362 9 msgs
#30258: [2.7] regrtest: handle child process crash
http://bugs.python.org/issue30258 8 msgs
#30263: regrtest: log the system load?
http://bugs.python.org/issue30263 8 msgs
#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273 8 msgs
#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276 8 msgs
#29943: PySlice_GetIndicesEx change broke ABI in 3.5 and 3.6 branches
http://bugs.python.org/issue29943 7 msgs
Issues closed (59)
==================
#15934: flaky test in test_ftplib
http://bugs.python.org/issue15934 closed by haypo
#19133: Transient test failure: test_with_statement (test_ftplib)
http://bugs.python.org/issue19133 closed by haypo
#22410: Locale dependent regexps on different locales
http://bugs.python.org/issue22410 closed by serhiy.storchaka
#22610: test_ftplib fails with sem_init: Too many open files
http://bugs.python.org/issue22610 closed by haypo
#23816: struct.unpack returns null pascal strings - [first] bug report
http://bugs.python.org/issue23816 closed by xiang.zhang
#27593: Deprecate sys._mercurial and create sys._git
http://bugs.python.org/issue27593 closed by brett.cannon
#28149: Incorrect indentation under “else” in _bsddb.c
http://bugs.python.org/issue28149 closed by martin.panter
#29448: Implement os.Pathlike for importlib.machinery
http://bugs.python.org/issue29448 closed by serhiy.storchaka
#29621: telnetlib.Telnet.write gives confusing error message when a st
http://bugs.python.org/issue29621 closed by berker.peksag
#29641: ./configure --enable-optimizations && make && make install com
http://bugs.python.org/issue29641 closed by haypo
#29915: Drop Mac OS X Tiger support in Python 3.7?
http://bugs.python.org/issue29915 closed by haypo
#29925: test_uuid fails on OS X Tiger
http://bugs.python.org/issue29925 closed by haypo
#29967: "AMD64 FreeBSD 9.x 3.x" tries to rebuild Include/opcode.h, tim
http://bugs.python.org/issue29967 closed by haypo
#30103: uu package uses old encoding
http://bugs.python.org/issue30103 closed by xiang.zhang
#30104: clang 4.0 miscompiles dtoa.c, giving broken float parsing and
http://bugs.python.org/issue30104 closed by haypo
#30106: test_asyncore: test_handle_write() fails in tearDown()
http://bugs.python.org/issue30106 closed by haypo
#30107: python.core file created when running tests on AMD64 FreeBSD C
http://bugs.python.org/issue30107 closed by haypo
#30119: (ftplib) A remote attacker could possibly attack by containing
http://bugs.python.org/issue30119 closed by berker.peksag
#30124: Fix C aliasing issue in Python/dtoa.c to use strict aliasing o
http://bugs.python.org/issue30124 closed by haypo
#30125: test_SEH() of test_ctypes logs "Windows fatal exception: acces
http://bugs.python.org/issue30125 closed by haypo
#30126: CheckTraceCallbackContent of test_sqlite3 fails on OS X Tiger
http://bugs.python.org/issue30126 closed by haypo
#30131: test_logging leaks a "dangling" thread
http://bugs.python.org/issue30131 closed by haypo
#30158: Deprecation warnings emitted in test_importlib
http://bugs.python.org/issue30158 closed by serhiy.storchaka
#30169: test_multiprocessing_spawn crashed on AMD64 Windows8.1 Non-Deb
http://bugs.python.org/issue30169 closed by haypo
#30173: x86 Windows7 3.x buildbot has the issue #26624 bug
http://bugs.python.org/issue30173 closed by haypo
#30175: Random test_imaplib.test_logincapa_with_client_certfile failur
http://bugs.python.org/issue30175 closed by haypo
#30184: Add tests for invalid use of PyArg_ParseTupleAndKeywords
http://bugs.python.org/issue30184 closed by serhiy.storchaka
#30185: forkserver process should silence KeyboardInterrupt
http://bugs.python.org/issue30185 closed by pitrou
#30190: unittest's assertAlmostEqual improved error message
http://bugs.python.org/issue30190 closed by berker.peksag
#30196: Add __matmul__ to collections.Counter
http://bugs.python.org/issue30196 closed by rhettinger
#30197: Enhance swap_attr() and swap_item() in test.support
http://bugs.python.org/issue30197 closed by serhiy.storchaka
#30199: Warning -- asyncore.socket_map was modified by test_ssl
http://bugs.python.org/issue30199 closed by haypo
#30200: tkinter ListboxSelect
http://bugs.python.org/issue30200 closed by Frank Pae
#30201: [3.5] RecvmsgIntoSCMRightsStreamTest fails with "OSError: [Err
http://bugs.python.org/issue30201 closed by haypo
#30204: socket.setblocking(0) changes socket.type
http://bugs.python.org/issue30204 closed by giampaolo.rodola
#30205: socket.getsockname() type mismatch with AF_UNIX on Linux
http://bugs.python.org/issue30205 closed by pitrou
#30206: data parameter for binascii.b2a_base64 is not positional-only
http://bugs.python.org/issue30206 closed by xiang.zhang
#30207: Rename test.test_support to test.support in 2.7
http://bugs.python.org/issue30207 closed by serhiy.storchaka
#30208: Small typos in IDLE doc
http://bugs.python.org/issue30208 closed by terry.reedy
#30209: some UTF8 symbols
http://bugs.python.org/issue30209 closed by terry.reedy
#30215: Make re.compile() locale agnostic
http://bugs.python.org/issue30215 closed by serhiy.storchaka
#30221: Deprecate assertNotAlmostEqual
http://bugs.python.org/issue30221 closed by serhiy.storchaka
#30225: EBADF error on x86 Tiger 3.x buildbot
http://bugs.python.org/issue30225 closed by haypo
#30232: configure: support Git worktree
http://bugs.python.org/issue30232 closed by haypo
#30233: [3.5] Warning -- locale was modified by test_idle
http://bugs.python.org/issue30233 closed by terry.reedy
#30234: Remove duplicate checks in test_isinstance
http://bugs.python.org/issue30234 closed by Jim Fasarakis-Hilliard
#30236: Add options --match and --failfast for test.regrtest in 2.7
http://bugs.python.org/issue30236 closed by serhiy.storchaka
#30239: test_asyncore: test_handle_expt() fails on macOS Sierra
http://bugs.python.org/issue30239 closed by haypo
#30240: Add daemon keyword argument to Thread constructor
http://bugs.python.org/issue30240 closed by zach.ware
#30243: Core dump when use uninitialized _json objects
http://bugs.python.org/issue30243 closed by serhiy.storchaka
#30254: [2.7] regrtest announces 401 tests, but there are 304 tests?
http://bugs.python.org/issue30254 closed by haypo
#30255: test_xml_etree: python: Objects/sliceobject.c:176: _PySlice_Ad
http://bugs.python.org/issue30255 closed by haypo
#30257: _bsddb: else misleadingly indented
http://bugs.python.org/issue30257 closed by haypo
#30261: Spam
http://bugs.python.org/issue30261 closed by zach.ware
#30264: [Windows] test_sax: Warning -- files was modified by test_sax
http://bugs.python.org/issue30264 closed by haypo
#30265: [2.7] test.support.unlink() should fail or emit a warning on W
http://bugs.python.org/issue30265 closed by haypo
#30269: [2.7] test_release_task_refs() of test_multiprocessing.py is u
http://bugs.python.org/issue30269 closed by haypo
#30277: Speeds up compiling cases-insensitive regular expressions
http://bugs.python.org/issue30277 closed by serhiy.storchaka
#30279: Remove unused Python/thread_foobar.h
http://bugs.python.org/issue30279 closed by haypo
[View Less]
1
0