Mailman 3 November 2015 - Python-Dev

Summary of Python tracker Issues
by Python tracker Nov. 20, 2015

Nov. 20, 2015

ACTIVITY SUMMARY (2015-11-13 - 2015-11-20) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5260 (+28) closed 32181 (+38) total 37441 (+66) Open issues with patches: 2311 Issues opened (51) ================== #25589: test_ascii_formatd fails on Mac when built with Intel compiler http://bugs.python.org/issue25589 reopened by christopher.hogan #25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind http://bugs.python.org/issue25622 opened by martin.panter #25623: Keep the link to Python implementation of OrderedDict http://bugs.python.org/issue25623 opened by serhiy.storchaka #25624: shutil.make_archive makes invalid directory entries http://bugs.python.org/issue25624 opened by gumblex #25625: "chdir" Contex manager for pathlib http://bugs.python.org/issue25625 opened by Jáchym Barvínek #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 opened by Ben Cipollini #25627: distutils : file "bdist_rpm.py" does not quote filenames when http://bugs.python.org/issue25627 opened by TheRegRunner #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 opened by rhettinger #25631: Segmentation fault with invalid Unicode command-line arguments http://bugs.python.org/issue25631 opened by itaibn #25632: Document BUILD_*_UNPACK opcodes http://bugs.python.org/issue25632 opened by Antony.Lee #25633: The documentation for urllib.request should mention http.clien http://bugs.python.org/issue25633 opened by jon orebro #25635: urllib2 cannot fully read FTP file http://bugs.python.org/issue25635 opened by keakon #25636: Let launcher autoselect best 32-bit Python on 64-bit system http://bugs.python.org/issue25636 opened by Christian.Ullrich #25637: Move non-collections-related ABCs out of collections.abc http://bugs.python.org/issue25637 opened by brett.cannon #25638: Verify the etree_parse and etree_iterparse benchmarks are work http://bugs.python.org/issue25638 opened by brett.cannon #25639: open 'PhysicalDriveN' on windows fails (since python 3.5) with http://bugs.python.org/issue25639 opened by rokozh #25640: EmailMessage should throw an error if policy is set to compat3 http://bugs.python.org/issue25640 opened by Matthieu Pepin #25641: urllib/request.py/getproxies_environment() throws "dictionary http://bugs.python.org/issue25641 opened by alphalfalfa #25642: Setting maxsize breaks asyncio.JoinableQueue/Queue http://bugs.python.org/issue25642 opened by Chris Seto #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 opened by serhiy.storchaka #25644: Unable to open IDLE on Windows 10 with Python 2.7.10 http://bugs.python.org/issue25644 opened by jatanig #25646: Distutils and Windows SDK 7.1 http://bugs.python.org/issue25646 opened by JGoutin #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 opened by vrutsky #25649: Come up with a good way to handle module aliasing http://bugs.python.org/issue25649 opened by brett.cannon #25650: Mismatching documentation <=> behaviour for typing.Any http://bugs.python.org/issue25650 opened by ebarry #25651: Confusing output for TestCase.subTest(0) http://bugs.python.org/issue25651 opened by ezio.melotti #25652: collections.UserString.__rmod__() raises NameError http://bugs.python.org/issue25652 opened by Jonathan Goble #25653: ctypes+callbacks+fork+selinux = crash http://bugs.python.org/issue25653 opened by arigo #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 opened by martin.panter #25655: Python errors related to failures loading DLL's lack informati http://bugs.python.org/issue25655 opened by never-eat-yellow-snow #25656: multiprocessing.dummy: pool.map hangs on empty list http://bugs.python.org/issue25656 opened by tomer70 #25657: virtualenv's activate does not update LD_LIBRARY_PATH http://bugs.python.org/issue25657 opened by serge-sans-paille #25658: PyThread assumes pthread_key_t is an integer, which is against http://bugs.python.org/issue25658 opened by EdSchouten #25659: ctypes.Array.from_buffer segmentation fault when trying to cre http://bugs.python.org/issue25659 opened by spearsem(a)gmail.com #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 opened by yselivanov #25661: tokenize.untokenize does not maintain the order of tabbed inde http://bugs.python.org/issue25661 opened by Blake Hilliard #25663: Make rlcompleter avoid duplicate global names http://bugs.python.org/issue25663 opened by martin.panter #25664: Unexpected UnicodeDecodeError in logging module http://bugs.python.org/issue25664 opened by zephor #25666: Python unexpectedly ignores a signal after fork http://bugs.python.org/issue25666 opened by jon orebro #25667: Supply dual-stack (IPv4/IPv6) socket bind routine http://bugs.python.org/issue25667 opened by jason.coombs #25668: Deadlock in logging caused by a possible race condition with " http://bugs.python.org/issue25668 opened by fviard #25669: unittest.assertEqual() on un-equal types inheriting from colle http://bugs.python.org/issue25669 opened by Nathan Herring #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 opened by Lucretiel #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 opened by Lukasa #25674: test_ssl (test_algorithms) failures on bolen-ubuntu slaves: sh http://bugs.python.org/issue25674 opened by db3l #25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, http://bugs.python.org/issue25675 opened by ealfie #25677: Syntax error caret confused by indentation http://bugs.python.org/issue25677 opened by Michael Layzell #25678: int() from a buffer reads past the buffer boundaries http://bugs.python.org/issue25678 opened by svenberkvens #25680: Selector.select() hangs when there is nothing to select http://bugs.python.org/issue25680 opened by Aleksey Kladov #25682: __package__ not set to None under pdb in Python 3 http://bugs.python.org/issue25682 opened by Keith Prussing #25683: __context__ for yields inside except clause http://bugs.python.org/issue25683 opened by yselivanov Most recent 15 issues with no replies (15) ========================================== #25683: __context__ for yields inside except clause http://bugs.python.org/issue25683 #25682: __package__ not set to None under pdb in Python 3 http://bugs.python.org/issue25682 #25675: doc for BaseEventLoop.run_in_executor() says its a coroutine, http://bugs.python.org/issue25675 #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 #25664: Unexpected UnicodeDecodeError in logging module http://bugs.python.org/issue25664 #25661: tokenize.untokenize does not maintain the order of tabbed inde http://bugs.python.org/issue25661 #25658: PyThread assumes pthread_key_t is an integer, which is against http://bugs.python.org/issue25658 #25657: virtualenv's activate does not update LD_LIBRARY_PATH http://bugs.python.org/issue25657 #25653: ctypes+callbacks+fork+selinux = crash http://bugs.python.org/issue25653 #25651: Confusing output for TestCase.subTest(0) http://bugs.python.org/issue25651 #25650: Mismatching documentation <=> behaviour for typing.Any http://bugs.python.org/issue25650 #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 #25644: Unable to open IDLE on Windows 10 with Python 2.7.10 http://bugs.python.org/issue25644 #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 #25636: Let launcher autoselect best 32-bit Python on 64-bit system http://bugs.python.org/issue25636 Most recent 15 issues waiting for review (15) ============================================= #25678: int() from a buffer reads past the buffer boundaries http://bugs.python.org/issue25678 #25677: Syntax error caret confused by indentation http://bugs.python.org/issue25677 #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 #25671: Fix venv activate.fish to maintain $status http://bugs.python.org/issue25671 #25663: Make rlcompleter avoid duplicate global names http://bugs.python.org/issue25663 #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 #25647: Return of asyncio.coroutine from asyncio.coroutine doesn't wor http://bugs.python.org/issue25647 #25643: Python tokenizer rewriting http://bugs.python.org/issue25643 #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 #25627: distutils : file "bdist_rpm.py" does not quote filenames when http://bugs.python.org/issue25627 #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 #25624: shutil.make_archive makes invalid directory entries http://bugs.python.org/issue25624 #25623: Keep the link to Python implementation of OrderedDict http://bugs.python.org/issue25623 #25622: Enable ctypes.test.test_values.Win_ValuesTestCase for non-Wind http://bugs.python.org/issue25622 Top 10 most discussed issues (10) ================================= #25626: Gzip fails for file over 2**32 bytes http://bugs.python.org/issue25626 13 msgs #25612: nested try..excepts don't work correctly for generators http://bugs.python.org/issue25612 12 msgs #25660: tabs don't work correctly in python repl http://bugs.python.org/issue25660 8 msgs #25654: test_multiprocessing_spawn ResourceWarning with -Werror http://bugs.python.org/issue25654 7 msgs #25672: set SSL_MODE_RELEASE_BUFFERS http://bugs.python.org/issue25672 7 msgs #23883: __all__ lists are incomplete http://bugs.python.org/issue23883 6 msgs #25638: Verify the etree_parse and etree_iterparse benchmarks are work http://bugs.python.org/issue25638 6 msgs #25646: Distutils and Windows SDK 7.1 http://bugs.python.org/issue25646 6 msgs #25628: Make namedtuple "verbose" and "rename" parameters into keyword http://bugs.python.org/issue25628 5 msgs #25640: EmailMessage should throw an error if policy is set to compat3 http://bugs.python.org/issue25640 5 msgs Issues closed (36) ================== #6973: subprocess.Popen.send_signal doesn't check whether the process http://bugs.python.org/issue6973 closed by gregory.p.smith #8823: urllib2 does not catch httplib.BadStatusLine http://bugs.python.org/issue8823 closed by martin.panter #17131: subprocess.Popen.terminate can raise exception on Posix http://bugs.python.org/issue17131 closed by gregory.p.smith #20220: test_datetime, test_tarfile, test_strptime time zone failures http://bugs.python.org/issue20220 closed by martin.panter #20387: tokenize/untokenize roundtrip fails with tabs http://bugs.python.org/issue20387 closed by jason.coombs #21779: test_multiprocessing_spawn fails when ran with -Werror http://bugs.python.org/issue21779 closed by serhiy.storchaka #23564: Patch fixing sanity check for ordered fd sequence in _posixsub http://bugs.python.org/issue23564 closed by gregory.p.smith #24172: Errors in resource.getpagesize module documentation http://bugs.python.org/issue24172 closed by martin.panter #24623: Parser: broken line numbers for triple-quoted strings http://bugs.python.org/issue24623 closed by serhiy.storchaka #25017: htmllib deprecated: Which library to use? Missing sane default http://bugs.python.org/issue25017 closed by martin.panter #25168: test_datetime.test_strptime() random failures on "s390x SLES" http://bugs.python.org/issue25168 closed by martin.panter #25388: tokenizer crash/misbehavior -- heap use-after-free http://bugs.python.org/issue25388 closed by serhiy.storchaka #25472: Typing: Specialized subclasses of generics cannot be unpickled http://bugs.python.org/issue25472 closed by gvanrossum #25498: Python 3.4.3 regression in ctypes.frombuffer() http://bugs.python.org/issue25498 closed by martin.panter #25557: Optimize LOAD_NAME bytecode http://bugs.python.org/issue25557 closed by haypo #25578: Memory leak in SSLSocket.getpeercert() with 0-length AIA exten http://bugs.python.org/issue25578 closed by python-dev #25583: os.makedirs with exist_ok=True raises PermissionError on Windo http://bugs.python.org/issue25583 closed by martin.panter #25590: tab-completition on instances repeatedly accesses attribute/de http://bugs.python.org/issue25590 closed by martin.panter #25611: test_deque failure on Gentoo and OpenIndiana buildbots on 2.7 http://bugs.python.org/issue25611 closed by rhettinger #25615: Document unsorted behaviour of glob.glob http://bugs.python.org/issue25615 closed by martin.panter #25618: gdk attribute error http://bugs.python.org/issue25618 closed by r.david.murray #25619: Exception AttributeError: "'NoneType'.... thrown on exit http://bugs.python.org/issue25619 closed by r.david.murray #25620: Bytes Issue http://bugs.python.org/issue25620 closed by SilentGhost #25621: ast.literal_eval() fails on docstrings http://bugs.python.org/issue25621 closed by r.david.murray #25629: Move set fill/used updates out of set_insert_clean() http://bugs.python.org/issue25629 closed by rhettinger #25630: SIGSEGV in os.putenv() http://bugs.python.org/issue25630 closed by python-dev #25634: Add a dedicated subclass for attribute missing errors http://bugs.python.org/issue25634 closed by r.david.murray #25645: Reference leak in test_capi, on "import _pickle" in a subinter http://bugs.python.org/issue25645 closed by python-dev #25648: asyncio.coroutine fails if asyncio debug mode is enabled and w http://bugs.python.org/issue25648 closed by gvanrossum #25662: _tkinter.TclError: bad event type or keysym "Alt" http://bugs.python.org/issue25662 closed by ronixx #25665: typing.NamedTuple instances are not picklable. http://bugs.python.org/issue25665 closed by gvanrossum #25670: ast.NodeTransformer: remove duplicate getattr() http://bugs.python.org/issue25670 closed by haypo #25673: unittest assertLessEqual not working properly with lists http://bugs.python.org/issue25673 closed by ezio.melotti #25676: Spam http://bugs.python.org/issue25676 closed by zach.ware #25679: Fix typo in Doc/reference/executionmodel.rst http://bugs.python.org/issue25679 closed by r.david.murray #25681: Assignment of one element in nested list changes multiple elem http://bugs.python.org/issue25681 closed by haypo

1 0

nested try..except bug in generators
by Yury Selivanov Nov. 20, 2015

Nov. 20, 2015

Hi, For anyone interested in tickling the low-level CPython internals (in this case eval loop and generators implementations), here's an interesting bug: http://bugs.python.org/issue25612. I have a patch that fixes it, but I'm not sure that it's correct, and I desperately need someone to review it. While debugging the issue, I found another oddity: class MainError(Exception): pass class SubError(Exception): pass def main(): try: raise MainError() except MainError: yield coro = main() coro.send(None) coro.throw(SubError()) On the last line of the above snippet, SubError will propagate. However, it won't have it's __context__ set to MainError. Martin Panter suggests that it's not a bug, and after giving this some thought, I now don't think it's a bug too -- SubError was instantiated outside of MainError from a code location where there was no error. Thoughts? Here's a separate issue for the __context__ bug/feature: http://bugs.python.org/issue25683 Thanks, Yury

1 0

Improving the reading part of REPL
by Adam Bartoš Nov. 20, 2015

Nov. 20, 2015

It seems that there will be some refactoring of the tokenizer code. Regarding this, I'd like to recall my proposal on readline hooks. It would be nice if char* based PyOS_Readline API was replaced by a Python str based hook customizable by Python code. I propose to add function sys.readlinehook accepting optional prompt and returning a line read interactively from a user. There would also be sys.__readlinehook__ containing the original value of sys.readlinehook (similarly to sys.(__)displayhook(__), sys.(__)excepthook(__) and sys.(__)std(in/out/err)(__)). Currently, the input is read from C stdin even if sys.stdin is changed (see http://bugs.python.org/issue17620). This complicates fixing http://bugs.python.org/issue1602 – the standard sys.std* streams are not capable of communicating in Unicode with Windows console, and replacing the streams with custom ones is not enough – one has also to install a custom readline hook, which is currently complicated. And even after installing a custom readine hook one finds out that Python tokenizer cannot handle UTF-16, so he has to wrap the custom stream objects just to let their encoding attribute have a different value, because readlinehook currently returns char* rather than a Python string. For more details see the documentation of my package: https://github.com/Drekin/win-unicode-console. The pyreadline package also sets up a custom readline so it would benefit if doing so would be easier. Moreover, the two consumers of PyOS_Readline API – the input function and the tokenizer – assume a different encoding of the bytes returned by the readlinehook. Effectively, one assumes sys.stdout.encoding and the other sys.stdin.encoding, so if these two are different, there is no way to implement a correct readline hook. If sys.readlinehook was added, the builting input function would be just a thin wrapper over sys.readlinehook removing the newline character and turning no input into EOFError. I thing that the best default value for sys.readlinehook on Windows would be stdio_readline – just write the prompt to sys.stdout and read a line from sys.stdin. On Linux, the default implementation would call GNU readline if it is available and sys.stdin and sys.stdout are standard TTYs (the check present in the current implementation of the input function), and it would call stdio_readline otherwise. Regards, Adam Bartoš

1 1

Reading Python source file
by Serhiy Storchaka Nov. 19, 2015

Nov. 19, 2015

I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python source file). The code is complicated. For now there are such cases: * Reading from the string in memory. * Interactive reading from the file. * Reading from the file: - Raw reading ignoring encoding in parser generator. - Raw reading UTF-8 encoded file. - Reading and recoding to UTF-8. The file is read by the line. It makes hard to check correctness of the first line if the encoding is specified in the second line. And it makes very hard problems with null bytes and with desynchronizing buffered C and Python files. All this problems can be easily solved if read all Python source file in memory and then parse it as string. This would allow to drop a large complex and buggy part of code. Are there disadvantages in this solution? As for memory consumption, the source text itself will consume only small part of the memory consumed by AST tree and other structures. As for performance, reading and decoding all file can be faster then by the line. [1] http://bugs.python.org/issue25643

10 27

Python stdlib ssl.SSLContext is missing mode setting ability
by Ben Bangert Nov. 19, 2015

Nov. 19, 2015

In Python 2 and 3, the ssl module's SSLContext object has a way to set SSL options, but not to set SSL modes. The set_mode command and some of the available modes: https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl library allows the setting of SSLContext modes, it seems very odd that the Python 2/3 ssl modules do not. Though I could understand that perhaps not all SSL libraries Python might build against would have this mode thing available. (BoringSSL sets this mode by default its considered such an obvious win) If there is some way to set this I happened to miss, apologies, I only went looking through the docs for it, not the code. On a side-note, in my testing, Python 3.4 had about 20kb/connection of overhead for using SSL, but Python 3.5 jumped to 30kb/connection of SSL overhead. These numbers for SSL overhead are far too high for any reasonable use of Python+SSL on highly concurrent systems. Test repo for testing SSL overhead here: https://github.com/bbangert/ssl-ram-testing/ Cheers, Ben

3 4

Re: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix versionadded
by Martin Panter Nov. 19, 2015

Nov. 19, 2015

On 18 November 2015 at 17:45, yury.selivanov <python-checkins(a)python.org> wrote: > https://hg.python.org/cpython/rev/b34c42e46e7b > changeset: 99204:b34c42e46e7b > branch: 3.4 > parent: 99201:89d66f912671 > user: Yury Selivanov <yselivanov(a)sprymix.com> > date: Wed Nov 18 12:44:31 2015 -0500 > summary: > asyncio.docs: Fix versionadded > > files: > Doc/library/asyncio-task.rst | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst > --- a/Doc/library/asyncio-task.rst > +++ b/Doc/library/asyncio-task.rst > @@ -699,4 +699,4 @@ > Unlike the functions above, :func:`run_coroutine_threadsafe` requires the > *loop* argument to be passed explicitely. > > - .. versionadded:: 3.4.4 > + .. versionadded:: 3.4.4, 3.5.1 This doesn’t get marked up properly; have a look at <https://docs.python.org/3.4/library/asyncio-task.html#asyncio.run_coroutine…>.

1 0

Benchmark results across all major Python implementations
by Brett Cannon Nov. 18, 2015

Nov. 18, 2015

I gave the opening keynote at PyCon CA and then gave the same talk at PyData NYC on the various interpreters of Python (Jupyter notebook of my presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I figured people here might find the benchmark numbers interesting so I'm sharing the link here. I'm still hoping someday speed.python.org becomes a thing so I never have to spend so much time benchmarking so may Python implementations ever again and this sort of thing is just part of what we do to keep the implementation ecosystem healthy.

10 14

Re: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10
by Brett Cannon Nov. 17, 2015

Nov. 17, 2015

Just an FYI there seems to be a consistent, minor refcount leak found by test_capi that has been there for what seems like a couple of weeks. On Mon, 16 Nov 2015 at 00:42 <solipsis(a)pitrou.net> wrote: > results for 97e2a6810f7f on branch "default" > -------------------------------------------- > > test_asyncio leaked [0, 0, 3] memory blocks, sum=3 > test_capi leaked [1, 1, 1] references, sum=3 > test_functools leaked [0, 2, 2] memory blocks, sum=4 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200'] > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

2 1

Clarification of PEP 394 for scripts that run under Python 2 and 3
by Damien George Nov. 15, 2015

Nov. 15, 2015

Hi python-dev, We have a Python script that runs correctly under Python 2.6, 2.7 and 3.3+. It is executed on a *nix system using the "python" executable (ie not python2 or python3 specifically). This works just fine for systems that have Python 2 installed, or 2 and 3, or just 3 and symlink "python" to "python3" (eg Arch Linux). But it fails for systems that have only Python 3 and do not create a "python" symlink, ie only "python3" exists as an executable. We thought that PEP 394 would come to the rescue here but it seems to be unclear on this point. In particular it says: - 4th point of the abstract: "so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3" - 6th point of the recommendation section: "One exception to this is scripts that are deliberately written to be source compatible with both Python 2.x and 3.x. Such scripts may continue to use python on their shebang line without affecting their portability" - 8th point in the migration notes: "If these conventions are adhered to, it will become the case that the python command is only executed in an interactive manner as a user convenience, or to run scripts that are source compatible with both Python 2 and Python 3." Well, that's pretty clear to me: one can expect the "python" executable to be available to run scripts that are compatible with versions 2.x and 3.x. The confusion comes because there are systems that install Python 3 without creating a "python" symlink (hence breaking the above). And Guido said that "'python' should always be the same as 'python2'" (see https://mail.python.org/pipermail/python-dev/2014-September/136389.html). Further, Nick Coghlan seemed to agree that "when there's only python3 installed, there should be no /usr/bin/python" (see https://mail.python.org/pipermail/python-dev/2014-September/136527.html). My questions are: 1. What is the true intent of PEP 394 when only Python 3 is installed? Is "python" available or not to run scripts compatible with 2.x and 3.x? 2. Is it possible to write a shebang line that supports all variations of Python installations on *nix machines? 3. If the answer to 2 is no, then what is the recommended way to support all Python installations with one standalone script? Thanks! Regards, Damien.

3 2

Support of UTF-16 and UTF-32 source encodings
by Serhiy Storchaka Nov. 15, 2015

Nov. 15, 2015

For now UTF-16 and UTF-32 source encodings are not supported. There is a comment in Parser/tokenizer.c: /* Disable support for UTF-16 BOMs until a decision is made whether this needs to be supported. */ Can we make a decision whether this support will be added in foreseeable future (say in near 10 years), or no? Removing commented out and related code will help to refactor the tokenizer, and that can help to fix some existing bugs (e.g. issue14811, issue18961, issue20115 and may be others). Current tokenizing code is too tangled. If the support of UTF-16 and UTF-32 is planned, I'll take this to attention during refactoring. But in many places besides the tokenizer the ASCII compatible encoding of source files is expected.

16 28