Python-Dev November 2015

python-dev@python.org

69 participants
47 discussions

by Adam Bartoš

It seems that there will be some refactoring of the tokenizer code. Regarding this, I'd like to recall my proposal on readline hooks. It would be nice if char* based PyOS_Readline API was replaced by a Python str based hook customizable by Python code. I propose to add function sys.readlinehook accepting optional prompt and returning a line read interactively from a user. There would also be sys.__readlinehook__ containing the original value of sys.readlinehook (similarly to sys.(__)displayhook(__), sys.(__)excepthook(__) and sys.(__)std(in/out/err)(__)). Currently, the input is read from C stdin even if sys.stdin is changed (see http://bugs.python.org/issue17620). This complicates fixing http://bugs.python.org/issue1602 – the standard sys.std* streams are not capable of communicating in Unicode with Windows console, and replacing the streams with custom ones is not enough – one has also to install a custom readline hook, which is currently complicated. And even after installing a custom readine hook one finds out that Python tokenizer cannot handle UTF-16, so he has to wrap the custom stream objects just to let their encoding attribute have a different value, because readlinehook currently returns char* rather than a Python string. For more details see the documentation of my package: https://github.com/Drekin/win-unicode-console. The pyreadline package also sets up a custom readline so it would benefit if doing so would be easier. Moreover, the two consumers of PyOS_Readline API – the input function and the tokenizer – assume a different encoding of the bytes returned by the readlinehook. Effectively, one assumes sys.stdout.encoding and the other sys.stdin.encoding, so if these two are different, there is no way to implement a correct readline hook. If sys.readlinehook was added, the builting input function would be just a thin wrapper over sys.readlinehook removing the newline character and turning no input into EOFError. I thing that the best default value for sys.readlinehook on Windows would be stdio_readline – just write the prompt to sys.stdout and read a line from sys.stdin. On Linux, the default implementation would call GNU readline if it is available and sys.stdin and sys.stdout are standard TTYs (the check present in the current implementation of the input function), and it would call stdio_readline otherwise. Regards, Adam Bartoš

4 years, 10 months

Reading Python source file

by Serhiy Storchaka

I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python source file). The code is complicated. For now there are such cases: * Reading from the string in memory. * Interactive reading from the file. * Reading from the file: - Raw reading ignoring encoding in parser generator. - Raw reading UTF-8 encoded file. - Reading and recoding to UTF-8. The file is read by the line. It makes hard to check correctness of the first line if the encoding is specified in the second line. And it makes very hard problems with null bytes and with desynchronizing buffered C and Python files. All this problems can be easily solved if read all Python source file in memory and then parse it as string. This would allow to drop a large complex and buggy part of code. Are there disadvantages in this solution? As for memory consumption, the source text itself will consume only small part of the memory consumed by AST tree and other structures. As for performance, reading and decoding all file can be faster then by the line. [1] http://bugs.python.org/issue25643

4 years, 10 months

Python stdlib ssl.SSLContext is missing mode setting ability

by Ben Bangert

In Python 2 and 3, the ssl module's SSLContext object has a way to set SSL options, but not to set SSL modes. The set_mode command and some of the available modes: https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl library allows the setting of SSLContext modes, it seems very odd that the Python 2/3 ssl modules do not. Though I could understand that perhaps not all SSL libraries Python might build against would have this mode thing available. (BoringSSL sets this mode by default its considered such an obvious win) If there is some way to set this I happened to miss, apologies, I only went looking through the docs for it, not the code. On a side-note, in my testing, Python 3.4 had about 20kb/connection of overhead for using SSL, but Python 3.5 jumped to 30kb/connection of SSL overhead. These numbers for SSL overhead are far too high for any reasonable use of Python+SSL on highly concurrent systems. Test repo for testing SSL overhead here: https://github.com/bbangert/ssl-ram-testing/ Cheers, Ben

4 years, 10 months

Re: [Python-Dev] [Python-checkins] cpython (3.4): asyncio.docs: Fix versionadded

by Martin Panter

On 18 November 2015 at 17:45, yury.selivanov <python-checkins(a)python.org> wrote: > https://hg.python.org/cpython/rev/b34c42e46e7b > changeset: 99204:b34c42e46e7b > branch: 3.4 > parent: 99201:89d66f912671 > user: Yury Selivanov <yselivanov(a)sprymix.com> > date: Wed Nov 18 12:44:31 2015 -0500 > summary: > asyncio.docs: Fix versionadded > > files: > Doc/library/asyncio-task.rst | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst > --- a/Doc/library/asyncio-task.rst > +++ b/Doc/library/asyncio-task.rst > @@ -699,4 +699,4 @@ > Unlike the functions above, :func:`run_coroutine_threadsafe` requires the > *loop* argument to be passed explicitely. > > - .. versionadded:: 3.4.4 > + .. versionadded:: 3.4.4, 3.5.1 This doesn’t get marked up properly; have a look at <https://docs.python.org/3.4/library/asyncio-task.html#asyncio.run_coroutine…>.

4 years, 10 months

Benchmark results across all major Python implementations

by Brett Cannon

I gave the opening keynote at PyCon CA and then gave the same talk at PyData NYC on the various interpreters of Python (Jupyter notebook of my presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I figured people here might find the benchmark numbers interesting so I'm sharing the link here. I'm still hoping someday speed.python.org becomes a thing so I never have to spend so much time benchmarking so may Python implementations ever again and this sort of thing is just part of what we do to keep the implementation ecosystem healthy.

4 years, 10 months

Re: [Python-Dev] [Python-checkins] Daily reference leaks (97e2a6810f7f): sum=10

by Brett Cannon

Just an FYI there seems to be a consistent, minor refcount leak found by test_capi that has been there for what seems like a couple of weeks. On Mon, 16 Nov 2015 at 00:42 <solipsis(a)pitrou.net> wrote: > results for 97e2a6810f7f on branch "default" > -------------------------------------------- > > test_asyncio leaked [0, 0, 3] memory blocks, sum=3 > test_capi leaked [1, 1, 1] references, sum=3 > test_functools leaked [0, 2, 2] memory blocks, sum=4 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200'] > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

4 years, 10 months

Clarification of PEP 394 for scripts that run under Python 2 and 3

by Damien George

Hi python-dev, We have a Python script that runs correctly under Python 2.6, 2.7 and 3.3+. It is executed on a *nix system using the "python" executable (ie not python2 or python3 specifically). This works just fine for systems that have Python 2 installed, or 2 and 3, or just 3 and symlink "python" to "python3" (eg Arch Linux). But it fails for systems that have only Python 3 and do not create a "python" symlink, ie only "python3" exists as an executable. We thought that PEP 394 would come to the rescue here but it seems to be unclear on this point. In particular it says: - 4th point of the abstract: "so python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3" - 6th point of the recommendation section: "One exception to this is scripts that are deliberately written to be source compatible with both Python 2.x and 3.x. Such scripts may continue to use python on their shebang line without affecting their portability" - 8th point in the migration notes: "If these conventions are adhered to, it will become the case that the python command is only executed in an interactive manner as a user convenience, or to run scripts that are source compatible with both Python 2 and Python 3." Well, that's pretty clear to me: one can expect the "python" executable to be available to run scripts that are compatible with versions 2.x and 3.x. The confusion comes because there are systems that install Python 3 without creating a "python" symlink (hence breaking the above). And Guido said that "'python' should always be the same as 'python2'" (see https://mail.python.org/pipermail/python-dev/2014-September/136389.html). Further, Nick Coghlan seemed to agree that "when there's only python3 installed, there should be no /usr/bin/python" (see https://mail.python.org/pipermail/python-dev/2014-September/136527.html). My questions are: 1. What is the true intent of PEP 394 when only Python 3 is installed? Is "python" available or not to run scripts compatible with 2.x and 3.x? 2. Is it possible to write a shebang line that supports all variations of Python installations on *nix machines? 3. If the answer to 2 is no, then what is the recommended way to support all Python installations with one standalone script? Thanks! Regards, Damien.

4 years, 10 months

Support of UTF-16 and UTF-32 source encodings

by Serhiy Storchaka

For now UTF-16 and UTF-32 source encodings are not supported. There is a comment in Parser/tokenizer.c: /* Disable support for UTF-16 BOMs until a decision is made whether this needs to be supported. */ Can we make a decision whether this support will be added in foreseeable future (say in near 10 years), or no? Removing commented out and related code will help to refactor the tokenizer, and that can help to fix some existing bugs (e.g. issue14811, issue18961, issue20115 and may be others). Current tokenizing code is too tangled. If the support of UTF-16 and UTF-32 is planned, I'll take this to attention during refactoring. But in many places besides the tokenizer the ASCII compatible encoding of source files is expected.

4 years, 10 months

Rietveld is broken

by Andrew Svetlov

When I try to press 'retry' on http://bugs.python.org/review/25074/ page and send a message server responds with 500 error. Going through the page for viewing diff for concrete file http://bugs.python.org/review/25074/diff/15535/Lib/test/test_asyncio/test_b… with adding a comment works well. The behavior is very annoying -- I was forced to write missed message twice. -- Thanks, Andrew Svetlov

4 years, 10 months

doc tests failing

by Ethan Furman

What am I doing wrong? I have tried: -------------------------------------------------------- hg update 3.5 # and hg update default make distclean && ./configure --with-pydebug && make -j2 cd Doc make doctest -------------------------------------------------------- and in both cases I get page after page of errors. I have tried installing python-sphinx and python3-sphinx; I have tried adding PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to no avail. Here's a random sample of the errors: ********************************************************************** File "library/shlex.rst", line ?, in default Failed example: remote_command Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "<doctest default[1]>", line 1, in <module> remote_command NameError: name 'remote_command' is not defined ********************************************************************** File "howto/sorting.rst", line ?, in default Failed example: sorted([5, 2, 4, 1, 3], cmp=numeric_compare) Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "<doctest default[1]>", line 1, in <module> sorted([5, 2, 4, 1, 3], cmp=numeric_compare) NameError: name 'numeric_compare' is not defined ********************************************************************** File "library/ipaddress.rst", line ?, in default Failed example: n2 = ip_network('192.0.2.1/32') Exception raised: Traceback (most recent call last): File "/usr/lib/python2.7/doctest.py", line 1315, in __run compileflags, 1) in test.globs File "<doctest default[1]>", line 1, in <module> n2 = ip_network('192.0.2.1/32') NameError: name 'ip_network' is not defined -- ~Ethan~

4 years, 10 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev November 2015