It seems that there will be some refactoring of the tokenizer code.
Regarding this, I'd like to recall my proposal on readline hooks. It would
be nice if char* based PyOS_Readline API was replaced by a Python str based
hook customizable by Python code. I propose to add function
sys.readlinehook accepting optional prompt and returning a line read
interactively from a user. There would also be sys.__readlinehook__
containing the original value of sys.readlinehook (similarly to
sys.(__)displayhook(__), sys.(__)excepthook(__) and
sys.(__)std(in/out/err)(__)).
Currently, the input is read from C stdin even if sys.stdin is changed (see
http://bugs.python.org/issue17620). This complicates fixing
http://bugs.python.org/issue1602 – the standard sys.std* streams are not
capable of communicating in Unicode with Windows console, and replacing the
streams with custom ones is not enough – one has also to install a custom
readline hook, which is currently complicated. And even after installing a
custom readine hook one finds out that Python tokenizer cannot handle
UTF-16, so he has to wrap the custom stream objects just to let their
encoding attribute have a different value, because readlinehook currently
returns char* rather than a Python string. For more details see the
documentation of my package: https://github.com/Drekin/win-unicode-console.
The pyreadline package also sets up a custom readline so it would benefit
if doing so would be easier. Moreover, the two consumers of PyOS_Readline
API – the input function and the tokenizer – assume a different encoding of
the bytes returned by the readlinehook. Effectively, one assumes
sys.stdout.encoding and the other sys.stdin.encoding, so if these two are
different, there is no way to implement a correct readline hook.
If sys.readlinehook was added, the builting input function would be just a
thin wrapper over sys.readlinehook removing the newline character and
turning no input into EOFError. I thing that the best default value for
sys.readlinehook on Windows would be stdio_readline – just write the prompt
to sys.stdout and read a line from sys.stdin. On Linux, the default
implementation would call GNU readline if it is available and sys.stdin and
sys.stdout are standard TTYs (the check present in the current
implementation of the input function), and it would call stdio_readline
otherwise.
Regards, Adam Bartoš
I'm working on rewriting Python tokenizer (in particular the part that
reads and decodes Python source file). The code is complicated. For now
there are such cases:
* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
- Raw reading ignoring encoding in parser generator.
- Raw reading UTF-8 encoded file.
- Reading and recoding to UTF-8.
The file is read by the line. It makes hard to check correctness of the
first line if the encoding is specified in the second line. And it makes
very hard problems with null bytes and with desynchronizing buffered C
and Python files. All this problems can be easily solved if read all
Python source file in memory and then parse it as string. This would
allow to drop a large complex and buggy part of code.
Are there disadvantages in this solution? As for memory consumption, the
source text itself will consume only small part of the memory consumed
by AST tree and other structures. As for performance, reading and
decoding all file can be faster then by the line.
[1] http://bugs.python.org/issue25643
In Python 2 and 3, the ssl module's SSLContext object has a way to set
SSL options, but not to set SSL modes.
The set_mode command and some of the available modes:
https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_set_mode.html
The most critical mode is SSL_MODE_RELEASE_BUFFERS, which can drop the
SSL overhead *per connection* from around 25kb to ~7kb. The pyopenssl
library allows the setting of SSLContext modes, it seems very odd that
the Python 2/3 ssl modules do not. Though I could understand that
perhaps not all SSL libraries Python might build against would have
this mode thing available.
(BoringSSL sets this mode by default its considered such an obvious win)
If there is some way to set this I happened to miss, apologies, I only
went looking through the docs for it, not the code.
On a side-note, in my testing, Python 3.4 had about 20kb/connection of
overhead for using SSL, but Python 3.5 jumped to 30kb/connection of
SSL overhead. These numbers for SSL overhead are far too high for any
reasonable use of Python+SSL on highly concurrent systems. Test repo
for testing SSL overhead here:
https://github.com/bbangert/ssl-ram-testing/
Cheers,
Ben
I gave the opening keynote at PyCon CA and then gave the same talk at
PyData NYC on the various interpreters of Python (Jupyter notebook of my
presentation can be found at bit.ly/pycon-ca-keynote; no video yet). I
figured people here might find the benchmark numbers interesting so I'm
sharing the link here.
I'm still hoping someday speed.python.org becomes a thing so I never have
to spend so much time benchmarking so may Python implementations ever again
and this sort of thing is just part of what we do to keep the
implementation ecosystem healthy.
Just an FYI there seems to be a consistent, minor refcount leak found by
test_capi that has been there for what seems like a couple of weeks.
On Mon, 16 Nov 2015 at 00:42 <solipsis(a)pitrou.net> wrote:
> results for 97e2a6810f7f on branch "default"
> --------------------------------------------
>
> test_asyncio leaked [0, 0, 3] memory blocks, sum=3
> test_capi leaked [1, 1, 1] references, sum=3
> test_functools leaked [0, 2, 2] memory blocks, sum=4
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogBLsY2a', '--timeout', '7200']
> _______________________________________________
> Python-checkins mailing list
> Python-checkins(a)python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
Hi python-dev,
We have a Python script that runs correctly under Python 2.6, 2.7 and 3.3+.
It is executed on a *nix system using the "python" executable (ie not
python2 or python3 specifically). This works just fine for systems that
have Python 2 installed, or 2 and 3, or just 3 and symlink "python" to
"python3" (eg Arch Linux).
But it fails for systems that have only Python 3 and do not create a
"python" symlink, ie only "python3" exists as an executable.
We thought that PEP 394 would come to the rescue here but it seems to be
unclear on this point. In particular it says:
- 4th point of the abstract: "so python should be used in the shebang line
only for scripts that are source compatible with both Python 2 and 3"
- 6th point of the recommendation section: "One exception to this is
scripts that are deliberately written to be source compatible with both
Python 2.x and 3.x. Such scripts may continue to use python on their
shebang line without affecting their portability"
- 8th point in the migration notes: "If these conventions are adhered to,
it will become the case that the python command is only executed in an
interactive manner as a user convenience, or to run scripts that are source
compatible with both Python 2 and Python 3."
Well, that's pretty clear to me: one can expect the "python" executable to
be available to run scripts that are compatible with versions 2.x and 3.x.
The confusion comes because there are systems that install Python 3 without
creating a "python" symlink (hence breaking the above). And Guido said
that "'python' should always be the same as 'python2'" (see
https://mail.python.org/pipermail/python-dev/2014-September/136389.html).
Further, Nick Coghlan seemed to agree that "when there's only python3
installed, there should be no /usr/bin/python" (see
https://mail.python.org/pipermail/python-dev/2014-September/136527.html).
My questions are:
1. What is the true intent of PEP 394 when only Python 3 is installed? Is
"python" available or not to run scripts compatible with 2.x and 3.x?
2. Is it possible to write a shebang line that supports all variations of
Python installations on *nix machines?
3. If the answer to 2 is no, then what is the recommended way to support
all Python installations with one standalone script?
Thanks!
Regards,
Damien.
For now UTF-16 and UTF-32 source encodings are not supported. There is a
comment in Parser/tokenizer.c:
/* Disable support for UTF-16 BOMs until a decision
is made whether this needs to be supported. */
Can we make a decision whether this support will be added in foreseeable
future (say in near 10 years), or no?
Removing commented out and related code will help to refactor the
tokenizer, and that can help to fix some existing bugs (e.g. issue14811,
issue18961, issue20115 and may be others). Current tokenizing code is
too tangled.
If the support of UTF-16 and UTF-32 is planned, I'll take this to
attention during refactoring. But in many places besides the tokenizer
the ASCII compatible encoding of source files is expected.
What am I doing wrong?
I have tried:
--------------------------------------------------------
hg update 3.5 # and hg update default
make distclean && ./configure --with-pydebug && make -j2
cd Doc
make doctest
--------------------------------------------------------
and in both cases I get page after page of errors. I have tried
installing python-sphinx and python3-sphinx; I have tried adding
PYTHON=../python and PYTHON=python3 to the `make doctest` line -- all to
no avail.
Here's a random sample of the errors:
**********************************************************************
File "library/shlex.rst", line ?, in default
Failed example:
remote_command
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python2.7/doctest.py", line 1315, in __run
compileflags, 1) in test.globs
File "<doctest default[1]>", line 1, in <module>
remote_command
NameError: name 'remote_command' is not defined
**********************************************************************
File "howto/sorting.rst", line ?, in default
Failed example:
sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python2.7/doctest.py", line 1315, in __run
compileflags, 1) in test.globs
File "<doctest default[1]>", line 1, in <module>
sorted([5, 2, 4, 1, 3], cmp=numeric_compare)
NameError: name 'numeric_compare' is not defined
**********************************************************************
File "library/ipaddress.rst", line ?, in default
Failed example:
n2 = ip_network('192.0.2.1/32')
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python2.7/doctest.py", line 1315, in __run
compileflags, 1) in test.globs
File "<doctest default[1]>", line 1, in <module>
n2 = ip_network('192.0.2.1/32')
NameError: name 'ip_network' is not defined
--
~Ethan~
