Paul Moore (p.f.moore at gmail.com) on Fri Sep 2 05:23:04 EDT 2016 wrote
> On 2 September 2016 at 03:35, Steve Dower <steve.dower at python.org <https://mail.python.org/mailman/listinfo/python-dev>> wrote:
> >* I'd need to test to be sure, but writing an incomplete code point should
> *>* just truncate to before that point. It may currently raise OSError if that
> *>* truncated to zero length, as I believe that's not currently distinguished
> *>* from an error. What behavior would you propose?
> For "correct" behaviour, you should retain the unwritten bytes, and
> write them as part of the next call (essentially making the API
> stateful, in the same way that incremental codecs work). I'm pretty
> sure that this could cause actual problems, for example I think invoke
> (https://github.com/pyinvoke/invoke) gets byte streams from
> subprocesses and dumps them direct to stdout in blocks (so could
> easily end up splitting multibyte sequences). It''s arguable that it
> should be decoding the bytes from the subprocess and then re-encoding
> them, but that gets us into "guess the encoding used by the
> subprocess" territory.
> The problem is that we're not going to simply drop some bad data in
> the common case - it's not so much the dropping of the start of an
> incomplete code point that bothers me, as the encoding error you hit
> at the start of the *next* block of data you send. So people will get
> random, unexplained, encoding errors.
> I don't see an easy answer here other than a stateful API.
Isn't the buffered IO wrapper for this?
> >* Reads of less than four bytes fail instantly, as in the worst case we need
> *>* four bytes to represent one Unicode character. This is an unfortunate
> *>* reality of trying to limit it to one system call - you'll never get a full
> *>* buffer from a single read, as there is no simple mapping between
> *>* length-as-utf8 and length-as-utf16 for an arbitrary string.
> And here - "read a single byte" is a not uncommon way of getting some
> data. Once again see invoke:
> used at
> I'm not saying that there's an easy answer here, but this *will* break
> code. And actually, it's in violation of the documentation: seehttps://docs.python.org/3/library/io.html#io.RawIOBase.read
> Read up to size bytes from the object and return them. As a
> convenience, if size is unspecified or -1, readall() is called.
> Otherwise, only one system call is ever made. Fewer than size bytes
> may be returned if the operating system call returns fewer than size
> If 0 bytes are returned, and size was not 0, this indicates end of
> file. If the object is in non-blocking mode and no bytes are
> available, None is returned.
> You're not allowed to return 0 bytes if the requested size was not 0,
> and you're not at EOF.
That's why it should be rather signaled by an exception. Even when one
doesn't transcode UTF-16 to UTF-8, reading just one byte is still
impossible I would argue that also incorrect here. I raise ValueError in
We've prepared an updated version of PEP 526:
This changes the title to "Syntax for Variable Annotations", now that
we've settled on global, class, instance, and local variables as the
things you might annotate.
There is one substantial change: where the previous version supported only
TARGET: TYPE = VALUE
the new PEP removes the distinction and just allows
TARGET: TYPE [= VALUE]
This simplifies the explanations a bit and enables type checkers to
support separating the annotation from the assignment for instance
variables in the __init__ method, e.g.
if <some test>:
self.name = <something>
self.name = <something else>
The other changes are all minor editing nits, or clarifications about
the scope of the PEP. The scope clarification is important: while I
really want the new syntax settled in 3.6, I have no intention to pin
down the way type checkers use this syntax, apart from the observation
TARGET: TYPE = VALUE
is just meant as a cleaner way to write what you'd currently write
using PEP 484 as
TARGET = VALUE # type: TYPE
The PEP does *not* claim that you have to use variable annotations --
in fact we'd prefer that they were unnecessary, but the prevalence of
type comments in code we've annotated so far makes it clear that there
are plenty of uses for them, and we'd rather have a clean syntax for
them that tools can see in the AST.
--Guido van Rossum (python.org/~guido)
I’m using asyncio and paramiko to multiplex different channels into a single SSH connection. Things were working fine till recently but suddenly started seeing that python getting crashed whenever I tried to write to the channel. I have very limited knowledge on how python interpreter works, so I’m finding difficulty in understanding the stack trace. Can you please help in understanding the below backtarce.
bash-4.2$ gdb /usr/software/bin/python3.4.3 core.60015
Traceback (most recent call last):
File "<string>", line 70, in <module>
File "<string>", line 67, in GdbSetPythonDirectory
File "/usr/software/share/gdb/python/gdb/__init__.py", line 19, in <module>
ImportError: No module named _gdb
GNU gdb (GDB) 7.5
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
Reading symbols from /usr/software/bin/python3.4.3...done.
warning: core file may not match specified executable file.
[New LWP 60015]
[New LWP 60018]
[New LWP 60019]
[New LWP 60020]
[New LWP 60021]
[New LWP 60022]
[New LWP 60023]
[New LWP 60024]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/software/lib/libthread_db.so.1".
Core was generated by `/usr/software/bin/python3.4.3 /x/eng/bbrtp/users/amresh/sshproxy_3896926_160824'.
Program terminated with signal 11, Segmentation fault.
#0 _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
1159 Objects/obmalloc.c: No such file or directory.
#0 _PyObject_Malloc (ctx=0x0, nbytes=52) at Objects/obmalloc.c:1159
#1 0x00007ff2e511474a in PyUnicode_New (maxchar=<optimized out>, size=3) at Objects/unicodeobject.c:1093
#2 PyUnicode_New (size=3, maxchar=<optimized out>) at Objects/unicodeobject.c:1033
#3 0x00007ff2e5139da2 in _PyUnicodeWriter_PrepareInternal (writer=writer@entry=0x7fff3d5c8640, length=<optimized out>, maxchar=<optimized out>, maxchar@entry=127) at Objects/unicodeobject.c:13327
#4 0x00007ff2e513f38b in PyUnicode_DecodeUTF8Stateful (s=s@entry=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=size@entry=3,
errors=errors@entry=0x7ff2dee5dd70 "strict", consumed=consumed@entry=0x0) at Objects/unicodeobject.c:4757
#5 0x00007ff2e5140690 in PyUnicode_Decode (s=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", size=3, encoding=0x7ff2dee5df28 "utf-8", errors=0x7ff2dee5dd70 "strict")
#6 0x00007ff2de49bfdf in unpack_callback_raw (o=<synthetic pointer>, l=3, p=0x7ff2e3572f78 "tcp\245reuse\001\253socket_type\244pull\251transport\246zeromq", u=0x7fff3d5c8840, b=<optimized out>)
#7 unpack_execute<true> (ctx=ctx@entry=0x7fff3d5c8840, data=0x7ff2e3572ec0 "\205\245_auth\300\245_call\246expect\243_i", <incomplete sequence \332>, len=<optimized out>, off=off@entry=0x7fff3d5c8820)
#8 0x00007ff2de49fe3d in __pyx_pf_7msgpack_9_unpacker_2unpackb (__pyx_v_packed=__pyx_v_packed@entry=0x7ff2e3572ea0, __pyx_v_object_hook=__pyx_v_object_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>,
__pyx_v_list_hook=__pyx_v_list_hook@entry=0x7ff2e54934b0 <_Py_NoneStruct>, __pyx_v_use_list=1, __pyx_v_encoding=0x7ff2dee5df08, __pyx_v_unicode_errors=0x7ff2dee5dd50,
__pyx_v_object_pairs_hook=0x7ff2e54934b0 <_Py_NoneStruct>, __pyx_v_ext_hook=0x13db2d8, __pyx_v_max_str_len=__pyx_v_max_str_len@entry=2147483647,
__pyx_v_max_bin_len=__pyx_v_max_bin_len@entry=2147483647, __pyx_v_max_array_len=2147483647, __pyx_v_max_map_len=2147483647, __pyx_v_max_ext_len=__pyx_v_max_ext_len@entry=2147483647,
__pyx_self=<optimized out>) at msgpack/_unpacker.pyx:139
#9 0x00007ff2de4a1395 in __pyx_pw_7msgpack_9_unpacker_3unpackb (__pyx_self=<optimized out>, __pyx_args=<optimized out>, __pyx_kwds=<optimized out>) at msgpack/_unpacker.pyx:102
#10 0x00007ff2e5174ed3 in do_call (nk=<optimized out>, na=<optimized out>, pp_stack=0x7fff3d5d2b80, func=0x7ff2df20ddc8) at Python/ceval.c:4463
#11 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d2b80) at Python/ceval.c:4264
#12 PyEval_EvalFrameEx (f=f@entry=0x7ff2def02208, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#13 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x7ff2deefec30, kwcount=0, defs=0x0,
defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#14 0x00007ff2e51734da in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3d5d2e10, func=0x7ff2dee9b7b8) at Python/ceval.c:4344
#15 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d2e10) at Python/ceval.c:4262
#16 PyEval_EvalFrameEx (f=f@entry=0x7ff2deefea98, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#17 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, argcount=argcount@entry=1, kws=0x14566c8, kwcount=0,
defs=0x7ff2deeaedb8, defcount=1, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#18 0x00007ff2e51734da in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fff3d5d30a0, func=0x7ff2dee2dd90) at Python/ceval.c:4344
#19 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d30a0) at Python/ceval.c:4262
#20 PyEval_EvalFrameEx (f=f@entry=0x1456478, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#21 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=locals@entry=0x0, args=args@entry=0x7ff2d87364c0, argcount=1, kws=kws@entry=0x7ff2dee1de40,
kwcount=kwcount@entry=3, defs=defs@entry=0x7ff2e0820fd8, defcount=defcount@entry=3, kwdefs=0x0, closure=0x0) at Python/ceval.c:3588
#22 0x00007ff2e50d3320 in function_call (func=0x7ff2df1e9a60, arg=0x7ff2d87364a8, kw=0x7ff2d8738248) at Objects/funcobject.c:632
#23 0x00007ff2e50a76ca in PyObject_Call (func=func@entry=0x7ff2df1e9a60, arg=arg@entry=0x7ff2d87364a8, kw=kw@entry=0x7ff2d8738248) at Objects/abstract.c:2040
#24 0x00007ff2e50be55d in method_call (func=0x7ff2df1e9a60, arg=0x7ff2d87364a8, kw=0x7ff2d8738248) at Objects/classobject.c:347
#25 0x00007ff2e50a76ca in PyObject_Call (func=0x7ff2dee30e88, arg=arg@entry=0x7ff2e433d048, kw=kw@entry=0x7ff2d8738248) at Objects/abstract.c:2040
#26 0x00007ff2e51d9301 in partial_call (pto=0x7ff2deee1db8, args=<optimized out>, kw=0x0) at ./Modules/_functoolsmodule.c:127
#27 0x00007ff2e50a76ca in PyObject_Call (func=func@entry=0x7ff2deee1db8, arg=arg@entry=0x7ff2e433d048, kw=kw@entry=0x0) at Objects/abstract.c:2040
#28 0x00007ff2e51700a0 in ext_do_call (nk=-466366392, na=0, flags=<optimized out>, pp_stack=0x7fff3d5d3540, func=0x7ff2deee1db8) at Python/ceval.c:4561
#29 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2878
#30 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3710, func=0x7ff2e1540730) at Python/ceval.c:4334
#31 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3710) at Python/ceval.c:4262
#32 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#33 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d38f0, func=0x7ff2e12f2f28) at Python/ceval.c:4334
#34 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d38f0) at Python/ceval.c:4262
#35 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#36 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3ad0, func=0x7ff2e12f0c80) at Python/ceval.c:4334
#37 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3ad0) at Python/ceval.c:4262
#38 PyEval_EvalFrameEx (f=<optimized out>, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#39 0x00007ff2e51756a9 in fast_function (nk=<optimized out>, na=1, n=1, pp_stack=0x7fff3d5d3cb0, func=0x7ff2df1e9ae8) at Python/ceval.c:4334
#40 call_function (oparg=<optimized out>, pp_stack=0x7fff3d5d3cb0) at Python/ceval.c:4262
#41 PyEval_EvalFrameEx (f=f@entry=0xf796b8, throwflag=throwflag@entry=0) at Python/ceval.c:2838
#42 0x00007ff2e5175f45 in PyEval_EvalCodeEx (_co=_co@entry=0x7ff2e400c660, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488, args=args@entry=0x0, argcount=argcount@entry=0,
kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, defcount=defcount@entry=0, kwdefs=kwdefs@entry=0x0, closure=closure@entry=0x0) at Python/ceval.c:3588
#43 0x00007ff2e517601b in PyEval_EvalCode (co=co@entry=0x7ff2e400c660, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488) at Python/ceval.c:775
#44 0x00007ff2e519c09e in run_mod (arena=0xfc2990, flags=0x7fff3d5d3f50, locals=0x7ff2e42df488, globals=0x7ff2e42df488, filename=0x7ff2e41d64b0, mod=0x103b4f0) at Python/pythonrun.c:2180
#45 PyRun_FileExFlags (fp=fp@entry=0xf77b60, filename_str=filename_str@entry=0x7ff2e41d81d0 "/x/eng/bbrtp/users/amresh/sshproxy_3896926_1608240818/test/nate/lib/NATE/Service/SSHProxy.py",
start=start@entry=257, globals=globals@entry=0x7ff2e42df488, locals=locals@entry=0x7ff2e42df488, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:2133
#46 0x00007ff2e519ced5 in PyRun_SimpleFileExFlags (fp=fp@entry=0xf77b60, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:1606
---Type <return> to continue, or q <return> to quit---
#47 0x00007ff2e519df09 in PyRun_AnyFileExFlags (fp=fp@entry=0xf77b60, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7fff3d5d3f50) at Python/pythonrun.c:1292
#48 0x00007ff2e51b6af5 in run_file (p_cf=0x7fff3d5d3f50, filename=0xf0e7f0 L"/x/eng/bbrtp/users/amresh/sshproxy_3896926_1608240818/test/nate/lib/NATE/Service/SSHProxy.py", fp=0xf77b60) at Modules/main.c:319
#49 Py_Main (argc=argc@entry=6, argv=argv@entry=0xee6010) at Modules/main.c:751
#50 0x0000000000400aa6 in main (argc=6, argv=<optimized out>) at ./Modules/python.c:69
Thanks and Regards,
On Fri, Sep 02, 2016 at 10:47:41AM -0700, Steve Dower wrote:
> "I'm not seeing what distinction you think you are making here. What
> distinction do you see between:
> x: int = func(value)
> x = func(value) #type: int"
> Not sure whether I agree with Mark on this particular point, but the
> difference I see here is that the first describes what types x may
> ever contain, while the latter describes what type of being assigned
> to x right here. So one is a variable annotation while the other is an
> expression annotation.
Ultimately Python is a dynamically typed language, and that's not
changing. This means types are fundamentally associated with *values*,
not *variables* (names). But in practice, you can go a long way by
pretending that it is the variable that carries the type. That's the
point of the static type checker: if you see that x holds an int here,
then assume (unless told differently) that x should always be an int.
Because in practice, most exceptions to that are due to bugs, or at
least sloppy code.
Of course, it is up to the type checker to decide how strict it wants to
be, whether to treat violations as a warning or a error, whether to
offer the user a flag to set the behaviour, etc. None of this is
relevant to the PEP. The PEP only specifies the syntax, leaving
enforcement or non-enforcement to the checker, and it says:
PEP 484 introduced type hints, a.k.a. type annotations. While its
main focus was function annotations, it also introduced the notion
of type comments to annotate VARIABLES [emphasis added]
not expressions. And:
This PEP aims at adding syntax to Python for annotating the types
of variables and attributes, instead of expressing them through
which to me obviously implies that the two ways (type comment, and
variable type hint) are intended to be absolutely identical in
semantics, at least as far as the type-checker is concerned.
(They will have different semantics at runtime: the comment is just a
comment, while the type hint will set an __annotations__ mapping.)
But perhaps the PEP needs to make it explicit that they are to be
treated exactly the same.