Mailman 3 December 2017 - Python-Dev

[ssl] The weird case of IDNA
by Christian Heimes Jan. 2, 2018

Jan. 2, 2018

Hi, tl;dr This mail is about internationalized domain names and TLS/SSL. It doesn't concern you if you live in ASCII-land. Me and a couple of other developers like to change the ssl module in a backwards-incompatible way to fix IDN support for TLS/SSL. Simply speaking the IDNA standards (internationalized domain names for applications) describe how to encode non-ASCII domain names. The DNS system and X.509 certificates cannot handle non-ASCII host names. Any non-ASCII part of a hostname is punyencoded. For example the host name 'www.bücher.de' (books) is translated into 'www.xn--bcher-kva.de'. In IDNA terms, 'www.bücher.de' is called an IDN U-label (unicode) and 'www.xn--bcher-kva.de' an IDN A-label (ASCII). Please refer to the TR64 document [1] for more information. In a perfect world, it would be very simple. We'd only had one IDNA standard. However there are multiple standards that are incompatible with each other. The German TLD .de demands IDNA-2008 with UTS#46 compatibility mapping. The hostname 'www.straße.de' maps to 'www.xn--strae-oqa.de'. However in the older IDNA 2003 standard, 'www.straße.de' maps to 'www.strasse.de', but 'strasse.de' is a totally different domain! CPython has only support for IDNA 2003. It's less of an issue for the socket module. It only converts text to IDNA bytes on the way in. All functions support bytes and text. Since IDNA encoding does change ASCII and IDNA-encoded data is ASCII, it is also no problem to pass IDNA2008-encoded text or bytes to all socket functions. Example: >>> import socket >>> import idna # from PyPI >>> names = ['straße.de', b'strasse.de', idna.encode('straße.de'), idna.encode('straße.de').encode('ascii')] >>> for name in names: ... print(name, socket.getaddrinfo(name, None, socket.AF_INET, socket.SOCK_STREAM, 0, socket.AI_CANONNAME)[0][3:5]) ... straße.de ('strasse.de', ('89.31.143.1', 0)) b'strasse.de' ('strasse.de', ('89.31.143.1', 0)) b'xn--strae-oqa.de' ('xn--strae-oqa.de', ('81.169.145.78', 0)) xn--strae-oqa.de ('xn--strae-oqa.de', ('81.169.145.78', 0)) As you can see, 'straße.de' is canonicalized as 'strasse.de'. The IDNA 2008 encoded hostname maps to a different IP address. On the other hand ssl module is currently completely broken. It converts hostnames from bytes to text with 'idna' codec in some places, but not in all. The SSLSocket.server_hostname attribute and callback function SSLContext.set_servername_callback() are decoded as U-label. Certificate's common name and subject alternative name fields are not decoded and therefore A-labels. The *must* stay A-labels because hostname verification is only defined in terms of A-labels. We even had a security issue once, because partial wildcard like 'xn*.example.org' must not match IDN hosts like 'xn--bcher-kva.example.org'. In issue [2] and PR [3], we all agreed that the only sensible fix is to make 'SSLContext.server_hostname' an ASCII text A-label. But this is an backwards incompatible fix. On the other hand, IDNA is totally broken without the fix. Also in my opinion, PR [3] is not going far enough. Since we have to break backwards compatibility anyway, I'd like to modify SSLContext.set_servername_callback() at the same time. Questions: - Is everybody OK with breaking backwards compatibility? The risk is small. ASCII-only domains are not affected and IDNA users are broken anyway. - Should I only fix 3.7 or should we consider a backport to 3.6, too? Regards, Christian [1] https://www.unicode.org/reports/tr46/ [2] https://bugs.python.org/issue28414 [3] https://github.com/python/cpython/pull/3010

10 18

Re: [Python-Dev] Concerns about method overriding and subclassing with dataclasses
by Nick Coghlan Dec. 30, 2017

Dec. 30, 2017

On 30 Dec. 2017 11:01 am, "Ethan Smith" <ethan(a)ethanhs.me> wrote: On Fri, Dec 29, 2017 at 4:52 PM, Guido van Rossum <guido(a)python.org> wrote: > I still think it should overrides anything that's just inherited but > nothing that's defined in the class being decorated. > > Could you explain why you are of this opinion? Is it a concern about complexity of implementation? Adding a new method to a base class shouldn't risk breaking existing subclasses. If folks want to retain the base class implementation, they can request that explicitly (and doing so isn't redundant at the point of subclass definition the way it is for methods defined in the class body). Cheers, Nick.

2 1

Supporting functools.singledispatch with classes.
by Ethan Smith Dec. 29, 2017

Dec. 29, 2017

Hello all, In https://bugs.python.org/issue32380 I was hoping to add support for singledispatch with methods. Unfortunately, this cannot be achieved internally without ugly attribute or stack hacks. Therefore, I was thinking it would be nice if singledispatch supported a keyword argument of the argument index to dispatch on, thus one can say: class A: @singledispatch(arg=1) def method(self, a): return 'base' @method.register(int) def method(self, a): return 'int' The other option that could work is to define a special decorator for methods def methodsingledispatch(func): """Single-dispatch generic method decorator.""" wrapped = singledispatch(func) def wrapper(*args, **kw): return wrapped.dispatch(args[1].__class__)(*args, **kw) wrapper.register = wrapped.register update_wrapper(wrapper, func) return wrapper Since this is an API change, Ivan said I should post here to get feedback. I prefer the first design, as it is more generic and flexible. There is also the issue of classmethod and staticmethod. Since these are descriptors, I'm not sure they will work with singledispatch at all. if you do @classmethod @singledispatch def foo(cls, arg): ... You lose register on foo, breaking everything. I believe this would require changing classmethod thus is a non-starter. If you do @singledispatch @classmethod def foo(arg): ... The wrapper in singledispatch needs to be called as the __func__ in classmethod, but __func__ is readonly. So at the moment, I don't think it is possible to implement singledispatch on classmethod or staticmethod decorated functions. I look forward to people's thoughts on these issues. Cheers, Ethan Smith

3 5

Summary of Python tracker Issues
by Python tracker Dec. 29, 2017

Dec. 29, 2017

ACTIVITY SUMMARY (2017-12-22 - 2017-12-29) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6355 (+13) closed 37843 (+24) total 44198 (+37) Open issues with patches: 2472 Issues opened (31) ================== #30722: Tools/demo/redemo.py broken https://bugs.python.org/issue30722 reopened by serhiy.storchaka #32411: Idlelib.browser: stop sorting dicts created by pyclbr https://bugs.python.org/issue32411 opened by terry.reedy #32412: help() of bitwise operators should mention sets as well https://bugs.python.org/issue32412 opened by steven.daprano #32413: Document that locals() may return globals() https://bugs.python.org/issue32413 opened by steven.daprano #32414: PyCapsule_Import fails when name is in the form 'package.modul https://bugs.python.org/issue32414 opened by lekma #32417: fromutc does not respect datetime subclasses https://bugs.python.org/issue32417 opened by p-ganssle #32418: Implement Server.get_loop() method https://bugs.python.org/issue32418 opened by asvetlov #32419: Add unittest support for pyc projects https://bugs.python.org/issue32419 opened by brgirgis #32420: LookupError : unknown encoding : [0x7FF092395AD0] ANOMALY https://bugs.python.org/issue32420 opened by Kitamura #32421: Keeping an exception in cache can segfault the interpreter https://bugs.python.org/issue32421 opened by zunger #32423: The Windows SDK version 10.0.15063.0 was not found https://bugs.python.org/issue32423 opened by isuruf #32424: Synchronize copy methods between Python and C implementations https://bugs.python.org/issue32424 opened by gphemsley #32425: Allow non-default XML parsers to take advantage of a _parse_wh https://bugs.python.org/issue32425 opened by gphemsley #32426: Tkinter.ttk Widget does not define wich option exists to set t https://bugs.python.org/issue32426 opened by alex.75 #32427: Rename and expose dataclasses._MISSING https://bugs.python.org/issue32427 opened by eric.smith #32428: dataclasses: make it an error to have initialized non-fields i https://bugs.python.org/issue32428 opened by eric.smith #32429: Outdated Modules/Setup warning is invisible https://bugs.python.org/issue32429 opened by mdk #32430: Simplify Modules/Setup{,.dist,.local} https://bugs.python.org/issue32430 opened by mdk #32431: Two bytes objects of zero length don't compare equal https://bugs.python.org/issue32431 opened by jonathanunderwood #32433: Provide optimized HMAC digest https://bugs.python.org/issue32433 opened by christian.heimes #32434: pathlib.WindowsPath.reslove(strict=False) returns absoulte pat https://bugs.python.org/issue32434 opened by mliska #32435: tarfile recognizes .gz file as tar https://bugs.python.org/issue32435 opened by spetrunin #32436: Implement PEP 567 https://bugs.python.org/issue32436 opened by yselivanov #32438: PyLong_ API cleanup https://bugs.python.org/issue32438 opened by erik.bray #32439: Clean up the code for compiling comparison expressions https://bugs.python.org/issue32439 opened by serhiy.storchaka #32441: os.dup2 should return the new fd https://bugs.python.org/issue32441 opened by benjamin.peterson #32443: Add Linux's signalfd() to the signal module https://bugs.python.org/issue32443 opened by gregory.p.smith #32444: python -m venv symlink dependency on how python binary is call https://bugs.python.org/issue32444 opened by seliger #32445: Skip creating redundant wrapper functions in ExitStack.callbac https://bugs.python.org/issue32445 opened by ncoghlan #32446: ResourceLoader.get_data() should accept a PathLike https://bugs.python.org/issue32446 opened by barry #32447: IDLE shell won't open on Mac OS 10.13.1 https://bugs.python.org/issue32447 opened by sm1979 Most recent 15 issues with no replies (15) ========================================== #32447: IDLE shell won't open on Mac OS 10.13.1 https://bugs.python.org/issue32447 #32446: ResourceLoader.get_data() should accept a PathLike https://bugs.python.org/issue32446 #32445: Skip creating redundant wrapper functions in ExitStack.callbac https://bugs.python.org/issue32445 #32441: os.dup2 should return the new fd https://bugs.python.org/issue32441 #32439: Clean up the code for compiling comparison expressions https://bugs.python.org/issue32439 #32436: Implement PEP 567 https://bugs.python.org/issue32436 #32433: Provide optimized HMAC digest https://bugs.python.org/issue32433 #32427: Rename and expose dataclasses._MISSING https://bugs.python.org/issue32427 #32426: Tkinter.ttk Widget does not define wich option exists to set t https://bugs.python.org/issue32426 #32423: The Windows SDK version 10.0.15063.0 was not found https://bugs.python.org/issue32423 #32418: Implement Server.get_loop() method https://bugs.python.org/issue32418 #32410: Implement loop.sock_sendfile method https://bugs.python.org/issue32410 #32404: fromtimestamp does not call __new__ in datetime subclasses https://bugs.python.org/issue32404 #32403: date, time and datetime alternate constructors should take fas https://bugs.python.org/issue32403 #32400: inspect.isdatadescriptor false negative https://bugs.python.org/issue32400 Most recent 15 issues waiting for review (15) ============================================= #32441: os.dup2 should return the new fd https://bugs.python.org/issue32441 #32439: Clean up the code for compiling comparison expressions https://bugs.python.org/issue32439 #32436: Implement PEP 567 https://bugs.python.org/issue32436 #32433: Provide optimized HMAC digest https://bugs.python.org/issue32433 #32431: Two bytes objects of zero length don't compare equal https://bugs.python.org/issue32431 #32429: Outdated Modules/Setup warning is invisible https://bugs.python.org/issue32429 #32427: Rename and expose dataclasses._MISSING https://bugs.python.org/issue32427 #32424: Synchronize copy methods between Python and C implementations https://bugs.python.org/issue32424 #32418: Implement Server.get_loop() method https://bugs.python.org/issue32418 #32414: PyCapsule_Import fails when name is in the form 'package.modul https://bugs.python.org/issue32414 #32413: Document that locals() may return globals() https://bugs.python.org/issue32413 #32411: Idlelib.browser: stop sorting dicts created by pyclbr https://bugs.python.org/issue32411 #32410: Implement loop.sock_sendfile method https://bugs.python.org/issue32410 #32404: fromtimestamp does not call __new__ in datetime subclasses https://bugs.python.org/issue32404 #32403: date, time and datetime alternate constructors should take fas https://bugs.python.org/issue32403 Top 10 most discussed issues (10) ================================= #32424: Synchronize copy methods between Python and C implementations https://bugs.python.org/issue32424 16 msgs #17611: Move unwinding of stack for "pseudo exceptions" from interpret https://bugs.python.org/issue17611 11 msgs #32429: Outdated Modules/Setup warning is invisible https://bugs.python.org/issue32429 9 msgs #31639: http.server and SimpleHTTPServer hang after a few requests https://bugs.python.org/issue31639 7 msgs #32431: Two bytes objects of zero length don't compare equal https://bugs.python.org/issue32431 6 msgs #32360: Save OrderedDict imports in various stdlibs. https://bugs.python.org/issue32360 5 msgs #32420: LookupError : unknown encoding : [0x7FF092395AD0] ANOMALY https://bugs.python.org/issue32420 5 msgs #21288: hashlib.pbkdf2_hmac Hash Constructor https://bugs.python.org/issue21288 4 msgs #32145: Wrong ExitStack Callback recipe https://bugs.python.org/issue32145 4 msgs #32419: Add unittest support for pyc projects https://bugs.python.org/issue32419 4 msgs Issues closed (24) ================== #24960: Can't use lib2to3 with embeddable zip file. https://bugs.python.org/issue24960 closed by benjamin.peterson #26133: asyncio: ugly error related to signal handlers at exit if the https://bugs.python.org/issue26133 closed by asvetlov #26666: File object hook to modify select(ors) event mask https://bugs.python.org/issue26666 closed by asvetlov #28236: In xml.etree.ElementTree Element can be created with empty and https://bugs.python.org/issue28236 closed by rhettinger #29084: C API of OrderedDict https://bugs.python.org/issue29084 closed by serhiy.storchaka #29504: blake2: compile error with -march=bdver2 https://bugs.python.org/issue29504 closed by christian.heimes #29780: Interpreter hang on self._epoll.poll(timeout, max_ev) https://bugs.python.org/issue29780 closed by asvetlov #31721: assertion failure in FutureObj_finalize() after setting _log_t https://bugs.python.org/issue31721 closed by asvetlov #31983: Officially add Py_SETREF and Py_XSETREF https://bugs.python.org/issue31983 closed by serhiy.storchaka #31988: Saving bytearray to binary plist file doesn't work https://bugs.python.org/issue31988 closed by serhiy.storchaka #32261: Online doc does not include inspect.classify_class_attrs https://bugs.python.org/issue32261 closed by csabella #32324: [Security] "python3 directory" inserts "directory" at sys.path https://bugs.python.org/issue32324 closed by ncoghlan #32335: Failed Python build on Fedora 27 https://bugs.python.org/issue32335 closed by amitg-b14 #32363: Deprecate task.set_result() and task.set_exception() https://bugs.python.org/issue32363 closed by yselivanov #32372: Optimize out __debug__ at the AST level https://bugs.python.org/issue32372 closed by serhiy.storchaka #32401: No module named '_ctypes' https://bugs.python.org/issue32401 closed by YoSTEALTH #32402: Coverity: CID 1426868/1426867: Null pointer dereferences in t https://bugs.python.org/issue32402 closed by inada.naoki #32415: Add Task.get_loop() and Future.get_loop() https://bugs.python.org/issue32415 closed by yselivanov #32416: Refactor and add new tests for the f_lineno setter https://bugs.python.org/issue32416 closed by serhiy.storchaka #32422: Reduce lru_cache memory overhead. https://bugs.python.org/issue32422 closed by serhiy.storchaka #32432: [BUG] Python vs Macbook High Sierra 10.13.2 https://bugs.python.org/issue32432 closed by Felipe Filgueira Barral #32437: UnicodeError: 'IDNA does not round-trip' https://bugs.python.org/issue32437 closed by berker.peksag #32440: Use HTTPS in help() https://bugs.python.org/issue32440 closed by Mariatta #32442: Result of pathlib.Path.resolve() with UNC path is not very use https://bugs.python.org/issue32442 closed by uranusjr

1 0

Re: [Python-Dev] Heap allocate type structs in native extension modules?
by Hugh Fisher Dec. 29, 2017

Dec. 29, 2017

> Date: Fri, 29 Dec 2017 11:25:14 +0200 > From: Serhiy Storchaka <storchaka(a)gmail.com> > To: python-dev(a)python.org > Subject: Re: [Python-Dev] Heap allocate type structs in native > extension modules? [ munch ] > Yes, you can create heap types by using PyType_FromSpecWithBases(). > > But be aware of caveats (https://bugs.python.org/issue26979). Thanks! I'll give it a go. I've already run into and solved the tp_new issue, so that won't be a problem. -- cheers, Hugh Fisher

1 0

(no subject)
by Yogev Hendel Dec. 29, 2017

Dec. 29, 2017

I don't know if this is the right place to put this, but I've found the following lines of code results in an incredibly long processing time. Perhaps it might be of use to someone. *import re* *pat = re.compile('^/?(?:\\w+)/(?:[%\\w-]+/?)+/?$')* *pat.match('/t/a-aa-aa-aaaaa-aa-aa-aa-aa-aa-aa./')*

4 3

Heap allocate type structs in native extension modules?
by Hugh Fisher Dec. 29, 2017

Dec. 29, 2017

I have a Python program which generates the boilerplate code for native extension modules from a Python source definition. (http://bitbucket.org/hugh_fisher/fullofeels if interested.) The examples in the Python doco and the "Python Essential Reference" book all use a statically declared PyTypeObject struct and PyType_Ready in the module init func, so I'm doing the same. Then Python 3.5 added a check for statically allocated types inheriting from heap types, which broke a couple of my classes. And now I'm trying to add a __dict__ to native classes so end users can add their own attributes, and this is turning out to be painful with static PyTypeObject structs Would it be better to use dynamically allocated type structs in native modules? -- cheers, Hugh Fisher

4 4

Re: [Python-Dev] 'continue'/'break'/'return' inside 'finally' clause
by Guido van Rossum Dec. 28, 2017

Dec. 28, 2017

Looks to me the prohibition was to prevent a crash. It makes more sense to fix it. On Dec 28, 2017 03:39, "Serhiy Storchaka" <storchaka(a)gmail.com> wrote: Currently 'continue' is prohibited inside 'finally' clause, but 'break' and 'return' are allowed. What is the r 'continue' was prohibited in https://bugs.python.org/issue1542451. Should we prohibit also 'break' and 'return' or allow 'continue'? 'break' and 'return' are never used inside 'finally' clause in the stdlib. _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% 40python.org

2 2

Documenting types outside of typing
by Barry Warsaw Dec. 28, 2017

Dec. 28, 2017

In his review of PR#4911, Antoine points to the documentation of two type definitions in importlib.resources, Package and Resource. https://github.com/python/cpython/pull/4911/files#diff-2a479c407f7177f3d7cb… One question is what markup to use for type definitions. I’m using class:: because that’s what’s used in typing and there doesn’t seem to be any better alternative. More to the point, Antoine questions whether these two types should be documented at all: https://github.com/python/cpython/pull/4911#discussion_r158801065 "What I mean is that a class is supposed to specify concrete behaviour, but being a type, Package doesn't have any methods or attributes of its own. So I don't see the point of mentioning it in the docs.” I suggest that they are worth documenting because they help to organize the discussion about what API is expected from the arguments to the functions, without having to duplicate that information in every function description. I also think that since you’ll see those types in the code, they are worth documenting. I don’t think you *lose* anything by including their documentation. But Antoine makes a good point that we probably don’t have a lot of precedence here, so suggests we discuss it on python-dev to come up with some useful conventions. I haven’t kept up on the dataclasses discussion, but given that types are important in that API too, have the same issues come up there and if so, how are they being handled? Cheers, -Barry

2 2

'continue'/'break'/'return' inside 'finally' clause
by Serhiy Storchaka Dec. 28, 2017

Dec. 28, 2017

Currently 'continue' is prohibited inside 'finally' clause, but 'break' and 'return' are allowed. What is the r 'continue' was prohibited in https://bugs.python.org/issue1542451. Should we prohibit also 'break' and 'return' or allow 'continue'? 'break' and 'return' are never used inside 'finally' clause in the stdlib.

1 0