Mailman 3 September 2016 - Python-Dev

Can CPython on GitHub use the "Merge" button on pull requests (now that they support "squash and merge")?
by Ben Hoyt 13 Sep '16

13 Sep '16

I noticed in [PEP 512 - Document steps to commit a pull request]( https://www.python.org/dev/peps/pep-0512/#document-steps-to-commit-a-pull-r…) it says that CPython on GitHub won't be able to use GitHub's "Merge" button on pull requests, because we want a linear history with one commit per change/issue. However, GitHub recently (actually on April 1, 2016 -- but it's not a joke :-) added support for "commit squashing". See https://github.com/blog/2141-squash-your-commits and https://help.github.com/articles/about-pull-request-merges/ ... basically you can do "old-GitHub-style merge" commits or "squash and merge" commits, and you can even set a repo to only allow "squash and merge" commits on that repo. Will CPython be able to use this? I think that using GitHub's integrated pull request and merge features will make it much easier for contributors (and core developers for that matter). And from personal experience, pressing that big green button is very satisfying. :-) P.S. While I'm here: is there a timeline for the various stages of PEP 512? -Ben

2 2

Fwd: Cssdbpy is a simple SSDB client written on Cython. Faster standart SSDB client.
by Yurij Alexandrovich 13 Sep '16

13 Sep '16

Hello! Post my open source client for take feedback. Cssdbpy is a simple SSDB client written on Cython. Faster standart SSDB client. https://github.com/deslum/cssdbpy Thanks.

2 1

PEP 528: Change Windows console encoding to UTF-8
by Steve Dower 13 Sep '16

13 Sep '16

I'm about to be offline for a few days, so I wanted to get my current draft PEPs out for people can read and review. I don't believe there is a lot of change as a result of either PEP, but the impact of what change there is needs to be weighed against the benefits. If anything, I'm likely to have underplayed the impact of this change (though I've had a *lot* of support for this one). Just stating my biases up-front - take it as you wish. See https://bugs.python.org/issue1602 for the current proposed patch for this PEP. I will likely update it after my upcoming flights, but it's in pretty good shape right now. Cheers, Steve --- https://github.com/python/peps/blob/master/pep-0528.txt --- PEP: 528 Title: Change Windows console encoding to UTF-8 Version: $Revision$ Last-Modified: $Date$ Author: Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Aug-2016 Post-History: 01-Sep-2016 Abstract ======== Historically, Python uses the ANSI APIs for interacting with the Windows operating system, often via C Runtime functions. However, these have been long discouraged in favor of the UTF-16 APIs. Within the operating system, all text is represented as UTF-16, and the ANSI APIs perform encoding and decoding using the active code page. This PEP proposes changing the default standard stream implementation on Windows to use the Unicode APIs. This will allow users to print and input the full range of Unicode characters at the default Windows console. This also requires a subtle change to how the tokenizer parses text from readline hooks, that should have no backwards compatibility issues. Specific Changes ================ Add _io.WindowsConsoleIO ------------------------ Currently an instance of ``_io.FileIO`` is used to wrap the file descriptors representing standard input, output and error. We add a new class (implemented in C) ``_io.WindowsConsoleIO`` that acts as a raw IO object using the Windows console functions, specifically, ``ReadConsoleW`` and ``WriteConsoleW``. This class will be used when the legacy-mode flag is not in effect, when opening a standard stream by file descriptor and the stream is a console buffer rather than a redirected file. Otherwise, ``_io.FileIO`` will be used as it is today. This is a raw (bytes) IO class that requires text to be passed encoded with utf-8, which will be decoded to utf-16-le and passed to the Windows APIs. Similarly, bytes read from the class will be provided by the operating system as utf-16-le and converted into utf-8 when returned to Python. The use of an ASCII compatible encoding is required to maintain compatibility with code that bypasses the ``TextIOWrapper`` and directly writes ASCII bytes to the standard streams (for example, [process_stdinreader.py]_). Code that assumes a particular encoding for the standard streams other than ASCII will likely break. Add _PyOS_WindowsConsoleReadline -------------------------------- To allow Unicode entry at the interactive prompt, a new readline hook is required. The existing ``PyOS_StdioReadline`` function will delegate to the new ``_PyOS_WindowsConsoleReadline`` function when reading from a file descriptor that is a console buffer and the legacy-mode flag is not in effect (the logic should be identical to above). Since the readline interface is required to return an 8-bit encoded string with no embedded nulls, the ``_PyOS_WindowsConsoleReadline`` function transcodes from utf-16-le as read from the operating system into utf-8. The function ``PyRun_InteractiveOneObject`` which currently obtains the encoding from ``sys.stdin`` will select utf-8 unless the legacy-mode flag is in effect. This may require readline hooks to change their encodings to utf-8, or to require legacy-mode for correct behaviour. Add legacy mode --------------- Launching Python with the environment variable ``PYTHONLEGACYWINDOWSSTDIO`` set will enable the legacy-mode flag, which completely restores the previous behaviour. Alternative Approaches ====================== The ``win_unicode_console`` package [win_unicode_console]_ is a pure-Python alternative to changing the default behaviour of the console. Code that may break =================== The following code patterns may break or see different behaviour as a result of this change. All of these code samples require explicitly choosing to use a raw file object in place of a more convenient wrapper that would prevent any visible change. Assuming stdin/stdout encoding ------------------------------ Code that assumes that the encoding required by ``sys.stdin.buffer`` or ``sys.stdout.buffer`` is ``'mbcs'`` or a more specific encoding may currently be working by chance, but could encounter issues under this change. For example:: sys.stdout.buffer.write(text.encode('mbcs')) r = sys.stdin.buffer.read(16).decode('cp437') To correct this code, the encoding specified on the ``TextIOWrapper`` should be used, either implicitly or explicitly:: # Fix 1: Use wrapper correctly sys.stdout.write(text) r = sys.stdin.read(16) # Fix 2: Use encoding explicitly sys.stdout.buffer.write(text.encode(sys.stdout.encoding)) r = sys.stdin.buffer.read(16).decode(sys.stdin.encoding) Incorrectly using the raw object -------------------------------- Code that uses the raw IO object and does not correctly handle partial reads and writes may be affected. This is particularly important for reads, where the number of characters read will never exceed one-fourth of the number of bytes allowed, as there is no feasible way to prevent input from encoding as much longer utf-8 strings:: >>> stdin = open(sys.stdin.fileno(), 'rb') >>> data = stdin.raw.read(15) abcdefghijklm b'abc' # data contains at most 3 characters, and never more than 12 bytes # error, as "defghijklm\r\n" is passed to the interactive prompt To correct this code, the buffered reader/writer should be used, or the caller should continue reading until its buffer is full.:: # Fix 1: Use the buffered reader/writer >>> stdin = open(sys.stdin.fileno(), 'rb') >>> data = stdin.read(15) abcedfghijklm b'abcdefghijklm\r\n' # Fix 2: Loop until enough bytes have been read >>> stdin = open(sys.stdin.fileno(), 'rb') >>> b = b'' >>> while len(b) < 15: ... b += stdin.raw.read(15) abcedfghijklm b'abcdefghijklm\r\n' Copyright ========= This document has been placed in the public domain. References ========== .. [process_stdinreader.py] Twisted's process_stdinreader.py (https://github.com/twisted/twisted/blob/trunk/src/twisted/test/process_stdi…) .. [win_unicode_console] win_unicode_console package (https://pypi.org/project/win_unicode_console/)

6 17

Python 3.6 what's new
by Yury Selivanov 12 Sep '16

12 Sep '16

Hi, Elvis and I authored What's New in Python 3.5. We'd like to volunteer to do the same for 3.6. If there are no objections, we can make the first editing pass in a couple of weeks. Yury

4 3

Python 3.7: remove all private C functions from the Python C API?
by Victor Stinner 12 Sep '16

12 Sep '16

Hi, Currently, Python has 3 C API: * python core API * regular API: subset of the core API * stable API (ABI?), the Py_LIMITED_API thing: subset of the regular API For practical purpose, all functions are declared in Include/*.h. Basically, Python exposes "everything". There are private functions which are exported using PyAPI_FUNC(), whereas they should only be used inside Python "core". Technically, I'm not sure that we can get ride of PyAPI_FUNC() because the stdlib also has extensions which use a few private functions. For Python 3.7, I propose that we move all these private functions in separated header files, maybe Include/private/ or Include/core/, and not export them as part of the "regular API". The risk is that too many C extensions rely on all these tiny "private" functions. Maybe for performance. I don't know. What do you think? See also the issue #26900, "Exclude the private API from the stable API": http://bugs.python.org/issue26900 Victor

7 7

Let's make the SSL module sane
by Christian Heimes 12 Sep '16

12 Sep '16

Hi, (CC TLS gurus) For 3.6 I like to make the SSL more sane and more secure by default. Yes, I'm a bit late but all my proposals are implemented, documented, partly tested and existing tests are passing. I'm going to write more tests and documentation after beta1. First I like to deprecated some old APIs and favor of SSLCotext. We have multiple ways to create a SSL socket or to configure libraries like urllib. The general idea is to make SSLContext the central object for TLS/SSL configuration. My patch deprecates ssl.wrap_socket() and SSLSocket constructor in favor of SSLContext.wrap_socket(). The patch also deprecates certfile, keyfile an similar arguments in network protocol libraries. I also considered to make cert validation enabled by default for all protocol in 3.6, Victor has rising some concerns. How about we change the behavior in 3.7 and just add a warning to 3.6? http://bugs.python.org/issue28022 https://github.com/tiran/cpython/commits/feature/feature/ssl_deprecation -------- Next up SSLContext default configuration. A bare SSLContext comes with insecure default settings. I'd like to make SSLContext(PROTOCOL_SSLv23) secure bu default. Changelog: The context is created with more secure default values. The options OP_NO_COMPRESSION, OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE, OP_NO_SSLv2 (except for PROTOCOL_SSLv2), and OP_NO_SSLv3 (except for PROTOCOL_SSLv3) are set by default. The initial cipher suite list contains only HIGH ciphers, no NULL ciphers and MD5 ciphers (except for PROTOCOL_SSLv2). http://bugs.python.org/issue28043 https://github.com/tiran/cpython/commits/feature/ssl_sane_defaults -------- Finally (and this is the biggest) I like to change how the protocols work. OpenSSL 1.1.0 has deprecated all version specific protocols. Soon OpenSSL will only support auto-negotiation (formerly known as PROTOCOL_SSLv23). My patch #26470 added PROTOCOL_TLS as alias for PROTOCOL_SSLv23. If the last idea is accepted I will remove PROTOCOL_TLS again. It hasn't been released yet. Instead I'm going to add PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER (see https://www.openssl.org/docs/manmaster/ssl/SSL_CTX_new.html TLS_server_method(), TLS_client_method()). PROTOCOL_TLS_CLIENT is like PROTOCOL_SSLv23 but only supports client-side sockets and PROTOCOL_TLS_SERVER just server-side sockets. In my experience we can't have a SSLContext with sensible and secure settings for client and server at the same time. Hostname checking and cert validation is only sensible for client-side sockets. Starting in 3.8 (or 3.7?) there will be only PROTOCOL_TLS_CLIENT and PROTOCOL_TLS_SERVER. I haven't created a ticket yet, code is at https://github.com/tiran/cpython/commits/feature/openssl_client_server -------- How will my proposals change TLS/SSL code? Application must create a SSLContext object. Applications are recommended to keep the context around to benefit from session reusage and reduce overload of cert parsing. Client side, ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT): * works with TLSv1.0, TLSv1.1, TLSv1.2 and new protocols * Options OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION are set * Only HIGH cipher suites are enabled, MD5 and NULL are disabled * all other ciphers are still enabled, MD5 for SSLv2 * cert_required = CERT_REQUIRED * check_hostname = True * ctx.wrap_socket() creates a client-side socket * ctx.wrap_socket(server_side=True) will not work * root certs are *not* loaded I don't load any certs because it is not possible to remove a cert or X509 lookup once it is loaded. create_default_context() just have to load the certs and set more secure ciper suites. Server side, ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_PROTOCOL): * works with TLSv1.0, TLSv1.1, TLSv1.2 and new protocols * Options OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_COMPRESSION are set * OP_CIPHER_SERVER_PREFERENCE, OP_SINGLE_DH_USE, OP_SINGLE_ECDH_USE are set * Only HIGH cipher suites are enabled, MD5 and NULL are disabled * all other ciphers are still enabled, MD5 for SSLv2 * cert_required = CERT_NONE (no client cert validation) * check_hostname = False * no root CA certs are loaded * only ctx.wrap_socket(server_side=True) works I hope this mail makes sense. Christian

4 7

Re: [Python-Dev] cpython: Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal.
by Christian Heimes 12 Sep '16

12 Sep '16

On 2016-09-09 18:53, guido.van.rossum wrote: > https://hg.python.org/cpython/rev/804b71d43c85 > changeset: 103415:804b71d43c85 > user: Guido van Rossum <guido(a)dropbox.com> > date: Fri Sep 09 09:36:26 2016 -0700 > summary: > Issue #27999: Make "global after use" a SyntaxError, and ditto for nonlocal. > Patch by Ivan Levkivskyi. > > files: > Doc/reference/simple_stmts.rst | 5 +- > Lib/test/test_syntax.py | 18 +++- > Misc/NEWS | 3 + > Python/symtable.c | 104 +++++++------------- > 4 files changed, 59 insertions(+), 71 deletions(-) > [...] > @@ -1337,31 +1313,23 @@ > long cur = symtable_lookup(st, name); > if (cur < 0) > VISIT_QUIT(st, 0); > - if (cur & DEF_ANNOT) { > - PyErr_Format(PyExc_SyntaxError, > - "annotated name '%U' can't be nonlocal", > - name); > + if (cur & (DEF_LOCAL | USE | DEF_ANNOT)) { > + char* msg; > + if (cur & DEF_ANNOT) { > + msg = NONLOCAL_ANNOT; > + } > + if (cur & DEF_LOCAL) { > + msg = NONLOCAL_AFTER_ASSIGN; > + } > + else { > + msg = NONLOCAL_AFTER_USE; > + } > + PyErr_Format(PyExc_SyntaxError, msg, name); Hi Guido, did you mean if / else if / else here? It's not completely clear if the code means to set msg a second time if both cur & DEF_ANNOT and cur & DEF_LOCAL are true. Christian

2 4

Re: [Python-Dev] [Python-checkins] cpython: Fixes test_getargs2 to get the buildbots working again.
by Steve Dower 11 Sep '16

11 Sep '16

On 11Sep2016 1944, steve.dower wrote: > https://hg.python.org/cpython/rev/7793d34609cb > changeset: 103679:7793d34609cb > user: Steve Dower <steve.dower(a)microsoft.com> > date: Sun Sep 11 19:43:51 2016 -0700 > summary: > Fixes test_getargs2 to get the buildbots working again. > > files: > Lib/test/test_getargs2.py | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/test_getargs2.py b/Lib/test/test_getargs2.py > --- a/Lib/test/test_getargs2.py > +++ b/Lib/test/test_getargs2.py > @@ -471,7 +471,7 @@ > > ret = get_args(*TupleSubclass([1, 2])) > self.assertEqual(ret, (1, 2)) > - self.assertIs(type(ret), tuple) > + self.assertIsInstance(ret, tuple) > > ret = get_args() > self.assertIn(ret, ((), None)) I'm not sure this is the fix we want to keep here, but it was sufficient to get the test going and unblock all the buildbots. I'm not entirely sure when the break appeared (essentially we seem to not be copying *args into a new tuple), but I'd guess it's to do with the fast calling improvements. Cheers, Steve

2 2

PEP520 and absence of __definition_order__
by Ethan Furman 11 Sep '16

11 Sep '16

Per Victor's advice I'm posting this here. PEP 520 has been accepted, but without the __definition_order__ attribute. The accompanying comment: > "Note: Since compact dict has landed in 3.6, __definition_order__ has > been removed. cls.__dict__ now mostly accomplishes the same thing > instead." The "mostly" is what concerns me. Much like having a custom __dir__ lets a class fine-tune what is of interest, a custom __definition_order__ allows a class to present a unified view of the class creation process. This could be important to classes that employ __getattr__ (or __getattribute__) to provide virtual attributes, such as Enum or proxy classes. With __definition_order__ Enum can display the actual creation order of enum members and methods, while relying on Enum.__dict__.keys() presents a jumbled mess with many attributes the user never wrote, the enum members either appearing /after/ all the methods (even if actually written before), or entirely absent. For example, this class: >>> class PassBy(Enum): ... value = 1 ... reference = 2 ... name = 3 ... object = name ... def used_by_python(self): ... return self.name == 'name' ... shows this: >>> PassBy.__dict__.keys() dict_keys([ '_generate_next_value_', '__module__', 'used_by_python', '__doc__', '_member_names_', '_member_map_', '_member_type_', '_value2member_map_', 'reference', 'object', '__new__', ]) Notice that two of the members are missing, and all are after the method. If __definition_order__ existed it would be this: >>> PassBy.__definition_order__ ['value', 'reference', 'name', 'object', 'used_by_python'] Which is a much more accurate picture of the user's class. -- ~Ethan~

7 13

Re: [Python-Dev] [Webmaster] A broken link!
by Steve Holden 11 Sep '16

11 Sep '16

Hi Karen, Thanks for your note. I just checked the source of the document in question, and it appears that link has been changed to reference https://www.mercurial-scm.org/guide, so it appears that we may be publishing an out-of-date document there. I'm copying this reply to the python-dev list, and the release manager may or may not choose to update the published version. regards Steve Steve Holden On Fri, Sep 9, 2016 at 3:37 PM, Karen Little < karen.little(a)whoishostingthismail.com> wrote: > Hi, > > Just wanted to let you know about a link that seems to be broken on this > page https://docs.python.org/3.2/whatsnew/3.2.html. > > It is this link http://mercurial.selenic.com/guide/, but the page doesn’t > seem to be active any more. I thought you might want to update. > > If you are looking for an alternative please check out > http://wiht.link/Mercurial-intro, it may make a suitable replacement. > > Kind Regards, > Karen > > > > Don't want emails from us anymore? Reply to this email with the word > "UNSUBSCRIBE" in the subject line. > WhoIsHostingThis, BM Box 3667, Old Gloucester Street London, WC1N 3XX, > United Kingdom > _______________________________________________ > Webmaster mailing list > Webmaster(a)python.org > https://mail.python.org/mailman/listinfo/webmaster > >

2 2