Mailman 3 April 2016 - Python-Dev

Should secrets include a fallback for hmac.compare_digest?
by Steven D'Aprano April 15, 2016

April 15, 2016

Now that PEP 506 has been approved, I've checked in the secrets module, but an implementation question has come up regarding compare_digest. Currently, the module tries to import hmac.compare_digest, and if that fails, then it falls back to a Python version. But since compare_digest has been available since 3.3, I'm now questioning whether the fallback is useful at all. Perhaps for alternate Python implementations? So, two questions: - should secrets include a fallback? - if so, what is the preferred way of doing this? # option 1: fallback if compare_digest is missing try: from hmac import compare_digest except ImportError: def compare_digest(a, b): ... # option 2: "C accelerator idiom" def compare_digest(a, b): ... try: from hmac import compare_digest except ImportError: pass Option 1 is closer to how I would write hybrid 2/3 code, but option 2 is how PEP 399 suggests it should be written. https://www.python.org/dev/peps/pep-0399/ Currently, hmac imports compare_digest from _operator. There's no Python version in operator either. Should there be? -- Steve

4 5

PEP 506 secrets module
by Steven D'Aprano April 15, 2016

April 15, 2016

Hi, As extensively discussed on Python-Ideas, the secrets module and PEP 506 is (I hope) ready for pronouncement. https://www.python.org/dev/peps/pep-0506/ There is code and tests here: https://bitbucket.org/sdaprano/secrets or you can run hg clone https://sdaprano@bitbucket.org/sdaprano/secrets The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4. -- Steve

13 28

Most 3.x buildbots are green again, please don't break them and watch them!
by Victor Stinner April 15, 2016

April 15, 2016

Hi, Last months, most 3.x buildbots failed randomly. Some of them were always failing. I spent some time to fix almost all Windows and Linux buildbots. There were a lot of different issues. So please try to not break buildbots again and remind to watch them sometimes: http://buildbot.python.org/all/waterfall?category=3.x.stable&category=3.x.u… Next weeks, I will try to backport some fixes to Python 3.5 (if needed) to make these buildbots more stable too. Python 2.7 buildbots are also in a sad state (ex: test_marshal segfaults on Windows, see issue #25264). But it's not easy to get a Windows with the right compiler to develop on Python 2.7 on Windows. -- Maybe it's time to move more 3.x buildbots to the "stable" category? http://buildbot.python.org/all/waterfall?category=3.x.stable By the way, I don't understand why "AMD64 OpenIndiana 3.x" is considered as stable since it's failing with multiple issues since many months and nobody is working on these failures. I suggest to move this buildbot back to the unstable category. -- We have many offline buildbots. What's the status of these buildbots? Should we expect that they come back soon? Or would it be possible to hide them? It would help to check the status of all buildbots. -- Failing buildbots: - AMD64 FreeBSD CURRENT 3.x: http://bugs.python.org/issue26566 -- I installed a fresh FreeBSD CURRENT in a VM and I'm unable to reproduce failures. Maybe the buildbot slave is oudated and FreeBSD must be upgraded? - AMD64 OpenIndiana 3.x, x86 OpenIndiana 3.x: test_socket failures on sendfile. Sorry but I'm not really interested by this OS. - PPC64 AIX 3.x: failing tests: test_httplib, test_httpservers, test_socket, test_distutils, test_asyncio, (...); random timeout failure in test_eintr, etc. I don't have access to AIX and I'm not interested to acquire an AIX license, nor to install it. I'm not sure that it's useful to have an AIX buildbot and no core developer have access to AIX, and nobody is working on AIX failures. Maybe HP wants to help us to support AIX? (Provide manpower, access to AIX servers, or something like that.) - x86 OpenBSD 3.x: 5 tests failed, test_crypt test_socket test_ssl test_strptime test_time. This OS needs some love ;-) - the 4 ICC buildbots are failing with stack overflow, segfault, etc. Again, I'm not sure that these buildbots are useful since it looks like we don't support this compiler yet. Or does it help to work on supporting this compiler? Who is working on ICC support? -- FYI I also made some enhancements on regrtest (our test runner for the test suite), mostly to debug failures: - display the duration of tests taking longer than 30 seconds - new timestamp prefix, used to debug buildbot hangs - when parallel tests are interrupted, display progress on waiting for completion - add timeout to main process when using -jN: it should help to debug buildbot hang - "Run tests in parallel using 3 child processes" or "Run tests sequentially" message which helps to understand how tests are running. There is the -j1 trap which has no effect: tests are still run sequentially. By the way, I proposed to really use subprocesses when -j1 is used: http://bugs.python.org/issue25285 The default timeout changed from 1 hour to 15 min, it's the maximum duration to run a single test file (ex: test_os.py). On my Linux box, running the whole test suite in parallel (10 child processes for my 4 CPU cores with hyperthreading) with Python compiled in debug mode (slow) takes 4 min 37 sec. Tell me if the default timeout is too low. It can be configured per buildbot if needed (TESTTIMEOUT env var). -- By the way, I'm always surprised by the huge difference of time needed to run a build on the different slaves: from a few minutes to more than 3 hours. The fatest Windows slave takes 28 minutes (run tests in parallel using 4 child processes), whereas the 3 others (run tests sequentially and) take between 2 hours and more than 3 hours! Why running tests on Windows takes so long? Maybe we should make sure that no buildbot run tests sequentially, because it creates a lot of annoying side effects (even if sometimes it helps to find tricky bugs, sometimes bugs restricted to the tests themself) and because a lot of time simply wait a few seconds. So running mutliple tests in parallel don't burn your CPU, it's just faster. IMHO the risk of random timeout failures is low compared to the speedup. -- The most interesting bug was a deadlock in locale.setlocale() on Windows 7: the bug made the buildbot to hang "sometimes" (randomly). Jeremy Kloth identified the bug, but Steve Dower noticed us that it's already fixed in Visual Studio 2015 Update 1: so please update VS if it's not the case yet. Steve added a post-build test to check if the ucrtbase/ucrtbased DLL has the known bug. => http://bugs.python.org/issue26624 Victor

10 11

Re: [Python-Dev] MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

On Thu, Apr 14, 2016 at 8:27 PM, Guido van Rossum <guido(a)python.org> wrote: > Great analysis! What might stand in the way of adoption is concern for > bytecode manipulation libraries that would have to be changed. What > might encourage adoption would be a benchmark showing this saves a lot > of time. > > Personally I'm expecting it won't make much of a difference for real > programs since almost always the cost of creating the function is > dwarfed by the (total) cost of running it. But Python does create a > lot of functions, and there's also lambdas. This change alone is very unlikely to have a measurable performance impact. The intention is to clean up ceval.c/compile.c a bit, nothing more. If many other opcodes were somehow slimmed down in the similar fashion, then we might (or might not) see perf gains. For example, most slot dispatch opcodes can be compressed into a single opcode+slot index with inlined dispatch logic, instead of each one individually calling C API functions... > There's also talk of switching to wordcode, in a different thread. > Maybe the idea would be easier to introduce there? (Bytecode libraries > would have to change anyways, so the additional concern for this > change would be minimal.) Wordcode can benefit from this change, because it guarantees single-byte MAKE_FUNCTION oparg. I think that Python should make bytecode explicitly unstable and subject to change with any major release. The potential for a faster Python interpreter (or simple JIT) is huge; requiring bytecode compatibility will slow down any progress in this area.

4 3

Re: [Python-Dev] MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

On Thu, Apr 14, 2016 at 8:32 PM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > > Would you like to work on a patch to implement that change? I'll work on a patch. Should I post it to bugs.python.org? > Since Python 3.6 may get a new bytecode format (wordcode, see the > other thread on this mlailing list), I think that it's ok to change > MAKE_FUNCTION in the same release. Wordcode looks like pure win from (projected) 25% bytecode size reduction alone.

2 1

Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!
by Victor Stinner April 14, 2016

April 14, 2016

Le 14 avr. 2016 11:16 AM, "Serhiy Storchaka" <storchaka(a)gmail.com> a écrit : > A desirable but nonexistent feature is to write emails to authors of commits that broke buildbots. How hard to implement this? Yeah I also had this idea since many years but buildbots were quite unstable. Maybe we should be more strict to consider a buildbot as stable? I propose to experiment sending notifications of failure to the authors of changes *and* to a new mailing list. I would subscribe to such list. An even safer starting point would be to only start with the mailing list. FYI I'm connected to the #python-dev IRC channel which already contain these notifications. But I agree that mails are better. > What are you think about backporting recent regrtest to 2.7? Most needed features to me are the -m and -G options. Regrtest changed a lot in python 3.6 (new test.libregrtest library). I suggest to start from python 3.5. For -m: if it doesn't need to modify the unittest module, I agree. I don't know -G option. > Would be nice to add a feature for running every test in separate subprocess. This will isolate the effect of failed tests. See my email :-) I proposed to modify -j1 to run tests in subrpocesses. I even mentionned my issue. I suggest to use -jN on all buildbot, at least -j1. Maybe -j2 is even better since many tests are waiting on IO or simple sleep. Victor

2 1

MAKE_FUNCTION simplification
by Nikita Nemkin April 14, 2016

April 14, 2016

MAKE_FUNCTION opcode is complex due to the way it receives input arguments: 1) default args, individually; 2) default kwonly args, individual name-value pairs; 3) a tuple of parameter names (single constant); 4) annotation values, individually; 5) code object; 6) qualname. The counts for 1,2,4 are packed into oparg bitfields, making oparg large. My suggestion is to pre-package 1-4 before calling MAKE_FUNCTION, i.e. explicitly emit BUILD_TUPLE for defaults args and BUILD_MAPs for keyword defaults and annotations. Then, MAKE_FUNCTION will become a dramatically simpler 5 argument opcode, taking 1) default args tuple (optional); 2) default keyword only args dict (optional); 3) annotations dict (optional); 4) code object; 5) qualname. These arguments correspond exactly to __annotations__, __kwdefaults__, __defaults__, __code__ and __qualname__ attributes. For optional args, oparg bits should indicate individual arg presence. (This also saves None checks in opcode implementation.) If we add another optional argument (and oparg bit) for __closure__ attribute, then separate MAKE_CLOSURE opcode becomes unnecessary. Default args tuple is likely to be a constant and can be packaged whole, compensating for the extra size of explicit BUILD_* instructions. Compare the current implementation: https://github.com/python/cpython/blob/master/Python/ceval.c#L3262 with this provisional implementation (untested): TARGET(MAKE_FUNCTION) { PyObject *qualname = POP(); PyObject *codeobj = POP(); PyFunctionObject *func; func = (PyFunctionObject *)PyFunction_NewWithQualName( codeobj, f->f_globals, qualname); Py_DECREF(codeobj); Py_DECREF(qualname); if (func == NULL) goto error; /* NB: Py_None is not an acceptable value for these. */ if (oparg & 0x08) func->func_closure = POP(); if (oparg & 0x04) func->func_annotations = POP(); if (oparg & 0x02) func->func_kwdefaults = POP(); if (oparg & 0x01) func->func_defaults = POP(); PUSH((PyObject *)func); DISPATCH(); } compile.c also gets a bit simpler, but not much. What do you think?

3 3

Not receiving bug tracker emails
by Martin Panter April 14, 2016

April 14, 2016

For the last ~36 hours I have stopped receiving emails for messages posted in the bug tracker. Is anyone else having this problem? Has anything changed recently? I have had it set to send to my gmail.com address since the beginning. At the moment the last bug message email is <https://bugs.python.org/issue19959#msg262569> with “Date: Mon, 28 Mar 2016 12:19:49 +0000”. I have checked spam and they are not going there. Earlier this year I had to set up a rule to avoid lots of tracker emails suddenly going to spam. I suspect there was something about the emails that Google doesn’t like (though I don’t understand the technical details). Maybe this has recently gotten worse at the Google end?

6 16

Re: [Python-Dev] pathlib - current status of discussions
by Random832 April 14, 2016

April 14, 2016

1 0

Re: [Python-Dev] pathlib - current status of discussions
by Random832 April 13, 2016

April 13, 2016

On Apr 13, 2016 19:06, Brett Cannon <brett(a)python.org> wrote: > On Wed, 13 Apr 2016 at 15:46 Nikolaus Rath <Nikolaus(a)rath.org> wrote: >> When passing an object that is of type str and has a __fspath__ >> attribute, all approaches return the value of __fspath__(). >> >> However, when passing something of type bytes, the second approach >> returns the object, while the third returns the value of __fspath__(). >> >> Is this intentional? I think a __fspath__ attribute should always be >> preferred. > > > It's very much intentional. If we define __fspath__() to only return strings but still want to minimize boilerplate of allowing bytes to simply pass through without checking a path argument to see if it is bytes then approach #2 is warranted. But if __fspath__() can return bytes then approach #3 allows for it. Er, the difference comes in when the object passed to os.fspath is a subclass of bytes that, itself, has a __fspath__ method (which may return a str). It's unlikely to occur in the wild, but is a semantic difference between this case and all other objects with __fspath__ methods.

1 0