Mailman 3 June 2018 - Python-Dev

Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
by Victor Stinner 08 Jun '18

08 Jun '18

Hi, I fixed a few tests which failed randomly. There are still a few, but the most annoying have been fixed. I will continue to keep an eye on our CIs: Travis CI, AppVeyor and buildbots. Sometimes, even when I report a regression, the author doesn't fix the bug. But when a test fails on Travis CI or AppVeyor, we cannot merge a pull request, so it blocks our whole workflow. I will restart the policy that I proposed last year: if a change introduces a regression and I'm unable to fix it quickly (say in less than 2 hours and the author isn't available), I will simply revert the change. Please don't take it personally, the purpose is just to unblock our workflow and be able to detect other regressions. It's just an opportunity for the change author to fix the change without the pressure of the broken CI. Buildbots only send email notifications to buildbot-status(a)python.org when the state changes from success (green) to failure (red). It's much simpler for me to spot a regression when most buildbots are green. By the way, while miss-lington is really an amazing tool (thanks Mariatta!!!), most core developers (including me!) forgot that 2 years ago, the policy was to check if a change doesn't break buildbots before backporting a change (well, 2 years ago we forward-ported changes, but it's the same idea ;-)). Please remind that we only run tests on Linux and Windows as pre-commit checks on GitHub pull requests, whereas it's very common to spot bugs on buildbots thanks to the diversity of platforms and architectures (and different performance). All buildbot builders can be watched at: http://buildbot.python.org/all/#/builders But I prefer to watch notifications on IRC (#python-dev on Freenode) and the buildbot-status mailing list. https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ -- Night gathers, AND NOW MY WATCH BEGINS. IT SHALL NOT END UNTIL MY DEATH. I shall take no wife, hold no lands, father no children. I shall wear no crowns and win no glory. I shall live and die at my post. I am the sword in the darkness. I am the watcher on the walls. I am the fire that burns against cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men. I pledge my life and honor to the Night's Watch, for this night and all the nights to come. Victor

3 5

Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
by Ivan Pozdeev 06 Jun '18

06 Jun '18

No, replying only to you wasn't intended. https://docs.travis-ci.com/user/running-build-in-debug-mode/ is the official doc on how to debug a Travis CI build via ssh. On 04.06.2018 22:31, Victor Stinner wrote: > FYI you only replied to me in private. Is it on purpose? > > I'm interested if I can learn how to get a SSH access to Travis CI! > > 2018-06-04 21:05 GMT+02:00 Ivan Pozdeev <vano(a)mail.mipt.ru>: >> On 04.06.2018 19:31, Victor Stinner wrote: >>> 2018-05-30 11:33 GMT+02:00 Victor Stinner <vstinner(a)redhat.com>: >>>> I fixed a few tests which failed randomly. There are still a few, but >>>> the most annoying have been fixed. >>> Quick update a few days later. >>> >>> For an unknown reason, >>> test_multiprocessing_forkserver.TestIgnoreEINTR.test_ignore() started >>> to fail very frequently but only on Travis CI. I have no explanation >>> why only Travis CI. I failed to reproduce the issue on a Ubuntu Trusty >>> container or in a Ubuntu Trusty VM. After hours of debug, I found the >>> bug and wrote a fix. But the fix didn't work in all cases. A second >>> fix and now it seems like the issue is gone! >> >> FYI Travis claim they provide ssh access on request to debug particularly >> pesky issues. >> Last time I tried, got no response from them though. >> >>> https://bugs.python.org/issue33532 if you are curious about the >>> strange multiprocessing send() which must block but it doesn't :-) >>> >>> Except Windows 7 which has issues with test_asyncio and >>> multiprocessing tests because this buildbot is slow, it seems like >>> most CIs are now stable. >>> >>> Known issues: >>> >>> * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: >>> https://bugs.python.org/issue33630 >>> * AIX: always red >>> * USBan: experimental buildbot >>> * Alpine: platform not supported yet (musl issues) >>> >>> Victor >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev(a)python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru >> >> -- >> Regards, >> Ivan >> -- Regards, Ivan

5 6

Docstrings on builtins
by Chris Barker 06 Jun '18

06 Jun '18

Over on python-ideas, someone is/was proposing literals for timedeltas. I don't expect that will come to anything, but it did make me take a look at the docstring for datetime.timedelta. I use iPython's ? a lot for a quick overview of how to use a class/function. This is what I get: In [8]: timedelta? Init signature: timedelta(self, /, *args, **kwargs) Docstring: Difference between two datetime values. File: ~/miniconda2/envs/py3/lib/python3.6/datetime.py Type: type That is, well, not so useful. I'd like to see at least the signature: datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0 And ideally much of the text in the docs. I've noticed similarly minimal docstrings on a number of builtin functions and methods. If I wanted to contribute a PR to enhance these docstrings, where would they go? I've seen mention of "argument clinic", but really don't know quite what that is, or how it works, but it appears to be related. Anyway -- more comprehensive docstrings on buildins could really help Python's usability for command line usage. Thanks, - Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

5 9

Re: [Python-Dev] Stable ABI
by Jeroen Demeyer 05 Jun '18

05 Jun '18

On 2018-06-01 17:18, Nathaniel Smith wrote: > Unfortunately, very few people use the stable ABI currently, so it's > easy for things like this to get missed. So there are no tests for the stable ABI in Python?

7 9

Withdraw PEP 546? Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
by Victor Stinner 05 Jun '18

05 Jun '18

Hi, tl; dr I will withdraw the PEP 546 in one week if noboy shows up to finish the implementation. Last year,I wrote the PEP 546 with Cory Benfield: "Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7" https://www.python.org/dev/peps/pep-0546/ The plan was to get a Python 2.7 implementation of Cory's PEP 543: "A Unified TLS API for Python" https://www.python.org/dev/peps/pep-0543/ Sadly, it seems like Cory is no longer available to work on the projec (PEP 543 is still a draft)t. The PEP 546 is implemented: https://github.com/python/cpython/pull/2133 Well, I closed it, but you can still get it as a patch with: https://patch-diff.githubusercontent.com/raw/python/cpython/pull/2133.patch But tests fail on Travis CI whereas I'm unable to reproduce the issue on my laptop (on Fedora). The failure seems to depend on the version of OpenSSL. Christian Heimes has a "multissl" tool which automates tests on multiple OpenSSL versions, but I failed to find time to try this tool. Time flies and one year later, the PR of the PEP 546 is still not merged, tests are still failing. One month ago, when 2.7.15 has been released, Benjamin Peterson, Python 2.7 release manager, simply proposed: "The lack of movement for a year makes me wonder if PEP 546 should be moved to Withdrawn status." Since again, I failed to find time to look at the test_ssl failure, I plan to withdraw the PEP next week if nobody shows up :-( Sorry Python 2.7! Does anyone would benefit of MemoryBIO in Python 2.7? Twisted, asyncio, trio, urllib3, anyone else? If yes, who is volunteer to finish the MemoryBIO backport (and maintain it)? Victor

5 7

Why not using "except: (...) raise" to cleanup on error?
by Victor Stinner 04 Jun '18

04 Jun '18

Hi, I just read a recent bugfix in asyncio: https://github.com/python/cpython/commit/9602643120a509858d0bee4215d7f150e6… + try: + await waiter + except Exception: + transport.close() + raise Why only catching "except Exception:"? Why not also catching KeyboardInterrupt or MemoryError? Is it a special rule for asyncio, or a general policy in Python stdlib? For me, it's fine to catch any exception using "except:" if the block contains "raise", typical pattern to cleanup a resource in case of error. Otherwise, there is a risk of leaking open file or not flushing data on disk, for example. Victor

5 10

Add __reversed__ methods for dict
by Rémi Lapeyre 02 Jun '18

02 Jun '18

Hi, since dict keys are sorted by their insertion order since Python 3.6 and that it’s part of Python specs since 3.7 a proposal has been made in bpo-33462 to add the __reversed__ method to dict and dict views. Concerns have been raised in the comments that this feature may add too much bloat in the core interpreter and be harmful for other Python implementations. Given the different issues this change creates, I see three possibilities: 1. Accept the proposal has it is for dict and dict views, this would add about 300 lines and three new types in dictobject.c 2. Accept the proposal only for dict, this would add about 80 lines and one new type in dictobject.c while still being useful for some use cases 3. Drop the proposal as the whole, while having some use, reversed(dict(a=1, b=2)) may not be very common and could be done using OrderedDict instead. What’s your stance on the issue ? Best regards, Rémi Lapeyre

8 13

PyIndex_Check conflicts with PEP 384
by Christian Tismer 01 Jun '18

01 Jun '18

Hi friends, when implementing the limited API for PySide2, I recognized a little bug where a function was replaced by a macro. The file number.rst on python 3.6 says """ .. c:function:: int PyIndex_Check(PyObject *o) Returns ``1`` if *o* is an index integer (has the nb_index slot of the tp_as_number structure filled in), and ``0`` otherwise. """ Without notice, this function was replaced by a macro a while ago, which reads #define PyIndex_Check(obj) \ ((obj)->ob_type->tp_as_number != NULL && \ (obj)->ob_type->tp_as_number->nb_index != NULL) This contradicts PEP 384, because there is no way for non-heaptype types to access the nb_index field. If nobody objects, I would like to submit a patch that adds the function back when the limited API is active. I think to fix that before Python 3.7 is out. Ciao -- Chris -- Christian Tismer-Sperling :^) tismer(a)stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam : GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023

3 3

The history of PyXML
by Serhiy Storchaka 01 Jun '18

01 Jun '18

Does anyone has the full copy of the PyXML repository, with the complete history? This library was included in Python 2.1 as the xml package and is not maintained as a separate project since 2004. It's home on SourceForge was removed. I have found sources of the last PyXML version (0.8.4), but without history. I'm trying to figure out some intentions and fix possible bugs in the xml package. The history of all commits could help.

6 10

How to watch buildbots?
by Victor Stinner 01 Jun '18

01 Jun '18

Hi, I would like to delegate the maintenance task "watch buildbots", since I'm already very busy with many other maintenance tasks. I'm looking for volunteers to handle incoming emails on buildbot-status. I already started to explain to Pablo Galindo Salgado how to do that, but it would be great to have at least two people doing this task. Otherwise, Pablo wouldn't be able to take holiday or just make a break for any reason. Buildbots are evil beast which require care every day. Otherwise, they quickly turn red and become less useful :-( It seems like the first blocker issue is that we have no explicit documentation "how to deal with buildbots?" (the devguide documentation is incomplete, it doesn't explain what I'm explaining below). Let me start with a few notes of how I watch buildbots. I'm getting buildbot notifications on IRC (#python-dev on Freenode) and on the buildbot-status mailing list: https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ When a buildbot fails, I look at tests logs and I try to check if an issue has already been reported. For example, search for the test method in title (ex: "test_complex" for test_complex() method). If no result, search using the test filename (ex: "test_os" for Lib/test/test_os.py). If there is no result, repeat with full text searchs ("All Text"). If you cannot find any open bug, create a new one: * The title should contain the test name, test method and the buildbot name. Example: " test_posix: TestPosixSpawn fails on PPC64 Fedora 3.x". * The description should contain the link to the buildbot failure. Try to identify useful parts of tests log and copy them in the description. * Fill the Python version field (ex: "3.8" for 3.x buildbots) * Select at least the "Tests" Component. You may select additional Components depending on the bug. If a bug was already open, you may add a comment to mention that there is a new failure: add at least a link to buildbot name and a link to the failure. And that's all! Simple, isn't it? At this stage, there is no need to investigate the test failure. To finish, reply to the failure notification on the mailing list with a very short email: add a link to the existing or the freshly created issue, maybe copy one line of the failure and/or the issue title. Recent bug example: https://bugs.python.org/issue33630 -- Later, you may want to analyze these failures, but I consider that it's a different job (different "maintenance task"). If you don't feel able to analyze the bug, you may try to find someone who knows more than you about the failure. For better bug reports, you can look at the [Changes] tab of a build failure, and try to identify which recent change introduced the regression. This task requires to follow recent commits, since sometimes the failure is old, it's just that the test fails randomly depending on network issues, system load, or anything else. Sometimes, previous tests have side effects. Or the buildbot owner made a change on the system. There are many different explanation, it's hard to write a complete list. It's really on a case by case basis. Hopefully, it's now more common that a buildbot failure is obvious and caused by a very specific recent changes which can be found in the [Changes] tab. -- If you are interested to help me on watching our CIs: please come on the python-buildbot(a)python.org mailing list! Introduce yourself and explain how do you plan to help. I may propose to mentor you to assist you the first weeks. As I wrote, maybe a first step would be to write down a documentation how to deal with buildbots and/or update and complete existing documentations. https://devguide.python.org/buildbots/ Victor

4 5