Mailman 3 October 2018 - Python-Dev

Custom AIX build last week...
by Michael Felt Oct. 30, 2018

Oct. 30, 2018

I noticed that there was a "custom" build queued for my AIX build-bot last week. (https://buildbot.python.org/all/#/builders/159/builds/1). It failed, but I hope that was due to the issue with install.sh. If it could be run again - and if it fails again, please let me know what the test was, and I'll look at it on my server manually. If it succeeds - I would also appreciate a heads-up! Thanks, Michael

2 2

short-circuiting runtime errors/exceptions in python debugger.
by Ed Peschko Oct. 30, 2018

Oct. 30, 2018

all, I was debugging a very long script that I was not all that familiar with, and I was doing my familiar routine of being very careful in evaluating expressions to make sure that I didn't hit such statements as: TypeError: unsupported operand type(s) for +: 'int' and 'str' anyways the script has a runtime of hours, so this was tedious work, and I hit one-too-many times where I missed a condition and had to start all over again. So that got me thinking: the main problem with exceptions and runtime errors is that they short-circuit the program context. You have this context, but you can't change it to avoid the failure; ie: with 1. aa = 'a' 2. bb = 1 + a 3. print bb you'll never get to line 3 if you go through line 2. If you are lucky you can catch it, modify aa to be an integer to continue, otherwise your only recourse is to start over again. So I was wondering if it would be possible to keep that context around if you are in the debugger and rewind the execution point to before the statement was triggered. so you could say: python script.py (Pdb) c then hit the exception, say: (Pdb) aa = 2 then run again (Pdb) c to work your way through the exception point. You could then fix the script inline in an editor. I can't emphasize exactly how much time and effort this would save. At best, debugging these types of issues is annoying, at worst it is excruciating, because they are sometimes intermittent and are not easily repeatable. Using RemotePdb or Pdb to attach to a long-running process and having the assurance that the underlying script won't die because of an inane coding error would do wonders for the reliability and integrity of scripting. so - is this possible? just from my experiments it doesn't look so, but perhaps there is a trick out there that would give this functionality.. if it isn't possible, how easy would it be to implement? Thanks much, Ed

9 9

Re: [Python-Dev] short-circuiting runtime errors/exceptions in the python debugger.
by Ed Peschko Oct. 29, 2018

Oct. 29, 2018

Steve, thanks for the response, and yes, I've experimented with reverse debugging, and yes for the reasons specified in that article you gave it isn't really practical with anything but small projects because of the massive amounts of memory usage. But that's really not what I'm asking for here. I'm simply asking that python give developers the opportunity to stop execution and open up a debugger *before* the exception is hit, and be able to instrument the code at that point using that debugger. The way that I mimic this in perl is by wrapping places where die, confess, croak, AUTOLOAD, etc are hit with something that looks like this: sub _confess { $DB::single = 1; my $trap = 1; if ($trap) { confess(@_); } } that way, I can short-circuit evals before they happen and save the state to both examine and modify. It's a hack, but it's a manageable hack since there are only so many places where things can go off the rails in perl, and I cover each exit. I can set $trap = 0 and lo-and-behold it will continue as if no trap was issued. Now, if there is global reverse debugging implemented in python, this has got to be doable as well. I'd dare say it could be done with very little overhead in memory. > You know you can set breakpoints in the debugger? You don't have to > single-step all the way through from the beginning. It isn't clear from > your post how experienced you are. ok just to be on the totally clear side, yes I'm quite aware of pdb.set_trace(), etc. They however don't do much to help the pain of debugging and maintaining code which is *not your own*. For all good they are, you might as well step through each line because you have no clue where the mines in a particular script or library reside, and when you are going to hit them. And of course depending on how good the debugging implementation is, when you hit these mines they may be well-nigh impossible to remove without something like this. Which btw is the main reason WHY I'm asking for this, and this is why I think it would be such a productivity booster. Not for my code which I control, but for others. I'll take one particular experience here to make my point. I've been experimenting with an automation framework written in python here that shall otherwise remain nameless (except it is a fairly popular one). Its ssh layer is *notoriously* obtuse, it freezes up on multiple occasions,the support lists are clogged with issues surrounding it, yet release-after-release things don't get fixed. Why? I'd argue because the code is hard to parse but even harder to debug. Having something like this would be a great boon to fixing any such issues. So that's basically it. No - I'm not looking for a full-blown reverse debugger for the reasons you state. I'm looking for something more limited and therefore more workable. Ed

1 0

AIX build-bots broken for 11 days
by Michael Felt Oct. 29, 2018

Oct. 29, 2018

a) I wish I seen this earlier, but I have been away from OSS for the last weeks as my RL job has priority. b) to answer a very different question I needed to look at the current (Always FAIL) status of the the two AIX buildbots - and I saw that 11 days ago they both started failing with the following messages: renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8 ./install-sh -c -m 644 ./Tools/gdb/libpython.py python-gdb.py ./install-sh: not found make: 1254-004 The error code from the last command is 1. renaming build/scripts-3.8/2to3 to build/scripts-3.8/2to3-3.8 ./install-sh -c -m 644 ./Tools/gdb/libpython.py python-gdb.py make: execvp: ./install-sh: The file access permissions do not allow the specified action. Makefile:662: recipe for target 'python-gdb.py' failed I am quite surprised, because, afaik, gdb is not used on either of the AIX bots. In any case These two runs failed in the "normal" way https://buildbot.python.org/all/#/builders/10/builds/1644/steps/4/logs/stdio https://buildbot.python.org/all/#/builders/161/builds/327/steps/4/logs/stdio And these fail during the build. https://buildbot.python.org/all/#/builders/10/builds/1645/steps/2/logs/stdio https://buildbot.python.org/all/#/builders/161/builds/328/steps/2/logs/stdio Help appreciated. Michael

1 0

Helping with Documentation
by Kenneth Reitz Oct. 29, 2018

Oct. 29, 2018

Hello all, I’d like to become a core contributor to Python, by contributing polish to its documentation (adding missing pieces, modernize it a bit in spots, add more usage examples (itertools), etc). Is anyone already working on this, and if so, can I join forces with you? -- Kenneth Reitz

3 2

Re: [Python-Dev] Python Language Governance Proposals
by Jeroen Demeyer Oct. 27, 2018

Oct. 27, 2018

On 2018-10-26 19:17, Brett Cannon wrote: > But since you're asking about wanting to "review PEPs", you can review > them now. Unfortunately not everybody agrees on that... See https://mail.python.org/pipermail/python-dev/2018-October/155441.html in particular I really hope that I won't have to wait 5 more months before a decision can be made on PEP 580.

1 0

Re: [Python-Dev] Python Language Governance Proposals
by Jeroen Demeyer Oct. 26, 2018

Oct. 26, 2018

What is the timeframe for the installation of the new governance? In other words, when will it be possible to review PEPs?

3 2

Summary of Python tracker Issues
by Python tracker Oct. 26, 2018

Oct. 26, 2018

ACTIVITY SUMMARY (2018-10-19 - 2018-10-26) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6823 (-12) closed 40005 (+62) total 46828 (+50) Open issues with patches: 2716 Issues opened (34) ================== #25750: tp_descr_get(self, obj, type) is called without owning a refer https://bugs.python.org/issue25750 reopened by serhiy.storchaka #35028: Off by one error in cgi.FieldStorage(max_num_fields) https://bugs.python.org/issue35028 opened by Matthew Belisle #35031: test_asyncio test_start_tls_server_1 fails in AMD64 FreeBSD CU https://bugs.python.org/issue35031 opened by pablogsal #35032: Remove the videos from faq/Windows https://bugs.python.org/issue35032 opened by matrixise #35033: Column or row spanning cells are not implemented. https://bugs.python.org/issue35033 opened by mdk #35035: Documentation for email.utils is named email.util.rst https://bugs.python.org/issue35035 opened by zmwangx #35037: PYLONG_BITS_IN_DIGIT differs between MinGW and MSVC https://bugs.python.org/issue35037 opened by scoder #35040: [functools] provide an async-compatible version of functools.l https://bugs.python.org/issue35040 opened by Liran Nuna #35042: Use the role :pep: for the PEP \d+ https://bugs.python.org/issue35042 opened by matrixise #35045: test_min_max_version (test.test_ssl.ContextTests) fails on Fed https://bugs.python.org/issue35045 opened by cstratak #35047: Better error messages in unittest.mock https://bugs.python.org/issue35047 opened by Petter S #35048: Can't reassign __class__ despite the assigned class having ide https://bugs.python.org/issue35048 opened by bup #35050: Off-by-one bug in AF_ALG https://bugs.python.org/issue35050 opened by christian.heimes #35052: Coverity scan: copy/paste error in Lib/xml/dom/minidom.py https://bugs.python.org/issue35052 opened by cstratak #35054: Add more index entries for symbols https://bugs.python.org/issue35054 opened by serhiy.storchaka #35055: Error when we try to download the epub archive https://bugs.python.org/issue35055 opened by matrixise #35056: Test leaks of memory not managed by Python allocator https://bugs.python.org/issue35056 opened by serhiy.storchaka #35058: Unable to Install Python on Windows https://bugs.python.org/issue35058 opened by alexbach #35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions https://bugs.python.org/issue35059 opened by vstinner #35061: Specify libffi.so soname for ctypes https://bugs.python.org/issue35061 opened by tturbs #35062: io.IncrementalNewlineDecoder assign out-of-range value to bitw https://bugs.python.org/issue35062 opened by xiang.zhang #35063: Checking for abstractmethod implementation fails to consider M https://bugs.python.org/issue35063 opened by Antony.Lee #35064: COUNT_ALLOCS build export inc_count() and dec_count() function https://bugs.python.org/issue35064 opened by vstinner #35065: Reading received data from a closed TCP stream using `StreamRe https://bugs.python.org/issue35065 opened by vxgmichel #35066: Inconsistency between dangling '%' handling in time.strftime() https://bugs.python.org/issue35066 opened by mjsaah #35067: Use vswhere instead of _distutils_findvs https://bugs.python.org/issue35067 opened by steve.dower #35068: [2.7] Possible crashes due to incorrect error handling in pyex https://bugs.python.org/issue35068 opened by ZackerySpytz #35070: test_posix fails on macOS 10.14 Mojave https://bugs.python.org/issue35070 opened by barry #35071: Canot send real string from c api to module (corrupted string) https://bugs.python.org/issue35071 opened by Yhojann Aguilera #35072: re.sub does not play nice with chr(92) https://bugs.python.org/issue35072 opened by Samuel Warfield #35074: source install [3.7.1] on debian jessie https://bugs.python.org/issue35074 opened by foxleoly #35075: Doc: pprint example uses dead URL https://bugs.python.org/issue35075 opened by inada.naoki #35076: FAIL: test_min_max_version (test.test_ssl.ContextTests) with l https://bugs.python.org/issue35076 opened by jean-michel #35077: Make TypeError message less ambiguous https://bugs.python.org/issue35077 opened by coyot Most recent 15 issues with no replies (15) ========================================== #35076: FAIL: test_min_max_version (test.test_ssl.ContextTests) with l https://bugs.python.org/issue35076 #35075: Doc: pprint example uses dead URL https://bugs.python.org/issue35075 #35068: [2.7] Possible crashes due to incorrect error handling in pyex https://bugs.python.org/issue35068 #35067: Use vswhere instead of _distutils_findvs https://bugs.python.org/issue35067 #35064: COUNT_ALLOCS build export inc_count() and dec_count() function https://bugs.python.org/issue35064 #35063: Checking for abstractmethod implementation fails to consider M https://bugs.python.org/issue35063 #35062: io.IncrementalNewlineDecoder assign out-of-range value to bitw https://bugs.python.org/issue35062 #35061: Specify libffi.so soname for ctypes https://bugs.python.org/issue35061 #35048: Can't reassign __class__ despite the assigned class having ide https://bugs.python.org/issue35048 #35045: test_min_max_version (test.test_ssl.ContextTests) fails on Fed https://bugs.python.org/issue35045 #35035: Documentation for email.utils is named email.util.rst https://bugs.python.org/issue35035 #35024: Incorrect logging in importlib when '.pyc' file creation fails https://bugs.python.org/issue35024 #35018: Sax parser provides no user access to lexical handlers https://bugs.python.org/issue35018 #35009: argparse throws UnicodeEncodeError for printing help with unic https://bugs.python.org/issue35009 #35000: aexit called after loop close https://bugs.python.org/issue35000 Most recent 15 issues waiting for review (15) ============================================= #35068: [2.7] Possible crashes due to incorrect error handling in pyex https://bugs.python.org/issue35068 #35067: Use vswhere instead of _distutils_findvs https://bugs.python.org/issue35067 #35065: Reading received data from a closed TCP stream using `StreamRe https://bugs.python.org/issue35065 #35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions https://bugs.python.org/issue35059 #35054: Add more index entries for symbols https://bugs.python.org/issue35054 #35050: Off-by-one bug in AF_ALG https://bugs.python.org/issue35050 #35047: Better error messages in unittest.mock https://bugs.python.org/issue35047 #35042: Use the role :pep: for the PEP \d+ https://bugs.python.org/issue35042 #35035: Documentation for email.utils is named email.util.rst https://bugs.python.org/issue35035 #35032: Remove the videos from faq/Windows https://bugs.python.org/issue35032 #35031: test_asyncio test_start_tls_server_1 fails in AMD64 FreeBSD CU https://bugs.python.org/issue35031 #35028: Off by one error in cgi.FieldStorage(max_num_fields) https://bugs.python.org/issue35028 #35024: Incorrect logging in importlib when '.pyc' file creation fails https://bugs.python.org/issue35024 #35022: MagicMock should support `__fspath__` https://bugs.python.org/issue35022 #35021: Assertion failures in datetimemodule.c. https://bugs.python.org/issue35021 Top 10 most discussed issues (10) ================================= #35059: Convert Py_INCREF() and PyObject_INIT() to inlined functions https://bugs.python.org/issue35059 17 msgs #33015: Fix function cast warning in thread_pthread.h https://bugs.python.org/issue33015 13 msgs #35052: Coverity scan: copy/paste error in Lib/xml/dom/minidom.py https://bugs.python.org/issue35052 12 msgs #34576: [EASY doc] http.server, SimpleHTTPServer: warn users on securi https://bugs.python.org/issue34576 10 msgs #35066: Inconsistency between dangling '%' handling in time.strftime() https://bugs.python.org/issue35066 9 msgs #35042: Use the role :pep: for the PEP \d+ https://bugs.python.org/issue35042 8 msgs #35070: test_posix fails on macOS 10.14 Mojave https://bugs.python.org/issue35070 8 msgs #35055: Error when we try to download the epub archive https://bugs.python.org/issue35055 7 msgs #9263: Try to print repr() when an C-level assert fails (in the garba https://bugs.python.org/issue9263 6 msgs #35037: PYLONG_BITS_IN_DIGIT differs between MinGW and MSVC https://bugs.python.org/issue35037 6 msgs Issues closed (60) ================== #1621: Do not assume signed integer overflow behavior https://bugs.python.org/issue1621 closed by vstinner #8525: Display exceptions' subclasses in help() https://bugs.python.org/issue8525 closed by ncoghlan #20216: Misleading docs for sha1, sha256, sha512, md5 modules https://bugs.python.org/issue20216 closed by vstinner #21196: Name mangling example in Python tutorial https://bugs.python.org/issue21196 closed by vstinner #21967: Interpreter crash upon accessing frame.f_restricted of a frame https://bugs.python.org/issue21967 closed by serhiy.storchaka #29843: errors raised by ctypes.Array for invalid _length_ attribute https://bugs.python.org/issue29843 closed by taleinat #30863: Rewrite PyUnicode_AsWideChar() and PyUnicode_AsWideCharString( https://bugs.python.org/issue30863 closed by serhiy.storchaka #32236: open() shouldn't silently ignore buffering=1 in binary mode https://bugs.python.org/issue32236 closed by vstinner #32256: Make patchcheck.py work for out-of-tree builds https://bugs.python.org/issue32256 closed by izbyshev #32321: Add pure Python fallback for functools.reduce https://bugs.python.org/issue32321 closed by vstinner #32798: mmap.flush() on Linux does not accept the "offset" and "size" https://bugs.python.org/issue32798 closed by vstinner #32890: os: Some functions may report bogus errors on Windows https://bugs.python.org/issue32890 closed by vstinner #33594: add deprecation since 3.5 for a few methods of inspect. https://bugs.python.org/issue33594 closed by vstinner #33708: Doc: Asyncio's Event documentation typo. https://bugs.python.org/issue33708 closed by vstinner #33712: OrderedDict can set an exception in tp_clear https://bugs.python.org/issue33712 closed by serhiy.storchaka #33726: Add short descriptions to PEP references in seealso https://bugs.python.org/issue33726 closed by vstinner #33947: Dataclasses can raise RecursionError in __repr__ https://bugs.python.org/issue33947 closed by eric.smith #34070: Superfluous call to isatty in open() when buffering >= 0 https://bugs.python.org/issue34070 closed by vstinner #34081: Sphinx duplicate label warning in docs https://bugs.python.org/issue34081 closed by mdk #34260: shutil.copy2 is not the same as cp -p https://bugs.python.org/issue34260 closed by vstinner #34282: Enum._convert shadows members named _convert https://bugs.python.org/issue34282 closed by orlnub123 #34482: datetime: Tests for potential crashes due to non-UTF-8-encodab https://bugs.python.org/issue34482 closed by taleinat #34551: Redundant store can be removed from _PyFunction_FastCallDict https://bugs.python.org/issue34551 closed by lukasz.langa #34573: Simplify __reduce__() of set and dict iterators. https://bugs.python.org/issue34573 closed by vstinner #34574: OrderedDict iterators are exhausted during pickling https://bugs.python.org/issue34574 closed by serhiy.storchaka #34748: Incorrect HTML link in functools.partial https://bugs.python.org/issue34748 closed by xiang.zhang #34789: Make xml.sax.make_parser accept iterables https://bugs.python.org/issue34789 closed by taleinat #34824: _ssl.c: Possible null pointer dereference https://bugs.python.org/issue34824 closed by vstinner #34839: doctest: Change example under warnings section https://bugs.python.org/issue34839 closed by mdk #34890: Support functools.partial in inspect.is*function() checks https://bugs.python.org/issue34890 closed by pablogsal #34901: Missing isolated (-I) flag in sys.flags table https://bugs.python.org/issue34901 closed by ned.deily #34912: Update overflow checks in resize_buffer https://bugs.python.org/issue34912 closed by vstinner #34936: tkinter.Spinbox.selection_element() raises TclError https://bugs.python.org/issue34936 closed by serhiy.storchaka #34970: Protect tasks weak set manipulation in asyncio.all_tasks() https://bugs.python.org/issue34970 closed by ned.deily #34973: Crash in bytes constructor with mutating list https://bugs.python.org/issue34973 closed by serhiy.storchaka #34980: KillPython target doesn't detect 64-bit processes https://bugs.python.org/issue34980 closed by jkloth #34983: expose symtable.Symbol.is_nonlocal() https://bugs.python.org/issue34983 closed by pablogsal #34984: Improve error messages in bytes and bytearray constructors https://bugs.python.org/issue34984 closed by serhiy.storchaka #34991: variable type list [] referential integrity data loss https://bugs.python.org/issue34991 closed by ammar2 #35017: socketserver accept a last request after shutdown https://bugs.python.org/issue35017 closed by vstinner #35020: Add multisort recipe to sorting docs https://bugs.python.org/issue35020 closed by rhettinger #35025: Compiling `timemodule.c` can fail on macOS due to availability https://bugs.python.org/issue35025 closed by benjamin.peterson #35027: distutils.core.setup does not raise TypeError when if classifi https://bugs.python.org/issue35027 closed by xtreak #35029: Convert SyntaxWarning exception raised at code generation time https://bugs.python.org/issue35029 closed by serhiy.storchaka #35030: Python 2.7 OrderedDict creates circular references https://bugs.python.org/issue35030 closed by rhettinger #35034: Add closing and iteration to threading.Queue https://bugs.python.org/issue35034 closed by rhettinger #35036: logger failure in suspicious.py https://bugs.python.org/issue35036 closed by pablogsal #35038: AttributeError: 'frame' object has no attribute 'f_restricted' https://bugs.python.org/issue35038 closed by matrixise #35039: remove unused vars in Lib/turtledemo module https://bugs.python.org/issue35039 closed by vstinner #35041: urllib.parse.quote safe Parameter Not Optional https://bugs.python.org/issue35041 closed by ammar2 #35043: functools.reduce doesn't work properly with itertools.chain https://bugs.python.org/issue35043 closed by Vasantha Ganesh Kanniappan #35044: Use the :exc: role for the exceptions in the doc https://bugs.python.org/issue35044 closed by vstinner #35046: logging.StreamHandler performs two syscalls when one would do https://bugs.python.org/issue35046 closed by vinay.sajip #35049: argparse.ArgumentParser fails on arguments with leading dash a https://bugs.python.org/issue35049 closed by paul.j3 #35051: Fix pep8 on Lib/turtledemo module https://bugs.python.org/issue35051 closed by brett.cannon #35053: Enhance tracemalloc to trace properly free lists https://bugs.python.org/issue35053 closed by vstinner #35057: Email header refolding adds additional \r in nested parse tree https://bugs.python.org/issue35057 closed by r.david.murray #35060: subprocess output seems to depend on size of terminal screen https://bugs.python.org/issue35060 closed by eryksun #35069: Unexecuted import in function causes UnboundLocalError https://bugs.python.org/issue35069 closed by steven.daprano #35073: 'from app import __init__' behaves differently with native imp https://bugs.python.org/issue35073 closed by qagren

1 0

"Deprecation" of os.system in favor of subprocess?
by Stephane Wirtel Oct. 26, 2018

Oct. 26, 2018

Good morning/afternoon/evening/night ;-) In the documentation of os.system [1], there is this paragraph, where we suggest to use subprocess instead of os.system. """ The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function. See the Replacing Older Functions with the subprocess Module section in the subprocess documentation for some helpful recipes. """ But I have found some references (between 15 and 23 (if we include setup.py)) of os.system. Do we need to keep the code like that or could we start to use subprocess for these "references" because it is preferable? If you think we could move to subprocess, I would like to open an issue and try to fix it. 1. Add the 'deprecated' directive in the doc 2. Use subprocess for these references What is your opinion? Thank you, Stéphane [1] https://docs.python.org/3.8/library/os.html?highlight=os%20system#os.system -- Stéphane Wirtel - https://wirtel.be - @matrixise

14 16

Re: Python Language Governance Proposals
by eamanu15 Oct. 23, 2018

Oct. 23, 2018

Are there any list of candidates?

2 1