Mailman 3 September 2018 - Python-Dev

What is the purpose of the _PyThreadState_Current symbol in Python 3?
by Gabriele 29 Sep '18

29 Sep '18

Hi Victor, > I understand that you are writing a debugger and you can only *read* > modify, not execute code, right? I'm working on a frame stack sampler that runs independently from the Python process. The project is "Austin" (https://github.com/P403n1x87/austin). Whilst I could, in principle, execute code with other system calls, I prefer not to in this case. > In the master branch, it's now _PyRuntime.gilstate.tstate_current. If > you run time.sleep(3600) and look into > _PyRuntime.gilstate.tstate_current using gdb, you can a NULL pointer > (tstate_current=0) because Python releases the GIL.. I would like my application to make as few assumptions as possible. The _PyRuntime symbol might not be available if all the symbols have been stripped out of the binaries. That's why I was trying to rely on _PyThreadState_Current, which is in the .dynsym section. Judging by the output of nm -D `which python3` (I'm on Python 3.6.6 at the moment) I cannot see anything more useful than that. My current strategy is to try and make something out of this symbol and then fall back to a brute force approach to scan the .bss section for valid PyInterpreterState instances (which works reliably well and is quite fast too, but a bit ugly). > There is also _PyGILState_GetInterpreterStateUnsafe() which gives > access to the current Python interpreter: > _PyRuntime.gilstate.autoInterpreterState. From the interpreter, you > can use the linked list of thread states from interp->tstate_head. > > I hope that I helped :-) Yes thanks! Your comment made me realise why I can use PyThreadState_Current at the very beginning, and it is because Python is going through the intensive startup process, which involves, among other things, the loading of frozen modules (I can clearly see most if not all the steps in the output of Austin, as mentioned in the repo's README). During this phase, the main (and only thread) holds the GIL and is quite busy doing stuff. The long-running applications that I was trying to attach to have very long wait periods where they sit idle waiting for a timer to trigger the next operations, that fire very quickly and put the threads back to sleep again. If this is what the _PyThreadState_Current is designed for, then I guess I cannot really rely on it, especially when attaching Austin to another process. Best regards, Gabriele

2 4

switch statement
by e2qb2a44f＠prolan-power.hu 29 Sep '18

29 Sep '18

Hi, A humble proposal for a switch-like statement syntax for Python: - - - switch blah in (100, 2, 30, 'bumm'): dosomething1() x = 88 case blah in (44, 55): otherstuff(9) case blah in (8): boo() else: wawa() - - - So, let's use, and allow only *tuples*. As early as possible, build a jump table, based on (foreknown) small integer values. As in other languages. Strings may have to be hashed (in "compile time"), to obtain small integer value. Some secondary checking may have to be done for exact content equality. (Alternative: do no allow strings at all.) For gaps in the integer range: maybe apply some very basic dividing/shifting to "compact" the range. (As compilers optimize in other languages, I guess -- but I may be totally wrong.) (For example find "unused bits" in the numbers (in 2-base representation). newnum = orignum >> 3 & 6 | orignum & ~6. newnum is smaller (roughly 1/8) than orignum.) The (achievable) goal is to be faster than hash table lookup. (A hash table with keys 100, 2, 30, 'bumm' etc.) And approach the speed of single array-index lookup. (Or even faster in some cases as there will be just jumps instead of calls?) (I am not an "expert"!) Let allow fallthrough or not? - To be decided. (Either is compatible with the above.) I know about PEP 3103 and https://docs.python.org/3.8/faq/design.html?highlight=switch#why-isn-t-ther… (I do not know how to comment on a PEP, where to discuss a PEP. If this is inappropriate place, please forward it.) --

6 5

Summary of Python tracker Issues
by Python tracker 28 Sep '18

28 Sep '18

ACTIVITY SUMMARY (2018-09-21 - 2018-09-28) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6781 (-14) closed 39803 (+80) total 46584 (+66) Open issues with patches: 2703 Issues opened (54) ================== #12782: Multiple context expressions do not support parentheses for co https://bugs.python.org/issue12782 reopened by lukasz.langa #28655: Tests altered the execution environment in isolated mode https://bugs.python.org/issue28655 reopened by vstinner #32528: Change base class for futures.CancelledError https://bugs.python.org/issue32528 reopened by yselivanov #34768: Add documentation explaining __init__.py in packages https://bugs.python.org/issue34768 opened by bkestelman #34769: _asyncgen_finalizer_hook running in wrong thread https://bugs.python.org/issue34769 opened by twisteroid ambassador #34771: test_ctypes failing on Linux SPARC64 https://bugs.python.org/issue34771 opened by kelledin-3 #34773: sqlite3 module inconsistently returning only some rows from a https://bugs.python.org/issue34773 opened by shankargopal #34774: IDLE: use theme colors for help viewer https://bugs.python.org/issue34774 opened by terry.reedy #34775: pathlib.PurePath division raises TypeError instead of returnin https://bugs.python.org/issue34775 opened by Roger Aiudi #34776: Postponed annotations break inspection of dataclasses https://bugs.python.org/issue34776 opened by drhagen #34778: Memoryview for column-major (f_contiguous) arrays from bytes i https://bugs.python.org/issue34778 opened by lgautier #34779: IDLE internals show up in tracebacks when returning objects th https://bugs.python.org/issue34779 opened by ppperry #34780: Hang on startup if stdin refers to a pipe with an outstanding https://bugs.python.org/issue34780 opened by izbyshev #34781: infinite waiting in multiprocessing.Pool https://bugs.python.org/issue34781 opened by coells #34782: Pdb crashes when code is executed in a mapping that does not d https://bugs.python.org/issue34782 opened by ppperry #34784: Heap-allocated StructSequences https://bugs.python.org/issue34784 opened by eelizondo #34785: pty.spawn -- auto-termination after child process is dead (a z https://bugs.python.org/issue34785 opened by jarryshaw #34788: ipaddress module fails on rfc4007 scoped IPv6 addresses https://bugs.python.org/issue34788 opened by Jeremy McMillan #34789: Make xml.sax.make_parser accept iterables https://bugs.python.org/issue34789 opened by adelfino #34790: Deprecate passing coroutine objects to asyncio.wait() https://bugs.python.org/issue34790 opened by yselivanov #34791: xml package does not obey sys.flags.ignore_environment https://bugs.python.org/issue34791 opened by christian.heimes #34792: Tutorial doesn''t discuss / and * function arguments https://bugs.python.org/issue34792 opened by diekhans #34793: Remove support for "with (await asyncio.lock):" https://bugs.python.org/issue34793 opened by yselivanov #34794: memory leak in TkApp:_createbytearray https://bugs.python.org/issue34794 opened by dtalkin #34795: loop.sock_recv failure because of delayed callback handling https://bugs.python.org/issue34795 opened by kyuupichan #34796: Tkinter scrollbar issues on Mac. https://bugs.python.org/issue34796 opened by terry.reedy #34797: Convert heapq to the argument clinic https://bugs.python.org/issue34797 opened by pablogsal #34798: pprint ignores the compact parameter for dicts https://bugs.python.org/issue34798 opened by Nicolas Hug #34799: When function in tracing returns None, tracing continues. https://bugs.python.org/issue34799 opened by fabioz #34800: email.contentmanager raises error when policy.max_line_length= https://bugs.python.org/issue34800 opened by silane #34801: codecs.getreader() splits lines containing control characters https://bugs.python.org/issue34801 opened by nascheme #34804: Repetition of 'for example' in documentation https://bugs.python.org/issue34804 opened by rarblack #34805: Explicitly specify `MyClass.__subclasses__()` returns classes https://bugs.python.org/issue34805 opened by pekka.klarck #34806: distutils tests fail with recent 3.7 branch https://bugs.python.org/issue34806 opened by doko #34807: pathlib.[r]glob fails when the toplevel directory is not reada https://bugs.python.org/issue34807 opened by Antony.Lee #34810: Maximum and minimum value of C types integers from Python https://bugs.python.org/issue34810 opened by scls #34811: test_gdb fails with latest gdb https://bugs.python.org/issue34811 opened by cstratak #34812: support.args_from_interpreter_flags() doesn't inherit -I (isol https://bugs.python.org/issue34812 opened by vstinner #34814: makesetup: must link C extensions to libpython when compiled i https://bugs.python.org/issue34814 opened by vstinner #34816: ctypes + hasattr https://bugs.python.org/issue34816 opened by lfriedri #34817: Ellipsis docs has extra dot in the markdown that makes it look https://bugs.python.org/issue34817 opened by xtreak #34818: test.test_ssl.ThreadedTests.test_tls1_3 fails in 2.7 with Attr https://bugs.python.org/issue34818 opened by xnox #34820: binascii.c:1578:1: error: the control flow of function ‘bina https://bugs.python.org/issue34820 opened by wencan #34821: Crash after run Python interpreter from removed directory https://bugs.python.org/issue34821 opened by Игорь Никитин #34822: Simplify AST for slices https://bugs.python.org/issue34822 opened by serhiy.storchaka #34823: libffi detection doesn’t work in my setup https://bugs.python.org/issue34823 opened by stapelberg #34824: _ssl.c: Possible null pointer dereference https://bugs.python.org/issue34824 opened by ZackerySpytz #34825: Add more entries to os module to pathlib reference table https://bugs.python.org/issue34825 opened by xtreak #34826: io.BufferedReader crashes in 2.7 on memoryview attribute acces https://bugs.python.org/issue34826 opened by gregory.p.smith #34828: sqlite.iterdump does not work for (most) databases with autoin https://bugs.python.org/issue34828 opened by itssme #34829: Add missing selection_ methods to tkinter Spinbox https://bugs.python.org/issue34829 opened by j-4321-i #34831: Asyncio Tutorial https://bugs.python.org/issue34831 opened by cjrh #34832: "Short circuiting" in base64's b64decode, decode, decodebytes https://bugs.python.org/issue34832 opened by pw.michael.harris #34833: [CI] Azure Pipeline: Initialize Agent failed https://bugs.python.org/issue34833 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #34832: "Short circuiting" in base64's b64decode, decode, decodebytes https://bugs.python.org/issue34832 #34831: Asyncio Tutorial https://bugs.python.org/issue34831 #34829: Add missing selection_ methods to tkinter Spinbox https://bugs.python.org/issue34829 #34825: Add more entries to os module to pathlib reference table https://bugs.python.org/issue34825 #34824: _ssl.c: Possible null pointer dereference https://bugs.python.org/issue34824 #34823: libffi detection doesn’t work in my setup https://bugs.python.org/issue34823 #34822: Simplify AST for slices https://bugs.python.org/issue34822 #34812: support.args_from_interpreter_flags() doesn't inherit -I (isol https://bugs.python.org/issue34812 #34807: pathlib.[r]glob fails when the toplevel directory is not reada https://bugs.python.org/issue34807 #34805: Explicitly specify `MyClass.__subclasses__()` returns classes https://bugs.python.org/issue34805 #34801: codecs.getreader() splits lines containing control characters https://bugs.python.org/issue34801 #34799: When function in tracing returns None, tracing continues. https://bugs.python.org/issue34799 #34797: Convert heapq to the argument clinic https://bugs.python.org/issue34797 #34796: Tkinter scrollbar issues on Mac. https://bugs.python.org/issue34796 #34785: pty.spawn -- auto-termination after child process is dead (a z https://bugs.python.org/issue34785 Most recent 15 issues waiting for review (15) ============================================= #34829: Add missing selection_ methods to tkinter Spinbox https://bugs.python.org/issue34829 #34828: sqlite.iterdump does not work for (most) databases with autoin https://bugs.python.org/issue34828 #34825: Add more entries to os module to pathlib reference table https://bugs.python.org/issue34825 #34824: _ssl.c: Possible null pointer dereference https://bugs.python.org/issue34824 #34822: Simplify AST for slices https://bugs.python.org/issue34822 #34818: test.test_ssl.ThreadedTests.test_tls1_3 fails in 2.7 with Attr https://bugs.python.org/issue34818 #34814: makesetup: must link C extensions to libpython when compiled i https://bugs.python.org/issue34814 #34800: email.contentmanager raises error when policy.max_line_length= https://bugs.python.org/issue34800 #34797: Convert heapq to the argument clinic https://bugs.python.org/issue34797 #34794: memory leak in TkApp:_createbytearray https://bugs.python.org/issue34794 #34791: xml package does not obey sys.flags.ignore_environment https://bugs.python.org/issue34791 #34790: Deprecate passing coroutine objects to asyncio.wait() https://bugs.python.org/issue34790 #34789: Make xml.sax.make_parser accept iterables https://bugs.python.org/issue34789 #34785: pty.spawn -- auto-termination after child process is dead (a z https://bugs.python.org/issue34785 #34784: Heap-allocated StructSequences https://bugs.python.org/issue34784 Top 10 most discussed issues (10) ================================= #34751: Hash collisions for tuples https://bugs.python.org/issue34751 56 msgs #34814: makesetup: must link C extensions to libpython when compiled i https://bugs.python.org/issue34814 18 msgs #22490: Using realpath for __PYVENV_LAUNCHER__ makes Homebrew installs https://bugs.python.org/issue22490 12 msgs #32892: Remove specific constant AST types in favor of ast.Constant https://bugs.python.org/issue32892 12 msgs #34162: idlelib/NEWS.txt for 3.8.0 (and backports) https://bugs.python.org/issue34162 9 msgs #34521: Multiple tests (test_socket, test_multiprocessing_*) fail due https://bugs.python.org/issue34521 9 msgs #34781: infinite waiting in multiprocessing.Pool https://bugs.python.org/issue34781 9 msgs #34806: distutils tests fail with recent 3.7 branch https://bugs.python.org/issue34806 9 msgs #28655: Tests altered the execution environment in isolated mode https://bugs.python.org/issue28655 8 msgs #31405: shutil.which doesn't find files without PATHEXT extension on W https://bugs.python.org/issue31405 7 msgs Issues closed (79) ================== #5950: Make zipimport work with zipfile containing comments https://bugs.python.org/issue5950 closed by barry #12458: Tracebacks should contain the first line of continuation lines https://bugs.python.org/issue12458 closed by serhiy.storchaka #16360: argparse: comma in metavar causes assertion failure when forma https://bugs.python.org/issue16360 closed by paul.j3 #23584: test_doctest lineendings fails in verbose mode https://bugs.python.org/issue23584 closed by xtreak #24937: Multiple problems in getters & setters in capsulethunk.h https://bugs.python.org/issue24937 closed by petr.viktorin #24997: mock.call_args compares as equal and not equal https://bugs.python.org/issue24997 closed by berker.peksag #26000: Crash in Tokenizer - Heap-use-after-free https://bugs.python.org/issue26000 closed by xtreak #26144: test_pkg test_4 and/or test_7 sometimes fail https://bugs.python.org/issue26144 closed by xtreak #26452: Wrong line number attributed to comprehension expressions https://bugs.python.org/issue26452 closed by xtreak #28418: Raise Deprecation warning for tokenize.generate_tokens https://bugs.python.org/issue28418 closed by xtreak #30350: devguide suggests to use VS 2008 to build Python 2.7, but VS 2 https://bugs.python.org/issue30350 closed by vstinner #30964: Mention ensurepip in package installation docs https://bugs.python.org/issue30964 closed by brett.cannon #31007: ERROR: test_pipe_handle (test.test_asyncio.test_windows_utils. https://bugs.python.org/issue31007 closed by vstinner #31425: Expose AF_QIPCRTR in socket module https://bugs.python.org/issue31425 closed by taleinat #31511: test_normalization: test.support.open_urlresource() doesn't ha https://bugs.python.org/issue31511 closed by vstinner #31535: configparser unable to write comment with a upper cas letter https://bugs.python.org/issue31535 closed by xtreak #31737: Documentation renders incorrectly https://bugs.python.org/issue31737 closed by mdk #31837: ParseError in test_all_project_files() https://bugs.python.org/issue31837 closed by xtreak #31986: [2.7] test_urllib2net.test_sites_no_connection_close() randoml https://bugs.python.org/issue31986 closed by vstinner #32117: Tuple unpacking in return and yield statements https://bugs.python.org/issue32117 closed by gvanrossum #32282: When using a Windows XP compatible toolset, `socketmodule.c` f https://bugs.python.org/issue32282 closed by steve.dower #32533: SSLSocket read/write thread-unsafety https://bugs.python.org/issue32533 closed by steve.dower #32552: Improve text for file arguments in argparse.ArgumentDefaultsHe https://bugs.python.org/issue32552 closed by paul.j3 #32556: support bytes paths in nt _getdiskusage, _getvolumepathname, a https://bugs.python.org/issue32556 closed by steve.dower #32557: allow shutil.disk_usage to take a file path on Windows also https://bugs.python.org/issue32557 closed by steve.dower #32718: Install PowerShell activation scripts for venv for all platfor https://bugs.python.org/issue32718 closed by brett.cannon #33016: nt._getfinalpathname may use uninitialized memory https://bugs.python.org/issue33016 closed by steve.dower #33091: ssl.SSLError: Invalid error code (_ssl.c:2217) https://bugs.python.org/issue33091 closed by steve.dower #33180: Flag for unusable sys.executable https://bugs.python.org/issue33180 closed by steve.dower #33309: Unittest Mock objects do not freeze arguments they are called https://bugs.python.org/issue33309 closed by xtreak #33415: When add_mutually_exclusive_group is built without argument, t https://bugs.python.org/issue33415 closed by paul.j3 #33442: Python 3 doc sidebar dosnt follow page scrolling like 2.7 doc https://bugs.python.org/issue33442 closed by xtreak #33782: VSTS Windows-PR: internal error https://bugs.python.org/issue33782 closed by steve.dower #33871: Possible integer overflow in iov_setup() https://bugs.python.org/issue33871 closed by serhiy.storchaka #33937: test_socket: SendmsgSCTPStreamTest.testSendmsgTimeout() failed https://bugs.python.org/issue33937 closed by vstinner #33977: [Windows] test_compileall fails randomly with PermissionError: https://bugs.python.org/issue33977 closed by vstinner #34046: subparsers -> add_parser doesn't support hyphen char '-' https://bugs.python.org/issue34046 closed by paul.j3 #34188: Allow dict choices to "transform" values in argpagse https://bugs.python.org/issue34188 closed by paul.j3 #34223: PYTHONDUMPREFS=1 ./python -c pass does crash https://bugs.python.org/issue34223 closed by vstinner #34248: dbm errors should contain file names https://bugs.python.org/issue34248 closed by berker.peksag #34267: find_python.bat doesn't find installed Python 3.7 https://bugs.python.org/issue34267 closed by steve.dower #34320: Creating dict from OrderedDict doesn't preserve order https://bugs.python.org/issue34320 closed by inada.naoki #34372: Parenthesized expression has incorrect line numbers https://bugs.python.org/issue34372 closed by serhiy.storchaka #34472: zipfile: does not include optional descriptor signature https://bugs.python.org/issue34472 closed by serhiy.storchaka #34533: Apply PEP384 to _csv module https://bugs.python.org/issue34533 closed by berker.peksag #34537: test_gdb fails with LC_ALL=C https://bugs.python.org/issue34537 closed by vstinner #34539: namedtuple's exec() throws segmentation fault https://bugs.python.org/issue34539 closed by xtreak #34548: IDLE: Make TextView use the configured theme colors https://bugs.python.org/issue34548 closed by taleinat #34575: Python 3.6 compilation fails on AppVeyor: libeay.lib was creat https://bugs.python.org/issue34575 closed by vstinner #34582: VSTS builds should use jobs, pools, and test results https://bugs.python.org/issue34582 closed by steve.dower #34609: Importing certain modules while debugging raises an exception https://bugs.python.org/issue34609 closed by terry.reedy #34610: Incorrect iteration of Manager.dict() method of the multiproce https://bugs.python.org/issue34610 closed by serhiy.storchaka #34659: Inconsistency between functools.reduce & itertools.accumulate https://bugs.python.org/issue34659 closed by lisroach #34683: Caret positioned wrong for SyntaxError reported by ast.c https://bugs.python.org/issue34683 closed by gvanrossum #34687: asyncio: is it time to make ProactorEventLoop as the default e https://bugs.python.org/issue34687 closed by vstinner #34728: deprecate *loop* argument for asyncio.sleep https://bugs.python.org/issue34728 closed by willingc #34734: Azure linux buildbot failure https://bugs.python.org/issue34734 closed by xtreak #34736: Confusing base64.b64decode output https://bugs.python.org/issue34736 closed by taleinat #34744: New %(flag)s format specifier for argparse.add_argument help s https://bugs.python.org/issue34744 closed by paul.j3 #34759: Possible regression in ssl module in 3.7.1 and master https://bugs.python.org/issue34759 closed by njs #34760: Regression in abc in combination with passing a function to is https://bugs.python.org/issue34760 closed by levkivskyi #34761: str(super()) != super().__str__() https://bugs.python.org/issue34761 closed by eric.snow #34762: Change contextvars C API to use PyObject https://bugs.python.org/issue34762 closed by yselivanov #34770: pyshellext.cpp: Possible null pointer dereference https://bugs.python.org/issue34770 closed by xiang.zhang #34772: Python will suddenly not plot https://bugs.python.org/issue34772 closed by ammar2 #34777: urllib.request accepts anything as a header parameter for some https://bugs.python.org/issue34777 closed by xtreak #34783: [3.7] segmentation-fault/core dump when try to run non-existin https://bugs.python.org/issue34783 closed by vstinner #34786: ProcessPoolExecutor documentation reports wrong exception bein https://bugs.python.org/issue34786 closed by xiang.zhang #34787: imghdr raise TypeError for PNG https://bugs.python.org/issue34787 closed by christian.heimes #34802: asyncio.iscoroutine() documentation is wrong https://bugs.python.org/issue34802 closed by yselivanov #34803: argparse int type does not accept scientific notation https://bugs.python.org/issue34803 closed by paul.j3 #34808: bytes[0] != bytes[0:1] https://bugs.python.org/issue34808 closed by ucyo #34809: On MacOSX with 3.7 python getting "Symbol not found: _PyString https://bugs.python.org/issue34809 closed by zach.ware #34813: child process disappears when removing a print statement after https://bugs.python.org/issue34813 closed by calimeroteknik #34815: Change Py_Ellipse __str__ behavior. https://bugs.python.org/issue34815 closed by photofone #34819: Executor.map and as_completed timeouts are able to deviate https://bugs.python.org/issue34819 closed by pitrou #34827: Make argparse.NameSpace iterable https://bugs.python.org/issue34827 closed by serhiy.storchaka #34830: functools.partial is weak referncable https://bugs.python.org/issue34830 closed by mdk #1529353: Squeezer - squeeze large output in IDLE's shell https://bugs.python.org/issue1529353 closed by terry.reedy

1 0

Questions about signal handling.
by Eric Snow 28 Sep '18

28 Sep '18

Hi all, I've got a pretty good sense of how signal handling works in the runtime (i.e. via a dance with the eval loop), but still have some questions: 1. Why do we restrict calls to signal.signal() to the main thread? 2. Why must signal handlers run in the main thread? 3. Why does signal handling operate via the "pending calls" machinery and not distinctly? More details are below. My interest in the topic relates to improving in-process interpreter isolation. #1 & #2 ----------- Toward the top of signalmodule.c we find the following comment [1] (written in 1994): /* NOTES ON THE INTERACTION BETWEEN SIGNALS AND THREADS When threads are supported, we want the following semantics: - only the main thread can set a signal handler - any thread can get a signal handler - signals are only delivered to the main thread I.e. we don't support "synchronous signals" like SIGFPE (catching this doesn't make much sense in Python anyway) nor do we support signals as a means of inter-thread communication, since not all thread implementations support that (at least our thread library doesn't). We still have the problem that in some implementations signals generated by the keyboard (e.g. SIGINT) are delivered to all threads (e.g. SGI), while in others (e.g. Solaris) such signals are delivered to one random thread (an intermediate possibility would be to deliver it to the main thread -- POSIX?). For now, we have a working implementation that works in all three cases -- the handler ignores signals if getpid() isn't the same as in the main thread. XXX This is a hack. */ At the very top of the file we see another relevant comment: /* XXX Signals should be recorded per thread, now we have thread state. */ That one was written in 1997, right after PyThreadState was introduced. So is the constraint about the main thread just a historical artifact? If not, what would be an appropriate explanation for why signals must be strictly bound to the main thread? #3 ----- Regarding the use of Py_MakePendingCalls() for signal handling, I can imagine the history there. However, is there any reason signal handling couldn't be separated from the "pending calls" machinery at this point? As far as I can tell there is no longer any strong relationship between the two. -eric [1] https://github.com/python/cpython/blob/master/Modules/signalmodule.c#L71

11 28

What is the purpose of the _PyThreadState_Current symbol in Python 3?
by Gabriele 27 Sep '18

27 Sep '18

In trying to find the location of a valid instance of PyInterpreterState in the virtual memory of a running Python (3.6) application (using process_vm_read on Linux), I have noticed that I can only rely on _PyThreadState_Current.interp at the very beginning of the execution. If I try to attach to a running Python process, then _PythreadState_Current.interp doesn't seem to point to anything useful to derive the currently running threads and the frame stacks for each of them. This makes me wonder about the purpose of this symbol in the .dynsym section. Apart from a brute force approach for finding a valid PyInterpreterState, is there a more reliable approach for the version of Python that I'm targeting? Thanks, Gabriele

2 1

Postponed annotations break inspection of dataclasses
by David Hagen 27 Sep '18

27 Sep '18

The new postponed annotations have an unexpected interaction with dataclasses. Namely, you cannot get the type hints of any of the data classes methods. For example, I have some code that inspects the type parameters of a class's `__init__` method. (The real use case is to provide a default serializer for the class, but that is not important here.) ``` from dataclasses import dataclass from typing import get_type_hints class Foo: pass @dataclass class Bar: foo: Foo print(get_type_hints(Bar.__init__)) ``` In Python 3.6 and 3.7, this does what is expected; it prints `{'foo': <class '__main__.Foo'>, 'return': <class 'NoneType'>}`. However, if in Python 3.7, I add `from __future__ import annotations`, then this fails with an error: ``` NameError: name 'Foo' is not defined ``` I know why this is happening. The `__init__` method is defined in the `dataclasses` module which does not have the `Foo` object in its environment, and the `Foo` annotation is being passed to `dataclass` and attached to `__init__` as the string `"Foo"` rather than as the original object `Foo`, but `get_type_hints` for the new annotations only does a name lookup in the module where `__init__` is defined not where the annotation is defined. I know that the use of lambdas to implement PEP 563 was rejected for performance reasons. I could be wrong, but I think this was motivated by variable annotations because the lambda would have to be constructed each time the function body ran. I was wondering if I could motivate storing the annotations as lambdas in class bodies and function signatures, in which the environment is already being captured and is code that usually only runs once.

6 9

[RELEASE] Python 3.7.1rc1 and 3.6.7rc1 now available for testing
by Ned Deily 27 Sep '18

27 Sep '18

Python 3.7.1rc1 and 3.6.7rc1 are now available. 3.7.1rc1 is the release preview of the first maintenance release of Python 3.7, the latest feature release of Python. 3.6.7rc1 is the release preview of the next maintenance release of Python 3.6, the previous feature release of Python. Assuming no critical problems are found prior to 2018-10-06, no code changes are planned between these release candidates and the final releases. These release candidates are intended to give you the opportunity to test the new security and bug fixes in 3.7.1 and 3.6.7. We strongly encourage you to test your projects and report issues found to bugs.python.org as soon as possible. Please keep in mind that these are preview releases and, thus, their use is not recommended for production environments. You can find these releases and more information here: https://www.python.org/downloads/release/python-371rc1/ https://www.python.org/downloads/release/python-367rc1/ -- Ned Deily nad(a)python.org -- []

2 1

Bug in _portable_fseek on Windows in 2.7.x
by Myria 25 Sep '18

25 Sep '18

Sorry for mailing about a bug instead of putting in a bug tracker ticket. The bug tracker's login system just sits there for a minute then says "an error has occurred". This line of code is broken in Windows: https://github.com/python/cpython/blob/v2.7.15/Objects/fileobject.c#L721 _lseeki64 only modifies the kernel's seek position, not the cached position stored in stdio. This sometimes leads to problems where fgetpos does not return the correct file pointer. The solution is to not do that "SIZEOF_FPOS_T >= 8" code at all on Windows, and just do this instead (or make a new HAVE_FSEEKI64 macro): #elif defined(MS_WINDOWS) return _fseeki64(fp, offset, whence); 3.x is unaffected because it uses the Unix-like FD API in Windows instead of stdio, in which its usage of _lseeki64 is correct. Thanks, Melissa

1 0

Re: [Python-Dev] Documenting the private C API (was Re: Questions about signal handling.)
by Jeroen Demeyer 25 Sep '18

25 Sep '18

On 2018-09-25 16:01, Barry Warsaw wrote: > Maybe this is better off discussed in doc-sig but I think we need to consider documenting the private C API. Even the *public* C API is not fully documented. For example, none of the PyCFunction_... functions appears in the documentation.

2 1

Store startup modules as C structures for 20%+ startup speed improvement?
by Larry Hastings 24 Sep '18

24 Sep '18

What follows is the text of issue 34690: https://bugs.python.org/issue34690 The PR is here: https://github.com/python/cpython/pull/9320 I don't know if we should be discussing this here on python-dev, or on bpo, or on Zulip, or on the soon-to-be-created Discourse. But maybe we can talk about it somewhere! //arry/ ---- This patch was sent to me privately by Jeethu Rao at Facebook. It's a change they're working with internally to improve startup time. What I've been told by Carl Shapiro at Facebook is that we have their blessing to post it publicly / merge it / build upon it for CPython. Their patch was written for 3.6, I have massaged it to the point where it minimally works with 3.8. What the patch does: it takes all the Python modules that are loaded as part of interpreter startup and deserializes the marshalled .pyc file into precreated objects stored as static C data. You add this .C file to the Python build. Then there's a patch to Python itself (about 250 lines iirc) that teaches it to load modules from these data structures. I wrote a quick dumb test harness to compare this patch vs 3.8 stock. It runs a command line 500 times and uses time.perf_counter to time the process. On a fast quiescent laptop I observe a 21-22% improvement: cmdline: ['./python', '-c', 'pass'] 500 runs: sm38 average time 0.006302303705982922 best 0.006055746000129147 worst 0.00816565500008437 clean38 average time 0.007969956444008858 best 0.007829047999621253 worst 0.008812210000542109 improvement 0.20924239043734505 % cmdline: ['./python', '-c', 'import io'] 500 runs: sm38 average time 0.006297688038004708 best 0.005980765999993309 worst 0.0072462130010535475 clean38 average time 0.007996319670004595 best 0.0078091849991324125 worst 0.009175700999549008 improvement 0.21242667903482038 % The downside of the patch: for these modules it ignores the Python files on disk--it doesn't even stat them. If you add stat calls you lose half of the speed improvement. I believe they added a work-around, where you can set a flag (command-line? environment variable? I don't know, I didn't go looking for it) that tells Python "don't use the frozen modules" and it loads all those files from disk. I don't propose to merge the patch in its current state. I think it would need a lot of work both in terms of "doing things the way Python does it" as well as just code smell (the serializer is implemented in both C and Python and jumps back and forth, also the build process for the serialized modules is pretty tiresome). Is it worth working on?

16 32