Mailman 3 July 2018 - Python-Dev

New _Py_InitializeFromConfig() function (PEP 432)
by Victor Stinner 04 Aug '18

04 Aug '18

Hi, I finished my work on the _PyCoreConfig structure: it's a C structure in Include/pystate.h which has many fields used to configure Python initialization. In Python 3.6 and older, these parameters were scatted around the code, and it was hard to get an exhaustive list of it. This work is linked to the Nick Coghlan's PEP 432 "Restructuring the CPython startup sequence": https://www.python.org/dev/peps/pep-0432/ Right now, the new API is still private. Nick Coghlan splitted the initialization in two parts: "core" and "main". I'm not sure that this split is needed. We should see what to do, but it would be nice to make the _PyCoreConfig API public! IMHO it's way better than the old way to configuration Python initialization. -- It is now possible to only use _PyCoreConfig to initialize Python: it overrides old ways to configure Python like environment variables (ex: PYTHONPATH), global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()). I added tests to test_embed on the different ways to configure Python initialization: * environment variables (ex: PYTHONPATH) * global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()) * _PyCoreConfig I found and fixed many issues when writing these tests :-) Reading the current configuration, _PyCoreConfig_Read(), no longer changes the configuration. Now the code to read the configuration and the code to apply the configuration is properly separated. The work is not fully complete, there are a few remaining corner cases and some parameters (ex: Py_FrozenFlag) which cannot be set by _PyCoreConfig yet. My latest issue used to work on this API: https://bugs.python.org/issue34170 I had to refactor a lot of code to implement all of that. -- The problem is that Python 3.7 got the half-baked implementation, and it caused issues: * Calling Py_Main() after Py_Initialize() fails with a fatal error on Python 3.7.0 https://bugs.python.org/issue34008 * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize() https://bugs.python.org/issue34247 I fixed the first issue, I'm now working on the second one to see how it can be fixed. Other option would be to backport the code from master to the 3.7 branch, since the code in master has a way better design. But it requires to backport a lot of changes. I'm not sure yet what is the best option. Victor

5 13

Finding Guido's replacement
by Chris Angelico 03 Aug '18

03 Aug '18

Guido's term as Benevolent Dictator For Life has been a long one, but in the wake of his resignation, we have an opportunity to correct some fundamental flaws in the system. Among them: * Guido lacks patience, as evidenced by the brevity of his acceptance posts. See https://mail.python.org/pipermail/python-dev/2017-December/151038.html and https://mail.python.org/pipermail/python-dev/2011-November/114545.html and particularly https://mail.python.org/pipermail/python-dev/2016-May/144646.html where Guido specifically cites his own lack of patience. * Lately, all Guido's actions have been to benefit his employer, not the Common Pythonista. We have proof of this from reliable reporting sources such as Twitter and social media. * Finally, "For Life" is far too long. We need to change our rulers periodically. I propose a new way to appoint a project head. All candidates shall be flown to an island owned by the Python Secret Underground (which emphatically does NOT exist, but an island that would be owned by it if it did), whereupon they parachute down, search for guns, and proceed to fight each other until only one is left alive. The survivor shall be treated to a chicken dinner and proclaimed Patient, Understanding, Benevolent Governor, a title which shall be retained for one fortnight, after which we repeat the exercise. If this plan meets with broad approval, I shall write up PEP 3401, in honour of the prior art in PEP 401. ChrisA

10 17

Re: [Python-Dev] Error message for wrong number of arguments
by Jeroen Demeyer 03 Aug '18

03 Aug '18

On 2018-07-30 20:22, Chris Barker wrote: > is it possible for the interpreter to know when this error is > generated that this is a bound method expecting a "self", rather than an > arbitrary function with n parameters? That would be quite hard. The error message is generated by the underlying function. At that point, the information of how it was called (as bound method or not) is already gone. Jeroen.

4 4

Accessing mailing list archives
by Bob Purvy 31 Jul '18

31 Jul '18

hi all, I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. Alternatively, they're maybe *already* in a database, and I just need API access to do the querying? Can someone help me out? Bob

4 4

Re: [Python-Dev] Let's change to C API!
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 12:56, Victor Stinner wrote: > We try to make CPython build as simple as possible. I'm quite sure > that Cython rely on the stdlib. It does rely on modules like "re" and "functools". > Would depending on Cython open a > chicken-and-egg issue? Yes, that's a problem but it's not unsolvable. For example, we could use the PEP 399 pure Python modules for running Cython. Or we could keep certain "core" C modules (which are used by Cython) implemented directly in C. Note that Cython is not all-or-nothing: it is easy to mix pure Python modules, Cython modules and pure C modules. You can also combine pure C code and Cython code in the same module. Anyway, I know that this is probably not going to happen, but I just wanted to bring it up in case people would find it a great idea. But maybe not many CPython core developers actually know and use Cython? > I would be nice to be able to use something to "generate" C > extensions, maybe even from pure Python code. Cython has a "pure Python mode" which does exactly that. There are several ways to include typing information, to ensure that a module remains Python-compatible but can be compiled by Cython in an optimized way. Jeroen.

4 3

Clarifying Cygwin support in CPython
by Erik Bray 31 Jul '18

31 Jul '18

Hi folks, As some people here know I've been working off and on for a while to improve CPython's support of Cygwin. I'm motivated in part by a need to have software working on Python 3.x on Cygwin for the foreseeable future, preferably with minimal graft. (As an incidental side-effect Python's test suite--especially of system-level functionality--serves as an interesting test suite for Cygwin itself too.) This is partly what motivated PEP 539 [1], although that PEP had the advantage of benefiting other POSIX-compatible platforms as well (and in fact was fixing an aspect of CPython that made it unfriendly to supporting other platforms). As far as I can tell, the first commit to Python to add any kind of support for Cygwin was made by Guido (committing a contributed patch) back in 1999 [2]. Since then, bits and pieces have been added for Cygwin's benefit over time, with varying degrees of impact in terms of #ifdefs and the like (for the most part Cygwin does not require *much* in the way of special support, but it does have some differences from a "normal" POSIX-compliant platform, such as the possibility for case-insensitive filesystems and executables that end in .exe). I don't know whether it's ever been "officially supported" but someone with a longer memory of the project can comment on that. I'm not sure if it was discussed at all or not in the context of PEP 11. I have personally put in a fair amount of effort already in either fixing issues on Cygwin (many of these issues also impact MinGW), or more often than not fixing issues in the CPython test suite on Cygwin--these are mostly tests that are broken due to invalid assumptions about the platform (for example, that there is always a "root" user with uid=0; this is not the case on Cygwin). In other cases some tests need to be skipped or worked around due to platform-specific bugs, and Cygwin is hardly the only case of this in the test suite. I also have an experimental AppVeyor configuration for running the tests on Cygwin [3], as well as an experimental buildbot (not available on the internet, but working). These currently rely on a custom branch that includes fixes needed for the test suite to run to completion without crashing or hanging (e.g. https://bugs.python.org/issue31885). It would be nice to add this as an official buildbot, but I'm not sure if it makes sense to do that until it's "green", or at least not crashing. I have several other patches to the tests toward this goal, and am currently down to ~22 tests failing. Before I do any more work on this, however, it would be best to once and for all clarify the support for Cygwin in CPython, as it has never been "officially supported" nor unsupported--this way we can avoid having this discussion every time a patch related to Cygwin comes up. I could provide some arguments for why I believe Cygwin should supported, but before this gets too long I'd just like to float the idea of having the discussion in the first place. It's also not exactly clear to me how to meet the standards in PEP 11 for supporting a platform--in particular it's not clear when a buildbot is considered "stable", or how to achieve that without getting necessary fixes merged into the main branch in the first place. Thanks, Erik [1] https://www.python.org/dev/peps/pep-0539/ [2] https://github.com/python/cpython/commit/717d1fdf2acbef5e6b47d9b4dcf48ef182… [3] https://ci.appveyor.com/project/embray/cpython

3 6

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 09:36, INADA Naoki wrote: > I think PEP 580 is understandable only for people who tried to implement > method objects. Is this really a problem? Do we expect that all Python developers can understand all PEPs, especially on a technical subject like this? To give a different example, I would say that PEP 567 is also quite technical and not understandable by people who don't care about about context variables. If PEP 580 is accepted, we can make it very clear in the documentation that this is only meant for implementing fast function/method classes and that ordinary "extension writers" can safely skip that part. For example, you write > They should learn PyCCallDef and CCALL_* flags in addition > to PyMethodDef and METH_*. but that's not true: they can easily NOT learn those flags, just like they do NOT need to learn about context variables if they don't need them. >> I would like to stress that PEP 580 was designed for maximum >> performance, both today and for future extensions (such as calling with >> native C types). >> > > I don't know what the word *stress* mean here. (Sorry, I'm not good at English > enough for such hard discussion). > But I want to see PoC of real benefit of PEP 580, as I said above. "to stress" = to draw attention to, to make it clear that So, PEP 580 is meant to keep all existing optimizations for functions/methods. It can also be extended in the future (for example, to support direct C calling) by just adding extra flags and structure fields to PyCCallDef. > Hm, My point was providing easy and simple way to support FASTCALL > in callable object like functools.partial or functools.lru_cache. That can be done easily with only PEP 580. Jeroen.

4 4

Re: [Python-Dev] Let's change to C API!
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 15:34, Victor Stinner wrote: > But I never used Cython nor cffi, so I'm not sure which one is the > most appropriate depending on the use case. Cython is a build-time tool, while cffi is a run-time tool. But Cython does a lot more than just FFI. It is a Python->C compiler which can be used for FFI but also for many other things. > A major "rewrite" of such large code base is > very difficult since people want to push new things in parallel. Or is > it maybe possible to do it incrementally? Yes, that's not a problem: you can easily mix pure Python code, Cython code and C code. I think that this kind of mixing is an important part in Cython's philosophy: for stuff where you don't care about performance: use Python. For most stuff where you do care: use Cython. For very specialized code which cannot easily be translated to Cython: use C. Jeroen.

1 0

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 12:10, INADA Naoki wrote: > Surely, they should understand they must use CCALL_* flags instead of > METH_* flags when implementing fast-callable object. Yes indeed. But implementing a fast-callable object is quite specialized, not something that ordinary extension writers would care about. And if they don't care about performance, tp_call remains supported. More generally: with PEP 580, everything from the documented C API remains supported. So people can write extensions exactly as before. They only need to care about PEP 580 if they want to use the new features that PEP 580 adds (or if they used undocumented internals). Jeroen.

2 1

Re: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580
by Jeroen Demeyer 31 Jul '18

31 Jul '18

On 2018-07-31 12:10, INADA Naoki wrote: > After spent several days to read PEP 580 and your implementation, I think > I can implement it. I think it's not easy, but it's not impossible too. The signature of "extended_call_ptr" in PEP 576 is almost the same as the signature of a CCALL_FUNCARG|CCALL_FASTCALL|CCALL_KEYWORDS function in PEP 580 (the only difference is a "self" argument which can be ignored if you don't need it). So, if you can implement it using PEP 576, it's not a big step to implement it using PEP 580.

1 0