On Sat, Jun 1, 2019 at 12:29 AM Terry Reedy wrote:

We try not to. There is already too much work. ... That was removed from the PR before it was merged. ... That was merged. ... Another chunk was added to What's New.

Woot to all of these! :)

On 2019-05-31, Simon Cross wrote:

As the maintainer of Genshi, one the libraries affected by the CodeType and similar changes, I thought I could add a users perspective to the discussion: [...]

Thanks. I think this change to PyCode_New() could have been handled a bit better. Couldn't we introduce a new CPP define that enables the revised API of PyCode_New()? For 3.8 extensions, they would get the backwards compatible API (and a warning) unless they set the define. For 3.9, we would enable the new API by default. That gives 3rd party extensions one release cycle to catch up to the change. Perhaps something similar could be done for CodeType called from within Python code. In this case, it seems that introducing a new API like PyCode_NewEx() is not the way. However, just changing an API like that is not very friendly to 3rd party extensions, even if we don't claim it is a stable API. For me, change to PyCode_New() means that I can't test the 3.8.0b1 because the 3rd party extensions I rely on don't compile with it. Normally I try to test my application code with the latest alpha and beta releases. It would be great if we had a system that did CI testing with the top PyPI modules. E.g. pull the latest versions of the top 100 PyPI modules and test them with the latest CPython branch. With that, at least we would know what the fallout would be from some incompatible CPython change. Setting that system up would be a fair amounnt of work but I suspect the PSF could fund someone who puts together a plan to do it. Such a system would be even more useful if we start moving stuff out of stdlib into PyPI.

Neil Schemenauer schrieb am 08.06.19 um 22:46:

It would be great if we had a system that did CI testing with the top PyPI modules. E.g. pull the latest versions of the top 100 PyPI modules and test them with the latest CPython branch.

FWIW, travis-ci provides the latest CPython master builds (and some latest dev branches). We use them in Cython for our CI tests. One of the problems is that their images include some widely used libraries like NumPy, some of which in turn depend on Cython these days, so it happened once already that they failed to provide updated images because they were lacking a Cython version that worked with them, and we didn't notice that a change in Cython was needed because the CI builds were continuing to use an outdated CPython master version. :) Ah, circular dependencies… I think they fixed something about that, though. It wasn't a problem this time, at least. We also have the "Cython testbed", which we (irregularly) use before releases to check that we didn't break more than was broken before the release. https://travis-ci.org/cython-testbed It's pretty much what was asked for here, just for Cython, and it turns out to be a considerable amount of work to keep this from breaking arbitrarily for the included projects, even without changing something in Cython along the way. Thus, personally, I would prefer a decentralised CI approach, where interested/important projects test themselves against CPython master (which many of them do already), and have them report back when they notice an undermotivated breakage. Some projects do that with Cython (and CPython) already, and that works quite well so far and seems the least work for everyone. Stefan

Serhiy Storchaka schrieb am 01.06.19 um 09:02:

I have a related proposition. Yesterday I have reported two bugs (and Pablo quickly fixed them) related to handling positional-only arguments. These bugs were occurred due to subtle changing the meaning of co_argcount. When we make some existing parameters positional-only, we do not add new arguments, but mark existing parameters. But co_argcount now means the only number of positional-or-keyword parameters. Most code which used co_argcount needs now to be changed to use co_posonlyargcount+co_argcount.

I propose to make co_argcount meaning the number of positional parameters (i.e. positional-only + positional-or-keyword). This would remove the need of changing the code that uses co_argcount.

Sounds reasonable to me. The main distinction points are positional arguments vs. keyword arguments vs. local variables. Whether the positional ones are positional or positional-only is irrelevant in many cases.

PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or PyCode_NewEx() with different signature.

It's not a commonly used function, and it's easy for C code to adapt. I don't think it's worth adding a new function to the C-API here, compared to just changing the signature. Very few users would benefit, at the cost of added complexity. Stefan

Victor Stinner schrieb am 12.06.19 um 00:09:

So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice.

I think Cython's deal with regard to this is: """ If you use Cython, we will try hard to cover up the adaptations for upcoming (and existing) CPython versions for you. But you'll likely have to rerun Cython on your project before a new CPython major release comes out. """ That's obviously not ideal for projects that end up being unmaintained. But then again, you can't freeze time forever, and /any/ change to a dependency can end up being fatal to them. I personally think that rerunning Cython when a new CPython release comes out is an acceptable price to pay for a project. In the worst case, this can even be done by others, as you suggested as a common Fedora practice elsewhere, Victor. (To be fair, I'll have to add that new Cython versions are also part of those dependencies that might end up introducing incompatible changes and forcing your code to be adapted. The upcoming Cython 3.0 release will be such a case. However, given our compatibility record so far, I still consider Cython a very fair deal.) Stefan

Petr Viktorin wrote:

On 6/12/19 7:40 AM, Stefan Behnel wrote:

...

Victor Stinner schrieb am 12.06.19 um 00:09: So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice. I think Cython's deal with regard to this is:

""" If you use Cython, we will try hard to cover up the adaptations for upcoming (and existing) CPython versions for you. But you'll likely have to rerun Cython on your project before a new CPython major release comes out. """

That's obviously not ideal for projects that end up being unmaintained. But then again, you can't freeze time forever, and /any/ change to a dependency can end up being fatal to them.

I personally think that rerunning Cython when a new CPython release comes out is an acceptable price to pay for a project. In the worst case, this can even be done by others, as you suggested as a common Fedora practice elsewhere, Victor. I hope this is something that improvements in Python's packaging story (specifically, PEP 518) should help with. I see the current practice of including Cython's output in releases as a workaround for the fact that you can't (reasonably) specify Cython as a build dependency. Cython is a much lighter dependency than a C compiler -- though a less common one. When there's a reliable way to specify build-time dependencies, running Cython on each build will hopefully become the obvious way to do it.

Or is there something else blocking that future?

PEP 518 lets you specify Cython as a build dependency, but it's still up to _some_ build tool to specify that the project used Cython and then to know how to run Cython as appropriate. So if you're got setuptools working then listing setuptools, wheel, and cython as build dependencies _should_ work as you want.

Victor Stinner wrote:

Update.

Le ven. 31 mai 2019 à 10:49, Petr Viktorin

...

PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount".

Pablo proposed a PR to revert PyCode_New() API to Python 3.7 API: https://github.com/python/cpython/pull/13959

I dislike "PyCode_NewEx" name, it will become worse when another parameter will be added in the future. I prefer "With" naming. I prefer "PyCode_NewWithPosArgs()" name.

I agree if a new name is chosen (I haven't decided if the parameter expansion was wrong or not in practice or if this was just poorly communicated). -Brett

For types.CodeType constructor, well, I added CodeType.replace() to help projects to be prepared for next CodeType constructor change :-) But *I* am fine with breaking this constructor since very few projects create explicitly code objects.

...

Our policy for such changes seems to be fragmented tribal knowledge. I'm writing to check if my understanding is reasonable, so I can apply it and document it explicitly. The doc has been updated: https://bugs.python.org/issue36896

"If you instantiate any of these types, note that signatures may vary between Python versions."

...

There is a surprisingly large ecosystem of tools that create code objects. The expectation seems to be that these tools will need to be adapted for each minor version of Python. For PyCode_New() change, it seems like (almost?) only projects using Cython are affected, no?

To not require Cython when they are installed, many projects include C files generated by Cython in their tarball (and other released files).

So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice.

For this reason, I'm in favor of reverting PyCode_New() API and add a new function with the extra positional only parameter.

...

Also, the expected level of API stability should be documented in the docs. Note: code.h header is excluded from the stable API :-) The whole file is surrounded by "#ifndef Py_LIMITED_API".

Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Hi Antoine, In Python 3.4, I added "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)" to the public C API. Problem: we wanted to add a new "calloc" field to PyMemAllocator structure. I chose to rename the structure to PyMemAllocatorEx to ensure that all C extensions using this function get a compilation error, rather than leaving the new "calloc" field uninitialized. The function became: "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)" I'm not sure what will be the new structure name when we will add new fields (like malloc_aligned) :-( For PyCodeOptions: does it contain posonlyargcount? If not, how do you pass posonlyargcount. I'm not sure if you plan to handle the backward compatibility. Note: In the PEP 587, I added a private "_config_version" field to PyConfig structure to prepare the structure to future extension, but keep backward compatibility. The structure will be versionned ;-) Victor Le jeu. 13 juin 2019 à 01:08, Antoine Pitrou a écrit :

On Wed, 12 Jun 2019 00:09:10 +0200 Victor Stinner wrote:

...

Update.

Le ven. 31 mai 2019 à 10:49, Petr Viktorin a écrit :

...

PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount".

Pablo proposed a PR to revert PyCode_New() API to Python 3.7 API: https://github.com/python/cpython/pull/13959

I dislike "PyCode_NewEx" name, it will become worse when another parameter will be added in the future. I prefer "With" naming. I prefer "PyCode_NewWithPosArgs()" name.

Or perhaps the alternative is a struct-based approach:

typedef struct { // ... } PyCodeOptions;

PyObject* PyCode_New(PyCodeOptions opts);

?

Regards

Antoine.

_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/6K2R62CV...

-- Night gathers, and now my watch begins. It shall not end until my death.