Larry Hastings wrote:

M.-A. Lemburg wrote:

...

Backporting PyCapsule is fine, but the changes you made to all those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module) are used in lots of 3rd party extension modules and changing them from PyCObject to PyCapsule is a major change in the module API.

You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons:

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely.

4. When Python publishes an API using a CObject, it describes the thing the CObject points to in a header file. In nearly all cases that header file also provides a macro or inline function that does the importing work for you. I changed those to use capsules too. So if the third-party code uses the macro or inline function, all you need do is recompile it against 2.7 and it works fine. Sadly I know of one exception: pyexpat.expat_CAPI. The header file just describes the struct pointed to by the CObject, but callers

I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. This would allow third-party libraries to run against 2.7 unchanged. The major problem with this is that third-party libraries would still be vulnerable to the mix-and-match CObject crash. A secondary, minor concern: obviously we'd store the CObject attribute with the existing name, and the capsule attribute would have to get some new name. But in Python 3.1, these attributes already expose a capsule. Therefore, people who convert to using the capsules now would have to convert again when moving to 3.1.

Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and return the pointer within. (We could probably also map the capsule "context" to the CObject "desc", if any of the Python use cases needed it.) I wouldn't change anything else about CObjects; creating and using them would continue to work as normal. This would also allow third-party libraries to run against Python 2.7 unchanged. The only problem is that it's unsafe, as indeed allowing any use of PyCObject_AsVoidPtr() is unsafe.

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1.

Your thoughts?

More later, have to run. For now, I'm with Antoine on this one: * adding support for PyCapsules to 2.7 is fine. * exposing the various module C APIs as PyCapsules in addition to the existing PyCObject C APIS is fines as well and indeed a good idea, since it makes porting 2.7 applications to 3.1 easier. * adding more C APIs using PyCapsules in Python 2.7 is fine as well, e.g. for the decimal module. * removing the PyCObject C APIs is no option in Python 2.7. Please remember that you're dealing with Python 2.7. Major backwards incompatible changes are not allowed in that branch and you're pretty much breaking all 3rd party modules that have just started using e.g the datetime module C API. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 25 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Hello Larry,

You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons:

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

Ok, but this is IMHO not a good enough reason to break the API. No prior announcement (prior to 2.7) has been made that PyCObject APIs would be replaced by their PyCapsule equivalents. You must leave people time to adapt.

2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared.

Not /forcing them/ to switch, though. If you agree to put a bit more work into it, and make the PyCObject APIs coexist with the corresponding PyCapsule APIs in 2.7, then I think it would be fine. The point where we force people to switch their APIs is the 2.x -> 3.x transition. That's the whole point of breaking compatibility at one defined point. If we force them to switch when migrating to 2.7 instead, it becomes messy and quite unfriendly to them.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely.

I don't think there is a point in removing it in 2.8. If 2.8 ever exists, its purpose will be to serve users who don't /want/ to switch to 3.x yet, so breaking compatibility for them kind of ruins the purpose.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. [...]

Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and return the pointer within. [...]

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1.

I think solution #1 would be the best one. #2 looks too complicated. #3 is not much friendlier than the current compatibility breakage. #4 is the necessary fallback if you don't want to implement #1. Regards Antoine.

Larry Hastings wrote:

M.-A. Lemburg wrote:

...

Backporting PyCapsule is fine, but the changes you made to all those PyCObject uses does not look backwards compatible.

The C APIs exposed by the modules (e.g. the datetime module) are used in lots of 3rd party extension modules and changing them from PyCObject to PyCapsule is a major change in the module API.

You're right, my changes aren't backwards compatible. I thought it was reasonable for four reasons:

Just as reminder of the process we have in place for such changes: Please discuss any major breakage on python-dev before checking in the patch.

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

That's good, but then again: deliberate wrong use of APIs will always cause crashes and at least I don't know of any report about PyCObjects posing a problem in all their years of existence.

2. As I just mentioned, Python 3.1 already uses capsules everywhere instead of CObjects. Since part of the purpose of Python 2.7 is to prepare developers for the to upgrade to 3.1, getting them to switch to capsules now is just one more way they are prepared.

Sure, but forcing them is not a good idea, even less so if you can easily expose both a PyCObject and PyCapsule interface to the same C API.

3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we ever made a 2.8 I want to remove it completely.

Please remember that PyCObjects are not only used internally in CPython, but also in other 3rd party modules to expose C APIs and those will generally have to support more than just the latest Python release. If you deprecate those C APIs, the process will at least have to cover one more release, i.e. run through the whole deprecation process: 1. pending deprecation (2.7) 2. deprecation (2.8) 3. removal (2.9) I think it's better to add a -3 warning when using PyCObjects in 2.7.

4. When Python publishes an API using a CObject, it describes the thing the CObject points to in a header file. In nearly all cases that header file also provides a macro or inline function that does the importing work for you. I changed those to use capsules too. So if the third-party code uses the macro or inline function, all you need do is recompile it against 2.7 and it works fine. Sadly I know of one exception: pyexpat.expat_CAPI. The header file just describes the struct pointed to by the CObject, but callers

I know about those macros... I introduced that idea with mxDateTime and then added the same logic in CPython in a couple of places a long while ago, e.g. socketmodule.h. IIRC, Jim Fulton added PyCObjects for exchanging module C APIs a few years before that. Later on PyCObject_Import() was added to simplify the C API import a bit. A recompile is certainly a possibility to have 3rd party modules switch to the capsule interfaces of the stdlib modules, but we should still be more careful about this. Wouldn't it be possible to have PyCObject_AsVoidPtr() et al. work with PyCapsules as well ? After all, the caller is only interested in the pointer and doesn't really care about what object was used to wrap it.

I can suggest four ways to ameliorate the problem.

First, we could do as Antoine Pitrou suggests on the bug (issue 7992): wherever the CObject used to be published as a module attribute to expose an API, we could provide both a CObject and a capsule; internally Python would only use the capsules. This would allow third-party libraries to run against 2.7 unchanged. The major problem with this is that third-party libraries would still be vulnerable to the mix-and-match CObject crash.

True, but we've been happy with this vulnerability for years, just as we've been happy with the fact that it's easy to crash the VM by passing hand-crafted byte-code to it, or using ctypes to call an OS function with the wrong parameters, etc. Like I said: there are many ways to deliberately crash Python. We don't have a concept of interface signatures in Python, it's mostly based on trust.

A secondary, minor concern: obviously we'd store the CObject attribute with the existing name, and the capsule attribute would have to get some new name. But in Python 3.1, these attributes already expose a capsule. Therefore, people who convert to using the capsules now would have to convert again when moving to 3.1.

This should be manageable with some aliasing.

Second, we could make CObject internally support unpacking capsules. If you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and return the pointer within. (We could probably also map the capsule "context" to the CObject "desc", if any of the Python use cases needed it.) I wouldn't change anything else about CObjects; creating and using them would continue to work as normal. This would also allow third-party libraries to run against Python 2.7 unchanged. The only problem is that it's unsafe, as indeed allowing any use of PyCObject_AsVoidPtr() is unsafe.

That's a good idea. Yes, playing with fire is unsafe, but fire is useful in a lot of places as well :-)

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

I'm not sure whether that would worth the effort. 3rd party modules for Python 2.x are likely not going to use them, since they typically have to support more than just Python 2.7. E.g. look at Plone - they are still running on Python 2.4 and use lots of 3rd party modules. It took Zope years to switch from 2.4 to 2.6. You have the same situation with other large systems, whether OSS or not. If a required 3rd party module would suddenly only support Python 2.7, developers would either have to find a replacement that (still) works for them (rather unlikely), or try to port their whole app to 2.7, which often enough is not easily possible due to other restrictions. This aspect is often not considered when discussing such backwards incompatible changes - at least not in the early stages. Fortunately, we have so far - and most of the time - found a way to keep everyone happy.

Fourth, we could back out of the changes to published APIs and convert them back to CObjects. -1.

See above. That's still an option as well, but I agree that either making PyCObjects compatible with PyCapsules (via changes to the PyCObject functions) or having them exist side-by-side is a better option. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 26 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Well, it's been a couple of days, and nobody has stepped forward to speak in favor of my aggressive PyCapsule schedule. So I will bow to the wishes of those who have spoken up, and scale it back. Specifically, I propose writing a patch that will: * allow the CObject API to (unsafely) dereference a capsule * add support for the new files added by capsule to the Visual Studio builds (fixes an unrelated second complaint about my checkin) * add a -3 warning for calls to CObject * add a PendingDeprecation warning to CObject I've gotten in the habit of doing code reviews at work, and I now greatly prefer working that way. Can I get a volunteer to review the patch when it's ready? Please email me directly. Thanks! M.-A. Lemburg wrote:

Just as reminder of the process we have in place for such changes: Please discuss any major breakage on python-dev before checking in the patch.

I'm aware this is a good idea. I simply didn't consider this a major breakage. Recompiling against the 2.7 header files fixes it for everybody. (Except external users of pyexpat, if any exist. Google doesn't show any, though this is not proof that they don't exist.) If you suggest that any breakage with previous versions is worth mentioning, no matter how small, then I'll remember that in the future. Certainly the Python community has had many thrilling and dynamic conversations over minutae, so I guess it wouldn't be that surprising if this were true.

...

1. The CObject API isn't safe. It's easy to crash Python 2.6 in just a few lines by mixing and matching CObjects. Switching Python to capsules prevents a class of exploits. I've included a script at the bottom of this message that demonstrates three such crashes. The script runs in Python 2 and 3, but 3.1 doesn't crash because it's using capsules.

That's good, but then again: deliberate wrong use of APIs will always cause crashes and at least I don't know of any report about PyCObjects posing a problem in all their years of existence.

[...] we've been happy with this vulnerability for years, just as we've been happy with the fact that it's easy to crash the VM by passing hand-crafted byte-code to it, or using ctypes to call an OS function with the wrong parameters, etc.

I'm surprised you find that acceptable. I thought the community was in agreement: our goal is to make the interpreter uncrashable from pure Python. I would describe ctypes as "impure" Python. If you wanted to allow untrusted developers to run code through a Python interpreter, you'd harden it: take away ctypes, and the ability to feed in arbitrary bytecode, and reduce the stack limit. You probably wouldn't take away "datetime", "cStringIO", and "cPickle"--yet I can crash 2.6 using those.

...

Third, I've been pondering writing a set of preprocessor macros, shipped in their own header file distributed independently of Python and released to the public domain, that would make it easy to use either CObjects or capsules depending on what version of Python you were compiling against. Obviously, using these macros would require a source code change in the third-party library. But these macros would make it a five-minute change. This could compliment the first or second approaches.

I'm not sure whether that would worth the effort. 3rd party modules for Python 2.x are likely not going to use them, since they typically have to support more than just Python 2.7.

The point of these macros would be to support older versions of Python. The macros would boil down to either PyCapsule or CObject calls, depending on what version of Python you were compiling against. For Python 2.6 and before it would alway use CObjects. One set of macros would switch to PyCapsule when building against 2.7; another would use CObject permanently in 2.x. (Or maybe it should switch in 2.8? 2.9? But now we're in the realm of speculative fiction.) I'd be happy to write these if there was demand, but my sneaking suspicion is that nobody would use them. If you're genuinely interested in these macros please email me privately. /larry/