Thu Mar 25 19:05:52 CET 2010

Larry Hastings wrote:
> 
> M.-A. Lemburg wrote:
>> Backporting PyCapsule is fine, but the changes you made to all
>> those PyCObject uses does not look backwards compatible.
>>
>> The C APIs exposed by the modules (e.g. the datetime module)
>> are used in lots of 3rd party extension modules and changing
>> them from PyCObject to PyCapsule is a major change in the
>> module API.
> 
> You're right, my changes aren't backwards compatible.  I thought it was
> reasonable for four reasons:
> 
> 1. The CObject API isn't safe.  It's easy to crash Python 2.6 in just a
> few lines by mixing and matching CObjects.  Switching Python to capsules
> prevents a class of exploits.  I've included a script at the bottom of
> this message that demonstrates three such crashes.  The script runs in
> Python 2 and 3, but 3.1 doesn't crash because it's using capsules.
> 
> 2. As I just mentioned, Python 3.1 already uses capsules everywhere
> instead of CObjects.  Since part of the purpose of Python 2.7 is to
> prepare developers for the to upgrade to 3.1, getting them to switch to
> capsules now is just one more way they are prepared.
> 
> 3. Because CObject is unsafe, I want to deprecate it in 2.7, and if we
> ever made a 2.8 I want to remove it completely.
> 
> 4. When Python publishes an API using a CObject, it describes the thing
> the CObject points to in a header file.  In nearly all cases that header
> file also provides a macro or inline function that does the importing
> work for you.  I changed those to use capsules too.  So if the
> third-party code uses the macro or inline function, all you need do is
> recompile it against 2.7 and it works fine.  Sadly I know of one
> exception: pyexpat.expat_CAPI.  The header file just describes the
> struct pointed to by the CObject, but callers
> 
> 
> I can suggest four ways to ameliorate the problem.
> 
> First, we could do as Antoine Pitrou suggests on the bug (issue 7992):
> wherever the CObject used to be published as a module attribute to
> expose an API, we could provide both a CObject and a capsule; internally
> Python would only use the capsules.  This would allow third-party
> libraries to run against 2.7 unchanged.  The major problem with this is
> that third-party libraries would still be vulnerable to the
> mix-and-match CObject crash.  A secondary, minor concern: obviously we'd
> store the CObject attribute with the existing name, and the capsule
> attribute would have to get some new name.  But in Python 3.1, these
> attributes already expose a capsule.  Therefore, people who convert to
> using the capsules now would have to convert again when moving to 3.1.
> 
> Second, we could make CObject internally support unpacking capsules.  If
> you gave a capsule to PyCObject_AsVoidPtr() it would unpack it and
> return the pointer within.  (We could probably also map the capsule
> "context" to the CObject "desc", if any of the Python use cases needed
> it.)  I wouldn't change anything else about CObjects; creating and using
> them would continue to work as normal.  This would also allow
> third-party libraries to run against Python 2.7 unchanged.  The only
> problem is that it's unsafe, as indeed allowing any use of
> PyCObject_AsVoidPtr() is unsafe.
> 
> Third, I've been pondering writing a set of preprocessor macros, shipped
> in their own header file distributed independently of Python and
> released to the public domain, that would make it easy to use either
> CObjects or capsules depending on what version of Python you were
> compiling against.  Obviously, using these macros would require a source
> code change in the third-party library.  But these macros would make it
> a five-minute change.  This could compliment the first or second
> approaches.
> 
> Fourth, we could back out of the changes to published APIs and convert
> them back to CObjects.  -1.
> 
> 
> Your thoughts?

More later, have to run. For now, I'm with Antoine on this one:

 * adding support for PyCapsules to 2.7 is fine.

 * exposing the various module C APIs as PyCapsules in addition to
   the existing PyCObject C APIS is fines as well and indeed a good
   idea, since it makes porting 2.7 applications to 3.1 easier.

 * adding more C APIs using PyCapsules in Python 2.7 is fine as well,
   e.g. for the decimal module.

 * removing the PyCObject C APIs is no option in Python 2.7.

Please remember that you're dealing with Python 2.7. Major backwards
incompatible changes are not allowed in that branch and you're pretty
much breaking all 3rd party modules that have just started using e.g
the datetime module C API.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Mar 25 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/