[Python-Dev] New PEP: Using ssize_t as the index type

[M.-A. Lemburg]

Martin v. Löwis wrote:
> M.-A. Lemburg wrote:
>> I don't see a good solution for these other than introducing
>> a set of new APIs (and maybe doing some macro magic as Martin
>> did for PyArg_ParseTuple()). Due to the fact that changes in
>> the types of output parameters require changes in the
>> extension variable type layout itself, they introduce a large
>> number of type changes in the extension and make writing
>> backwards compatible extensions harder than necessary.
> 
> That's not true. It is very easy to write extensions that
> receive such values and are still backwards-compatible.
> 
> Suppose you had
> 
>   int pos;
>   PyObject *k, *v;
> 
>   PyDict_Next(dict, &pos, &k, &v);
> 
> You just change this to
> 
>   /* beginning of file */
>   #ifdef Py_HEX_VERSION < 2.5
>   typedef int Py_ssize_t;
>   #endif
> 
>   /* later */
>   Py_ssize_t pos;
>   PyObject *k, *v;
> 
>   PyDict_Next(dict, &pos, &k, &v);
> 
> That's it!

If it were this easy, I wouldn't have objections. But it's
not.

The variables you use with these APIs tend to propagate
through the extension, you use them in other calls,
make assignments, etc.

If you implement extension types, you end up having to
convert all the length related struct variables to
Py_ssize_t.

If you're writing against 3rd party APIs which don't
use ssize_t or size_t, you have to convert Py_ssize_t
to int where necessary.

All this is fine, but it's also a lot of work which
can be made easier. Recompiling an extension is well
within range of many Python users, manually checking,
fixing and porting it to the new API is certainly not.

>> Furthermore, all extensions for Python 2.4 would have to be
>> ported to the new Python API and porting is not going to
>> be a simple recompile, but will require C skills.
> 
> Not all extensions. Only those that call functions that expect
> int* output parameters - which is fairly uncommon.

The set of functions that will require Py_ssize_t
is getting larger in your branch which is why I started
this discussion.

In the first checkin you only had
the rarely used slice APIs converted. In the meantime
the buffer APIs, the Unicode APIs and others have
been added to the list.

These APIs are used a lot more often than the slice
APIs.

I'm not saying that it's a bad idea to adjust these
to Py_ssize_t, it's just the backwards incompatible
way this is done which bothers me.

>> Martin, please add the above points to the PEP. I'd also
>> like to see it published, because it's hard to track a PEP
>> in the mailing
> 
> It's very difficult to get a PEP number assigned. I wrote
> peps at python.org with no response.

Would it be possible to host the PEP in the python.org
wiki or maybe in the sandbox on svn.python.org ?

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Jan 10 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::