[Python-Dev] unicode vs buffer (array) design issue can crash interpreter
M.-A. Lemburg
mal at egenix.com
Fri Mar 31 11:04:59 CEST 2006
Martin v. Löwis wrote:
> Neal Norwitz wrote:
>> See http://python.org/sf/1454485 for the gory details. Basically if
>> you create a unicode array (array.array('u')) and try to append an
>> 8-bit string (ie, not unicode), you can crash the interpreter.
>>
>> The problem is that the string is converted without question to a
>> unicode buffer. Within unicode, it assumes the data to be valid, but
>> this isn't necessarily the case. We wind up accessing an array with a
>> negative index and boom.
>
> There are several problems combined here, which might need discussion:
>
> - why does the 'u#' converter use the buffer interface if available?
> it should just support Unicode objects. The buffer object makes
> no promise that the buffer actually is meaningful UCS-2/UCS-4, so
> u# shouldn't guess that it is.
> (FWIW, it currently truncates the buffer size to the next-smaller
> multiple of sizeof(Py_UNICODE), and silently so)
>
> I think that part should just go: u# should be restricted to unicode
> objects.
'u#' is intended to match 's#' which also uses the buffer
interface. It expects the buffer returned by the object
to a be a Py_UNICODE* buffer, hence the calculation of the
length.
However, we already have 'es#' which is a lot safer to use
in this respect: you can explicity define the encoding you
want to see, e.g. 'unicode-internal' and the associated
codec also takes care of range checks, etc.
So, I'm +1 on restricting 'u#' to Unicode objects.
> - should Python guarantee that all characters in a Unicode object
> are between 0 and sys.maxunicode? Currently, it is possible to
> create Unicode strings with either negative or very large Py_UNICODE
> elements.
>
> - if the answer to the last question is no (i.e. if it is intentional
> that a unicode object can contain arbitrary Py_UNICODE values): should
> Python then guarantee that Py_UNICODE is an unsigned type?
Py_UNICODE must always be unsigned. The whole implementation
relies on this and has been designed with this in mind (see
PEP 100). AFAICT, the configure does check that Py_UNICODE
is always unsigned.
Regarding the permitted range of values, I think the necessary
overhead to check that all Py_UNICODE* array values are within
the currently permitted range would unnecessarily slow down
the implementation.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Mar 31 2006)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
More information about the Python-Dev
mailing list