[Python-Dev] A codecs nit

[M.-A. Lemburg]

Barry Warsaw wrote:
> On Wed, 2006-02-15 at 22:07 +0100, M.-A. Lemburg wrote:
> 
>> Those are not pseudo-encodings, they are regular codecs.
>>
>> It's a common misunderstanding that codecs are only seen as serving
>> the purpose of converting between Unicode and strings.
>>
>> The codec system is deliberately designed to be general enough
>> to also work with many other types, e.g. it is easily possible to
>> write a codec that convert between the hex literal sequence you
>> have above to a list of ordinals:
> 
> Slightly off-topic, but one thing that's always bothered me about the
> current codecs implementation is that str.encode() (and friends)
> implicitly treats its argument as module, and imports it, even if the
> module doesn't live in the encodings package.  That seems like a mistake
> to me (and a potential security problem if the import has side-effects).

It was a mistake, yes, and thanks for bringing this up.

Codec packages should implement and register their own
codec search functions.

> I don't know whether at the very least restricting the imports to the
> encodings package would make sense or would break things.
> 
>>>> import sys
>>>> sys.modules['smtplib']
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> KeyError: 'smtplib'
>>>> ''.encode('smtplib')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> LookupError: unknown encoding: smtplib
>>>> sys.modules['smtplib']
> <module 'smtplib' from '/usr/lib/python2.4/smtplib.pyc'>
> 
> I can't see any reason for allowing any randomly importable module to
> act like an encoding.

The encodings package search function will try to import
the module and then check the module signature. If the
module fails to export the codec registration API, then
it raises the LookupError you see above.

At the time, it was nice to be able to write codec
packages as Python packages and have them readily usable
by just putting the package on the sys.path.

This was a side-effect of the way the encodings search
function worked. The original design idea was to have
all 3rd party codecs register themselves with the
codec registry. However, this implies that the application
using the codecs would have to run the registration
code at least ones. Since the encodings package search
function provided a more convenient way, this was used
by most codec package programmers.

In Py 2.5 we'll change that. The encodings package search
function will only allow codecs in that package to be
imported. All other codec packages will have to provide
their own search function and register this with the
codecs registry.

The big question is: what to do about 2.3 and 2.4 - adding
the same patch will cause serious breakage, since popular
codec packages such as Tamito's Japanese package rely
on the existing behavior.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 18 2006)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::