[Python-Dev] Why can't I encode/decode base64 without importing a module?

Glenn Linderman v+python at g.nevcal.com
Wed Apr 24 11:18:06 CEST 2013

On 4/24/2013 1:22 AM, M.-A. Lemburg wrote:
> On 23.04.2013 19:24, Guido van Rossum wrote:
>> On Tue, Apr 23, 2013 at 9:04 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> On 23.04.2013 17:47, Guido van Rossum wrote:
>>>> On Tue, Apr 23, 2013 at 8:22 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>>> Just as reminder: we have the general purpose
>>>>> encode()/decode() functions in the codecs module:
>>>>> import codecs
>>>>> r13 = codecs.encode('hello world', 'rot-13')
>>>>> These interface directly to the codec interfaces, without
>>>>> enforcing type restrictions. The codec defines the supported
>>>>> input and output types.
>>>> As an implementation mechanism I see nothing wrong with this. I hope
>>>> the codecs module lets you introspect the input and output types of a
>>>> codec given by name?
>>> At the moment there is no standard interface to access supported
>>> input and output types... but then: regular Python functions or
>>> methods also don't provide such functionality, so no surprise
>>> there ;-)
>> Not quite the same though. Each function has its own unique behavior.
>> But codecs support a standard interface, *except* that the input and
>> output types sometimes vary.
> The codec system itself
>>> It's mostly a matter of specifying the supported type
>>> combinations in the codec documentation.
>>> BTW: What would be a use case where you'd want to
>>> programmatically access such information before calling
>>> the codec ?
>> As you know, in Python 3, most code working with bytes doesn't also
>> work with strings, and vice versa (except for a few cases where we've
>> gone out of our way to write polymorphic code -- but users rarely do
>> so, and any time you use a string or bytes literal you basically limit
>> yourself to that type).
>> Suppose I write a command-line utility that reads a file, runs it
>> through a codec, and writes the result to another file. Suppose the
>> name of the codec is a command-line argument (as well as the
>> filenames). I need to know whether to open the files in text or binary
>> mode based on the name of the codec.
> Ok, so you need to know which codecs your tool can support and
> which of those need text input and which bytes input.
> I've been thinking about this some more: I think that type
> information alone is not flexible enough to cover such
> use cases.

Maybe MIME type and encoding would be sufficient type information, but 
probably not str vs. bytes.

> In your use case you'd want to only permit use of a certain
> set of codecs, not simply all of them, since some might
> not implement what you actually want to achieve with the tool,
> e.g. a user might have installed a codec set that adds
> support for reading and writing image data, but your
> intended use was to only support text data.

MIME type supports this sort of concept, with the two-level hierarchy of 
naming the type... text/xml text/plain image/jpeg

> So what we need is a way to allow the codecs to say e.g.
> "I work on text", "I support encoding bytes and text",
> "I encode to bytes", "I'm reversible", "I transform
> input data", "I support bytes and text, and will create
> same type output", "I work on image data", "I work on
> X509 certificates", "I work on XML data", etc.

Guess what I think you are re-inventing here....
Nope, guess again....
Yep, MIME types _plus_ encodings.

> In other words, we need a form of tagging system, with a
> set of standard tags that each codec can publish and
> which also allows non-standard tags (which can then at
> some point be made standard, if there's agreement on them).

Hmm.  Sounds just like the registry for, um, you guessed it: MIME types.

> Given a codec name you could then ask the codec registry for
> the codec tags and verify that the chosen codec handles
> text data, needs bytes or text encoding input and
> creates bytes as encoding output. If the registry returns
> codec tags that don't include the "I work on text" tag,
> the tool could then raise an error.

For just doing text encoding transformations,  text/plain would work as 
a MIME type, and the encodings of interest for the encodings.

Seems like "str" always means "Unicode" but the MIME type can vary; 
"bytes" might mean encoded text, and the MIME type can also vary.

For non-textual transformations, "encoding" might mean Base 64, BinHex, 
or other such representations... but those can also be applied to text, 
so it might be a 3rd dimension, or it might just be a list of encodings 
rather than a single encoding.

Compression could be another dimension, or perhaps another encoding.

But really, then, a transformation needs to be a list of steps; a codec 
can sign up to perform one or more of the steps, a sequence of codecs 
would have to be found, capable of performing a subsequence of the 
steps, and then run in the appropriate order.

This all sounds so general, that probably the Python compiler could be 
implemented as a codec :)  Or any compiler. Probably a web server could 
be implemented as a codec too :)  Well, maybe not, codecs have limited 
error handling and reporting abilities.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130424/afeb3f6c/attachment.html>

More information about the Python-Dev mailing list