[Python-Dev] urllib.quote and unicode bug resuscitation attempt

[Mike Brown]

Stefan Rank wrote:
> on 12.07.2006 07:53 Martin v. Löwis said the following:
> > Anthony Baxter wrote:
> >>> The right thing to do is IRIs. 
> >> For 2.5, should we at least detect that it's unicode and raise a 
> >> useful error?
> > 
> > That can certainly be done, sure.
> > 
> > Martin
> 
> That would be great.
> 
> And I agree that updating urllib.quote for unicode should be part of a 
> grand plan that updates all of urllib[2] and introduces an irilib / 
> urischemes / uriparse module in 2.6 as Martin and John J Lee suggested.
>   =)
> 
> cheers,
> stefan

Put me down as +1 on raising a useful error instead of a KeyError or whatever,
and +1 on having an irilib, but -1 on working toward accepting unicode in the
URI-oriented urllib.quote(), because (a.) user expectations for strings that
contain non-ASCII-range characters will vary, and (b.) percent-encoding is
supposed to only operate on a byte-encoded view of non-URI information, not
the information itself.

Longer explanation:

I, too, initially thought that quote() was outdated since it choked on unicode
input, but eventually I came to realize that it's wise to reject such input,
because to attempt to percent-encode characters, rather than bytes, reflects a
fundamental misunderstanding of the level at which percent-encoding is
intended to operate.

This is one of the hardest aspects of URI processing to grok, and I'm not
very good at explaining it, even though I've tried my best in the Wikipedia
articles. It's basically these 3 points:

1. A URI can only consist of 'unreserved' characters, as I'm sure you know. 
It's a specific set that has varied slightly over the years, and is a subset 
of printable ASCII.

2. A URI scheme is essentially a mapping of non-URI information to a sequence
of URI characters. That is, it is a method of producing a URI from non-URI
information within a particular information domain ...and vice-versa.

3. A URI scheme should (though may not do so very clearly, especially the
older it is!) tell you that the way to represent a particular bit of non-URI
information, 'info', in a URI is to convert_to_bytes(info), and then, as per
STD 66, make the bytes that correspond, in ASCII, to unreserved characters
manifest as those characters, and all others manifest as their percent-encoded
equivalents. In urllib parlance, this step is 'quoting' the bytes.

3.1. [This isn't crucial to my argument, but has to be mentioned to complete
the explanation of percent-encoding.] In addition, those bytes corresponding,
in ASCII, to some 'reserved' characters are exempt from needing to be
percent-encoded, so long as they're not being used for their reserved purpose
(if any) in whatever URI component they're going into -- Semantically, there's
no difference between such bytes when expressed in the URI as a literal
reserved character or as a percent-encoded byte. URI scheme specs vary greatly
in how they deal with this nuance. In any case, urllib.quote() has the 'safe' 
argument which can be used to specify the exempt reserved characters.



In the days when the specs that urllib was based on were relevant, 99% of the
time, the bytes being 'quoted' were ASCII-encoded strings representing ASCII
character-based non-URI information, so quite a few of us, including many URI
scheme authors, were tempted to think that what was being
'quoted'/percent-encoded *was* the original non-URI information, rather than a
bytewise view of it mandated by a URI scheme.  That's what I was doing when I
thought that quote(some_unicode_path) should 'work', especially in light of
Python's "treat all strings alike" guideline.  But if you accept all of the
above, which is what I believe the standard requires, then unicode input is a
very different situation from str input; it's unclear whether and how the
caller wants the input to be converted to bytes, if they even understand what
they're doing at all.

See, right now, quote('abc 123%') returns 'abc%20123%25', as you would expect. 
Similarly, everyone would probably expect u'abc 123%' to return
u'abc%20123%25', and if we were to implement that, there'd probably be no harm 
done.

But look at quote('\xb7'), which, assuming you accept everything I've said
above is correct, rightfully returns '%B7'.  What would someone expect
quote(u'\xb7') to return?  Some might want u'%B7' because they want the same
result type as the input they gave, with no other changes from how it would
normally be handled. Some might want u'%C2%B7' because they're conflating the
levels of abstraction and expect, say, UTF-8 conversion to be done on their
input.  Some (like me) might want a TypeError or ValueError because we
shouldn't be handing such ambiguous data to quote() in the first place. And 
then there's the u'\u0100'-and-up input to worry about; what does a user
expect to be done with that?

I would prefer to see quote() always reject unicode input with a TypeError. 
Alternatively, if it accepts unicode, it should produce unicode, and since it
can only reasonably assume what the user wants done with ASCII-range
characters, it should only accept input < u'\x80'.

In any case, quote() should be better documented to explain what it accepts 
( a byte sequence )
and why 
( it is intended to be used at the stage of URI production where non-URI 
info, such as a unicode filesystem path, has already been converted to bytes 
according to the requirements of a URI scheme, and now needs to be represented 
as a URI-safe character sequence )
and exactly what it produces
( a str representing URI character s).

Mike