[ python-Bugs-639311 ] urllib.basejoin() mishandles ''

[SourceForge.net]

Bugs item #639311, was opened at 2002-11-16 04:34
Message generated for change (Comment added) made by bcannon
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=639311&group_id=5470

Category: Python Library
Group: Python 2.2.1
Status: Closed
Resolution: Invalid
Priority: 5
Submitted By: Mike Brown (mike_j_brown)
>Assigned to: Brett Cannon (bcannon)
Summary: urllib.basejoin() mishandles ''

Initial Comment:
It's not entirely clear whether urllib.basejoin() intends to 
implement RFC 2396's "resolution of relative URI 
references to absolute form" faithfully, but it seems to 
behave improperly when given an empty string as the 
relative URI to make absolute.

>>> from urllib import basejoin
>>> basejoin('http://host/foo/bar.xml','')
'http://host/foo/'

I believe it should return the base as-is, because the 
empty string is a reference to the document that 
contains that reference... and presumably the 
document's URI is what you're passing in as the base.

----------------------------------------------------------------------

>Comment By: Brett Cannon (bcannon)
Date: 2004-03-25 07:47

Message:
Logged In: YES 
user_id=357491

There is a misunderstanding about what I meant by "documented".  I 
meant it is not a specific part of the public API since it is not documented 
at http://www.python.org/dev/doc/devel/lib/module-urllib.html which is 
the official docs for urllib.  Having a docstring does not mean the code is 
considered part of the public API.  Sorry I didn't clarify that.

Now you obviously know a lot about this Mike, so can you give me your 
opinion on whether replacing urllib.basejoin with urlparse.urljoin is a 
reasonable move?

----------------------------------------------------------------------

Comment By: Mike Brown (mike_j_brown)
Date: 2004-03-23 15:15

Message:
Logged In: YES 
user_id=371366

Well, it is documented as "Utility to combine a URL with a 
base URL to form a new URL".

The notion of 'base URL' is described in the RFCs mentioned in 
urllib.__doc__ (all of which predate RFC 2396, I notice, 
though it doesn't matter) and is exclusively applicable to 
relative URL/URI reference resolution, a process in which an 
empty path denotes a same-document reference / no 
traversal from the origin. So I don't think it was unreasonable 
of me to have an expectation that basejoin() would be at 
least somewhat conformant to the specs that its parent 
module purports to implement.

And it actually *is* conformant when the relative URL 
consists of just a query or fragment; the empty path doesn't 
result in the tail of the base URL being chopped off before the 
query or fragment is added/replaced. So this lends further 
credence to the notion that the intent was to be conformant.

Furthermore, in the one and only place where it is used in 
core python libs at all (in urllib.FancyURLOpener), basejoin() 
is used in a way that would give a wrong result, if left as-is: 
to derive an absolute URL so a redirect can be followed, when 
given a relative URL in the Location header of an HTTP 302 
response. Granted, the odds of this header being empty are 
slim, but still. It's a bug in FancyURLOpener.redirect_internal, 
at least.

Nevertheless, I am now of the opinion that anyone serious 
about conformant URL handling will avoid urllib. In 4Suite I 
have implemented wrappers and replacements for the 
functions we need. See also bug #649962.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2004-03-23 13:37

Message:
Logged In: YES 
user_id=357491

I disagree with what you are expecting.  For instance, if you run 
``urllib.basejoin("http://python.org/index.html", "/doc")`` it returns 
"http://python.org/dev", which makes sense.  So changing its behavior 
based on it being an empty string would not strictly match how the 
function works when compared to being given any string.

And on top of things the function is not even documented, so you really 
shouldn't be expecting any specific behavior.

Closing this as invalid.

----------------------------------------------------------------------

Comment By: Brett Cannon (bcannon)
Date: 2003-05-21 20:14

Message:
Logged In: YES 
user_id=357491

Perhaps urllib.basejoin (which is not documented) should just become a 
wrapper for urlparse.urljoin ?  It won't solve this bug but it would cut back 
on unneeded code.

----------------------------------------------------------------------

Comment By: Mike Brown (mike_j_brown)
Date: 2002-11-26 02:41

Message:
Logged In: YES 
user_id=371366

I was partly mistaken; the document's URI is not necessarily 
the base. A reference with an empty path (e.g., an empty 
string or just a fragment identifier) is a reference to the current 
document, regardless of the base URI you are resolving 
against. A base URI is only for resolving relative URIs that are 
not referencing the current document. See some discussion 
at http://lists.w3.org/Archives/Public/uri/2002Jan/0015.html

So neither urllib.basejoin() nor urlparse.urljoin() fully 
implement the RFC 2396 "resolution to absolute form", since 
there would need to be a way to indicate "current document" 
other than returning the base.

Nevertheless, basejoin()'s behavior differs from urlparse.urljoin
()'s when presented with the empty string, and it's not clear 
whether that is intentional.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=639311&group_id=5470