[Mailman-Developers] Fwd: Anatomy of list ids does not keep with that of urls causes some REST end points to return 404 always

[Ankush Sharma]

---------- Forwarded message ----------
From: Ankush Sharma <ankush.sharma.ece12 at iitbhu.ac.in>
Date: Sun, Mar 8, 2015 at 3:49 PM
Subject: Re: [Mailman-Developers] Anatomy of list ids does not keep with
that of urls causes some REST end points to return 404 always
To: Richard Damon <Richard at damon-family.org>


Well, thanks Richard for replying.

On Sat, Mar 7, 2015 at 8:04 PM, Richard Damon <Richard at damon-family.org>
wrote:

> On 3/7/15 5:27 AM, Ankush Sharma wrote:
>
>> Hello everyone,
>>
>> I want to discuss a serious issue regarding mailman architecture in
>> context
>> with bug #1429366 <https://bugs.launchpad.net/mailman/+bug/1429366>.
>> The *hash(#)
>> *is a valid character as far as the local part of the email addresses is
>> concerned as per the email RFCs. So, as the mailing list addresses are
>> email addresses too, we can use # in the list names too. And, in context
>> with mailman it works well. We can create a list with  list_id
>> *sam#hashed.host.org
>> <http://hashed.host.org>* for the address *sam#hashed at host.org
>> <hashed at host.org> *. This works fine. But it makes the *list_id *to
>> contain
>> the hash character and therefore the REST endpoint for retrieving list
>> wise
>> info becomes invalid, i.e :
>>
>> *<api-root>/lists/sam#hashed.host.org <http://hashed.host.org>*
>>
>> Because in an URL the stuff after # is treated as *document starting
>> point*
>> i.e an id identifier or something of a dom element. This is not a valid
>> PATH for the server. Therefore the *falcon wsgi request object* does not
>> contain information of that and the *req.path* simply returns *sam* as the
>> *list_id* ( http://bazaar.launchpad.net/~mailman-coders/mailman/3.0/
>> view/head:/src/mailman/rest/wsgiapp.py#L65
>> ) giving a 404 because there is no any list with list id *sam*.
>> The mailman client works fine, it sends a GET to
>> *<api-root>lists/sam#hashed.host.org
>> <http://hashed.host.org>. *
>>
>> This causes the REST end points which needs *list_id* to return 404 or in
>> worse we can have a list_id clash between ids* sam#XXXXX* and *sam*.
>> Further more if the list_id starts with a # character then the server
>> finds
>> list_id to be *empty string* and therefore we get a KEY ERROR because
>> *fqdn_listname
>> *is not set too. The bug highly effects postorius too. The lists index
>> template at */postorius/lists/ *cannot be rendered as it uses the former
>> REST endpoint and again a 404 is given. And, until we delete this list
>> from
>> the database, we can't do anything except of getting a 404 and KEY ERROR
>> each time.
>> As far as security is concerned, if an another user created a public list
>> using a hash character, then public list indexing would also fail.
>>
>> *Possible fixes:*
>> 1. Put a constraint over the use of this kind of characters, but it's like
>> falling back.
>> 2. Do not derive list ids directly from the list addresses, use some kind
>> of hashing or encoding technique to generate list_ids. Need to change a
>> lot
>> in architecture;  :( .
>> 3. Use obscure techniques to retain the hash part as discussed here
>> http://stackoverflow.com/questions/317760/how-to-get-
>> url-hash-from-server-side
>>   .
>>
>> I think this is quite a serious issue in mailman architecture; i.e
>> obtaining list_ids directly from list addresses. I have been trying to fix
>> this bug using some obscure techniques. But as it is related to mailman
>> architecture, I thought it should be worthy to discuss on the mailing list
>> first.
>>
>> Thanks,
>> Ankush Sharma
>> github.com/black-perl
>>
>>  The simple, and standard complient, solution would seem to be to define
> that special characters get URL encoded as needed (and decoded on the
> receiving end.


I had a feeling of using URL encoding universally for all the URL building
routines but the issue I faced is of identifying the set of URL building
routines that actually require to encode URLs to prevent chaos. That's why
I didn't listed in on the possible fixes list. Because in that case it
would be difficult to identify these guys with full confidence.


> It should be standard procedure that ANY routine building a URL would use
> URL encoding on the pieces unless it is positive that it is not needed.


But now are things more clear. I will try to keep a positive approach
towards all these URL building routines keeping them in the suspected list
till I am not sure about one.

(URL encoding is replacing the problem chars with %nn where nn is the hex
value of the character, this now makes % a problem char so it is sent as
part of the URL as %25)

>
> --
> Richard Damon
>
> _______________________________________________
> Mailman-Developers mailing list
> Mailman-Developers at python.org
> https://mail.python.org/mailman/listinfo/mailman-developers
> Mailman FAQ: http://wiki.list.org/x/AgA3
> Searchable Archives: http://www.mail-archive.com/
> mailman-developers%40python.org/
> Unsubscribe: https://mail.python.org/mailman/options/mailman-
> developers/ankush.sharma.ece12%40itbhu.ac.in
>
> Security Policy: http://wiki.list.org/x/QIA9
>

Can it be a security concern as if any other user created such a public
list then public listing would simply fail ? I haven't tried reproducing
this part on my machine yet. But I am trying now.

Thanks,
Ankush Sharma
github.com/black-perl