---------- Forwarded message ---------- From: Ankush Sharma <ankush.sharma.ece12@iitbhu.ac.in> Date: Sun, Mar 8, 2015 at 3:49 PM Subject: Re: [Mailman-Developers] Anatomy of list ids does not keep with that of urls causes some REST end points to return 404 always To: Richard Damon <Richard@damon-family.org>
Well, thanks Richard for replying.
On Sat, Mar 7, 2015 at 8:04 PM, Richard Damon <Richard@damon-family.org> wrote:
On 3/7/15 5:27 AM, Ankush Sharma wrote:
Hello everyone,
I want to discuss a serious issue regarding mailman architecture in context with bug #1429366 <https://bugs.launchpad.net/mailman/+bug/1429366>. The *hash(#) *is a valid character as far as the local part of the email addresses is concerned as per the email RFCs. So, as the mailing list addresses are email addresses too, we can use # in the list names too. And, in context with mailman it works well. We can create a list with list_id *sam#hashed.host.org <http://hashed.host.org>* for the address *sam#hashed@host.org <hashed@host.org> *. This works fine. But it makes the *list_id *to contain the hash character and therefore the REST endpoint for retrieving list wise info becomes invalid, i.e :
*<api-root>/lists/sam#hashed.host.org <http://hashed.host.org>*
Because in an URL the stuff after # is treated as *document starting point* i.e an id identifier or something of a dom element. This is not a valid PATH for the server. Therefore the *falcon wsgi request object* does not contain information of that and the *req.path* simply returns *sam* as the *list_id* ( http://bazaar.launchpad.net/~mailman-coders/mailman/3.0/ view/head:/src/mailman/rest/wsgiapp.py#L65 ) giving a 404 because there is no any list with list id *sam*. The mailman client works fine, it sends a GET to *<api-root>lists/sam#hashed.host.org <http://hashed.host.org>. *
This causes the REST end points which needs *list_id* to return 404 or in worse we can have a list_id clash between ids* sam#XXXXX* and *sam*. Further more if the list_id starts with a # character then the server finds list_id to be *empty string* and therefore we get a KEY ERROR because *fqdn_listname *is not set too. The bug highly effects postorius too. The lists index template at */postorius/lists/ *cannot be rendered as it uses the former REST endpoint and again a 404 is given. And, until we delete this list from the database, we can't do anything except of getting a 404 and KEY ERROR each time. As far as security is concerned, if an another user created a public list using a hash character, then public list indexing would also fail.
*Possible fixes:*
Put a constraint over the use of this kind of characters, but it's like falling back.
Do not derive list ids directly from the list addresses, use some kind of hashing or encoding technique to generate list_ids. Need to change a lot in architecture; :( .
Use obscure techniques to retain the hash part as discussed here http://stackoverflow.com/questions/317760/how-to-get- url-hash-from-server-side
I think this is quite a serious issue in mailman architecture; i.e obtaining list_ids directly from list addresses. I have been trying to fix this bug using some obscure techniques. But as it is related to mailman architecture, I thought it should be worthy to discuss on the mailing list first.
Thanks, Ankush Sharma github.com/black-perl
The simple, and standard complient, solution would seem to be to define that special characters get URL encoded as needed (and decoded on the receiving end.
I had a feeling of using URL encoding universally for all the URL building routines but the issue I faced is of identifying the set of URL building routines that actually require to encode URLs to prevent chaos. That's why I didn't listed in on the possible fixes list. Because in that case it would be difficult to identify these guys with full confidence.
It should be standard procedure that ANY routine building a URL would use URL encoding on the pieces unless it is positive that it is not needed.
But now are things more clear. I will try to keep a positive approach towards all these URL building routines keeping them in the suspected list till I am not sure about one.
(URL encoding is replacing the problem chars with %nn where nn is the hex value of the character, this now makes % a problem char so it is sent as part of the URL as %25)
-- Richard Damon
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/ mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman- developers/ankush.sharma.ece12%40itbhu.ac.in
Security Policy: http://wiki.list.org/x/QIA9
Can it be a security concern as if any other user created such a public list then public listing would simply fail ? I haven't tried reproducing this part on my machine yet. But I am trying now.
Thanks, Ankush Sharma github.com/black-perl