On Nov 26, 2014, at 02:30 PM, Abhilash Raj wrote:

I am working on porting mailman3 to python3. There are few doubts I have which may sound stupid, but I have a little knowledge about encoding and charsets which I think is important for the port. Also this is my first time working on python3 so forgive me for asking anything obvious( I did try googling first) The code is in very preliminary stages with tons of errors. Its up on launchpad.

lp:~raj-abhilash1/mailman/py3

Yay! Note that I also have been playing around with it a bit. I wouldn't say it's a functional branch, but you might compare notes:

lp:~barry/mailman/py3

1. pkg_resources.resource_string used to read the configuration files while initialization and testing now returns bytes instead of string(like in python2).

Right. "resource_string" is really a misnomer. In another Python 3 project I work on, I use:

from pkg_resource import resource_string as resource_bytes

just so that the call sites are accurate.

So instead I use pkg_resources.resource_stream and deocode to the steam to a utf-8 encoding(why? to avoid errors for the time being). I wanted to ask what would be the best for this case?

You can of course also decode the bytestring that "resource_bytes" returns. That's generally what I do in that other project.

2. To create a hash everywhere unicode strings must be encoded. So again the same question what should I encode it to? UTF-8 or US-ASCII? Or someway to determine which encoding should be used?

I think utf-8 is generally the right encoding to use, except in contexts where you are given a better hint. E.g. if you're decoding say the payload of a message with a Content-Type header, and that header has a charset attribute, it will name the encoding that you're supposed to use.

There may be cases where us-ascii or latin-1 would be better, but I think those should be determined on a case-by-case basis.

Also if anybody has suggestions on porting please let me know. I am online sporadically on irc as maxking on #mailman (I see all of the messages sent while I am away).

The trunk should now be python2.7 -3 clean, so it's ready to be ported. I'd probably not try to run the full test suite yet, but start at the lowest level, e.g. database/model and work your way up. If there are changes that make sense even in Python 2 (i.e. for bilingual support for now) try to put those in separate branches and merge proposals, so they can be merged into trunk even before a full port.

This page will be indispensable:

https://wiki.python.org/moin/PortingToPy3k/BilingualQuickRef

Cheers, -Barry