Fil wrote:

...

Incidentally, since Fil seems to be taking all the credit for what I started I think he should actually put a proper credit in there for me, linking to my business website at http://www.orenet.co.uk/ rather than the vague passing mentions that are in the docs/code of his version of the system at the present time.

I'm sorry if I did anything not appropriate. From what you see in the code http://trac.rezo.net/trac/rezo/browser/Mailman/MySQLMemberAdaptor/MysqlMembe... it seems that the mention you require is there. Yes, that's good, although it's just a copy from my original code ;-) Please tell me how to make it better and I will correct it.

"real people" don't read source code, if you can, something in the README as well would be good for me, thanks.

More to the point, I fixed a few issues and added some features. Do you plan to take these into account in "your" version? If not we are necessarily "forking" this piece of code, regrettably, because I need "my" version in production.

That wasn't what I was on about.

Don't get me wrong here, I want the same as everyone else; excellent quality SQL support integrated into Mailman. (note SQL, not MySQL, as a portable backend system is not used by my original adaptor [it's using the MySQL-specific MYSQLdb?] or I believe your fork of it!?), and I don't mind that you have snarfed my code and extended it in your own way, I'm glad of it. I'd just like a credit that 'real' users might actually read ;-)

In order for that to happen, and for either version to be incorporated into Mailman 'proper', an agreement needs to be reached (and I may be out of date here, and one already has) betwen you, perhaps me, and the core mailman developers about how to solve at least the following:

• The conflict between the old pickle way of doing things of iterating over a get-singular-record method numerous times rather than a grab-multiple-records-and-return-in-the-right-format which is more the way SQL works effectively. Whether that's a rewrite, or some way of overriding the existing methods to implement them better, I don't know. Perhaps I can look into this soon. Any hints from the core guys?

• The two pages of suggested patches and extra changes to the core of mailman, eg. the CGI's and how everything should be merged together.

• Implementation of getMembersMatching() - You *CAN* do regexps in mysql ;-) Although perhaps that would necessitate use of a mysql version check at instantiation time, or in the ping() function?

That is what I *don't* like about your patch, I appreciate you needed to solve certain problems to get it going, but in doing so, I figured you have made it less able to integrate into the Mailman core, which is why I didn't just point everyone to your version, and kept mine as-is until I found some time for it.

I really like the 'extend.py' method you came up with because it's waaay better than my method of importing the thing, and your documentation is probably better than mine, too.

I note that you are in France (according to whois), which isn't impossibly far from London, where I am, perhaps we can get our heads together on this directly, after all France is the home of Kronenburg, which is a good reason for me to go there. I'm afraid I can't think of any others apart from Mailman and Kronenburg, though ;-)

K.

PS. Is Trac really as good as people seem to think it is? I've not started using it, but I certainly like the *look* of it?

-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ Linux/Security Contractor/LAMP Coder/ISP, via http://www.orenet.co.uk/ DJ via http://www.hellnoise.co.uk/

...

Yes, that's good, although it's just a copy from my original code ;-)

I'm guilty of referring to "Fil's MySQLMemberAdaptor" in various list postings. I'll try to be more careful in the future. Thanks. I'm not really sure what naming convention I would prefer, apart from just getting fully functional and suitably fast SQL support in there somehow.

ISTR Python 'pickles' and similar methods were what caused a low-memory server of mine to take 3+ hours to upgrade using Yum, so personally I guess I'd like to dispose of them from Mailman ;-)

...

• The conflict between the old pickle way of doing things of iterating over a get-singular-record method numerous times rather than a grab-multiple-records-and-return-in-the-right-format which is more the way SQL works effectively. Whether that's a rewrite, or some way of overriding the existing methods to implement them better, I don't know. Perhaps I can look into this soon. Any hints from the core guys?

Mailman 3 supports a real user database back end. Barry has been doing all the work on this, and I'm not up to speed having spent all my time on 2.1/2.2, so I can't comment intelligently on details, but that's where we're headed.

Ok, well I did a quick check, but could not find a download link on any Wiki's or Sourceforge. I've subscribed to the MM3 developers list, though. I hope it's not too high traffic...

Can someone tell me where to download the MM3 source as-is, or which source repository/tags to use to get hold of it so I can put my 2p in should I notice anything that could help, or find time to code anything in?

Also, since there does not appear to be anything firm in regards a release schedule for MM3, I'd appreciate your input on how (just the name of the function I would need to override/rewrite would be enough) I might accomplish better integration for a "grab-multiple-records-and-return-in-the-right-format" method of getting member records, so I can see if there's a way to sanely do so in MM2.2. I can't even remember if the MySQL Adaptor works with 2.1 or not?? No doubt I could find it given time and picking through code, but I'd appreciate a pointer...

It is quite strange, perhaps, but I suddenly have a lot more Mailman-motivation than I've had in a long while. Perhaps because I am bored stiff of PHP web development! ;-)

K.

-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ Linux/Security Contractor/LAMP Coder/ISP, via http://www.orenet.co.uk/ DJ via http://www.hellnoise.co.uk/

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 26, 2008, at 4:57 PM, Mark Sapiro wrote:

The MM3 developers list is essentially dead. Look at <http://mail.python.org/pipermail/mailman3-dev/.>

I'd totally forgotten about that list. Any objections if I just
delete it (keeping the archives of course)?

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkjdWNgACgkQ2YZpQepbvXHNdACdEmqkjq/lrWSZ6VvY0ixptskM /UMAnAgtOEh52n7NHo+YKYBWYuFnhevt =Zsle -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sep 26, 2008, at 1:48 PM, kyrian (List) wrote:

Don't get me wrong here, I want the same as everyone else; excellent
quality SQL support integrated into Mailman. (note SQL, not MySQL,
as a portable backend system is not used by my original adaptor
[it's using the MySQL-specific MYSQLdb?] or I believe your fork of
it!?), and I don't mind that you have snarfed my code and extended
it in your own way, I'm glad of it. I'd just like a credit that
'real' users might actually read ;-)

Note that Mailman 3 is backed by the Storm ORM <http://storm.canonical.com

. Storm supports SQLite, MySQL and PostgreSQL out of the box, and
it is my intent that MM3 should support all three out of the box, with
either a simple configuration setting or at most a plugin written
against documented and supported APIs. All the current tests use
SQLite since that comes with Python 2.5.

I would welcome developers to download and try the code, and to submit
branches or patches that improve the compatibility with MySQL or
PostgreSQL. I personally have not tried it with either database.

In order for that to happen, and for either version to be
incorporated into Mailman 'proper', an agreement needs to be reached
(and I may be out of date here, and one already has) betwen you,
perhaps me, and the core mailman developers about how to solve at
least the following:

IMO, Mailman 2.2 should not change the pickle-based approach to
persistency. I also think we should be careful about changing any
APIs to the data model that MM2 uses internally. Mark is an excellent
programmer and tests the code very well, but I always cringe when I
think about how few unit tests there are in the MM2.x code base.
That's another mistake of mine that I'm trying to correct in MM3.

Mailman 3 also uses Zope interfaces heavily, and the code is careful
to be written against these formal interfaces. In theory this should
mean that it doesn't matter whether the data persists in a supported
database, LDAP, Excel spreadsheet or flat files, with a properly
written implementation of those interfaces, it should Just Work. Such
implementations should be easily hooked into MM3 via its documented
plugin interface.

I'm soon going to be undertaking a bit of reorganization of the code.
I want to make it absolutely clear what is core functionality and what
is add-on. For example, pickup of email, moderation decisions,
munging operations, and delivery are all core functionality. Email
commands, RESTful admin control, command line operation, web interface
are all add-ons. A little bit of reorganization will make that more
clear, and I intend to release the next alpha once the core is
functional.

• The conflict between the old pickle way of doing things of
  iterating over a get-singular-record method numerous times rather
  than a grab-multiple-records-and-return-in-the-right-format which is
  more the way SQL works effectively. Whether that's a rewrite, or
  some way of overriding the existing methods to implement them
  better, I don't know. Perhaps I can look into this soon. Any hints
  from the core guys?

I hope the MM3 APIs are much more efficient for this.

• The two pages of suggested patches and extra changes to the core
  of mailman, eg. the CGI's and how everything should be merged
  together.

I'm not quite sure what you're asking for here.

• Implementation of getMembersMatching() - You *CAN* do regexps in
  mysql ;-) Although perhaps that would necessitate use of a mysql
  version check at instantiation time, or in the ping() function?

Can you explain what this is used for? In MM3, the core membership
construct is the 'roster'. The interesting thing is that a roster is
just a database query (for the SQL backend), so you could define any
number of queries to answer various membership questions.

I note that you are in France (according to whois), which isn't
impossibly far from London, where I am, perhaps we can get our heads
together on this directly, after all France is the home of
Kronenburg, which is a good reason for me to go there. I'm afraid I
can't think of any others apart from Mailman and Kronenburg,
though ;-)

Aside: I'm going to be in London the last two weeks of October, and
while it'll be very work focussed, I'd love to arrange a meet-up with
any UK Mailman hackers.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkjdWBYACgkQ2YZpQepbvXG9FQCfWPD5nlkiIju2aUz0MXkUxVxq QMMAn3awr3PVSosJjgEjRx3qOThi6VgP =lja4 -----END PGP SIGNATURE-----

Apologies for any confusion cause by posting by CC to this list, while emailing people directly. I was mainly responding to Fil, CC the list.

Fil wrote:

I've added the proper URLs in the README.

Thanks.

As for further developments I'll have to pass, as I'll be switchwing to MM3 before I have time to develop more that small fixes to Mailman and/or this adapter.

I think this is about the right way to go about things.

I were you, I would be inclined incorporate the small-ish character set fixes I've put into my code (or something like them, given it's as yet untested) into your fork of the MySQL Member Adaptor for MM2.x...

I'm "considering" pointing people from my own download location to your version at this point, FWIW.

Others wrote variously...

About MM2 code:

Thanks, I will poke around in "getRegularMemberKeys and getDigestMemberKeys" to see if I can see a sane method of integration happening 'in band'.

I can see where Fil is coming from a bit more having poked around his Trac archive, and it's probably not as much of an issue as I imagined where he's diverged from existing MM structure.

If I have it right, the only really significant "out of band" change is that he's basically just replaced the search backend for the admin pages to optimise fetching data on multiple list members at once, which I knew from the start would suck badly given the way the API worked 'in band'. The remainder of those changes seem to be docs, cosmetics and add-ons.

Some diff patches wouldn't go amiss though I suppose for more sane application of the changes, which is not so dependent on precise MM version.

About MM3 resources:

Thanks, these will be noted in my Wiki for whenever I find time to look at it.

And about Trac:

Thanks, err, about the same response there, either Wiki or Brain.

Last but by no means least, about beer in london:

Certainly, there are many worthy pubs in central london which would be ideal. Probably best worked out offlist between any interested parties, though. I guess the main questions are: types of beer, preferences for type of establishment, noise levels, music, etc?

K.

-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ Linux/Security Contractor/LAMP Coder/ISP, via http://www.orenet.co.uk/ DJ via http://www.hellnoise.co.uk/

kyrian (List) writes:

PS. Is Trac really as good as people seem to think it is? I've not started using it, but I certainly like the *look* of it?

Well, the MacPorts people switched to Trac advertising it as "our wonderful new Trac", and indeed for the first year I wondered how to use it, especially for searching bugs.

I ended up simply ignoring all courtesy, and submitted everything as a new bug and let the core devs classify them or find dupes. The number of dupes I've run into suggests that's what everybody does on MacPorts. :-(

At this point that Trac has gotten somewhat better, but I would say that when you start up you probably need to pay careful attention to configuration, especially to priming it with useful searches.

Fil wrote:

...

It should according to the docs I found, provided I've applied them correctly, fix the non-ASCII character encoding problems (I've relied on the assertions about list members to cover that angle, and just checked encoding on the 'user supplied' parameters to various function calls).

I finally got the time to look at your character encoding problem and patch, and I was not able to make it work. I still getting the dreaded UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 1: ordinal not in range(128)

Seems my "lucky guess" was not so lucky ;-)

I'm pretty sure I said I though it would work but hadn't actually tested it, anyways. I don't currently have a 'live' mailman+mysql instance anywhere to test with, for a start. I've got a dedicated development server set up but not migrated anything like that to it yet.

Also I'm afraid that I'm quite engrossed in writing a plugin for Pidgin and playing with Asterisk VoIP at the moment, and hadn't really settled on a time to look at MM again for a while. My first priority would be to resolve the horrible performance problem without losing the name info as per a previous email, and then I might look at this, when I get started back on it again.

The gist of the problem seems to be that you need to treat the strings as utf-8 or iso-8859-1 encoded 'objects' rather than standard ASCII string types within the code, and I don't know for sure how to do that. I'd have to research it as much as you would, unless anyone on-list has a suggestion about it, or can send me a copy of the Python book that I can reference for this.

K.

-- Kev Green, aka Kyrian. E: kyrian@ore.org WWW: http://kyrian.ore.org/ Linux/Security Contractor/LAMP Coder/ISP, via http://www.orenet.co.uk/ DJ via http://www.hellnoise.co.uk/

Human Rights left unattended may be Removed, or Destroyed, or Damaged by the Security Services.

kyrian (List) wrote:

The gist of the problem seems to be that you need to treat the strings as utf-8 or iso-8859-1 encoded 'objects' rather than standard ASCII string types within the code, and I don't know for sure how to do that.

And you have to know which because there are iso-8859-1 encoded characters which aren't valid utf-8 codes and there are utf-8 encoded characters which get garbled if decoded as iso-8859-1.

Thus, code like

    try:
        unicode(value, "ascii")
    except UnicodeError:
        value = unicode(value, "utf-8")
    else:
        # value was valid ASCII data
        pass

which I think is no different from simply

    value = unicode(value, "utf-8")

since if value is ascii to begin with, calling it utf-8 is OK,

doesn't work if value is actually iso-8859-1 encoded and contains bytes which aren't valid utf-8 or which decode differently from utf-8.

-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

OK I get something working with the following:

def escape(self, value):
    try:
        b = unicode(value,'utf-8')
    except:
        try:
            b = unicode(value,'latin-1')
        except:
            b = value
    return unicode(MySQLdb.escape_string(b.encode('utf-8')),'utf-8')

will try a little more and commit if it works