[TriZPUG] Project Night Project

[Chris Calloway]

Executive Summary:

A good project night project would be to make an mbox file suitable for 
importing into our current email list archives from the cooked archives 
of our old email list.

And Now For Something Completely Different (from an abstract):

A the last after-meeting, there were five of us hanging out until the 
bitter end. Michael Hrivnak asked what goes on at project night? Are 
people really pair programming?

I said it was a bit more organically do-acratic than that. I described 
that most people were working on their own things, some people even 
working on work. But that people generally spoke up if they wanted to 
work with someone else, had an idea or question, or wanted help. I said 
I generally came to project night with something to work on and rarely 
ended up working on that thing because I'd end up working with someone 
else on whatever they needed. Pairing happens at project night, just not 
necessarily by design or intention and not in call cases.

The organizer of any particular the project night sets the tone to his 
or her liking. The project nights Calvin has been calling he classifies 
as "informal" and that's been working pretty well for most people who 
come. There's an occasional "what's going on here?" freak-out, but most 
people get it as soon as they see it.

There was a project night where I announced beforehand that I'd work on 
the Django Bootcamp take home assignment with anybody who wanted to show 
up and work on them, and then nobody from the bootcamp showed up at the 
next project night. Then again, there was a project night where I was 
working on core Python issues and was hoping to involve someone else in 
that work, and instead ended up spending the the evening answering new 
to Python questions. That's just life. Things happen organically rather 
than the way we plan.

Alex Kesling said it would be good to have projects for people to work 
on at project night. I kind of thought, well, there are so many things 
people are already working on. And often I see people who aren't 
finishing the projects they already started get new ideas and want to 
recruit other people to do them. There are always project ideas for 
people with nothing else to do: (see 
http://whartoniteseekscodemonkey.tumblr.com/). The thing is, most people 
already have enough to do and most people have their own ideas. They are 
just looking for like-minded people around whom to work.

However, there is a lot of merit to what Alex suggested, despite my 
pessimism. There are always people new to Python who just want some 
direction regarding which direction to take. I'm pretty sure the next 
new to Python person who asks me what to do next, I'll point at the 
Python Koans. It really is a Pythonic Zen experience. The brilliance of 
the guy who gave the Python Koans "talk" at PyOhio, where the whole 
session of the talk was silence as people worked through the koans, 
still just makes me laugh.

But beyond new to Python or "what's going on here?" questions, I hear a 
lot of, "How can I help?" and "What can I do to help?," questions. These 
questions are sometimes tough because although I often need a lot of 
help, sometimes helping people help, or getting help in a form that is 
actually helpful, can be more difficult than doing something yourself 
even if you don't have the time. I've had bosses whose whole outlook on 
work was how to avoid getting "help," because there's no work quite as 
painful as work where the upper management has concluded that you need 
some extra "help" (cue management consultants and the whole script from 
Office Space). But I've been thinking a lot about this idea of 
ready-made projects and how someone who wants to help can help and have 
it be actually helpful.

Now that project is staring me right in the face. If someone wants a 
project, or if there needs to be a project for project night, or if the 
question is "How can I help?," then I have a project that I should 
particularly farm out, especially at this time when there are far too 
many projects to juggle.

Background:

TriZPUG was one of the first Python user groups. It was certainly the 
first Python user group with its own website separate from python.org 
wiki pages. It was also one of the first localized Python user groups 
with an email list.

When TriZPUG got started, the PSF had only just formed. The main Python 
organization before the PSF was the Python Software Association (PSA), 
sometimes called the Python Software Activity, which did things like put 
on the first nine or so Python conferences in Washington, DC. The PSA 
was a lot more do-acratic and less bureaucratic than the current PSF. 
(Today's PSF bureaucracy is actually a necessary thing, not a bad thing, 
in order to do legal software property conservancy.)

The PSA was pretty loose: you paid $25 and your were in. It was mostly 
government employees in and around Washington, DC, most of whom, 
including GVR, would go on to join Digital Creations (now Zope 
Corporation) at which time the PSF was formed to keep the Python 
copyright from flowing to whomever GVR happened to be employed by at any 
moment.

This is kind of a gloss, and ignores the intermediate Python Software 
Consortium. But one of the do-acratic things various members of the PSA 
did was run some Python infrastructure, namely python.net.

Python.net provided the first TriZPUG email list:

http://starship.python.net/mailman/listinfo/triangle-zpug

By 2007, python.net was kind of an orphan. The PSF's python.org had 
become the hang-out for Pythonistas. Python.net mostly provided 
homepages and web space for most of the earliest Python pioneers. But it 
didn't really have the support it had before the PSF took over 
python.org. Python.net suffered a couple of nasty crashes. Many of the 
early Python pioneers did not continue to support it after the crashes. 
It had become too easy to host your own site and get your own blog.

After the second crash, some of us in TriZPUG decided it would be better 
if we followed the lead of some more recent Python user groups and moved 
our email list to the more stable python.org. The second crash had 
severely disrupted our volunteer coordination for the biggest event we'd 
ever staged. The old email list had become a problem.

It took about a year after that decision to find the right people in 
python.org to make this happen as this was also about the time that 
mail.python.org was going through some management lapses. I even tried 
to find someone in TriZPUG to be the python.org postmaster after a call 
for help went out from the PSF. But eventually a new python.org 
postmaster was found, Brad Knowles, who was super nice and got our email 
list moved to where it has lived since May 16, 2008:

http://mail.python.org/mailman/listinfo/trizpug

As part of the move to python.org Brad offered to move our python.net 
archives to python.org. All he needed was the mbox file for our email 
list. Unfortunatly, our mbox file had not survived the python.net 
crashes. At least not a comprehensive mbox file. Only an mbox file since 
the last crash or two. All that really survived comprehensively were the 
"cooked" archives that MailMan makes from an mbox file:

http://starship.python.net/pipermail/triangle-zpug/

For MailMan to absorb old archives into a new email list, it needs to 
recreate the cooked archives from an mbox file. There might be a way to 
hack MailMan's archive import feature to accepted cooked archives, but 
it was beyond where our volunteer postmaster help at python.org was 
willing to go. They did make a valiant effort at reconstructing an mbox 
from the cooked archives.

But the format of the cooked archives had changed over the years 
2002-2008, and so it wasn't as simple as it first seemed. Things like 
the format of the email headers changed over the life of the email list. 
We arrived at a point where we had exhausted the reasonable efforts of 
the python.org postmasters to import the old archives into the new list. 
So to this day, we have archives split between python.net and python.org.

However, as part of our TriZPUG to TriPython name change, I have been 
talking to Brad Knowles once again.  With a little advance notice Brad 
is happy to change the name on our email list when we need. And he 
offered once again to try to import our old python.net archives. I 
reminded him of the previous effort to do so. But once reminded, we 
talked about making another more intensive effort to recreate and mbox 
file from the python.net cooked archives. Python's support for mbox 
manipulation has come a long way in the last four years:

http://docs.python.org/library/mailbox.html

Brad said if we can recreate the mbox file from the cooked archives, 
he'd like to try to import the archives once again so we would have 
comprehensive archives all in one place.

The Project:

1) Get the old cooked gzipped archives at:

http://docs.python.org/library/mailbox.html

using something like the urllib module or the requests package.

2) Expand the cooked archives with something like the gzip module.

3) Examine the archives and figure out a way to parse them into 
individual messages with header metadata. Classic text parsing.

4) Understand where the parsing has to deal with different metadata 
formats in the cooked archives. Deal with it. Classic real world text 
parsing.

5) From the parsed data, reconstruct an mbox file called 
triangle-zpug.mbox using something like the mailbox module.

6) Be prepared to make changes if necessary. Life is contingent in many 
ways. Things often don't work as planned. There are some old cooked 
archives beyond May 2008. Sometimes people posted to the old list by 
mistake after May 2008. One of the best pure Python discussions this 
group ever had was in April 2009 on the old list. It went on for days 
before someone realized, hey, how come this thread is on the old email 
list? You might want to reconstruct an mbox with all the old archives, 
and be prepared to reconstruct them again if the MailMan import cannot 
mix old and new archives from the same month.

You might want to do some testing. Testing might be pretty difficult. 
Mock only gets you as far as unit testing. This will require real 
functional and system testing. You might need to make test builds of 
MailMan. You will probably have to work closely with the python.org 
postmasters who are running the import steps. You might need to get the 
current mbox file from the python.org postmasters in order to do 
realistic testing.

The building and testing of this project could be involved. It may be a 
good candidate for continuous integration, especially if a lot of 
changes arise. And I would just anticipate a lot of changes for this 
project.

7) Be prepared to fail. Life is contingent in many ways. Things often 
don't work as planned. It might be there there will never be sufficient 
information in the cooked archives to make the MailMan import work in 
ways acceptable to the python.org postmasters.

8) This project would really do well to have a least one pair of people 
working on it. This doesn't sound like a really good loner project.

9) This project really needs to be in a publicly visible repository. You 
might get tired of it and other people could pick it up. You might get 
stuck and need more eyes on it. You might need more than a couple of 
people to help make it work. Your team might be geographically 
distributed around the Triangle. You want it so that when you are done, 
anyone can download the finished product, run it, and get the mbox file 
needed.

10) I'm certainly willing to help anyone who wants to work on this 
project at project night (as in help get you pointed in a right 
direction, help explain something that may not be clear to you, or help 
you get unstuck, rather than do it for you or even with you though I 
may). I'll be at the September project night. I'll have to miss the one 
in October. (At least the one in Carrboro. There's a possibility a 
project night at NCSU could be spun up by then.) But then I should be 
around for all the following project nights for awhile.

11) You also don't really need me at all to do this project. The project 
parameters are pretty well laid out here. This could be a self-starter 
for the right self-starter.

12) It would be great if you could let the email list know if you are 
working on this. We wouldn't want several people working on this in 
isolation without knowledge of each other.

13) It would be great if when you say you are working on this, you are 
actually working on this, rather than planning, intending, or wanting to.

14) Whenever possible, join the people already working on a project 
rather than making a new project.

15) It's not necessary for this to be done in time for the October name 
change. It's a do-acratic thing. It will be done when you do it.

-- 
Sincerely,

Chris Calloway http://nccoos.org/Members/cbc
office: 3313 Venable Hall   phone: (919) 599-3530
mail: Campus Box #3300, UNC-CH, Chapel Hill, NC 27599