Killing off Pipermail and the effects on scrubbing in Mailman 3
There's an aspect of the scrubber that isn't going to work in a Mailman 3 world where we have multiple, possibly external, archivers and especially where we don't have such tight integration with Pipermail (or Pipermail at all <wink>).
We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them.
I can think of a few ways to handle this.
The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues.
This makes sense because with a model of external archiving, those archivers may make different decisions about what should be removed or displayed from the original message. We can still include a little blurb saying that a part was scrubbed out, and since the messages can have the pre-calculated url to the message in one or more archivers, the user is always free to just click on the url to see the full message, displayed with whatever policy the archiver is configured with.
One possibility is to save the scrubbed part inside the core and provide a url to the REST API for accessing this attachment. This can't be inserted into the scrubbed message directly though because this would be a non-public url to the resource, and it would have to be proxied by the web ui. We need better configuration for integrating the web ui with the core any way (e.g. to calculate the url to the user's options page), so this could be part of that. The interactions are trickier though because you would then have to inform the web ui that there's a new attachment it should proxy.
The other, more elaborate option is to define an IScrubber interface, or alternatively a "primary" IArchiver, that the message can pass through, which would give it an opportunity to provide urls for each of the parts that will be scrubbed out. This is trickier because there can really be only one such thing defined in the system. I think it would be confusing if you received a message that had something like this:
text/html part scrubbed, view it at one of the following:
http://example.com/attachments/foo.html
http://example.org/some/extra/path/bar.html
http://another.archive.example.net/whatever/baz.htmlBesides, this may be nearly impossible to do without in-band communication with that external archiver, which is exactly what the RFC 5064 + message-id-hash was supposed to avoid. I think we definitely don't want to have to force such in-band communications to occur in order to scrub messages of unwanted parts.
For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler!
Any other suggestions?
Cheers, -Barry
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 3/15/2012 10:41 PM, Barry Warsaw wrote:
We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them.
There are two things going on. There is content filtering, i.e., removal from the message of parts with unwanted MIME types or filename extensions. These parts are simply removed by pipeline/mime_delete.py (which probably needs some changes ported from 2.1, aargh...).
Then there is what pipeline/scrubber.py does with the remaining message which is remove those message parts which can't be rendered well in a flat, text/plain message and store them aside and replace them by links in the message. The part we can't do in MM 3 is calculate a URL to display/download them.
I can think of a few ways to handle this.
The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues.
I'm not sure about the *before* with respect to usenet and digest and certainly outgoing. Currently in 2.1, we don't scrub (as opposed to content filter) non-digest deliveries unless scrub_nondigest is Yes. We maybe should just drop that option.
We also don't scrub messages for the MIME digest.
I also don't think we scrub messages destined for usenet. I think we let usenet worry about that in the same way we propose to let whatever archiver is configured worry about it.
I don't see a need to handle these differently in MM 3.
[...]
For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler!
Perhaps we could keep the scrubber as is except for modifying it to not store scrubbed parts and put some kind of apology in the message rather than the link to the no longer stored content.
Then my lp:~msapiro/mailman/scrubber-fix branch would still be relevant ;)
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32)
iD4DBQFPY3QuVVuXXpU7hpMRAkN2AKCVzhwvBUITlLVgDMwg+V+da0cyJACXed7Q jAvaD7jeN2/4armc/nIxBw== =MsB3 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Mar 16, 2012, at 10:11 AM, Mark Sapiro wrote:
There are two things going on. There is content filtering, i.e., removal from the message of parts with unwanted MIME types or filename extensions. These parts are simply removed by pipeline/mime_delete.py (which probably needs some changes ported from 2.1, aargh...).
Yeah, that's embarrassing ;). I've started down the road of adding unittests for the code in that module. You'll see the start of that land momentarily.
Then there is what pipeline/scrubber.py does with the remaining message which is remove those message parts which can't be rendered well in a flat, text/plain message and store them aside and replace them by links in the message. The part we can't do in MM 3 is calculate a URL to display/download them.
Yep.
The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues.
I'm not sure about the *before* with respect to usenet and digest and certainly outgoing. Currently in 2.1, we don't scrub (as opposed to content filter) non-digest deliveries unless scrub_nondigest is Yes. We maybe should just drop that option.
We also don't scrub messages for the MIME digest.
I also don't think we scrub messages destined for usenet. I think we let usenet worry about that in the same way we propose to let whatever archiver is configured worry about it.
I don't see a need to handle these differently in MM 3.
ISTM that essence of the scrubber is to turn any remaining text/html parts into plain text, by various means. I think the MM2 scrubber.py module is essentially hopeless, but the basic functionality is useful. I've decided to remove the scrubber in the Pipermail-eradication branch, which will also land momentarily. I think it would be useful though to rewrite the scrubber, boil it down to its essential functionality, and add that to the appropriate spot in the pipeline.
How would you like to take a crack at that?
For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler!
Perhaps we could keep the scrubber as is except for modifying it to not store scrubbed parts and put some kind of apology in the message rather than the link to the no longer stored content.
Then my lp:~msapiro/mailman/scrubber-fix branch would still be relevant ;)
Yeah, sorry about that. ;) I think scrubber.py was just too nasty to salvage. Something much simpler would still be useful though.
Cheers,
- -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJPZMQ9AAoJEBJutWOnSwa/nFEP/iMDCM+ETv1KV36nP8r/cZfB C50m+K1MUm/MaZpkpQI8980J96QWC1RoWvQ7sQGg2difvDvNwI0JZP4gMBJkHVUu sO/hJZu0BDa28cC9Ww94fRX4ujelm/jesc8td0v02s54FSHUIOgxxDr+sfWNFPvI OpLDJZVtC6LJbDt1IqI2ozxbq/b3hhuaXDbmzIsWqotyZZ/+fQDjgM4L9SCEjhrT tDwQjFhsZmH3m58pFRkP/cOJCV2lKs0MnMGMhELHGkatMGKtVFAuP1e3r24N20yX EVDX/7Dg20BzacNYnAVGnO28sYqb4JltRAb14+IvIMcRzIO+WKKAyJioKX3cohcT 14fhb0agtDPlMMBJw8J5AD9VEimMcZaMmISLpRY6jqkaHRu/4RxZlG3RRWtcBwdS dN0WZnnNx6B+wV5VUJ7Q5WaDO1Xtp0jGHuT96vOQlHDm/+iwwmWWvGH3DQg1yVDN gT2/JyLeXpDprP+qXNPLyWlMlADQjUCq7uvD51J0gcCC6aLanPnM9CuCQXdJRlFl 7g+zI9a17qCdniQcbNUgq+87ektXLi7JCp6nA1yEm0Zaelp3wJC2cB7up9ZaVR7O SX8qpMFnfqFkvsQLC2pLH7plplHpboXWOjLALITFBzasth4hS98oHH+gOJktTKni Erk1f+FsVR9l0Geu2q++ =z6a7 -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 3/17/2012 10:05 AM, Barry Warsaw wrote:
ISTM that essence of the scrubber is to turn any remaining text/html parts into plain text, by various means. I think the MM2 scrubber.py module is essentially hopeless, but the basic functionality is useful. I've decided to remove the scrubber in the Pipermail-eradication branch, which will also land momentarily. I think it would be useful though to rewrite the scrubber, boil it down to its essential functionality, and add that to the appropriate spot in the pipeline.
How would you like to take a crack at that?
Sure. Now that I actually have a bit of an idea of what's going on in the MM 3 core, I'm happy to give it a go.
Next step for me may be to learn more about how the unittests fit within their framework so I can create some.
Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one.
Anyway, Cygwin is not going to cut it for MM 3. At the sprint, I tried installing MM 3 in a vagrant VM, but there was too much missing (e.g., no Python.h) and even 'apt-get install python-dev' didn't fix that. I ended up working remotely inside a virtualenv on my production server. That actually seems to work OK, but I'm afraid that as I get deeper into it, there will be things I need to do that I won't want to do on my production box.
Anyway, if anyone has any suggestions for me besides the obvious "bite the bullet now and scrap Windows - it will only be worse later", I'm interested. I suppose I could always dual-boot Windows and some Linux side by side.
Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time.
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32)
iD8DBQFPZSGEVVuXXpU7hpMRAnRAAKDSSRUhDdQe8HoIBzOh3coe8elMIQCfU+dP fKbzWiMB+H1wm4Jou28BV7g= =Ehz5 -----END PGP SIGNATURE-----
On 03/17/2012 11:43 PM, Mark Sapiro wrote:
Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one.
Anyway, Cygwin is not going to cut it for MM 3. At the sprint, I tried installing MM 3 in a vagrant VM, but there was too much missing (e.g., no Python.h) and even 'apt-get install python-dev' didn't fix that. I ended up working remotely inside a virtualenv on my production server. That actually seems to work OK, but I'm afraid that as I get deeper into it, there will be things I need to do that I won't want to do on my production box.
Anyway, if anyone has any suggestions for me besides the obvious "bite the bullet now and scrap Windows - it will only be worse later", I'm interested. I suppose I could always dual-boot Windows and some Linux side by side.
Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time.
I would definitively suggest to get rid of Windows, but if you don't want or can't do the big step there is no reason to do a dual-boot, a just create a Linux virtual machine and you're done :)
Unless you don't have much RAM is the best solution, and there are even pre-made virtual machines here: http://virtualboximages.com/
so it doesn't really take long to set up something working :)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Mar 17, 2012, at 04:43 PM, Mark Sapiro wrote:
Next step for me may be to learn more about how the unittests fit within their framework so I can create some.
It's *relatively* simple. Essentially all unittests live in a test_foo.py module inside a tests/ directory (which has an __init__.py so it's a package). I generally name unittest.TestCase subclasses as TestFoo and individual tests inside there as test_foo() methods. There are lots of examples spread throughout. It's only a little extra work if you have to create a new tests/ subdirectory or test_foo.py module, but again, it should be pretty straightforward.
These all get auto-discovered by the zope.testing framework, so there's
nothing else necessary to hook it all up. The one other tricky thing is the
layers, which are a zope.testing mechanism to share more generic resources
among a suite of tests. E.g. if you need the rest server to be started, you'd
use the RESTLayer (see mailman.testing.layers for what's currently defined).
To enable a layer, you just need to set the layer attribute of the TestFoo
class. Almost all the tests will require at least the ConfigLayer since that
brings up the Zope component architecture, connfiguration system, etc.
zope.testing is a bit weird here because the normal rules of subclasses don't actually apply. E.g. the LMTPLayer derives from ConfigLayer which derives from MockAndMonkeyLayer. All the appropriate setUp()s get called by zope.testing so you don't need to up-call explicitly in the layer class. Yes, this kind of sucks since it's magical, and it's one reason why eventually I'd like to switch to something like testresources and a more sane testing framework, but that's a lot of effort for little immediate value.
Note that doctests are all set up in test_documentation.py so they're not really all that special.
Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one.
There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows. Of course, all the Linuxes should give you a way to dual-boot your Windows machine if you're so inclined. You may even be able to boot off a USB drive or stick, though that will be slower. Commercial options such as VMware are also a possibility, though of course you should try the free options first. :)
Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time.
Definitely. :)
- -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJPZxexAAoJEBJutWOnSwa/jlQQAKqzGaL5Vrnm5vS8+NSlN+LH gY/Gyh/aT8F4V6IYd3Hn47kPMtdMPv6m+FNhSTOGkwf1EU1UtPgNvkcYUNuVsHBe A9Bh04DDk0t5S1WOdcKBSdkwt65YR3yNkvsiCZQpOXD4xDF4EDlSRwNTjtGk6ze7 84xBUL5DXJiO1L5OxBsZusYlR4vpO80AwCszil3uon9oDQIxsHuXvPfVwjJM8CfN hFf3Rq7LePLbOO0uFNLxsDxMyRZx0wrN/vg+2H81bnLNdheaqlmRjoZn2TkFZmTb eYN+XeFW9l2jlGgYfQAm8R5e+mNhE+a3HG7LwO2IyYn8JzR+YTFQeiR91XdOFYrt OPfMrXbUaRAb6QEwwWJY2HmbqIiihgFOPtmSBSRcIoiU/cWjC6LbajZN8QV24xhq yTrbkBxZBTGPPQ7tgd05bb/PYlqdWCRoXeL8zlbcDK98MFzPgfBMfmHqWObPd+8E q49pDoeDUlhW+SmhfTOqM3zQvuolIu6WYDTcTeIWYLzMsjT3pDJ6R1NIw4jNGmtl TqLYL+WuUzBXyNfGD234C2VnJscVyJLFduew2pig6dFG0MX9gAN/zTI3i8wZDlUC xDKDagCZ7uSNzjVEDQLzoZmrs3EzuPvCY5h4kzlVObUiKBvg8ALPEV8MJS6M8QDj IeBbG8Iniz01AoBaixZ1 =rxit -----END PGP SIGNATURE-----
On Mon, Mar 19, 2012 at 07:25:37AM -0400, Barry Warsaw wrote:
Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one.
There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows.
http://xkcd.com/424/springs to mind.
(I'm aware a fair few people are moving away from Ubuntu, in favor of Mint <http://linuxmint.com/>).
Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time.
With a python script? ;o)
(a former friend & colleague, Chris Lightfoot[1] wrote evil.c one day; I mirror it at http://hendricks.amyl.org.uk/~adam/tmp/evil.c-txt -- to Debianize a FBSD box. Console server *highly* recommended. It worked the few times we used it)
[1] http://www.ex-parrot.com/~chris/wwwitter/20070305-chris_lightfoot_1978-2007....
-- War is Peace Freedom is Slavery Ignorance is Strength 1984
On 03/19/2012 11:25 AM, Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Mar 17, 2012, at 04:43 PM, Mark Sapiro wrote:
Next step for me may be to learn more about how the unittests fit within their framework so I can create some. It's *relatively* simple. Essentially all unittests live in a test_foo.py module inside a tests/ directory (which has an __init__.py so it's a package). I generally name unittest.TestCase subclasses as TestFoo and individual tests inside there as test_foo() methods. There are lots of examples spread throughout. It's only a little extra work if you have to create a new tests/ subdirectory or test_foo.py module, but again, it should be pretty straightforward.
These all get auto-discovered by the zope.testing framework, so there's nothing else necessary to hook it all up. The one other tricky thing is the layers, which are a zope.testing mechanism to share more generic resources among a suite of tests. E.g. if you need the rest server to be started, you'd use the RESTLayer (see mailman.testing.layers for what's currently defined). To enable a layer, you just need to set the
layerattribute of the TestFoo class. Almost all the tests will require at least the ConfigLayer since that brings up the Zope component architecture, connfiguration system, etc.zope.testing is a bit weird here because the normal rules of subclasses don't actually apply. E.g. the LMTPLayer derives from ConfigLayer which derives from MockAndMonkeyLayer. All the appropriate setUp()s get called by zope.testing so you don't need to up-call explicitly in the layer class. Yes, this kind of sucks since it's magical, and it's one reason why eventually I'd like to switch to something like testresources and a more sane testing framework, but that's a lot of effort for little immediate value.
Note that doctests are all set up in test_documentation.py so they're not really all that special.
Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code..
Cheers, Andrea
On Mar 19, 2012, at 12:03 PM, Andrea Crotti wrote:
Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code..
Care to adapt my post into a merge proposal? :)
-Barry
On 03/19/2012 12:27 PM, Barry Warsaw wrote:
On Mar 19, 2012, at 12:03 PM, Andrea Crotti wrote:
Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code.. Care to adapt my post into a merge proposal? :)
-Barry
Yes sure I will do that ASAP..
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 3/19/2012 4:25 AM, Barry Warsaw wrote:
There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows.
Thanks to all for the advice. I have some learning to do.
I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well.
When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file.
I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h.
At that point, I gave up and went to a virtualenv on my production server.
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (MingW32)
iD8DBQFPZ6GnVVuXXpU7hpMRAlwwAJ91EQ7k4TtOLBvyexmrmyspzJZ8RACg3iFQ fdgRt+ylQxrcuoL0d3W+VMY= =lP82 -----END PGP SIGNATURE-----
On 03/19/2012 09:14 PM, Mark Sapiro wrote:
Thanks to all for the advice. I have some learning to do.
I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well.
When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file.
I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h.
At that point, I gave up and went to a virtualenv on my production server.
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Well but why using Vagrant? Vagrant is great to create VMS for automatic testing, but if you want to create your own VM with all your things that you need I don't see how it can help, just install a normal Ubuntu/whatever on Virtualbox and you're fine..
And another possible advice is to always copy the errors you get, also because most of the times you can get the answer you want simply by a quick google search ;)
On Monday 2012-03-19 14:40 (-0700), Andrea Crotti <andrea.crotti.0@gmail.com> wrote:
Well but why using Vagrant? Vagrant is great to create VMS for automatic testing, but if you want to create your own VM with all your things that you need I don't see how it can help, just install a normal Ubuntu/whatever on Virtualbox and you're fine..
I'd recommend this too. (Vagrant for repeatable deployments, you only need one).
If you like Ubuntu, Ubuntu Server (an LTS version) is well worth using for this sort of thing as it is already headless, no desktop (or though you can add one if you want).
Chris
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Mar 19, 2012, at 02:14 PM, Mark Sapiro wrote:
I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h.
Hmm, I just tried this in a Lucid chroot and it worked for me. :/
I do get test failures (could be 2.6 related) but everything built.
- -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJPZ6mFAAoJEBJutWOnSwa/5IIP/3bo0dOX/ESg4wDbtd//sjcr 9AHwT7X3oOUIB9H8IAqRkSFNTIZDoxTtLrv7BzIHLgfIAkS9KaKcRSK3IGlulPrG 6KtCf93nVbtd8KdRYDxT2TmWhoR377g9DDnGDztcf6N39HiSVrPNzxLx2oSMrfYh cvLhQnGnI7xOloVox6uEvEB7fU1DQaLJe6G4obZxzr0YZGViZGv1u4h1Qi0dgPgO 7Av1cne29t7+4iyNbgoo8nvCmSBcE4Lbmg3e+mHr25BMpKUKuM0sce323RN3Pp/+ 5GZOtFuiWKYB/GHmm0OpIRCUCMX1oPf2SyVw94+MYW32wspDbUpOBevgtGPbFmQx WtKm1l6ps2TGX+q5WatNVemtmoZWVLx7N6CfxdcyZ7kdIvnGkxVqylq7o5RUWUQM 83rqo4EEDPZjnF7Aos7RgdSUAbdh99Rus4/8VoCJgg4sTbMjFFcH/PRiYeiMR+GK GJtFiZilYjiXUnCZ6jye3s6mBsy/YHtUd3ytYrjUtritVgMcMuPF0EwRTwb04OIN TWmE5BNvlgF8mn0U1UA3+V5A42GqjnK7PajZ9VpTwyymU65QgK7PSwIYyXNzATQb wHHokKct8sE4eNWt9Bs4Ai6MlMJKLoAFaOJvJQtZ2r0OBpuBcNomHb0c2yddlGSr Yj04lLFc/Sp3FFko9EbU =Ib4D -----END PGP SIGNATURE-----
I, too, am attempting to get MM3 running on my new laptop (Mac OSX 10.7). Because of the way Xcode 4.3 and Python are set up, compiling the .c extensions in storm is failing.
I need to figure out where I can tell setup.py to add an additional "include" path when it calls the c compiler.
Anyone know how?
Richard
On Mar 19, 2012, at 4:47 PM, Barry Warsaw wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Mar 19, 2012, at 02:14 PM, Mark Sapiro wrote:
I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h.
Hmm, I just tried this in a Lucid chroot and it worked for me. :/
I do get test failures (could be 2.6 related) but everything built.
- -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux)
iQIcBAEBCAAGBQJPZ6mFAAoJEBJutWOnSwa/5IIP/3bo0dOX/ESg4wDbtd//sjcr 9AHwT7X3oOUIB9H8IAqRkSFNTIZDoxTtLrv7BzIHLgfIAkS9KaKcRSK3IGlulPrG 6KtCf93nVbtd8KdRYDxT2TmWhoR377g9DDnGDztcf6N39HiSVrPNzxLx2oSMrfYh cvLhQnGnI7xOloVox6uEvEB7fU1DQaLJe6G4obZxzr0YZGViZGv1u4h1Qi0dgPgO 7Av1cne29t7+4iyNbgoo8nvCmSBcE4Lbmg3e+mHr25BMpKUKuM0sce323RN3Pp/+ 5GZOtFuiWKYB/GHmm0OpIRCUCMX1oPf2SyVw94+MYW32wspDbUpOBevgtGPbFmQx WtKm1l6ps2TGX+q5WatNVemtmoZWVLx7N6CfxdcyZ7kdIvnGkxVqylq7o5RUWUQM 83rqo4EEDPZjnF7Aos7RgdSUAbdh99Rus4/8VoCJgg4sTbMjFFcH/PRiYeiMR+GK GJtFiZilYjiXUnCZ6jye3s6mBsy/YHtUd3ytYrjUtritVgMcMuPF0EwRTwb04OIN TWmE5BNvlgF8mn0U1UA3+V5A42GqjnK7PajZ9VpTwyymU65QgK7PSwIYyXNzATQb wHHokKct8sE4eNWt9Bs4Ai6MlMJKLoAFaOJvJQtZ2r0OBpuBcNomHb0c2yddlGSr Yj04lLFc/Sp3FFko9EbU =Ib4D -----END PGP SIGNATURE-----
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/richard%40nfsnet.o...
Security Policy: http://wiki.list.org/x/QIA9
On Mar 19, 2012, at 05:41 PM, Richard Wackerbarth wrote:
I, too, am attempting to get MM3 running on my new laptop (Mac OSX 10.7). Because of the way Xcode 4.3 and Python are set up, compiling the .c extensions in storm is failing.
I need to figure out where I can tell setup.py to add an additional "include" path when it calls the c compiler.
Anyone know how?
I've never done it, but I found this. Hopefully that helps.
http://www.buildout.org/docs/tutorial.html#custom-egg-building
-Barry
On 03/19/2012 03:14 PM, Mark Sapiro wrote:
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Incidentally, how much interest would there be in having a Mailman-developer VM (say, for VirtualBox) around and easy to download? Some modern distro, appropriate dev tools, and a checkout of moderately recent code, maybe even ready to run with instructions on the desktop or something? It seems like it might be a nice thing to hand out to prospective GSoC students as well as Mark. ;) The Systers folk made one a few years ago for 2.1 dev and maybe it's time for an updated version?
Terri
On 3/19/2012 4:02 PM, Terri Oda wrote:
Incidentally, how much interest would there be in having a Mailman-developer VM (say, for VirtualBox) around and easy to download? Some modern distro, appropriate dev tools, and a checkout of moderately recent code, maybe even ready to run with instructions on the desktop or something? It seems like it might be a nice thing to hand out to prospective GSoC students as well as Mark. ;)
I think it would be great. I'm not sure how fast I will progress with this, but I may be the one to make it. If it were available right now, I would certainly use it.
I'm currently thinking about VirtualBox vs. dual boot. I'm used to working with Cygwin command windows on my Windows desktop. There are various issues with Cygwin, both itself and WRT MM 3, so I need to move away from that. VirtualBox would allow me to continue to work in a hybrid environment with some Windows applications and some Unix like applications running together, but dual boot might avoid potential hardware interface issues and would motivate me to move completely away from Windows (although I would probably still need access to Windows to diagnose user problems with the web site I admin).
Of course, I could always do both. The two machines I would do this on both have two 200GB+ hard drives with lots of available space so disc is not a problem. My travel computer has 8GB of ram and a dual-core 64 bit CPU, so that's plenty I'm sure. My main, at home computer is over 10 years old and has only 2GB of ram and a Pentium 4 CPU, but I think even it would be OK.
-- Mark Sapiro <mark@msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan
On Mar 19, 2012, at 06:17 PM, Mark Sapiro wrote:
I'm currently thinking about VirtualBox vs. dual boot. I'm used to working with Cygwin command windows on my Windows desktop. There are various issues with Cygwin, both itself and WRT MM 3, so I need to move away from that. VirtualBox would allow me to continue to work in a hybrid environment with some Windows applications and some Unix like applications running together, but dual boot might avoid potential hardware interface issues and would motivate me to move completely away from Windows (although I would probably still need access to Windows to diagnose user problems with the web site I admin).
From a technical standpoint, you can run Windows in a virtual machine hosted on Linux. :)
-Barry
On Mar 19, 2012, at 05:02 PM, Terri Oda wrote:
On 03/19/2012 03:14 PM, Mark Sapiro wrote:
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Incidentally, how much interest would there be in having a Mailman-developer VM (say, for VirtualBox) around and easy to download? Some modern distro, appropriate dev tools, and a checkout of moderately recent code, maybe even ready to run with instructions on the desktop or something? It seems like it might be a nice thing to hand out to prospective GSoC students as well as Mark. ;) The Systers folk made one a few years ago for 2.1 dev and maybe it's time for an updated version?
+1 is an understatement. :)
-Barry
- On 19 Mar 2012, Terri Oda wrote:
Incidentally, how much interest would there be in having a Mailman-developer VM (say, for VirtualBox) around and easy to download? Some modern distro,
I'm not very active around here anymore, but I'll toss this to the table: maybe a public ec2 AMI? A sponsoring foundation could even host an account with "IAM" (Amazon IDM) to run them under and pay for the AMI storage.
-- David Champion • dgc@uchicago.edu • IT Services • University of Chicago
On 03/19/2012 11:02 PM, Terri Oda wrote:
On 03/19/2012 03:14 PM, Mark Sapiro wrote:
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Incidentally, how much interest would there be in having a Mailman-developer VM (say, for VirtualBox) around and easy to download? Some modern distro, appropriate dev tools, and a checkout of moderately recent code, maybe even ready to run with instructions on the desktop or something? It seems like it might be a nice thing to hand out to prospective GSoC students as well as Mark. ;) The Systers folk made one a few years ago for 2.1 dev and maybe it's time for an updated version?
Terri
That sounds like a good idea :) In this case, maybe something declarative with puppet/vagrant might really be a very good idea, since it's much easier to ship a configuration file and it has to be deployed / updated easily...
On 03/19/2012 09:14 PM, Mark Sapiro wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 3/19/2012 4:25 AM, Barry Warsaw wrote:
There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows.
Thanks to all for the advice. I have some learning to do.
I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well.
When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file.
I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h.
At that point, I gave up and went to a virtualenv on my production server.
At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot.
Anyway, it's going to be interesting to figure this out.
Since I was curious to see I created a vagrant VM with lucid and I only needed to install the following:
To install (Lucid32):
- python
- python-dev
- bzr
- python-setuptools
- gccTo make everything work and be able to run buildout and bin/test in mailman bzr version. So now I was thinking to transform this manual steps in a Vagrantfile, and we could use this to ship for people that want to try, other things to do:
Optional packages:
- Gnome / XFCE / Other
- vim
To check/set:
- launchpad login, possibly passing the authentication and the SSH
key somehow to the vagrant file
- dimension of the VM to make sure it's not too smallPS. I get 6 failures running the tests though, as File "/home/vagrant/mailman/src/mailman/model/docs/users.rst", line 172, in users.rst Failed example: user_2.preferred_address = anne Differences (ndiff with -expected +actual): Traceback (most recent call last): - ... - UnverifiedAddressError: Anne Person <anne@example.com> + File "/usr/lib/python2.6/doctest.py", line 1248, in __run + compileflags, 1) in test.globs + File "<doctest users.rst[41]>", line 1, in <module> + user_2.preferred_address = anne + File "/home/vagrant/mailman/src/mailman/model/user.py", line 112, in preferred_address + raise UnverifiedAddressError(address) + UnverifiedAddressError: <unprintable UnverifiedAddressError object>
participants (8)
-
Adam McGreggor -
Andrea Crotti -
Barry Warsaw -
Chris Clark -
David Champion -
Mark Sapiro -
Richard Wackerbarth -
Terri Oda