Mailman 3 April 2016 - Twisted

[Twisted-Python] web.client.readbody
by John Aherne 04 Apr '16

04 Apr '16

To Glyph This works fine if I use response.deliverBody I get a json response back. I'll look at what Cory says and see where I get. Thanks John -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

1 0

[Twisted-Python] web.client.readBody
by John Aherne 04 Apr '16

04 Apr '16

I have been using deliverBody to process in responses but decided that readBody might be a better fit. So far it works with http, but as soon as I switch to https it fails. I get a response http code 200 OK but no data. I've puzzled over this but can't see what the problem is. Has anyone else seen this problem and found out what the solution is. Thanks Response received <twisted.web._newclient.Response object at 0x0362EB50> Response version: ('HTTP', 1, 1) Response code: 200 Response phrase: OK Response headers: Headers({'alternate-protocol': ['443:quic'], 'x-xss-protection': ['1; mode=block'], 'accept-ranges': ['none'], 'expires': ['Mon, 04 Apr 2016 13:32:31 GMT'], 'vary': ['Accept-Language,Accept-Encoding'], 'server': ['mafe'], 'cache-control': ['public, max-age=86400'], 'date': ['Sun, 03 Apr 2016 13:32:31 GMT'], 'x-frame-options': ['SAMEORIGIN'], 'alt-svc': ['quic=":443"; ma=2592000; v="32,31,30,29,28,27,26,25"'], 'content-type': ['application/json; charset=UTF-8']}) LENGTH twisted.web.iweb.UNKNOWN_LENGTH [('Alternate-Protocol', ['443:quic']), ('X-XSS-Protection', ['1; mode=block']), ('Accept-Ranges', ['none']), ('Expires', ['Mon, 04 Apr 2016 13:32:31 GMT']), ('Vary', ['Accept-Language,Accept-Encoding']), ('Server', ['mafe']), ('Cache-Control', ['public, max-age=86400']), ('Date', ['Sun, 03 Apr 2016 13:32:31 GMT']), ('X-Frame-Options', ['SAMEORIGIN']), ('Alt-Svc', ['quic=":443"; ma=2592000; v="32,31,30,29,28,27,26,25"']), ('Content-Type', ['application/json; charset=UTF-8'])] Got Error error res [Failure instance: Traceback (failure with no frames): <class 'twisted.web.client.PartialDownloadError'>: 200 OK ] SHUTDOWN -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] SSL wincertstore Ticket 6371
by John Aherne 04 Apr '16

04 Apr '16

I have picked up on this again. There would appear to be a failing test in test_agent.py. I have added a new test to check that certificateOptions.trustRoot is an instance of OpensSSLWindowsCertificateAuthorities if wincertstore is available And I have modified the existing test test_setsTrustRootOnContextDefaultTrustRoot to be skipped if wincertstore is available. I have run some simple tests using Agent against googlemaps to confirm the host verification works and fails and a similar test with treq. So looks good. But before I send up the patch I need to be happy that trial twisted completes with out error. And I keep getting an error on one test but I can't work out what the problem is. Hopefully someone can fathom it from the traceback and point me in the right direction. Thanks [ERROR] Traceback (most recent call last): Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug) <DelayedCall 0x42e4f30 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('d.root-servers.net', <Deferred at 0x42e4a80>)> <DelayedCall 0x42e4328 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('e.root-servers.net', <Deferred at 0x42e4b48>)> <DelayedCall 0x42e4d28 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('f.root-servers.net', <Deferred at 0x42e4fa8>)> <DelayedCall 0x42e4ad0 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('h.root-servers.net', <Deferred at 0x42e4a58>)> <DelayedCall 0x4299aa8 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('l.root-servers.net', <Deferred at 0x42ce3a0>)> <DelayedCall 0x42e4cb0 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('a.root-servers.net', <Deferred at 0x42e40d0>)> <DelayedCall 0x42e4d50 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('b.root-servers.net', <Deferred at 0x42e4ee0>)> <DelayedCall 0x42e45d0 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('c.root-servers.net', <Deferred at 0x42e4af8>)> <DelayedCall 0x42e4670 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('g.root-servers.net', <Deferred at 0x42e4468>)> <DelayedCall 0x42e4968 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('i.root-servers.net', <Deferred at 0x42e4e68>)> <DelayedCall 0x42e4558 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('j.root-servers.net', <Deferred at 0x42e4fd0>)> <DelayedCall 0x42cee68 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('k.root-servers.net', <Deferred at 0x42e4b70>)> <DelayedCall 0x4355f08 [60.0s] called=0 cancelled=0 ThreadedResolver._cleanup('m.root-servers.net', <Deferred at 0x4355e68>)> twisted.names.test.test_tap.OptionsTests.test_recursiveConfiguration ------------------------------------------------------------------------------- Ran 11184 tests in 84.952s FAILED (skips=2946, errors=1, successes=8237) -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] Running trial on windows
by John Aherne 04 Apr '16

04 Apr '16

When I run trial in a virtualenv it seems to miss some imports and skips some tests. It would seem to be mainly wincertstore that it can't import. To make it work I have to include wincertstore in my main python setup and then it starts to test the wincertstore tests. Anyone else seen this happening. This is Windows 10 and python2.7.11 32 bit, virtualenvwrapper-win 1.2.1 Thanks -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] web.client.readbody
by John Aherne 04 Apr '16

04 Apr '16

I was obviously half-asleep when I posted the previous email. Doing more tests, it would seem that only googlemaps is giving me this problem. -- *John Aherne* *www.rocs.co.uk <http://www.rocs.co.uk>* 020 7223 7567

2 1

[Twisted-Python] Problem with Deferreds
by Chris Norman 02 Apr '16

02 Apr '16

Hi all, I recently got over myself and forced myself to read about Deferreds rather than using threading opperations. I've used them successfully in a couple of places, but this one has me flummoxed: Here's the code for the Deferred and it's sub-commands: def do_command(self, cmd, **kwargs): """Process a command in true Deferred style.""" if cmd.permissions(self.connection): cmd(self.connection, **kwargs) else: logger.warning('Blocked from running command %s which is secured with %s.', cmd.__name__, cmd.permissions.__name__) raise CommandError('You have insufficient privileges to perform this action. The staff have been notified.') def handle_error(self, err): """Handle an error from do_command.""" if isinstance(err, CommandError): return self.send_error(e.message, disconnect = e.disconnect) else: self.log(e, level = 'exception') self.send_error('Sorry, but a problem with the server means your command was not executed. The staff have been notified.') def lineReceived(self, line): """Parse an incoming command.""" global lines lines += 1 # Increment the line count. data = line.decode(settings.ENCODING) try: command, kwargs = json.loads(data) if not isinstance(command, string_types) or not isinstance(kwargs, dict): raise TypeError('Expecting [str, dict]. Got [%s, %s] instead.' % (type(command), type(kwargs))) except (TypeError, ValueError) as e: self.log('Invalid command string: %s', data, level = 'error') self.log(e, level = 'exception') return self.send_error('Invalid command.', disconnect = True) cmd = commands.commands.get(command, None) if cmd is None: self.log('Unrecognised command: %s.', command, level = 'warning') elif self.connection.player or not cmd.login_required: d = defer.Deferred() print('Adding callback.') d.addCallback(self.do_command, **kwargs) print('Adding errback.') d.addErrback(self.handle_error) print('Calling callback.') d.callback(cmd) print('Called.') # Never gets this far. return d else: return self.send_error('Not authenticated.', disconnect = True) Here's the traceback I get when the callback gets called: Unhandled Error Traceback (most recent call last): File "server/functions.py", line 88, in server_start reactor.run() File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", line 1194, in run self.mainLoop() File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/base.py", line 1206, in mainLoop self.doIteration(t) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/epollreactor.py", line 396, in doPoll log.callWithLogger(selectable, _drdw, selectable, fd, event) --- <exception caught here> --- File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", line 101, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/log.py", line 84, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", line 610, in _doReadOrWrite self._disconnectSelectable(selectable, why, inRead) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/internet/posixbase.py", line 258, in _disconnectSelectable selectable.connectionLost(failure.Failure(why)) File "/usr/local/lib/python3.5/site-packages/Twisted-16.0.0-py3.5.egg/twisted/python/failure.py", line 232, in __init__ tb = self.value.__traceback__ builtins.AttributeError: 'Deferred' object has no attribute '__traceback__' Anyone have any ideas? Cheers, Chris

2 2

[Twisted-Python] upcoming changes to twistedmatrix.com mail infrastructure
by Glyph 02 Apr '16

02 Apr '16

Over the last few months, twistedmatrix.com's mailman installation has been used increasingly frequently to execute denial-of-service attacks against people's mailboxes. This is accomplished by sending huge numbers of subscription requests to our website, which in turn sends huge numbers of confirmation emails to their inbox. Based on some information that some targeted users have sent me, I now believe that this is to cause those users' mail quotas to be exceeded so that password reset or login notification emails won't reach them. This has been going on for some time, but the frequency and severity of the attacks seem to be increasing; I only recently realized that this was considerably worse than an annoyance for those affected. I now have at least 1 confirmed report of this attack being a part of a (partially successful) identity theft. This isn't the only problem we have with email: We're running our own infrastructure which puts load on our already beyond-overloaded volunteer system administration team. Despite running our own infrastructure, we are not dogfooding Twisted at all in the process, so we're not even learning anything useful from the pain; "exim is bad" is a lesson we've already learned many times, we do not need to keep learning it. Given how hard it is for us to upgrade Mailman in our current system, we aren't even dogfooding our fellow community project terribly well. Our infrastructure runs on the same host as the website and the buildmaster, overloading a very creaky system. In addition to mailing lists, we run a mail forwarder. Our server's sender reputation is ... not great. We don't have SPF records, we don't do DKIM, and we don't provide authenticated SMTP for users, so emails just come from "wherever" when they are sent from, e.g. 'glyph(a)twistedmatrix.com' :-). In order to address this, as soon as I can reasonably manage to do so, I will be moving Twisted's email infrastructure to mailgun.com <http://mailgun.com/>, a product that I've been successfully using for a range of personal domains (in particular, the divmod.com <http://divmod.com/> email forwarder - yes, I still operate that, when the Twisted community promises you an email address for life you get it ;-)). Additionally, Mailgun uses a bunch of Twisted within their infrastructure, so (although we won't be operating it) we will actually be dogfooding considerably more. (Mailgun is a product of my employer, Rackspace, but they've given us a generous open source discount so there's no conflict of interest; the Twisted project won't be spending money on this.) There will be a couple of inconveniences immediately after the transition: At first, there will be no self-service subscription to mailing lists any more. If you want to subscribe, you'll have to send a message to twisted-python-owner(a)twistedmatrix.com <mailto:twisted-python-owner@twistedmatrix.com> and the list administrator (right now, probably just me) will manually add your address. (Self-service unsubscription will still be possible.) I'm not sure if I'll be able to keep the list archives at https://twistedmatrix.com/pipermail/ <https://twistedmatrix.com/pipermail/> updated, at least at first. I would encourage everyone to use http://news.gmane.org/gmane.comp.python.twisted <http://news.gmane.org/gmane.comp.python.twisted> and http://news.gmane.org/gmane.comp.python.twisted.web <http://news.gmane.org/gmane.comp.python.twisted.web> in the meanwhile. Speaking of the contents of that sad URL, many disused mailing lists will be deleted. I doubt anyone will notice since there haven't been any posts to most of them in many years. If you presently send email from a twistedmatrix.com <http://twistedmatrix.com/> address, you will probably want to start using the mailgun forwarder so that your messages will have nice shiny DKIM/SPF headers; I suspect you may start having more deliverability problems than you already do once other mail servers notice that we have said records if you're not using them. I'll distribute SMTP credentials via GPG-encrypted email to everyone I'm aware of who uses such an address. There will be considerable benefits though: For those of you with @twistedmatrix.com <http://twistedmatrix.com/> addresses, Mailgun operates a pretty conservative low-pass spam filter, but in looking at the analytics from my own personal domains, it really helps a lot and it is definitely more effective than the setup we've got right now. Deliverability and mail-sending performance should be much improved; messages should arrive faster because they will be quarantined or deferred-bounced by major senders like GMail et. al. far less often, because we'll be forwarding less spam and legitimate messages will have appropriate anti-spam headers. Trac will get faster at certain times because email DoSes should stop hitting the server. Administrative overhead will decrease; we can just stop maintaining email ourselves. Last but certainly not least, we'll stop being a collective unwilling accessory to cybercrime. Probably these changes will all be pretty subtle, and most folks won't notice, but I wanted it to be clear in advance that they were intentional, in case there is some disruption associated with them :-). If anyone wants to give me a hand with parts of this (for example, setting up a smarthost configuration so that trac can still send email) please let me know. -glyph

4 5