Mailman 3 October 2013 - Twisted

[Twisted-Python] Garbage in logfiles?
by Conor Sayres Oct. 16, 2013

Oct. 16, 2013

Hi All, I'm working for an astro observatory and we are upgrading our telescope control software. We are swapping to twisted under the hood, and I have run into an issue regarding logging. I have set up twisted logging to record all communication between our devices. When we simulate our system with a high frequency log rollover rate (~5 seconds), many of the logs contain only garbage. Here is an example of a few lines in a given bad log file: 3230 3133 2d31 302d 3136 2031 303a 3131 3a34 352d 3037 3030 205b 2d5d 2044 6576 436d 6428 5354 4154 5553 290a 3230 3133 2d31 302d 3136 2031 303a 3131 3a34 352d 3037 3030 205b 2d5d 2044 6576 436d 6428 5354 4154 5553 290a 3230 3133 2d31 302d Here is an example of a few lines from a good log file: 2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS) 2013-10-16 10:11:39-0700 [-] Galil Reply(: 1, 1, 1, 1, 1 axis homed) 2013-10-16 10:11:39-0700 [-] To All Users(1 1 i axisHomed=1, 1, 1, 1, 1) 2013-10-16 10:11:39-0700 [-] Galil Reply(000000000, 000000000, 000000000, 000000000, 000000000 commanded position) 2013-10-16 10:11:39-0700 [-] To All Users(1 1 i cmdMount=0, 0, 0, 0, 0) 2013-10-16 10:11:39-0700 [_SocketProtocol,client] MirrorCtrl Reply(1 1 i axisHomed=1, 1, 1, 1, 1) 2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS) I haven't been able to reproduce the problem in a simpler framework than our simulation. I am triggering the rollover by directly calling LogFile.rollover on a timer. The problem also exists if I define a small file size limit for automatic rollover and let twisted do it automatically. Sound familiar to anyone out there? Cheers, Conor

4 6

Re: [Twisted-Python] Twisted-Python Digest, Vol 115, Issue 19
by Tom Sheffler Oct. 16, 2013

Oct. 16, 2013

Don't know if this helps: >>> chr(0x32) + chr(0x30) + chr(0x31) + chr(0x33) + chr(0x2d) + chr(0x31) + chr(0x30) + chr(0x2d) + chr(0x31) + chr(0x36) + chr(0x20) + chr(0x31) + chr(0x30) + chr(0x3a) + chr(0x31) + chr(0x31) '2013-10-16 10:11' >>> On Wed, Oct 16, 2013 at 1:14 PM, <twisted-python-request(a)twistedmatrix.com>wrote: > Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. Re: Garbage in logfiles? (Conor Sayres) > 2. Re: Garbage in logfiles? (Christopher Armstrong) > 3. Re: Garbage in logfiles? (Glyph) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 16 Oct 2013 11:12:12 -0700 > From: Conor Sayres <sayresc(a)gmail.com> > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Subject: Re: [Twisted-Python] Garbage in logfiles? > Message-ID: <81DD4767-8F1E-4A55-857E-A7A61BE9E3D4(a)gmail.com> > Content-Type: text/plain; charset="us-ascii" > > > On Oct 16, 2013, at 10:44 AM, Christopher Armstrong < > radix(a)twistedmatrix.com> wrote: > > > On Wed, Oct 16, 2013 at 12:33 PM, Conor Sayres <sayresc(a)gmail.com> > wrote: > > Hi All, > > > > I'm working for an astro observatory and we are upgrading our telescope > control software. We are swapping to twisted under the hood, and I have > run into an issue regarding logging. > > > > I have set up twisted logging to record all communication between our > devices. When we simulate our system with a high frequency log rollover > rate (~5 seconds), many of the logs contain only garbage. Here is an > example of a few lines in a given bad log file: > > > > 3230 3133 2d31 302d 3136 2031 303a 3131 > > 3a34 352d 3037 3030 205b 2d5d 2044 6576 > > 436d 6428 5354 4154 5553 290a 3230 3133 > > 2d31 302d 3136 2031 303a 3131 3a34 352d > > 3037 3030 205b 2d5d 2044 6576 436d 6428 > > 5354 4154 5553 290a 3230 3133 2d31 302d > > > > Here is an example of a few lines from a good log file: > > > > 2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS) > > 2013-10-16 10:11:39-0700 [-] Galil Reply(: 1, 1, 1, 1, 1 axis homed) > > 2013-10-16 10:11:39-0700 [-] To All Users(1 1 i axisHomed=1, 1, 1, 1, 1) > > 2013-10-16 10:11:39-0700 [-] Galil Reply(000000000, 000000000, > 000000000, 000000000, 000000000 commanded position) > > 2013-10-16 10:11:39-0700 [-] To All Users(1 1 i cmdMount=0, 0, 0, 0, 0) > > 2013-10-16 10:11:39-0700 [_SocketProtocol,client] MirrorCtrl Reply(1 1 i > axisHomed=1, 1, 1, 1, 1) > > 2013-10-16 10:11:39-0700 [_SocketProtocol,client] Galil Reply(XQ#STATUS) > > > > I haven't been able to reproduce the problem in a simpler framework than > our simulation. I am triggering the rollover by directly calling > LogFile.rollover on a timer. The problem also exists if I define a small > file size limit for automatic rollover and let twisted do it automatically. > > > > > > I don't think there's any method named "rollover" anywhere in Twisted > (in fact, I just confirmed this with grep). > > I'm sorry, the method is rotate my mistake > > > > > Your message is pretty vague and context-free; could you try to add more > necessary detail? > > Is the problem with the output you're seeing the *content* of the > numbers/letters, or just the fact that they're appearing at all? Maybe > there's relevant output before or after that block of numbers/letters? Did > you leave out the timestamp prefixes when pasting to this email, or is > there really no timestamp prefixing the lines of that output? > > The bad log files are 100% full of non-relevant output. They contain no > time stamps, solely lines of 4 character blocks (hex?). They should > contain information like the example of a good log I showed above. > > The good log files are 100% full of relevant output, no corruption. They > contain the timestamps (as they should). > > It seems somewhat random whether or not a given log file will be good or > bad. > > > > > I don't know where those numbers/letters could be coming from. Does your > code (or any libraries you use) ever print to stdout? By default, Twisted > redirects all prints (and any writing to sys.stdout) to the log file. > > > > The log was configured to NOT capture print statements to stdout. > > > > > -- > > Christopher Armstrong > > http://radix.twistedmatrix.com/ > > http://planet-if.com/ > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python(a)twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > >

1 0

[Twisted-Python] AutobahnTestsuite 0.5.6
by Tobias Oberstein Oct. 13, 2013

Oct. 13, 2013

Hi, we've just release a new version of AutobahnTestsuite - including WebSocket compression testing. Full announcement: https://groups.google.com/forum/#!topic/autobahnws/_WIk5Z_xJ4U Reports: http://autobahn.ws/testsuite#reports_20131013 Takeaways: 1) Compression: => go to section 12 in reports => compression on real-world JSON can achieve ratios of up to 20! 2) Performance: Autobahn/Twisted on CPython using wsaccel (Cython based acceleration for certain things) or PyPy are really competitive. http://autobahn.ws/testsuite/reports/servers/index.html => section 9 => Testing details: https://github.com/tavendo/AutobahnTestSuite/tree/master/examples/publicrep… There are a couple of interesting aspects rgd CPy+wsaccel and PyPy: on large volumes of data, CPy+wsaccel wins. On small data, but lots of, PyPy is faster. PyPy is good at accelerating all the loops and conditional code, but Cython is faster when doing mass data crunching. I expect more differences under real, concurrent load, since the GCs are quite different. Anyway, Cheers, /Tobias

1 0

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Oct. 13, 2013

Oct. 13, 2013

1 0

[Twisted-Python] web templating with html
by kevin mcintyre Oct. 12, 2013

Oct. 12, 2013

https://twistedmatrix.com/documents/current/web/howto/twisted-templates.htm… I see the example with quoting, but what if I want to render html? I'm trying to get the text/html part of an email, and want to render as is. Thanks in advance.

2 1

[Twisted-Python] ANN: Crochet 0.9.0 - Use Twisted Anywhere!
by Itamar Turner-Trauring Oct. 8, 2013

Oct. 8, 2013

Crochet is an MIT-licensed library that makes it easier for blocking or threaded applications like Flask or Django to use the Twisted networking framework. Crochet provides the following features: * Runs Twisted's reactor in a thread it manages. * The reactor shuts down automatically when the process' main thread finishes. * Hooks up Twisted's log system to the Python standard library logging framework. Unlike Twisted's built-in logging bridge, this includes support for blocking Handler instances. * A blocking API to eventual results (i.e. Deferred instances). This last feature can be used separately, so Crochet is also useful for normal Twisted applications that use threads. You can download Crochet at: http://pypi.python.org/pypi/crochet Documentation can be found at: http://crochet.readthedocs.org Bugs and feature requests should be filed at the project page: https://github.com/itamarst/crochet *What's New in 0.9.0* New features: * Expanded and much improved documentation, including a new section with design suggestions. * New decorator @wait_for_reactor added, a simpler alternative to @run_in_reactor. * Refactored @run_in_reactor, making it a bit more responsive. * Blocking operations which would otherwise never finish due to reactor having stopped (EventualResult.wait() or @wait_for_reactor decorated call) will be interrupted with a ReactorStopped exception. Bug fixes: * @run_in_reactor decorated functions (or rather, their generated wrapper) are interrupted by Ctrl-C. * On POSIX platforms, a workaround is installed to ensure processes started by reactor.spawnProcess have their exit noticed. See Twisted ticket 6378 <http://tm.tl/6738> for more details about the underlying issue.

2 2

[Twisted-Python] AutobahnPython 0.6.3 - WebSocket compression and more
by Tobias Oberstein Oct. 8, 2013

Oct. 8, 2013

Hi, AutobahnPython 0.6.3 was just released to PyPi https://pypi.python.org/pypi/autobahn with lots of new features, including _WebSocket compression_, an upcoming extension to the WebSocket protocol. We have seen compression ratios of 2-15x with JSON payload over WebSocket, which is great, in particular for mobile and embedded devices / IoT! AutobahnPython supports WS compression for both clients and servers, and with all knobs and options. Besides AutobahnPython, the extension is currently implemented in Chromium (very near shipping in Chrome), WebSocket++ (client and server), pywebsocket (server only) and Jetty (server only, not yet released, partial implementation). Complete list of new features is here: https://github.com/tavendo/AutobahnPython/blob/master/CHANGELOG.md /Tobias PS: should it be frowned upon posting announcements like this, please let me know. If it's useful/ok, please let me know also - I am hesitating each time to do postings like this to the list ..

6 20

[Twisted-Python] Weekly Bug Summary
by trac＠twistedmatrix.com Oct. 6, 2013

Oct. 6, 2013

1 0

[Twisted-Python] twisted linux netfilter_log protocol
by David Stainton Oct. 6, 2013

Oct. 6, 2013

Hi, I am wondering how I should combine various twisted interfaces in a way that makes sense. It probably shows here... that I'm new to twisted. The Linux iptables can log packets to the netfilter_log which can give access to user space. I wrote a simple twisted Reader (IReadDescriptor implementation) that is working functional code... https://gist.github.com/david415/6789612 But since these are packets it returns... should I implement a "read only Protocol"? The protocol's dataReceive() method could be called from the NFLogReader's doRead() method. Does this mean that NFLogReader would be responsible for calling buildProtocol to construct the NFLogProtocol? In the normal Twisted examples the buildProtocol seems to be called from the react loop.. Should I pass the protocol factory and the reader to the service? If I want to drop privileges right after retrieving the filedescriptor for netfilter_log, shall I use a Twisted Service/Application to drop the root privs to a non-superuser? Should the privileges be dropped by the privilegedStartService() method? The manual says that method is for preparing to drop services... What I want out of all of this is the most high performance and general solution to utilizing netfilter_log in twisted... David

3 7

[Twisted-Python] txsocksx 1.13.0.0 released (SOCKS clients for Twisted)
by Aaron Gallagher Oct. 6, 2013

Oct. 6, 2013

Hi! txsocksx <https://pypi.python.org/pypi/txsocksx> 1.13.0.0 has been released--the first stable 1.x release. This version has full implementations of client endpoints which wrap client endpoints and perform SOCKS 4, 4a, or 5 proxy negotiation. Additionally, txsocksx comes with support code to allow HTTP requests to be made over a SOCKS connection using Twisted's Agent API, and another wrapper endpoint for performing TLS negotiation after proxy negotiation. (The TLS wrapper endpoint will eventually land in twisted as a part of <http://tm.tl/5642>, but are experimentally in txsocksx for now.) Documentation is available on ReadTheDocs, including examples and an API reference: <http://txsocksx.readthedocs.org/en/latest/>.

2 2