Mailman 3 July 2013 - Twisted

[Twisted-Python] Buildbot downtime
by Tom Prince 04 Mar '22

04 Mar '22

I am plannig to upgrade twisted's buildbot in the near future. In preperation for that, I am going to take down the production buildbot on Tuesday, June 26, around 1600 UTC[1]. During that time, the result of any builds done will be lost. Thus, I'd ask that people avoid committing to trunk during that time. I will send a reminder before I take down the buildbot both here, and on #twisted and #twisted-dev. Tom [1] http://timeanddate.com/worldclock/meetingtime.html?iso=20120626&p1=80&p2=22…

2 8

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz 06 Aug '21

06 Aug '21

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] Real Viagra Professional - for real sex pros
by Tamika 01 May '19

01 May '19

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <center> <a href="http://rqf.entercompassion.com"><img src="http://mediapix.ru/pics/685207b3dd437eff7f15c0744b5c1d3a.gif"> <title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" href="http://gfx8.nnt.com/mail/12.0.1187.0824/positioning.css" /> </head> <body style="overflow:auto;"> <div id="PrintShellToolbar"> <div class="ContainerPadding"> <span class="FloatRight"> <a href= > <img alt="" src="http://gfx1.nfi.com/mail/w2/pr02/ltr/glyph_close_rest.gif" /> <Close window> <a href= > <img alt="" src="http://gfx2.czb.com/mail/w2/pr02/ltr/i_print.gif" /> <Print> </a> </span> </div> </div> <div> <div class="BorderContainerOutline ContainerPadding"> <div class="ClearBoth"> <span class="HeaderData" style="font-size:large;"> <TuPostal.com Cambia de nombre?> </span> </div> <div class="ClearBoth"> <span class="HeaderLabel"> <From:> </span> <span class="HeaderData"> <b><TuPostal.com></b> </span> </div> <div class="ClearBoth"> <span class="HeaderLabel"> <Sent:> </span> <span class="HeaderData"> <div class="ClearBoth"> <span class="HeaderLabel"> To: </span> <span class="HeaderData"> Mikelelliott (mikelelliott(a)rlo.com) </span> </div> <br /> </div> </div> <div class="MessageBody" style=""> <br /> <div id="MessageBodyText" class="ExternalClass" style="border: 1px solid #fff ;"> <pre>Hola!<br> <br>Tupostal es ahora parte de Sonico!<br> <br>Para conocer nuestro nuevo sitio, haz clic en el siguiente link:<br><a href='http://www.gwpcom/index1.php' target='_blank'>http://www.bjncom/index1.php</a><br> <br>Sonico es un sitio donde puedes...<br>- Guardar fotos ilimitadas<br>- Re encontrarte con viejos amigos<br>- Recibir novedades de tus amigos.<br> <br>Te Esperamos! Ya somos mas de 21 millones de usuarios!!<br> <br>Sonico - Amigos Conectados<br> <br> <br>PD: Ahora tambien podras guardar tus postales<br> <br> <br>------------------------------------------<br>Haz recibido este email porque te encuentras registrado en fdm.com.<br>Si no deseas recibir mas mensajes como este, por favor haz clic en el siguiente link:<br><a href='http://www.cehcom/unsubscribe.php?type=2' target='_blank'>http://www.fyocom/unsubscribe.php?type=2</a><br> <br></pre> </div> </div> </div> </body> </html>

8 10

[Twisted-Python] [ANN] txZMQ released
by Andrey Smirnov 21 Jul '14

21 Jul '14

Hi! I would like to announce txZMQ, ZeroMQ bindings for Twisted. txZMQ is based on pyzmq and requires recent ØMQ version (>=2.1). txZMQ uses ØMQ APIs to get file descriptor that is used to signal pending actions from ØMQ library IO thread running in separate thread. txZMQ should be usable with any Twisted reactor. ØMQ is: * The socket library that acts as a concurrency framework. * Carries messages across inproc, IPC, TCP, and multicast. * Connect N-to-N via fanout, pubsub, pipeline, request-reply. * Fast enough for clustered products and supercomputing. * Asynch I/O for scalable multicore message-passing apps. Bridging ØMQ and Twisted makes a nice match: fast simple messaging between Twisted instances solving complex problems :) Links: * PyPi: http://pypi.python.org/pypi/txZMQ/ * Source code: https://github.com/smira/pyzmq * pyzmq: http://pypi.python.org/pypi/pyzmq * ØMQ: http://www.zeromq.org/ -- Andrey Smirnov, Qik Web Team Lead

8 11

[Twisted-Python] regarding xml elements
by Raj kumar 02 Oct '13

02 Oct '13

Hi, I have an xml file in my application, I have created an element using Example goes like this......... document.createElement("abc") and i appeneded it by using append() method. But how i can reflect this change to my xml file? and one more thing is i want to create element with some other parameters.... like............. <abc m=" " n=" "> </abc> and i have m and n values as strings with me. can anybody help me to create this element and write it to the existing xml file as a child of an existing element? Thanks in advance.. Raj.. Forgot the famous last words? Access your message archive online at http://in.messenger.yahoo.com/webmessengerpromo.php

9 14

[Twisted-Python] logging
by Glyph 18 Sep '13

18 Sep '13

Hello everybody. Today I'd like to talk about everyone's favorite subsystem within Twisted: logging. There are a number of problems with Twisted's logging system. The general theme of these problems is that Twisted log system was designed with some intended use-cases in mind, but its implementation was stopped halfway through and none of those use-cases were really well satisfied. At various conferences, and in person at various times, we've gotten together to express frustration at the logging system, but I don't think that the problems with it have ever been really thoroughly written down. Wilfredo has been working on logging here: http://trac.calendarserver.org/browser/CalendarServer/trunk/twext/python/lo…. This began as an attempt to just address some issues with our own logging wrappers, but has been growing into a design that is more fully capable of replacing logging within Twisted wholesale. I hope one of us can contribute it to Twisted soon, but in the meanwhile, please feel free to provide feedback - hopefully if some people can provide a bit of informal feedback now, the formal review process can be shorter and more pleasant :). Now, back to some explanation of the current problems. The main intended use-case for Twisted's logging system is to avoid doing unnecessary work in the case where it is not required. One of the reasons we even still have our own logging system is that the stdlib logging module is notorious for doing too much work per log message, and doing it too prematurely. One aspect of this is that logging would just use the existing 'kwargs' dictionary as its message type, and not actually do any formatting; that is, string-copying, until the message was logged. On Python 1.5.2, that meant no need to allocate anything since the interpreter needed a dict for the kwargs anyway! Woohoo premature optimization! Obviously, at least on PyPy, that optimization is no longer relevant (and we don't have performance tests for logging anyway). But, I don't think we should change that because a dict is still a good object for the sort of extensibility that log messages need. But, we have never written any filtering logic, nor have we instrumented any of our existing log messages with formatting stuff. Every message with a format string just goes to the log. For a long time, there's been consensus among at least a few core Twisted developers that log messages ought to be specified in terms of a format message and several relevant keyword arguments, so that tools can extract the values of the keyword arguments without parsing ad-hoc messages or using regexes. In other words, like this: log.msg(format="Foo happened. Relevant information includes: " "bar=%(bar)s baz=%(baz)s boz=%(boz)s", bar=bar, baz=baz, boz=boz) Yet, nothing within Twisted itself does this. There's a bit of a chicken and egg problem: since we have no tools for filtering log messages based on their structured values, there's no obvious motivation to break the structured contents of a message out into those values. And since no messages break out their values into nice structured fields, there's no motivation to write filtering tools that inspect those fields. And, even if we did have filtering, we don't have a facility to easily defer work. You could, of course, just do something like this: class GenerateOnlyIfNeeded(object): def __str__(self): return doSomethingExpensiveToGenerateAString() log.msg(format="debugging: %(expensive)s", logLevel=debug, expensive=GenerateOnlyIfNeeded()) But that's an awful lot of thinking for a single log message, not to mention the temporary class and the method you needed to allocate. Such deferral of work should be easy. And, speaking of the things that one might filter on, log messages don't typically include information about what module and class they're from; or, for that matter, what object is logging them. This information should be included by default in idiomatic usage of the API, so that a filtering system can inspect them. Another reason that we - both Twisted itself and Twisted users - don't typically pass keyword arguments and a format string, is that the default argument to msg is, simply, a "message". (It also does a dumb thing where it tries to emulate the argument-passing convention of 'print' and stringify its arguments and join them with spaces; we should get rid of that.) Instead, that argument always ought to be a format string. We should also have a log writer within Twisted that preserves structure, and dumps the logs using JSON (or similar, perhaps more than one of these; AMP would be another good thing). It's also not possible to use format strings when you are writing an error log message in Twisted; instead, 'why' is treated completely differently than 'message'. The 'why' argument to 'err', which should be mandatory (what were you doing when you tried to log a message?) is optional, which means that error messages are even more inscrutable. The portions of the dictionary used by the logging system are not namespaced in any way, even by convention. Therefore it's easy for a user to conflict with the keys used by the system. If users of Twisted actually used the system as intended more pervasively, I'm pretty sure that we would end up seeing people conflict with 'time' and 'system' arguments often. Speaking of those keys: the 'time' key is unconditionally overwritten by log.msg, which means that re-publishing an event that happened previously is destructive; there's no way to track historical time. Messages logged before the logging system starts are just lost. It would be good for the initial log observer to keep a (limited-size, of course) buffer of log messages which could be dumped upon failing. This would make it easier, for example, to debug tap plugins that use the log system to report something about their state during makeService. (And, as the maintainer of a phenomenally complex makeService, I can tell you that this would definitely be helpful.) The naming of methods is bad and confusing. msg? err? Easy to type, but hard to understand. We could have equally short names that are not confusing. The contract between parts of the system is poorly specified. For example, trial -j has a hard time with log output that includes any non-bytes keys. I imagine this will only become worse when trial actually supports Python 3. And speaking of encoding, the log system is written in terms of bytes instead of in terms of unicode. Now, there's good reason to be particularly permissive of garbage data in the logging system (logging a message should never provoke an error), but at some point - probably before the data gets to user-written observers - everything should be normalized into a structure that has enough properties that the user can depend on it. On that note, when presented with unicode encoding errors, Twisted's logger will directly raise an exception back to application code, something that it will otherwise only do when presented with KeyboardInterrupt. That I just don't understand. We use old, semi-deprecated '%' formatting style. If we're going to introduce a new API, we should use PEP 3101 formatting; in addition to being new and improved instead of old and inferior, it's a lot easier to type {foo} correctly than to type %(foo)s without forgetting the s. The whole LogContext thing is poorly designed. Mea culpa; this was entirely my fault. One problem with its design was that it was somewhat poorly explained, and so its intent has not carried over years of maintenance. The idea was supposed to be that every event-source within Twisted would provide a useful system key, which would indicate what event-source object (TCP connection, listening port, timed event, UDP port) was being activated when the given message was logged. However, a lot of factors conspire to make this information less than useful. For one thing, especially in database-backed applications, or those that are careful about rate-limiting or flow-control, nothing ever happens except in response to a callFromThread or a callLater; this means that the system argument is always "-". I think that the feature this is trying to support needs to be fundamentally re-thought (and in the intervening decade I've had a lot of ideas about how it might work better). Maybe we can build on some of Chris Armstrong's recent work on Deferred debugging in tracking asynchronous context in Deferreds to help identify the real cause(s) of work being executed. OK, I think that's it for now. (NB: The implementation linked above doesn't address all of the issues I've enumerated in this message, and I don't think that it should address every single one of them before being added to Twisted; writing this one message took me long enough, but we should possibly also discuss which ones are fundamental issues with the logging system and which are features that should be considered separately.) -glyph

12 40

[Twisted-Python] Python 3: bytes vs. str in twisted.python.filepath
by Harry Bock 16 Sep '13

16 Sep '13

Hi all, My name is Harry Bock. I'm interested in helping out porting Twisted to Python 3, and I've popped in IRC a few times to introduce myself and ask a few questions. A few developers agreed that working on trial dependencies would be a big help. In doing some porting work on trial, I stumbled upon a previous porting effort (possibly by Itamar?) for twisted.python.filepath and related modules. It seemed like the porting effort included forcing all pathname inputs to be byte strings instead of native strings. After some investigation, I believe this is the wrong approach, but I wanted to start a discussion here first. Some thoughts: (a) As of Python 3.3, use of the ANSI API in Windows is deprecated[1], so many functions in os and os.path raise DeprecationWarning when given byte strings as input. Although win32 is not an initial target of the porting effort, we'll have to support it eventually and the API should be supported before then. (b) Misunderstandings at the application level about the underlying filesystem's path encoding is not the problem of the Twisted API. Correct me if I'm wrong, but that's the responsibility of the system administrator or individual user (at least on UNIX) to set the LANG environment variable, or for the application to call setlocale(3) to explicitly override it. (c) If we do not allow unicode strings, we will be forcing the application developer to decide how to encode paths when using the FilePath API. Per (b) above, the user will have to call sys.getfilesystemencoding()[2] to divine what encoding to use before using the API at all, which to me is terribly annoying and would just add str.encode calls everywhere. Thus, my vote is that on Python 2.x, Twisted should accept either the native str or unicode types for path names, and on Python 3.x, only accept the str type to prevent deprecation issues with system calls. I have a patch set that will make this happen including unittest modifications; if there's a consensus I'm happy to open a ticket and submit the patches. Thanks! [1] http://bugs.python.org/issue13374 [2] http://docs.python.org/3/library/sys.html#sys.getfilesystemencoding

5 7

[Twisted-Python] SQL ORM for Twisted & PostgreSQL?
by Jeffrey Ollie 21 Aug '13

21 Aug '13

I know that this has been asked before, but it's been a while and I'm hoping for some good news. Is there a SQL ORM that works well with Twisted and PostgreSQL? In particular I'm hoping to find something that works with txpostgres as that's the library I prefer to access the database with. -- Jeff Ollie

9 17

[Twisted-Python] Twisted and Qt5
by Burak Nehbit 12 Aug '13

12 Aug '13

Hi, I have a question and a general guidance request on handling Qt and Twisted together. This is nearly as much a Qt question as it is a Twisted one, so I apologise in advance if this is off–topic. To give a little bit of a background, I am building a desktop GUI application which I am coding using Python and Javascript. The back–end is Python, which makes use of SQLAlchemy, Twisted and several other Python frameworks. The GUI is in Qt, currently linked to Python with PyQt, but I will probably replace it with PySide before release. I am using Qt to draw a QtWebkit frame, in which I run Angular JS to serve up my visual interface, plus a host of other data presentation logic. Which is, it turns out, quite an impasse, because if I use Qt4, the GUI of my application renders in … 12 seconds. Qt5 and its version of QtWebkit loads the same thing in 300 milliseconds. I am therefore forced to use Qt5. This is how I end up with an application where both Twisted and Qt5 are essential requirements. Now, my questions are these: 1) I know there is a Qt4 reactor (which I managed to get to work and pass the twisted test suite) for Twisted. Is there such a thing for Qt5? Is it planned? 2) In the case it's not, are there any alternatives to having a Qt5 reactor? My Twisted knowledge is still woefully inexact (I'm still reading the Twisted book), so I am not very well versed on alternative ways to accomplish this. The one way I can see is to modify the Twisted Qt4 reactor to work with Qt5, but I have almost no knowledge of C++, so this seems far–fetched. This is my only unsolved question about my application— a project of love and learning rather than a commercial one—so if I can figure this out, I'll be well on my way. In case there is nothing to be done about this and a Qt5 reactor is an imperative, I would be willing to finance this to the best of my ability; I'm a student, but I can try to pay whatever is needed to have this reactor done, open sourced—and added to Twisted core, if desired. If there is such a project on its way, I can be a tester. If nobody is interested I would be very happy if you could give me a lead on how to make my own reactor; if I succeed I can release it as open source. Likely I am not a programmer good enough to make that happen, though! Qt is pretty much the only decent GUI framework in existence and almost the only way to have a Python application with a web–stack GUI, which is one of the most popular ways desktop applications are designed these days. If you know of another way to run Webkit as a Python GUI, I'd love to hear that too, but to the extent of my research this looks impossible without Qt. Thanks! Best, Burak

5 10

[Twisted-Python] transport.write performance.
by zipxing 08 Aug '13

08 Aug '13

A simple TCP echo server using epoll reactor:this server process take 60% cpu on 4000 request/s. If use self.transport.getHandle().send instead of self.transport.write,it take 30% cpu on 4000 request/s. Why transport.write take more user cpu?Why twisted performance so poor?(echosvr.c using libevent only take 12% cpu on 4000 request/s) tsvr.py-----------------------------------------------------------import sys, time, random, socket, tracebackfrom twisted.internet import epollreactorepollreactor.install()from twisted.internet import defer, reactor, taskfrom twisted.internet.protocol import Protocol, Factoryfrom protocol import TCPServerProtocol def main(): tcpprotocol = TCPServerProtocol factory = Factory() factory.protocol = tcpprotocol reactor.listenTCP(9976, factory) reactor.run() if __name__ == '__main__': main() protocol.py--------------------------------------------------------- import socketimport datetimeimport tracebackfrom twisted.protocols.basic import LineReceiverfrom twisted.internet import protocol class TCPServerProtocol(LineReceiver): req_count = 0 req_time = datetime.datetime.now() def lineReceived(self, data): TCPServerProtocol.req_count+=1 if TCPServerProtocol.req_count%10000==0: ct = datetime.datetime.now() dt = ct-TCPServerProtocol.req_time pps = 10000/(dt.seconds+dt.microseconds/1000000.0) TCPServerProtocol.req_time=ct print('RPS='+str(pps)) try: #self.transport.write(data) self.transport.getHandle().send(data) except: traceback.print_exc() tcli.py -----------------------------------------------------------------import sysimport socketimport tracebackimport timeimport datetime host = 'localhost'port = 9976loopcount = 300sockcount = 5000RPS = 4000 ss=[]for x in xrange(sockcount): ss.append(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) ss[x].connect((host, port)) ss[x].settimeout(120) for x in xrange(10000000): st = datetime.datetime.now() for y in xrange(loopcount): try: if ss[x%sockcount]!=None: ss[x%sockcount].sendall('1234567890\r\n') ss[x%sockcount].recv(1024) except: print y sys.exit() time.sleep(0.1) dt = (datetime.datetime.now()-st) plc = loopcount/(dt.seconds+dt.microseconds/1000000.0) print loopcount/(dt.seconds+dt.microseconds/1000000.0) #auto adjust RPS if plc<RPS: if RPS-plc>50: loopcount+=10 else: if plc-RPS>50: loopcount-=10 echosvr.c ----------------------------------------------------------------------#include <stdio.h>#include <stdlib.h>#include <errno.h>#include <assert.h> #include <event2/event.h>#include <event2/bufferevent.h> #define LISTEN_PORT 9976#define LISTEN_BACKLOG 32 #ifdef FD_SETSIZE#undef FD_SETSIZE#endif#define FD_SETSIZE 65536 void do_accept(evutil_socket_t listener, short event, void *arg);void read_cb(struct bufferevent *bev, void *arg);void error_cb(struct bufferevent *bev, short event, void *arg);void write_cb(struct bufferevent *bev, void *arg); int main(int argc, char *argv[]){ int ret; evutil_socket_t listener; listener = socket(AF_INET, SOCK_STREAM, 0); assert(listener > 0); evutil_make_listen_socket_reuseable(listener); struct sockaddr_in sin; sin.sin_family = AF_INET; sin.sin_addr.s_addr = 0; sin.sin_port = htons(LISTEN_PORT); if (bind(listener, (struct sockaddr *)&sin, sizeof(sin)) < 0) { perror("bind"); return 1; } if (listen(listener, LISTEN_BACKLOG) < 0) { perror("listen"); return 1; } printf ("Listening...\n"); evutil_make_socket_nonblocking(listener); struct event_base *base = event_base_new(); assert(base != NULL); struct event *listen_event; listen_event = event_new(base, listener, EV_READ|EV_PERSIST, do_accept, (void*)base); event_add(listen_event, NULL); event_base_dispatch(base); printf("The End."); return 0;} void do_accept(evutil_socket_t listener, short event, void *arg){ struct event_base *base = (struct event_base *)arg; evutil_socket_t fd; struct sockaddr_in sin; socklen_t slen; fd = accept(listener, (struct sockaddr *)&sin, &slen); if (fd < 0) { perror("accept"); return; } if (fd > FD_SETSIZE) { perror("fd > FD_SETSIZE\n"); return; } printf("ACCEPT: fd = %u\n", fd); struct bufferevent *bev = bufferevent_socket_new(base, fd, BEV_OPT_CLOSE_ON_FREE); bufferevent_setcb(bev, read_cb, NULL, error_cb, arg); bufferevent_enable(bev, EV_READ|EV_WRITE|EV_PERSIST);} void read_cb(struct bufferevent *bev, void *arg){#define MAX_LINE 256 char line[MAX_LINE+1]; int n; evutil_socket_t fd = bufferevent_getfd(bev); while (n = bufferevent_read(bev, line, MAX_LINE), n > 0) { line[n] = '\0'; //printf("fd=%u, read line: %s\n", fd, line); bufferevent_write(bev, line, n); }} void write_cb(struct bufferevent *bev, void *arg) {} void error_cb(struct bufferevent *bev, short event, void *arg){ evutil_socket_t fd = bufferevent_getfd(bev); printf("fd = %u, ", fd); if (event & BEV_EVENT_TIMEOUT) { printf("Timed out\n"); //if bufferevent_set_timeouts() called } else if (event & BEV_EVENT_EOF) { printf("connection closed\n"); } else if (event & BEV_EVENT_ERROR) { printf("some other error\n"); } bufferevent_free(bev);}

7 19

Twisted July 2013