Mailman 3 July 2003 - Twisted

[Twisted-Python] Postgres client
by Sune Kirkeby July 27, 2003

July 27, 2003

Hello twisted people. People have expressed interest in the postgres client I wrote for twisted waaay back, so here it is. It's got the packet parsing/formatting, and a weird sort of interface, so afaik all it really needs is integration with the rest of twisted (enterprise?), and postgres -> python type mappings (/usr/include/postgresql/server/catalog/pg_type.h has the postgres type identifiers listed). The code is license-less as it is in the patch, but you may consider it licensed under the LGPL, and I give Glyph non-exclusive copyrights to the code. There, that should make it twisted-friendly. I hope. Enjoy, /s -- Sune Kirkeby | If humans were supposed to fly they'd | be born with stewardesses.

1 0

[Twisted-Python] Connection poolin
by John Goerzen July 27, 2003

July 27, 2003

Here is something I haven't been able to figure out yet. I am using IMAP4Client from Twisted. The behavior I used to use is this: When I wanted to speak to the IMAP server in my multithreaded application, I would call a function that returns me an IMAP connection. There was a limit on the number of connections open -- let's say 5 for this example. If an open connection was available, that function would return it to me immediately. If 5 were already open, it would block until one of those was freed. Otherwise, it would establish a new connection and return that to me. My task as a caller was to always release the connection when done. My pattern, then, was generally one of: imap = self.imapserver.acquireconnection() try: # select the appropriate mailbox # do stuff here return result finally: self.imapserver.release(imap) I have been trying to figure out how to do this the Twisted way. Obviously blocking is right out. I've thought that in each method I have that needs a connection, I could write two methods: one that calls the Twisted version of acquireconnection and sets its callback to the second, which does the real work (and possibly chains to others via callbacks.) There are some problems with that approach. One is that it is very verbose and requires a lot of code all over the place to manage. Another is that there's no easy way to tell in Twisted when we are done with the given connection. I suppose I could always add errbacks all along the way plus a callback, each of which releases the connection, but that seems inconvenient and error-prone. Basically, what I'm saying is that I can invent something that works, but everything I've come up with seems like a nasty kludge, and I'm looking for the Right Way to do it. Thanks, John

2 1

[Twisted-Python] How can ftpclient.py work?
by John Goerzen July 27, 2003

July 27, 2003

Hi, I've been reading over the Twisted examples to learn the system and one of them, ftpclient.py, has me flummoxed. Specifically, the connectionMade function near the end. It issues a number of commands right in a row, without waiting for the results to come back from the deferreds that they return. To me this seems bad because: 1. The system would have no way of knowing which command a particular response belongs to 2. Sending additional commands before the server is done with the first one may cause problems (for instance, if it takes a little while to fully transmit the first command, the second one could be transmitted in the middle) 3. There is no error-checking in there, so commands are issued that are dependant on the success of earlier ones (rnuning nlst after cdup, for instance), but they're issued potentially before the earlier ones have a chance to return an error. Now, I'm assuming that these problems do not actually exist. Yet I cannot work out why not. Can anybody shed some light? Thanks, John

3 4

[Twisted-Python] Help w/ low adbapi connection pool limits?
by Steve Freitas July 27, 2003

July 27, 2003

Hi all, I'm new to Twisted, and it's about to show. :-) I hope you can help me. I'm writing a PostgreSQL-backed XML-RPC server with Twisted 1.0.6 on Linux. My problem is that if I hammer it with repeated, high-speed queries from a client, the client receives a Fault 8002. I'm probably just not setting some limit variable somewhere high enough. Here's my server code: from twisted.web import xmlrpc, server from twisted.enterprise import adbapi class takeOne(xmlrpc.XMLRPC): def xmlrpc_dbtest(self): dbpool = adbapi.ConnectionPool("pyPgSQL.PgSQL", host='127.0.0.1', \ user='postgres', database='template1') dbpool.connect() d = adbapi.Augmentation(dbpool) result = d.runQuery("select * from auth where uname='foo'") dbpool.close() return result (In an earlier iteration I'd omitted the "dbpool.close()" call. Putting that in made it work considerably better, but it still fails under heavy load.) Individual queries from the interactive interpreter work fine: [steve@lothar]$ python >>> import xmlrpclib >>> s = xmlrpclib.Server('http://localhost:7080/') >>> s.dbtest() [{'baseObj': ['foo', 'd00d']}] But when I hit it hard for a simple load test, it goes for a few seconds and then falls over: >>> while i < 500: ... x = s.dbtest() ... i = i + 1 ... Traceback (most recent call last): File "<stdin>", line 2, in ? File "/usr/lib/python2.2/xmlrpclib.py", line 821, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.2/xmlrpclib.py", line 975, in __request verbose=self.__verbose File "/usr/lib/python2.2/xmlrpclib.py", line 853, in request return self.parse_response(h.getfile()) File "/usr/lib/python2.2/xmlrpclib.py", line 896, in parse_response return u.close() File "/usr/lib/python2.2/xmlrpclib.py", line 571, in close raise apply(Fault, (), self._stack[0]) xmlrpclib.Fault: <Fault 8002: 'error'> Afterwards the connection pool seems to still be available: >>> s.dbtest() [{'baseObj': ['foo', 'd00d']}] But I've got lots of connections in process still: [steve@lothar]$ netstat -atn tcp 0 0 127.0.0.1:5432 127.0.0.1:48841 TIME_WAIT tcp 0 0 127.0.0.1:7080 127.0.0.1:48839 TIME_WAIT (repeat these guys about 150 times each, plus several being serviced) The connections eventually go away as (I presume) they get serviced. So I expect I'm just hitting a database connection pooling limitation, as a similar test against an XML-RPC function which didn't access a database ran just fine. I'm not sure if the limitation is in Twisted, or in PyPgSQL, and I don't know enough about how to enable traceback logging in Twisted (is there any?) to tell. Ideally I'd like it to handle several thousand backlogged requests without burping. Help! :-) Thanks for your time, Steve Freitas

3 6

[Twisted-Python] submodel caching in woven
by Michael Shyne July 26, 2003

July 26, 2003

I'm trying to create a web-app in woven using already existing data objects. I'm starting by just trying to display a name. This is what I've done: components.registerAdapter(model.ObjectWrapper, myObj, interfaces.IModel) class DisplayObj(page.Page): templateFile = "display-obj.html" def wmfactory_obj(self, request): id = request.args.get('id', (0,))[0] if id: return objFactory.getObj(id) return None The template only referes to "obj/name". This all works fine... except when I change the id in the URL and reload the page. The name displayed doesn't change. The problem is with the submodel caching in the root model (the page). I've attached the changes I made to fix it. I'm new to woven (and twisted in general) so I don't know if I'm doing something wrong, or if my changes have far-reaching effects. I should note that these changes merely improved the situation to mostly working (which, of course, is much better than never working). The caching is keyed off of "id(request)". However, it seems there is object caching occuring with the request objects, "id(request)" often returns identical results. I believe that the submodel caching will also have to look at a timestamp to work consistently. This was too big of a change to make at this point. But if the list agrees with my current changes and thinks adding a timestamp comparison is the right way to go, I would be more than happy to do it. -Michael Shyne

1 0

[Twisted-Python] ProxyClient error using ReverseProxy
by David McCurley July 25, 2003

July 25, 2003

I'm working on full reverse proxy functionality and have run into an error (twisted 1.0.6) during some basic testing. SYNOPSIS: - Made a root ReverseProxyResource that goes to google from http://localhost:8080/ in my web.tap - Using Mozilla 1.4, hit the local url, all is okay - Now click the refresh button and an exception is thrown in the console window running the tap. - Reload again and no error, reload again and exception, repeatable every other time. I'm not clear on the full call details but the refresh that causes the exception has a request already queued, so when the ProxyClient.handleRequest calls the self.father.transport.loseConnection method, the exception triggers because the transport is a StringTransport in the queued request and it doesn't have a "loseConnection" method. I've been trying to trace this to submit a patch, but its not clear to me what the queuing is for or why this is happening, or when to call loseConnection. DETAILS: Made an .rpy file and a web tap that made it the root resource: t1.rpy =========================== import twisted.web.proxy resource = twisted.web.proxy.ReverseProxyResource('www.google.com', 80, '') =========================== In mozilla 1.4 get http://localhost:8080/ Console output: ==================== 2003/07/25 19:03 EDT [HTTPChannel,0,127.0.0.1] Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x840eccc> 2003/07/25 19:03 EDT [ProxyClient,client] Stopping factory <twisted.web.proxy.ProxyClientFactory instance at 0x840eccc> 2003/07/25 19:03 EDT [HTTPChannel,1,127.0.0.1] Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x8416fb4> ==================== Now click reload after a few seconds (Mozilla icon stays spinning this time like it has an open connection its waiting on): Console output: ==================== 2003/07/25 19:04 EDT [HTTPChannel,1,127.0.0.1] Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x8411b3c> 2003/07/25 19:04 EDT [ProxyClient,client] Traceback (most recent call last): File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 472, in doSelect _logrun(selectable, _drdw, selectable, method, dict) File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 65, in callWithLogger callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib/python2.2/site-packages/twisted/python/log.py", line 52, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib/python2.2/site-packages/twisted/python/context.py", line 32, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib/python2.2/site-packages/twisted/internet/default.py", line 481, in _doReadOrWrite why = getattr(selectable, method)() File "/usr/lib/python2.2/site-packages/twisted/internet/tcp.py", line 216, in doRead return self.protocol.dataReceived(data) File "/usr/lib/python2.2/site-packages/twisted/protocols/basic.py", line 184, in dataReceived return self.rawDataReceived(data) File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 354, in rawDataReceived self.handleResponseEnd() File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 336, in handleResponseEnd self.handleResponse(b) File "proxy.py", line 72, in handleResponse File "/usr/lib/python2.2/site-packages/twisted/protocols/http.py", line 285, in __getattr__ return getattr(self.__dict__['s'], attr) exceptions.AttributeError: loseConnection 2003/07/25 19:04 EDT [ProxyClient,client] Stopping factory <twisted.web.proxy.ProxyClientFactory instance at 0x8411b3c> ==================== Reload again and this time the following output happens at the console and the Mozilla icon stops spinning like all connections are closed. Console output: ==================== 2003/07/25 19:04 EDT [HTTPChannel,2,127.0.0.1] Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x84266ac> 2003/07/25 19:04 EDT [ProxyClient,client] Stopping factory <twisted.web.proxy.ProxyClientFactory instance at 0x84266ac> 2003/07/25 19:04 EDT [HTTPChannel,3,127.0.0.1] Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x84454a4> ==================== Reload again and it starts spinning and gives same type of exception trace as above. Any pointers on where to look or how to fix this? Thanks, David

1 0

[Twisted-Python] Using ResourceSubscription with vhost
by Alex Levy July 25, 2003

July 25, 2003

I am using twisted-web to serve a number of virtual hosts. I'd like one of these virtual hosts to point to a ResourceSubscription I've got set up. The code in the first example works, the second does not. --- This works - /etc/twisted-web/local.d/50vhost.py --- rs = distrib.ResourceSubscription('unix', 'socketpath') file = static.File('/somewhere') file.putChild('rs', rs) root.addHost('my.domain.com', file) --- This doesn't --- rs = distrib.ResourceSubscription('unix', 'socketpath') root.addHost('my.domain.com', rs) --- snip snip --- If I use the second example, navigating to http://my.domain.com/anything results in a 'No such child resource' error message. So, am I missing something that should be more obvious, or is this a real bug? -- Alex Levy WWW: http://mesozoic.geecs.org "Never let your sense of morals prevent you from doing what is right." -- Salvor Hardin, Isaac Asimov's _Foundation_

1 0

[Twisted-Python] Mail Server
by David A. Leedom July 25, 2003

July 25, 2003

I am new to Twisted Matrix. I am trying to setup an mail system that will use a database as the datastore for emails, domains and user access (both pop3/imap and stmp). What I am looking for is a code snipit example of how to setup a POP3 and SMTP server using frame work without using the TAP utility. I figure once I have this example then I can begin to understand the documentation better. There is something missing in the explanation of how things work. It just goes over my head. Thanks You, Dave Leedom The Hightower Group, Inc. Information Technology Consulting 165 West Airport Road/Lititz, PA 17543 V:717-560-4002, 877-560-4002 x: 114 F:717-560-2825 www.hightowergroup.com --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.498 / Virus Database: 297 - Release Date: 7/8/2003

7 8

[Twisted-Python] unlistenTCP not working?
by Michael Porter July 25, 2003

July 25, 2003

Hi, I'm having a problem with unlistenTCP - it doesn't seem to work if application.listenTCP was called after the reactor is started. Am I missing something here or is this a bug? See test code below for demonstration... def listen(application, port): from twisted.web import server, resource print "called listen %d" % port application.listenTCP(port, server.Site(resource.Resource())) def unlisten(application, port): print "called unlisten %d" % port application.unlistenTCP(port) def test(listenFirst): from twisted.python import log import sys log.startLogging(sys.stdout, 0) from twisted.internet import app, reactor application = app.Application("test") if listenFirst: listen(application, 8080) else: reactor.callLater(0, listen, application, 8080) reactor.callLater(5, unlisten, application, 8080) application.run(save=0) If you call test(listenFirst = 1) then all works as expected (log shows it starts listening immediately and stops after 5 secs). But if you start listening by scheduling a call after application.run (call test(listenFirst = 0)) the listenTCP works but when unlisten is called 5 secs later it doesn't seem to stop listening and nothing appears in the log. (I'm running on Windows with twisted 1.0.6) Any ideas? Thanks, Mike.

4 5

[Twisted-Python] Patch for thinko/typo in components howto
by Stephen C. Waterbury July 24, 2003

July 24, 2003

The CVS version is slightly different from what's on the web, but both have this error. (The one that's on the web additionally has the error that "the method definitions inside the [Interface] class block do not have any self parameters", but this has of course been fixed in CVS ...) Steve. Index: components.html =================================================================== RCS file: /cvs/Twisted/doc/howto/components.html,v retrieving revision 1.6 diff -c -u -r1.6 components.html --- components.html 18 Jul 2003 06:33:59 -0000 1.6 +++ components.html 24 Jul 2003 16:30:44 -0000 @@ -263,7 +263,7 @@ from twisted.python import components class IAmericanSocket(components.Interface): - def voltage(): + def voltage(self): """Return the voltage produced by this socket object, as an integer. """

2 1