Mailman 3 August 2004 - Twisted

[Twisted-Python] Question on using processes
by Matt Feifarek Aug. 10, 2004

Aug. 10, 2004

I'm using twisted.internet.utils.getProcessOutput() to run commands from a reactor. I've pretty much got this working okay (with a few false starts related to how args are put together... you can't send in a big long string; you literally have to make a list of argument words... what a PITA). Now, I've got a command that needs to pipe something through another command, something like this: /bin/cat foo.txt | /usr/bin/tail If I set up my "process" like this: CMD = "/bin/cat" ARGS = [ "foo.txt", "|", "/usr/bin/tail" ] d = utils.getProcessOutput( CMD, ARGS ) It doesn't work. Now, this is a stupidly-trivial example, but it seems to have trouble with thinking that the pipe symbol is a filename. My less trivial example merely hangs up the console and does nothing. I figured that this had something to do with the pipe (obviously) so I tried going deeper and using reactor.spawnProcess... This is a little over my head, but I tried using an instance of twisted.internet.protocol.ProcessProtocol as the protocol, and turning on usePTY. Still, no dice. So, my next theory is that it has something to do with the way that the process is literally launched. This is extremely low level, and has resisted my attempts to figure it out even with a debugger and stepping through. My question: what's the best way to go about this? Options: - write my own Process protocol somehow? - figure out what child descriptors may be required to fix this? - simply make a bash script that wraps up the pipe inside of a black box and make the reactor call that? Thanks!

5 4

[Twisted-Python] xml-rpc with cred authentication
by Guy Robinson Aug. 10, 2004

Aug. 10, 2004

Hello, I'm trying to write an application framework around Twisted xmlrpc but with authentication. Cred seems to be what I need but I have the following (newbie) questions: Is my understanding correct? I need to write the following: My understanding is the portal is used in defining the protocol? So if xmlrpc is a 'protocol' on top of http does http or xmlrpc already have a portal implemented for it? Or is my understanding all wrong? The CredentialChecker as I want to store the username/password/credentials in zodb or atop The Realm as this is what xmlrpc will use to get a user's credentials. TIA, Guy ps: If anyone can point me to a twisted xmlrpc implementation with authentication it would be appreciated. (no point reinventing the wheel)

2 2

[Twisted-Python] module interface checking tool
by James Y Knight Aug. 10, 2004

Aug. 10, 2004

So, I was a bit irritated recently because there is no way to test for private name usage in a module's namespace. '__all__' is a good start, but it is somewhat poor for a few reasons: 1) it doesn't affect import modname; modname.someprivatename at all so you may be using private interfaces without having any way of knowing. 2) Because it doesn't actually affect code people write, it often doesn't get updated when new public interfaces get added. 3) until recently it didn't even affect help(module), so users don't even know they're using a private name. So, I wrote a tool to do the access checking. My goal is to have it used for testing (solving problems 1&2 *within* twisted, at least), not as something that you'd run your production server under. sandbox/foom/private.py. Currently, it just spits out warnings because that way I can actually run through all the twisted tests, but that's controllable as a flag. Ideally every module in twisted would define __all__ and we could run the entire testsuite with access control enabled without raising illegal access exceptions. To run the twisted tests with this, I temporarily stuck the following in twisted/__init__.py: import private private.privatizeModulesWithPrefix('twisted') Much of the output (attached at the end) is from the IMAP tests trying to access private bits in order to test them. I'm not sure exactly what to do about that -- tests _should_ sometimes be able to access private parts, but they should also be restricted, normally, because you want to test against the public interface as well. So I may have to create a way to annotate some tests as being able to access private attributes while disallowing it by default. Also, rebuild accesses private classes, and it is supposed to, so should be given special dispensation. Besides those, the other things that show up currently are: twisted.internet.abstract.isIPAddress from twisted.internet.iocpreactor.tcp (?) twisted.internet.protocol.ConsumerToProtocolAdapter from twisted.protocols.ftp (?) twisted.internet.threads._putResultInDeferred from twisted.enterprise.adbapi (_deferToThread) Some of those perhaps should be public (?), but _putResultInDeferred, probably should not. IMO these are bugs, either of omission in __all__ or access from an inappropriate place. I dunno which. Also, only 35 out of 559 modules in twisted use __all__. That's sad, and should be fixed. I'm sure if that was done, more of these kinds of things would show up. Here's the full list of accesses to private names (I took the output and sort|uniq'd it). '=> Yes' means access was granted based on the two modules living in the same package. '=> No' means it would have raised an exception if I had that enabled. <= private twisted.internet.abstract.isIPAddress from twisted.internet.iocpreactor.tcp (?): => No <= private twisted.internet.protocol.BaseProtocol from twisted.python.rebuild (latestClass): => No <= private twisted.internet.protocol.ConsumerToProtocolAdapter from twisted.protocols.ftp (?): => No <= private twisted.internet.threads._putResultInDeferred from twisted.enterprise.adbapi (_deferToThread): => No <= private twisted.mail.imap4.Command from twisted.mail.test.test_imap (append): => No <= private twisted.mail.imap4.DontQuoteMe from twisted.mail.test.test_imap (testQuoteAvoider): => No <= private twisted.mail.imap4.FileProducer from twisted.mail.test.test_imap (testFileProducer): => No <= private twisted.mail.imap4.MessageProducer from twisted.mail.test.test_imap (testMultipleMultiPart): => No <= private twisted.mail.imap4.MessageProducer from twisted.mail.test.test_imap (testSingleMultiPart): => No <= private twisted.mail.imap4.MessageProducer from twisted.mail.test.test_imap (testSinglePart): => No <= private twisted.mail.imap4.MessageSet from twisted.mail.test.test_imap (?): => No <= private twisted.mail.imap4.MessageSet from twisted.mail.test.test_imap (testSetFlags): => No <= private twisted.mail.imap4._FetchParser from twisted.mail.test.test_imap (testFetchParserBody): => No <= private twisted.mail.imap4._FetchParser from twisted.mail.test.test_imap (testFetchParserMacros): => No <= private twisted.mail.imap4._FetchParser from twisted.mail.test.test_imap (testFetchParserSimple): => No <= private twisted.mail.imap4._formatHeaders from twisted.mail.test.test_imap (testFetchHeaders): => No <= private twisted.mail.imap4._formatHeaders from twisted.mail.test.test_imap (testHeaderFormatter): => No <= private twisted.mail.imap4.collapseNestedLists from twisted.mail.test.test_imap (testFiles): => No <= private twisted.mail.imap4.collapseNestedLists from twisted.mail.test.test_imap (testQuoteAvoider): => No <= private twisted.mail.imap4.collapseStrings from twisted.mail.test.test_imap (testStringCollapser): => No <= private twisted.mail.imap4.decoder from twisted.mail.test.test_imap (testDecode): => No <= private twisted.mail.imap4.parseIdList from twisted.mail.test.test_imap (testIdListParser): => No <= private twisted.mail.imap4.parseNestedParens from twisted.mail.test.test_imap (_searchWork): => No <= private twisted.mail.imap4.parseNestedParens from twisted.mail.test.test_imap (testLiterals): => No <= private twisted.mail.imap4.parseNestedParens from twisted.mail.test.test_imap (testParenParser): => No <= private twisted.mail.imap4.splitQuoted from twisted.mail.test.test_imap (testQuotedSplitter): => No <= private twisted.mail.imap4.wildcardToRegexp from twisted.mail.test.test_imap (testWildcard): => No <= private twisted.mail.imap4.wildcardToRegexp from twisted.mail.test.test_imap (testWildcardNoDelim): => No <= private twisted.persisted.dirdbm.ALLOW_TWISTED_REBUILD from twisted.python.rebuild (rebuild): => No <= private twisted.web.xmlrpc.addIntrospection from twisted.web.test.test_xmlrpc (?): => No <= private twisted.internet.abstract.isIPAddress from twisted.internet.base (resolve): => Yes <= private twisted.internet.abstract.isIPAddress from twisted.internet.tcp (resolveAddress): => Yes <= private twisted.internet.abstract.isIPAddress from twisted.internet.udp (connect): => Yes <= private twisted.internet.abstract.isIPAddress from twisted.internet.udp (startListening): => Yes <= private twisted.internet.base.BaseConnector from twisted.internet.ssl (?): => Yes <= private twisted.internet.base.BaseConnector from twisted.internet.ssl (__init__): => Yes <= private twisted.internet.base.BaseConnector from twisted.internet.tcp (?): => Yes James

4 10

[Twisted-Python] stopTrying, cancel connection in progress
by Bosnjak Zoran ITWEP Aug. 10, 2004

Aug. 10, 2004

I have a subclass of ReconnectingClientFactory. If I do - reactor.connectTCP(ip, port, myFactory) and - myFactory.stopTrying() - del myFactory ... it seems like stopTrying does nothing. When the connection is finally made (for the first time), myFactory.buildProtocol is called and new protocol instance is created... So, how do I cancel connection in progress, so that myProtocol is never created? Thanks! Zoran Bosnjak

1 0

[Twisted-Python] "Twisted Web" Book
by Ed Suominen Aug. 9, 2004

Aug. 9, 2004

Glad to see there's a book, but why isn't it under the O'Reilly name? Hmmm... http://tinyurl.com/4adsh Ed Suominen Registered Patent Agent Open-Source Software Author (yes, both...) Web Site: http://www.eepatents.com

2 1

[Twisted-Python] Problem with SYN_SENT's under SCO
by Richard.Sharp＠t-online.de Aug. 9, 2004

Aug. 9, 2004

Hello, I recently had a problem under SCO that I had to solve. The program was piling up open socket objects all hanging in the state of SYN_SENT. Eventually the maximum number of open files was exceeded and the application hung badly. I eventually narrowed the problem down with a perfectly normal python program to trying to connect as a client to a server that is not listening. The first connect returns the error EINPROGRESS, the second connect then returns the error "Connection refused". Running a simple program that repeatedly tried to connect, I could use "netstat -a" to watch the SYN_SENT's increasing. This only appears to happen under SCO and even here I am not sure whether it is not version specific. I discovered that closing the socket object after the "connection refused" seemed to release any open objects and removed the SYN_SENTS, so I changed failIfNotConnected to what you see below. There may be a better or more appropriate solution but this has at least got rid of my problem. Version 1.3.0 internet/tcp.py from line 355: def failIfNotConnected(self, err): if sys.platform[0:3] != 'sco': if (self.connected or self.disconnected or not (hasattr(self, "connector"))): return else: self.socket.close() if not (hasattr(self,"connector")): return self.connector.connectionFailed(failure.Failure(err)) if hasattr(self, "reactor"): # this doesn't happens if we failed in __init__ self.stopReading() self.stopWriting() del self.connector greetings, Richard Sharp

1 0

[Twisted-Python] How to write to log in the context of a tap-builder?
by Stephen Waterbury Aug. 7, 2004

Aug. 7, 2004

Seems that twistd controls the log while it runs the stuff in the tap, so if I do a log.msg('blah') in there it's ignored ... is there a way to "borrow" the log from twistd and write something to it (like have my service say something when it starts up)? Steve

1 0

[Twisted-Python] [p2p-hackers] source code (in python, using twisted) to my reliable udp rpc protocol (fwd from angryhicKclown@netscape.net)
by Eugen Leitl Aug. 7, 2004

Aug. 7, 2004

----- Forwarded message from angryhicKclown(a)netscape.net ----- From: angryhicKclown(a)netscape.net Date: Sat, 07 Aug 2004 14:23:26 -0400 To: p2p-hackers(a)zgp.org Subject: [p2p-hackers] source code (in python, using twisted) to my reliable udp rpc protocol X-Mailer: Atlas Mailer 2.0 Reply-To: "Peer-to-peer development." <p2p-hackers(a)zgp.org> I was paging thru the archives and realized I never posted it. Works for me, but I've never done a large-scale test of it. Perhaps someone would like to and share the results? It consists of several layers, a stack, if you will: Reliable datagram - handles acks, etc Datagram stream - allows one to send messages larger than the MTU CBOB - binary format for storing structured data, similar to xml, except it doesn't suck TinyRPC - RPC protocol build upon all of those layers Enjoy. Requires www.twistedmatrix.com. ---- cut here ---- # TinyRPC protocol # bsd license from twisted.internet import reactor, protocol, interfaces, defer from twisted.protocols import policies import socket import struct import binascii import cStringIO as StringIO import time CBOB_BOOL = 0 CBOB_INT = 1 CBOB_FLOAT = 2 CBOB_STR = 3 CBOB_USTR = 4 CBOB_TUPLE = 5 CBOB_LIST = 6 CBOB_DICT = 7 CBOB_OBJ = 8 CBOB_NONE = 9 def cbob_encode(obj, buf=StringIO.StringIO()): if isinstance(obj, bool): if obj: buf.write(struct.pack("! BB", CBOB_BOOL, 1)) else: buf.write(struct.pack("! BB", CBOB_BOOL, 0)) elif isinstance(obj, int): buf.write(struct.pack("! Bi", CBOB_INT, obj)) elif isinstance(obj, float): buf.write(struct.pack("! Bf", CBOB_FLOAT, obj)) elif isinstance(obj, str): buf.write(struct.pack("! BH", CBOB_STR, len(obj)) + obj) elif isinstance(obj, unicode): buf.write(struct.pack("! BH", CBOB_USTR, len(obj)) + obj) elif isinstance(obj, tuple): buf.write(struct.pack("! BB", CBOB_TUPLE, len(obj))) for o in obj: cbob_encode(o, buf) elif isinstance(obj, list): buf.write(struct.pack("! BH", CBOB_LIST, len(obj))) for o in obj: cbob_encode(o, buf) elif isinstance(obj, dict): buf.write(struct.pack("! BH", CBOB_DICT, len(obj))) for o in obj: cbob_encode(o, buf) cbob_encode(obj[o], buf) elif isinstance(obj, object): members = obj.__dict__.items() #inspect.getmembers(obj) buf.write(struct.pack("! BHB", CBOB_OBJ, len(members), len(obj.__class__.__name__))) buf.write(obj.__class__.__name__) for member in members: cbob_encode(member[0], buf) cbob_encode(member[1], buf) elif isinstance(obj, NoneType): buf.write(struct.pack("! B", CBOB_NONE)) else: raise "couldn't serialize " + str(obj) def cbob_decode(buf): d = buf.read(1) typ = struct.unpack("! B", d)[0] if typ == CBOB_BOOL: v = struct.unpack("! B", buf.read(1))[0] return v == 1 elif typ == CBOB_INT: return struct.unpack("! i", buf.read(4))[0] elif typ == CBOB_FLOAT: return struct.unpack("! f", buf.read(4))[0] elif typ == CBOB_STR: l = struct.unpack("! H", buf.read(2))[0] return buf.read(l) elif typ == CBOB_USTR: l = struct.unpack("! H", buf.read(2))[0] return unicode(buf.read(l)) elif typ == CBOB_TUPLE: l = struct.unpack("! B", buf.read(1))[0] v = () for i in range(0, l): v = v + (cbob_decode(buf),) return v elif typ == CBOB_LIST: l = struct.unpack("! H", buf.read(2))[0] v = [] for i in range(0, l): v.append(cbob_decode(buf)) return v elif typ == CBOB_DICT: l = struct.unpack("! H", buf.read(2))[0] v = {} for i in range(0, l): key,value = cbob_decode(buf),cbob_decode(buf) v[key] = value return v elif typ == CBOB_OBJ: membercount,clsnamesize = struct.unpack("! HB", buf.read(3)) clsname = buf.read(clsnamesize) v = globals()[clsname]() for i in range(0, membercount): name,value = cbob_decode(buf),cbob_decode(buf) setattr(v, name, value) return v elif typ == CBOB_NONE: return None else: raise "invalid typecode " + typ class DispatcherTransport: __implements__ = interfaces.IUDPConnectedTransport def __init__(self, dispatcher, addr, proto): self.dispatcher = dispatcher self.addr = addr self.protocol = proto def getHost(self): return self.dispatcher.getHost() def getPeer(self): return ("INET",) + self.addr def write(self, packet): return self.dispatcher.transport.write(packet, self.addr) def loseConnection(self): self.protocol.stopProtocol() del self.protocol class Dispatcher(protocol.DatagramProtocol): # TODO: call startFactory()? def __init__(self, factory): self.handlers = {} factory.dispatcher = self self.factory = factory def __getitem__(self, addr): if addr not in self.handlers: return self.open_new(addr) return self.handlers[addr] def datagramReceived(self, data, addr): self[addr].datagramReceived(data) def getHost(self): return self.transport.getHost() def open_new(self, addr): assert addr not in self.handlers, "Already connected" p = self.factory.buildProtocol(addr) p.transport = DispatcherTransport(self, addr, p) p.startProtocol() self.handlers[addr] = p return p def startProtocol(self): return self.factory.startFactory() def stopProtocol(self): return self.factory.stopFactory() class ReliableDatagramProtocol(protocol.ConnectedDatagramProtocol): "acks and stuff" NUM_RETRANSMITS = 2 #10 RETRANSMIT_INTERVAL = .200 MAX_LAST_RECEIVED = 50 OP_SEND = 0 OP_ACK = 128 # msb set def startProtocol(self): self.retransmits = {} # crc->(data,num retransmits, bits, delayedcall for retransmit, Deferred when packet is acked) self.last_received = [] def stopProtocol(self, reason): print "protocol stopped",reason def send(self, data, bits=0): crc = binascii.crc32(data) d = defer.Deferred() self.retransmits[crc] = (data, self.NUM_RETRANSMITS, bits, reactor.callLater(self.RETRANSMIT_INTERVAL, self.retransmit, crc), d) self.send_packet(self.OP_SEND, bits, data) return d def retransmit(self, crc): t = self.retransmits[crc] self.send_packet(self.OP_SEND, t[2], t[0]) if t[1] > 0: self.retransmits[crc] = (t[0], t[1] - 1, t[2], reactor.callLater(self.RETRANSMIT_INTERVAL, self.retransmit, crc), t[4]) else: self.packetLost(retransmits[crc][0]) del self.retransmits[crc] def send_packet(self, op, bits, data): self.transport.write(struct.pack("! B", op | bits) + data) def send_ack(self, crc): if len(self.last_received) >= self.MAX_LAST_RECEIVED: del self.last_received[0] self.last_received.append(crc) self.transport.write(struct.pack("! Bi", self.OP_ACK, crc)) def got_ack(self, crc): self.retransmits[crc][3].cancel() d = self.retransmits[crc][4] del self.retransmits[crc] d.callback(crc) def datagramReceived(self, data): opbyte, = struct.unpack("! B", data[0]) op = opbyte & 128 data = data[1:] if op == self.OP_SEND: crc = binascii.crc32(data) if crc not in self.last_received: self.dataReceived(data, opbyte) self.send_ack(crc) else: self.got_ack(struct.unpack("! i", data)[0]) def dataReceived(self, data, bits): print "received",data,"from",self.transport.getPeer() def packetLost(self): self.transport.loseConnection() class ReliableDatagramStreamMessageProtocol(ReliableDatagramProtocol): "Sends large messages (> mtu size) and sends them in a buffered block..." MTU = 512 OP_BEGINMSG = 64 # 2nd msb set OP_DATA = 0 def startProtocol(self): ReliableDatagramProtocol.startProtocol(self) self.outmessagequeue = [] # data len, stream, bits self.inmessage = None self.packetcount = 0 self.numpackets = 0 def sendMessage(self, data, bits=0): self.outmessagequeue.append((len(data), StringIO.StringIO(data), bits)) if len(self.outmessagequeue) == 1: self.send_first_block() def send_first_block(self, dummy=""): length,stream,bits = self.outmessagequeue[0] length = length - self.MTU + 4 if length < 0: length = 0 length = length + 1 self.send(struct.pack("! I", length / self.MTU + length % self.MTU) + stream.read(self.MTU - 4), self.OP_BEGINMSG | bits).addCallback(self.send_next_block) def send_next_block(self, dummy=""): length,stream,bits = self.outmessagequeue[0] d = stream.read(self.MTU) if len(d) > 0: self.send(d, self.OP_DATA | bits).addCallback(self.send_next_block) else: stream.close() del self.outmessagequeue[0] if len(self.outmessagequeue) > 0: self.send_next_block() def dataReceived(self, data, bits): if bits & 64 == self.OP_DATA: self.packetcount = self.packetcount + 1 self.inmessage.write(data) else: self.inmessage = StringIO.StringIO() self.packetcount = 1 mtu = len(data) # the entire packet size is the size of the mtu self.numpackets = struct.unpack("! I", data[:4])[0] self.inmessage.write(data[4:]) if self.packetcount >= self.numpackets: self.messageReceived(self.inmessage.getvalue(), bits) self.inmessage = StringIO.StringIO() def messageReceived(self, message, bits): print "got message", message class DatagramRPCProtocol(ReliableDatagramStreamMessageProtocol): OP_CALL = 32 OP_RETURN = 0 MAX_CALLID = 512 def startProtocol(self): ReliableDatagramStreamMessageProtocol.startProtocol(self) if hasattr(self.factory, "buildRemote"): self.remotes = self.factory.buildRemote() self.calls = {} self.callid = 0 def messageReceived(self, message, bits): message = StringIO.StringIO(message) if bits & 32 == self.OP_CALL: t, objname, methodname, args, kwargs = cbob_decode(message) r = getattr(self.remotes[objname], methodname)(*args,**kwargs) #print "calling " + objname + "." + methodname if isinstance(r, defer.Deferred): r.addCallback(lambda result: self.send_result(t, result)) else: self.send_result(t, r) else: t,r = cbob_decode(message) self.calls[t].callback(r) def send_result(self, t, r): sio = StringIO.StringIO() cbob_encode((t, r), sio) self.sendMessage(sio.getvalue(), self.OP_RETURN) def callRemote(self, objname, methodname, *args, **kwargs): t = self.callid if self.callid == self.MAX_CALLID: self.callid = 0 else: self.callid = self.callid + 1 self.calls[t] = defer.Deferred() sio = StringIO.StringIO() cbob_encode((t, objname, methodname, args, kwargs), sio) self.sendMessage(sio.getvalue(), self.OP_CALL) return self.calls[t] #class LazyRMIProtocol(DatagramRPCProtocol): # raise "Todo: implement" class TestObject: def hello(self, name): return "Hello, " + name.first + " " + name.last class Name: first = "" last = "" def test(): reactor.callLater(0, test2) reactor.run() def test2(): f = protocol.ServerFactory() #f.protocol = ReliableDatagramProtocol #f.protocol = ReliableDatagramStreamMessageProtocol f.protocol = DatagramRPCProtocol f.buildRemote = lambda: {"myObj" : TestObject()} d = Dispatcher(f) d2 = Dispatcher(f) p = d[socket.gethostbyname("localhost"), 8889] p2 = d[socket.gethostbyname("localhost"), 8888] reactor.listenUDP(8888, d) reactor.listenUDP(8889, d2) reactor.callLater(0, test3, p, p2) def test3(p, p2): name = Name() name.first = "great" name.last = "world" p.callRemote("myObj", "hello", name).addCallback(test4) def test4(r): print "result of call", r if __name__ == "__main__": test() __________________________________________________________________ Switch to Netscape Internet Service. As low as $9.95 a month -- Sign up today at http://isp.netscape.com/register Netscape. Just the Net You Need. New! Netscape Toolbar for Internet Explorer Search from anywhere on the Web and block those annoying pop-ups. Download now at http://channels.netscape.com/ns/search/install.jsp _______________________________________________ p2p-hackers mailing list p2p-hackers(a)zgp.org http://zgp.org/mailman/listinfo/p2p-hackers _______________________________________________ Here is a web page listing P2P Conferences: http://www.neurogrid.net/twiki/bin/view/Main/PeerToPeerConferences ----- End forwarded message ----- -- Eugen* Leitl <a href="http://leitl.org">leitl</a> ______________________________________________________________ ICBM: 48.07078, 11.61144 http://www.leitl.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE http://moleculardevices.org http://nanomachines.net

1 0

[Twisted-Python] intermittent traceback using adbapi with pyPgSQL
by Stephen Waterbury Aug. 7, 2004

Aug. 7, 2004

Twisted gurus, I didn't want to create a bug report yet because I'm not sure that it's a Twisted bug. I have only observed it while using pyPgSQL as the adaptor; haven't seen it yet (and have run the exact same code that produces it) with psycopg. I see this traceback intermittently occur at the end of chains of deferreds that include 2 or more adbapi calls. I can provide some example code, but I should really develop a concise example (as opposed to my app code, which is spread over 2 or 3 modules) that might be useful as the basis for a unit test in case this is a Twisted problem -- it's not clear to me where the problem is: my code, Twisted, or pyPgSQL. My natural assumption would be the former, but the intermittency and the adaptor-dependence makes me wonder. I can run the exact same code several times without incident and then suddenly get this traceback, which makes me think it might be some allocated limit being exceeded (threads in the thread pool? sorry, clueless here). Some observations: * all database operations in the deferred chain complete successfully before the traceback occurs. * it only occurs with pyPgSQL -- not with psycopg (at least I haven't yet seen it with psycopg, and believe me I've tried to make it happen! ... perhaps psycopg's thread management prevents it?) * this always occurs at the end of a "test" in which there is a succeed(result) callback that prints the result and calls reactor.stop() -- I wonder if stopping the reactor (which wouldn't normally happen in an app, of course) breaks the pipe? * I'm using - PostgreSQL 7.4.2 - pyPgSQL 2.4 - psycopg 1.1.15 - Twisted 1.3.0 - Python Python 2.3.3 (#1, Jul 22 2004, 15:58:51) [GCC 3.3.2 20031022 (Red Hat Linux 3.3.2-1)] on linux2 - Fedora Core 1, 2.4.22-1.2197.nptl Anyway, here's the traceback: --------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/threading.py", line 436, in __bootstrap self.run() File "/usr/local/lib/python2.3/threading.py", line 416, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/local/lib/python2.3/site-packages/twisted/python/threadpool.py", line 157, in _worker context.call(ctx, function, *args, **kwargs) File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 64, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.3/site-packages/twisted/python/context.py", line 43, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.3/site-packages/twisted/internet/threads.py", line 42, in _putResultInDeferred reactor.callFromThread(deferred.callback, result) File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 201, in callFromThread self.wakeUp() File "/usr/local/lib/python2.3/site-packages/twisted/internet/base.py", line 207, in wakeUp self.waker.wakeUp() File "/usr/local/lib/python2.3/site-packages/twisted/internet/default.py", line 399, in wakeUp self.o.flush() exceptions.IOError: [Errno 32] Broken pipe --------------------------------------------------------------------- Has anyone else observed this? Cheers, Steve

2 1

[Twisted-Python] Twisted co from subversion repository won't install
by Sergio Trejo Aug. 6, 2004

Aug. 6, 2004

I'm new to using subversion but I checked out today r4658 and after running "python setup.py build" (which seemed to work Ok) on Mac OS X, I then ran "python setup.py install" and received the following errors. It may be something peculiar to my system as I've installed Python 2.3.4 via Fink for OS X 10.3.4. Serg # python setup.py install running install running build running build_py running build_ext Checking if C extensions can be compiled, don't be alarmed if a few compile errors are printed. gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/sw/include/python2.3 -c conftest.c -o conftest.o gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/sw/include/python2.3 -c conftest.c -o conftest.o running build_scripts error: file 'bin/im' does not exist _________________________________________________________________ MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*. http://join.msn.com/?page=features/virus

2 2