Mailman 3 September 2010 - Twisted

[Twisted-Python] How to cap the buffering size of data to be sent in Protocol class
by Weikai Xie 22 Sep '10

22 Sep '10

Folks, I have a server program which basically use a protocol.Protocol class to send a continuous byte stream to clients on TCP connections. However, I found , for whatever reason, if a client hung up and failed to remove the data from its receiving socket buffer, then the memory usage of the server program will go up endlessly. I think it is because the TCP sendQ buffer of the svr was building up as a result of the other side does not remove the data from its recvQ buffer (this guess was verified by the output of the netstat command), which in turn results the twisted Protocol to buffer the data submitted by upper layer via transport.write() to its internal buffer. My questions is that whether there is any mechanism in Twisted which can be used to set the maximum buffering size of the Protocol; and whenever if this cap is reached, I can be informed to give a chance to know the client is experiencing problem Regards. Weikai Xie

2 1

[Twisted-Python] Python 3
by James Mansion 21 Sep '10

21 Sep '10

I found it hard to tell what the level of support is for Python 3 - particularly on 64 bit Windows 7. I found some tickets in trac, but othewise no clear statement. Is Twisted 10.x OK on Python 3? James

4 4

[Twisted-Python] how to implement sleep?
by ruslan usifov 19 Sep '10

19 Sep '10

Hello Twisted is great, but how cant i emulate sleep behaviour in syested. I know that exists callLater, but how to use it in yield scenario, like here: @inlineCallbacks def work(self): l_attempts = 0; while l_attemps < 3: try: resp = yield <some twisted call> break; except: yield sleep(3); l_attempts +=1 else: log.err("cant communicate"); Here i make 3 attempts to connect busy service, with pause 3 seconds between attempts. How cant i implement this in twisted

6 6

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 18 Sep '10

18 Sep '10

1 0

[Twisted-Python] kqueue reactor
by ruslan usifov 16 Sep '10

16 Sep '10

Which kqueue reactor in twisted is best? Twisted have kqreactor - but it need PyKQueue patched version, and PyKQueue doesn't supported Also i found https://trac.calendarserver.org/browser/CalendarServer/branches/users/glyph…, but doesn't known is it possible use in production?

2 1

[Twisted-Python] any change of the .exe builds on PyPI?
by Chris Withers 16 Sep '10

16 Sep '10

ie: http://pypi.python.org/pypi/Twisted/10.1.0 It's great that they're being done now, but easy_install and friends on Windows still fail to find them unless they're on PyPI :-( Chris

4 5

[Twisted-Python] Selectable SerialPort Windows/Linux
by Markus Hubig 14 Sep '10

14 Sep '10

Hi @all! I'm trying to write a python library module for a special serial communication protocol called IMPBUS. To use the serial interface for sending and receiving packets as for now I'm sub-classing pyserial. My code looks like this: from serial import Serial, SerialExceptionfrom serial import EIGHTBITS, PARITY_ODD, STOPBITS_TWOimport binasciiclass SerialDevice(Serial): def __init__(self, port): Serial.__init__(self) self.port = port self.baudrate = 57600 self.bytesize = EIGHTBITS self.parity = PARITY_ODD self.stopbits = STOPBITS_TWO self.timeout = 0 self.xonxoff = 0 self.rtscts = 0 self.dsrdtr = 0 def _write(self, packet): fileno = self.fileno() while True: readable, writeable, excepts = select([], [fileno], [], 0.1) if fileno in writeable: length = self.write(packet) break return length def _read(self): fileno = self.fileno() while True: readable, writeable, excepts = select([], [fileno], [], 0.1) if fileno in readable: header = self.read(7) length = int(binascii.b2a_hex(header[3]), 16) data = self.read(length) packet = header + data break return packet def talk(self, packet): self._write(packet) responce = self._read() return responce But the problem is that I can't use select with pyserial on Windows, because it don't provide the fileno() methode. So after some googling I found twisted.internet.serialport "A select()able serial device, acting as a transport." I never used twisted before so I'm a little overwhelmed by how I can replace pyserial with twisted in the code above ... maybe someone can point me to the right direction. It seems I need a "Protocol" and a "receiver" ... - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig(a)imko.de internet: www.imko.de _________________________________________________________________

4 7

[Twisted-Python] Just curious about EINTR in socket.recv()
by Alexey 14 Sep '10

14 Sep '10

As far as I know, socket.recv() may raise socket.error exception in case of error return from recv(2) system call. My system's (Linux i386 2.6.32) man page say there could be these error values: EAGAIN or EWOULDBLOCK EBADF ECONNREFUSED EFAULT EINTR EINVAL ENOMEM ENOTCONN ENOTSOCK Now, looking into twisted.internet.tcp, method doRead() on line 443, (twisted 10.1.0) we may find, that code: try: data = self.socket.recv(self.bufferSize) except socket.error, se: if se.args[0] == EWOULDBLOCK: return else: return main.CONNECTION_LOST if not data: return main.CONNECTION_DONE return self.protocol.dataReceived(data) This is the point in the twisted framework, that decides how TCP socket will get closed, whether that be in "clean" or "non clean" fashion. But some error codes, besides EWOULDBLOCK, are not in any way related to reporting a permanent error, that deserves closing the socket. Question to Python system library gurus: is socket.recv() supposed to raise socket.error with EINTR code ? In that case we should return None too, since that operation should be restarted, and reporting main.CONNECTION_LOST is a bug. What about ENOMEM case? Is system supposed to have destroyed the socket after that event, or it can safely recover later, so there is no sense in closing the socket in twisted either? Sorry, if that is a non-issue, or already tracked, I'm just curious. -- Alexey.

2 1

[Twisted-Python] twisted cred: why does avatarId need to be a str?
by Laurens Van Houtven 11 Sep '10

11 Sep '10

Hi, As some of you probably know I'm trying to beat OAuth2.0 into submission. I'm using twisted.cred to do it. The restriction that avatarIds need to be strs is somewhat getting in my way, and I'm not sure if the correct way to do it is: 1) ignore the restriction (credentials types are so specific to OAuth that it's very unlikely you'll find anything already pluggable...) 2) use the mind object (but then it becomes somewhat wonky what goes on the mind object and what should be the avatarId) I'll give you a practical example: An OAuth client makes a request for an access token. This basically means he trades some credentials in for an access token (I use the expression trade-in because most of the credentials can only be used once). It presents some credentials, which can be grouped into two parts: 1) credentials that authenticate the client itself 2) credentials for the access grant (eg I've got this thing that says I should have access) Because (1) is pretty much always the same (and is common to other parts of txOAuth) and (2) depends on the specific kind of request, this results in two portals: 1) for authenticating the client, which returns an IClient 2) for authenticating the credentials (which includes the authenticated client, since credentials are client-specific), which returns the token Coming from Perspective Broker I think the Client would be an apt mind object for the second portal, since it represents the entity on the other side trying to do auth. The appropriate credentials checker in the second portal checks (and invalidates, since most of the credentials are single use) the credentials. What avatarId is sent to the Realm? The Realm is also responsible for requesting the new token, so it needs to know details about the credential that the credential checker just checked (and in doing so, invalidated). These are not available through the IClient mind object. The alternative design is that all of the credentials checkers know how to request tokens, and instead give the tokenId as avatarId. This is still insufficient, because tokens can include extra information, such as scope, expiration time... That can again be solved by sending an opaque string value which contains all of the data in the token response, but in that case the Realm is completely useless and the checkers do all of the work. Additionally, this seems stupid because cred checkers are for checking credentials, not checking credentials + a bunch of other stuff. Here's my theory: 13:43 <lvh> aa_: fwiw I *think* it's because cred's supposed to be pluggable 13:43 <lvh> aa_: so avatarId is supposed to be the equivalent of a username 13:44 <lvh> aa_: and it's str not unicode because there's a helluva lot more things that store bytes than unicode objects So um, any thoughts from people that understand cred? Is it ever okay to send richer things than strings to a Realm? thanks lvh

4 9

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 11 Sep '10

11 Sep '10

1 0