[Twisted-Python] Selectable SerialPort Windows/Linux
Hi @all! I'm trying to write a python library module for a special serial communication protocol called IMPBUS. To use the serial interface for sending and receiving packets as for now I'm sub-classing pyserial. My code looks like this: from serial import Serial, SerialExceptionfrom serial import EIGHTBITS, PARITY_ODD, STOPBITS_TWOimport binasciiclass SerialDevice(Serial): def __init__(self, port): Serial.__init__(self) self.port = port self.baudrate = 57600 self.bytesize = EIGHTBITS self.parity = PARITY_ODD self.stopbits = STOPBITS_TWO self.timeout = 0 self.xonxoff = 0 self.rtscts = 0 self.dsrdtr = 0 def _write(self, packet): fileno = self.fileno() while True: readable, writeable, excepts = select([], [fileno], [], 0.1) if fileno in writeable: length = self.write(packet) break return length def _read(self): fileno = self.fileno() while True: readable, writeable, excepts = select([], [fileno], [], 0.1) if fileno in readable: header = self.read(7) length = int(binascii.b2a_hex(header[3]), 16) data = self.read(length) packet = header + data break return packet def talk(self, packet): self._write(packet) responce = self._read() return responce But the problem is that I can't use select with pyserial on Windows, because it don't provide the fileno() methode. So after some googling I found twisted.internet.serialport "A select()able serial device, acting as a transport." I never used twisted before so I'm a little overwhelmed by how I can replace pyserial with twisted in the code above ... maybe someone can point me to the right direction. It seems I need a "Protocol" and a "receiver" ... - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig@imko.de internet: www.imko.de _________________________________________________________________
On 09:24 am, mhubig@imko.de wrote:
See the serialport examples on the website. A couple are linked from http://twistedmatrix.com/documents/current/core/examples/. You do indeed want a Protocol subclass. The code from your "self.read(7)" line to the end of that loop will probably end up in a dataReceived method - but note that you'll have to do some buffering, as you can't be assured that dataReceived will get called with exactly the number of bytes you want at a time, so you may have to collect data from multiple dataReceived calls (likewise you may get more than you want at once, and need to split it up). Jean-Paul
On Tue, Sep 14, 2010 at 2:08 PM, <exarkun@twistedmatrix.com> wrote:
See the serialport examples on the website. A couple are linked from http://twistedmatrix.com/documents/current/core/examples/.
The examples I found are mouse.py and gpsfix.py and there only listening on the serial line. It seems that with twisted I have some sort of a thread running which will trigger some methods (e.g. dataRecieved) as it reads something from the serial line. How could I send data then?
You do indeed want a Protocol subclass.
Ok I see, and the Protocol subclass provides the dataReceived method!?
The code from your "self.read(7)" line to the end of that loop will probably
end up in a dataReceived method - but note that you'll have to do some buffering, as you can't be assured that dataReceived will get called with
exactly
the number of bytes you want at a time, so you may have to collect data from
multiple dataReceived calls (likewise you may get more than you want at once, and need to split it up).
Hmm ok so I have to design dataReceived() to buffer the received data until it get's the whole answer packet and than let it call e.g. packetRecieved() to process the packet ... I'm mot sure if this asynchronous approach is what I need, because the protocol I'm implementing is strict Master-Slave. - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig@imko.de internet: www.imko.de _________________________________________________________________
Markus, I wonder if you've seen this really excellent tutorial: http://krondo.com/?page_id=1327 It is one of the finest tutorials I've seen anywhere on the web, on any topic. In fact, I think the Twisted leadership should put links to this tutorial all over their home page. While the finger "tutorial" is OK, this tutorial is much much much better, actually explaining *why* you make certain classes, etc. Vic On Tue, Sep 14, 2010 at 9:32 AM, Markus Hubig <mhubig@imko.de> wrote:
-- “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.” -- Antoine de Saint Exupéry
On 01:32 pm, mhubig@imko.de wrote:
Call the method for sending data. Generally, the protocol's transport's write method. Serial ports are not special in this regard, it's how almost all protocol implementations in Twisted work. See some of the other documentation (for example the client or server howtos linked from <http://twistedmatrix.com/documents/current/core/howto/>) for more details on this.
You do indeed want a Protocol subclass.
Ok I see, and the Protocol subclass provides the dataReceived method!?
Typically.
I'm not sure I understand this. What is a "Master-Slave" protocol? Why would someone not benefit from a Twisted-based implementation of such a protocol? Jean-Paul
On Tue, Sep 14, 2010 at 3:46 PM, <exarkun@twistedmatrix.com> wrote:
OK I'm not sure either ;-) this is why I ask. But in my case I have the following situation: behind my serial port I have a bus system made of me (the master) and some slaves. The slaves are absolute passive until I send them a command. After I've send a command to one of my slaves the salve ALWAY reply by sending a packet. So what's the benefit if I have a "deamon thread" waiting the hole time to get something from the serial line if the only time it can get something is immediately after the master has send a packet? The communication is totally synchronous. - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig@imko.de internet: www.imko.de _________________________________________________________________
On 02:00 pm, mhubig@imko.de wrote:
Do you ever want to interact with more than one slave? Do your slaves ever have any bugs that prevent them from responding? Do you want to be able to unit test your protocol code easily? If you want any of this, you might want to use Twisted. Also, there generally aren't any threads involved when you're using Twisted's serial port support. Jean-Paul
Here is my code for Plug 'n Play detection of serial devices. It's not yet ready for showtime, but it works with my serial attached Wacom. Best regards, eulores # -*- coding: utf-8 -*- import sys if sys.platform == 'win32': from twisted.internet import win32eventreactor win32eventreactor.install() from twisted.internet import reactor, protocol from twisted.internet.task import deferLater, LoopingCall from twisted.internet.defer import Deferred, inlineCallbacks, returnValue from twisted.internet.serialport import * import re def wait(seconds): return deferLater(reactor, seconds, lambda:None) class StructuredPnP(object): ''' The structured PnP field is available with these fields: data: string, required. Original unformatted string. other: string, optional rev: integer, required. Revision of the PnP standard. 100 is standard version 1.00 eisa: string, required product: string, required serial: string, optional klass: string, optional compat: string, optional. Other older products with similar functionality user: string, optional. Free flowing field useful for the end-user ''' def __init__(self, data): #data = '\\96,N,8,1(\x01$WAC0608\\\\PEN\\WAC0000\\WACOM UD\r\nUD-0608-R,V1.4-4\r\nF4)' # test string for a 7-bit character string #data = 'aaa(bbcccdddd\\eeeeeeee\\fff\\gggg\\hhhhii)' # test string for a 6-bit character string #data = 'AAA\x08BBCCCDDDD<EEEEEEEE<FFF<GGGG<HHHHII\x09' self.data = data for key in "other eisa product serial klass compat user".split(): setattr(self, key, '') self.rev = '\0\0' prologue = r'(?P<other>[^(]{,16}?)' matrix = r'(?P<rev>..)(?P<eisa>...)(?P<product>....)(?:@(?P<serial>[^@]{,8}))?(?:@(?P<klass>[^@]{,32}))?(?:@(?P<compat>[^@]{,40}))?(?:@(?P<user>.{,40}?))?(?:..)?' needle1 = prologue + '\\(' + matrix.replace('@','\\\\') + '\\)' needle2 = prologue + '\\\x08' + matrix.replace('@','\\\x3C') + '\\\x09' dct = dict() mo = re.match(needle1, data, re.S) if mo: dct = mo.groupdict() else: mo = re.match(needle2, data, re.S) if mo: dct = mo.groupdict() for k in "eisa product serial klass compat user".split(): v = dct[k] dct[k] = ''.join([chr(ord(ch)+0x20) for ch in list(v)]) for k,v in dct.items(): setattr(self, k, v) self.rev = ((ord(self.rev[0])&0x3f)<<6) + (ord(self.rev[1])&0x3f) def __str__(self): return self.data def __repr__(self): l = ['<StructuredPnP object> %r' % self.data] for k in "other rev eisa product serial klass compat user".split(): l.append(' %-8s %r' % (k, getattr(self,k,False))) return '\n'.join(l) class PnPProtocol(protocol.Protocol): """ See Plug and Play External COM device Specification, rev 1.00 from Microsoft & Hayes, 1994-1995""" def __init__(self, deferred): self.deferred = deferred self.data = '' self.timeout = 1.4 self.T5 = reactor.callLater(self.timeout, self.deliverPnP) def deliverPnP(self): self.transport.loseConnection() if self.deferred is not None: d, self.deferred = self.deferred, None d.callback(StructuredPnP(self.data)) def dataReceived(self, data): self.T5.reset(self.timeout) self.data += data if len(self.data)>=256: self.T5.reset(0) @inlineCallbacks def connectionMade(self): while 1: # print "2.1.1" self.transport.setDTR(1) self.transport.setRTS(0) yield wait(0.2) if not self.transport.getDSR(): break # print "2.1.3 part A" self.transport.setDTR(0) yield wait(0.2) # print "2.1.3 part B" self.transport.setDTR(1) yield wait(0.2) # print "2.1.4" self.transport.setRTS(1) # timer T5 is now used for per-character timeout self.timeout = 0.2 yield wait(0.2) if self.data: break # print "2.1.5" self.transport.setDTR(0) self.transport.setRTS(0) yield wait(0.2) # print "2.1.6" self.transport.setDTR(1) self.transport.setRTS(1) yield wait(0.2) break if not self.data: self.T5.reset(0) returnValue(None) def connectionLost(self, reason='connectionDone'): print "Connection lost:", reason def pnpString(port=0): d = Deferred() protocol = PnPProtocol(d) try: #SerialPort(protocol, port, reactor, baudrate=1200, bytesize=SEVENBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, timeout=0, xonxoff=0, rtscts=0) SerialPort(protocol, port, reactor, baudrate=1200, bytesize=SEVENBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, xonxoff=0, rtscts=0) except serial.SerialException: d.callback(StructuredPnP('')) return d @inlineCallbacks def imRunning(): print "PnP string: %r" % (yield pnpString(3)) reactor.stop() if __name__ == "__main__": reactor.callWhenRunning(imRunning) reactor.run() On Tue, Sep 14, 2010 at 11:24 AM, Markus Hubig <mhubig@imko.de> wrote:
On 09:24 am, mhubig@imko.de wrote:
See the serialport examples on the website. A couple are linked from http://twistedmatrix.com/documents/current/core/examples/. You do indeed want a Protocol subclass. The code from your "self.read(7)" line to the end of that loop will probably end up in a dataReceived method - but note that you'll have to do some buffering, as you can't be assured that dataReceived will get called with exactly the number of bytes you want at a time, so you may have to collect data from multiple dataReceived calls (likewise you may get more than you want at once, and need to split it up). Jean-Paul
On Tue, Sep 14, 2010 at 2:08 PM, <exarkun@twistedmatrix.com> wrote:
See the serialport examples on the website. A couple are linked from http://twistedmatrix.com/documents/current/core/examples/.
The examples I found are mouse.py and gpsfix.py and there only listening on the serial line. It seems that with twisted I have some sort of a thread running which will trigger some methods (e.g. dataRecieved) as it reads something from the serial line. How could I send data then?
You do indeed want a Protocol subclass.
Ok I see, and the Protocol subclass provides the dataReceived method!?
The code from your "self.read(7)" line to the end of that loop will probably
end up in a dataReceived method - but note that you'll have to do some buffering, as you can't be assured that dataReceived will get called with
exactly
the number of bytes you want at a time, so you may have to collect data from
multiple dataReceived calls (likewise you may get more than you want at once, and need to split it up).
Hmm ok so I have to design dataReceived() to buffer the received data until it get's the whole answer packet and than let it call e.g. packetRecieved() to process the packet ... I'm mot sure if this asynchronous approach is what I need, because the protocol I'm implementing is strict Master-Slave. - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig@imko.de internet: www.imko.de _________________________________________________________________
Markus, I wonder if you've seen this really excellent tutorial: http://krondo.com/?page_id=1327 It is one of the finest tutorials I've seen anywhere on the web, on any topic. In fact, I think the Twisted leadership should put links to this tutorial all over their home page. While the finger "tutorial" is OK, this tutorial is much much much better, actually explaining *why* you make certain classes, etc. Vic On Tue, Sep 14, 2010 at 9:32 AM, Markus Hubig <mhubig@imko.de> wrote:
-- “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.” -- Antoine de Saint Exupéry
On 01:32 pm, mhubig@imko.de wrote:
Call the method for sending data. Generally, the protocol's transport's write method. Serial ports are not special in this regard, it's how almost all protocol implementations in Twisted work. See some of the other documentation (for example the client or server howtos linked from <http://twistedmatrix.com/documents/current/core/howto/>) for more details on this.
You do indeed want a Protocol subclass.
Ok I see, and the Protocol subclass provides the dataReceived method!?
Typically.
I'm not sure I understand this. What is a "Master-Slave" protocol? Why would someone not benefit from a Twisted-based implementation of such a protocol? Jean-Paul
On Tue, Sep 14, 2010 at 3:46 PM, <exarkun@twistedmatrix.com> wrote:
OK I'm not sure either ;-) this is why I ask. But in my case I have the following situation: behind my serial port I have a bus system made of me (the master) and some slaves. The slaves are absolute passive until I send them a command. After I've send a command to one of my slaves the salve ALWAY reply by sending a packet. So what's the benefit if I have a "deamon thread" waiting the hole time to get something from the serial line if the only time it can get something is immediately after the master has send a packet? The communication is totally synchronous. - Markus -- __________________________________________________________________ IMKO Micromodultechnik GmbH Markus Hubig System Administration & Development Im Stoeck 2 D-76275 Ettlingen / GERMANY HR: HRB 360936 Amtsgericht Mannheim President: Dipl.-Ing. (FH) Kurt Koehler Tel: 0049-(0)7243-5921-26 Fax: 0049-(0)7243-5921-40 e-mail: mhubig@imko.de internet: www.imko.de _________________________________________________________________
On 02:00 pm, mhubig@imko.de wrote:
Do you ever want to interact with more than one slave? Do your slaves ever have any bugs that prevent them from responding? Do you want to be able to unit test your protocol code easily? If you want any of this, you might want to use Twisted. Also, there generally aren't any threads involved when you're using Twisted's serial port support. Jean-Paul
Here is my code for Plug 'n Play detection of serial devices. It's not yet ready for showtime, but it works with my serial attached Wacom. Best regards, eulores # -*- coding: utf-8 -*- import sys if sys.platform == 'win32': from twisted.internet import win32eventreactor win32eventreactor.install() from twisted.internet import reactor, protocol from twisted.internet.task import deferLater, LoopingCall from twisted.internet.defer import Deferred, inlineCallbacks, returnValue from twisted.internet.serialport import * import re def wait(seconds): return deferLater(reactor, seconds, lambda:None) class StructuredPnP(object): ''' The structured PnP field is available with these fields: data: string, required. Original unformatted string. other: string, optional rev: integer, required. Revision of the PnP standard. 100 is standard version 1.00 eisa: string, required product: string, required serial: string, optional klass: string, optional compat: string, optional. Other older products with similar functionality user: string, optional. Free flowing field useful for the end-user ''' def __init__(self, data): #data = '\\96,N,8,1(\x01$WAC0608\\\\PEN\\WAC0000\\WACOM UD\r\nUD-0608-R,V1.4-4\r\nF4)' # test string for a 7-bit character string #data = 'aaa(bbcccdddd\\eeeeeeee\\fff\\gggg\\hhhhii)' # test string for a 6-bit character string #data = 'AAA\x08BBCCCDDDD<EEEEEEEE<FFF<GGGG<HHHHII\x09' self.data = data for key in "other eisa product serial klass compat user".split(): setattr(self, key, '') self.rev = '\0\0' prologue = r'(?P<other>[^(]{,16}?)' matrix = r'(?P<rev>..)(?P<eisa>...)(?P<product>....)(?:@(?P<serial>[^@]{,8}))?(?:@(?P<klass>[^@]{,32}))?(?:@(?P<compat>[^@]{,40}))?(?:@(?P<user>.{,40}?))?(?:..)?' needle1 = prologue + '\\(' + matrix.replace('@','\\\\') + '\\)' needle2 = prologue + '\\\x08' + matrix.replace('@','\\\x3C') + '\\\x09' dct = dict() mo = re.match(needle1, data, re.S) if mo: dct = mo.groupdict() else: mo = re.match(needle2, data, re.S) if mo: dct = mo.groupdict() for k in "eisa product serial klass compat user".split(): v = dct[k] dct[k] = ''.join([chr(ord(ch)+0x20) for ch in list(v)]) for k,v in dct.items(): setattr(self, k, v) self.rev = ((ord(self.rev[0])&0x3f)<<6) + (ord(self.rev[1])&0x3f) def __str__(self): return self.data def __repr__(self): l = ['<StructuredPnP object> %r' % self.data] for k in "other rev eisa product serial klass compat user".split(): l.append(' %-8s %r' % (k, getattr(self,k,False))) return '\n'.join(l) class PnPProtocol(protocol.Protocol): """ See Plug and Play External COM device Specification, rev 1.00 from Microsoft & Hayes, 1994-1995""" def __init__(self, deferred): self.deferred = deferred self.data = '' self.timeout = 1.4 self.T5 = reactor.callLater(self.timeout, self.deliverPnP) def deliverPnP(self): self.transport.loseConnection() if self.deferred is not None: d, self.deferred = self.deferred, None d.callback(StructuredPnP(self.data)) def dataReceived(self, data): self.T5.reset(self.timeout) self.data += data if len(self.data)>=256: self.T5.reset(0) @inlineCallbacks def connectionMade(self): while 1: # print "2.1.1" self.transport.setDTR(1) self.transport.setRTS(0) yield wait(0.2) if not self.transport.getDSR(): break # print "2.1.3 part A" self.transport.setDTR(0) yield wait(0.2) # print "2.1.3 part B" self.transport.setDTR(1) yield wait(0.2) # print "2.1.4" self.transport.setRTS(1) # timer T5 is now used for per-character timeout self.timeout = 0.2 yield wait(0.2) if self.data: break # print "2.1.5" self.transport.setDTR(0) self.transport.setRTS(0) yield wait(0.2) # print "2.1.6" self.transport.setDTR(1) self.transport.setRTS(1) yield wait(0.2) break if not self.data: self.T5.reset(0) returnValue(None) def connectionLost(self, reason='connectionDone'): print "Connection lost:", reason def pnpString(port=0): d = Deferred() protocol = PnPProtocol(d) try: #SerialPort(protocol, port, reactor, baudrate=1200, bytesize=SEVENBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, timeout=0, xonxoff=0, rtscts=0) SerialPort(protocol, port, reactor, baudrate=1200, bytesize=SEVENBITS, parity=PARITY_NONE, stopbits=STOPBITS_ONE, xonxoff=0, rtscts=0) except serial.SerialException: d.callback(StructuredPnP('')) return d @inlineCallbacks def imRunning(): print "PnP string: %r" % (yield pnpString(3)) reactor.stop() if __name__ == "__main__": reactor.callWhenRunning(imRunning) reactor.run() On Tue, Sep 14, 2010 at 11:24 AM, Markus Hubig <mhubig@imko.de> wrote:
participants (4)
-
eulores -
exarkun＠twistedmatrix.com -
Markus Hubig -
Victor Norman