[Tutor] multipart socket streaming problem: reading the stream

richard kappler richkappler at gmail.com
Tue Jan 5 10:38:51 EST 2016


This is a continuation of the thread 'reading an input stream' I had to
walk away from for a few days due to the holidays and then other work
considerations, and I figured it best to break my confusion into separate
chunks, I hope that's appropriate. In short, my script needs to read a
stream of xml data from a socket (port 2008), the data coming in from as
many as 30 different machines, but usually 4 or less, as many as 3 messages
per second from each machine at times, messages block format delimited by
stx(\x02) and etx (\x03), send the data in those blocks to a parser
(already built using lxml and an xslt file) and send it out to splunk using
a native 'event writer'.

I am a bit lost in the woods on 'reading' the stream. My first attempt
tried to read the stream directly (from the buffer 'I think'):

#!/usr/bin/env python


import socket

import lxml.etree as ET


# receive socket

sockrx = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sockrx_address = ('', 2008)

print 'opening sockrx on %s port %s' % sockrx_address

try:

    sockrx.bind(sockrx_address)

except socket.error as msg:

    print 'Bind failed. Error code: ' + str(msg)

    sys.exit()


sockrx.listen(5)

print 'listening'


while True:

    # wait for a connection

    connection, client_address = sockrx.accept()

    data = connection.recv(8192)

    if data:

        print 'receiving data'

        f1 = open('parser.out', 'a')

        xslt = ET.parse('stack13.xsl')

        dom = ET.parse(data)

        transform = ET.XSLT(xslt)

        newdom = transform(dom)

        f1.write(str(newdom))



# close sockrx

connection.close()


f1.close()


This did not work:

Traceback (most recent call last):

  File "streamer-01.py", line 27, in <module>

    dom = ET.parse(data)

  File "lxml.etree.pyx", line 3239, in lxml.etree.parse
(src/lxml/lxml.etree.c:69955)

  File "parser.pxi", line 1748, in lxml.etree._parseDocument
(src/lxml/lxml.etree.c:102066)

  File "parser.pxi", line 1774, in lxml.etree._parseDocumentFromURL
(src/lxml/lxml.etree.c:102330)

  File "parser.pxi", line 1678, in lxml.etree._parseDocFromFile
(src/lxml/lxml.etree.c:101365)

  File "parser.pxi", line 1110, in lxml.etree._BaseParser._parseDocFromFile
(src/lxml/lxml.etree.c:96817)

  File "parser.pxi", line 582, in
lxml.etree._ParserContext._handleParseResultDoc
(src/lxml/lxml.etree.c:91275)

  File "parser.pxi", line 683, in lxml.etree._handleParseResult
(src/lxml/lxml.etree.c:92461)

  File "parser.pxi", line 620, in lxml.etree._raiseParseError
(src/lxml/lxml.etree.c:91722)

IOError: Error reading file

and then the error message printed out the entire file to screen, which I
won't here  for brevity.


I was, however, able to receive the the stream as above, write it to a
file, then read the file to the parser and it worked. I tired using
makefile() but that did not work either. I'm looking for some general
direction here, more specific questions to follow.

regards, Richard


More information about the Tutor mailing list