[Tutor] looping generator

[richard kappler]

We got it working, here's the code for critique and guidance on potential
improvement. All that's left to do is the threading, which seems fairly
trivial. We will spawn the threads between the     connection,
client_address = sock.accept() statement and the while True: that follows
it, putting everything under the while True: but before the f1.close() as a
function that is the thread target. Please note, during your code review,
that the socket is intentionally left open, this will run 24/7/365 and must
be listening as such. Also, f1 (parser out) is for dev only and will be
replaced with a call to splunk's event writer. I look forward to your
comments.

regards, Richard

#!/usr/bin/env python

import socket
import lxml.etree as ET

stx = '\x02'
etx = '\x03'

def waiting_for_stx(buf):
    start = buf.find(stx)
    if start >= 0:
        return (buf[start + 1:], waiting_for_etx, True)
    else:
        return ('', waiting_for_stx, False)

def waiting_for_etx(buf):
    end = buf.find(etx)
    if (end < 0):
        return (buf, waiting_for_etx, False)
    else:
        messagetoprocess = buf[:end]
        dataParse(messagetoprocess)
        return (buf[end + 1:], waiting_for_stx, True)

def dataParse(message):
    print 'parsing'
    xslt = ET.parse('stack13.xsl')
    dom = ET.XML(message)
    transform = ET.XSLT(xslt)
    newdom = transform(dom)
    f1.write(str(newdom))


sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock_addr = ('', 2008)
sock.bind(sock_addr)
sock.listen(5)
print 'listening'

f1 = open('parser.out', 'a')
print "opening parser.out"

curr_state = waiting_for_stx
buf = ''

while True:
    # wait for a connection
    connection, client_address = sock.accept()
    while True:
        data_chunks = connection.recv(8192)
        for chunk in data_chunks:
            buf += chunk
            stillProcessing = True
            while stillProcessing:
                (buf, curr_state, stillProcessing) = curr_state(buf)

f1.close()


On Thu, Jan 7, 2016 at 12:02 PM, richard kappler <richkappler at gmail.com>
wrote:

> I have a stream of incoming xml data. I can receive the data, parse the
> data, etc, so long as I don't get fancy and I have a miniscule delay in
> between each message. If I get rid of the time delay, which I need to, I
> need the script to continuously process the incoming messages. Here's what
> I have:
>
> #!/usr/bin/env python
>
> import socket
> import lxml.etree as ET
>
> def dataRecv(connection):
>     print 'receiving'
>     while True:
>         data = connection.recv(65536)
>         while True:
>             print "writing to data.in"
>             f2.write(data)
>             start = data.find('\x02')
>             end = data.find('\x03')
>             message = data[start+1:end]
>             print "writing to messages.out"
>             f3.write(message)
>             yield message
>
> def dataParse(message):
>     print 'parsing'
>     xslt = ET.parse('stack13.xsl')
>     dom = ET.XML(message)
>     transform = ET.XSLT(xslt)
>     newdom = transform(dom)
>     f1.write(str(newdom))
>
>
> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> sock_addr = ('', 2008)
> #data = sock.makefile('r')
> sock.bind(sock_addr)
> sock.listen(5)
> print 'listening'
>
> f1 = open('parser.out', 'a')
> print "opening parser.out"
> f2 = open('data.in', 'a')
> print "opening data.in"
> f3 = open('messages.out', 'a')
> print "opening messages.out"
>
> while True:
>     # wait for a connection
>     connection, client_address = sock.accept()
>     q = dataRecv(connection)
>     dataParse(q.next())
>
> # close sockrx
> #connection.close()
>
> f1.close()
>
>
> In the dataRecv function, I have tried (where you see while True) if data,
> while data and while True. Regardless, it doesn't loop, it receives al ten
> messages from the test file being sent, but only processes the first
> message then stops (not exits). I feel like I'm missing something obvious
> but can't find it.
>
> regards, Richard
>