[python-uk] Question from uniq dojo about fileinput

Stestagg stestagg at gmail.com
Thu Jan 8 00:30:00 CET 2015

When I try to run it, I get a hang too, in my case (there may be other
factors for you) what's happening is because of the openpty stuff.

In the parent process, you're creating two file handles, master and slave.
Subprocess then forks, meaning the child inherits them.  At this point,
both the parent and child processes have the same PTY handles open.

In my version of the code, there's a traceback in uniq.py that's preventing
making the child bail out early.  At this point the child's FDs are closed,
but because the master still has the 'slave' FD open, the pty isn't being
shut down.

Add an os.close() after the subprocess call:

process = Popen("python -u uniq.py", shell=True, stdin=PIPE, stdout=slave)

This will cause the pty to collapse when the child exits, which should stop
the hanging read().

After adding some UTF-8 decoding to the uniq.py file, I managed to get this:

>       assert expected == result
E       assert ['bar\n', 'fo...o\n', 'end\n'] == ['bar\r\n', 'f...n',
E         At index 0 diff: 'bar\n' != 'bar\r\n'
E         Full diff:
E         - ['bar\n', 'foo\n', 'bar\n', 'foo\n', 'end\n']
E         + ['bar\r\n', 'foo\r\n', 'bar\r\n', 'foo\r\n', 'end\r\n']
E         ?       ++         ++         ++         ++         ++

Which seems quite close :)


On Wed Jan 07 2015 at 10:10:45 PM Tom Viner <tom at viner.tv> wrote:

> Hi all,
> So before the London dojo meets again tomorrow night I wanted to resolve a
> question from last month.
> The task last month was to make our own GNU uniq commands. After the dojo
> I got our team's code working with the fileinput module
> <https://docs.python.org/2/library/fileinput.html>. This allowed the
> flexibility of seamlessly reading from either stdin or taking filenames as
> arguments.
> With fileinput the code
> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/uniq.py#L11>
> can just say:
> for line in fileinput.input():
> and that gives you all these usages:
> python uniq.py my_filename.txt other_file.txt
> echo -e "hello\nhello2" | python uniq.py
> python uniq.py <(python print_sleep_print.py)
> With the print_sleep_print.py script
> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/print_sleep_print.py>
> I can even see about 8Kb at a time will be buffered into uniq.py, made
> unique and printed, and then more data will be passed in.
> The problem came when testing this feature. I found a way to connect to
> both the input and output of a subprocess running the command:
>     master, slave = pty.openpty()
>     process = Popen("python uniq.py", shell=True, stdin=PIPE, stdout=slave)
>     stdin_handle = process.stdin
>     stdout_handle = os.fdopen(master)
> I then write some data in, and read data out. It all works except for one
> thing: even if I close the stdin_handle the stdout_handle will just block
> once I've read all the output. Full test here
> <https://github.com/tomviner/pydojo-uniq-s6e4/blob/master/team1/test_uniq.py#L37>
> .
> Appreciate any insight!
> Tom
> _______________________________________________
> python-uk mailing list
> python-uk at python.org
> https://mail.python.org/mailman/listinfo/python-uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-uk/attachments/20150107/8a020ef6/attachment-0001.html>

More information about the python-uk mailing list