[Python-Dev] Re: PEP 324 (process module)

[Trent Mick]

[Peter Astrand wrote]
> 
> > I believe ActiveState's Trent Mick has released something along these
> > lines.  Does your proposed API match theirs?
> 
> No, not at all. There are some issues with this module, both regards the
> API and the implementation, which I'm not happy with. I tried to establish
> a discussion with Trent about this, but if I remember correctly, I never
> recieved a response.

Yes, I am an ass. I am sorry about that Peter. I have had my blinders on
for quite a while with work stuff.

I have just started looking at your module so I can't be sure that we
could find a happy common ground but I hope so. I realize that my
process.py has some warts (including some you have pointed out below)
and I am willing to work on those.

> Here's some things which I'm not happy with:
> 
> * There are three different "factory" classes: Process, ProcessOpen,
> ProcessProxy. I have only one, covering all cases. I think three different
> classes are confusing to users. Even I have trouble understanding the
> difference between them.

Part of the confusion, I think, is that my process.py's docstrings have
lagged a little bit behind development and there is one glaring problem
that ProcessOpen.__init__.__doc__ mistakenly has the description from
ProcessProxy.__init__. 

Process:
    Create a process without doing without doing anything with
    stdin/stderr/stdout. Kind of like os.system in that regard.
ProcessOpen:
    ...add stdin/stdout/stderr attributes (file-like objects). Kind of
    like os.popen3 in that regard. 

Separating the two allows Process' implementation to be simpler: no
dup'ing (and inheritability handling, etc) of the std handlers is
required. If general opinion is that this separation is not useful, then
I am fine with merging the two. (Note that my vague memory is that this
separation may be necessary to allow for new console handling with
subsystem:windows apps on Windows. I can't remember exactly though.)

ProcessProxy:
    ...a behemoth using a thread for each of stdin/stdout/stderr to
    allow the user to get an event-like IO interface.
    
This is something that was required for Komodo (for my job) to allow
interactive running (and stdin handling) of processes in a GUI terminal
(i.e., if you know Komodo, to run commands in the "Command Output" tab).
If this is deemed too controversial for the Python core then I think it
would be possible to move this out to a separate module -- though I know
that some people outside of Komodo have found this useful.

> 
> * The code size is very large. The code is complex.
> 
> * Trent's module always executes things through the shell, and deals with
> every ugly cornercase that comes from this.

Some of the large code size is _for_ execution through the shell. :)  I
think that execution via the shell should be a feature of a module like
this (so that users can use some shell features) and I even think that
it should be the default (so that, for example, Windows users don't have
to learn what cmd.exe or command.com are to run "dir"). However, I
absolutely agree that one should be able to run withOUT the shell (i.e.
have an option for this).

Other reasons for the size/complexity of my process.py over yours:
- My process objects have a .kill() method -- which is actually quite a
  pain on Windows.
- My module contains a work-around (113 lines) for a known bug in
  LinuxThreads where one cannot .wait() on a created process from a
  subthread of the thread that created the process.
  (1) This feature was a requirement for Komodo. Pulling it out to a
      separate module and just documenting the limitation (which really
      isn't a big deal for most common uses) is probably an option for me.
      In fact the workaround is probably not generally acceptible and
      should, at the very least, be made optional.
  (2) I haven't had a chance to check but I _think_ that recent Linuxs
      may have switched to the newer threading libraries out there that
      fix this. At least I remember reading about the new threading
      libraries (there was more that one competitor) a couple of years
      ago.
- The ProcessProxy "behemoth" is responsible for about half of my
  process.py module.
- My module includes some handling that for subsystem:windows vs.
  subsystem:console apps on Windows that I don't think yours does.
  (That isn't totally fair: I think yours has a few features that mine
  doesn't.)


> * The modules uses destructors, which I'm usually avoiding.

Is that evil? I have gotten mixed signals from the occassional lurking
that I have done on the Python lists. Currently refcounts will ensure
that my __del__'s get called. And I *do* have .close() methods for
proper usage. Though that seem anathema to some Python programmers.


> So, the chances of getting our modules API-compatible are very small.

Presuming that I don't go dark again I hope that maybe we can reach some
concensus. In the meantime, if you'd be willing to change your module's
name to something other than process.py I think that would help
discussions. (I don't know if that is a pain for you at this point. You
mentioned "subprocess". Alternatively, how about "posixprocess"? Though,
despite PEP 324's title, I don't know if that is completely accurate
anymore.)


Trent

-- 
Trent Mick
TrentM at ActiveState.com