[IPython-dev] Fwd: iPipe and Hotwire Shell

[Colin Walters]

[have now taught GMail to use my right email so I don't get caught in
the spamtrap]

On Jan 31, 2008 6:58 AM, Walter Dörwald <walter at livinglogic.de> wrote:
>
> Currently ipipe's ifile class is a thin wrapper around Jason Orendorff's
> path class (actually it's a subclass).

Looking more closely, this class is very different from Hotwire's File
object, which is designed for asynchronous use in a GUI, gathering all
information it can in one go from a separate thread; then property
access from the main thread never touches disk.

>The only thing ifile does is add
> the properties that will we used when displaying the output and
> implementing the method __xattrs__() which tells ipipe which attributes
> to display

Ah, I see.  The problem of which attributes to display is something I
haven't really tackled yet in Hotwire.  Right now the way it works is
essentially that the renderers define it.

But it would be good to share this, because then Python module authors
could use it generically.  Maybe call it __dispattrs__ to make it more
clear it's related to display?

> hotwire propably does its own command line parsing, ipipe tables just
> implement the  __or__/__ror__ magic methods. But thats a superficial
> difference, ipipe could switch to command line parsing, or hotwire might
> add the required magic.

Right, as you mention ipipe really is Python, just using the __or__
trick.  This may sound a bit heretical here but I'm actually thinking
of the HotwirePipe language as independent of Python; in the long term
I think what we really want in a system shell is that you can
seamlessly use whichever (sane) language you want - but using a
common, shared class library on top of a common, shared runtime
(bytecode, VM etc.).  That's why I called the Hotwire builtin which
takes a snippet of Python code "py-map", and not just "map".  And that
the everyday tools you use in that shell are just really thin
convenience wrappers around that common shared library.  I'm dreaming,
I know...

But let me try to put together a strawman here for the near-term future =)

Hotwire builtins can become wrappers on top of IPipe library "i"
functions.  There's a close correspondence between the names of
Hotwire builtins and the names of IPipe functions.  The Hotwire
language uses "-" in names (following Unix tradition), but we can
bidirectionally map from/to "_" for IPipe functions.
Options to Hotwire builtins would map to keyword arguments in IPipe
functions.  For example, the LsBuiltin "-a" option maps to a
hypothetical "all=True" keyword argument for ils.  Hotwire builtin
arguments map to "*args"-style argument captures for IPipe functions.

IPipe would gain the concept of a multithreaded pipe (you need this,
otherwise the system can deadlock), joined by iterable queues.  I can
try to do a bit of work to untangle this from the rest of the Hotwire
core.  There is a bit of a question as to how this should map to IPipe
because in Hotwire there is the concept of GStreamer-style negotation
between components; for example, if you do:

/bin/ls -al | /bin/grep x

This expands to a pipe between two SysBuiltin threads, each spawning a
system binary.  But, we don't really want to parse the data in between
as Unicode lines (the default I/O for SysBuiltin) - it would be pure
overhead, because /bin/grep like all Unix binaries takes byte arrays.
So the first SysBuiltin says to the second "I can output
bytearray/chunked", and the second says "I can accept
bytearray/chunked", and thus they just pass "str" objects.  Though,
thinking about it a bit more in this case I should really have the
first actually pass the file descriptor for the OS-level pipe to the
second to use as stdin, so the Hotwire process isn't involved at all.

If we wanted to support something like that in IPipe, it'd need to
change probably from being Python to being a language of its own.

To implement the above, we also need to share the underlying object
library, such as File and Process objects.  Probably what I should
really do is try to submit this stuff to Python upstream...

> Unfortunately I'm on OS X, so I can't really test hotwire, so I'm
> limited to reading source and looking at screenshots.

Yeah, I need to put together a package with the native GTK+-OSX bits and PyGTK.

> As Fernando stated, we can't use any of the GPL licensed parts of
> hotwire, but if I've read COPYING/COPYING.API/COPYING.UI correctly, that
> only affects the GUI parts.

Right, all the stuff I'm talking about here is under a MIT-style
license which is basically the same as the IPython license.