[IPython-dev] iPipe and Hotwire Shell

[Walter Dörwald]

Colin Walters sagte:

> On Jan 31, 2008 6:58 AM, Walter Dörwald <walter at livinglogic.de> wrote:
>>
>> Currently ipipe's ifile class is a thin wrapper around Jason Orendorff's
>> path class (actually it's a subclass).
>
> Looking more closely, this class is very different from Hotwire's File
> object, which is designed for asynchronous use in a GUI, gathering all
> information it can in one go from a separate thread; then property
> access from the main thread never touches disk.

Indeed. In ifile almost *every* property access touches disk.

>> The only thing ifile does is add
>> the properties that will we used when displaying the output and
>> implementing the method __xattrs__() which tells ipipe which attributes
>> to display
>
> Ah, I see.  The problem of which attributes to display is something I
> haven't really tackled yet in Hotwire.  Right now the way it works is
> essentially that the renderers define it.

So is the basic output mode of hotwire a table or something else?

Another thing I noticed is that in hotwire the "container" (e.g. LsBuiltin etc.) tells the system what kind of objects it's
going to produce. In ipipe the display object displays whatever comes out of the pipe. This makes it possible to display
anything that is a sequence or iterator.

> But it would be good to share this, because then Python module authors
> could use it generically.  Maybe call it __dispattrs__ to make it more
> clear it's related to display?

Why not? Then we'd have to rename the generic function xattrs() too.

What xattrs() produces is a list of Descriptor objects, which are more flexible than simple attribute names.

Python 2.6 will allow to customize what the dir() function returns by implementing the __dir__() magic method. So maybe we
should go that road (or at least use it as a fallback)?

Another advantage of xattrs() is that it's a generic function, so you can configure the display attributes for classes you
didn't write without having to monkeypatch the class, i.e. you can do stuff like:

@ipipe.xattrs.when_type(complex):
def xattrs_compres(obj, mode="default"):
	return ("real", "imag")

>> hotwire propably does its own command line parsing, ipipe tables just
>> implement the  __or__/__ror__ magic methods. But thats a superficial
>> difference, ipipe could switch to command line parsing, or hotwire might
>> add the required magic.
>
> Right, as you mention ipipe really is Python, just using the __or__
> trick.  This may sound a bit heretical here but I'm actually thinking
> of the HotwirePipe language as independent of Python;

No problem. Actually I'd prefer if I could type:

   >>> ils ~

instead of having to type

   >>> ils("~")

> in the long term
> I think what we really want in a system shell is that you can
> seamlessly use whichever (sane) language you want - but using a
> common, shared class library on top of a common, shared runtime
> (bytecode, VM etc.).

Sounds like .NET.

> That's why I called the Hotwire builtin which
> takes a snippet of Python code "py-map", and not just "map".  And that
> the everyday tools you use in that shell are just really thin
> convenience wrappers around that common shared library.  I'm dreaming,
> I know...
>
> But let me try to put together a strawman here for the near-term future =)
>
> Hotwire builtins can become wrappers on top of IPipe library "i"
> functions.  There's a close correspondence between the names of
> Hotwire builtins and the names of IPipe functions.  The Hotwire
> language uses "-" in names (following Unix tradition), but we can
> bidirectionally map from/to "_" for IPipe functions.

Most exiting ipipe names are single words anyway, so there's no mapping required.

> Options to Hotwire builtins would map to keyword arguments in IPipe
> functions.  For example, the LsBuiltin "-a" option maps to a
> hypothetical "all=True" keyword argument for ils.  Hotwire builtin
> arguments map to "*args"-style argument captures for IPipe functions.

This sounds like a great idea. Configuration could be done by an optparse option parser. The long option names could be the
names of the keyword arguments.

However we still have the generator/threading mismatch and the "container declares the output" mismatch.

> IPipe would gain the concept of a multithreaded pipe (you need this,
> otherwise the system can deadlock), joined by iterable queues.

As long as both sides of this look like normal generator this would be *really* useful. The ipipe object iteration could stay a
simple iterator/generator, the piping operator would pipe this into a threading queue. Getting the results out of the queue
should be a simple iterator/generator again.

> I can
> try to do a bit of work to untangle this from the rest of the Hotwire
> core.  There is a bit of a question as to how this should map to IPipe
> because in Hotwire there is the concept of GStreamer-style negotation
> between components; for example, if you do:
>
> /bin/ls -al | /bin/grep x
>
> This expands to a pipe between two SysBuiltin threads, each spawning a
> system binary.  But, we don't really want to parse the data in between
> as Unicode lines (the default I/O for SysBuiltin) - it would be pure
> overhead, because /bin/grep like all Unix binaries takes byte arrays.
> So the first SysBuiltin says to the second "I can output
> bytearray/chunked", and the second says "I can accept
> bytearray/chunked", and thus they just pass "str" objects.  Though,
> thinking about it a bit more in this case I should really have the
> first actually pass the file descriptor for the OS-level pipe to the
> second to use as stdin, so the Hotwire process isn't involved at all.

So when two system commands are adjacent in the pipe encoding/decoding or even passing Python string objects around could be
optimized away. But that should be a specials case that doesn't affect Python tools in the pipe.

> If we wanted to support something like that in IPipe, it'd need to
> change probably from being Python to being a language of its own.

Currently ipipe doesn't do anything like this (calling external commands). The only command that executes external code is ix()
(which could greatly benefit from a threaded implementation)

> To implement the above, we also need to share the underlying object
> library, such as File and Process objects.  Probably what I should
> really do is try to submit this stuff to Python upstream...
>
>> Unfortunately I'm on OS X, so I can't really test hotwire, so I'm
>> limited to reading source and looking at screenshots.
>
> Yeah, I need to put together a package with the native GTK+-OSX bits and PyGTK.

That would be really helpful.

>> As Fernando stated, we can't use any of the GPL licensed parts of
>> hotwire, but if I've read COPYING/COPYING.API/COPYING.UI correctly, that
>> only affects the GUI parts.
>
> Right, all the stuff I'm talking about here is under a MIT-style
> license which is basically the same as the IPython license.

Good!

Servus,
   Walter