[Python-Dev] Towards native fileevents in Python (Was Re: Python multiplexing is too hard)

[Alexandre Ferrieux]

Guido van Rossum wrote:
> 
> > Yup. One easy answer is 'just copy from Tcl'...
> 
> Tcl seems to be your only frame of reference.

Nope, but I'll welcome any proof of existence of similar abstractions
(for multiplexing) elsewhere.

>  I think it's too early
> to say that borrowing Tcl's design is right for Python.  Don't forget
> that part of Tcl's design was guided by the desire for backwards
> compatibility with Tcl's strong (stronger than Python I find!) Unix
> background.

I don't quite get how the 'unix background' comes into play here, since
[fileevent] is now implemented and works correctly on all platforms.

If you are talinkg about the API as seen from above, I don't understand
why 'hooking a callback' and 'multiplexing event sources' are a unix
specificity, and/or why it should be avoided outside unix.

> > Seriously, I'm really too new to Python to suggest the details or even
> > the *style* of this 'level 2 API to multiplexing'. However, I can sketch
> > the implementation since select() (from C or Tcl) is the one primitive I
> > most depend on !
> >
> > Basically, as shortly mentioned before, the key problem is the
> > heterogeneity of seemingly-selectable things in Windoze. On unix, not
> > only does select() work with
> > all descriptor types on which it makes sense, but also the fd used by
> > Xlib is accessible; hence clean multiplexing even with a GUI package is
> > trivial. Now to the real (rotten) meat, that is M$'s. Facts:
> 
> Note that on Windows, select() is part of SOCKLIB, which explains why
> it only understands sockets.  Native Windows code uses the
> wait-for-event primitives that you are describing, and these are
> powerful enough to wait on named pipes, sockets, and GUI events.
> Complaining about the select interface on Windows isn't quite fair.

Sorry, you missed the point. Here I used the term 'select()' as a 
generic one (I didn't want to pollute a general discussion with
OS-specific names...). On windows it means MsgWaitForMultipleObjects.

Now as you said "these are powerful enough to wait on named pipes,
sockets, and GUI events"; I won't deny the obvious truth. However,
again, they don't work on *unnamed pipes* (which are the only ones in
'95). That's my sole reason for complaining, and I'm afraid it is fair
;-)

> >       1. 'Handle' types are not equal. Unnames pipes are (surprise!) not
> > selectable. Why ? Ask a relative in Redmond...
> 
> Can we cut the name-calling?

Yes we can :^P

> 
> >       2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true
> > handles. They are selectable, but for example you can't use'em for
> > redirections. Okay in our case we don't care. I only mention it cause
> > its scary and could pop back into your face some time later.
> 
> Handles are a much more low-level concept than file descriptors.  get
> used to it.

Take it easy, I meant to help. Low level as they be, can you explain why
*some* can be passed to CreateProcess as redirections, and *some* can't
?
Obviously there *is* some attempt to unify things in Windows (if only
the
single name of 'handle'); and just as clearly it is not completely
successful.

> >       3. The GUI API doesn't expose a descriptor (handle), but fortunately
> > (though disgustingly) there is a special syscall to wait on both "the
> > message queue" and selectable handles: MsgWaitForMultipleObjects. So its
> > doable, if not beautiful.
> >
> > The Tcl solution to (1.), which is the only real issue,
> 
> Why is (1) the only issue?

Because for (2) we don't care (no need for redirections in our case) and
for (3) the judgement is only aesthetic. 

>  Maybe in Tcl-land...

Come on, I'm emigrating from Tcl to Python with open palms, as Cameron
puts it.
I've already mentioned the outstanding beauty of Python's internal
design, and in comparison Tcl is absolutely awful. Even at the (script)
API level, Some of the early choices in Tcl are disgusting (and some
recent ones too...). I'm really turning to Python with the greatest
pleasure - please don't interpret my arguments as yet another Lang1 vs.
Lang2 flamewar.

> > is to have a
> > separate thread blockingly read 1 byte from the pipe, and then post a
> > message back to the main thread to awaken it (yes, ugly code to handle
> > that extra byte and integrate it with the buffering scheme).
> 
> Or the exposed API could deal with this in a different way.

Please elaborate ?

> > In summary, why not peruse Tcl's hard-won experience on
> > selecting-on-windoze-pipes ?
> 
> Because it's designed for Tcl.

I said 'why not' as a positive suggestion.
I didn't expect you to actually say why not...

Moreover, I don't understand 'designed for Tcl'. What's specific to Tcl
in
unifying descriptor types ?

> > Then, for the API exposed to the Python programmer, the Tclly exposed
> > one is a starter:
> >
> >       fileevent $channel readable|writable callback
> >       ...
> >       vwait breaker_variable
> >
> > Explanation for non-Tclers: fileevent hooks the callback, vwait does a
> > loop of select(). The callback(s) is(are) called without breaking the
> > loop, unless $breaker_variable is set, at which time vwait returns.
> 
> Sorry, you've lost me here.  Fortunately there's more info at
> http://dev.scriptics.com/man/tcl8.3/TclCmd/fileevent.htm.  It looks
> very complicated,

Ahem, self-destroying argument: "Fortunately ... very complicated".

While I agree the fileevent manpage is longer than it should be, I fail
to see
what's complicated in the model of 'hooking a callback for a given kind
of events'.

> and I'm not sure why you rejected my earlier
> suggestion to use threads outright as "too complicated".

Not on the same level. You're complaining about the script-level API (or
its documentation, more precisely !). I dismissed the thread-based
*implementation*
as an overkill in terms of resource consumption (thread context +
switching + ITC)
on platforms which can use select() (for anon pipes on Windows, as
already explained, the thread is unavoidable).

>  After
> reading that man page, threads seem easy compared to the caution one
> has to exert when using non-blocking I/O.

Oh, I get it. The problem is, *that* manpage unfortunately tries to
explain event-based and non-blocking I/O at the same time (presumably
because the average user will never follow the 'See Also' links). That's
a blatant pedagogic mistake. Let me try:

	fileevent <channel> readable|writable <script>

	Hooks <script> to be called back whenever the given <channel> becomes
readable|writable. 'Whenever' here means from within event processing
primitives (vwait, update).

	Example:

		# whenever a new line comes down the socket, display it.

		set s [socket $host $port]
		fileevent $s readable gotdata
		proc gotdata {} {global s;puts "New data: [gets $s]"}
		vwait forever

To answer a potential question about blockingness, yes, in the example
above the [gets] will block until a complete line is received. But
mentioning this fact in the manpage is uselessly misleading because  the
fileevent mechanism obviously allows to implement any kind of protocol,
line-based or not, terminator- or size-header- based or not. Uses with
blocking and nonblocking [read] and mixes thereof are immediate
consequences of this classification.

Hope this helps.

> Vwait seems to be part of the Tcl event model.

Hardly. It's just the Tcl name for the primitive that (blockingly) calls
select() (generic term - see above)

>  Maybe we would need to think about an event model for Python?

With pleasure - please define 'model'. Do you mean callbacks vs.
explicit decoding of an event strucutre ? Do you mean blocking select()
vs. something more asynchronous like threads or signals ?

> On the other hand, Python is
> at the mercy of the event model of whatever GUI package it is using --
> which could be Tk, or wxWindows, or Gtk, or native Windows, or native
> MacOS, or any of a number of other event models.

Why should Python be alone to be exposed to this diversity ? Don't
assume
that Tk is the only option for Tcl. The Tcl/C API even exposes the
proper hooks
to integrate any new event source, like a GUI package.

Again, I'm not interested in Tcl vs. Python here (and anyway Python wins
!!!). I just want to extract what's truly orthogonal to specific design
choices. As it turns out, what you call 'the Tcl event model' can
happily be transported to any (imperative) lang.

I can even be more precise: a random GUI package can be used this way
iff the two following conditions hold:

	(a) Its queue can awaken a select()-like primitive.
	(b) Its queue can be Peek'ed (to check for buffered msgs
                                      before blockigng again)

> Perhaps this is an issue that each GUI package available to Python
> will have to deal with separately...

The characterization is given just above. To me it looks generic enough
to build an abstraction upon it. It's been done for Tcl, and is utterly
independent from its design peculiarities. Now everything depends on
whether abstraction is sought or not...

-Alex