[Python-Dev] Re: Towards native fileevents in Python (Was Re: Python multiplexing is too hard)

[Cameron Laird]

	From alexandre.ferrieux@cnet.francetelecom.fr  Mon May 22 03:40:13 2000
			.
			.
			.
	> Alex, it's disappointing to me too!  There just isn't anything
	> currently in the library to do this, and I haven't written apps that
	> needs this often enough to have a good feel for what kind of
	> abstraction is needed.

	Thanks for the empathy. Apologies for my slight overreaction.

	> However perhaps we can come up with a design for something better?  Do
	> you have a suggestion here?

	Yup. One easy answer is 'just copy from Tcl'...

	Seriously, I'm really too new to Python to suggest the details or even
	the *style* of this 'level 2 API to multiplexing'. However, I can sketch
	the implementation since select() (from C or Tcl) is the one primitive I
	most depend on !

	Basically, as shortly mentioned before, the key problem is the
	heterogeneity of seemingly-selectable things in Windoze. On unix, not
	only does select() work with
	all descriptor types on which it makes sense, but also the fd used by
	Xlib is accessible; hence clean multiplexing even with a GUI package is
	trivial. Now to the real (rotten) meat, that is M$'s. Facts:

		1. 'Handle' types are not equal. Unnames pipes are (surprise!) not
	selectable. Why ? Ask a relative in Redmond...

		2. 'Handle' types are not equal (bis). Socket 'handles' are *not* true
	handles. They are selectable, but for example you can't use'em for
	redirections. Okay in our case we don't care. I only mention it cause
	its scary and could pop back into your face some time later.

		3. The GUI API doesn't expose a descriptor (handle), but fortunately
	(though disgustingly) there is a special syscall to wait on both "the
	message queue" and selectable handles: MsgWaitForMultipleObjects. So its
	doable, if not beautiful.

	The Tcl solution to (1.), which is the only real issue, is to have a
	separate thread blockingly read 1 byte from the pipe, and then post a
	message back to the main thread to awaken it (yes, ugly code to handle
	that extra byte and integrate it with the buffering scheme).

	In summary, why not peruse Tcl's hard-won experience on
	selecting-on-windoze-pipes ?

	Then, for the API exposed to the Python programmer, the Tclly exposed
	one is a starter:

		fileevent $channel readable|writable callback
		...
		vwait breaker_variable

	Explanation for non-Tclers: fileevent hooks the callback, vwait does a
	loop of select(). The callback(s) is(are) called without breaking the
	loop, unless $breaker_variable is set, at which time vwait returns.

	One note about 'breaker_variable': I'm not sure I like it. I'd prefer
	something based on exceptions. I don't quite understand why it's not
	already this way in Tcl (which has (kindof) first-class exceptions), but
	let's not repeat the mistake: let's suggest that (the equivalent of)
	vwait loops forever, only to be broken out by an exception from within
	one of the callbacks.
			.
			.
			.
I've copied everything Alex wrote, because he writes for
me, also.

As much as I welcome it, I can't answer Guido's question,
"What should the API look like?"  I've been mulling this
over, and concluded I don't have sufficiently deep know-
ledge to be trustworthy on this.

Instead, I'll just give a bit of personal testimony.  I
made the rather coy c.l.p posting, in which I sincerely
asked, "How do you expert Pythoneers do it?" (my para-
phrase), without disclosing either that Alex and I have
been discussing this, or that the Tcl interface we both
know is simply a delight to me.

Here's the delight.  Guido asked, approximately, "What's
the point?  Do you need this for more than the keeping-
the-GUI-responsive-for-which-there's-already-a-notifier-
around case?"  The answer is, yes.  It's a good question,
though.  I'll repeat what Alex has said, with my own em-
phasis:  Tcl gives a uniform command API for
* files (including I/O ports, ...)
* subprocesses
* TCP socket connections
and allows the same fcntl()-like configuration of them
all as to encodings, blocking, buffering, and character
translation.  As a programmer, I use this stuff
CONSTANTLY, and very happily.  It's not just for GUIs; 
several of my mission-critical delivered products have
Tcl-coded daemons to monitor hardware, manage customer
transactions, ...  It's simply wonderful to be able to
evolve a protocol from a socket connection to an fopen()
read to ...

Tcl is GREAT at "gluing".  Python can do it, but Tcl has
a couple of years of refinement in regard to portability
issues of managing subprocesses.  I really, *really*
miss this stuff when I work with a language other than
Tcl.

I don't often whine, "Language A isn't language B."  I'm
happy to let individual character come out.  This is,
for me, an exceptional case.  It's not that Python doesn't
do it the Tcl way; it's that the Tcl way is wonderful, and
moreover that Python doesn't feel to me to have much of an
alternative answer.  I conclude that there might be some-
thing for Python to learn here.

A colleague has also write an even higher-level wrapper in
Tcl for asynchronous sockets.  I'll likely explain more
about it <URL:http://www-users.cs.umn.edu/~dejong/tcl/EasySocket.tar.gz>
in a follow-up.

Conclusion for now:  Alex and I like Python so much that we
want you guys to know that better piping-gluing-networking
truly is possible, and even worthwhile.  This is sort of
like the emigrants who've reported, "Yeah, here's the the
stuff about CPAN that's cool, and how we can have it, too."
Through it all, we absolutely want Python to continue to be
Python.