Mailman 3 July 1999 - Python-Dev

RE: [Python-Dev] RE: NT select.select?
by Brian Lloyd July 30, 1999

July 30, 1999

Guido wrote: > > Brian, > > (Also in response to your bug report.) I'm a little worried that > upping the limit to 1024 would cause some performance problems if > you're making a lot of select() calls. The select allocates three > arrays of length FD_SETSIZE+3; each array item is 12 bytes. This is a > total allocation of more than 36K for a meager select() call! > And all > that memory also has to be cleared by the FD_ZERO() call. > > If you actually have that many sockets, that's worth paying for (the > socket objects themselves use up just as much memory, and your Python > data structures for the sockets, no matter how small, are probably > several times bigger), but for a more typical program, I see > this as a > lot of overhead. > > Is there a way that this can be done more dynamically, e.g. by making > the set size as big as needed on windows but no bigger? > > (Before you suggest allocating that memory statically, remember it's > possible to call select from multiple threads. Allocating 36K of > thread-local space for each thread also doesn't sound too pleasant.) > > --Guido van Rossum (home page: http://www.python.org/~guido/) Hmm - after going through all of the Win32 sdks, it doesn't appear to be possible to do it any other way than as a -D option at compile time, so optimizing for the common case (folks who _don't_ need large numbers of fds) is reasonable. Since we distribute a python15.dll with Zope on windows, this isn't that big a deal for us - we can just compile in a higher limit in our distributed dll. I was mostly thinking of the win32 users who don't have the ability to rebuild their dll, but maybe this isn't that much of a problem; I suspect that the people who write significant socket apps that would run into this problem probably have access to a compiler if they need it. Brian Lloyd brian(a)digicool.com Software Engineer 540.371.6909 Digital Creations http://www.digicool.com

3 2

RE: NT select.select?
by Brian Lloyd July 30, 1999

July 30, 1999

> Is there some low limit on maximum number of sockets you can > have in the > Python-NT's select call? A program that happens to work > perfectly on Linux > seems to die on NT around 64(?) sockets to the 'too many file > descriptors > in call' error. > > Any portable ways to bypass it? > > -Markus Hi Markus, It turns out that NT has a default 64 fd limit on arguments to select(). The good news is that you can actually bump the limit up to whatever number you want by specifying a define when compiling python15.dll. If you have the ability to rebuild your python15.dll, you can add the define: FD_SETSIZE=1024 to the preprocessor options for the python15 project to raise the limit to 1024 fds. The default 64 fd limit is too low for anyone trying to run an async server that handles even a modest load, so I've submitted a bug report to python.org asking that the define above find its way into the next python release... Brian Lloyd brian(a)digicool.com Software Engineer 540.371.6909 Digital Creations http://www.digicool.com

2 1

Re: [Python-Dev] End of the line
by Jack Jansen July 29, 1999

July 29, 1999

> [Jack seems to like an asynch IO model] > > > def foo(): > > obj = stdin.delayed_read() > > obj2 = stdout.delayed_write("data") > > do_lengthy_computation() > > data = obj.get() # Here we wait for the read to complete > > del obj2 # Here we wait for the write to > > complete. > > > > This gives a fairly nice programming model. > > Indeed. Taking this a little further, I come up with something like: > > inlock = threading.Lock() > buffer = stdin.delayed_read(inlock) > > outlock = threading.Lock() > stdout.delayed_write(outlock, "The data") > > fired = threading.Wait(inlock, outlock) # new fn :-) > > if fired is inlock: # etc. I think this is exactly what I _didn't_ want:-) I'd like the delayed read to return an object that will automatically wait when I try to get the data from it, and the delayed write object to automatically wait when I garbage-collect it. Of course, there's no reason why you couldn't also wait on these objects (or, on unix, pass them to select(), or whatever). On second thought the method of the delayed read should be called read() in stead of get(), of course. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

3 3

Tcl news
by David Ascher July 28, 1999

July 28, 1999

8.2b1 is released: Some surprising news: they now use cygwin tools to do the windows build. Not surprising news: they still haven't incorporated some bug fixes I submitted eons ago =) http://www.scriptics.com/software/relnotes/tcl8.2b1 --david

1 0

Re: [Python-Dev] End of the line
by Jack Jansen July 27, 1999

July 27, 1999

> What Im getting at is that a Python IO model should maybe go a little > further than "tradtional" IO - asynchronous IO and synchronisation > capabilities should also be specified. Of course, these would be optional, > but it would be excellent if a platform could easily slot into pre-defined > Python semantics if possible. What Python could do with reasonable ease is a sort of "promise" model, where an I/O operation returns an object that waits for the I/O to complete upon access or destruction. Something like def foo(): obj = stdin.delayed_read() obj2 = stdout.delayed_write("data") do_lengthy_computation() data = obj.get() # Here we wait for the read to complete del obj2 # Here we wait for the write to complete. This gives a fairly nice programming model. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

2 1

continuations for the curious
by Christian Tismer July 27, 1999

July 27, 1999

Howdy, my modules are nearly ready. I will be out of my office for two weeks, but had no time to finalize and publish yet. Stackless Python has reached what I wanted it to reach: A continuation can be saved at every opcode. The continuationmodule has been shrunk heavily. Some extension is still needed, continuations are still frames, but they can be picked like Sam wanted it originally. Sam, I'm pretty sure this is more than enough for coroutines. Just have a look at getpcc(), this is now very easy. All involved frames are armed so that they *can* save themselves, but will do so just if necessary. The cheapest solution I could think of, no other optimization is necessary. If your coroutine functions like to swap two frames, and if they manage to do so that the refcount of the target stays at one, no extra frame will be generated. That's it, really. If someone wants to play, get the stackless module, replace ceval.c, and build continuationmodule.c as a dll or whatever. testct.py contains a lot of crap. The first implementation of class coroutine is working right. The second one is wrong by concept. later - chris ftp://ftp.pns.cc/pub/veryfar.zip -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

3 3

I'd like list.pop to accept an optional second argument giving a default value
by Jim Fulton July 27, 1999

July 27, 1999

I like the list pop method because it provides a way to use lists as thread safe queues and stacks (since append and pop are protected by the global interpreter lock). With pop, you can essentially test whether the list is empty and get a value if it isn't in one atomic operation: try: foo=queue.pop(0) except IndexError: ... empty queue case else: ... non-empty case, do something with foo Unfortunately, this incurs exception overhead. I'd rather do something like: foo=queue.pop(0,marker) if foo is marker: ... empty queue case else: ... non-empty case, do something with foo I'd be happy to provide a patch. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

9 42

RE: [Python-Dev] End of the line
by Brian Lloyd July 27, 1999

July 27, 1999

> [Tim, notes that Perl line-at-a-time text mode input runs 3x > faster than > Python's on his platform] > > And much to my surprise, it turns out Perl reads lines a > character at a time > too! And they do not reimplement stdio. But they do cheat. > > [some notes on the cheating and PerlIO api snipped] > > The usual *implementation* of these guys is as straight macro > substitution > to the corresponding C stdio call. It's possible to > implement them some > other way, but I don't see anything in the source that > suggests anyone has > done so, except possibly to build it all on AT&T's SFIO lib. Hmm - speed bonuses not withstanding, an implementation of such a beast in the Python sources would've helped a lot to reduce the ugly hairy gymnastics required to get Python going on Win CE, where (until very recently) there was no concept of most of the things you expect to find in stdio... Brian Lloyd brian(a)digicool.com Software Engineer 540.371.6909 Digital Creations http://www.digicool.com

3 2

continuationmodule.c preview
by Christian Tismer July 25, 1999

July 25, 1999

Howdy, please find attached my latest running version of continuationmodule.c which is really able to do continuations. You need stackless Python 0.3 for it, which I just submitted. This module is by no means ready. The central functions are getpcc() and putcc. Call/cc is at the moment to be done like: def callcc(fun, *args, **kw): cont = getpcc() return apply(fun, (cont,)+args, kw) getpcc(level=1) gets a parent's current continuation. putcc(cont, val) throws a continuation. At the moment, these are still frames (albeit special ones) which I will change. They should be turned into objects which have a link to the actual frame, which can be unlinked after a shot or by hand. This makes it easier to clean up circular references. I have a rough implementation of this in Python, also a couple of generators and coroutines, but all not pleasing me yet. Due to the fact that my son is ill, my energy has dropped a little for the moment, so I thought I'd better release something now. I will make the module public when things have been settled a little more. ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

5 8

Importing extension modules
by M.-A. Lemburg July 22, 1999

July 22, 1999

I'm currently testing a pure Python version of mxDateTime (my date/time package), which uses a contributed Python version of the C extension. Now, to avoid problems with pickled DateTime objects (they include the complete module name), I would like to name *both* the Python and the C extension version mxDateTime. With the current lookup scheme (shared mods are searched before Python modules) this is no problem since the shared mod is found before the Python version and used instead, so getting this working is rather simple. The question is: will this setup remain a feature in future versions of Python ? (Does it work this way on all platforms ?) Cheers, -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

1 0