Mailman 3 December 1999 - Python-Dev

Missing POSIX functions: the list
by Andrew M. Kuchling 11 Dec '99

11 Dec '99

After poking around in the O'Reilly POSIX book, here's a list of POSIX functions that don't seem to be available in Python. Not all of them seem worth supporting. Ironically, Greg Ward's daemonize() Perl subroutine, which started me on this, doesn't actually seem to need anything that Python doesn't have. I'm looking for corrections to the list; are there other POSIX functions I've missed, or are some of them actually in Python? I think implementing most of these functions is straightforward, with the exception of opendir/readdir/closedir. Worth adding? ============= opendir(), readdir(), closedir() -- most of their functionality is available through os.listdir(), but it might be useful to have a direct interface. Downside is that this would require a new extension type for the C DIR struct. My (lazy) inclination is to not bother. Worth adding: ============= abort() -- used in Py_FatalError(), but not accessible to Python code ctermid(), ctermid_r() -- returns the terminal pathname -- probably just add ctermid(), but use ctermid_r() for thread-safety fpathconf(fd, name) -- Get configuration limit for a file -- would need constants from unistd.h getlogin() -- returns user's login name -- could do something similar with pwd.getpwuid( os.getuid() )[0], but getlogin() apparently looks in utmp getgroups(gidsetsize, grouplist) -- Gets supplementary group IDs pathconf(path, name) -- Gets config variables for a path -- would need constants from unistd.h sysconf(int name) -- Gets system configuration information -- would need constants from unistd.h Not worth adding: ================= clearerr() -- looks like fileobjects call clearerr() before raising errors cuserid() -- returns user's login name -- ORA book says "Do not use this function" -- removed in 1990 POSIX difftime -- seems only required in C "because no addition properties are defined for time_t" (Solaris man page) tmpfile(), tmpnam() -- Create temp file, generate temp filename -- Similar functionality available in tempfile.py mblen(), mbstowcs(), mbtowc(), wcstombs(), wctomb() -- Multi-byte character functions: -- Don't bother; wait for the Unicode type. -- A.M. Kuchling http://starship.python.net/crew/amk/ I'm sorry I became abusive just now ... calling you worms... I was just speaking relatively, you understand. -- Dekko, in ZOT! #3

6 9

Zip format (was: Questions about distutils strategy )
by Jack Jansen 11 Dec '99

11 Dec '99

Is it possible nowadays to have two files with the same name but different paths (i.e. foo/bar.py and foo/spam/bar.py) in the same archive? That's the one thing that always struck me as very very silly about zipfiles. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

5 4

dbm clone with serious specs wanted
by Guido van Rossum 11 Dec '99

11 Dec '99

Someone has asked me for a dbm clone that can store 16M keys of 350 bytes each, and runs on Linux, HPUX, and NT. That's 5.6 Gigabyte in keys alone! I presume most classic approaches won't cut it since total file size is typicall limited by the seek system call, internal data structures and/or file index format to 2Gb (signed longs) or 4Gb (unsigned longs). Does anyone have an idea where to start looking? Would a Python extension already exist? --Guido van Rossum (home page: http://www.python.org/~guido/)

3 3

posix module test suite
by Fred L. Drake, Jr. 10 Dec '99

10 Dec '99

There's not a test for the posix or os modules; if anyone would like to contribute one, this would be a good time! ;-) -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives

2 1

forwarded message from Fred L. Drake
by Fred L. Drake, Jr. 10 Dec '99

10 Dec '99

OK, I've checked in some changes to the posix module to add support for a few of the POSIX interfaces Andrew expressed interest in seeing (and some he said weren't such a good idea, or at least not necessary, but about which I decided I disagreed after all). For those of you who aren't on the checkins list (??), I've attached the message so you'll know what functions were added. -Fred -- Fred L. Drake, Jr. <fdrake(a)acm.org> Corporation for National Research Initiatives

3 3

Python 1.6 status
by Guido van Rossum 08 Dec '99

08 Dec '99

Greg Stein recently reminded me that he was holding off on 1.6 patches because he was under the impression that I wasn't accepting them yet. The situation is rather more complicated than that. There are a great deal of things that need to be done, and for many of them I'd be most happy to receive patches! For other things, however, I'm still in the requirements analysis phase, and patches might be premature (e.g., I want to redesign the import mechanisms, and while I like some of the prototypes that have been posted, I'm not ready to commit to any specific implementation). How do you know for which things I'm ready for patches? Ask me. I've tried to make lists before, and there are probably some hints in the TODO FAQ wizard as well as in the "requests" section of the Python Bugs List. Greg also suggested that I might receive more patches if I opened up the CVS tree for checkins by certain valued contributors. On the one hand I'm reluctant to do that (I feel I have a pretty good track record of checking in patches that are mailed to me, assuming I agree with them) but on the other hand there might be something to say for this, because it gives contributors more of a sense of belonging to the inner core. Of course, checkin privileges don't mean you can check in anything you like -- as in the Apache world, changes must be discussed and approved by the group, and I would like to have a veto. However once a change is approved, it's much easier if the contributor can check the code in without having to go through me all the time. A drawback may be that some people will make very forceful requests to be given checkin privileges, only to never use them; just like there are some members of python-dev who have never contributed. I definitely want to limit the number of privileged contributors to a very small number (e.g. 10-15). One additional detail is the legal side -- contributors will have to sign some kind of legal document similar to the current (wetsign.html) release form, but guiding all future contributions. I'll have to discuss this with CNRI's legal team. Greg, I understand you have checkin privileges for Apache. What is the procedure there for handing out those privileges? What is the procedure for using them? (E.g. if you made a bogus change to part of Apache you're not supposed to work on, what happens?) I'm hoping for several kind of responses to this email: - uncontroversial patches - questions about whether specific issues are sufficiently settled to start coding a patch - discussion threads opening up some issues that haven't been settled yet (like the current, very productive, thread in i18n) - posts summarizing issues that were settled long ago in the past, requesting reverification that the issue is still settled - suggestions for new issues that maybe ought to be settled in 1.6 - requests for checkin privileges, preferably with a specific issue or area of expertise for which the requestor will take responsibility --Guido van Rossum (home page: http://www.python.org/~guido/)

14 54

Re: [Python-Dev] Re: .DLL vs .PYD search order
by Jack Jansen 07 Dec '99

07 Dec '99

On the Mac I've introduced "magic cookies" into sys.path, which allow you to do interesting searches (like searching for a DLL or PYC-resource in the application itself) at known places in the import process. There isn't a cookie for "search along the standard MacOS dll search path" (which is somewhat similar to the Windows dll search path) because I haven't seen a reason for it, but there's nothing to stop it. And if you'd insert that cookie it would be perfectly clear (at least, it should be) that only dll modules will be found in that step, not .py modules. Actually I'm so happy with the magic cookie scheme that I've advocated at various times in the past that something similar also be used for determining where builtin modules and frozen modules appear in sys.path... -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen(a)oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.oratrix.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm

9 24

Re: [Python-Dev] Re: .DLL vs .PYD search order
by Fredrik Lundh 06 Dec '99

06 Dec '99

> > you obviously attempted to use imputil to implement > > non-standard import behaviour on top of the standard > > storage system -- while we've used it to implement > > standard import behaviour on top of non-standard > > storage systems. > > No, I tried to make the imputil approach work as replacement > for the standard builtin importer. I'm confused. earlier, you said (or rather, I think you said) that you looked at imputil to see if it could "handle the problems you had at the time"... and now you say that you tried to use it as a drop-in replacement for the "standard path importer". I must be missing something here... > After I got that to work, I added some caching > to avoid duplicated stats. The resulting importer was > around twice as slow as the builtin one for the following > imports: > > # the default one Python does at startup, plus: > from mx import HTMLTools,DateTime,ODBC > > This is a pretty common setup for my scripts, so its > preformance is relevant to me. did you try stuffing all your PYC's into an archive file, and running them from there? </F>

1 0

Re: [Python-Dev] Import redesign [LONG]
by Greg Stein 03 Dec '99

03 Dec '99

[ taking the liberty to CC: this back to python-dev ] On Fri, 19 Nov 1999, David Ascher wrote: > > > (2) a file in a directory that's on sys.path can be a zip/jar file; > > > its contents will be considered as a package (note that this is > > > different from (1)!) > > > > No problem. This will slow things down, as a stat() for *.zip and/or *.jar > > must be done, in addition to *.py, *.pyc, and *.pyo. > > Aside: it strikes me that for Python programs which import lots of files, > 'front-loading' the stat calls could make sense. When you first look at a > directory in sys.path, you read the entire directory in memory, and > successive imports do a stat on the directory to see if it's changed, and > if not use the in-memory data. Or am I completely off my rocker here? Not at all. I thought of this last night after my email. Since the Importer can easily retain state, it can hold a cache of the directory listings. If it doesn't find the file in its cached state, then it can reload the information from disk. If it finds it in the cache, but not on disk, then it can remove the item from its cache. The problem occurs when you path is [A, B], the file is in B, and you add something to A on-the-fly. The cache might direct the importer at B, missing your file. Of course, with the appropriate caveats/warnings, the system would work quite well. It really only breaks during development (which is one reason why I didn't accept some caching changes to imputil from MAL; but that was for the Importer in there; Python's new Importer could have a cache). I'm also not quite sure what the cost of reading a directory is, compared to issuing a bunch of stat() calls. Each directory read is an opendir/readdir(s)/closedir. Note that the DBM approach is kind of similar, but will amortize this cost over many processes. Cheers, -g -- Greg Stein, http://www.lyra.org/

5 9

updated imputil
by Greg Stein 03 Dec '99

03 Dec '99

I've updated imputil... The main changes is that I added SysPathImporter and BuiltinImporter. I also did some restructing to help with bootstrapping the module (remove dependence on os.py). For testing a revamped Python import system, you can importing the thing and call imputil._test_revamp() to set it up. This will load normal, builtin, and frozen modules via imputil. Dynamic modules are still handled by Python, however. I ran a timing comparisons of importing all modules in /usr/lib/python1.5 (using standard and imputil-based importing). The standard mechanism can do it in about 8.8 seconds. Through imputil, it does it in about 13.0 seconds. Note that I haven't profiled/optimized any of the Importer stuff (yet). The point about dynamic modules actually discovered a basic problem that I need to resolve now. The current imputil assumes that if a particular Importer loaded the top-level module in a package, then that Importer is responsible for loading all other modules within that package. In my particular test, I tried to import "xml.parsers.pyexpat". The two package modules were handled by SysPathImporter. The pyexpat module is a dynamic load module, so it is *not* handled by the Importer -- bam. Failure. Basically, each part of "xml.parsers.pyexpat" may need to use a different Importer... Off to ponder, -g -- Greg Stein, http://www.lyra.org/

8 23