From mal@lemburg.com Tue Jun 1 09:14:55 1999 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 01 Jun 1999 10:14:55 +0200 Subject: [Distutils] extensions in packages References: <374BA5C5.706EFB1E@lemburg.com> <14155.63283.936747.273032@weyr.cnri.reston.va.us> <374C20DD.53B458DC@lemburg.com> <14156.28035.739990.919106@weyr.cnri.reston.va.us> <374D0077.2699505F@lemburg.com> <374D17A7.436D8260@lyra.org> <374E4B41.13A2A1FD@lemburg.com> <374F0980.171E34F6@lyra.org> Message-ID: <3753967F.60521C5D@lemburg.com> Greg Stein wrote: > > M.-A. Lemburg wrote: > > > > Greg Stein wrote: > > >... > > > IMO, the interpreter core should perform as little searching as > > > possible. Basically, it should only contain bootstrap stuff. It should > > > look for a standard importing module and load that. After it is loaded, > > > the import mechanism should defer to Python for all future imports. (the > > > cost of running Python code is minimal against the I/O used by the > > > import) > > > > > > IMO #2, the standard importing module should operate along the lines of > > > imputil.py. > > > > You mean moving the whole import mechanism away from C and into > > Python ? Have you tried such an approach with your imputil.py ? > > Yes and yes. > > Using Python's import hook effectively means that you completely take > over Python's import mechanism (one of its failings, IMO). imputil.py is > designed to provide for iterating through a list of importers, looking > for one that works. > > In any case... yes, I've use imputil to the exclusion of Python's import > logic. You still need imp.new_module() and imp.get_magic(). But that > does implies that you can axe a lot of stuff outside of that. My tests > don't have loading of dynamic libraries, so you would still need an imp > function to load that (but strip the *searching* for the module). So all that's needed is some DLL loader magic in C, some Win32 Registry APIs and the Mac fork stuff. If imp were extended to provide those APIs instead of using them itself and then moving all the other code to Python we should arrive at a solution that would also cover things like internet import, import via pipes, import from databases, etc. > > And just curious: why did Guido recode ni.py in C if he could have > > used ni.py in your proposed way instead ? > > For two reasons that I can think of: > > 1) people had to manually import "ni" Well, I guess ni could have been imported by Python at startup -- just like exceptions is right now. The problem here: what if it doesn't find the import logic module ? Plus, how does it do import ;) ? > 2) it took over the import hook which effectively prevents further use > of it (or if somebody *did* use it, then they would wipe out ni's > functionality; again, this is why I dislike the current hook approach > and like a list-based approach which is possible via imputil) Right. BTW: that's a general problem with all kinds of hooks, e.g. the system exit hook is another problem area. A generic solution to this wouldn't be a bad idea either. Here is one way to do this for the sys.exitfunc hook that I'm currently using: """ Central Registry for sys.exitfunc()-type functions. """ __version__ = "0.1" import sys,traceback class ExitFunctionDispatcher: """ Singleton that manages exit functions. These function will be called upon system exit in reverse order of their registering. """ def __init__(self): """ Install the dispatcher as sys.exitfunc() """ self.exitfunc_list = [] if hasattr(sys,'exitfunc'): self.old_exitfunc = sys.exitfunc else: self.old_exitfunc = None sys.exitfunc = self.exitfunc def exitfunc(self, write=sys.stderr.write,print_exc=traceback.print_exc, stderr=sys.stderr): """ This is the exitfunc that we install to dispatch the processing to the registered other functions """ for f in self.exitfunc_list: try: f() except: write('Error while executing Exitfunction %s:\n' % f.__name__) print_exc(10,stderr) # Now that we're finished, call the previously installed exitfunc() if self.old_exitfunc: self.old_exitfunc() def register(self,f,position=0): """ Register f as exit function. These functions must not take parameters. - position = 0: register the function at the beginning of the list; these functions get called before the functions already in the list (default) - position = -1: register the function at the end of the list; the function will get called after all other functions """ if position < 0: position = position + len(self.exitfunc_list) + 1 self.exitfunc_list.insert(position,f) def deregister(self,f): """ Remove the function f from the exitfunc list; if it is not found, the error is silently ignored. """ try: self.exitfunc_list.remove(f) except: pass # Create the singleton ExitFunctions = ExitFunctionDispatcher() -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 213 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From arcege@shore.net Tue Jun 1 16:42:47 1999 From: arcege@shore.net (Michael P. Reilly) Date: Tue, 1 Jun 1999 11:42:47 -0400 (EDT) Subject: [Distutils] shared extensions that use other shared extensions Message-ID: <199906011542.LAA14576@northshore.shore.net> Hi folks, I ran into a small problem this weekend while working on a potential redesign of _tkinter.c that I thought I should bring up here. This is not so much a change to distutils code, or the addition of a feature. But it is a big "gotcha" on UNIX, where shared objects could reside in different directories (as has been discussed here lately). Just to give a little rundown of the problem. A few weeks ago, someone on c.l.py asked if there was a Tcl interpreter extension. Some people (including myself) stated that there was one in Tkinter, but couldn't divorce yourself from Tk. So I got it in my head to make a Tcl module then redesign _tkinter.c to use that module instead. Now the problem: To be thread safe (especially with Tcl8.0), there is a global lock object used when entering all the Tcl/Tk code. This object should be shared between the two modules (it is using the same Tcl_Interp object). Not a problem, right? In the Python init_tkinter() function, simply import the Tcl module and get access to the global data in the shared module. The import.c code would load the dynamic modules and handle it. Not quite. When the _tkinter.so loads, it attempts to load the Tcl module shared object as well, not based on sys.path but on LD_LIBRARY_PATH (or what every was set at link-time). I got around this by making a PyCObject out of the Tcl lock object in the module, then accessing that inside init_tkinter(), thereby making no references to the Tcl module. Now to my point to all this. I have two, in fact. First, when writing modules, try not rely on global data in other Python modules unless it is thru the Python API. You might find that your runtime link-editor can't handle finding the shared library file. Second, it may not always be enough to place the shared object modules in site-packages or the like. I can easily imagine that SWIG might make references to something inadvertently that is outside the view that import.c (or the imp module) has of the world. -Arcege -- ------------------------------------------------------------------------ | Michael P. Reilly, Release Engineer | Email: arcege@shore.net | | Salem, Mass. USA 01970 | | ------------------------------------------------------------------------ From gward@cnri.reston.va.us Wed Jun 2 19:07:48 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Wed, 2 Jun 1999 14:07:48 -0400 Subject: [Distutils] complete GNU readline support, packaging issues In-Reply-To: <199905271632.JAA26633@hpcll563.cup.hp.com>; from Ovidiu Predescu on Thu, May 27, 1999 at 09:32:41AM -0700 References: <199905271632.JAA26633@hpcll563.cup.hp.com> Message-ID: <19990602140747.A22811@cnri.reston.va.us> On 27 May 1999, Ovidiu Predescu said: > I started to investigate the ways to package the work I've done and I > discovered the distutil package. I'm trying to write a setup.py file, but I > need the following things and I didn't figure out how to express them: > > - the main C file is obtained by running the m4 program on a .m4 file. How can > I specify this dependency and the rule for generating the C file? > > - I need to define a C preprocessor macro that contains the version of the > readline library. The way the things are setup now is by running a configure > script that determines the version of the library. Can I do this with setup.py? Alas, the Distutils suite just doesn't have this capability yet: there is no support for building extension modules. All the current version can do is copy .py and .pyc files around. The key thing is that it codifies a standard way for copying them around, and provides a fairly flexible OO framework for adding stuff like building/installing extension modules. Unfortunately, I just haven't gotten around to doing that stuff yet. Sorry... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Tue Jun 8 03:14:22 1999 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 7 Jun 1999 22:14:22 -0400 Subject: [Distutils] Two fixes (at last) Message-ID: <19990607221421.A1343@cnri.reston.va.us> Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From da@ski.org Tue Jun 8 17:58:05 1999 From: da@ski.org (David Ascher) Date: Tue, 8 Jun 1999 09:58:05 -0700 (Pacific Daylight Time) Subject: [Distutils] Two fixes (at last) In-Reply-To: <19990607221421.A1343@cnri.reston.va.us> Message-ID: On Mon, 7 Jun 1999, Greg Ward wrote: > Hope someone (Amos?) can try the new version under NT. Any takers for > Mac OS? One buglet: in util.py, copy_file, the order of the chmod and utime need to be swapped to: if preserve_times: os.utime (dst, (st[ST_ATIME], st[ST_MTIME])) if preserve_mode: os.chmod (dst, S_IMODE (st[ST_MODE])) (otherwise the chmod can prevent the utime from doing its thing). --david From da@ski.org Tue Jun 15 00:13:44 1999 From: da@ski.org (David Ascher) Date: Mon, 14 Jun 1999 16:13:44 -0700 (Pacific Daylight Time) Subject: [Distutils] inertia Message-ID: I'm reorganizing the PyOpenGL source tree. I'd love to move to a distutils distribution process instead of my current complicated homegrown system. However, I'm not willing to do that without the ability to generate .so's simply on Unices (I can do the win32 portion, see below). Is anyone working on this? I'd hoped to have the distribution story moved along further by the time the ORA conference rolls around, but it's not looking likely. Here's a dare: if someone does the Unix .c->.so->'simple installation' process, I'll do the win32 equivalent. My guess is that the first step is a pseudo-parse of the Makefile.pre.in file, no? Once the compilation process is in place (even buggy), it makes it more likely for others to contribute snippets for e.g. RPM generation, etc. --david From jjacobs@aceltd.com Thu Jun 17 19:22:09 1999 From: jjacobs@aceltd.com (Jeffrey C. Jacobs) Date: Thu, 17 Jun 1999 14:22:09 -0400 Subject: [Distutils] RE: [Crew] Wizards' Resolution to Zope/PIL/mxDateTime conflict? Message-ID: <237BD340B58ED111857500A0C99DBA9C3953A1@ace-pdc.aceltd.com> > -----Original Message----- > From: Tom Bryan [SMTP:tbryan@server.python.net] > Sent: June 17, 1999 12:19 PM > To: Crew List > Subject: Re: [Crew] Wizards' Resolution to Zope/PIL/mxDateTime > conflict? > > Perhaps this discussion should be moved to the PSA list or to the > comp.lang.python newsgroup. The Starship isn't going to be the only > computer with this problem. We juat are going to be one of the few > machines that currently has *many* third-party extensions to Python. As > Python grows in popularity, this problem is only going to get worse. In > the worst case scenario, people won't be able to use a Python solution > because the required third-party extensions conflict and they don't want > to kludge it all together themselves. > It seems to me perhaps the simplest solution is, dare I say it, Bureaucracy! Okay, now that I got you all riled up, let me explain. Right now, anyone can go write a Python package and post it up all over the place, helter-skelter. It may even appear on www.python.org. However, if I'm writing Package A and you have written package B unbeknownst to me, I might be stepping on your toes. That is to say, if my package does the same thing, so what? more choice to the hacker. The problem is when I CALL my module/package the same thing as you. It is this naming convention that needs to be cleaned up, as I see it. Rather than come up with some complex scheme of installation or some major changes to the python language, wouldn't it just be simpler to standardize our naming now, before Python does reach the 10,000,000 user mark? That is, rather then trying to come up with an installation-level solution, why not just have someone review all packages for conflicts on a first-come-first-serve basis? Set up a committee of Python certification which will basically go through each new package and install it on a max-install station. If there are no conflicts, the package is certified and the author gets official recognition. If there is conflict, a polite e-mail can be sent to the author suggesting he or she modify / change the name(s) chosen for his or her package to avoid future conflict potential. As for currently existing conflicts, there is of course the problem with install-base. The best thing I can suggest is a renaming of 1 or more existing packages to conform to the Python standard extension model. That is to say, there would then be two copies of the package available -- 1 with backwards-compatible naming and 1 which is the Python-certified name. Based on the desire of the package author / maintainer, the old names could be phased out as time goes by in new releases. Typically, the more ingrained a package is in an application, the less likely it is that a conflicting package will be installed on that system. Thus *hopefully* this should not effect any previous install of Python. It would probably be best when conflicts *do* occur to ask *both* pre-existing Package authors to try to come up with new names, so as to be more even-handed, though I'd suggest larger packages get priority. The best thing is, when Microsoft starts to see Python as a threat to its Visual xxx monopoly and starts to "embrace" Python with their new Visual P++ integrated development environment, and starts adding weird packages left, right and center, the Python certification committee can just "uncertify" the extensions and (hopefully) prevent the destruction of the language the same way they did Java. I think Mark Hammond would be heading Microsoft off at the pass, as it were WRT this issue though. What's more, we here at Starship seem to be the perfect candidate to be the "Python Package" machine, at least on the Unix / Linux side -- I should hope a similar all-install machine could be set up for Microsoft-based system verification, as well as Mac and any other OS which has a unique package list. > Does someone want to make an intelligent, concise post to one of the other > forums? comp.lang.python has enough posters who want to "improve" the > language. Perhaps we could direct their energy into a useful change. > I am sending a copy of this over there... Be Seeing You, Jeffrey. From akuchlin@mems-exchange.org Thu Jun 17 19:45:36 1999 From: akuchlin@mems-exchange.org (Andrew M. Kuchling) Date: Thu, 17 Jun 1999 14:45:36 -0400 (EDT) Subject: [Distutils] Wizards' Resolution to Zope/PIL/mxDateTime conflict? In-Reply-To: <237BD340B58ED111857500A0C99DBA9C3953A1@ace-pdc.aceltd.com> References: <237BD340B58ED111857500A0C99DBA9C3953A1@ace-pdc.aceltd.com> Message-ID: <14185.16976.172870.869562@amarok.cnri.reston.va.us> Jeffrey C. Jacobs writes: >packages for conflicts on a first-come-first-serve basis? Set up a >committee of Python certification which will basically go through each new >package and install it on a max-install station. If there are no conflicts, >the package is certified and the author gets official recognition. If there Too much bureaucracy; won't fly. Either Java's org.python.core arrangement will have to be used, or some other automatic disambiguation method will have to be used; intervention of an outside committee isn't acceptable, because then I can't name anything without contacting them. -- A.M. Kuchling http://starship.python.net/crew/amk/ Nothing I've ever written has reached 1.0. -- Greg Ward at IPC7, on using small version numbers From arcege@shore.net Thu Jun 17 19:48:55 1999 From: arcege@shore.net (Michael P. Reilly) Date: Thu, 17 Jun 1999 14:48:55 -0400 (EDT) Subject: [Distutils] RE: [Crew] Wizards' Resolution to Zope/PIL/mxDateTime conflict? In-Reply-To: <237BD340B58ED111857500A0C99DBA9C3953A1@ace-pdc.aceltd.com> from "Jeffrey C. Jacobs" at "Jun 17, 99 02:22:09 pm" Message-ID: <199906171848.OAA24744@northshore.shore.net> [Charset iso-8859-1 unsupported, filtering to ASCII...] > > > -----Original Message----- > > From: Tom Bryan [SMTP:tbryan@server.python.net] > > Sent: June 17, 1999 12:19 PM > > To: Crew List > > Subject: Re: [Crew] Wizards' Resolution to Zope/PIL/mxDateTime > > conflict? > > > > Perhaps this discussion should be moved to the PSA list or to the > > comp.lang.python newsgroup. The Starship isn't going to be the only > > computer with this problem. We juat are going to be one of the few > > machines that currently has *many* third-party extensions to Python. As > > Python grows in popularity, this problem is only going to get worse. In > > the worst case scenario, people won't be able to use a Python solution > > because the required third-party extensions conflict and they don't want > > to kludge it all together themselves. > It seems to me perhaps the simplest solution is, dare I say > it, Bureaucracy! Okay, now that I got you all riled up, let me explain. > Right now, anyone can go write a Python package and post it up all over the > place, helter-skelter. It may even appear on www.python.org. However, if > I'm writing Package A and you have written package B unbeknownst to me, I > might be stepping on your toes. That is to say, if my package does the same > thing, so what? more choice to the hacker. The problem is when I CALL my > module/package the same thing as you. It is this naming convention that > needs to be cleaned up, as I see it. Rather than come up with some complex > scheme of installation or some major changes to the python language, > wouldn't it just be simpler to standardize our naming now, before Python > does reach the 10,000,000 user mark? That is, rather then trying to come up > with an installation-level solution, why not just have someone review all > packages for conflicts on a first-come-first-serve basis? Set up a > committee of Python certification which will basically go through each new > package and install it on a max-install station. If there are no conflicts, > the package is certified and the author gets official recognition. If there > is conflict, a polite e-mail can be sent to the author suggesting he or she > modify / change the name(s) chosen for his or her package to avoid future > conflict potential. [Other discussion snipped] > Be Seeing You, > > Jeffrey. Jeffrey, there is already a service for this, Guido set it up some time ago: go to http://www.python.org/search/ and select the link "The Python module registry at NIST". This service needs more press and perhaps more "peer pressure" from module authors. -Arcege From jim@interet.com Fri Jun 25 19:29:30 1999 From: jim@interet.com (James C. Ahlstrom) Date: Fri, 25 Jun 1999 14:29:30 -0400 Subject: [Distutils] Put Python library into binary Message-ID: <3773CA8A.43691D3E@interet.com> Hello everyone, It looks like you have been working on putting the Python library into the binary. I looked in Tools/freeze at freeze.py. Please check my understanding of the following: 1) Use of freeze.py creates a frozen module with the Python library in it. 2) There is always a __main__.py program incorporated. 3) No further use of frozen modules is permitted since there is only one frozen module allowed. Number (2) and (3) are a problem for me, but maybe I am wrong about how this all works. Jim Ahlstrom From jim@interet.com Mon Jun 28 14:07:26 1999 From: jim@interet.com (James C. Ahlstrom) Date: Mon, 28 Jun 1999 09:07:26 -0400 Subject: [Distutils] Anyone home? Message-ID: <3777738E.DA38BB40@interet.com> Hi, Is this sig active? Is anyone out there? Jim Ahlstrom From jim@interet.com Wed Jun 30 15:21:06 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 30 Jun 1999 10:21:06 -0400 Subject: [Distutils] Add multiple frozen modules Message-ID: <377A27D2.96BD5CE8@interet.com> Hi, I have been working with the Tools/freeze/freeze.py etc. utilities in order to build a Python binary with part of the Python library built in. I found out you-all have been working on this. But AFAIK we are not there yet. I see that using freeze.py will create a binary Python with the libs included in the frozen module. But: 1) The "script" argument to freeze.py is not optional. Some module is always frozen with the name __main__ and so it is always executed when Python starts up. This makes it impossible to create a python15.dll (or python15.so) which is general-purpose. That is, a shared binary Python with libraries but not a main program built in. Since the Python interpreter and libs must use exactly the same version, it is useful to link them together. 2) There is only one frozen module allowed, so no further use of frozen modules is possible. The frozen feature has been devoted to the Python libraries. This makes it impossible to make a python15.dll which can be used with a variety of python.exe programs with further frozen modules. Unix translation: A python15.so shared library for use with further python main programs containing frozen modules. I have code which solves both problems. But perhaps I have not understood Tools/freeze, and there is already a way to do this? I see there is a function PyImport_ExtendInittab() in import.c which adds entries to the table of built-in modules, which is sort of analogous to (2), but that this function is unused. (My code uses a different method.) Please let me know what you think. Is this useful enough to be added to Python? Or is it already there? Jim Ahlstrom From MHammond@skippinet.com.au Wed Jun 30 17:07:53 1999 From: MHammond@skippinet.com.au (Mark Hammond) Date: Thu, 1 Jul 1999 02:07:53 +1000 Subject: [Distutils] Add multiple frozen modules In-Reply-To: <377A27D2.96BD5CE8@interet.com> Message-ID: <008e01bec312$b646bf30$0801a8c0@bobcat> [Jim laments about freeze and some good enhancements he whipped up] Here is my personal take on this. It is formed simply by working with freeze, at one stage attempting to generate interest in helping extend freeze, and watching what other people do when faced with a similar problem. I get the impression freeze is dieing. It is dieing in favour of techniques that avoid requiring a C compiler and allow an even more generic concept of where "frozen" code can come from. Although avoiding the C compiler is particularly attractive on Windows (where users can't be assumed to have one), it offers a number of other benefits on all platforms. Specifically, Greg Stien and Gordon McMillan (and plenty of people before them :-) seem to have what is considered "state of the art" in where this is heading. But "where it is heading" is the rub. Freeze is still well ahead in a number of significant areas. The ideas you describe are definately worth-while, but freeze works OK for me now - so whenever I next get a decent block of time to play with this stuff some more, I am likely to focus on these other efforts other than freeze. I dont think you will find many people arguing either way on this. My only comment is that re Point 2: PyImport_ExtendInittab() is indeed able to give the functionality you describe. Each seperate frozen package can call this function to add its frozen modules. So this means your patch for this should be unnecessary (or at least slightly different to take advantage of this). Re point 1: This is possible (or nearly :) for Windows freeze builds, but AFAIK not for Unix builds, so this definately seems worthwhile. And-it-still-needs-more-hacks-to-complete-COM-support-ly, Mark. From jim@interet.com Wed Jun 30 17:59:26 1999 From: jim@interet.com (James C. Ahlstrom) Date: Wed, 30 Jun 1999 12:59:26 -0400 Subject: [Distutils] Add multiple frozen modules References: <008e01bec312$b646bf30$0801a8c0@bobcat> Message-ID: <377A4CEE.C4EC0C9C@interet.com> [Hmm.... Jim looks at clock and wonders what Mark is doing down in Australia in the middle of the night...] Mark Hammond wrote: > > I get the impression freeze is dieing. It is dieing in favour of > techniques that avoid requiring a C compiler and allow an even more generic > concept of where "frozen" code can come from. Although avoiding the C > compiler is particularly attractive on Windows (where users can't be > assumed to have one), it offers a number of other benefits on all > platforms. Specifically, Greg Stien and Gordon McMillan (and plenty of > people before them :-) seem to have what is considered "state of the art" > in where this is heading. I don't know of techniques which aren't Windows specific. Greg?? Gordon?? > I dont think you will find many people arguing either way on this. My only > comment is that re Point 2: PyImport_ExtendInittab() is indeed able to give > the functionality you describe. I don't think so. It adds to the list of available compiled-in modules, so that were they imported, they would be available for import. But the problem is how to get them imported. In particular, "__main__" would never be called unless it were explicitly a frozen module. The list of frozen modules PyImport_FrozenModules is not the same as PyImport_Inittab. Actually, I am not sure how to solve this even in my current code, except in a Windows-specific way. > Re point 1: This is possible (or nearly :) for Windows freeze > builds, but AFAIK not for Unix builds, so this definately seems worthwhile. Jim Ahlstrom