Distutils-SIG July 1999

distutils-sig@python.org

15 participants
14 discussions

by Greg Ward

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 years, 10 months

Package versioning

by Timothy Docker

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

2 years, 10 months

[Distutils-sig] distutils charter and interscript

by John Skaller

2 years, 10 months

Re: Add multiple frozen modules

by Gordon McMillan

(Doing my bi-monthly perusal of Distutils activity, I find...) [Mark] > .... Specifically, Greg Stien and Gordon McMillan (and plenty of > people before them :-) seem to have what is considered "state of the > art" in where this is heading. [Jim Ahlstrom] I don't know of techniques which aren't Windows specific. Greg??Gordon?? Mark is referring to Greg's imputil.py (http://www.lyra.org/greg/small/) and some of the places I've taken it. My Win32-specific installer (ftp://ftp.python.org/pub/python/contrib/System/Installer_r_01.exe) makes use of this, but not all of it is Windows specific. (I originally packaged it so the cross-platform stuff was available separately, but there was no apparent interest from non-Windows users.) Greg's imputil.py basically makes it possible to create a chain of importers, with the standard mechanism pushed to the end. Writing an importer is easy. You set up the chain is site.py. I created a way of building archives of .pyc's (or .pyo's, though I've never worked with them). These are compressed with zlib. The standard library fits in about 500K and (subjectively) it is no slower and perhaps faster than the regular method. The mechanism handles modules and packages, and the building of an archive can be done in all kinds of ways (including using modulefinder from freeze). I also created another kind of archive that can package up arbitrary stuff, and is fairly easily unpacked from C code. All of the above is cross-platform. On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory: myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want] (The Window's installation / standalone stuff goes further than this). These .pyz's do work on Linux, but the steps Python goes through to determine where it lives is simpler on Windows. I'm not sure what it would take to get a single directory Python installation (independent of any other installed Pythons) working on *nix. - Gordon

21 years, 6 months

Re: Add multiple frozen modules

by Gordon McMillan

My installer package now has a homepage on my starship site: http://starship.python.net/crew/gmcm/install.html This page gives a pretty complete writeup. The package itself has not yet been updated (either with Robin Dunn's enhancements, or a couple bug fixes I have planned). Also from this page you can download a tar.gz of just the archiving stuff. For you Unix-weenies, I've even stripped the Windows line endings! - Gordon

21 years, 7 months

Re: [Distutils] autoconf in distutils?

by lannert＠lannert.rz.uni-duesseldorf.de

[I'm only subscribed to the digest; please bear with me if I'm out of sync with the current status of the discussion] Greg Ward wrote: > Sooner or > later (probably sooner), somebody is going to want to say: > > if compiler == 'gcc' and platform == 'linux-x86': > compiler_options.append ('-funroll-loops') > > ...and that's probably only the beginning. ... and, later on, suggested variables like OS class, OS name, OS version, OS vendor, architecture, etc. etc. IIRC, Marc-Andre's suggestion pointed into a somewhat different direction: that the properties of a certain [instance of an] OS would be described by a set of variables indicating system "behaviour", rather than the formal vendor/version/architecture description. I would second this idea (and, IIDNRC, first it <wink>); maybe there are some situations where I'd like to know if a certain system is a SuSE or RedHat Linux, but usually I'd rather be interested in whether it has SysV or Simple init (and where the init.d directory is), if ps and friends want BSD or SysV style operands, if we have cc or gcc or egcs, and the like. The "intelligence" about the peculiarities of a vendor-x/version-y installation should IMO not be built into a tool like distutils; it would fail anyway for mangled or personalized systems (e.g. a SuSE Linux updated with a few RH and some home-grown rpm's). All the vendor-, installation- and configuration-specific data should be specified as individual variables assignments, probably in several files which are processed in a hierarchical order (like /etc/system.inst: /var/lib/inst/*.inst:~/.inst/*.inst:.instrc, just to give a silly example). They could be provided by (contributors to) distutils as a first start, but ideally by the vendors or distribution builders themselves <0.99 dream>. I'd even go so far and suggest shell variable assignment format for these files (name=value or name="value with blanks" etc.); it's far from ideal, but since it could be used from non-pythonic tools there would be an incentive for others to support and provide these files. Python programs can, of course, parse these files, although it's not trivial. I've written a ShellAssignment parser which understands "", '', $var, ${var}, \ escapes, at least in the most common cases; I wished there was a string.parsequotedstring() function ... Detlef

21 years, 7 months

Distributing 'external packages'

by Martijn Faassen

Hi there, Recently in the discussion on autoconf I've seen it mentioned that lots of Python extensions build on external non-python packages (that may be configured by autoconf or in many other ways). A developer using these external packages will obviously have them installed, and it's not too strange to expect co-developers to download these packages and install them as well. However, now we get to the two other audiences of distutils: the distributor/packager, who prepackages everything for one or more particular platforms, and most importantly the user who doesn't want to know about anything, just wants to run the program. If I'm a plain user and want to try out Python-Powered XML-database Warpdrive Enhancer version 4.2, I don't want to be required to download and install the Warpdrive package, and the XML package, etc, if I can avoid it. I just want to install and go. For many systems of course the separate install is unavoidable; we can't package Oracle for download, for instance. But for many smaller packages (such as libraries) that may or may not be on the user's system it becomes more important. So, as a reminder, how are we going to handle this? Some random requirements and ramblings: * The packager/distributor would like a standard way to somehow include these packages (or at least check for them). This doesn't mean *building* these packages, but it does mean a standard way to pack them up and install them. * We don't have any control over these external packages. Still, we don't want everybody to grow their own way to deal with particular packages. * If two Python packages are installed that both use FooLib, we don't want the disutils to install FooLib twice. Same if FooLib is already there. That's also why we need a standard way to handle these. * An idea I aired previously was to provide somekind of disutils wrapper for common packages. For instance, if FooLib is often used by software written in Python, we make a wrapper for it. The wrapper calls Foolib's installation/configuration methods (which may for instance be rpm commands, or a windows installer program) where necessary: * Install FooLib * If this is not possible without manual intervention, some way to tell the user what to do in easy steps. * Check if FooLib is already there (and what properties it has) * Also check if is it already registered with a Disutils Wrapper; if not, some way to try to add the wrapper so that the next pyapps that get installed won't need to go through this process again. * Possibly also uninstall FooLib, though this may be far too tricky. Eventually if this gets popular a Disutils Wrapper can even be distributed with FooLib itself, but the Disutils Archive should also offer some consistent way to get at known external package wrappers. All of this may be simply too hard, as packages can vary in many ways, but I think there are least *some* common things one can standardize. The idea is to hide all the nonstandard ways behind some standard interface, as far as possible, to help packagers and developers. It occurs to me that all of this is somewhat analogous to autoconf again; autoconf checks out capabilities of the system, in particular of the C compiler, and libraries. But as far as I know autoconf isn't modular as this package approach would be. Now you all start shooting this down. :) Regards, Martijn

21 years, 7 months

Re: Distutils-SIG digest, Vol 1 #79 - 1 msg

by Guido van Rossum

Greg, Only a few comments: 1) There's no command line interface on the Mac, but the compiler they use (Metroworks) can be controlled through an external "event" interface (Apple's response to COM, except Apple had it long before COM even existed :-). So there's no reason why it couldn't be done on the Mac. 2) I'm not sure if you're trying to add a method for each of the typical cc options (I recognized -I, -D, -U, etc.). Looks like you're missing -R, which is like -L but works at runtime, and is needed if you're using shared libraries that live in non-standard places. 3) I stringly prefer shared lib over dynamic lib (on Windows everyone calls them "DLL" anyway). 4) Do you really not want to support cross-compiling? The Windows/ce folks will regret that ;-) But I don't think you need to support -I (without a directory) for that; typically the compiler has a different name and all is well. I bet -I without args is intended for people who want to experiment with a hacked set of system includes; they can add it to the compiler command name if they want to. 5) When creating a shared lib from libraries only, you need to specify an entry point. Is this useful to support? 6) I haven't been following distutils very closely, so forgive me if the above makes no sense :-) --Guido van Rossum (home page: http://www.python.org/~guido/)

21 years, 7 months

autoconf in distutils?

by gward＠cnri.reston.va.us

Hmmm, just as well I was away last week: I *think* I agree with almost all the points-of-view expressed concerning accessing autoconf-like features from distutils. I would have had a hard time participating in that discussion, agreeing with *everyone*. For my next trick, I'll try to figure out a position that's not inherently self-contradictory. ;-) But seriously; I think Marc-Andre's initial post had two points: * testing compiler/header/library features, existence, etc. (the bread and butter of autoconf, but only the beginning of what it is used for) * getting a more detailed platform description (compiler/linker name and version, and an OS/hardware description more detailed than os.name) The autoconf-ish stuff would be fun to do in Python, and a good test of how general my compiler framework is. It wouldn't be as hairy as autoconf itself, because all the horrible things about shell programming magically disappear. As Fred pointed out, it would also be more portable than the M4-generated shell script approach. But it would probably be trickier than I like to admit, and not necessary in most cases. Consider: many (most?) Python extensions are probably thin glue layers to larger C libraries. It is the large C libraries that have configure scripts to adapt themselves to a particular platform; given the existence of Python, a C compiler and library, and the big C library being wrapped, the Python glue extension should build trouble-free. I suspect a similar situation in Perl-land; I've just posted a question to perl-xs(a)perl.org to see if the experts over there agree with my assessment that autoconf-like features are occasionally handy, but not necessary for most extensions. (I further suspect that most Perl extension developers who need something autoconf-ish have rolled their own in Makefile.PL; if we do nothing autoconf-ish in the distutils, then I suspect that Python extension developers will do similarly in setup.py. It would be nice to have some basic "Does this header file exist?" "Does this library define that symbol?" type functionality, though.) Bottom line is, I suspect it's not essential for building basic Python extensions, and would be a layer on top of the platform-specific CCompiler subclasses... so it's not an immediate concern. Regarding Marc-Andre's other wish: yes, this is important! Sooner or later (probably sooner), somebody is going to want to say: if compiler == 'gcc' and platform == 'linux-x86': compier_options.append ('-funroll-loops') ...and that's probably only the beginning. The CCompiler framework only has a barebones notion of what's needed to compile and link Python extensions (and will know about binary executables, in order to build a new static Python). I'm not even sure of how to handle optimization/ debugging options in a portable way, much less something like -Wall or -funroll-loops (considering only one popular compiler). Ultimately some sort of "Oh, here, just add your own damn compiler options" cop-out will be needed, and the only way to make that workable is to allow the extension developer to make decisions based on the platform and compiler. Perhaps a distutils.platform module is called for, which could expose all these things in a standard way. How's this sound for a start: * OS class ('posix', 'windows') (aka os.name) * OS ('linux', 'solaris', 'irix', 'winnt') * OS version ('2.2.10', '2.7', '6.1', '4.0') * OS vendor ('redhat', 'suse', 'sun', 'sgi', 'microsoft') * architecture ('x86', 'sparc', 'mips', 'alpha', 'ppc') * processor ('586', '686', ... ???) * compiler ('gcc', 'mipspro', 'sunpro', 'msvc') * compiler version ('2.8.1', '7.5', '??', '5.0') (does Sun even have version numbers on their compiler?) OK, flame away. I'm sure someone will tell me that this is a good start, but not nearly detailed enough, and Greg Stein will tell me to stop being so obsessive about finicky little details. I'm inclined to the latter... anyone want to try their hand at hacking up such a module? Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

21 years, 7 months

Re: Add multiple frozen modules

by Gordon McMillan

Arcege wrote: > Gordon, > I haven't looked at your own installer (I've been meaning to). But > the one issue I have with it is about the compression. When > compiling the Python distribution, zlib is not standard. This > means that pyz files aren't altogether portable. > BTW, what again is the URL of your installer? I have my own that I > had written (spamcan) and wanted to compare it. zlib is standard on Windows and comes in the RH RPMs. That's close enough for me <wink>. It's on the contrib site under System, but that's in the Windows-only form (an exe). It will soon have a homepage on starship. I'll again offer the inner pieces as tar.gz, but when I did that before (on my now defunct corporate site) I didn't get any takers. [BTW, I'm not on this SIG, I just check it periodically]. - Gordon

21 years, 7 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG July 1999