Mailman 3 February 1999 - Distutils-SIG

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Installer
by Gordon McMillan 23 Feb '99

23 Feb '99

Hi all, I've taken Greg Stein's "small" distribution and (with his advice(*) and help) created a mechanism for building compressed single-file importable python libraries. This is cross-platform, depending only on zlib (and Greg's imputil.py). There are a number of ways you can create these archives - grabbing directories, packages or computing the dependencies of a particular script. Then, for Win32 users, I've taken it a few steps further. I've mixed this with Christian Tismer's sqfreeze (based on /F's squeeze) and freeze and created a compiler-less Python installer. You can also use it much like freeze, except you don't need a compiler. The major difference is that, once installed, it won't be a completely standalone executable. The installer will unpack the dependencies into the exe's directory. There will be no dependencies outside that directory(**). You can pack in extension modules, dlls and anything else you like. The python lib support will be contained in a single archive file. So the equivalent of a frozen pure-python script would be one self-installing exe that expands to 5 files - the "squeezed" main script, python15.dll, zlib.dll, zlib.pyd and your private support lib (something.pyz). Check it out - http://www.mcmillan-inc.com/install.html Oh yes, while the code has my copyright, it's released under a do-what-you-want license. Mostly, I glued together what others had done (freeze, sqfreeze, Greg's stuff...). - Gordon (*) Not always followed. Don't blame Greg. (**) Note that if you want to install something that makes use of Mark's COM extensions (particularly, Python COM servers), you can't get away with this.

3 3

The $0.02 of a packager ;-)
by Oliver Andrich 10 Feb '99

10 Feb '99

Hi, after being invited by Greg Ward to this list, I tried to read all documents available concerning the topic of the list and I also tried to get uptodate with the mailing list by reading the archives. Now I like to give my $0.02 as someone who tries to keep a quite big distribution uptodate which also includes a lot of third party modules. This mail is rather long, cause I like to comment all all things in one mail, cause I think that quite a lot relates to each other. And please keep in mind that I am speaking as an enduser in one or another way. That means I will have to use this as a developer and will have to use it as a packager of third party developer. So I won't comment on any implementation issue where it is not absolutely critical. And please keep also in mind that may be I am talking about things you have already talked about, but I am quite new to this list and I only read the archives of this and the last month. ;-) - Tasks and division of labour I think that the packager and installer role are a little bit mixed up. In my eyes the packagers job is to build and test the package on a specific platform and also build the binary distribution. This is also what you wrote on the webpage. But the installers role should only be to install the prebuilt package, cause his normal job is to provide uptodate software that the users of the system he manages use. And he has enough to do with that. - The proposed Interface Basically the that I can say that I like the idea that I can write in my RPM spec file setup.py build setup.py test setup.py install and afterwards I have installed the package somewhere on my machine. And I am absolutely sure that it works as indented. I think that this is the way it works for most Perl modules. But I have problems with bdist option of setup.py cause I think that this is hard to implement. If I got this right I as a RPM and debian package maintainer should be able to say setup.py bdist rpm setup.py bdist debian And afterwards I have a debian linux and rpm package of the Python package. Nice in theory but this would require that setup.py or the distutils packages how to create these packages, that means we have to implement a meta packaging system on top of existing packaging systems which are powerful themselves. So what would it look like when I call these commands above? Would the distutils stuff create a spec file (input file to create a rpm) and then call rpm -ba <specfile>? And inside the rpm build process setup.py is called again to compile and install the packages content? Finally rpm creates the two normal output files, which are the actual binary package and the other is the source rpm from which you can recompile the binary package on your machine. This is the same for debian linux, slackaware linux, rpm based linux versions, Solaris packages and BeOS software packages. The last is only a vague guess cause I only looked into the Be system very shortly. - What I would suggest what setup.py should do The options that I have no problem with are build_py - copy/compile .py files (pure Python modules) build_ext - compile .c files, link to .so in blib build_doc - process documentation (targets: html, info, man, ...?) build - build_py, build_ext, build_doc dist - create source distribution test - run test suite install - install on local machine What should make_blib do ? But I require is that I can tell the build_ext which compiler switches to use, cause may be I need on my system different switches then the original developer can use. I also like to provide the install option with an argument to tell where the files should be installed, cause I can tell rpm for example that it should compile the extension package as if it would be installed in /usr/lib/python1.5 but could it in the install stage to install it in /tmp/py-root/usr/lib/python1.5. So I can build and install the package without overwriting an existing installation of a older version and I also have a clean way to determine what files actually got installed. install should also be split up into install and install_doc and installdoc should also be able to take an argument where I tell it where to install the files to. I would remove the bdist option cause it would introduce a lot of work, cause you not only have to tackle various systems but also various packaging systems. I would add an option files instead which returns a list of files this packages consists of. And consequently an option doc_files is also required cause I like to stick to the way rpm manages doc files, I simply tell it what files are doc files and it installs them the right way. Another that would be fine if I could extract the package information with setup.py. Something like setup description returns the full description and so on. And I would also add an option system to the command line options, cause I like to tell the setup.py script an option from which it can determine on which system it is running. Why this is required will follow. - ARCH dependent sections should be added What is not clear in my eyes, may be I have missed something, but how do you deal with different architectures? What I would suggest here is we should use a dictionary instead of plain definitions of cc, ccshared, cflags and ldflags. Such a dictionary may look like that compilation_flags = { "Linux" : { "cc": "gcc", "cflags": "-O3", ...}, "Linux2.2": { "cc": "egcs", ....}, "Solaris": { "cc": "cc", ....} } And now I would call setup.py like that setup.py -system Linux build or whatever convention you want to use for command line arguments. - Subpackages are also required Well, this is something that I like very much and what I really got accustomed to. They you build PIL and also a Tkinter version that supports PIL, then like to create both packages and also state that PIL-Tkinter requires PIL. Conclusion (or whatever you want to call it) I as a packager don't require the distutils stuff to be some kind of meta packaging system that generates from some kind of meta information the actual package creation file from which it is called again. And I don't believe that have to develop a complete new packaging system, cause for quite a lot systems such systems exist. And I also think that we introduce such a system the acceptance wouldn't be very high. The people want to maintain the software basis with their natural tools. A RedHat Linux user would like to use rpm, a Solaris user would like to use pkg and a WindowsUser would like to use INstallShield (or whatever is the standard). The target of distutils should be to develop a package which can be configured to compile and install the extension package. The developed software should be usable by the packager to extract all required information to create his native package and the installer, should use the prebuilt packages at best or should be able to install the package by calling setup install. I hope that I described as good as possible what I require as packager and I think that is not a business of distutils but of the native packaging system. Any comments are welcome and I am willing to discuss this, as I am absolutely aware that we need a standard way of installing python extensions. Best regards, Oliver -- Oliver Andrich, RZ-Online, Schlossstrasse Str. 42, D-56068 Koblenz Telefon: 0261-3921027 / Fax: 0261-3921033 / Web: http://rhein-zeitung.de Private Homepage: http://andrich.net/

4 8

Re: new import mechanism and a "small" distribution
by Greg Stein 01 Feb '99

01 Feb '99

Just van Rossum wrote: > > At 8:00 AM -0800 1/31/99, Greg Stein wrote: > >... > >Okay... enough background and rambling. If you're interested, go look at the > >four messages in the thread titled "Freeze and new import architecture" in > >the distutils-sig archive at: > >http://www.python.org/pipermail/distutils-sig/1998-December/thread.html > > I'm not on that sig so I missed your post originally. I agree with most you > say here: > http://www.python.org/pipermail/distutils-sig/1998-December/000077.html > Especially that entries in sys.path should be loader instances (or > directory paths). > > Some questions: > - what is the interface of a loader By "loader", I will presume that you mean an instance of an Importer subclass that is defining get_code(). Here is the method as defined by imputil.Importer: def get_code(self, parent, modname, fqname): """Find and retrieve the code for the given module. parent specifies a parent module to define a context for importing. It may be None, indicating no particular context for the search. modname specifies a single module (not dotted) within the parent. fqname specifies the fully-qualified module name. This is a (potentially) dotted name from the "root" of the module namespace down to the modname. If there is no parent, then modname==fqname. This method should return None, a 2-tuple, or a 3-tuple. * If the module was not found, then None should be returned. * The first item of the 2- or 3-tuple should be the integer 0 or 1, specifying whether the module that was found is a package or not. * The second item is the code object for the module (it will be executed within the new module's namespace). * If present, the third item is a dictionary of name/value pairs that will be inserted into new module before the code object is executed. This provided in case the module's code expects certain values (such as where the module was found). """ raise RuntimeError, "get_code not implemented" That method is the sole interface used by Importer subclasses. To define a custom import mechanism, you would just derive from imputil.Importer and override that one method. I'm not sure if that answers your question, however. Please let me know if something is unclear so that I can correct the docstring. Oh, geez. And I totally spaced on one feature of imputil.py. There is a way to define an import mechanism for very simple uses. I created a subclass named FuncImporter that delegates the get_code() method to a user-supplied function. This allows a user to do something like this: import imputil def get_code(parent, modname, fqname): ...do something... imputil.install_with(get_code) The install_with() utility simply creates a FuncImporter with the specified function and then installs the importer. No need to mess with subclasses. > - how are packages identified If get_code() determines that the requested module is a package, then it should return the integer 1 along with the code object for that package's module. In the standard package system, the code object is loaded from __init__.py. An example: let's say that get_code() is called with (None, "mypkg", "mypkg") for its arguments. get_code() finds "mypkg" in whatever path it is configured for, and then determines that it represents a directory. It looks in the directory for __init__.py or .pyc. If it finds it, then mypkg is a real package. It loads code from __init__.py and returns (1, code). The Importer superclass will create a new module for "mypkg" and execute the code within it, and then label it as a package (for future reference). Internally, packages are labelled with a module-level name: __ispkg__. That is set to 0 or 1 accordingly. The Importer that actually imports a module places itself into the module with "__importer__ = self". This latter variable is to allow Importers to only work on modules that they imported themselves, and helps with identifying the context of an importer (whether an import is being performed by a module within a package, or not). > - can it be in Python 1.6 or sooner? I imagine that it could be in Python 1.6, but I doubt that it would go into Python 1.5.2, as it has not had enough exposure yet. Guido's call on all counts, though :-) > PS: I was just wondering, why doesn't reload() use the standard import hook? The standard import hook returns a *new* module object. reload() must repopulate the module object. I'm not sure that I answered your questions precisely, as they were rather non-specific. If I haven't answered well enough, then please ask again and I'll try again :-). Cheers, -g -- Greg Stein, http://www.lyra.org/

2 5

Re: new import mechanism and a "small" distribution
by Greg Stein 01 Feb '99

01 Feb '99

Gordon McMillan wrote: > > [Greg puts code where his mouth is] > > > A while back, Mark Hammond brought up a "new import architecture" > > proposal after some discussion with Jack Jansen and Guido. I > > responded in my usual diplomatic style and said "feh. the two-step > > import style is bunk." > > [snip...] > > This is outrageously cool! > > It appears that you're aiming for a rewrite of import.c (and > friends) and giving new meaing to "sys.path". (I love it - Java says > "it can be a directory or a zip file" and Greg says "why stop there? > It can be anything at all" - hee hee). Yup :-) Last month, I even argued that the import mechanism could simply be shifted entirely to Python, too, since the overhead of interpreted Python code is minimal next to the parsing, execution, and/or I/O involved with importing. hehe... > What is enabling this magic in this prototype? We pick up your > site.py automatically, but is your python.exe doing something in > between Py_Initialize and Py_Main? Or is this all based on making a > chain out of __builtin__.__import__? If python.exe doesn't find its registry settings, then it assumes a default sys.path (which includes the current directory). Using that, it loads exceptions.py and then site.py. Once site.py loads, then I install the custom import hook so that all future imports will be yanked from py15.pyl. The builtin modules aren't in there, of course, so those fall down the chain to the builtin importer. Cheers, -g -- Greg Stein, http://www.lyra.org/

1 0