(Doing my bi-monthly perusal of Distutils activity, I find...)
[Mark]
.... Specifically, Greg Stien and Gordon McMillan (and plenty of people before them :-) seem to have what is considered "state of the art" in where this is heading.
[Jim Ahlstrom] I don't know of techniques which aren't Windows specific. Greg??Gordon??
Mark is referring to Greg's imputil.py (http://www.lyra.org/greg/small/) and some of the places I've taken it. My Win32-specific installer (ftp://ftp.python.org/pub/python/contrib/System/Installer_r_01.exe) makes use of this, but not all of it is Windows specific. (I originally packaged it so the cross-platform stuff was available separately, but there was no apparent interest from non-Windows users.)
Greg's imputil.py basically makes it possible to create a chain of importers, with the standard mechanism pushed to the end. Writing an importer is easy. You set up the chain is site.py.
I created a way of building archives of .pyc's (or .pyo's, though I've never worked with them). These are compressed with zlib. The standard library fits in about 500K and (subjectively) it is no slower and perhaps faster than the regular method. The mechanism handles modules and packages, and the building of an archive can be done in all kinds of ways (including using modulefinder from freeze).
I also created another kind of archive that can package up arbitrary stuff, and is fairly easily unpacked from C code.
All of the above is cross-platform.
On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:
myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]
(The Window's installation / standalone stuff goes further than this).
These .pyz's do work on Linux, but the steps Python goes through to determine where it lives is simpler on Windows. I'm not sure what it would take to get a single directory Python installation (independent of any other installed Pythons) working on *nix.
- Gordon
Gordon McMillan wrote:
On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:
myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]
But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path.
Jim Ahlstrom
James C. Ahlstrom wrote:
Gordon McMillan wrote:
On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:
myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]
But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path.
In my "small" distribution, I was able to take advantage of the fact that python.exe looks in the current directory for site.py if there is nothing in the registry. The site.py then proceeds to bootstrap the import system.
Gordon has an excellent writeup of the process on his website and/or in the README in his distribution (forget which). My small distribution, links to some discussion about it, and a reference to Gordon's distro is available at: http://www.lyra.org/greg/small/
Cheers, -g
-- Greg Stein, http://www.lyra.org/
Greg Stein wrote:
In my "small" distribution, I was able to take advantage of the fact that python.exe looks in the current directory for site.py if there is nothing in the registry. The site.py then proceeds to bootstrap the import system.
Gordon:
Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely. For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.
Christian:
This is what I use for standalone apps. We just drop the whole tree into the application. site.py lives in the Python directory, as a special version. This suffices, since then no other path than "." is needed to boot.
And that's the only annoyance: You need to take care of environment space, which is unfortunately not big enough very often. I think a special python.exe for this purpose would be handy, which just ignores all registry, sets the path to the executable's directory, and done.
A Windows user normally sets the "Start in" directory in the shortcut to something other than the directory of the binary. In other words, the current directory "." is unreliable. Users can right-click their icon and change it.
I don't like changing other people's registry either. In fact our bank customers hate it when we do. I don't trust the environment PYTHONPATH either. So we can't be sure the site.py bootstrap is bullet-proof. Please stand by while I try to clarify my thoughts...
Jim Ahlstrom
"James C. Ahlstrom" wrote: ...
A Windows user normally sets the "Start in" directory in the shortcut to something other than the directory of the binary. In other words, the current directory "." is unreliable. Users can right-click their icon and change it.
I don't like changing other people's registry either. In fact our bank customers hate it when we do. I don't trust the environment PYTHONPATH either. So we can't be sure the site.py bootstrap is bullet-proof. Please stand by while I try to clarify my thoughts...
I'd say we write a startup program which sits in the same directory as the python15.dll (which is the Python directory for my customers! I don't want to interfere with anything.) This extracts it's executable's program path, chdirs to there, sets the environment internally to "." (which holds now), and the site.py trick always should work.
We (PNS) go even further (but at the moment without the extra .exe which we will do soon). Site.py does some fiddling with the Python path, tries to find ./lib and so on, and then looks for ../app.py which has to be there. From there, the bootstrap loads the application. By this, we can replace the whole Python installation by moving a single directory tree. It does not contain any application dependant stuff and just knows that the app.py sits on top.
-chris
Christian Tismer wrote:
I'd say we write a startup program which sits in the same directory as the python15.dll (which is the Python directory for my customers! I don't want to interfere with anything.) This extracts it's executable's program path, chdirs to there, sets the environment internally to "." (which holds now), and the site.py trick always should work.
We would need to remember the original getcwd() and chdir back so the user's "Start in" setting still works.
Also, I think it would be better to use the directory of python15.dll, since all of Python is in it, and that is where imports are done. It is not really a feature of the app.
Jim Ahlstrom
"James C. Ahlstrom" wrote:
Christian Tismer wrote:
I'd say we write a startup program which sits in the same directory as the python15.dll (which is the Python directory for my customers! I don't want to interfere with anything.) This extracts it's executable's program path, chdirs to there, sets the environment internally to "." (which holds now), and the site.py trick always should work.
We would need to remember the original getcwd() and chdir back so the user's "Start in" setting still works.
Well, that's easy. Whatever directory the user "was in", the path of the executable (i.e. what's in sys.executable) will be allways enough to figure out where you are now. This might not be a full path but a relative one, and it might (will) be the "short" version under Win9X, but anyway it was enough for the exec launcher to launch the .exe.
Also, I think it would be better to use the directory of python15.dll, since all of Python is in it, and that is where imports are done. It is not really a feature of the app.
In my case, python15.dll and python.exe *are* in the same directory. This is not the app, app.py is a file on top of this python directory, which is like plug-in (drop-in) and always the same. *int* the same python drop-in-dir I also have the simplified site.py which just takes care that the path settings are completed, that application scripts are found (by reaching out, off the directory, into app.py) and imports are prepended to ./lib stuff.
Works fine, regardless where ther is another Python installation, even if another one is running.
ciao - chris
Jim Ahlstrom wrote:
Gordon:
Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely. For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.
A Windows user normally sets the "Start in" directory in the shortcut to something other than the directory of the binary. In other words, the current directory "." is unreliable. Users can right-click their icon and change it.
OK - os.path.dirname(sys.argv[0]).
In my case it doesn't matter. I have just concatenated my archive (of python15.dll, any .pyzs, exceptions.py, site.py etc.) onto the end of the executable. When run, it opens itself as a file and unpacks the archive into the current directory. Then it does a _putenv("PYTHONPATH=.") and dynamically loads python15.dll. If you're using Run.exe, it even cleans up whatever it unpacked at end of run.
More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...)
- Gordon
Gordon McMillan wrote:
OK - os.path.dirname(sys.argv[0]).
Oh, I see. Your main program is given as a complete path in the icon target. So "python D:/my/path/myprog.py". I start my main as the frozen module named "__main__", so sys.argv[0] is useless.
In my case it doesn't matter. I have just concatenated my archive (of python15.dll, any .pyzs, exceptions.py, site.py etc.) onto the end of the executable. When run, it opens itself as a file and unpacks the archive into the current directory. Then it does a _putenv("PYTHONPATH=.") and dynamically loads python15.dll. If you're using Run.exe, it even cleans up whatever it unpacked at end of run.
Cool. I didn't realize you could append to an .exe and still have it run. How do you know where your appended data starts?
More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...)
I think this may be a good idea. We could use a Windows user-defined resource named with a magic name such as "PythonPyc", and all resource modules could be loaded in Py_Initialize(). The .rc resource file names the .pyc file so there is no need to convert the bytes to C and compile. The .rc file syntax is the line: string PythonPyc C:/lib/string.pyc You use FindResource(), SizeofResource(), LoadResource() and LockResource() to access these resources.
It is possible to replace resources in an exe or dll using BeginUpdateResource(), UpdateResource() and EndUpdateResource(). If we wrote a Python interface into a .pyd, then users don't even need a compiler nor resource compiler. Note that there is Mac code in the distribution which accesses Mac resources. The only reason I have not written all this is it is not usable on Unix.
Jim Ahlstrom
"James C. Ahlstrom" wrote:
Gordon McMillan wrote:
OK - os.path.dirname(sys.argv[0]).
Oh, I see. Your main program is given as a complete path in the icon target. So "python D:/my/path/myprog.py". I start my main as the frozen module named "__main__", so sys.argv[0] is useless.
Freezing is not so handy as Gordon's trick. Which came partially out of my trick, which came from Fredrik's squeeze trick, or so.
[the trick]
Cool. I didn't realize you could append to an .exe and still have it run. How do you know where your appended data starts?
Is it still so? look at the end for the magic cookie, use its info to find the start of a cookie, and there's the data?
More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...)
I think this may be a good idea. We could use a Windows user-defined resource named with a magic name such as "PythonPyc", and all resource modules could be loaded in Py_Initialize(). The .rc resource file names the .pyc file so there is no need to convert the bytes to C and compile. The .rc file syntax is the line: string PythonPyc C:/lib/string.pyc You use FindResource(), SizeofResource(), LoadResource() and LockResource() to access these resources.
Yes, this is still an open issue. Greg proposed a similar way. I never came to it, since the drawback of the simple approach is that it works, and it isn't so obvious how. By a resource, I open the app to Joe Hacker and invite him to play with resources. So why should I, if everything can be done in a copy /b style? :-)
It is possible to replace resources in an exe or dll using BeginUpdateResource(), UpdateResource() and EndUpdateResource(). If we wrote a Python interface into a .pyd, then users don't even need a compiler nor resource compiler. Note that there is Mac code in the distribution which accesses Mac resources. The only reason I have not written all this is it is not usable on Unix.
Yes, and to complete it: The append trick *does* work on Unix. And a resource like management of things could be written with Python, defining our own resources. Why care about an OS?
Ahem :-) ciao - chris
Christian Tismer wrote:
Yes, and to complete it: The append trick *does* work on Unix. And a resource like management of things could be written with Python, defining our own resources. Why care about an OS?
[Jim ponders the append trick...]
OK, a really simple idea is to do this: cat python.exe string.pyc site.pyc > myprog.exe That is, just append a bunch of *.pyc to the executable. Anybody can do that. Then make the import mechanism be able to find them. So, look for the *.pyc magic numbers. To recover the module names, use the base file name of the code object attribute co.co_filename.
Same with Python15.dll, append *.pyc to it too.
Jim Ahlstrom
James C. Ahlstrom wrote:
Christian Tismer wrote:
Yes, and to complete it: The append trick *does* work on Unix. And a resource like management of things could be written with Python, defining our own resources. Why care about an OS?
[Jim ponders the append trick...]
OK, a really simple idea is to do this: cat python.exe string.pyc site.pyc > myprog.exe That is, just append a bunch of *.pyc to the executable. Anybody can do that. Then make the import mechanism be able to find them. So, look for the *.pyc magic numbers. To recover the module names, use the base file name of the code object attribute co.co_filename.
Same with Python15.dll, append *.pyc to it too.
You can make it a lot easier on yourself if you remember the byte offset of each of those .pyc files. Then, you append a marshalled dictionary mapping the names to the offsets (and remember the byte offset of this marshalled dict). Finally, you append the byte offset of the dict.
Recovery is then very easy: seek to (-4) from the end and read the offset. Seek there and unmarshall the dict. Seek to whichever module you want and unmarshal that.
I used this technique in constructing my Python library of modules in the small distribution. The offset was at the beginning cuz it was a self-defined format so I just seeked back to the start to write out the file's magic value + offset. But the technique works for appending, too.
Personally, I think it is a hack to simply append stuff to an executable (Unix or Windows). I would much rather see a proper COFF or ELF section inserted in there which contains the modules.
Cheers, -g
-- Greg Stein, http://www.lyra.org/
Greg Stein wrote:
You can make it a lot easier on yourself if you remember the byte offset of each of those .pyc files. Then, you append a marshalled dictionary
Yes, I see your point. I am looking into this now. But it does require a special Python set up program. I was trying to make it easier on the user. If a Lib module turns up missing you just cat it onto python15.dll.
Personally, I think it is a hack to simply append stuff to an executable (Unix or Windows). I would much rather see a proper COFF or ELF section inserted in there which contains the modules.
Yes, I agree. But presumably every different system has its own executable format. I do not know how standard they are even just on Unix.
I am currently writing some experimental code and will report back.
Jim Ahlstrom
I got a little time to think about this over the weekend and propose this design. It is a way to package *.pyc files into a single-file portable archive for distribution with commercial, CGI (like Marc is doing) or otherwise simplified distributions. It is dumb-stupid-simple, my personal favorite and a requirement for commercial software. This is not concerned with an "installer", which is a separate problem. Few of these ideas are mine.
1. There is a Python library file format (PYL) which can hold *.pyc files. The format is: a.pyc, b.pyc, marshalled dict, offset of dict, magic number.
The a.pyc, b.pyc,... are the bytes of the *.pyc files including the eight byte header. The dict has keys "name", values are the seek offset of the start of the *.pyc files. The "offset of dict" is the seek offset for finding the dict. The magic number identifies the file as a PYL file.
The "names" can be normal names such as "os", or dotted names such as "foo.bar". Although initially devoted to *.pyc files, we note that it is possible to include other resources with non-module names such as "more*data".
A PYL file is portable to any computer, Unix, Windows, etc. Compression is not included, and should be done at the "installer" level if desired.
2. The PYL file always has the same directory and file name as the binary module containing import.c. but with a .pyl ending:
Python Binary PYL File Name /usr/local/bin/python /usr/local/bin/python.pyl /my/so/dir/python.so /my/so/dir/python.pyl C:/python/python15.dll C:/python/python15.pyl
These Python binary names are available as sys.executable (the main) and sys.dllfullpath (the shared library or DLL).
3) Since the PYL file can be efficiently read backwards, it can, if desired, be appended the the Python binary itself: cat python15.pyl >> python15.dll
4) The PYL file is created with the Python program makepyl.py and no C compiler is necessary.
5) There is a new optional built-in module "importer" which may be included by editing Setup. It is imported in Py_Initialize() after "sys" is set up, but before any other imports such as "exceptions". It is not an error if it is absent. If present, it replaces __import__ just like imputils.py does. The replacement importer searches for PYL files as a .pyl file, in the current sys.executable, and in the current sys.dllfullpath (name of DLL). Note that importer can use multiple PYL files. Importer is able to import the modules exceptions, site and sitecustomize, and probably most other modules. Importer has methods to print out the names of PYL modules available. You could still override importer using sitecustomize and imputils if desired, in which case it may be convenient to use importer's methods.
6) Alternative to (5): Modules exceptions, site, sitecustomize and imputils are frozen (using frozen modules) into the interpreter binary, and sitecustomize boots imputils. Thereafter the Python logic in sitecustomize and imputils implements the logic of (5). Sitecustomize has methods to print out the names of PYL modules available.
7) The Python main program is enhanced to start "__main__" as the main program if it can be imported, in which case all command line arguments are sent to __main__. This enables you to create a Python program "myprog", and start it with the command "myprog -a arg1 ..." just like a normal program.
8) The Python main can start any module "foo" (which may be in a PYL file) as the main program in response to a new command line argument. This enables you to ship multiple main programs in the PYL file.
9) The current frozenmain.c is eliminated and the enhanced main is used instead. This (I hope) results in a net decrease in code.
Where this is going:
It enables simplified distribution of Python programs.
Greg has demonstrated that import.c can be replaced with a Python module, and that in general at least some of Python should be written in itself. This heads that way.
Please send in your "hell no" comments now.
Jim Ahlstrom
[Jim talking about a Python Lib format]
You may find the attached DictFile.py useful. It does pretty much what you (and even includes a PYCFile subclass). Be warned that it will not work out of the box though, since it references some other modules I normally use. You can safely comment them out though.
The "Python Lib format" that Jim discusses is exactly what is used in my "small" distribution. Gordon used a slight variant and even created a nice base class to create similar files.
In other words, the code already exists in to match the exact requirements.
I strongly agree with Jim's thoughts on moving all Python importing out to the Python level (minus a minimalist set of C functions to dynamically load module M from directory D). Freezing that into the interpreter is a great Step 2.
IMO, I would also like to see a frozen Python parser, compiler, and interactive loop, but that is a separate discussion :-)
Cheers, -g
-- Greg Stein, http://www.lyra.org/
On Mon, 19 Jul 1999, M.-A. Lemburg wrote:
[Jim talking about a Python Lib format]
You may find the attached DictFile.py useful. It does pretty much what you (and even includes a PYCFile subclass). Be warned that it will not work out of the box though, since it references some other modules I normally use. You can safely comment them out though.
-- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 165 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/
After writing some code, I see that the library file needs to be a little different. We need to add the file size so all fseek()'s can be done relative to the end. That enables the file to be appended to the executable. The offsets written to the file are from ftell() and are relative to the beginning. The length is subtracted on access.
And I introduced a main dictionary TOC and a different dictionary for the *.pyc files pyc_dict. Although that may seem a little fancy, it is actually easier. And it is much more extensible. Data may be freely written to the main dict TOC without damaging the code for reading *.pyc using pyc_dict. And another section (I'm thinking of Gordon's installer) can be added as yet another dict. In another variation, core Python files are in pyc_dict, but encoded *.pyc files are in a different dict, and imputils is used to import either.
The Python library file format now is:
Data... consisting of *.pyc files including their 8-byte header; Possibly other data... TOC, a marshal'd dictionary table of contents; a twelve byte ascii decimal seek offset to the start of TOC, a NULL byte; a twelve byte ascii decimal total file size, a NULL byte; a 16-byte magic number to identify the file as a valid Python library file.
The TOC (table of contents) is a dictionary containing the following keys: "TIME" The time the library was created, time.time(). "ASCTIME" The ascii time the library was created. "VERSION" The version of the library file format, "1". "PYC*VERSION" The ascii version number of the PYC section, "19990720". "PYC" A dictionary with keys "name", and values a three-tuple "tup". The "name" is the module name. This is a plain name such as "os" or a dotted name such as "foo.bar". The "tup[0]" is the seek offset to the data, which is the bytes of the module .pyc file including the 8-byte header. The "tup[1]" is the full path of the .pyc file. The "tup[2]" is an integer used as flag bits. Flag bit 0x01 is one if the module name refers to a package, in which case the file is package.__init__.pyc.
This format is designed to be extensible. Just add atoms of data to TOC. To add a another whole section, add another name like "PYC".
I have almost finished makepyl.py, a Python program which creates a PYL file or lists its contents. And C-code changes to import.c to be able to import from a PYL file. However, I am a little short of time because I am leaving for two weeks vacation on this Saturday.
Jim Ahlstrom
After writing some code, I see that the library file needs to be a little different. We need to add the file size so all fseek()'s can be done relative to the end. That enables the file to be appended to the executable. The offsets written to the file are from ftell() and are relative to the beginning. The length is subtracted on access.
I havent been following this thread as closely as I would like, but I am a little confused. Are we building yet another packager here, or is this work building on the work Gordon currently "owns"?
I have almost finished makepyl.py, a Python program which creates a PYL file or lists its contents. And C-code changes to import.c to be able to import from a PYL file. However, I am a little short of
If I recall correctly, when Greg posted his version of what grew into Gordon's, there was discussion over the direction import.c should take. Are your patches in line with that? Im just thinking that poor Guido has already discussed this once, so may be reluctant to accept new hacks when the correct direction has been mapped out. As far as I can see, Gordon's code works fine without C level changes, so Im a little unsure what direction this is going, and exactly what this is solving that Gordon's doesnt.
Mark.
Mark Hammond wrote:
I havent been following this thread as closely as I would like,
That's OK, I missed out on the prior distutils-sig discussions anyway.
but I am a little confused. Are we building yet another packager here, or is this work building on the work Gordon currently "owns"?
It is not another packager. It provides a way to bootstrap Gordon's installer without requiring exceptions.py etc. to be found with the usual sys.path mechanism. All Python files can be in the library files. This has to be done with C code.
If I recall correctly, when Greg posted his version of what grew into Gordon's, there was discussion over the direction import.c should take. Are your patches in line with that?
I hope so. The "makepyl.py" program for making library files is almost identical to Greg's easygen.py, and I wrote his name at the top. But I did alter the file format so that it could be appended to python15.dll, and so source names could be listed.
I am open to any file format that works. I was hoping that we could finalize the format in this sig. It doesn't have to be my format.
I read all prior distutil-sig archives, but in case I missed something, could you provide a reference to the discussion you are referring to?
Im just thinking that poor Guido has already discussed this once, so may be reluctant to accept new hacks when the correct direction has been mapped out. As far as I can see, Gordon's code works fine without C level changes, so Im a little unsure what direction this is going, and exactly what this is solving that Gordon's doesnt.
I solves the bootstrap problem. Any Python-only solution has to read several Python *.pyc files before it can get started. This requires using the regular sys.path mechanism which is what I am trying to replace. My C code can be used to boot up Gordon's installer.
Also note Marc's mxCGIPython project. He has valiantly collected dozens of binary frozen library versions just so someone can install Python on their server without a hassle. A portable Python library file is a good idea.
I got serious about this because of the recent thread started by Mikael Lyngvig in c.l.p. We really need a way to enable bullet-proof Python program distribution on machines where Python is not installed, on machines which have an invalid Python distribution, and on machines which have a valid Python but the wrong version. It is my perception that we are not there yet.
Jim Ahlstrom
OK, I put my import Python library code on ftp://ftp.interet.com/pub/pylib.zip and ftp://ftp.interet.com/pub/pylib.tar.gz. The contents are identical. Uncompress in a temp directory.
Try listing the contents of the library file makepyl.pyl using "python makepyl.py -t", or make a new one with "python makepyl.py" but be aware that this OVERWRITES makepyl.pyl. You will need to change the config file makepyl.cfg.
Copy makepyl.pyl to python15.pyl (in the same directory) to make it active. Then import string, and enter "print string" to see that the module comes from python15.pyl.
There are two issues, the library file format and the C-code changes to import.c.
I published a proposed library file format here on distutils and it is described in the doc strings. But I didn't describe why I didn't use Greg's nor Gordon's format. Both formats remove the 8-byte .pyc header. But import.c validates this header for .pyc files and it should be retained. There was no file source information, but a user will want to print out the file names in the library. Greg's format could not be appended to another file. Gordon can build the library onto another file with a program, but can not append the same file to different programs. The new format can append the same library file to any other executable or dll using "cat library.pyl >> AnyPythonBinary[|.exe|.so|.dll]. It can also support multiple appends if desired.
The changes to import.c add another PY_LIBRARY import method (like PY_SOURCE, PY_COMPILED, etc.) and is programmed in a parallel fashion. The code may be a bit rough but it is sufficient to get an idea.
I am going away on vacation tomorrow, so I really hope all this works OK. I will catch up on all your comments when I get back.
Jim Ahlstrom
James C. Ahlstrom wrote:
I published a proposed library file format here on distutils and it is described in the doc strings. But I didn't describe why I didn't use Greg's nor Gordon's format. Both formats remove the 8-byte .pyc header. But import.c validates this header for .pyc files and it should be retained.
Why? So import.c can try to recompile from non-existant source?
If you let Greg or me build the archive, you'll find that the compiles get done if they're needed. Then the package contains a python and .pyc's that match that version.
With or without a check, a mismatch will produce an ugly mess.
There was no file source information, but a user will want to print out the file names in the library.
If they want to, the can get the name from the module's __importer__ attribute. They have everything they need to call archive.contents(), too, (if you're letting them get to an interpreter prompt).
Greg's format could not be appended to another file. Gordon can build the library onto another file with a program, but can not append the same file to different programs.
Um, you mean because there's a "os.remove()" in Builder.py?
Builder.py is mostly concerned with parsing the config file and determining dependencies. The actual archive building is done by the archive class, with some utility functions in archivebuilder.py.
The changes to import.c add another PY_LIBRARY import method (like PY_SOURCE, PY_COMPILED, etc.) and is programmed in a parallel fashion. The code may be a bit rough but it is sufficient to get an idea.
I realize you don't like having to have four .py files in the current directory. I don't like messing with python internals, especially when (among other things) you can use this to distribute a stripped down but otherwise standard python.
In addition, both Greg & I are hoping (with some encouraging signs) that his imputil will attain sanctified status, which would probably mean that something like this could be done with just one 'extra' file.
But in the meantime, are you sure you can't get the same effect by using normal embedding techniques in the .exe (ie, instead of just calling Py_Main?). (Well, not quite normal techniques, since you have to use GetProcAddress).
I am going away on vacation tomorrow, so I really hope all this works OK. I will catch up on all your comments when I get back.
Have a good time.
- Gordon
Sigh..., I am back from vacation and as far as I can see, there are no comments except from Gordon. And it looks like all I have managed to do is get Gordon mad at me :-(
But anyway:
Gordon McMillan wrote:
James C. Ahlstrom wrote:
I published a proposed library file format here on distutils and it is described in the doc strings. But I didn't describe why I didn't use Greg's nor Gordon's format. Both formats remove the 8-byte .pyc header. But import.c validates this header for .pyc files and it should be retained.
Why? So import.c can try to recompile from non-existant source?
If you let Greg or me build the archive, you'll find that the compiles get done if they're needed. Then the package contains a python and .pyc's that match that version.
With or without a check, a mismatch will produce an ugly mess.
I am not saying the myprog.pyl file is created incorrectly nor that the version does not match python.exe.
The point is that mylib.pyl is a separate file, and so it may be out of phase with python.exe because the user upgraded python.exe but not mylib.pyl.
Currently Python will import a file.pyc even if there is no file.py present. If that file.pyc has a bad magic number then Python throws an ImportError. I think a myprog.pyl file is exactly analogous.
There was no file source information, but a user will want to print out the file names in the library.
If they want to, the can get the name from the module's __importer__ attribute. They have everything they need to call archive.contents(), too, (if you're letting them get to an interpreter prompt).
I don't understand. I want to check which files were used to create myprog.pyl, and this can only be recorded in myprog.pyl. I don't want the module names "sub1", I want "C:/path/to/sub1.py". This could be used as a check before a product is shipped. But I guess I can live without it.
Greg's format could not be appended to another file. Gordon can build the library onto another file with a program, but can not append the same file to different programs.
Um, you mean because there's a "os.remove()" in Builder.py?
Builder.py is mostly concerned with parsing the config file and determining dependencies. The actual archive building is done by the archive class, with some utility functions in archivebuilder.py.
My statement stands. I was hoping to ship myprog.pyl and allow the recipient to use "cat myprog.pyl >> python.exe" if they wanted. I know I can re-make a new python2.exe for each system.
I realize you don't like having to have four .py files in the current directory. I don't like messing with python internals, especially when (among other things) you can use this to distribute a stripped down but otherwise standard python.
I agree that changing Python internals is usually bad, and your design has the very great advantage that no changes are required.
I will throw out my design and re-start using your software. But since I have tried this before I know it won't work. But maybe I can come up with a more minimal set of changes.
In addition, both Greg & I are hoping (with some encouraging signs) that his imputil will attain sanctified status, which would probably mean that something like this could be done with just one 'extra' file.
But in the meantime, are you sure you can't get the same effect by using normal embedding techniques in the .exe (ie, instead of just calling Py_Main?). (Well, not quite normal techniques, since you have to use GetProcAddress).
The best I came up with so far is to use freeze to incorporate my Python files in a DLL, and then use a special main to link in the frozen modules. This currently works, but it is not ideal.
I am going away on vacation tomorrow, so I really hope all this works OK. I will catch up on all your comments when I get back.
Have a good time.
Thanks. I went scuba diving for a week, and then went on a one week cruise with a large family group in honor of my mother's 80th birthday. It was great fun!
Jim Ahlstrom
"James C. Ahlstrom" wrote:
I will throw out my design and re-start using your software. But since I have tried this before I know it won't work. But maybe I can come up with a more minimal set of changes.
OK, I have a new design with a minimal set of changes. I am currently testing it by using it. All code is in ftp://ftp.interet.com/pylib.zip and pylib.tar (identical).
The only core changes are to getpathp.c and sysmodule.c in order to add sys.dllfullpath so that the name of Python15.dll (the interpreter shared library) is available. This is a trivial change.
The magic happens in sitecustomize.py. It installs a custom importer which uses Greg's imputil.py. The custom importer searches for the single file interet.pyl in the directory of the main executable. Obviously this is a custom name only usable by us, but that is what sitecustomize is for. Sitecustomize.py is based on Greg's easygen.py, but adds a few important features. It adds a global "Importer" as the SimpleArchive instance, and the methods enable() to turn the importer on and off, __repr__() to show the importer state, and _get_path() to get the importer name.
To boot all this, I have installed exceptions.pyc, site.pyc and sitecustomize.pyc as frozen modules. Actually I froze a dozen other library files too, since this is easier than putting them in interet.pyl.
The above works fine, and is an improvement over just using frozen modules. Our program files change frequently, and it is time consuming to convert the *.pyc to C strings and compile them.
However, this is insufficient to enable a Python programmer to easily ship Python programs. The remaining problems are:
1) Freezing sitecustomize.py etc. requires a compiler and not all people have one.
2) I am using Greg's PYL file format which lacks pyc headers, file paths, flag bits, seeks from the end (for append ability), and multiple sections. We need a standard PYL file format.
3) The executable main program can not be in the PYL file. We need the ability to run __main__() as the main if it is present (like freeze does now), and the ability to run any main from the PYL file.
4) On windows each Python program should have its own icon, so we need a way to include icons.
Comments? If there is agreement on what to do, I volunteer to write the code.
Jim Ahlstrom
James C. Ahlstrom asks:
Gordon McMillan wrote:
On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:
myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]
But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path.
Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely.
For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.
(I forgot imputil.py, archive.py and zlib.pyd in the above list).
- Gordon
Gordon McMillan wrote:
James C. Ahlstrom asks:
Gordon McMillan wrote:
On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:
...
But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path.
Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely.
This is what I use for standalone apps. We just drop the whole tree into the application. site.py lives in the Python directory, as a special version. This suffices, since then no other path than "." is needed to boot.
For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.
And that's the only annoyance: You need to take care of environment space, which is unfortunately not big enough very often. I think a special python.exe for this purpose would be handy, which just ignores all registry, sets the path to the executable's directory, and done.
ciao - chris (who really hates to muck with anyone's registry)
participants (6)
-
Christian Tismer -
Gordon McMillan -
Greg Stein -
James C. Ahlstrom -
M.-A. Lemburg -
Mark Hammond