Gordon McMillan wrote:

On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:

myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]

But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path. Jim Ahlstrom

James C. Ahlstrom asks:

Gordon McMillan wrote:

...

On Windows, this means you can have a complete Python installation (independent of any other Python installation) in a single directory:

myPython/ python.exe (and/or pythonw.exe) python15.dll (completely vanilla) py_lib.pyz exceptions.py (from the std distr) site.py (hacked to load all the .pyz's) [any other .pyd's or .dll's you want] [more .pyz's if you want] [more .py's if you want]

But the normal PYTHON/Registry stuff is used to find site.py, no? Is there any guarantee that the correct site.py and exceptions.py will be found? The myPython/ directory may not be on the path.

Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely. For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script. (I forgot imputil.py, archive.py and zlib.pyd in the above list). - Gordon

Greg Stein wrote:

In my "small" distribution, I was able to take advantage of the fact that python.exe looks in the current directory for site.py if there is nothing in the registry. The site.py then proceeds to bootstrap the import system.

Gordon:

Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely. For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.

Christian:

This is what I use for standalone apps. We just drop the whole tree into the application. site.py lives in the Python directory, as a special version. This suffices, since then no other path than "." is needed to boot.

And that's the only annoyance: You need to take care of environment space, which is unfortunately not big enough very often. I think a special python.exe for this purpose would be handy, which just ignores all registry, sets the path to the executable's directory, and done.

A Windows user normally sets the "Start in" directory in the shortcut to something other than the directory of the binary. In other words, the current directory "." is unreliable. Users can right-click their icon and change it. I don't like changing other people's registry either. In fact our bank customers hate it when we do. I don't trust the environment PYTHONPATH either. So we can't be sure the site.py bootstrap is bullet-proof. Please stand by while I try to clarify my thoughts... Jim Ahlstrom

Jim Ahlstrom wrote:

Gordon:

...

Not if you deliberately SET PYTHONPATH=. which cuts the registry out entirely. For my installer / standalone, I deliberately muck with the environment before loading python and executing the "main" script.

A Windows user normally sets the "Start in" directory in the shortcut to something other than the directory of the binary. In other words, the current directory "." is unreliable. Users can right-click their icon and change it.

OK - os.path.dirname(sys.argv[0]). In my case it doesn't matter. I have just concatenated my archive (of python15.dll, any .pyzs, exceptions.py, site.py etc.) onto the end of the executable. When run, it opens itself as a file and unpacks the archive into the current directory. Then it does a _putenv("PYTHONPATH=.") and dynamically loads python15.dll. If you're using Run.exe, it even cleans up whatever it unpacked at end of run. More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...) - Gordon

Gordon McMillan wrote:

OK - os.path.dirname(sys.argv[0]).

Oh, I see. Your main program is given as a complete path in the icon target. So "python D:/my/path/myprog.py". I start my main as the frozen module named "__main__", so sys.argv[0] is useless.

In my case it doesn't matter. I have just concatenated my archive (of python15.dll, any .pyzs, exceptions.py, site.py etc.) onto the end of the executable. When run, it opens itself as a file and unpacks the archive into the current directory. Then it does a _putenv("PYTHONPATH=.") and dynamically loads python15.dll. If you're using Run.exe, it even cleans up whatever it unpacked at end of run.

Cool. I didn't realize you could append to an .exe and still have it run. How do you know where your appended data starts?

More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...)

I think this may be a good idea. We could use a Windows user-defined resource named with a magic name such as "PythonPyc", and all resource modules could be loaded in Py_Initialize(). The .rc resource file names the .pyc file so there is no need to convert the bytes to C and compile. The .rc file syntax is the line: string PythonPyc C:/lib/string.pyc You use FindResource(), SizeofResource(), LoadResource() and LockResource() to access these resources. It is possible to replace resources in an exe or dll using BeginUpdateResource(), UpdateResource() and EndUpdateResource(). If we wrote a Python interface into a .pyd, then users don't even need a compiler nor resource compiler. Note that there is Mac code in the distribution which accesses Mac resources. The only reason I have not written all this is it is not usable on Unix. Jim Ahlstrom

"James C. Ahlstrom" wrote:

Gordon McMillan wrote:

...

OK - os.path.dirname(sys.argv[0]).

Oh, I see. Your main program is given as a complete path in the icon target. So "python D:/my/path/myprog.py". I start my main as the frozen module named "__main__", so sys.argv[0] is useless.

Freezing is not so handy as Gordon's trick. Which came partially out of my trick, which came from Fredrik's squeeze trick, or so. [the trick]

Cool. I didn't realize you could append to an .exe and still have it run. How do you know where your appended data starts?

Is it still so? look at the end for the magic cookie, use its info to find the start of a cookie, and there's the data?

...

More advanced would be to make the archive a legit resource section of the exe, but I can't see that as worth the effort (esp. since I don't expect my users to have compilers, and I've already gone to the work of finding the import sections in dll headers so I can find binary dependencies...)

I think this may be a good idea. We could use a Windows user-defined resource named with a magic name such as "PythonPyc", and all resource modules could be loaded in Py_Initialize(). The .rc resource file names the .pyc file so there is no need to convert the bytes to C and compile. The .rc file syntax is the line: string PythonPyc C:/lib/string.pyc You use FindResource(), SizeofResource(), LoadResource() and LockResource() to access these resources.

Yes, this is still an open issue. Greg proposed a similar way. I never came to it, since the drawback of the simple approach is that it works, and it isn't so obvious how. By a resource, I open the app to Joe Hacker and invite him to play with resources. So why should I, if everything can be done in a copy /b style? :-)

It is possible to replace resources in an exe or dll using BeginUpdateResource(), UpdateResource() and EndUpdateResource(). If we wrote a Python interface into a .pyd, then users don't even need a compiler nor resource compiler. Note that there is Mac code in the distribution which accesses Mac resources. The only reason I have not written all this is it is not usable on Unix.

Yes, and to complete it: The append trick *does* work on Unix. And a resource like management of things could be written with Python, defining our own resources. Why care about an OS? Ahem :-) ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

James C. Ahlstrom wrote:

Christian Tismer wrote:

...

Yes, and to complete it: The append trick *does* work on Unix. And a resource like management of things could be written with Python, defining our own resources. Why care about an OS?

[Jim ponders the append trick...]

OK, a really simple idea is to do this: cat python.exe string.pyc site.pyc > myprog.exe That is, just append a bunch of *.pyc to the executable. Anybody can do that. Then make the import mechanism be able to find them. So, look for the *.pyc magic numbers. To recover the module names, use the base file name of the code object attribute co.co_filename.

Same with Python15.dll, append *.pyc to it too.

You can make it a lot easier on yourself if you remember the byte offset of each of those .pyc files. Then, you append a marshalled dictionary mapping the names to the offsets (and remember the byte offset of this marshalled dict). Finally, you append the byte offset of the dict. Recovery is then very easy: seek to (-4) from the end and read the offset. Seek there and unmarshall the dict. Seek to whichever module you want and unmarshal that. I used this technique in constructing my Python library of modules in the small distribution. The offset was at the beginning cuz it was a self-defined format so I just seeked back to the start to write out the file's magic value + offset. But the technique works for appending, too. Personally, I think it is a hack to simply append stuff to an executable (Unix or Windows). I would much rather see a proper COFF or ELF section inserted in there which contains the modules. Cheers, -g -- Greg Stein, http://www.lyra.org/

I got a little time to think about this over the weekend and propose this design. It is a way to package *.pyc files into a single-file portable archive for distribution with commercial, CGI (like Marc is doing) or otherwise simplified distributions. It is dumb-stupid-simple, my personal favorite and a requirement for commercial software. This is not concerned with an "installer", which is a separate problem. Few of these ideas are mine. 1. There is a Python library file format (PYL) which can hold *.pyc files. The format is: a.pyc, b.pyc, marshalled dict, offset of dict, magic number. The a.pyc, b.pyc,... are the bytes of the *.pyc files including the eight byte header. The dict has keys "name", values are the seek offset of the start of the *.pyc files. The "offset of dict" is the seek offset for finding the dict. The magic number identifies the file as a PYL file. The "names" can be normal names such as "os", or dotted names such as "foo.bar". Although initially devoted to *.pyc files, we note that it is possible to include other resources with non-module names such as "more*data". A PYL file is portable to any computer, Unix, Windows, etc. Compression is not included, and should be done at the "installer" level if desired. 2. The PYL file always has the same directory and file name as the binary module containing import.c. but with a .pyl ending: Python Binary PYL File Name /usr/local/bin/python /usr/local/bin/python.pyl /my/so/dir/python.so /my/so/dir/python.pyl C:/python/python15.dll C:/python/python15.pyl These Python binary names are available as sys.executable (the main) and sys.dllfullpath (the shared library or DLL). 3) Since the PYL file can be efficiently read backwards, it can, if desired, be appended the the Python binary itself: cat python15.pyl >> python15.dll 4) The PYL file is created with the Python program makepyl.py and no C compiler is necessary. 5) There is a new optional built-in module "importer" which may be included by editing Setup. It is imported in Py_Initialize() after "sys" is set up, but before any other imports such as "exceptions". It is not an error if it is absent. If present, it replaces __import__ just like imputils.py does. The replacement importer searches for PYL files as a .pyl file, in the current sys.executable, and in the current sys.dllfullpath (name of DLL). Note that importer can use multiple PYL files. Importer is able to import the modules exceptions, site and sitecustomize, and probably most other modules. Importer has methods to print out the names of PYL modules available. You could still override importer using sitecustomize and imputils if desired, in which case it may be convenient to use importer's methods. 6) Alternative to (5): Modules exceptions, site, sitecustomize and imputils are frozen (using frozen modules) into the interpreter binary, and sitecustomize boots imputils. Thereafter the Python logic in sitecustomize and imputils implements the logic of (5). Sitecustomize has methods to print out the names of PYL modules available. 7) The Python main program is enhanced to start "__main__" as the main program if it can be imported, in which case all command line arguments are sent to __main__. This enables you to create a Python program "myprog", and start it with the command "myprog -a arg1 ..." just like a normal program. 8) The Python main can start any module "foo" (which may be in a PYL file) as the main program in response to a new command line argument. This enables you to ship multiple main programs in the PYL file. 9) The current frozenmain.c is eliminated and the enhanced main is used instead. This (I hope) results in a net decrease in code. Where this is going: It enables simplified distribution of Python programs. Greg has demonstrated that import.c can be replaced with a Python module, and that in general at least some of Python should be written in itself. This heads that way. Please send in your "hell no" comments now. Jim Ahlstrom

The "Python Lib format" that Jim discusses is exactly what is used in my "small" distribution. Gordon used a slight variant and even created a nice base class to create similar files. In other words, the code already exists in to match the exact requirements. I strongly agree with Jim's thoughts on moving all Python importing out to the Python level (minus a minimalist set of C functions to dynamically load module M from directory D). Freezing that into the interpreter is a great Step 2. IMO, I would also like to see a frozen Python parser, compiler, and interactive loop, but that is a separate discussion :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ On Mon, 19 Jul 1999, M.-A. Lemburg wrote:

[Jim talking about a Python Lib format]

You may find the attached DictFile.py useful. It does pretty much what you (and even includes a PYCFile subclass). Be warned that it will not work out of the box though, since it references some other modules I normally use. You can safely comment them out though.

-- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 165 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

After writing some code, I see that the library file needs to be a little different. We need to add the file size so all fseek()'s can be done relative to the end. That enables the file to be appended to the executable. The offsets written to the file are from ftell() and are relative to the beginning. The length is subtracted on access.

I havent been following this thread as closely as I would like, but I am a little confused. Are we building yet another packager here, or is this work building on the work Gordon currently "owns"?

I have almost finished makepyl.py, a Python program which creates a PYL file or lists its contents. And C-code changes to import.c to be able to import from a PYL file. However, I am a little short of

If I recall correctly, when Greg posted his version of what grew into Gordon's, there was discussion over the direction import.c should take. Are your patches in line with that? Im just thinking that poor Guido has already discussed this once, so may be reluctant to accept new hacks when the correct direction has been mapped out. As far as I can see, Gordon's code works fine without C level changes, so Im a little unsure what direction this is going, and exactly what this is solving that Gordon's doesnt. Mark.

OK, I put my import Python library code on ftp://ftp.interet.com/pub/pylib.zip and ftp://ftp.interet.com/pub/pylib.tar.gz. The contents are identical. Uncompress in a temp directory. Try listing the contents of the library file makepyl.pyl using "python makepyl.py -t", or make a new one with "python makepyl.py" but be aware that this OVERWRITES makepyl.pyl. You will need to change the config file makepyl.cfg. Copy makepyl.pyl to python15.pyl (in the same directory) to make it active. Then import string, and enter "print string" to see that the module comes from python15.pyl. There are two issues, the library file format and the C-code changes to import.c. I published a proposed library file format here on distutils and it is described in the doc strings. But I didn't describe why I didn't use Greg's nor Gordon's format. Both formats remove the 8-byte .pyc header. But import.c validates this header for .pyc files and it should be retained. There was no file source information, but a user will want to print out the file names in the library. Greg's format could not be appended to another file. Gordon can build the library onto another file with a program, but can not append the same file to different programs. The new format can append the same library file to any other executable or dll using "cat library.pyl >> AnyPythonBinary[|.exe|.so|.dll]. It can also support multiple appends if desired. The changes to import.c add another PY_LIBRARY import method (like PY_SOURCE, PY_COMPILED, etc.) and is programmed in a parallel fashion. The code may be a bit rough but it is sufficient to get an idea. I am going away on vacation tomorrow, so I really hope all this works OK. I will catch up on all your comments when I get back. Jim Ahlstrom

Sigh..., I am back from vacation and as far as I can see, there are no comments except from Gordon. And it looks like all I have managed to do is get Gordon mad at me :-( But anyway: Gordon McMillan wrote:

James C. Ahlstrom wrote:

...

I published a proposed library file format here on distutils and it is described in the doc strings. But I didn't describe why I didn't use Greg's nor Gordon's format. Both formats remove the 8-byte .pyc header. But import.c validates this header for .pyc files and it should be retained.

Why? So import.c can try to recompile from non-existant source?

If you let Greg or me build the archive, you'll find that the compiles get done if they're needed. Then the package contains a python and .pyc's that match that version.

With or without a check, a mismatch will produce an ugly mess.

I am not saying the myprog.pyl file is created incorrectly nor that the version does not match python.exe. The point is that mylib.pyl is a separate file, and so it may be out of phase with python.exe because the user upgraded python.exe but not mylib.pyl. Currently Python will import a file.pyc even if there is no file.py present. If that file.pyc has a bad magic number then Python throws an ImportError. I think a myprog.pyl file is exactly analogous.

...

There was no file source information, but a user will want to print out the file names in the library.

If they want to, the can get the name from the module's __importer__ attribute. They have everything they need to call archive.contents(), too, (if you're letting them get to an interpreter prompt).

I don't understand. I want to check which files were used to create myprog.pyl, and this can only be recorded in myprog.pyl. I don't want the module names "sub1", I want "C:/path/to/sub1.py". This could be used as a check before a product is shipped. But I guess I can live without it.

...

Greg's format could not be appended to another file. Gordon can build the library onto another file with a program, but can not append the same file to different programs.

Um, you mean because there's a "os.remove()" in Builder.py?

Builder.py is mostly concerned with parsing the config file and determining dependencies. The actual archive building is done by the archive class, with some utility functions in archivebuilder.py.

My statement stands. I was hoping to ship myprog.pyl and allow the recipient to use "cat myprog.pyl >> python.exe" if they wanted. I know I can re-make a new python2.exe for each system.

I realize you don't like having to have four .py files in the current directory. I don't like messing with python internals, especially when (among other things) you can use this to distribute a stripped down but otherwise standard python.

I agree that changing Python internals is usually bad, and your design has the very great advantage that no changes are required. I will throw out my design and re-start using your software. But since I have tried this before I know it won't work. But maybe I can come up with a more minimal set of changes.

In addition, both Greg & I are hoping (with some encouraging signs) that his imputil will attain sanctified status, which would probably mean that something like this could be done with just one 'extra' file.

But in the meantime, are you sure you can't get the same effect by using normal embedding techniques in the .exe (ie, instead of just calling Py_Main?). (Well, not quite normal techniques, since you have to use GetProcAddress).

The best I came up with so far is to use freeze to incorporate my Python files in a DLL, and then use a special main to link in the frozen modules. This currently works, but it is not ideal.

...

I am going away on vacation tomorrow, so I really hope all this works OK. I will catch up on all your comments when I get back.

Have a good time.

Thanks. I went scuba diving for a week, and then went on a one week cruise with a large family group in honor of my mother's 80th birthday. It was great fun! Jim Ahlstrom