At 11:22 AM +0200 5/21/99, hinsen@dirac.cnrs-orleans.fr wrote:

...

find_module() should then first check sys.builtin_module_names with the full name before doing anything else. (probably only when it is confirmed that "foo" is a package.)

All that would be doable, but the real problem is the name of the init function!

Right, I was being naive: I thought that was just "a" problem...

Only one module can define a global symbol "initbar". So the one for foo.bar would have to be called "initfoo.bar" (or something similar). On the other hand, when the same module is used dynamically, the init function must be called "initbar" again (unless the current import mechanism is changed).

So there are really two options: 1) Define a switch that C extensions can check to determine whether the init func should be called initbar or initfoo_bar (or something). This means it's up to the extension developer to cater for statically linked builtin submodules by doing something like this in the extension source: #ifdef PY_STATIC_SUBMODULES #define initbar initfoo_bar #endif 2) change the DL import mechanism so the init function *has* to be called initfoo_bar. But then, to remain backwards compatible you'd still have use a switch, so it doesn't help much now. Just

At 4:08 PM +0200 5/21/99, Konrad Hinsen wrote:

...

2) change the DL import mechanism so the init function *has* to be called initfoo_bar. But then, to remain backwards compatible you'd still have use a switch, so it doesn't help much now.

Backwards compatible with what? Currently builtin modules can't be in packages at all, so nothing's lost.

But DLLs *can* be (that's the whole point, no?). If the rules for the init func changes, I think at least Marc-Andre L. won't be too happy: all (?) of his extensions use DLLs as submodules, so he would need to add switches to remain compatible with 1.5.2. I'm sure he's not the only one. Just

FWIW, I _do_ use DLLs in packages, and it causes me no end of grief. I need to have special runtime hacks that works with __path__, I need a special __init__ in the package where the DLL is to "appear", and also need even further special casing for Freeze! So although I can see the problems with the mechanisms, IMO it is very important that packages be capable of treating DLLs as first-class citizens. Personally, I would not have a "compatibility" problem as such, but I would need to remove or update my hacks - but I find that reasonable. Mark.

...

Backwards compatible with what? Currently builtin modules can't be in packages at all, so nothing's lost.

But DLLs *can* be (that's the whole point, no?). If the rules for the init func changes, I think at least Marc-Andre L. won't be too happy: all (?) of his extensions use DLLs as submodules, so he would need to add switches to remain compatible with 1.5.2. I'm sure he's not the only one.

Mark Hammond wrote:

FWIW, I _do_ use DLLs in packages, and it causes me no end of grief. I need to have special runtime hacks that works with __path__, I need a special __init__ in the package where the DLL is to "appear", and also need even further special casing for Freeze!

I've been using DLL/SOs in packages with much success for some time now. Don't know why you need any hacks to get this going though: it works right out of the box for me. The situation is a little different for frozen apps without shared libs though: the extension modules will become top-level modules. Haven't frozen those kinds of apps yet, but it should still work out of the box (except maybe when you pass pickles from an app using top-level modules to one using in-package modules). -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 220 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg writes:

Note that this kind of search will be very costly due the amount of IO needed to search the path. Some sort of fastpath hook

Marc-Andre, Why does this need to be so costly? Compared to the current scheme, there's little to add. Once a package has been identified (and *only* then!), search the path for all the appropriate subdirectories (one stat() for each path entry). The current approach requires about a half dozen stats for each path entry: foo.py, foo.py[co], foomodule.so, foo.so, foo/ + foo/__init__.py + foo.__init__.py[co]. It will typically be even cheaper for sub-packages, because the original path will usually be much shorter than sys.path. Note that I'm not saying there shouldn't be some sort of directory caching; loading Grail is still dog slow, and I've no doubt that the 600+ stat() calls contribute to that! 1-) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

M.-A. Lemburg writes:

Well, I was referring to the additional lookup needed to find the next package dir of the same name. Say you put the Python package into site-packages and the binaries into plat-<platform>.

I didn't say it was free; just that the cost was insignificant compared to the current cost. My sys.path in an interactive interpreter contains 11 entries. If I want to add a package with both $prefix and $exec_prefix components, the worst case is that the directory holding the __init__.py* is the last path entry, and the other directory is in the immediately preceeding path entry. After the current mechanism locates the __init__.py* file, it needs to build the __path__ for the package. It takes 10 stat() calls to locate the additional directory. Considering that the initial search that caused the package module to be created took: 11 stats to see if the entries contained the appropriate directory + 2 stats to determine that the first directory of the package (the one that doesn't have __init__.py*) wasn't it + 36 to determine that the first 9 directories didn't contain a matching .so|module.so|.py|.py[co]. Plus at least one to actually find the __init__.pyc; two if only the .py is available. (I think I followed the code right. ;) That's 59 system calls (either stat() or open(), the later hidden inside fdopen()). I don't the added 10 to get the right __path__ is worth worrying about. It's the .py[co] files that are expensive to load! Once you've created the package, sub-modules are very cheap: you will typically have no more than two path entries to check even once all this is in place. I said:

caching; loading Grail is still dog slow, and I've no doubt that the 600+ stat() calls contribute to that! 1-)

Oops, after following through with the math, I'd have to adjust this to 6000 stat()/open() calls for Grail. Sorry! And back to Marc-Andre:

I would very much like to see some sort of caching in the interpreter. The fastpath hook I implemented uses a marshalled dict stored in the user's home dir for the lookup. Once created,

I don't think I'd store the cache; if a user's home directory is mounted via NFS (common), then it may often be wrong if the user actively works with a variety of hosts with different versions or installations of Python. The benefits of a cache are greatest for applications that import a lot of modules (like Grail!); the cache can be built using a directory scan as each directory is searched. (I think one of the guys from CWI did this at one point and had really good results; Jack?) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

M.-A. Lemburg wrote:

... Anyway, the dynamic and static versions are both implementable using the hook, so I'd opt for going into that direction rather than hard-wiring some logic into the interpreters core.

IMO, the interpreter core should perform as little searching as possible. Basically, it should only contain bootstrap stuff. It should look for a standard importing module and load that. After it is loaded, the import mechanism should defer to Python for all future imports. (the cost of running Python code is minimal against the I/O used by the import) IMO #2, the standard importing module should operate along the lines of imputil.py. Cheers, -g -- Greg Stein, http://www.lyra.org/

M.-A. Lemburg writes:

Wow, what an analysis.

And such fun, as well! ;-)

...

It's the .py[co] files that are expensive to load! Once you've created the package, sub-modules are very cheap: you will typically have no more than two path entries to check even once all this is in place.

I'm not sure I follow you here: do you mean with a package dir cache in place or using the system implemented in the current

Anything contained within a package is relatively cheap to load because the search path is shorter. Currently, if the __init__.py* does nothing to the __path__, there's only one entry! In the current scheme, the .py[co] files are the last thing checked within a directory during the search. Loading one of these costs more in searching than any other type of module. Of course, parsing Python isn't free either, so loading a .py file for which no .py[co] exists is really more expensive, it's just found a little sooner. I said:

caching; loading Grail is still dog slow, and I've no doubt that the 600+ stat() calls contribute to that! 1-)

And then I corrected myself:

Oops, after following through with the math, I'd have to adjust this to 6000 stat()/open() calls for Grail. Sorry!

Ok, I loaded Grail and looked more carefully. I was thinking it was loading about 100 modules. Well, that's at the point that it loads the users .grail/user/grailrc.py (if it exists). By the time my home page was loaded, there were 145 distinct module objects loaded into sys.modules, and 17 entries on sys.path. Lots of Grail modules are in packages these days, but there are also a lot loaded from the standard library. So lets say there are probably around 5000 stat()/open() calls (reduce the number due to package use, then increase it again because (a) there are more modules being loaded than I'd estimated, and (b) the standard library is quite a ways down sys.path.

This seems like something to worry about and probably also enough to try really hard to find a good solution, IMHO.

This is where a good caching system makes a lot of sense.

True, that's why the hook allows you to code the strategy in Python. Note that my current version uses the sys.path as key into a table of name:file mappings, so even when using different setups (which will certainly have some differences in sys.path), the cache should work. Maybe one should add some more information to the key... like the platform specifica or the even the mtimes of the directories on the path.

I'm not sure that keying on sys.path is sufficient. Around here, a Solaris/SPARC and Solaris/x86 box are likely to share the same sys.path. That doesn't mean the directories are the same; the differences are taken care of via NFS. Using the mtimes as part of the key means you don't have any way to clear the cache: an older mtime may just mean the version of the path for a different platform, which still wants to use the cache! Perhaps it could be keyed on (platform, dir), and the mtimes could be used to determine the need to refresh that directory. Doing this right is hard, and can be substantially affected by a site's filesystem layout. Avoiding problems due to issues like these is a good reason to use a runtime-only cache. A site for which this isn't sufficient can the use the "hook" mechanism to install something that can do better within the context of specific filesystem management policies.

Yep, remember that too. The problem with these scans is that directories may contain huge amounts of files and you would need to check all of them against the module extensions Python

They probably won't contain much other than Python modules in a reasonable installation. There's no need to filter the list; just include every file, and then test for the appropriate entries when attempting a specific import. This limits the up-front cost substantially. If we don't assume a reasonable installation (non-module files in the module dirs), it just gets slower and people have an incentive to clean up their installation. This is acceptable.

Anyway, the dynamic and static versions are both implementable using the hook, so I'd opt for going into that direction rather than hard-wiring some logic into the interpreters core.

I have no problems with using a "hook" to implement a more efficient mechanism. I just want the "standard" mechanism to be efficient, because that's the one I'll use. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives

Fred L. Drake wrote:

M.-A. Lemburg writes:

...

This seems like something to worry about and probably also enough to try really hard to find a good solution, IMHO.

This is where a good caching system makes a lot of sense.

...

True, that's why the hook allows you to code the strategy in Python. Note that my current version uses the sys.path as key into a table of name:file mappings, so even when using different setups (which will certainly have some differences in sys.path), the cache should work. Maybe one should add some more information to the key... like the platform specifica or the even the mtimes of the directories on the path.

I'm not sure that keying on sys.path is sufficient. Around here, a Solaris/SPARC and Solaris/x86 box are likely to share the same sys.path. That doesn't mean the directories are the same; the differences are taken care of via NFS. Using the mtimes as part of the key means you don't have any way to clear the cache: an older mtime may just mean the version of the path for a different platform, which still wants to use the cache! Perhaps it could be keyed on (platform, dir), and the mtimes could be used to determine the need to refresh that directory. Doing this right is hard, and can be substantially affected by a site's filesystem layout. Avoiding problems due to issues like these is a good reason to use a runtime-only cache. A site for which this isn't sufficient can the use the "hook" mechanism to install something that can do better within the context of specific filesystem management policies.

Right and that's the key point in optionally moving (at least) the lookup machinery into Python. Admins could then use site.py to add optimized lookup cache implementations for their site. The default implementation should probably be some sort of dynamic cache like the one you sketched below.

...

Yep, remember that too. The problem with these scans is that directories may contain huge amounts of files and you would need to check all of them against the module extensions Python

They probably won't contain much other than Python modules in a reasonable installation. There's no need to filter the list; just include every file, and then test for the appropriate entries when attempting a specific import. This limits the up-front cost substantially.

Ok, point taken.

If we don't assume a reasonable installation (non-module files in the module dirs), it just gets slower and people have an incentive to clean up their installation. This is acceptable.

True.

...

Anyway, the dynamic and static versions are both implementable using the hook, so I'd opt for going into that direction rather than hard-wiring some logic into the interpreters core.

I have no problems with using a "hook" to implement a more efficient mechanism. I just want the "standard" mechanism to be efficient, because that's the one I'll use.

The hook idea makes the implementation a little more open. Still, I think that even the "standard" lookup/caching scheme should be implemented in Python. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 217 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Greg Stein wrote:

M.-A. Lemburg wrote:

...

Greg Stein wrote:

...

... IMO, the interpreter core should perform as little searching as possible. Basically, it should only contain bootstrap stuff. It should look for a standard importing module and load that. After it is loaded, the import mechanism should defer to Python for all future imports. (the cost of running Python code is minimal against the I/O used by the import)

IMO #2, the standard importing module should operate along the lines of imputil.py.

You mean moving the whole import mechanism away from C and into Python ? Have you tried such an approach with your imputil.py ?

Yes and yes.

Using Python's import hook effectively means that you completely take over Python's import mechanism (one of its failings, IMO). imputil.py is designed to provide for iterating through a list of importers, looking for one that works.

In any case... yes, I've use imputil to the exclusion of Python's import logic. You still need imp.new_module() and imp.get_magic(). But that does implies that you can axe a lot of stuff outside of that. My tests don't have loading of dynamic libraries, so you would still need an imp function to load that (but strip the *searching* for the module).

So all that's needed is some DLL loader magic in C, some Win32 Registry APIs and the Mac fork stuff. If imp were extended to provide those APIs instead of using them itself and then moving all the other code to Python we should arrive at a solution that would also cover things like internet import, import via pipes, import from databases, etc.

...

And just curious: why did Guido recode ni.py in C if he could have used ni.py in your proposed way instead ?

For two reasons that I can think of:

1) people had to manually import "ni"

Well, I guess ni could have been imported by Python at startup -- just like exceptions is right now. The problem here: what if it doesn't find the import logic module ? Plus, how does it do import ;) ?

2) it took over the import hook which effectively prevents further use of it (or if somebody *did* use it, then they would wipe out ni's functionality; again, this is why I dislike the current hook approach and like a list-based approach which is possible via imputil)

Right. BTW: that's a general problem with all kinds of hooks, e.g. the system exit hook is another problem area. A generic solution to this wouldn't be a bad idea either. Here is one way to do this for the sys.exitfunc hook that I'm currently using: """ Central Registry for sys.exitfunc()-type functions. """ __version__ = "0.1" import sys,traceback class ExitFunctionDispatcher: """ Singleton that manages exit functions. These function will be called upon system exit in reverse order of their registering. """ def __init__(self): """ Install the dispatcher as sys.exitfunc() """ self.exitfunc_list = [] if hasattr(sys,'exitfunc'): self.old_exitfunc = sys.exitfunc else: self.old_exitfunc = None sys.exitfunc = self.exitfunc def exitfunc(self, write=sys.stderr.write,print_exc=traceback.print_exc, stderr=sys.stderr): """ This is the exitfunc that we install to dispatch the processing to the registered other functions """ for f in self.exitfunc_list: try: f() except: write('Error while executing Exitfunction %s:\n' % f.__name__) print_exc(10,stderr) # Now that we're finished, call the previously installed exitfunc() if self.old_exitfunc: self.old_exitfunc() def register(self,f,position=0): """ Register f as exit function. These functions must not take parameters. - position = 0: register the function at the beginning of the list; these functions get called before the functions already in the list (default) - position = -1: register the function at the end of the list; the function will get called after all other functions """ if position < 0: position = position + len(self.exitfunc_list) + 1 self.exitfunc_list.insert(position,f) def deregister(self,f): """ Remove the function f from the exitfunc list; if it is not found, the error is silently ignored. """ try: self.exitfunc_list.remove(f) except: pass # Create the singleton ExitFunctions = ExitFunctionDispatcher() -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 213 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/