[Python-Dev] difficulty of implementing phase 2 of PEP 302 in Python source

Brett Cannon brett at python.org
Thu Sep 28 02:26:14 CEST 2006

On 9/27/06, Phillip J. Eby <pje at telecommunity.com> wrote:
> At 04:11 PM 9/27/2006 -0700, Brett Cannon wrote:
> >On 9/27/06, Phillip J. Eby
> ><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote:
> >>At 02:11 PM 9/27/2006 -0700, Brett Cannon wrote:
> >> >But it has been suggested here that the import machinery be rewritten
> in
> >> >Python.  Now I have never touched the import code since it has always
> had
> >> >the reputation of being less than friendly to work with.  I am asking
> for
> >> >opinions from people who have worked with the import machinery before
> if
> >> >it is so bad that it is worth trying to re-implement the import
> semantics
> >> >in pure Python or if in the name of time to just work with the C
> >> >code.  Basically I will end up breaking up built-in, .py, .pyc, and
> >> >extension modules into individual importers and then have a chaining
> class
> >> >to act as a combined .pyc/.py combination importer (this will also
> make
> >> >writing out to .pyc files an optional step of the .py import).
> >>
> >>The problem you would run into here would be supporting zip imports.
> >
> >I have not looked at zipimport so I don't know the exact issue in terms
> of
> >how it hooks into the import machinery.  But a C level API will most
> >likely be needed.
> I was actually assuming you planned to reimplement that in Python as well,
> and hence the need for the storage/format separation.

I was not explictly planning on it.

>>   It
> >>would probably be more useful to have a mapping of file types to "format
> >>handlers", because then a filesystem importer or zip importer would then
> be
> >>able to work with any .py/.pyc/.pyo/whatever formats, along with any new
> >>ones that are invented, without reinventing the wheel.
> >
> >So you are saying the zipimporter would then pull out of the zip file the
> >individual file to import and pass that to the format-specific importer?
> No, I'm saying that the zipimporter would simply call the format importers
> in sequence, as in your original concept.  However, these importers would
> call *back* to the zipimporter to ask if the file they are looking for is
> there.

Ah, OK.  So for importing 'email', the zipimporter would call the .pyc
importer and it would ask the zipimporter, "can you get me email.pyc?" and
if it said no it would move on to asking the .py importer for email.py, etc.

>>Thus, whether it's file import, zip import, web import, or whatever, the
> >>same handlers would be reusable, and when people invent new extensions
> like
> >>.ptl, .kid, etc., they can just register format handlers instead.
> >
> >So a sepration of data store from data interpretation for
> importation.  My
> >only worry is a possible explosion of checks for the various data
> >types.  If you are using the file data store and had .py, .pyc, .so,
> >module.so , .ptl, and .kid registered that might suck in terms of
> >performance hit.
> Look at it this way: the parent importer can always pull a directory
> listing once and cache it for the duration of its calls to the child
> importers.  In practice, however, I suspect that the stat calls will be
> faster.  In the case of a zipimport parent, the zip directory is already
> cached.
> Also, keep in mind that most imports will likely occur *before* any
> special
> additional types get registered, so the hits will be minimal.  And the
> more
> of sys.path is taken up by zip files, the less of a hit it will be for
> each
> query.

That's fine.  Just thinking about how the current situation sucks for NFS
but how caching just isn't done.  But obvoiusly this could change.

>   And I am assuming for a web import that it would decide based on the
> > extension of the resulting web address?
> No - you'd effectively end up doing a web hit for each possible
> extension.  Which would suck, but that's what caching is
> for.  Realistically, you wouldn't want to do web-based imports without
> some
> disk-based caching anyway.
> >   And checking for the various types might not work well for other data
> > store types.  Guess you would need a way to register with the data store
> > exactly what types of data interpretation you might want to check.
> No, you just need a method on the parent importer like get_data().
> >Other option is to just have the data store do its magic and somehow know
> >what kind of data interpretation is needed for the string returned (e.g.,
> >a database data store might implicitly only store .py code and thus know
> >that it will only return a string of source).  Then that string and the
> >supposed file extension is passed ot the next step of creating a module
> >from that data string.
> Again, all that's way more complex than you need; you can do the same
> thing
> by just raising IOError from get_data() when asked for something that's
> not
> a .py.
> >>Format handlers could of course be based on the PEP 302 protocol, and
> >>simply accept a "parent importer" with a get_data() method.  So, let's
> say
> >>you have a PyImporter:
> >>
> >>      class PyImporter:
> >>          def __init__(self, parent_importer):
> >>              self.parent = parent_importer
> >>
> >>          def find_module(self, fullname):
> >>              path = fullname.split('.')[-1]+'.py'
> >>              try:
> >>                  source = self.parent.get_data(path)
> >>              except IOError:
> >>                  return None
> >>              else:
> >>                  return PySourceLoader(source)
> >>
> >>See what I mean?  The importers and loaders thus don't have to do direct
> >>filesystem operations.
> >
> >I think so.  Basically you want more of a way to stack imports so that
> the
> >basic importers are just passed the string of what it is supposed to load
> >from.  Other importers higher in the chain can handle getting that
> string.
> No, they're full importers; they're not passed "a string".  The only
> difference between this and your original idea of an importer chain is
> that
> I'm saying the chained format-specific importers need to know who their
> "parent" importer (the data store) is, so they can be data-store
> independent.  Everything else can be done with that, and perhaps a few
> extra parent importer methods for stat, save, etc.


>>Of course, to fully support .pyc timestamp checking and writeback, you'd
> >>need some sort of "stat" or "getmtime" feature on the parent importer,
> as
> >>well as perhaps an optional "save_data" method.  These would be
> extensions
> >>to PEP 302, but welcome ones.
> >
> >Could pass the string representing the location of where the string came
> >from.  That would allow for the required stat calls for .pyc files as
> >needed without having to implement methods just for this one use case.
> Huh?  In order to know if a .pyc is up to date, you need the st_mtime of
> the .py file.  That can't be done in the parent importer without giving it
> format knowledge, which goes against the point of the exercise.

Sorry, thought .pyc files based whether they needed to be recompiled based
on the stat info on the .py and .pyc file, not on data stored from within
the .pyc .

> something like stat() and save() methods need to be available on the
> parent, if it can support them.
> >>Anyway, based on my previous work with pkg_resource, pkgutil, zipimport,
> >>import.c , etc. I would say this is how I'd want to structure a
> >>reimplementation of the core system.  And if it were for Py3K, I'd
> probably
> >>treat sys.path and all the import hooks associated with it as a single
> >>meta-importer on sys.meta_path -- listed after a meta-importer for
> handling
> >>frozen and built-in modules.  (I.e., the meta-importer that uses
> sys.path
> >>and its path hooks would be last on sys.meta_path.)
> >
> >Ah, interesting idea!  Could even go as far as removing sys.path and just
> >making it an attribute of the base importer if you really wanted to make
> >it just meta_path for imports.
> Perhaps, but then that just means you have to have a new variable for
> 'sys.path_importer' or some such, just to get at it.  (i.e., code won't be
> able to assume it's always the last item in sys.meta_path).  So this seems
> wasteful and changing things just for the sake of change, vs. just keeping
> the other PEP 302 sys variables.  I just think the *implementation* of
> them
> can move to sys.meta_path, as that simplifies the main __import__ function
> down to just calling meta_path importers in sequence, modulo some package
> issues.
> One other rather tricky matter is that the sys.path meta-importer has to
> deal with package __path__ management...  and actually, meta_path
> importers
> are supposed to receive a copy of sys.path...  ugh.  Well, it was a nice
> idea, but I guess you can't actually implement sys.path using a meta_path
> importer.  :(  For Py3K, we could drop the path argument to find_module()
> and manage it, but it can't be done and still allow current meta_path
> hooks
> to work right.

Ah, true.

>>In other words, sys.meta_path is really the only critical import hook from
> >>the raw interpreter's point of view.  sys.path, however, (along with
> >>sys.path_hooks and sys.path_importer_cache) is critical from the
> >>perspective of users, applications, etc., as there has to be some way to
> >>get things onto Python's path in the first place.
> >
> >Yeah, I think I get it.  I don't know how much it simplifies things for
> >users but I think it might make it easier for alternative import writers.
> That was the idea, yes.  :)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/python-dev/attachments/20060927/6718c9f1/attachment.html 

More information about the Python-Dev mailing list