[Python-Dev] zipimport & import hooks

Kevin Jacobs jacobs@penguin.theopalgroup.com
Fri, 6 Dec 2002 08:32:21 -0500 (EST)


On Fri, 6 Dec 2002, Fredrik Lundh wrote:
> Just van Rossum wrote:
> > It seems the majority doesn't like non-strings on sys.path, which to me is
> > understandable, especially if you take PYTHONPATH into consideration. Let's try
> > to focus on that.
> 
> Has anyone presented any hard data, or are people just guessing?  As far
> as I can tell, I'm the only one here who has posted fragile code from a real
> shipping product -- and I still believe it's a bogus argument.

I've reviews several hundred thousand lines of code in our repository, and
here is the frequency breakdown of 588 usages of sys.path:

  84%: sys.path.insert(0, package_path)
  10%: sys.path.append(package_path)
   2%: sys.path.extend(path_list)
   1%: str(package_path) in sys.path
  <1%: random debugging/unit test code that prints sys.path or compares
       sys.path with a previous copy of sys.path.

The remaining ~1% use this fairly hairy idiom to implement a plug-in system:

  # in a package __init__.py
  # ...
  for filename in os.listdir(__path__[0]):
    if not filename.endswith('.py') or filename.startswith('__'):
      continue

    name = filename[:-3]

    plugin = __import__('%s.%s' % (base,name), None, None, [name])

    # ...do something with plugin...

Clearly, the above code can be redone in a much nicer way that doesn't
depend on __path__, the qualified base package name, and __import__.

Other than the above hard numbers, I have only some vague user-feelings to
add to the discussion based on a poll of our in-house developers:

  1) They all are very convinced that

       sys.path.append('/usr/path/stuff.zip') 

     should work just like
     
       > mkdir /usr/path/stuff
       > cd /usr/path/stuff
       > unzip -d stuff.zip

       sys.path.append('/usr/path/stuff')

     Any other proposal that magically ignores the zip file name is right
     out, even if it makes it marginally more challenging to wrap the python
     standard library in a zip file.

  2) They don't care if Python stores objects, strings, or bananas in
     sys.path, so long as 

       sys.path=map(str,sys.path) 

     results in a human-readable path and does not change how imports occur.

  3) They all want ZIP archives to be a builtin feature, but don't care if
     they have to write an extra line of code to register a TAR/BZ2/whatever
     import hook.

  4) They don't care if a small amount of code breaks.  It is trivial to
     notice when it does break and easy to fix.


Note that these comments are a result of a 15 minute discussion among a very
mix bag of programmers.  Some have been with Python for many years, others
are in their first week of learning.  Some come from primarily one of
Java/C/C++/VB/Smalltalk/Scheme/Pascal backgrounds, others are extreme language
polyglots.  So, given the amount of discussion on python-dev, I was surprised
how quickly the group converged upon the previous criteria with virtual
unanimity.  The only dissenting voice was our lone .NET fanatic who wouldn't
shut up about how .NET assemblies were by far the superior way of doing
things.  We all have long since learned to ignore him.

Hope this helps,
-Kevin

--
Kevin Jacobs
The OPAL Group - Enterprise Systems Architect
Voice: (216) 986-0710 x 19         E-mail: jacobs@theopalgroup.com
Fax:   (216) 986-0714              WWW:    http://www.theopalgroup.com