[Python-Dev] Require loaders set __package__ and __loader__

Brett Cannon brett at python.org
Sun Apr 15 00:50:30 CEST 2012

On Sat, Apr 14, 2012 at 18:32, Guido van Rossum <guido at python.org> wrote:

> On Sat, Apr 14, 2012 at 2:15 PM, Eric Snow <ericsnowcurrently at gmail.com>
> wrote:
> > On Sat, Apr 14, 2012 at 2:56 PM, Brett Cannon <brett at python.org> wrote:
> >> An open issue in PEP 302 is whether to require __loader__ attributes on
> >> modules. The claimed worry is memory consumption, but considering
> importlib
> >> and zipimport are already doing this that seems like a red herring.
> >> Requiring it, though, opens the door to people relying on its existence
> and
> >> thus starting to do things like loading assets with
> >> ``__loader__.get_data(path_to_internal_package_file)`` which allows
> code to
> >> not care how modules are stored (e.g. zip file, sqlite database, etc.).
> >>
> >> What I would like to do is update the PEP to state that loaders are
> expected
> >> to set __loader__. Now importlib will get updated to do that implicitly
> so
> >> external code can expect it post-import, but requiring loaders to set it
> >> would mean that code executed during import can rely on it as well.
> >>
> >> As for __package__, PEP 366 states that modules should set it but it
> isn't
> >> referenced by PEP 302. What I want to do is add a reference and make it
> >> required like __loader__. Importlib already sets it implicitly
> post-import,
> >> but once again it would be nice to do this pre-import.
> >>
> >> To help facilitate both new requirements, I would update the
> >> importlib.util.module_for_loader decorator to set both on a module that
> >> doesn't have them before passing the module down to the decorated
> method.
> >> That way people already using the decorator don't have to worry about
> >> anything and it is one less detail to have to worry about. I would also
> >> update the docs on importlib.util.set_package and
> importlib.util.set_loader
> >> to suggest people use importlib.util.module_for_loader and only use the
> >> other two decorators for backwards-compatibility.
> >
> > +1
> Funny, I was just thinking about having a simple standard API that
> will let you open files (and list directories) relative to a given
> module or package regardless of how the thing is loaded. If we
> guarantee that there's always a __loader__ that's a first step, though
> I think we may need to do a little more to get people who currently do
> things like open(os.path.join(os.path.basename(__file__),
> 'some_file_name') to switch. I was thinking of having a stdlib
> function that you give a module/package object, a relative filename,
> and optionally a mode ('b' or 't') and returns a stream -- and sibling
> functions that return a string or bytes object (depending on what API
> the user is using either the stream or the data can be more useful).
> What would we call thos functions and where would the live?

IOW go one level lower than get_data() and return the stream and then just
have helper functions which I guess just exhaust the stream for you to
return bytes or str? Or are you thinking that somehow providing a function
that can get an explicit bytes or str object will be more optimized than
doing something with the stream? Either way you will need new methods on
loaders to make it work more efficiently since loaders only have get_data()
which returns bytes and not a stream object. Plus there is currently no API
for listing the contents of a directory.

As for what to call such functions, I really don't know since they are
essentially abstract functions above the OS which work on whatever storage
backend a module uses.

For where they should live, it depends if you are viewing this as more of a
file abstraction or something that ties into modules. For the former it
seems like shutil or something that dealt with higher order file
manipulation. If it's the latter I would say importlib.util.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120414/badfa7fb/attachment.html>

More information about the Python-Dev mailing list