than necessary to implement what import needs to function.

On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:

--
>>>>> "Brett" == Brett Cannon <brett@python.org> writes:

> I opened
> https://bugs.python.org/issue25711 to specifically try to
> fix this issue once and for all and along the way modernize
> zipimport by rewriting it from scratch to be more
> maintainable

Every time I read about impementing a custom loader:

https://docs.python.org/3/library/importlib.html

I've wondered why python does not have some sort of virtual
filesystem layer to deal with locating modules/packages/support
files. Virtual file systems seem like a good way to store data on a
wide range of storage devices.

Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.

A VFSLoader object would interface with importlib and deal with:

- implementing a finder and loader

- Determine the actual type of file to load (.py, .pyc, .pyo,
__pycache__, etc).

- Do all of it's work by calling virtual functions such as:
* listdir(path)
* read(path)
* stat(path) # for things like mtime, size, etc
* write(path, data) # not all VFS implement this

Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.

Then things like a ziploader would just inherit from VFSLoader
implement the straight forward methods and everything should "just
work" :). I see no reason why every type of loader (real filesystem,
http, ssh, sql database, etc) would not do this as well.

All those examples except "real filesystem" are of very limited practical value.

Leave all
the details such as implicit namespace packages, presence of
__init__.py[oc] files, .pth files, etc in one single
location and put the details on how to interact with the actual
storage device in leaf classes which don't know or care about the high
level details. They would not even know they are loading python
modules. It is just blobs of data to them.

Actually the import loader API is much more suitable and less work to implement than a VFS.

I may try my hand at creating a prototype of this for just the
zipimporter and see how it goes.

That would nevertheless be an interesting exercise -- I hope you do it and report back.

> At this point the best option might be, Mike, if you do a
> code review for https://bugs.python.org/issue17633, even if
> it is simply a LGTM. I will then personally make sure the
> approved patch gets checked in for Python 3.6 in case the
> rewrite of zipimport misses the release.

Cool. I'll see what I can do. I was having a bit of trouble with
the register/login part of the bug tracker. Which is why I came
here. I'll battle with it one more time and see if I can get it to
log me in.

If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.

The patch should be fairly simple. In a nutshell it just does a:

path.replace('.', '/') + '/' in two locations. One where it checks
for the path being a directory entry in the zip file and the second to
return an implicit namespace path (instead of not found) if it is a
match. I'll check the patch on the tracker and see if it still works
with 3.5.1. If not I'll attach mine.

Well, Brett would like to see your feedback on the specific patch. Does it work for you?

--
--Guido van Rossum (python.org/~guido)

_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org