I just wanted to quickly say that Guido's observation as to how a VFS is overkill is right. Imagine implementing a loader using sqlite and you quickly realize that doing a dull VFS is more than necessary to implement what import needs to function. On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
> "Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
A VFSLoader object would interface with importlib and deal with:
- implementing a finder and loader
- Determine the actual type of file to load (.py, .pyc, .pyo, __pycache__, etc).
- Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this
Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.
Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well.
All those examples except "real filesystem" are of very limited practical value.
Leave all the details such as implicit namespace packages, presence of __init__.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them.
Actually the import loader API is much more suitable and less work to implement than a VFS.
I may try my hand at creating a prototype of this for just the zipimporter and see how it goes.
That would nevertheless be an interesting exercise -- I hope you do it and report back.
> At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release.
Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in.
If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.
The patch should be fairly simple. In a nutshell it just does a:
path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine.
Well, Brett would like to see your feedback on the specific patch. Does it work for you?
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org