zipimport.c broken with implicit namespace packages
BRIEF INTRODUCTION: I've been using python since the early 1.X releases. Mostly for application development. On occasion I've contributed bits to the core:
grep Romberg Misc/ACKS Mike Romberg
I've recently ported a large application to python3 (it started life as using 1.1 so it has been a long road for this codebase). The one big killer feature of python3 I'm attempting to use is implicit namespace packages. But they are broken with the zipimport.c module. It seems that zipimport.c never worked with these as it is comparing paths in the form 'a.b.c' to paths in the form 'a/b/c'. I created a patch that fixes this and makes zipimport work exactly the same as a standard filesystem import. I was getting my patch ready to submit when I found that this problem has already been reported: https://bugs.python.org/issue17633 Is there anything I can do to help fix this issue? I could polish up my patch create test cases and submit them. But it looks like the above patch does the same thing and is in "the process". But it has been "in the process" for three years. What else needs to be done? I'll help if I can. Mike
FIrst off, Mike, sorry about the bug. We have unfortunately let zipimport get into a sorry state that has made no one want to work on the code anymore. That being said, I opened https://bugs.python.org/issue25711 to specifically try to fix this issue once and for all and along the way modernize zipimport by rewriting it from scratch to be more maintainable (or whatever the module is named in case we break backwards-compatibility). At this point the best option might be, Mike, if you do a code review for https://bugs.python.org/issue17633, even if it is simply a LGTM. I will then personally make sure the approved patch gets checked in for Python 3.6 in case the rewrite of zipimport misses the release. On Sat, 2 Jan 2016 at 11:35 <mike.romberg@comcast.net> wrote:
BRIEF INTRODUCTION: I've been using python since the early 1.X releases. Mostly for application development. On occasion I've contributed bits to the core:
grep Romberg Misc/ACKS Mike Romberg
I've recently ported a large application to python3 (it started life as using 1.1 so it has been a long road for this codebase). The one big killer feature of python3 I'm attempting to use is implicit namespace packages. But they are broken with the zipimport.c module.
It seems that zipimport.c never worked with these as it is comparing paths in the form 'a.b.c' to paths in the form 'a/b/c'. I created a patch that fixes this and makes zipimport work exactly the same as a standard filesystem import. I was getting my patch ready to submit when I found that this problem has already been reported:
https://bugs.python.org/issue17633
Is there anything I can do to help fix this issue? I could polish up my patch create test cases and submit them. But it looks like the above patch does the same thing and is in "the process". But it has been "in the process" for three years. What else needs to be done? I'll help if I can.
Mike _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
--
"Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable Every time I read about impementing a custom loader: https://docs.python.org/3/library/importlib.html I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices. A VFSLoader object would interface with importlib and deal with: - implementing a finder and loader - Determine the actual type of file to load (.py, .pyc, .pyo, __pycache__, etc). - Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well. Leave all the details such as implicit namespace packages, presence of __init__.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them. I may try my hand at creating a prototype of this for just the zipimporter and see how it goes. > At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release. Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in. The patch should be fairly simple. In a nutshell it just does a: path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine. Mike
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
"Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
A VFSLoader object would interface with importlib and deal with:
- implementing a finder and loader
- Determine the actual type of file to load (.py, .pyc, .pyo, __pycache__, etc).
- Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this
Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.
Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well.
All those examples except "real filesystem" are of very limited practical value.
Leave all the details such as implicit namespace packages, presence of __init__.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them.
Actually the import loader API is much more suitable and less work to implement than a VFS.
I may try my hand at creating a prototype of this for just the zipimporter and see how it goes.
That would nevertheless be an interesting exercise -- I hope you do it and report back.
> At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release.
Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in.
If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.
The patch should be fairly simple. In a nutshell it just does a:
path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine.
Well, Brett would like to see your feedback on the specific patch. Does it work for you? -- --Guido van Rossum (python.org/~guido)
I just wanted to quickly say that Guido's observation as to how a VFS is overkill is right. Imagine implementing a loader using sqlite and you quickly realize that doing a dull VFS is more than necessary to implement what import needs to function. On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
> "Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
A VFSLoader object would interface with importlib and deal with:
- implementing a finder and loader
- Determine the actual type of file to load (.py, .pyc, .pyo, __pycache__, etc).
- Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this
Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.
Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well.
All those examples except "real filesystem" are of very limited practical value.
Leave all the details such as implicit namespace packages, presence of __init__.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them.
Actually the import loader API is much more suitable and less work to implement than a VFS.
I may try my hand at creating a prototype of this for just the zipimporter and see how it goes.
That would nevertheless be an interesting exercise -- I hope you do it and report back.
> At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release.
Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in.
If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.
The patch should be fairly simple. In a nutshell it just does a:
path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine.
Well, Brett would like to see your feedback on the specific patch. Does it work for you?
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sat, 2 Jan 2016, 20:42 Brett Cannon <brett@python.org> wrote:
I just wanted to quickly say that Guido's observation as to how a VFS is overkill is right. Imagine implementing a loader using sqlite and you quickly realize that doing a dull VFS is more
"dull" -> "full" than necessary to implement what import needs to function.
On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
>> "Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
A VFSLoader object would interface with importlib and deal with:
- implementing a finder and loader
- Determine the actual type of file to load (.py, .pyc, .pyo, __pycache__, etc).
- Do all of it's work by calling virtual functions such as: * listdir(path) * read(path) * stat(path) # for things like mtime, size, etc * write(path, data) # not all VFS implement this
Emulating a decent filesystem API requires you to implement functionality that would never be used by an import loader (write() is an example -- many of the stat() fields are another example). So it would just be overkill.
Then things like a ziploader would just inherit from VFSLoader implement the straight forward methods and everything should "just work" :). I see no reason why every type of loader (real filesystem, http, ssh, sql database, etc) would not do this as well.
All those examples except "real filesystem" are of very limited practical value.
Leave all the details such as implicit namespace packages, presence of __init__.py[oc] files, .pth files, etc in one single location and put the details on how to interact with the actual storage device in leaf classes which don't know or care about the high level details. They would not even know they are loading python modules. It is just blobs of data to them.
Actually the import loader API is much more suitable and less work to implement than a VFS.
I may try my hand at creating a prototype of this for just the zipimporter and see how it goes.
That would nevertheless be an interesting exercise -- I hope you do it and report back.
> At this point the best option might be, Mike, if you do a > code review for https://bugs.python.org/issue17633, even if > it is simply a LGTM. I will then personally make sure the > approved patch gets checked in for Python 3.6 in case the > rewrite of zipimport misses the release.
Cool. I'll see what I can do. I was having a bit of trouble with the register/login part of the bug tracker. Which is why I came here. I'll battle with it one more time and see if I can get it to log me in.
If you really can't manage to comment in the tracker (which sounds unlikely -- many people have succeeded :-) you can post your LGTM on the specific patch here.
The patch should be fairly simple. In a nutshell it just does a:
path.replace('.', '/') + '/' in two locations. One where it checks for the path being a directory entry in the zip file and the second to return an implicit namespace path (instead of not found) if it is a match. I'll check the patch on the tracker and see if it still works with 3.5.1. If not I'll attach mine.
Well, Brett would like to see your feedback on the specific patch. Does it work for you?
-- --Guido van Rossum (python.org/~guido) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
" " == Brett Cannon <brett@python.org> writes:
> I just wanted to quickly say that Guido's observation as to how > a VFS is overkill is right. Imagine implementing a loader using > sqlite and you quickly realize that doing a dull VFS is more > than necessary to implement what import needs to function. I fear I've made a poor choice in calling this abstract class a VFS (I'm terrible with names). I'm not thinking of anything along the lines of a full file system that supports open(), seek(), read() and everything else. That for sure would be overkill and way more complicated than it needs to be. All I'm really thinking about is a simple abstract interface that is used by an importer to actually locate and retrieve the binary objects that will be loaded. For the simple case I think just two methods would/could server a read only "blob/byte database": listdir(path) # returns an iterable container of "files"/"dirs" found # at path get(path) # returns a bytes object for the given path I think with those two virtual calls a more high level import layer can locate and retrieve modules to be loaded without even knowing where they came from. The higher level would know about things such as the difference between .py and .pyc "files" or the possible existance of __pycache__ directories and what may be found in them. Right now the zipimporter contains a list of file extensions to try and load (and in what order). It also lacks any knowledge of __pycache__ directories (which is one of the outstanding bugs with it). It just seems to me that this sorta logic would be better moved to a higher layer and the zip layer just translates paths into reads of byte blobs. I mentioned write()/put() for two reasons: 1) When I import a .py file then a .pyc file is created on my filesystem. I don't really know what piece of code created it. But a write to the filesystem (assuming it is writeable and permissions set etc) occurs. It might be nice for other storage systems (zip, sql, etc) could optionally support this. They could if the code that crated the .pyc simply did a put() to the object that pulled in the .py file. The interface is expanded by two calls (put() and delete()). 2) Integration with package data. I know there are modules/packages out there that help a module try and locate data files that may be associated with a package. I think it would be kinda cool for a module to instead be able to get a handle to the abstract class that loaded it. it could then use the same listdir() get() and possibly write methods the importer did. The writing bit of this may or may not be a good idea :) Anyway, hope I did not muddy the waters. I was just thinking a bit out loud and none of this may live past my own experiments. I was/am just trying to think of why the importers like the zipimporter don't work like a filesystem importer and how they would be cleaner if they just dealt with paths and byte blobs to store/get based on those paths. Mike
On 3 January 2016 at 15:29, <mike.romberg@comcast.net> wrote:
" " == Brett Cannon <brett@python.org> writes:
> I just wanted to quickly say that Guido's observation as to how > a VFS is overkill is right. Imagine implementing a loader using > sqlite and you quickly realize that doing a dull VFS is more > than necessary to implement what import needs to function.
I fear I've made a poor choice in calling this abstract class a VFS (I'm terrible with names). I'm not thinking of anything along the lines of a full file system that supports open(), seek(), read() and everything else. That for sure would be overkill and way more complicated than it needs to be.
All I'm really thinking about is a simple abstract interface that is used by an importer to actually locate and retrieve the binary objects that will be loaded. For the simple case I think just two methods would/could server a read only "blob/byte database":
listdir(path) # returns an iterable container of "files"/"dirs" found # at path
get(path) # returns a bytes object for the given path
We already have the latter: https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoade... It's the former that has taken a while to get to, as the 3rd party pkg_resources module (part of setuptools) already provides a pragmatic API that also has the virtue of being compatible with both Python 2 & 3, and there a few subtleties related to the possible use of temporary files that make a robust API design trickier than it first appears to be. For folks that are interested in that, folks that aren't following import-sig in addition to python-dev may want to take a look at Brett's design for the importlib.resources API: http://nbviewer.jupyter.org/gist/brettcannon/9c4681a77a7fa09c5347 Cheers, Nick. P.S. If anyone actually *does* want a full "virtual file system layer" API for non-filesystem storage locations: http://docs.pyfilesystem.org/en/latest/filesystems.html -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 3 January 2016 at 16:32, Nick Coghlan <ncoghlan@gmail.com> wrote:
For folks that are interested in that, folks that aren't following import-sig in addition to python-dev may want to take a look at Brett's design for the importlib.resources API: http://nbviewer.jupyter.org/gist/brettcannon/9c4681a77a7fa09c5347
Sorry, I meant to include a link to the import-sig thread as well: https://mail.python.org/pipermail/import-sig/2015-November/001041.html Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, 2 Jan 2016 at 21:31 <mike.romberg@comcast.net> wrote:
" " == Brett Cannon <brett@python.org> writes:
> I just wanted to quickly say that Guido's observation as to how > a VFS is overkill is right. Imagine implementing a loader using > sqlite and you quickly realize that doing a dull VFS is more > than necessary to implement what import needs to function.
I fear I've made a poor choice in calling this abstract class a VFS (I'm terrible with names). I'm not thinking of anything along the lines of a full file system that supports open(), seek(), read() and everything else. That for sure would be overkill and way more complicated than it needs to be.
All I'm really thinking about is a simple abstract interface that is used by an importer to actually locate and retrieve the binary objects that will be loaded. For the simple case I think just two methods would/could server a read only "blob/byte database":
listdir(path) # returns an iterable container of "files"/"dirs" found # at path
get(path) # returns a bytes object for the given path
I think with those two virtual calls a more high level import layer can locate and retrieve modules to be loaded without even knowing where they came from.
The higher level would know about things such as the difference between .py and .pyc "files" or the possible existance of __pycache__ directories and what may be found in them. Right now the zipimporter contains a list of file extensions to try and load (and in what order). It also lacks any knowledge of __pycache__ directories (which is one of the outstanding bugs with it). It just seems to me that this sorta logic would be better moved to a higher layer and the zip layer just translates paths into reads of byte blobs.
I mentioned write()/put() for two reasons:
1) When I import a .py file then a .pyc file is created on my filesystem. I don't really know what piece of code created it. But a write to the filesystem (assuming it is writeable and permissions set etc) occurs. It might be nice for other storage systems (zip, sql, etc) could optionally support this. They could if the code that crated the .pyc simply did a put() to the object that pulled in the .py file. The interface is expanded by two calls (put() and delete()).
2) Integration with package data. I know there are modules/packages out there that help a module try and locate data files that may be associated with a package. I think it would be kinda cool for a module to instead be able to get a handle to the abstract class that loaded it. it could then use the same listdir() get() and possibly write methods the importer did. The writing bit of this may or may not be a good idea :)
So it's possible that some abstraction might be possible, but up to this point there hasn't been a motivating factor for importlib to do this as zipimport is written in C and thus won't benefit from any abstraction that importlib uses in its Python code (hence why zipimport needs to be rewritten). Maybe after we have a zip-backed importer written in Python a common API will fall through and we can abstract that out, but I wouldn't want to guess until the code is written and someone can stare at the two implementations. It should also be said that there is nothing requiring that an importer support any concept of a file. Going back to the sqlite database example, you could make it nothing more than a table that maps module name to source code and bytecode: CREATE TABLE code ( name TEXT PRIMARY KEY NOT NULL, source BLOB, bytecode BLOB ); /* A trigger that re-generates `bytecode` when `source` is set would be nice in this example, but that's beyond my SQL abilities. */ That's enough to implement an importer and there is no mention of file paths anywhere. If you really wanted to could do a second table that acted as a database-backed file system for reading package data files, but that is not required to meet the minimum functionality -- and then some thanks to bytecode support -- for an importer.
Anyway, hope I did not muddy the waters. I was just thinking a bit out loud and none of this may live past my own experiments. I was/am just trying to think of why the importers like the zipimporter don't work like a filesystem importer and how they would be cleaner if they just dealt with paths and byte blobs to store/get based on those paths.
It's a reasonable thing to consider, but it would be better to get zipimport fixed for you, then rewritten, and then look at code for the rewritten zipimport and what importlib.machinery.SourceFileLoader has to see if there is a common API to abstract out and build upon.
" " == Brett Cannon <brett@python.org> writes:
... > So it's possible that some abstraction might be possible, but > up to this point there hasn't been a motivating factor for > importlib to do this as zipimport is written in C and thus > won't benefit from any abstraction that importlib uses in its > Python code (hence why zipimport needs to be rewritten). Maybe > after we have a zip-backed importer written in Python a common > API will fall through and we can abstract that out, but I > wouldn't want to guess until the code is written and someone > can stare at the two implementations. Fair enough. I too think it is a good idea to make base classes after their is a need for them and not before. Some argument could be made that there is a need now as zipimported modules/packages don't work exactly the same way as "normal" ones. But since you plan a rewrite of zipimport collecting the common stuff after that makes sense. > It should also be said that there is nothing requiring that an > importer support any concept of a file. Going back to the > sqlite database example, you could make it nothing more than a > table that maps module name to source code and bytecode: Yep. I was not thinking file either. I may have said file (again I'm terrible with names) but I'm thinking an array of bytes with a string id (which I call a path). The actual storage could simply be storing the byte array using the stringID/path. ... > That's enough to implement an importer and there is no mention > of file paths anywhere. If you really wanted to could do a > second table that acted as a database-backed file system for > reading package data files, but that is not required to meet > the minimum functionality -- and then some thanks to bytecode > support -- for an importer. Yea the python module name could be viewed as a path (package.subpackage.module) and stored in a hierarchical way. Or it could simply be viewed as a string and stored in some flat database. Anyway, back at the ranch... I've got an extended version of the patch for issue17633 working on my system. I've added to the test case to check for the proper functioning of implicit namespace packages spread between two zip files. I still need to add tests for a namespace package spread between a normal filesystem and a zip file. I expect I'll have that done in a day or two. I'll post it to the bug tracker. Mike
" " == Brett Cannon <brett@python.org> writes:
... > It's a reasonable thing to consider, but it would be better to > get zipimport fixed for you, then rewritten To that end, I've added a patch to the issue tracker: https://bugs.python.org/issue17633 My patch is issue17633-3.diff which builds upon issue17633-2.diff in that it fixes an issue with the enumerated value used by find_loader (find_loader was returning -1 which was not even a valid enumerated value). I also expanded the test cases for zipimport to cover namespace packages spread between multiple zip archives and zip archives and real filesystems. Please let me know if there is anything else I can do. Thanks, Mike
On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
"Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
Just to point out that it would be nice to have an easier way to use something other that zipfiles. I have a need to exploit a different solution and have to patch the bootstrap code (because the zipfile support is handled as a special case). BTW the need is to create iOS and Android executables from frozen Python code. Phil
On Sun, 3 Jan 2016 at 02:55 Phil Thompson <phil@riverbankcomputing.com> wrote:
On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
> "Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the
only time the VFS would come in handy would be for zipfiles, where we already have a solution.
Just to point out that it would be nice to have an easier way to use something other that zipfiles. I have a need to exploit a different solution and have to patch the bootstrap code (because the zipfile support is handled as a special case). BTW the need is to create iOS and Android executables from frozen Python code.
Not quite sure about how zip files are a special-case beyond just being put in sys.meta_path automatically. You can get the same results with a .pth file or a sitecustomize.py depending on how pervasive your need is. Otherwise feel free to file an issue at bugs.python.org and we can talk over there about what you specifically need and if it's reasonable to try and support.
On 3 Jan 2016, at 5:33 pm, Brett Cannon <brett@python.org> wrote:
On Sun, 3 Jan 2016 at 02:55 Phil Thompson <phil@riverbankcomputing.com> wrote: On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido@python.org> wrote:
On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg@comcast.net> wrote:
--
> "Brett" == Brett Cannon <brett@python.org> writes:
> I opened > https://bugs.python.org/issue25711 to specifically try to > fix this issue once and for all and along the way modernize > zipimport by rewriting it from scratch to be more > maintainable
Every time I read about impementing a custom loader:
https://docs.python.org/3/library/importlib.html
I've wondered why python does not have some sort of virtual filesystem layer to deal with locating modules/packages/support files. Virtual file systems seem like a good way to store data on a wide range of storage devices.
Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
Just to point out that it would be nice to have an easier way to use something other that zipfiles. I have a need to exploit a different solution and have to patch the bootstrap code (because the zipfile support is handled as a special case). BTW the need is to create iOS and Android executables from frozen Python code.
Not quite sure about how zip files are a special-case beyond just being put in sys.meta_path automatically. You can get the same results with a .pth file or a sitecustomize.py depending on how pervasive your need is. Otherwise feel free to file an issue at bugs.python.org and we can talk over there about what you specifically need and if it's reasonable to try and support.
I've created http://bugs.python.org/issue26007 and hope it's clear enough what the issue is. Thanks, Phil
participants (5)
-
Brett Cannon -
Guido van Rossum -
mike.romberg＠comcast.net -
Nick Coghlan -
Phil Thompson