I'd like to take this moment to note how awesome importing from zipfile is. Did you know you can execute a zip file with Python, and if it has a __main__.py in the root it will be executed, with the zipfile's contents on the PYTHONPATH? file.zip: __main__.py somedep.py other/code.py python file.zip # runs!
On Sat, Sep 15, 2012 at 8:19 PM, Daniel Holth <dholth@gmail.com> wrote:
I'd like to take this moment to note how awesome importing from zipfile is. Did you know you can execute a zip file with Python, and if it has a __main__.py in the root it will be executed, with the zipfile's contents on the PYTHONPATH?
I've known this, but can you say a little about the caveats (e.g. why you wouldn't always want to store your packages as zip files)? For example, IIRC you can't really use __file__ from within such modules. --Chris
Of course the major potentially non obvious caveats are that a lot of python code doesn't use the API to load data from the importer path, instead using __file__ and assuming everything is one, and that sometimes (shared libraries) the system can't load it unless its a real file. Daniel Holth On Sep 15, 2012, at 11:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
On Sat, Sep 15, 2012 at 8:19 PM, Daniel Holth <dholth@gmail.com> wrote:
I'd like to take this moment to note how awesome importing from zipfile is. Did you know you can execute a zip file with Python, and if it has a __main__.py in the root it will be executed, with the zipfile's contents on the PYTHONPATH?
I've known this, but can you say a little about the caveats (e.g. why you wouldn't always want to store your packages as zip files)? For example, IIRC you can't really use __file__ from within such modules.
--Chris
On Sun, 2012-09-16 at 08:07 -0400, Daniel Holth wrote:
Of course the major potentially non obvious caveats are that a lot of python code doesn't use the API to load data from the importer path, instead using __file__ and assuming everything is one, and that sometimes (shared libraries) the system can't load it unless its a real file.
This is easy to work around in brand new code using pkg_resources, so it's generally only a problem with older code, or code which can depend only on the stdlib. Unfortunately, the stdlib doesn't have much to help here. There's pkgutil.get_data but it doesn't really work well if you need to get a filelike handle to a resource, or you need to list the contents of a directory, and other things which pkg_resources does very nicely, so it seems like Python isn't really serious about allowing code in zipfiles to be a viable alternative to file-and-directory-based code. - C
This is easy to work around in brand new code using pkg_resources, so it's generally only a problem with older code, or code which can depend only on the stdlib. Unfortunately, the stdlib doesn't have much to help here. There's pkgutil.get_data but it doesn't really work well if you need to get a filelike handle to a resource, or you need to list the contents of a directory, and other things which pkg_resources does very nicely, so it seems like Python isn't really serious about allowing code in zipfiles to be a viable alternative to file-and-directory-based code.
I got the impression that killing zip imports is an explicit goal of packaging. This is sure to be a contentious issue. To all the haters, let me say that no one is going to make you import code from a zip if you don't want to. That easy_install mistake will not be repeated. However, those of us who like importing from places other than the filesystem should be able to enjoy continually improving support for doing so. Also, remember that thing about the debugger not showing the source code for zipped code? Fixed.
On 16 September 2012 18:41, Daniel Holth <dholth@gmail.com> wrote:
This is easy to work around in brand new code using pkg_resources, so it's generally only a problem with older code, or code which can depend only on the stdlib. Unfortunately, the stdlib doesn't have much to help here. There's pkgutil.get_data but it doesn't really work well if you need to get a filelike handle to a resource, or you need to list the contents of a directory, and other things which pkg_resources does very nicely, so it seems like Python isn't really serious about allowing code in zipfiles to be a viable alternative to file-and-directory-based code.
I got the impression that killing zip imports is an explicit goal of packaging. This is sure to be a contentious issue.
I think packaging possibly only considered installing *to* filesystem locations. But that's no different from distutils - at worst, packaging explicitly ignored the setuptools features around creating eggs (importable zip formats). It's not that you can't do it, just that packaging didn't want to get sucked into that area and so you're on your own. Python *itself* very definitely supports zip imports. And with the new implementation of the core import machinery using importlib in Python 3.3, it goes a lot further, supporting imports from any locations you care to write the code for. There's no core support for putting your package into (say) a sqlite database, but once it's in there, the core supports you adding hooks in so you can put that sqlite database on sys.path and have it "just work". That's a huge amount of flexibility, and no-one has even scratched the surface of it yet. Paul.
On Sep 16, 2012, at 1:23 PM, Chris McDonough wrote:
On Sun, 2012-09-16 at 08:07 -0400, Daniel Holth wrote:
Of course the major potentially non obvious caveats are that a lot of python code doesn't use the API to load data from the importer path, instead using __file__ and assuming everything is one, and that sometimes (shared libraries) the system can't load it unless its a real file.
This is easy to work around in brand new code using pkg_resources, so it's generally only a problem with older code, or code which can depend only on the stdlib. Unfortunately, the stdlib doesn't have much to help here. There's pkgutil.get_data but it doesn't really work well if you need to get a filelike handle to a resource, or you need to list the contents of a directory, and other things which pkg_resources does very nicely, so it seems like Python isn't really serious about allowing code in zipfiles to be a viable alternative to file-and-directory-based code.
Remind me again why pkg_resources (or an API exactly like it) isn't in the stdlib already? -Barry
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil. I am really impressed with the file after seeing how easy it was to add PEP-345 and PEP-376 support to pkg_resources.py without having to present a different API to the user, and seeing how it provides a lot of what an installer needs especially being able to iterate over all the distributions in a particular folder, but I am glad it is not in the stdlib because I wouldn't have been able to do that if it was.
On Sun, 2012-09-16 at 15:33 -0400, Daniel Holth wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
I think maybe there's also a latent notion floating around that "a resource (aka non-Python files) shouldn't be addressable within Python packages" aka. "Python should only make it easy to address Python files in Python packages, but no other kind of file". I'm not sure I've ever seen any of the core dev team express this opinion, but I've seen it floating around elsewhere. - C
On Sun, Sep 16, 2012 at 3:40 PM, Chris McDonough <chrism@plope.com> wrote:
On Sun, 2012-09-16 at 15:33 -0400, Daniel Holth wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
I think maybe there's also a latent notion floating around that "a resource (aka non-Python files) shouldn't be addressable within Python packages" aka. "Python should only make it easy to address Python files in Python packages, but no other kind of file". I'm not sure I've ever seen any of the core dev team express this opinion, but I've seen it floating around elsewhere.
I think we should talk about something else. Let's talk about how to replace the stdlib's implementation of the packaging API with another implementation, like import hooks. How would that work? Should we define a strict API/implementation separation by using the amazing apipkg, and then let Python choose the actual implementation based on an environment variable/site.py/.pth file with 'import; do something'; or perhaps a configure.zcml in site-packages with zope.interface?
On Sep 16, 2012, at 03:40 PM, Chris McDonough wrote:
On Sun, 2012-09-16 at 15:33 -0400, Daniel Holth wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
I think maybe there's also a latent notion floating around that "a resource (aka non-Python files) shouldn't be addressable within Python packages" aka. "Python should only make it easy to address Python files in Python packages, but no other kind of file". I'm not sure I've ever seen any of the core dev team express this opinion, but I've seen it floating around elsewhere.
Yeah. Personally, I think that's just silly. It's way, way, way too convenient to put some data files in your packages, and with pkg_resources, there's actually a principled way to do that consistently regardless of whether you're running from source or installed. I'm not even sure *not* putting your data files in your packages has any kind of common systematic support in Python. OTOH, it's definitely true that data files that people have to edit (e.g. configuration files) shouldn't live in your packages. That's just an invitation to violate all kinds of standards (he says, painfully aware of Mailman 2's mm_cfg.py file ;). -Barry
On Mon, Sep 17, 2012 at 10:38 AM, Barry Warsaw <barry@python.org> wrote:
On Sep 16, 2012, at 03:40 PM, Chris McDonough wrote:
On Sun, 2012-09-16 at 15:33 -0400, Daniel Holth wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
I think maybe there's also a latent notion floating around that "a resource (aka non-Python files) shouldn't be addressable within Python packages" aka. "Python should only make it easy to address Python files in Python packages, but no other kind of file". I'm not sure I've ever seen any of the core dev team express this opinion, but I've seen it floating around elsewhere.
Yeah. Personally, I think that's just silly. It's way, way, way too convenient to put some data files in your packages, and with pkg_resources, there's actually a principled way to do that consistently regardless of whether you're running from source or installed. I'm not even sure *not* putting your data files in your packages has any kind of common systematic support in Python.
OTOH, it's definitely true that data files that people have to edit (e.g. configuration files) shouldn't live in your packages. That's just an invitation to violate all kinds of standards (he says, painfully aware of Mailman 2's mm_cfg.py file ;).
-Barry
I don't think pkg_resources's much better version of pkgutil.get_data is seriously under dispute. What's really missing in action is a better api than zipimport._zip_directory_cache to list everything on the importer path, not just the modules. distutils2 does try to add some support for 'data files not on sys.path' Bento will write, on request only, a Python file with the root paths used during installation. You get to choose the name. Something like: yourpackage/__config.py: SCRIPTS='/path/to/scripts' I think this is a better idea than .dist-info/RESOURCES being a file with original-path -> installed-path for every file in the distribution.
On Sep 16, 2012, at 03:33 PM, Daniel Holth wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
Right, which is why I passively-aggressively suggested an API like it. :)
I am really impressed with the file after seeing how easy it was to add PEP-345 and PEP-376 support to pkg_resources.py without having to present a different API to the user, and seeing how it provides a lot of what an installer needs especially being able to iterate over all the distributions in a particular folder, but I am glad it is not in the stdlib because I wouldn't have been able to do that if it was.
Interesting, but of course, this is the same with any module in the stdlib. That's why it's (somewhat humorously) said the stdlib is where good modules go to die. ;) -Barry
On Sun, Sep 16, 2012 at 3:33 PM, Daniel Holth <dholth@gmail.com> wrote:
It is because the file has a controversial coding style, with up to 25 lines of whitespace between classes, and includes functionality for eggs and some other stuff that was unwanted on top of the resources stuff that partly made its way into pkgutil.
I don't see why that couldn't be fixed though. As it is, almost every time I make a patch to distribute I clean things up a bit, in part because I have vim configured by default to run autopep8 and some other scripts on every Python file I open so I don't have to look at *quite* as much ugly code (this has proven very useful at work).
I am really impressed with the file after seeing how easy it was to add PEP-345 and PEP-376 support to pkg_resources.py without having to present a different API to the user, and seeing how it provides a lot of what an installer needs especially being able to iterate over all the distributions in a particular folder, but I am glad it is not in the stdlib because I wouldn't have been able to do that if it was.
Yes--I frequently have found myself wishing pkg_resources' availability could be assumed. Especially for getting handles to non-code resources, and for things like get_distribution(). Erik
participants (6)
-
Barry Warsaw -
Chris Jerdonek -
Chris McDonough -
Daniel Holth -
Erik Bray -
Paul Moore