As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata. The current implementation, and documentation, is here: http://undefined.org/python/pkgdata-0.1.tgz """ pkgdata is a simple, extensible way for a package to acquire data file resources. The implementation is lightweight, and is intended to be included *inside* your Python packages. """ It's based on PyProtocols (but the implementation doesn't require it unless you want to override the default mechanism). Basically, it's useful for py2exe and OS X bundlebuilder type situations. It's a simple and flexible alternative to os.path.join(os.path.dirname(__file__), "myresource"). The source is simple, and has lots of examples. Read if you're interested. If you would like to see additional examples of its use, here are the interesting things to look at if you check out pygame CVS: pygame/lib/pkgdata.py (this is the pkgdata "client") pygame/lib/macosx.py (usage from python) pygame/src/font.c (usage from C) pygame/src/display.c (more usage from C) pygame/examples/macosx/aliens_app_example/ (this has the pkgdata "host adapter" for bundlebuilder type situations) The fact that it uses PyProtocols is just personal preference. It could work on top of any sort of global registry, but PyProtocols is a really nice global registry that made my implementation extremely terse. -bob
At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata.
Do you plan to support the PEP 302 'get_data()' facility?
On Mar 25, 2004, at 11:28 AM, Phillip J. Eby wrote:
At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata.
Do you plan to support the PEP 302 'get_data()' facility?
Probably not, I don't think anyone ever implemented it, and it seems more complicated than it needs to be. If someone really does use it, and thinks it should be implemented, please speak up. -bob
At 01:28 PM 3/25/04 -0500, Bob Ippolito wrote:
On Mar 25, 2004, at 11:28 AM, Phillip J. Eby wrote:
At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata.
Do you plan to support the PEP 302 'get_data()' facility?
Probably not, I don't think anyone ever implemented it, and it seems more complicated than it needs to be. If someone really does use it, and thinks it should be implemented, please speak up.
Perhaps I was unclear. What I meant was closer to, "why not expose this functionality as methods on a __loader__ attribute of the module, and update PEP 302 if needed?" I'd love there to be a standard facility in Python for doing this, and have it be supported for import-from-zip, py2exe, and all the other possible routes to importing a module. As far as I'm aware, PEP 302 is the only PEP that even vaguely touches on this, so it seems to make sense to me to update it. As you probably know, PEAK has 'config.fileNearModule()' to do this, but it doesn't have any mechanisms for dealing with non-file imports. I intend to try to implement the PEP 302 route once I get around to needing it. (Although, in all honesty I'm unlikely to embrace zipfile imports until there's a way to use C extensions in them, at which point it becomes a cool way to do simple binary installs of large packages.)
On Mar 25, 2004, at 2:37 PM, Phillip J. Eby wrote:
At 01:28 PM 3/25/04 -0500, Bob Ippolito wrote:
On Mar 25, 2004, at 11:28 AM, Phillip J. Eby wrote:
At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata.
Do you plan to support the PEP 302 'get_data()' facility?
Probably not, I don't think anyone ever implemented it, and it seems more complicated than it needs to be. If someone really does use it, and thinks it should be implemented, please speak up.
Perhaps I was unclear. What I meant was closer to, "why not expose this functionality as methods on a __loader__ attribute of the module, and update PEP 302 if needed?"
Because import hooks are a pain, and this is simple. pkgdata also, intentionally, doesn't need to be standardized in Python. It uses protocolForURI and can (and should!) actually live inside every package that you use it from.
I'd love there to be a standard facility in Python for doing this, and have it be supported for import-from-zip, py2exe, and all the other possible routes to importing a module. As far as I'm aware, PEP 302 is the only PEP that even vaguely touches on this, so it seems to make sense to me to update it.
That is a bit of a different problem. pkgdata solves the easy problem. If everyone starts using it, it will at least be possible to solve the hard problem, where the hard problem is intractable if all code keeps on using the "pragmatic" os.path.join(os.path.basedir(__file__), 'myfile') garbage.
As you probably know, PEAK has 'config.fileNearModule()' to do this, but it doesn't have any mechanisms for dealing with non-file imports. I intend to try to implement the PEP 302 route once I get around to needing it. (Although, in all honesty I'm unlikely to embrace zipfile imports until there's a way to use C extensions in them, at which point it becomes a cool way to do simple binary installs of large packages.)
The way we do C extensions in a zip file on the Mac is to put a python shim in place of the extension, and load the extension from somewhere else. If you were to put the actual extension in the zip file, you would have to extract it to disk somewhere anyway because most dynamic linkers won't load objects that aren't files (although it seems like Mac OS X 10.4 will support this). If you're particularly interested, the extension shim file template is this: def __load(): import imp, sys, os for p in sys.path: path = os.path.join(p, "%(filename)s") if os.path.exists(path): break else: assert 0, "file not found: %(filename)s" mod = imp.load_dynamic("%(name)s", path) __load() del __load -bob
"Phillip J. Eby"
At 01:28 PM 3/25/04 -0500, Bob Ippolito wrote:
On Mar 25, 2004, at 11:28 AM, Phillip J. Eby wrote:
At 11:11 AM 3/25/04 -0500, Bob Ippolito wrote:
As per the BOF last night, here is the pkgdata module that I mentioned. Currently the only project using it is pygame. The "sys.datapath" mechanism that I talked about *could actually be implemented* on top of this system, supposing an appropriate adapter was written to support it! Remember: pkgdata is *just a hook*. It doesn't do anything other give the eventual option of doing something smarter than os.path.join(os.path.basedir(__file__), data). When you figure out what something smarter is, or you change your mind about what the smarter solution is (inside of an app bundle vs. system wide install, etc.), you don't have to refactor any of the code that uses pkgdata.
Do you plan to support the PEP 302 'get_data()' facility?
Probably not, I don't think anyone ever implemented it, and it seems more complicated than it needs to be. If someone really does use it, and thinks it should be implemented, please speak up.
Perhaps I was unclear. What I meant was closer to, "why not expose this functionality as methods on a __loader__ attribute of the module, and update PEP 302 if needed?"
I'd love there to be a standard facility in Python for doing this, and have it be supported for import-from-zip, py2exe, and all the other possible routes to importing a module. As far as I'm aware, PEP 302 is the only PEP that even vaguely touches on this, so it seems to make sense to me to update it.
I understand what you mean, and had exactly the same idea. I don't yet understand the code in Bob's package, probably because I never looked at PEAK any more than 5 or 10 minutes. What I understand (and that may be misunderstanding) is that it's a more or less abstract api to implement this functionality, and extend it later. I had a probably related idea, to implement a function like 'is_frozen' or so, which would return whether a script is executed as standard python script, or run frozen in an executable (py2exe, Installer, freeze, cx_freeze). This function should hide the exact details how this is determined, and be universally available. This way modules needing to know this (for example the COM registration code in win32all^H^H^H^H^H^H^H^Hpywin32) could use this function to customize its behaviour, with relying on internal details of py2exe or Installer.
As you probably know, PEAK has 'config.fileNearModule()' to do this, but it doesn't have any mechanisms for dealing with non-file imports. I intend to try to implement the PEP 302 route once I get around to needing it. (Although, in all honesty I'm unlikely to embrace zipfile imports until there's a way to use C extensions in them, at which point it becomes a cool way to do simple binary installs of large packages.)
py2exe creates a pure Python loader for each extension module, which then dynamically loads it from the file system, without the need for the directory of the .pyds to be in sys.path. I don't think there is another way to load dlls on windows (except when you unpack them from the archive into the file system, and remove them later again - which I consider extremly ugly). Thomas
At 08:58 PM 3/25/04 +0100, Thomas Heller wrote:
I had a probably related idea, to implement a function like 'is_frozen' or so, which would return whether a script is executed as standard python script, or run frozen in an executable (py2exe, Installer, freeze, cx_freeze). This function should hide the exact details how this is determined, and be universally available. This way modules needing to know this (for example the COM registration code in win32all^H^H^H^H^H^H^H^Hpywin32) could use this function to customize its behaviour, with relying on internal details of py2exe or Installer.
What I'm suggesting is that PEP 302 already provides a natural place for this to exist: the __loader__ method of a module. The module itself, or other modules, can then access loader facilities via that object if it exists, or use a fallback if not. And, if PEP 302 is ever "finished" (in the sense that the builtin Python import mechanism becomes part of the sys.metapath, then in theory we should also see a '__loader__' object available for normal modules. Indeed, the only thing stopping this from happening, I believe, is that nobody's implemented it. There's still probably a window in which it could be done for 2.4, and the resulting standardization would be a blessing, especially if coupled with "package data" installation support in distutils.
participants (3)
-
Bob Ippolito
-
Phillip J. Eby
-
Thomas Heller