distlib updated to include "entry point" functionality
I've now updated distlib to include equivalent functionality for pkg_resources' entry points. There is no ability to install packages on-demand via "require()" functionality, but distributions are able to register things they implement, and the capability to query what's available is also present. The relevant information is stored in a dist-info file REGISTRY which has the same format as entry_points.txt files. There is no code at present to write these files; that will come later when focusing on what installer software will need from distlib. The distlib.depgraph code has now been merged into distlib.database, as indicated in an earlier post. I've also added a section on migrating from older APIs. This currently covers pkg_resources' resource and entry point APIs. The resources API now covers extraction of files from .zip files and caching of the extracted files in the file system. I noticed that pkg_resources has a function cleanup_resources() to clear the cache, but this is a no-op. Can someone tell me why this is? Regards, Vinay Sajip
On Mon, Oct 8, 2012 at 2:05 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I've now updated distlib to include equivalent functionality for pkg_resources' entry points. There is no ability to install packages on-demand via "require()" functionality, but distributions are able to register things they implement, and the capability to query what's available is also present. The relevant information is stored in a dist-info file REGISTRY which has the same format as entry_points.txt files.
Given that it's the same format, is there any reason to change the filename? Changing the filename just means more code in Daniel's wheel support code, and REGISTRY is an awfully vague name. Registry of what? ;-)
I noticed that pkg_resources has a function cleanup_resources() to clear the cache, but this is a no-op. Can someone tell me why this is?
When Bob Ippolito and I first designed that system, we thought there'd be a need to use temporary directories for extraction. Apparently, there never was. ;-)
PJ Eby <pje <at> telecommunity.com> writes:
Given that it's the same format, is there any reason to change the filename? Changing the filename just means more code in Daniel's wheel support code,
I'm not too hung up about the filename, and only chose REGISTRY because it is more aligned with the other filenames which appear in .dist-info directories. Since distlib is still in its very early days, I expect the filename can be changed without too much of an impact; the idea is to hide that sort of implementation detail behind an API.
and REGISTRY is an awfully vague name. Registry of what?
Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"
When Bob Ippolito and I first designed that system, we thought there'd be a need to use temporary directories for extraction. Apparently, there never was.
Thanks for the info, and the feedback. Regards, Vinay Sajip
On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"
Right, but isn't that then an entry point? I agree REGISTRY is not a very good name in any case.
Lennart Regebro <regebro <at> gmail.com> writes:
IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"
Right, but isn't that then an entry point?
How is it an entry point? Perhaps I'm misunderstanding what an entry point is, but to my mind it's the start of some code. Of course a module runs code when imported, but the module may already have been imported when the entry point is loaded - in which case, nothing would get run at load time, IIUC. Even if it wasn't imported before, I'm not sure whether you'd want to import a module just for side-effects caused by the import.
I agree REGISTRY is not a very good name in any case.
Like I said, I only chose it because it's similar to other filenames in .dist-info, and I'll entertain alternative bikeshed colours ;-) I've added read_registry and write_registry methods to Distribution which take an optional filename; this allows reading from alternative filenames, which is handy for testing, at least. Regards, Vinay Sajip
On Mon, Oct 8, 2012 at 3:59 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"
Right, but isn't that then an entry point?
I agree REGISTRY is not a very good name in any case.
I would prefer to keep the filename the same as long as the format is the same. (Not a suggestion to change the format.) bdist_wheel is easy to edit but pkg_resources.py is not because it has a long release cycle and if you make a mistake you will break almost every Python installation. Right now a new class attribute Distribution.PKG_INFO keeps track of whether the PEP-defined Metadata 1.0-1.3 is called PKG-INFO or METADATA. To support REGISTRY in pkg_resources you might do something similar, changing line ~2319 self._get_metadata('entry_points.txt'), self to self._get_metadata(self.ENTRY_POINTS_FILENAME), self or for name in (possible, entry, points, filenames): ... defining ENTRY_POINTS_FILENAME for both Distribution and DistInfoDistribution, and you would have to wait for a new distribute release. In other words, it is a hassle and the feature is the same. An entry_point is always an entry_point. It should no sooner be confused with an English "point of entry" than keywords like "print" should be confused with pressing books. Daniel
Daniel Holth <dholth <at> gmail.com> writes:
I would prefer to keep the filename the same as long as the format is the same.
See my comments about read_registry and write_registry methods in Distribution (in response to Lennart's post).
In other words, it is a hassle and the feature is the same. An entry_point is always an entry_point. It should no sooner be confused with an English "point of entry" than keywords like "print" should be confused with pressing books.
What do *you* mean by entry point? My understanding is that it represents the start of some executable code. Wikipedia: "In computer programming, an entry point is a memory address, corresponding to a point in the code of a computer program which is intended as the destination of a long jump, be it internal or external." So, in the Python sphere, I would interpret an entry point to be a callable. While that might be the most common usage of the feature, it's not the only one. For example, you could use name = value entries where value is not a callable, but for example a name of a resource to use for something, or a configuration option, or the name of a package or a module (which is not an entry point by my understanding of entry points being callables). Regards, Vinay Sajip
On Mon, Oct 8, 2012 at 5:14 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
What do *you* mean by entry point? My understanding is that it represents the start of some executable code. Wikipedia:
"In computer programming, an entry point is a memory address, corresponding to a point in the code of a computer program which is intended as the destination of a long jump, be it internal or external."
So, in the Python sphere, I would interpret an entry point to be a callable. While that might be the most common usage of the feature, it's not the only one. For example, you could use name = value entries where value is not a callable, but for example a name of a resource to use for something, or a configuration option, or the name of a package or a module (which is not an entry point by my understanding of entry points being callables).
In pkg_resources, entry points are references to any importable object. They don't have to be callables. Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard. ;-) In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.) If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import. I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times. That's why I baked those features into entry points from the beginning - I was battling what was at the time an excruciating slow startup for the Chandler application platform, and I borrowed the architectural pattern of delayed loading of plugins from Eclipse. (I also remember that Ian Bicking was talking about some problem, maybe on the Web-SIG, and I remember that the problem he was talking about could be solved by something like entry points, at which point I then had a solution for both Ian's problem and Chandler's problem.)
PJ Eby <pje <at> telecommunity.com> writes:
In pkg_resources, entry points are references to any importable object. They don't have to be callables.
Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard.
Yes, I acknowledged this in my response to Lennart's post.
In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.)
Right, they're just objects of one sort or another.
If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import.
As you say, entry points don't have to be actual entry points, and I didn't want to slavishly follow pkg_resources nomenclature; though it reflects the most common use case, it's not the only one. Your suggestion of "exports" seems very reasonable as a replacement for "registry", which I agree is too generic.
I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times.
I understand, and appreciate your pointing these things out. As I see it, the specific way that some of those features were implemented is what some people object to. In distlib, importing is still lazy, but sys.path is not updated before attempting imports. Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib, whereas distlib hopes to provide (some of) the useful features provided in those packages in a (hopefully) non-contentious way, with perhaps part of it going into the stdlib at some future date if there's a consensus about that. ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path. Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned? Regards, Vinay Sajip
On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib,
The two aren't 100% connected, even assuming the antecedent were true. Originally, Guido Pronounced that they *were* to be added to the stdlib (for Python 2.5), and I withdrew them due largely to an objection about the lack of third-party expertise in the underlying formats -- an expertise that is now considerably more widespread, especially thanks to you and Daniel. ;-) More to the point, though, if it's not possible for someone to create a wrapper to reimplement these portions of the pkg_resources API, then it's going to be harder to get people to port their code. The challenge isn't necessarily that there are dozens of platforms that rely on this feature explicitly; the challenge is that code which uses pkg_resources APIs for these things is *intentionally ignorant* of whether the feature is being used. The entire point of the API is to free the caller from worrying about the callee's needs, and the callee's dependencies can be transparently met by distributions that are not directly on sys.path, at least in the .egg or .wheel model of things. (If .wheel files are importable, anyway.)
ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path.
Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the extras required by an entry point? (i.e., is it possible for someone to make a common wrapper that re-adds sys.path management?)
Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned?
Now that 3.3 includes a faster import mechanism, probably so. Under older Pythons, adding a large number of directories to sys.path could badly affect startup performance, if plugins were installed as directories instead of as zipfiles. (As a general note, in case you aren't aware: the entire .egg format was invented originally as a distribution+use format for application plugins; Bob Ippolito suggested that in the general case, any Python library could also be considered a "plugin" to Python itself, and then things kind of took off from there...)
PJ Eby <pje <at> telecommunity.com> writes:
More to the point, though, if it's not possible for someone to create a wrapper to reimplement these portions of the pkg_resources API, then it's going to be harder to get people to port their code.
I will give some thought to migration concerns. I'm not sure exactly what the best approach is, or if identical semantics are achievable in all areas.
of whether the feature is being used. The entire point of the API is to free the caller from worrying about the callee's needs, and the callee's dependencies can be transparently met by distributions that are not directly on sys.path, at least in the .egg or .wheel model of things. (If .wheel files are importable, anyway.)
I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer.
Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the
At the moment, not for a completely arbitrary distribution, as we don't have the metadata in non-installed distributions. Of course, an arbitrary distribution could be unpacked to a temporary location and have the equivalent of egg-info run to get the information; but for this to work ideally, such information would need to be present in the archive metadata and also need to be in the index. There are a number of ways this could happen - it's not clear to me what the best way is, but I probably haven't thought about it enough.
extras required by an entry point? (i.e., is it possible for someone to make a common wrapper that re-adds sys.path management?)
I think it's too early to say.
Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned?
Now that 3.3 includes a faster import mechanism, probably so. Under older Pythons, adding a large number of directories to sys.path could badly affect startup performance, if plugins were installed as directories instead of as zipfiles.
I assume from the context above that that you actually mean "no" as the answer to the question "is it insufficient?", whereas I would take "probably so" to mean "yes, it's insufficient". Regards, Vinay Sajip
On Tue, Oct 9, 2012 at 7:28 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer.
Except that installation doesn't always equal placement on a sys.path. Application platforms can and do distinguish between a plugin that is "installed" and one that is "activated" or "enabled". Again, the entire egg infrastructure was originally designed to support pluggable applications, which is part of why it's so annoying to Linux packagers at times. (In roughly the same way as Firefox is annoying to them for having .xpi's.) I'm not saying distlib must support all these plugin usecases *itself*, but if it solves its chosen usecases in a way that can't be *adapted by others* to handle the app platform use cases, then there's not going to be an appealing alternative to sticking with pkg_resources or its derivatives.
I assume from the context above that that you actually mean "no" as the answer to the question "is it insufficient?", whereas I would take "probably so" to mean "yes, it's insufficient".
Oops - I misread that as "Do you feel that this is sufficient?". Sorry.
PJ Eby <pje <at> telecommunity.com> writes:
I'm not saying distlib must support all these plugin usecases *itself*, but if it solves its chosen usecases in a way that can't be *adapted by others* to handle the app platform use cases, then there's not going to be an appealing alternative to sticking with pkg_resources or its derivatives.
Well, distlib is pretty new on the block, and it'd be interesting to get feedback from others who were trying to build on it to meet other use cases. I can't see why it couldn't be built on, but if someone came up with specific sticking points, ideally based on wanting to achieve some specific outcomes, then perhaps those points could be addressed. Since we don't even have a 0.1 release yet, there's room for discussion. Regards, Vinay Sajip
On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
PJ Eby <pje <at> telecommunity.com> writes:
In pkg_resources, entry points are references to any importable object. They don't have to be callables.
Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard.
Yes, I acknowledged this in my response to Lennart's post.
In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.)
Right, they're just objects of one sort or another.
If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import.
As you say, entry points don't have to be actual entry points, and I didn't want to slavishly follow pkg_resources nomenclature; though it reflects the most common use case, it's not the only one. Your suggestion of "exports" seems very reasonable as a replacement for "registry", which I agree is too generic.
I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times.
I understand, and appreciate your pointing these things out. As I see it, the specific way that some of those features were implemented is what some people object to. In distlib, importing is still lazy, but sys.path is not updated before attempting imports. Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib, whereas distlib hopes to provide (some of) the useful features provided in those packages in a (hopefully) non-contentious way, with perhaps part of it going into the stdlib at some future date if there's a consensus about that.
ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path. Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned?
As far as my performance concerns are concerned, I just want to be able to avoid iterating through all the installed distributions twice for code that is using both APIs, which besides the porting hassle is why I might implement one API on top of the other implementation. I would also like to be able to store my installed package database in sqlite3 by implementing an appropriate distlib/pkg_resources backend and defining a standard post-(un)install "index this new package" hook but I doubt I will have the (unfunded) time. I'm sure sys.path manipulation could be implemented on top of a basic Distributions-from-path API to search for the entry points, without having to be integrated with it. FYI Ruby's gem seems to do this with the equivalent of a Python import hook (our sys.meta_path) and they do index the available gems in an actual not-just-a-bunch-of-files database. There is an interesting but long discussion at http://python.6.n6.nabble.com/Re-Python-Dev-PEP-365-Adding-the-pkg-resources... GvR starts off by wishing it wasn't 1700 lines long. It seems "easy to review" was a more important factor than the specific name of any particular feature, except for "easy_install" which was seen as too much of a setuptools brand name. Vinay wrote: I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer.
Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the
At the moment, not for a completely arbitrary distribution, as we don't have the metadata in non-installed distributions. Of course, an arbitrary distribution could be unpacked to a temporary location and have the equivalent of egg-info run to get the information; but for this to work ideally, such information would need to be present in the archive metadata and also need to be in the index. There are a number of ways this could happen - it's not clear to me what the best way is, but I probably haven't thought about it enough.
At this point of the runtime, the "arbitrary non-installed distribution" is just a dist that has been installed outside of sys.path. So the static metadata is there. Wheels are designed to be valid sys.path entries in the simple case. We don't encourage you to do so mostly because we don't want to include .pyc files for all the Pythons you might use, but it is handy for bootstrapping. python wheel-0.12.0-py2.py3-none-any.whl/wheel install wheel-0.12.0-py2.py3-none-any.whl installs wheel's own wheel with itself, even when pkg_resources.py is not installed. (You could do the same thing with PYTHONPATH=... python -m wheel install). You could imagine including with Python a similar bootstrap installer as a runnable .zip file (invent an extension .pya / .pyp?). Since the functionality is hidden in the .zip file, you would only have to commit to and argue about the command-line API instead of the much larger package management API. The bootstrap installer would install the real installer or build systems such as distutils, setuptools, Bento, distlib, providing that those projects were willing to publish themselves to pypi as pre-built wheel archives. If you want to play around with renaming the egg-info files, check out Bento's egg_utils.py line 106: https://github.com/cournape/Bento/blob/master/bento/commands/egg_utils.py#L1... which controls what each metadata file is named. file_table = { "pkg_info": "PKG-INFO", "sources": "SOURCES.txt", "install_requires": "requires.txt", "top_levels": "top_level.txt", "not_zip_safe": "not-zip-safe", "dependency_links": "dependency_links.txt", "entry_points": "entry_points.txt", "ipkg_info": "ipkg.info", }
On Tue, Oct 9, 2012 at 7:53 PM, Daniel Holth <dholth@gmail.com> wrote:
would also like to be able to store my installed package database in sqlite3 by implementing an appropriate distlib/pkg_resources backend and defining a standard post-(un)install "index this new package" hook but I doubt I will have the (unfunded) time.
How would a database even work in that scenario (given multiple independent directories on sys.path), and why would you expect it to perform better than a modern filesystem?
There is an interesting but long discussion at http://python.6.n6.nabble.com/Re-Python-Dev-PEP-365-Adding-the-pkg-resources...
GvR starts off by wishing it wasn't 1700 lines long. It seems "easy to review" was a more important factor than the specific name of any particular feature, except for "easy_install" which was seen as too much of a setuptools brand name.
He also said egg support in the stdlib was controversial, in part because there were so many individual things that would be being blessed. However, there are now PEPs covering a lot of those things, like versioning standards, metadata, etc. In any case, I doubt that distlib would be a problem just because it can be *extended* to support eggs. In fact, he was fine with PEP 365 supporting *installing* setuptools, just not *including* setuptools.
Wheels are designed to be valid sys.path entries in the simple case. We don't encourage you to do so mostly because we don't want to include .pyc files for all the Pythons you might use,
Wait... isn't wheel a binary format? Oh, I see what you mean, wheels are a *fat* binary format, so they can include multiple Python versions and multiple platforms. However, for the application platform plugins use case, wheels can potentially be quite awesome, because you can build one fat wheel for all your supported platforms. "We don't want to inlcude .pyc files for all the Pythons you might use" doesn't apply to the "we" who already build and release different platform-specific eggs to cover multiple platforms and Python versions. ;-)
On 10 October 2012 23:15, PJ Eby <pje@telecommunity.com> wrote:
However, for the application platform plugins use case, wheels can potentially be quite awesome, because you can build one fat wheel for all your supported platforms. "We don't want to inlcude .pyc files for all the Pythons you might use" doesn't apply to the "we" who already build and release different platform-specific eggs to cover multiple platforms and Python versions. ;-)
I really don't understand the whole idea of including .pyc files in binary distributions - all they do is *reduce* the flexibility of the distribution by tying it to a single Python version. Far better to just ship the .py files and let the system compile them on demand (or if you absolutely insist, compile them at install time). Note - wheels are a binary *distribution* format, and as such aren't intended to be placed on sys.path directly. The fact that they can be is not important, that's just a side effect of using a zipfile with a simple internal layout. I concede that things are different for eggs which were intended to be an importable format (AIUI, more so than a distribution format, even). Of course, "fat" wheels containing multiple versions of *compiled* code (maybe for different architectures, or API versions) are conceivable - although I don't think the current internal layout of the wheel format supports that. Paul.
On Wed, Oct 10, 2012 at 6:15 PM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Oct 9, 2012 at 7:53 PM, Daniel Holth <dholth@gmail.com> wrote:
would also like to be able to store my installed package database in sqlite3 by implementing an appropriate distlib/pkg_resources backend and defining a standard post-(un)install "index this new package" hook but I doubt I will have the (unfunded) time.
How would a database even work in that scenario (given multiple independent directories on sys.path), and why would you expect it to perform better than a modern filesystem?
I don't know, but it would be interesting to try. For example the operation that parses every RECORD (list of installed files) to do reference counts on uninstall would probably be much faster in a database.
Wheels are designed to be valid sys.path entries in the simple case. We don't encourage you to do so mostly because we don't want to include .pyc files for all the Pythons you might use,
Wait... isn't wheel a binary format? Oh, I see what you mean, wheels are a *fat* binary format, so they can include multiple Python versions and multiple platforms.
However, for the application platform plugins use case, wheels can potentially be quite awesome, because you can build one fat wheel for all your supported platforms. "We don't want to inlcude .pyc files for all the Pythons you might use" doesn't apply to the "we" who already build and release different platform-specific eggs to cover multiple platforms and Python versions. ;-)
If you were using wheels to implement a plugin system you could append .pyc files to the archive later, put up with slower start times, or unpack each wheel into a separate folder. A frustrating thing about eggs was that you had to replace them all even for pure-Python distributions when a new Python was released. A single wheel could mark itself as being compatible with Python 2.6-3.3, and we just let the user create the .pyc. The "fat" part has been left out of the implementation so far. The tags just mark a single wheel as working with several Pythons, usually only true for pure-Python dists. There is no provision for installing a different set of files based on the installation platform.
participants (5)
-
Daniel Holth -
Lennart Regebro -
Paul Moore -
PJ Eby -
Vinay Sajip