On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip wrote:

Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"

Right, but isn't that then an entry point? I agree REGISTRY is not a very good name in any case.

On Mon, Oct 8, 2012 at 3:59 PM, Lennart Regebro wrote:

On Mon, Oct 8, 2012 at 9:34 PM, Vinay Sajip wrote:

...

Registry of what ever you like - pointers to data, perhaps. Any use of the registry, just like entry points, is a matter of convention between consenting / cooperating developers and their applications/libraries, or am I missing something? "Entry points" are not always entry points, are they? IIUC, you're allowed to just define a module or package without the attributes appearing after the ":"

Right, but isn't that then an entry point?

I agree REGISTRY is not a very good name in any case.

I would prefer to keep the filename the same as long as the format is the same. (Not a suggestion to change the format.) bdist_wheel is easy to edit but pkg_resources.py is not because it has a long release cycle and if you make a mistake you will break almost every Python installation. Right now a new class attribute Distribution.PKG_INFO keeps track of whether the PEP-defined Metadata 1.0-1.3 is called PKG-INFO or METADATA. To support REGISTRY in pkg_resources you might do something similar, changing line ~2319 self._get_metadata('entry_points.txt'), self to self._get_metadata(self.ENTRY_POINTS_FILENAME), self or for name in (possible, entry, points, filenames): ... defining ENTRY_POINTS_FILENAME for both Distribution and DistInfoDistribution, and you would have to wait for a new distribute release. In other words, it is a hassle and the feature is the same. An entry_point is always an entry_point. It should no sooner be confused with an English "point of entry" than keywords like "print" should be confused with pressing books. Daniel

On Mon, Oct 8, 2012 at 5:14 PM, Vinay Sajip wrote:

What do *you* mean by entry point? My understanding is that it represents the start of some executable code. Wikipedia:

"In computer programming, an entry point is a memory address, corresponding to a point in the code of a computer program which is intended as the destination of a long jump, be it internal or external."

So, in the Python sphere, I would interpret an entry point to be a callable. While that might be the most common usage of the feature, it's not the only one. For example, you could use name = value entries where value is not a callable, but for example a name of a resource to use for something, or a configuration option, or the name of a package or a module (which is not an entry point by my understanding of entry points being callables).

In pkg_resources, entry points are references to any importable object. They don't have to be callables. Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard. ;-) In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.) If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import. I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times. That's why I baked those features into entry points from the beginning - I was battling what was at the time an excruciating slow startup for the Chandler application platform, and I borrowed the architectural pattern of delayed loading of plugins from Eclipse. (I also remember that Ian Bicking was talking about some problem, maybe on the Web-SIG, and I remember that the problem he was talking about could be solved by something like entry points, at which point I then had a solution for both Ian's problem and Chandler's problem.)

On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip wrote:

Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib,

The two aren't 100% connected, even assuming the antecedent were true. Originally, Guido Pronounced that they *were* to be added to the stdlib (for Python 2.5), and I withdrew them due largely to an objection about the lack of third-party expertise in the underlying formats -- an expertise that is now considerably more widespread, especially thanks to you and Daniel. ;-) More to the point, though, if it's not possible for someone to create a wrapper to reimplement these portions of the pkg_resources API, then it's going to be harder to get people to port their code. The challenge isn't necessarily that there are dozens of platforms that rely on this feature explicitly; the challenge is that code which uses pkg_resources APIs for these things is *intentionally ignorant* of whether the feature is being used. The entire point of the API is to free the caller from worrying about the callee's needs, and the callee's dependencies can be transparently met by distributions that are not directly on sys.path, at least in the .egg or .wheel model of things. (If .wheel files are importable, anyway.)

ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path.

Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the extras required by an entry point? (i.e., is it possible for someone to make a common wrapper that re-adds sys.path management?)

Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned?

Now that 3.3 includes a faster import mechanism, probably so. Under older Pythons, adding a large number of directories to sys.path could badly affect startup performance, if plugins were installed as directories instead of as zipfiles. (As a general note, in case you aren't aware: the entire .egg format was invented originally as a distribution+use format for application plugins; Bob Ippolito suggested that in the general case, any Python library could also be considered a "plugin" to Python itself, and then things kind of took off from there...)

On Tue, Oct 9, 2012 at 7:28 PM, Vinay Sajip wrote:

I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer.

Except that installation doesn't always equal placement on a sys.path. Application platforms can and do distinguish between a plugin that is "installed" and one that is "activated" or "enabled". Again, the entire egg infrastructure was originally designed to support pluggable applications, which is part of why it's so annoying to Linux packagers at times. (In roughly the same way as Firefox is annoying to them for having .xpi's.) I'm not saying distlib must support all these plugin usecases *itself*, but if it solves its chosen usecases in a way that can't be *adapted by others* to handle the app platform use cases, then there's not going to be an appealing alternative to sticking with pkg_resources or its derivatives.

I assume from the context above that that you actually mean "no" as the answer to the question "is it insufficient?", whereas I would take "probably so" to mean "yes, it's insufficient".

Oops - I misread that as "Do you feel that this is sufficient?". Sorry.

On Tue, Oct 9, 2012 at 5:45 PM, Vinay Sajip wrote:

PJ Eby writes:

...

In pkg_resources, entry points are references to any importable object. They don't have to be callables.

Note, however, that if you load an entry point which is a module, then of course the module code will be executed as a side-effect, so technically, a module still meets the "start of some executable code" standard.

Yes, I acknowledged this in my response to Lennart's post.

...

In any case, the name was chosen for the primary use case of designating executable entry points in Python code, often callables. However, given the nature of Python, there was absolutely no reason to restrict them to referring *only* to callables. (In fact, the feature's name and implementation was inspired by Eclipse's "extension points" architecture, where the advertised objects usually implement interfaces with more than one method.)

Right, they're just objects of one sort or another.

...

If the name *must* be changed, I'd suggest using "exports", for symmetry with "imports". Entry points are always the name of an importable object, and they are basically being advertised so that other code knows what is available to be imported, and the loading API allows one to handle other details such as updating sys.path as necessary, and doing the actual import.

As you say, entry points don't have to be actual entry points, and I didn't want to slavishly follow pkg_resources nomenclature; though it reflects the most common use case, it's not the only one. Your suggestion of "exports" seems very reasonable as a replacement for "registry", which I agree is too generic.

...

I understand that right now you're not supporting those other features in distlib, but I feel I should mention as a reminder that many of these pkg_resources features were created to support application platforms with plugins, such as Trac, Zope, Chandler, etc., and to prevent the need for wheel-reinvention between different Python plugin systems. For this type of platform, it's often an important core feature to be able to dynamically add plugins and their requirements to sys.path on an as-needed basis when the entry points are actually *used*, because early loading or additions to sys.path can dramatically affect application startup times.

I understand, and appreciate your pointing these things out. As I see it, the specific way that some of those features were implemented is what some people object to. In distlib, importing is still lazy, but sys.path is not updated before attempting imports. Clearly, distlib can't do things exactly the way they're done in pkg_resources and setuptools, since those are not considered suitable to go into the stdlib, whereas distlib hopes to provide (some of) the useful features provided in those packages in a (hopefully) non-contentious way, with perhaps part of it going into the stdlib at some future date if there's a consensus about that.

ISTM that the "exports" feature in distlib allows applications to cooperate in the same way as pkg_resources' entry points do, except for the fact that sys.path is not manipulated at all, meaning that all such cooperating applications are assumed to be installed on the path. Actual importing is delayed until needed; scanning of distribution metadata for exports also happens on demand. Do you feel that this is insufficient to meet the performance concerns you mentioned?

As far as my performance concerns are concerned, I just want to be able to avoid iterating through all the installed distributions twice for code that is using both APIs, which besides the porting hassle is why I might implement one API on top of the other implementation. I would also like to be able to store my installed package database in sqlite3 by implementing an appropriate distlib/pkg_resources backend and defining a standard post-(un)install "index this new package" hook but I doubt I will have the (unfunded) time. I'm sure sys.path manipulation could be implemented on top of a basic Distributions-from-path API to search for the entry points, without having to be integrated with it. FYI Ruby's gem seems to do this with the equivalent of a Python import hook (our sys.meta_path) and they do index the available gems in an actual not-just-a-bunch-of-files database. There is an interesting but long discussion at http://python.6.n6.nabble.com/Re-Python-Dev-PEP-365-Adding-the-pkg-resources... GvR starts off by wishing it wasn't 1700 lines long. It seems "easy to review" was a more important factor than the specific name of any particular feature, except for "easy_install" which was seen as too much of a setuptools brand name. Vinay wrote: I don't believe that wheels are meant to be directly importable, but I could be wrong about that. I am not sure why callee dependencies need to be transparently met by distributions not on sys.path; Broadly speaking, and leaving eggs out of the picture, I would expect a distribution to declare its dependencies, and for these to be installed in the appropriate site-packages by the installer.

...

Do you mean that it is not possible to identify the entry points of an arbitrary distribution? What about obtaining information about the

At the moment, not for a completely arbitrary distribution, as we don't have the metadata in non-installed distributions. Of course, an arbitrary distribution could be unpacked to a temporary location and have the equivalent of egg-info run to get the information; but for this to work ideally, such information would need to be present in the archive metadata and also need to be in the index. There are a number of ways this could happen - it's not clear to me what the best way is, but I probably haven't thought about it enough.

At this point of the runtime, the "arbitrary non-installed distribution" is just a dist that has been installed outside of sys.path. So the static metadata is there. Wheels are designed to be valid sys.path entries in the simple case. We don't encourage you to do so mostly because we don't want to include .pyc files for all the Pythons you might use, but it is handy for bootstrapping. python wheel-0.12.0-py2.py3-none-any.whl/wheel install wheel-0.12.0-py2.py3-none-any.whl installs wheel's own wheel with itself, even when pkg_resources.py is not installed. (You could do the same thing with PYTHONPATH=... python -m wheel install). You could imagine including with Python a similar bootstrap installer as a runnable .zip file (invent an extension .pya / .pyp?). Since the functionality is hidden in the .zip file, you would only have to commit to and argue about the command-line API instead of the much larger package management API. The bootstrap installer would install the real installer or build systems such as distutils, setuptools, Bento, distlib, providing that those projects were willing to publish themselves to pypi as pre-built wheel archives. If you want to play around with renaming the egg-info files, check out Bento's egg_utils.py line 106: https://github.com/cournape/Bento/blob/master/bento/commands/egg_utils.py#L1... which controls what each metadata file is named. file_table = { "pkg_info": "PKG-INFO", "sources": "SOURCES.txt", "install_requires": "requires.txt", "top_levels": "top_level.txt", "not_zip_safe": "not-zip-safe", "dependency_links": "dependency_links.txt", "entry_points": "entry_points.txt", "ipkg_info": "ipkg.info", }

On 10 October 2012 23:15, PJ Eby wrote:

However, for the application platform plugins use case, wheels can potentially be quite awesome, because you can build one fat wheel for all your supported platforms. "We don't want to inlcude .pyc files for all the Pythons you might use" doesn't apply to the "we" who already build and release different platform-specific eggs to cover multiple platforms and Python versions. ;-)

I really don't understand the whole idea of including .pyc files in binary distributions - all they do is *reduce* the flexibility of the distribution by tying it to a single Python version. Far better to just ship the .py files and let the system compile them on demand (or if you absolutely insist, compile them at install time). Note - wheels are a binary *distribution* format, and as such aren't intended to be placed on sys.path directly. The fact that they can be is not important, that's just a side effect of using a zipfile with a simple internal layout. I concede that things are different for eggs which were intended to be an importable format (AIUI, more so than a distribution format, even). Of course, "fat" wheels containing multiple versions of *compiled* code (maybe for different architectures, or API versions) are conceivable - although I don't think the current internal layout of the wheel format supports that. Paul.