Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script)
On 16 July 2013 13:42, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip...
Couldn't you just ship both variants of the exe wrappers in a single distribution and then use the correct one for the current installation? That's what I'm doing in py2app.
That's OK for source-style installs (which is what setuptools does, and what pip mostly cares about right now). But for bundling with Python it needs to be considered (although it's just getting the right one in the right installer). But for wheels it's a pain, because instead of having just a single pip wheel, we need 32-bit and 64-bit wheels solely for the wrappers. That sucks. Hard. (And it's not just for pip, nose will have the same problem, as will many other projects that use exe wrappers). And the bdist_wheel command currently doesn't recognise this *at all*. So wheels using wrappers are potentially broken. I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found). This is complex enough that I'm now concerned that we need reference "wheel install" code in the stdlib, just so that people don't make up their own on the basis that "wheel is simple to install manually" and screw it up. Also so that we only have one style of command line script wrapper to deal with going forward, not a multitude of mostly-compatible solutions. Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0? Paul PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP.
On 16 July 2013 23:08, Paul Moore <p.f.moore@gmail.com> wrote:
Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0?
I believe Daniel already covered it in PEP 427 - rather than baking the entry point wrappers into the wheel, installers can generate any needed entry point wrappers if the wheel includes Python scripts in {distribution}-{version}.data/scripts/ (see http://www.python.org/dev/peps/pep-0427/#recommended-installer-features) Now, there may be holes in that scheme, but it seemed solid enough when I approved the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 16 July 2013 14:21, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 July 2013 23:08, Paul Moore <p.f.moore@gmail.com> wrote:
Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0?
I believe Daniel already covered it in PEP 427 - rather than baking the entry point wrappers into the wheel, installers can generate any needed entry point wrappers if the wheel includes Python scripts in {distribution}-{version}.data/scripts/ (see http://www.python.org/dev/peps/pep-0427/#recommended-installer-features)
Now, there may be holes in that scheme, but it seemed solid enough when I approved the PEP.
The big problem is that implementations don't do that :-( AFAIK, none of distlib, pip or wheel itself do anything with script wrappers except rewrite #! lines (which is the other, much easier, item in that section). At the moment, bdist_wheel just puts the exe wrappers generated from the source into the wheel itself, which again is probably wrong in the context what the PEP suggests. As I said in my other email, I think this is subtle enough that we need a stdlib implementation to stop people making mistakes like this. It's certainly not fair to expect a mostly-Unix development team to get this sort of Windows arcana right without some help. Paul
On 16 July 2013 23:29, Paul Moore <p.f.moore@gmail.com> wrote:
As I said in my other email, I think this is subtle enough that we need a stdlib implementation to stop people making mistakes like this. It's certainly not fair to expect a mostly-Unix development team to get this sort of Windows arcana right without some help.
Are we talking about the pip developers or python-dev, here? I think Martin and Brian feel pretty lonely, too :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 16 Jul, 2013, at 15:08, Paul Moore <p.f.moore@gmail.com> wrote:
On 16 July 2013 13:42, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On the other hand, I'm missing something, as I don't see how the *current* exe wrappers avoid meaning that there need to be separate 32-bit and 64-bit versions of pip...
Couldn't you just ship both variants of the exe wrappers in a single distribution and then use the correct one for the current installation? That's what I'm doing in py2app.
That's OK for source-style installs (which is what setuptools does, and what pip mostly cares about right now). But for bundling with Python it needs to be considered (although it's just getting the right one in the right installer). But for wheels it's a pain, because instead of having just a single pip wheel, we need 32-bit and 64-bit wheels solely for the wrappers. That sucks. Hard. (And it's not just for pip, nose will have the same problem, as will many other projects that use exe wrappers). And the bdist_wheel command currently doesn't recognise this *at all*. So wheels using wrappers are potentially broken.
You could just have a wheel that contains two data files: wrapper-win32.exe and wrapper-win64.exe, then select the one that gets used as the wrapper for a specific script when you create that wrapper. That's assuming that the wrapper .exe gets "created" when a wheel is installed and is not included in the wheel.
I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found).
Yikes, that means my assumption is wrong. The section on "Recommended installer features" in the wheel spec[1] says that the wrapper executable should be created on installation, does pip not do this?
This is complex enough that I'm now concerned that we need reference "wheel install" code in the stdlib, just so that people don't make up their own on the basis that "wheel is simple to install manually" and screw it up. Also so that we only have one style of command line script wrapper to deal with going forward, not a multitude of mostly-compatible solutions.
I'd love to see comprehensive wheel support in the stdlib, but that may have to wait for 3.5 because the entire packaging systeem (wheels, metadata, ...) is moving forward quickly at the moment. That said, it would be nice if distutils would grow support for creating wheels and modern metadata in sdists as that would mean I could drop usage of setuptools for most of my software (for python 3.4).
Nick: See the above point re PEP 426 - do you agree that this needs addressing in Metadata 2.0?
Paul
PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP.
That depends on what the wrapper does, if it launches a regular python with the right command-line you might be able to get away with a single wrapper, if it loads python.dll and executes the script directory you do need separate wrappers for 32 and 64 bit. [1] http://www.python.org/dev/peps/pep-0427/#recommended-installer-features
On 16 July 2013 14:30, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found).
Yikes, that means my assumption is wrong. The section on "Recommended installer features" in the wheel spec[1] says that the wrapper executable should be created on installation, does pip not do this?
Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the moment, and exes are being added to the wheels at wheel build time, which is also wrong. That'll teach me to work from reality rather than specs :-( Daniel, Vinay, pip developers - it looks like we need to do some work in this area to make the code conform to the specs. The PEP only says this is a "recommended" feature, but frankly I think it needs to be mandatory, or script wrappers are going to be a mess we'll be dealing with for some time :-(
PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP.
That depends on what the wrapper does, if it launches a regular python with the right command-line you might be able to get away with a single wrapper, if it loads python.dll and executes the script directory you do need separate wrappers for 32 and 64 bit.
As I said in another message, looks like there's no real reason for separate wrappers. A 32-bit one should work regardless [1]. But wheels built on 64-bit systems at the moment won't work on 32-bit ones (because the wrappers will be 64-bit). [1] With the possible exception that Windows' magic shadowing of 32 and 64 bit "stuff" - the WOW64 magic that I know nothing about - could cause odd results in obscure cases. I propose ignoring this in the absence of actual bug reports :-) Paul
Paul Moore <p.f.moore <at> gmail.com> writes:
Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the moment, and exes are being added to the wheels at wheel build time, which is also wrong.
Not true for distlib - it doesn't add .exe wrappers to wheels at build time :-) It adds them to the target directory when installing under Windows. (You can also choose not to install any wrappers.) Regards, Vinay Sajip
Need a script that visits all the console-script entry points to regenerate the wrappers. Then there are also the non-console-scripts scripts... I consider scripts as an optional convenience, but I suppose that isn't always the case. On Tue, Jul 16, 2013 at 9:53 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the moment, and exes are being added to the wheels at wheel build time, which is also wrong.
Not true for distlib - it doesn't add .exe wrappers to wheels at build time :-) It adds them to the target directory when installing under Windows. (You can also choose not to install any wrappers.)
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
Not true for distlib - it doesn't add .exe wrappers to wheels at build time It adds them to the target directory when installing under Windows. (You can also choose not to install any wrappers.)
Sorry, some misinformation there - distlib does do this when invoked via distil. However, this can be turned off at the distlib level - I will update distil to not do this when adding wheels. Regards, Vinay Sajip
On 16 July 2013 15:01, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
Not true for distlib - it doesn't add .exe wrappers to wheels at build
time
It adds them to the target directory when installing under Windows. (You can also choose not to install any wrappers.)
Sorry, some misinformation there - distlib does do this when invoked via distil. However, this can be turned off at the distlib level - I will update distil to not do this when adding wheels.
OK. That sounds good. I'm starting to become uncertain as to whether we actually have an issue here. I think: (1) Builders can put anything they like into the scripts directory. That's more or less out of our control. From what I know of distil's approach, it doesn't actually execute setup.py so it keeps a lot more control than (say) bdist_wheel, which effectively runs setup.py install to a temporary directory then bundles what it finds. But in essence, as things stand right now, the scripts directory of an arbitrary wheel could contain arbitrary stuff. (2) Wheel installers should "make Python scripts work". All that I'm aware of fix up shebang lines and add execute bits. Only distlib adds exe wrappers, I believe. Others work on Windows because the builders put exe wrappers in place (see (1)) but that has some issues. I suspect there may be some "accidental" successes on Unix where setuptools scripts do or don't get a .py extension depending on the target platform, but I have no evidence of this myself. So there's a lot of potentially platform dependent stuff in "scripts" and wheel builders don't (can't) recognise and encode this in the tags. Wheels mostly work at the moment because not many people use them cross platform. But there's potential for issues down the line. FWIW, I believe that the whole "scripts" directory as a concept is too platform-specific. The only real use for it is to expose CLI applications, and a declarative approach like setuptools console_scripts entry points would be better. So longer term I'd argue for deprecating "scripts" altogether and replacing it with some form of "CLI entry point" metadata which may be exposed as part of metadata 2.0 or 3.0, or may simply be internal metadata communicated between the builder and the installer, but not exposed at the PyPI level. Oscar's email argues for exposing it as project metadata, though, and I can see the benefit. Paul
Paul Moore <p.f.moore <at> gmail.com> writes:
FWIW, I believe that the whole "scripts" directory as a concept is too platform-specific. The only real use for it is to expose CLI applications
Well, you can also expose GUI applications this way, though the applications are fewer - Qt applications could easily be cross-platform, for example. Also, you can put in scripts which are not entry-point related, which would be essential if you e.g. have existing scripts to bundle as part of an application, which could be written in other languages, say. Not common, perhaps, but not a case you want to arbitrarily restrict given that it works now. Of course, there the distributor of the package is responsible for ensuring cross-platform workability of such scripts using e.g. including .cmd files for Windows or whatever. Regards, Vinay Sajip
On 16 July 2013 16:51, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
FWIW, I believe that the whole "scripts" directory as a concept is too platform-specific. The only real use for it is to expose CLI applications
Well, you can also expose GUI applications this way, though the applications are fewer - Qt applications could easily be cross-platform, for example. Also, you can put in scripts which are not entry-point related, which would be essential if you e.g. have existing scripts to bundle as part of an application, which could be written in other languages, say. Not common, perhaps, but not a case you want to arbitrarily restrict given that it works now. Of course, there the distributor of the package is responsible for ensuring cross-platform workability of such scripts using e.g. including .cmd files for Windows or whatever.
You're right, I should have included GUI apps. I'm not aware of any cases of scripts that aren't entry-point related (and which couldn't be converted to entry points fairly easily) although that certainly doesn't mean there aren't any. OTOH, I do know that there used to be a *lot* of examples of scripts doing what entry points do that were either absolutely not cross-platform (Unix shell scripts, bat files) or were problematic/badly implemented (Python scripts without a py extension, etc) Many of these have now gone, thank goodness, migrated to setuptools entry points. If I were writing a firm proposal, I'd go for something like entry points as metadata, managed by installers, as the primary interface (with backward compatibility code to automatically migrate setuptools entry points so the impact on developers is minimal) and the scripts directory/setup argument being solely for backward compatibility, no management by installers at all, whatever is in there just gets dumped onto the target unchanged (and the contents of scripts is explicitly defined as *not* affecting the compatibility tags). So setuptools users get new entry point metadata automatically, which is processed at install time, and anyone still using the distutils scripts parameter still works as before but gets no particular support from the new tools. Paul
If I were writing a firm proposal, I'd go for something like entry points as metadata
My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. Example of the metadata for setuptools 0.8: "exports": { "distutils.commands": [ "alias = setuptools.command.alias:alias", "bdist_egg = setuptools.command.bdist_egg:bdist_egg", "bdist_rpm = setuptools.command.bdist_rpm:bdist_rpm", "build_ext = setuptools.command.build_ext:build_ext", "build_py = setuptools.command.build_py:build_py", "develop = setuptools.command.develop:develop", "easy_install = setuptools.command.easy_install:easy_install", "egg_info = setuptools.command.egg_info:egg_info", "install = setuptools.command.install:install", "install_lib = setuptools.command.install_lib:install_lib", "rotate = setuptools.command.rotate:rotate", "saveopts = setuptools.command.saveopts:saveopts", "sdist = setuptools.command.sdist:sdist", "setopt = setuptools.command.setopt:setopt", "test = setuptools.command.test:test", "install_egg_info = setuptools.command.install_egg_info:install_egg_info", "install_scripts = setuptools.command.install_scripts:install_scripts", "register = setuptools.command.register:register", "bdist_wininst = setuptools.command.bdist_wininst:bdist_wininst", "upload_docs = setuptools.command.upload_docs:upload_docs" ], "scripts": { "console": [ "easy_install = setuptools.command.easy_install:main" ] }, "egg_info.writers": [ "PKG-INFO = setuptools.command.egg_info:write_pkg_info", "requires.txt = setuptools.command.egg_info:write_requirements", "entry_points.txt = setuptools.command.egg_info:write_entries", "eager_resources.txt = setuptools.command.egg_info:overwrite_arg", "namespace_packages.txt = setuptools.command.egg_info:overwrite_arg", "top_level.txt = setuptools.command.egg_info:write_toplevel_names", "depends.txt = setuptools.command.egg_info:warn_depends_obsolete", "dependency_links.txt = setuptools.command.egg_info:overwrite_arg" ], "setuptools.file_finders": [ "svn_cvs = setuptools.command.sdist:_default_revctrl" ], "distutils.setup_keywords": [ "eager_resources = setuptools.dist:assert_string_list", "namespace_packages = setuptools.dist:check_nsp", "extras_require = setuptools.dist:check_extras", "install_requires = setuptools.dist:check_requirements", "tests_require = setuptools.dist:check_requirements", "entry_points = setuptools.dist:check_entry_points", "test_suite = setuptools.dist:check_test_suite", "zip_safe = setuptools.dist:assert_bool", "package_data = setuptools.dist:check_package_data", "exclude_package_data = setuptools.dist:check_package_data", "include_package_data = setuptools.dist:assert_bool", "packages = setuptools.dist:check_packages", "dependency_links = setuptools.dist:assert_string_list", "test_loader = setuptools.dist:check_importable", "use_2to3 = setuptools.dist:assert_bool", "convert_2to3_doctests = setuptools.dist:assert_string_list", "use_2to3_fixers = setuptools.dist:assert_string_list", "use_2to3_exclude_fixers = setuptools.dist:assert_string_list" ], "setuptools.installation": [ "eggsecutable = setuptools.command.easy_install:bootstrap" ] } Regards, Vinay Sajip
On 16 July 2013 19:18, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If I were writing a firm proposal, I'd go for something like entry points as metadata
My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows.
Nice. So we could say that wheel installers should create exe wrappers based on the 'scripts' key of the 'exports' metadata. Then we process the 'scripts' directory unaltered for backward compatibility. The only other part of the equation is to ensure that wheel *builders* do not put setuptools-generated entry point wrappers into the scripts directory. Before I embarrass myself again (:-)) is this also something you already have implemented in distlib? :-) When you're finished with Guido's time machine, make sure you put the keys back :-) Paul
Paul Moore <p.f.moore <at> gmail.com> writes:
The only other part of the equation is to ensure that wheel *builders* do not put setuptools-generated entry point wrappers into the scripts directory. Before I embarrass myself again () is this also something you already have implemented in distlib?
No, because I didn't want to embarrass you ;-) Seriously - no, because that is policy rather than mechanism. The way distil builds wheels is to perform an installation into a working area, and then call Wheel.build in distlib pointing to the work area. The distlib code just copies what's in the work area to the wheel. Currently the install-scripts part of distil installs all scripts into the work area, including ones defined in exports; it should be a five-minute job to ensure that scripts in exports are excluded from this, when building wheels. A version of distil with these updates should appear soon - I'm waiting for the dust to settle on the most recent changes to PEP 426. Regards, Vinay Sajip
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
defined in exports; it should be a five-minute job to ensure that scripts in exports are excluded from this, when building wheels.
It was a quick job, but thinking about it, I should probably update the Wheel.install API to take an optional process_exports=True argument, so that the exported-script processing can be done during installation from wheels in a standardised way. Regards, Vinay Sajip
On 16 July 2013 20:21, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
defined in exports; it should be a five-minute job to ensure that scripts in exports are excluded from this, when building wheels.
It was a quick job, but thinking about it, I should probably update the Wheel.install API to take an optional process_exports=True argument, so that the exported-script processing can be done during installation from wheels in a standardised way.
I'm not 100% sure what your proposal is here - I'm confused about the precise roles of setup.py/setuptools as "builder" vs distil as "builder" vs distlib as "wheel builder" vs distlib as "wheel installer". I'll try to get some time in the next day or so to review the code and make sure we're not talking at cross purposes here. Bear with me. I understood that distlib simply built wheels from whatever is in the directories supplied. I'm not clear where it would get the "exports" metadata to store in the wheel. Assuming that's sorted, though, then whether or not distlib processes the exports on install is not an issue to me (I think it always should). What is more of an issue is what "the thing that puts stuff into the directories" does. If that's a setuptools-based build, it will process the exports data *itself* and put wrappers into the scripts directory. That is what I think we should be trapping and suppressing. But we have to be careful, as if setuptools is used to install directly, *not* going via a wheel, it has to generate wrappers itself somehow. Using distil to do the build is a whole other route. My main concern here is keeping a reasonable level of backward compatibility/interoperability with all the assorted tools around. And in particular with pip managing (but technically not *doing*) the builds and installs. I'll have a think, and do some code reviews, and try not get sucked into sending more emails until I know the facts a bit more clearly :-) Paul
Paul Moore <p.f.moore <at> gmail.com> writes:
I'm not 100% sure what your proposal is here - I'm confused about the precise roles of setup.py/setuptools as "builder" vs distil as "builder" vs distlib as "wheel builder" vs distlib as "wheel installer". I'll try to get some time in the next day or so to review the code and make sure we're not talking at cross purposes here. Bear with me.
I'm not sure I've got a concrete proposal yet - I'm still thinking things out, too :-) Currently, the wheel module in distlib doesn't do anything too clever - just the mechanical stuff of archiving and unarchiving. Because the exports stuff wasn't in the PEP, distil wrote the generated scripts and .exe wrappers to the work area, and so they ended up in the wheel. (This is the case for all released versions of distil).
I understood that distlib simply built wheels from whatever is in the directories supplied. I'm not clear where it would get the "exports" metadata to store in the wheel. Assuming that's sorted, though, then whether or not
Right now it's getting the exports courtesy of distil when you use that to build the wheel - distlib can't rely on that information being available because it's not part of the PEP. If the PEP is updated to include the exports, they should be in the wheel no matter which tool builds it. Then in theory distlib could generate the scripts during installation, but there are a lot of options to consider - did setuptools put them in there already? Do we want native launchers? etc. which is perfectly doable in distlib, but I'm not sure that's the best place for it because I think wheel processing should be uncomplicated. Wheel.install already has quite a few kwargs: dry_run=False: Don't actually do anything executable=None:Custom executable for shebang lines (e.g. to support path searching) warner=None:Used to defer warning behaviour to calling application lib_only=False:Process site-packages contents only (you suggested this) I'd like to not have to add any more, unless it's unavoidable :-) Nevertheless, I will probably try implementing it in distlib as an experiment, too see how it looks.
distlib processes the exports on install is not an issue to me (I think it always should). What is more of an issue is what "the thing that puts stuff into the directories" does. If that's a setuptools-based build, it will process the exports data *itself* and put wrappers into the scripts directory. That is what I think we should be trapping and suppressing. But we have to be careful, as if setuptools is used to install directly, *not* going via a wheel, it has to generate wrappers itself somehow.
Exactly why I'm so leery of putting this logic in distlib, until we think it through and add it to the PEP. At the moment distil does it the same way as setuptools/pip only to remain compatible, not for any other reason.
Using distil to do the build is a whole other route.
Right. I'm aiming for distil to be able to do just about everything pip can (functionally, the code is pretty much there barring installs from DVCS URLs), but backward compatibility is always a concern and a challenge :-) Another complication for distlib is that I expect it to work in 2.6+, where you can't always rely on the py launcher being present - hence the wrappers in distlib, with flags to disable writing them out. Another area to consider for scripts is which of foo, fooX and foo-X.Y to write to the scripts folder. This is particularly important in user site-packages, where scripts for different Python versions will coexist in the same folder, and the possibility exists of overwriting, which sometimes leads to unexpected behaviour (e.g. if I install foo-dist using 2.x which installs foo and foo-2.x to ~/.local/bin, then install it using 3.x, it would write foo and foo-3.x to ~/.local/bin. Quite apart from headaches with native launchers, it could be that the foo script installed with 3.x doesn't work (e.g. if it tries to process Python code which is 2.x compatible but not 3.x compatible). Fun and games! Regards, Vinay Sajip
On 16 July 2013 22:41, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
If the PEP is updated to include the exports, they should be in the wheel no matter which tool builds it. Then in theory distlib could generate the scripts during installation, but there are a lot of options to consider - did setuptools put them in there already? Do we want native launchers? etc. which is perfectly doable in distlib, but I'm not sure that's the best place for it because I think wheel processing should be uncomplicated. Wheel.install already has quite a few kwargs:
I really don't want the wrappers to be present in the wheel, because if they are the wheel becomes architecture-specific. Also, consider that Unix targets should have the actual scripts written with no extension, whereas Windows targets should have foo-script.py and foo.exe. That should be decided at install time, bot at wheel creation time. As regards version-specific scripts, I'd assume it's the project's job to specify precisely what scripts they want. On that one, I'm on the side of providing infrastructure, not setting policy. Although I could be persuaded otherwise if there was a PEP on what commands a distribution should provide. In that case, let the project provide the command names, and let the installer implement the standard versioned-executable policy. Paul.
I really don't want the wrappers to be present in the wheel, because if they are the wheel becomes architecture-specific. Also, consider that Unix targets should have the actual scripts written with no extension, whereas Windows targets should have foo-script.py and foo.exe. That should be decided at install time, bot at wheel creation time.
I think we're agreed on that as a desirable. I've already changed the distil code to omit writing launchers to the wheel, and that probably won't need to change. But it does generate the scripts for exports - to avoid doing that, we need to get the exports into the wheel in a standardised way (via pydist.json, perhaps, or perhaps a separate file). Iterating over the exports in a distribution path needs to be fast - note that the exports are not just scripts, and it's not yet clear whether the script exports need to be separated out from the rest.
As regards version-specific scripts, I'd assume it's the project's job to specify precisely what scripts they want. On that one, I'm on the side of providing infrastructure, not setting policy.
This sounds like you mean that distlib needs to stay basic (mechanism/infrastructure), and the installer needs to do the script generation (policy - perhaps controlled by the user). Currently, distlib allows one to specify foo with optional variants fooX and foo-X.Y (before factoring native launchers into the mix). This is set on the ScriptMaker instance which generates the scripts in the target directory (including shebang rewriting, native launchers etc.) Regards, Vinay Sajip
On 17 Jul 2013 04:19, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:
If I were writing a firm proposal, I'd go for something like entry
points as metadata
My extended metadata already covers this, though I use the name "exports"
(suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows. Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the pre-JSON days). However, I now think it makes more sense to standardise them as an "exports" field in PEP 426. So run with the assumption that something like that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata. Cheers, Nick.
On Jul 16, 2013, at 8:00 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 17 Jul 2013 04:19, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:
If I were writing a firm proposal, I'd go for something like entry points as metadata
My extended metadata already covers this, though I use the name "exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows.
Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the pre-JSON days).
However, I now think it makes more sense to standardise them as an "exports" field in PEP 426. So run with the assumption that something like that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata.
Cheers, Nick. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Are these only the scripts portion of entry points, or the whole kit and caboodle of pluggable entry points? Because I think the first makes sense, the second I'm hesitant on. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 17 Jul 2013 10:03, "Donald Stufft" <donald@stufft.io> wrote:
On Jul 16, 2013, at 8:00 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 17 Jul 2013 04:19, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:
If I were writing a firm proposal, I'd go for something like entry
My extended metadata already covers this, though I use the name
"exports" (suggested by PJE) because you can share not just code but data, and "entry points" generally implies code. The current version of distil creates wrappers for both gui and console scripts, and adds the appropriate native executable wrappers (32- or 64-bit, according to the running Python) on Windows.
Yeah, originally we were going to postpone dealing with entry points to a metadata extension (Daniel even had a proto-PEP kicking around in the
However, I now think it makes more sense to standardise them as an
"exports" field in PEP 426. So run with the assumption that something like
Cheers, Nick.
_______________________________________________
Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Are these only the scripts portion of entry points, or the whole kit and caboodle of pluggable entry points? Because I think the first makes sense,
points as metadata pre-JSON days). that will be part of the standard metadata - either derived from entry_points.txt for existing metadata, or specified directly for next generation metadata. the second I'm hesitant on. Actually, it may be better to have a top level "scripts" field, distinct from a general export mechanism. I'm seeing value in an exports mechanism, though. Yes, *in theory* you can get the same effect with an extension, but extensions can do a lot of other things, too. Python's metaprogramming is built on a model of multiple tools with increasing levels of power, flexibility and complexity, so I'm thinking an exports vs extensions split may be a good approach in a similar vein. No decision on this front yet, but I think it's at least worth my trying it out to see how it looks in the context of the PEP. (After all, we already know entry points are quite a popular feature of the setuptools metadata) A couple of bonus features of standardisation are that we can tie it into the extras system and automatic analysis tools can check the exports can actually be imported without needing to understand arbitrary extensions. Cheers, Nick.
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
Nick Coghlan <ncoghlan <at> gmail.com> writes:
Actually, it may be better to have a top level "scripts" field, distinct from a general export mechanism. I'm seeing value in an exports mechanism, though.
The exports functionality is important and used enough to warrant support in the PEP, and not only for the scripts part. There should be some way that this data gets into .dist-info in a standardised way, so that there is an ability to query e.g. implementations of a particular interface. Currently, distlib supports this as an extension to the PEP by reading a file from .dist-info, which distil puts there. At the moment the PEP is silent on the subject, which could lead to fragmentation in the implementations - e.g. whether JSON Or ini-style format is used for the data. I'd like to suggest that the whole of the exports information be included in the PEP 426 metadata, without singling out the scripts part. That way, it ends up in .dist-info via pydist.json. It has been suggested by PJE that the exports information should be in a separate file for speed of searching - though that suggestion was made in a pre-JSON world, where the speed of parsing the metadata wasn't C-assisted. Should performance still be an issue, then the exports dict could still be written out separately in .dist-info as e.g. exports.json by an installer. Regards, Vinay Sajip
On Jul 17, 2013, at 3:40 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
It has been suggested by PJE that the exports information should be in a separate file for speed of searching - though that suggestion was made in a pre-JSON world, where the speed of parsing the metadata wasn't C-assisted.
I don't think it was speed of parsing the file that was his concern, rather that if it's a separate file that only exists when there are entry points, then you don't have to open a file for every installed distribution. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft <donald <at> stufft.io> writes:
I don't think it was speed of parsing the file that was his concern, rather that if it's a separate file that only exists when there are entry points, then you don't have to open a file for every installed distribution.
OK. In that case, exports.json would avoid the need to open pydist.json to look for exports - if exports.json is missing, it would mean that dist has no exports. Regards, Vinay Sajip
On 16 July 2013 14:38, Paul Moore <p.f.moore@gmail.com> wrote:
On 16 July 2013 14:30, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
I think the correct solution is to explicitly have declarative support for "console script entry point" metadata in PEP 426, as well as having tools like bdist_wheel and distil do some explicit backward compatibility hacking to remove legacy-style exe wrappers. The wheel install code should then explicitly install appropriate wrappers for the target platform (which may be exe wrappers similar to the current ones, but moving forward may be some other mechanism if one is found).
There are many other uses for console script entry point metadata. For example, it would be good to be able to query pip/pypi in order to find out which packages supply a particular console command. One feature of Ubuntu that I really like is the way that it automatically tells you how to install any missing command: oscar:~$ pypy The program 'pypy' is currently not installed. You can install it by typing: sudo apt-get install pypy Obviously it's not as useful when the command and the package have exactly the same name :)
Yikes, that means my assumption is wrong. The section on "Recommended installer features" in the wheel spec[1] says that the wrapper executable should be created on installation, does pip not do this?
Yes, Nick pointed me at that part of the PEP. Nobody's doing that at the moment, and exes are being added to the wheels at wheel build time, which is also wrong.
That'll teach me to work from reality rather than specs :-(
Daniel, Vinay, pip developers - it looks like we need to do some work in this area to make the code conform to the specs. The PEP only says this is a "recommended" feature, but frankly I think it needs to be mandatory, or script wrappers are going to be a mess we'll be dealing with for some time :-(
I think that it should be mandatory. It should be possible for someone releasing a script via pypi to ensure that their program can be invoked after installation under a name of their choosing (assuming the user has the Python Scripts/bin directory in PATH). AFAIK the only bullet-proof way to do this on Windows is with an .exe wrapper. If you only want the program to be invokable from cmd and PowerShell* then a .bat file should be fine. Depending on file extension to invoke .py files with py.exe is subject to input/output redirection bugs on some windows systems (this is solveable when using .py in PATHEXT instead of file associations for cmd at least). However, if you also want the program name to be invokable from e.g. subprocess with shell=False or from git-bash or Cygwin or many other things then neither .bat files nor PATHEXT are sufficient. Wrapper .exes are necessary to ensure that this works properly. Oscar * I don't actually use PowerShell and cannot confirm that running .bat files works fully i.e. without screwing up sys.argv encoding or input/output redirection or anything else.
On 16 July 2013 16:09, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
If you only want the program to be invokable from cmd and PowerShell* then a .bat file should be fine. Depending on file extension to invoke .py files with py.exe is subject to input/output redirection bugs on some windows systems (this is solveable when using .py in PATHEXT instead of file associations for cmd at least).
bat files have many, many problems. The worst ones are: * Not nestable. If pip is a bat file, saying "pip install foo" from within another bat file will fail silently (control never returns to the line after the pip command). * If you interrupt them you get the obnoxious "Do you want to terminate the batch file?" prompt. If anyone suggests using bat files, I'll cry :-) However, if you also want the program name to be invokable from e.g.
subprocess with shell=False or from git-bash or Cygwin or many other things then neither .bat files nor PATHEXT are sufficient. Wrapper .exes are necessary to ensure that this works properly.
Yes. I have been convinced that ultimately, wrapper exes are the only "transparent" means of writing command-line applications on Windows. Because of this, I'd quite like it if wrapper functionality were added to the py launcher (most of the functionality is already present, it would probably be a pretty small change) so that we had a "one obvious way" of writing wrappers. I may try to put together a patch for CPython to this effect... Paul
On 16 July 2013 16:22, Paul Moore <p.f.moore@gmail.com> wrote:
On 16 July 2013 16:09, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
However, if you also want the program name to be invokable from e.g. subprocess with shell=False or from git-bash or Cygwin or many other things then neither .bat files nor PATHEXT are sufficient. Wrapper .exes are necessary to ensure that this works properly.
Yes. I have been convinced that ultimately, wrapper exes are the only "transparent" means of writing command-line applications on Windows.
Because of this, I'd quite like it if wrapper functionality were added to the py launcher (most of the functionality is already present, it would probably be a pretty small change) so that we had a "one obvious way" of writing wrappers. I may try to put together a patch for CPython to this effect...
I don't know whether or not you intend to have wrappers also work for Python 2.7 (in a third-party package perhaps) but there is a slightly subtle point to watch out for when non-ASCII characters in sys.argv come into play. Python 2.x uses GetCommandLineA and 3.x uses GetCommandLineW. A wrapper to launch 2.x should use GetCommandLineA and CreateProcessA to ensure that the 8-bit argument strings are passed through unaltered. To launch 3.x it should use the W versions. If not then the MSVC runtime (or the OS?) will convert between the 8-bit and 16-bit encodings using its own lossy routines. Oscar
Oscar Benjamin <oscar.j.benjamin <at> gmail.com> writes:
Python 2.x uses GetCommandLineA and 3.x uses GetCommandLineW. A wrapper to launch 2.x should use GetCommandLineA and CreateProcessA to ensure that the 8-bit argument strings are passed through unaltered. To launch 3.x it should use the W versions. If not then the MSVC runtime (or the OS?) will convert between the 8-bit and 16-bit encodings using its own lossy routines.
There is a standalone launcher available (referenced in PEP 397) which was the reference implementation, but it uses Unicode throughout. I would have assumed that whatever provided the decoding behind GetCommandLineW would use the same encoding in CreateProcessW(..., unicode_command_line, ...). Do you have any specific examples showing potential failure modes? Regards, Vinay Sajip
From: Oscar Benjamin I don't know whether or not you intend to have wrappers also work for Python 2.7 (in a third-party package perhaps) but there is a slightly subtle point to watch out for when non-ASCII characters in sys.argv come into play.
Python 2.x uses GetCommandLineA and 3.x uses GetCommandLineW. A wrapper to launch 2.x should use GetCommandLineA and CreateProcessA to ensure that the 8-bit argument strings are passed through unaltered. To launch 3.x it should use the W versions. If not then the MSVC runtime (or the OS?) will convert between the 8-bit and 16-bit encodings using its own lossy routines.
The launcher should always use GetCommandLineW, because the command line is already stored in a 16-bit encoding. GetCommandLineA will decode to an 8-bit encoding using some code page/settings (I can probably find out exactly which ones, but I don't know/care off the top of my head), and CreateProcessA will convert back using (hopefully) the same code page. There is never any point passing data between *A APIs in Windows, because they are just doing the conversion in the background. All you gain is that the launcher will corrupt the command line before python.exe gets a chance to. Cheers, Steve
On 19 July 2013 20:48, Steve Dower <Steve.Dower@microsoft.com> wrote:
From: Oscar Benjamin I don't know whether or not you intend to have wrappers also work for Python 2.7 (in a third-party package perhaps) but there is a slightly subtle point to watch out for when non-ASCII characters in sys.argv come into play.
Python 2.x uses GetCommandLineA and 3.x uses GetCommandLineW. A wrapper to launch 2.x should use GetCommandLineA and CreateProcessA to ensure that the 8-bit argument strings are passed through unaltered. To launch 3.x it should use the W versions. If not then the MSVC runtime (or the OS?) will convert between the 8-bit and 16-bit encodings using its own lossy routines.
The launcher should always use GetCommandLineW, because the command line is already stored in a 16-bit encoding. GetCommandLineA will decode to an 8-bit encoding using some code page/settings (I can probably find out exactly which ones, but I don't know/care off the top of my head), and CreateProcessA will convert back using (hopefully) the same code page.
There is never any point passing data between *A APIs in Windows, because they are just doing the conversion in the background. All you gain is that the launcher will corrupt the command line before python.exe gets a chance to.
Okay, thanks for the correction. The issue that made me think this was to do with calling Python 2.x as a subprocess of 3.x and vice-versa. When I looked back at it now I saw that the problem was to do with explicitly encoding with sys.getfilesystemencoding() in Python and using the mbcs codec (which previously had no error handling apart from 'replace'). Oscar
On 16 July 2013 14:08, Paul Moore <p.f.moore@gmail.com> wrote:
PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP.
Hmm. I just did a quick test, and then based on the results checked the setuptools source code. I can see no reason why there needs to be 32 and 64 bit launcher exes. The launchers simply use CreateProcess to launch a separate Python process using the #! line of the script. So there's no DLL loading going on, and no reason that I can see for needing separate 32 and 64 bit builds. Jason - can you shed any light on why there are separate builds for 32 and 64 bits? Actually, the launcher is essentially identical to the "py" launcher for Windows, except that it gets a script name to execute from the name of the launcher. I'm wondering whether the correct approach here would be to enhance the launcher one more time to look for a suitably named script and auto-run it if it's present (i.e. merge the wrapper functionality into the launcher). Then we have a standard wrapper that everyone can use and not reinvent their own. Paul
There are two versions of launchers primarily because of my naiveté when addressing the UAC issue. 64-bit launchers were exempt from the UAC restrictions that caused them to launch in a separate window. I believed this to be a proper fix, when in fact those still using 32-bit launchers were still experiencing the problem. See https://bitbucket.org/tarek/distribute/issue/143/easy_install-opens-new-cons ole-cant-read for more detail. So I agree, it would probably be sufficient to only supply 32-bit executables. However, my preference would be to supply architecture-appropriate executables rather than relying on a compatibility layer. Furthermore, I dont believe the ARM architecture has a compatibility layer (meaning 64-bit executables are required for 64-bit ARM builds), so architecture and word size distinction is necessary. I believe youre right about leveraging the py launcher. Id like for setuptools to not have to supply launchers at all but depend on py launcher instead. The py launcher is bundled with Python 3.3 so should become ubiquitously available soon. I believe setuptools can begin to rely on it and not supply a launcher at all. The scripts currently installed by setuptools are suitable for launching by py launcher, so all that will need to happen is to stop supplying its own launcher. At least, thats how I imagine it happening. From: Paul Moore [mailto:p.f.moore@gmail.com] Sent: Tuesday, 16 July, 2013 09:32 To: Distutils; Jason R. Coombs Subject: Re: Wheels and console script entry point wrappers (Was: Replacing pip.exe with a Python script) On 16 July 2013 14:08, Paul Moore <p.f.moore@gmail.com <mailto:p.f.moore@gmail.com> > wrote: PS There is still the proviso that I haven't tested my assumption that the separate 32 and 64 bit wrappers are *needed* (setuptools and distlib use them, so I think it's a valid assumption, but I need to test). I will try to get time to check that ASAP. Hmm. I just did a quick test, and then based on the results checked the setuptools source code. I can see no reason why there needs to be 32 and 64 bit launcher exes. The launchers simply use CreateProcess to launch a separate Python process using the #! line of the script. So there's no DLL loading going on, and no reason that I can see for needing separate 32 and 64 bit builds. Jason - can you shed any light on why there are separate builds for 32 and 64 bits? Actually, the launcher is essentially identical to the "py" launcher for Windows, except that it gets a script name to execute from the name of the launcher. I'm wondering whether the correct approach here would be to enhance the launcher one more time to look for a suitably named script and auto-run it if it's present (i.e. merge the wrapper functionality into the launcher). Then we have a standard wrapper that everyone can use and not reinvent their own. Paul
participants (9)
-
Daniel Holth -
Donald Stufft -
Jason R. Coombs -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Ronald Oussoren -
Steve Dower -
Vinay Sajip