setuptools develop command garbles binary files specified as scripts setup() parameter on windows
Hi, In our environment, we are using the scripts parameter to package external binary files, which is really a practical use case: 1) it allows for bundling of executables in our package python distribution (with pertaining dlls etc) 2) the binary is available in an activated virtualenv For the install case, everything is ok, the binaries are correctly installed. But for the develop command, setuptools garbles the binary by copying the binary file opened in text mode and writing it in binary mode. This is because of the line ending translation. I implemented a naive fix, only to be surprised by the python3 unicode management, and caused a regression. (see https://bitbucket.org/pypa/setuptools/issue/210/on-windows-binary-files-spec...) The less naive approach to the problem was implemented in https://bitbucket.org/pypa/setuptools/pull-request/60/fix-regression-in-issu... but the pull request was rejected. But the problem still stands. How can I use the develop command and not have my binaries specified as scripts be garbled. Jason R. Coombs argued that this change would add undocumented special casing, but I am inclined to disagree because the use case works: * on *nix platforms (because the open function does not translate line endings (the "b" mode has no effect on this platform)) * it already works when we go through the install flow (on every platform) * there is a discrepancy between install and develop semantics, for the scripts parameter I would like to know what is the take in the community on this. Note that this is the first time I contribute a change, so I may not be up to speed with all the dos and don'ts. Thanks. David.
On 17 July 2014 17:04, David Genest <david.genest@ubisoft.com> wrote:
I would like to know what is the take in the community on this.
My view would be: 1. The scripts argument is for *scripts* not binary files, so it is perfectly correct to open them in text mode, do line ending translation, etc. 2. What you're doing is not supported behaviour (although I appreciate the fact that it's useful to you). 3. The scripts feature is generally discouraged in favour of entry points. 4. By including binaries, you make your package non-portable, and do so in a way that setuptools/wheel/pip cannot spot, so (for example) if you were to build a wheel it would not be recognised as architecture-specific. You can probably include a project-specific hack if you're so inclined, but this shouldn't be added as a feature of setuptools. Longer term, maybe your use case is something that we could support via Metadata 2.0. I'm honestly not sure at the moment, as your description isn't very specific. What "external binary files" do you package? If they are DLLs (as you mention) how would they work on Unix? I'm a bit confused here, as you mention the script handling being OK for you on Unix, and yet you're talking about DLLs, which are for Windows... If this is intended to be a single-platform solution, how do you anticipate communicating that fact to the build and install tools (wheel, pip, etc)? Thanks for raising the issue, it sounds like you have a use case that isn't well supported at the moment. But I agree with Jason that this shouldn't be forced into the scripts functionality, which is, after all, intended for *scripts*. Paul
On 17 Jul 2014 15:10, "Paul Moore" <p.f.moore@gmail.com> wrote:
Longer term, maybe your use case is something that we could support via Metadata 2.0.
For the record, the current draft of the python.commands extension in PEP 459 does indeed include support for reporting prebuilt commands: http://www.python.org/dev/peps/pep-0459/#the-python-commands-extension The draft metadata extensions haven't really been properly reviewed yet, though. For pip 1.6, Donald and I have been focused on hammering PEP 440 into shape. IIRC, we had one outstanding issue to resolve before bringing that back to the list, but travel has currently fried my brain, so I don't recall off the top of my head what it was. Cheers, Nick.
I'm honestly not sure at the moment, as your description isn't very specific. What "external binary files" do you package? If they are DLLs (as you mention) how would they work on Unix? I'm a bit confused here, as you mention the script handling being OK for you on Unix, and yet you're talking about DLLs, which are for Windows... If this is intended to be a single-platform solution, how do you anticipate communicating that fact to the build and install tools (wheel, pip, etc)?
Thanks for raising the issue, it sounds like you have a use case that isn't well supported at the moment. But I agree with Jason that this shouldn't be forced into the scripts functionality, which is, after all, intended for *scripts*.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 17 July 2014 17:04, David Genest <david.genest@ubisoft.com> wrote:
I would like to know what is the take in the community on this.
My view would be:
1. The scripts argument is for *scripts* not binary files, so it is perfectly correct to open them in text mode, do line ending translation, etc.
I see that. But on windows, beside naming, the scripts argument is used to populate the Scripts directory (which are not only scripts, but sometimes setuptools wrappers - clearly executable). Cross-platform can be achieved by bundling unix and windows variants at setup.py time.
2. What you're doing is not supported behaviour (although I appreciate the fact that it's useful to you).
It is de-facto. We have been using this "functionality" to package our windows only binary executables in an egg (using install). Wheels are supported also (but I agree, pip or wheel cannot infer the platform).
3. The scripts feature is generally discouraged in favour of entry points. 4. By including binaries, you make your package non-portable, and do so in a way that setuptools/wheel/pip cannot spot, so (for example) if you were to build a wheel it would not be recognised as architecture-specific.
I agree that they would not be portable, but in a Windows only situation, this does not pose a problem. I'm not using the feature to publish a cross-platform solution, it is used to publish a package in-house. But the current setuptools tooling is perfectly adapted to this. Also, the binary delivery mechanism is a needed feature, especially on windows. The current API permits it's use now (on unix, or more precisely, platforms where the 'b' mode has no effect when opening files). No checks are being performed. Why should it be different on Windows ?
very specific. What "external binary files" do you package? If they are DLLs
In our situation, it is an executable embedding python, but it could be any binary auxiliary executable useful for the package. Dlls are only another binary example of what is needed by the package (and loadable by windows because on the path).
(as you mention) how would they work on Unix? I'm a bit confused here, as you mention the script handling being OK for you on Unix, and yet you're talking about DLLs, which are for Windows... If this is intended to be a single- platform solution, how do you anticipate communicating that fact to the build and install tools (wheel, pip, etc)?
I was probably too vague. I meant that the existing functionality supports binary files on unix, albeit not in a cross-platform way. In other words, if you specify a binary file in scripts, it will be copied over without being broken. It is not the case on Windows. I am not up to date with metadata, but I agree that it should be communicated in the tools.
Thanks for raising the issue, it sounds like you have a use case that isn't well supported at the moment. But I agree with Jason that this shouldn't be forced into the scripts functionality, which is, after all, intended for *scripts*.
Acknowledging that the current script solution is not for cross-platform transportation of binaries, the problem is that it is difficult to package arbitrary binary executables that are available in a virtualenv. Helper executables, or any other auxiliary executables. Right now, AFAIK, on python2, (and python3) it is not possible to publish a binary executable by other means than the script parameter. Knowing that the install works right now, I would think that the develop command should at least behave the same. David.
For the record, the current draft of the python.commands extension in PEP 459 does indeed include support for reporting prebuilt commands: http://www.python.org/dev/peps/pep-0459/#the-python-commands-extension The draft metadata extensions haven't really been properly reviewed yet, though. For pip 1.6, Donald and I have been focused on hammering PEP 440 into shape.
I agree that this would help, and in fact solve the problem. But only in newer python releases.
On 17 Jul 2014 16:15, "David Genest" <david.genest@ubisoft.com> wrote:
For the record, the current draft of the python.commands extension in
PEP 459 does indeed include support for reporting prebuilt
commands: http://www.python.org/dev/peps/pep-0459/#the-python-commands-extension The draft metadata extensions haven't really been properly reviewed yet, though. For pip 1.6, Donald and I have been focused on hammering PEP 440 into shape.
I agree that this would help, and in fact solve the problem. But only in newer python releases.
No, the focus of packaging efforts has moved to pip and setuptools specifically to avoid that problem - when metadata 2.0 is adopted, it will be supported at least as far back as Python 2.7, and likely on 2.6 as well. The only requirement will be to use setuptools (or another metadata 2.0 compatible build system) and pip (or another metadata 2.0 compatible installer) rather than using distutils directly. Cheers, Nick.
participants (3)
-
David Genest -
Nick Coghlan -
Paul Moore