Building local versions with pip wheel
Hey all, Patching 3rd party repositories comes up every so often at $WORK and one of the things we do is build a local version that is generally some released version + a couple of local patches. We follow the usual version scheme of using the public version identifier + local version label. The steps are as follows: python setup.py egg_info -b "+local<build number>" sdist bdist_wheel twine upload dist/*.whl We only really care about the wheel though, and for all of our other dependencies we basically just run: pip wheel <some pip valid URL> and upload it to our local pypi instance. This also allows us to directly build from Github repositories. With for example: pip wheel git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo <git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo> However we would like to add the build number so that when a new release is provided upstream we can easily update our local repository (or preferably we have upstreamed our patch in the meantime) and build a new release with our patches. Is there some way to influence the egg info for the build when using pip wheel? Some thing like: pip wheel git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo&egg_info= <git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo&egg_info=>"-b +local<build number>" Or is there a better method for dealing with this scenario? Thanks, Bert JW Regeer
On Thu, 3 Jan 2019 at 11:13, Bert JW Regeer <xistence@0x58.com> wrote:
Is there some way to influence the egg info for the build when using pip wheel? Some thing like:
pip wheel git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo&egg_info="-b +local<build number>"
Or is there a better method for dealing with this scenario?
The way auditwheel approaches this is as a "post-process the already built wheel" problem, rather than as a "modify the wheel build process" problem: 1. Build the original wheel archive however you'd normally do so 2. Unpack the original wheel archive 3. Make any desired changes to wheel contents and/or metadata 4. Repack the modified wheel (potentially with an updated RECORD file and modified filename) auditwheel's own utilities for doing that kind of thing are here: https://github.com/pypa/auditwheel/blob/master/auditwheel/wheeltools.py Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, 4 Jan 2019 at 16:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, 3 Jan 2019 at 11:13, Bert JW Regeer <xistence@0x58.com> wrote:
Is there some way to influence the egg info for the build when using pip wheel? Some thing like:
pip wheel git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo&egg_info="-b +local<build number>"
Or is there a better method for dealing with this scenario?
The way auditwheel approaches this is as a "post-process the already built wheel" problem, rather than as a "modify the wheel build process" problem:
1. Build the original wheel archive however you'd normally do so 2. Unpack the original wheel archive 3. Make any desired changes to wheel contents and/or metadata 4. Repack the modified wheel (potentially with an updated RECORD file and modified filename)
auditwheel's own utilities for doing that kind of thing are here: https://github.com/pypa/auditwheel/blob/master/auditwheel/wheeltools.py
It's also worth noting that distlib offers a helper API for modifying wheel archives without needing to write your own archive unpacking and repacking code: https://distlib.readthedocs.io/en/latest/tutorial.html#modifying-wheels Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jan 4, 2019, at 12:36, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Fri, 4 Jan 2019 at 16:50, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Thu, 3 Jan 2019 at 11:13, Bert JW Regeer <xistence@0x58.com> wrote:
Is there some way to influence the egg info for the build when using pip wheel? Some thing like:
pip wheel git+https://github.com/org/somerepo.git@ourpatchset#egg=somerepo&egg_info="-b +local<build number>"
Or is there a better method for dealing with this scenario?
The way auditwheel approaches this is as a "post-process the already built wheel" problem, rather than as a "modify the wheel build process" problem:
1. Build the original wheel archive however you'd normally do so 2. Unpack the original wheel archive 3. Make any desired changes to wheel contents and/or metadata 4. Repack the modified wheel (potentially with an updated RECORD file and modified filename)
auditwheel's own utilities for doing that kind of thing are here: https://github.com/pypa/auditwheel/blob/master/auditwheel/wheeltools.py
It's also worth noting that distlib offers a helper API for modifying wheel archives without needing to write your own archive unpacking and repacking code: https://distlib.readthedocs.io/en/latest/tutorial.html#modifying-wheels
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/ZRKT4...
Thanks for this information Nick, this is very helpful. The other question I have is related to automated version number changes when using CI to do builds. At the moment we use Jenkins and we use the Jenkins build number to use egg_info with -b as well. Modifying the wheel after the fact just to add the version number seems a little silly, especially since we are likely building from source in the first place, but there is no way (other than --global-options, but that disables all wheel support in pip) to tell pip wheel to increase the version number. This really precludes us from having all of the features that pip wheel brings, such as isolated build environments and other such goodies (or even PEP-518/517 support). Would it make sense to add an optional extra_version or something along those lines to PEP-517's config_settings that is to be used by build backends to append to the existing version identifier? I can automate things like bumpversion, but setup tools has supported passing daily builds/tags for a while now and it feels weird that we are going backwards with the new interfaces where it is made more difficult to do that in an easy/sane manner. Especially in the case of continuous integration/deployment where there is no explicit version bump anymore and the code is always moving forward. The version number of or code base has been perpetually stuck at 1.0 with the only thing increasing is the build number making it: 1.0.<build number>. A new commit per build just to increase the version (thereby triggering CI again... causing fun loops ;-), especially for branch builds where a developer may still be making more changes and the build number just needs to increase so that each build of a branch has a unique version number. I will admit that I have yet to test if pip passes through the environment un-modified, because in that case I may just end up using the environment variable in setup.py directly. Bert JW Regeer
On Wed, 13 Feb 2019 at 21:10, Bert JW Regeer <xistence@0x58.com> wrote:
I will admit that I have yet to test if pip passes through the environment un-modified, because in that case I may just end up using the environment variable in setup.py directly.
While PEP 517 doesn't specifically call this out, build frontends pretty much need to ensure that environment variables will be passed through faithfully to build backends, otherwise language independent settings like SOURCE_DATE_EPOCH won't work properly, and neither will the environment variables for non-Python build tools invoked from the build backend. The pep517 helper library at the very least works that way: https://github.com/pypa/pep517/blob/2f97e1bd61e63a1c2d3979cafe7fd8c7689c9bde... Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (2)
-
Bert JW Regeer -
Nick Coghlan