Automatic building of wheels? (discuss ;-)
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches the latest package and performs the build). Has anyone done this? How would it keep up to date?
Richard
My apologies, I somehow missed the open issue I just found in the tracker that captures this very issue:
https://bitbucket.org/hpk42/devpi/issue/110/build-put-wheel-for-pypi-release...
On 10 July 2014 11:52, Richard Jones r1char...@gmail.com wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches the latest package and performs the build). Has anyone done this? How would it keep up to date?
Richard
On Thu, Jul 10, 2014 at 11:52 +1000, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches the latest package and performs the build). Has anyone done this? How would it keep up to date?
I think it's fine to discuss it here because the desirable flow of events is not clear, to me at least.
My own current scenario looks like this:
- i have an Index X on a remote server which I use for uploading and preparing releases for open source packages.
- i want to install numpy and pandas but don't want to recompile on every installation but instead "cache" it in wheel format related to the computer i am working on.
Apart from triggering the building of the wheel the question is where or how is this wheel uploaded, exactly. If we just say I can also use index X for uploading, then the required functionality boils down to downloading numpy/pandas packages, creating the wheel and uploading it like a normal release file.
FYI "devpi test" already downloads a package, tests it and uploads test results back to the devpi server. That's a very similar flow of data. We might just consider a "devpi wheelify NAME" command which implements the downloading, building of wheel and uploading, reusing some of the "devpi test" code. It would just use the current index X (*).
If this were to somewhat happen transparently it's clear that the building needs to happen on the client side as devpi-2.0 doesn't have means to build packages and might run on a different platform anyway. I guess we could think about making "devpi install" grow build/cache-wheel functionality -- it is currently just a dumb wrapper around pip.
Any insights and thoughts on the matter welcome.
best, holger
(*) we could also think about uploading to another stage Y which might be a base for my X but more generally holds wheels of a certain platform. For example, users on ubuntu-14.04 64bit systems could always inherit from an index Y which holds wheels specific to that system. This could help with disambiguating platforms because i think the wheel filename does not contain enough information (thus pypi.python.org refuses to accept linux wheels IIRC).
Richard-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
On 10 July 2014 17:21, holger krekel hol...@merlinux.eu wrote:
On Thu, Jul 10, 2014 at 11:52 +1000, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part
of a
Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux
wheels,
we'd need to have separate indexes for the wheels for Debian, RHEL,
Ubuntu,
SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package"
is
a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches
the
latest package and performs the build). Has anyone done this? How would
it
keep up to date?
I think it's fine to discuss it here because the desirable flow of events is not clear, to me at least.
My own current scenario looks like this:
i have an Index X on a remote server which I use for uploading and preparing releases for open source packages.
i want to install numpy and pandas but don't want to recompile on every installation but instead "cache" it in wheel format related to the computer i am working on.
Apart from triggering the building of the wheel the question is where or how is this wheel uploaded, exactly. If we just say I can also use index X for uploading, then the required functionality boils down to downloading numpy/pandas packages, creating the wheel and uploading it like a normal release file.
FYI "devpi test" already downloads a package, tests it and uploads test results back to the devpi server. That's a very similar flow of data. We might just consider a "devpi wheelify NAME" command which implements the downloading, building of wheel and uploading, reusing some of the "devpi test" code. It would just use the current index X (*).
This is very similar to how I was thinking - a "devpi wheelify" command - or perhaps a more general "devpi build" command. It'd have some of the same options as "devpi upload" in that it could build a variety of bnary formats (I see no reason to limit such a feature to just wheels).
If this were to somewhat happen transparently it's clear that the building needs to happen on the client side as devpi-2.0 doesn't have means to build packages and might run on a different platform anyway. I guess we could think about making "devpi install" grow build/cache-wheel functionality -- it is currently just a dumb wrapper around pip.
I'm not convinced that having the wheel building happen transparently is a good idea at all. Or even possible, in a lot of cases, as you'd need to move users over from using pip to install to using devpi to install and that might be a barrier that's too high.
I'd be happy enough with a hook that could fire (when a new source file is cached) an installation-specific build process that can build all the wheels (or exe installers or whatever) that a particular installation requires.
(*) we could also think about uploading to another stage Y which might
be a base for my X but more generally holds wheels of a certain platform. For example, users on ubuntu-14.04 64bit systems could always inherit from an index Y which holds wheels specific to that system. This could help with disambiguating platforms because i think the wheel filename does not contain enough information (thus pypi.python.org refuses to accept linux wheels IIRC).
Yep, linux wheels are a problem and this is the solution I arrived at :)
The linux wheel problem needs to be solved in discussion with the linux vendors (or more specifically their downstream packagers) and I hope to have some of this discussion with some of them at EuroPython. Whatever the solution, it's probably not going to be arrived at very quickly :(
Richard
On Thursday, 10 July 2014 02:52:31 UTC+1, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
What I would like is actually somewhat simpler. I'd like to be able to set things up so that whenever devpi serves an egg or wininst file, it also serves the equivalent wheel (by running wheel convert). The "obvious" way to do this would be whenever devpi mirrors a project from PyPI it (or a hook/plugin) runs wheel convert on any exe and egg files, and caches the resulting wheels as if they had come from PyPI.
Alternatively, and just as good from my POV, would be for the plugin to add the wheels to a local index of my choosing. That would preserve /root/pypi as a PyPI mirror, but I could add my wheel index as an extra index URL, to get the wheels that way.
Paul
On Thu, Oct 09, 2014 at 07:26 -0700, Paul Moore wrote:
On Thursday, 10 July 2014 02:52:31 UTC+1, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
What I would like is actually somewhat simpler. I'd like to be able to set things up so that whenever devpi serves an egg or wininst file, it also serves the equivalent wheel (by running wheel convert). The "obvious" way to do this would be whenever devpi mirrors a project from PyPI it (or a hook/plugin) runs wheel convert on any exe and egg files, and caches the resulting wheels as if they had come from PyPI.
Alternatively, and just as good from my POV, would be for the plugin to add the wheels to a local index of my choosing. That would preserve /root/pypi as a PyPI mirror, but I could add my wheel index as an extra index URL, to get the wheels that way.
The second option seems to fit better how things work internally currently. So something like:
1. setup X/pypi_with_wheels, inherit from root/pypi 2. implement a wheel builder plugin that implements a new devpi_server_on_pypi_upload(link) hook which triggers an async script that does download, convert, upload to root/pypi_with_wheels.
should be implementable without too much effort.
Somehow it'd be nice, i guess, if we could make things more real-time, i.e. you would always see wheel links on the simple page to begin with. There would be wheels made out of sdists, exe, whatnot and they would be created dynamically when actually accessed. This would probably make most sense for per-machine devpi deployments. Implementing this requires probably some more effort.
best, holger
On 9 October 2014 16:02, holger krekel hol...@merlinux.eu wrote:
Alternatively, and just as good from my POV, would be for the plugin to add the wheels to a local index of my choosing. That would preserve /root/pypi as a PyPI mirror, but I could add my wheel index as an extra index URL, to get the wheels that way.
The second option seems to fit better how things work internally currently. So something like:
- setup X/pypi_with_wheels, inherit from root/pypi
- implement a wheel builder plugin that implements a new devpi_server_on_pypi_upload(link) hook which triggers an async script that does download, convert, upload to root/pypi_with_wheels.
should be implementable without too much effort.
Cool. When you say "implements a new hook", can plugins implement hooks? If that's what you mean, I'll give it a go and see if I can write something (expect requests for help, I've not looked into writing plugins yet!) If you mean that devpi would need to implement a new hook internally first, I'm not sure I know enough to be able to do that...
Somehow it'd be nice, i guess, if we could make things more real-time, i.e. you would always see wheel links on the simple page to begin with. There would be wheels made out of sdists, exe, whatnot and they would be created dynamically when actually accessed. This would probably make most sense for per-machine devpi deployments. Implementing this requires probably some more effort.
(Near) real-time would be best for me. My ideal would be (for example) to have "pip install lxml" install a wheel built from the exe that's available on PyPI. If the wheel convert happens asynchronously, that would mean the first install would fail, but if I repeated the install it would work.
Paul
On Thu, Oct 09, 2014 at 16:25 +0100, Paul Moore wrote:
On 9 October 2014 16:02, holger krekel hol...@merlinux.eu wrote:
Alternatively, and just as good from my POV, would be for the plugin to add the wheels to a local index of my choosing. That would preserve /root/pypi as a PyPI mirror, but I could add my wheel index as an extra index URL, to get the wheels that way.
The second option seems to fit better how things work internally currently. So something like:
- setup X/pypi_with_wheels, inherit from root/pypi
- implement a wheel builder plugin that implements a new devpi_server_on_pypi_upload(link) hook which triggers an async script that does download, convert, upload to root/pypi_with_wheels.
should be implementable without too much effort.
Cool. When you say "implements a new hook", can plugins implement hooks? If that's what you mean, I'll give it a go and see if I can write something (expect requests for help, I've not looked into writing plugins yet!) If you mean that devpi would need to implement a new hook internally first, I'm not sure I know enough to be able to do that...
Indeed, you'd need to implement a new hook within devpi-server. They are defined in devpi_server/config.py's PluginManager. And would need to be called from views.py's pkgserv where it calls into entry.cache_remote_file() -- entry.url points to the remote file and entry.basename is the raw basename.
If you rather open an issue to introduce this hook and specify the info/semantics that you think are needed from the plugin, i can see to get to it sometime (but no time promises).
Somehow it'd be nice, i guess, if we could make things more real-time, i.e. you would always see wheel links on the simple page to begin with. There would be wheels made out of sdists, exe, whatnot and they would be created dynamically when actually accessed. This would probably make most sense for per-machine devpi deployments. Implementing this requires probably some more effort.
(Near) real-time would be best for me. My ideal would be (for example) to have "pip install lxml" install a wheel built from the exe that's available on PyPI. If the wheel convert happens asynchronously, that would mean the first install would fail, but if I repeated the install it would work.
real-time could be done maybe using two new plugin points:
- devpi_server_get_releaselinks(stage) which returns the links to release files and is used to create the simple page for an index. A plugin could add canonical wheel filenames, generated from the existing links.
- devpi_server_retrieve_file(entry) would remotely retrieve or build a file -- this happens during access to the release file so if it takes very long (lxml, ...) the "pip install" command would time out, however.
Because of the time out issue, it may make sense to rather stick with the first async and overall somewhat simpler approach.
best, holger
Paul
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.
On 9 October 2014 16:41, holger krekel hol...@merlinux.eu wrote:
f you rather open an issue to introduce this hook and specify the info/semantics that you think are needed from the plugin, i can see to get to it sometime (but no time promises).
Thanks for all the information. I'll let it sink in, and maybe see if I can see what would be needed to implement a new hook. If I get anywhere I'll open a PR, otherwise I'll just file an issue and wait for you :-)
Paul
On Thu, Oct 09, 2014 at 19:35 +0100, Paul Moore wrote:
On 9 October 2014 16:41, holger krekel hol...@merlinux.eu wrote:
f you rather open an issue to introduce this hook and specify the info/semantics that you think are needed from the plugin, i can see to get to it sometime (but no time promises).
Thanks for all the information. I'll let it sink in, and maybe see if I can see what would be needed to implement a new hook. If I get anywhere I'll open a PR, otherwise I'll just file an issue and wait for you :-)
There is one thought that might be interesting in addition. If we are talking about a setting where devpi-server runs on your local machine where you are also issuing pip install commands i wonder if you could use some wheelhouse functionality and have devpi-server pick the local wheel files up and integrate it into the index. Or would that not make sense in your situation?
holger
On 9 October 2014 19:39, holger krekel hol...@merlinux.eu wrote:
On Thu, Oct 09, 2014 at 19:35 +0100, Paul Moore wrote:
On 9 October 2014 16:41, holger krekel hol...@merlinux.eu wrote:
f you rather open an issue to introduce this hook and specify the info/semantics that you think are needed from the plugin, i can see to get to it sometime (but no time promises).
Thanks for all the information. I'll let it sink in, and maybe see if I can see what would be needed to implement a new hook. If I get anywhere I'll open a PR, otherwise I'll just file an issue and wait for you :-)
There is one thought that might be interesting in addition. If we are talking about a setting where devpi-server runs on your local machine where you are also issuing pip install commands i wonder if you could use some wheelhouse functionality and have devpi-server pick the local wheel files up and integrate it into the index. Or would that not make sense in your situation?
That's not a bad idea. I had done that in the past (it doesn't actually need devpi, in fact, using pip's --find-links option to point to a local wheelhouse directory works fine). The issue is that I tend not to remember to update the wheelhouse, and I never found a good way of doing so automatically. So "pip install lxml" would fail because a new source version has been released, so the wheel is out of date and gets ignored.
I'm also looking at solutions within pip to make this easier. There's https://github.com/pypa/pip/issues/2084 (allow the user to say that certain packages must use wheels, avoids the "out of date" issue) and the currently inactive proposal for pip to build and cache wheels if needed on install.
We'll see which solution works out best :-) Paul.
On 10 October 2014 01:26, Paul Moore p.f....@gmail.com wrote:
On Thursday, 10 July 2014 02:52:31 UTC+1, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
What I would like is actually somewhat simpler. I'd like to be able to set things up so that whenever devpi serves an egg or wininst file, it also serves the equivalent wheel (by running wheel convert). The "obvious" way to do this would be whenever devpi mirrors a project from PyPI it (or a hook/plugin) runs wheel convert on any exe and egg files, and caches the resulting wheels as if they had come from PyPI.
I must have not been clear but that was basically what I was asking for :)
[with the caveat I mentioned: timing could be tricky if the build takes a while]
I've since used devpi-builder with some success. It avoids the timing issue by having the wheel build be a separate process. Unfortunately a bug in pip (PR still under review https://github.com/pypa/pip/pull/1965) meant that devpi-builder is less useful than it could be (it currently requires versions in requirements.txt to be pinned).
Richard
On 12 October 2014 23:16, Richard Jones r1char...@gmail.com wrote:
On 10 October 2014 01:26, Paul Moore p.f....@gmail.com wrote:
On Thursday, 10 July 2014 02:52:31 UTC+1, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
What I would like is actually somewhat simpler. I'd like to be able to set things up so that whenever devpi serves an egg or wininst file, it also serves the equivalent wheel (by running wheel convert). The "obvious" way to do this would be whenever devpi mirrors a project from PyPI it (or a hook/plugin) runs wheel convert on any exe and egg files, and caches the resulting wheels as if they had come from PyPI.
I must have not been clear but that was basically what I was asking for :)
[with the caveat I mentioned: timing could be tricky if the build takes a while]
Sorry, you were clear, it's just that I then read the referenced issue which confused me (as it went onto talking about whitelists), and by the time I got back to posting here I'd managed to misread what you wrote...
I've since used devpi-builder with some success. It avoids the timing issue by having the wheel build be a separate process. Unfortunately a bug in pip (PR still under review https://github.com/pypa/pip/pull/1965) meant that devpi-builder is less useful than it could be (it currently requires versions in requirements.txt to be pinned).
I saw the link, but never followed up on it. Will do so now, thanks!
Is there an issue filed related to that PR? I can't follow the PR in isolation, but I'll take a look at it.
Paul
Sorry, no bug filed, just the explanation in my second comment on the PR.
On 13 October 2014 09:24, Paul Moore p.f....@gmail.com wrote:
On 12 October 2014 23:16, Richard Jones r1char...@gmail.com wrote:
On 10 October 2014 01:26, Paul Moore p.f....@gmail.com wrote:
On Thursday, 10 July 2014 02:52:31 UTC+1, Richard Jones wrote:
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part
of a
Jenkins trigger for an upload, but this is for cached packages from
PyPI.
What I would like is actually somewhat simpler. I'd like to be able to
set
things up so that whenever devpi serves an egg or wininst file, it also serves the equivalent wheel (by running wheel convert). The "obvious"
way to
do this would be whenever devpi mirrors a project from PyPI it (or a hook/plugin) runs wheel convert on any exe and egg files, and caches the resulting wheels as if they had come from PyPI.
I must have not been clear but that was basically what I was asking for
:)
[with the caveat I mentioned: timing could be tricky if the build takes a while]
Sorry, you were clear, it's just that I then read the referenced issue which confused me (as it went onto talking about whitelists), and by the time I got back to posting here I'd managed to misread what you wrote...
I've since used devpi-builder with some success. It avoids the timing
issue
by having the wheel build be a separate process. Unfortunately a bug in
pip
(PR still under review https://github.com/pypa/pip/pull/1965) meant that devpi-builder is less useful than it could be (it currently requires versions in requirements.txt to be pinned).
I saw the link, but never followed up on it. Will do so now, thanks!
Is there an issue filed related to that PR? I can't follow the PR in isolation, but I'll take a look at it.
Paul
participants (3)
-
holger krekel -
Paul Moore -
Richard Jones