On 10 July 2014 17:21, holger krekel
Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches
On Thu, Jul 10, 2014 at 11:52 +1000, Richard Jones wrote: the
latest package and performs the build). Has anyone done this? How would it keep up to date?
I think it's fine to discuss it here because the desirable flow of events is not clear, to me at least.
My own current scenario looks like this:
- i have an Index X on a remote server which I use for uploading and preparing releases for open source packages.
- i want to install numpy and pandas but don't want to recompile on every installation but instead "cache" it in wheel format related to the computer i am working on.
Apart from triggering the building of the wheel the question is where or how is this wheel uploaded, exactly. If we just say I can also use index X for uploading, then the required functionality boils down to downloading numpy/pandas packages, creating the wheel and uploading it like a normal release file.
FYI "devpi test" already downloads a package, tests it and uploads test results back to the devpi server. That's a very similar flow of data. We might just consider a "devpi wheelify NAME" command which implements the downloading, building of wheel and uploading, reusing some of the "devpi test" code. It would just use the current index X (*).
This is very similar to how I was thinking - a "devpi wheelify" command - or perhaps a more general "devpi build" command. It'd have some of the same options as "devpi upload" in that it could build a variety of bnary formats (I see no reason to limit such a feature to just wheels).
If this were to somewhat happen transparently it's clear that the building needs to happen on the client side as devpi-2.0 doesn't have means to build packages and might run on a different platform anyway. I guess we could think about making "devpi install" grow build/cache-wheel functionality -- it is currently just a dumb wrapper around pip.
I'm not convinced that having the wheel building happen transparently is a good idea at all. Or even possible, in a lot of cases, as you'd need to move users over from using pip to install to using devpi to install and that might be a barrier that's too high. I'd be happy enough with a hook that could fire (when a new source file is cached) an installation-specific build process that can build all the wheels (or exe installers or whatever) that a particular installation requires. (*) we could also think about uploading to another stage Y which might
be a base for my X but more generally holds wheels of a certain platform. For example, users on ubuntu-14.04 64bit systems could always inherit from an index Y which holds wheels specific to that system. This could help with disambiguating platforms because i think the wheel filename does not contain enough information (thus pypi.python.org refuses to accept linux wheels IIRC).
Yep, linux wheels are a problem and this is the solution I arrived at :) The linux wheel problem needs to be solved in discussion with the linux vendors (or more specifically their downstream packagers) and I hope to have some of this discussion with some of them at EuroPython. Whatever the solution, it's probably not going to be arrived at very quickly :( Richard