Is it possible (or desirable) that devpi be able to trigger builds of
wheels for packages that it caches? I realise it'd be possible as part of a
Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels,
we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu,
SUSE, etc. since the wheels are typically not compatible. That complexity
could be external to devpi - it's just a case of getting the trigger to
indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is
a little beyond me for the moment - the delay between the cache event and
build completing could be significant. In light of that it might be
desirable to have a separate process which does all this work (fetches the
latest package and performs the build). Has anyone done this? How would it
keep up to date?
Richard
Hi all,
several people asked me to convert the devpi repository from
hg/mercurial to git. Although i am mostly fine with hg/mercurial
myself i'd like to use git in some more depth. Therefore i plan
to convert the repository to git but continue to host it at bitbucket
for the time being. Probably going to happen sometime this week.
best,
holger
devpi-2.0.0: web, search, replication for PyPI indexes
==========================================================
The devpi system in version 2.0 brings tons of fixes
and new features for the private github-style pypi caching server,
most notably:
- a new web interface featuring search of metadata and documentation
as well as easier navigation and showing of test results per release
file.
- a new transactional storage system (based on sqlite) supporting
real-time replication over http.
- a new (experimental) hook system for extending server-side
functionality.
- ported to a solid web framework and wsgi-server: pyramid
and waitress.
Upgrade note: devpi-server-2.0 requires to ``--export`` your 1.2
server state and then using ``--import`` with the new version
before you can serve your private packages through devpi-server-2.0.0.
Also, please checkout the web plugin if you want to have a web interface::
http://doc.devpi.net/2.0/web.html
Here is a Quickstart tutorial for efficient pypi-mirroring
on your laptop::
http://doc.devpi.net/2.0/quickstart-pypimirror.html
And if you want to manage your releases or implement staging
as an individual or within an organisation::
http://doc.devpi.net/2.0/quickstart-releaseprocess.html
If you want to host a devpi-server installation with nginx/supervisor
and access it from clients from different hosts::
http://doc.devpi.net/2.0/quickstart-server.html
More documentation here::
http://doc.devpi.net/2.0/
many many thanks to Florian Schulze who implemented the new ``devpi-web``
package and helped with many other improvements.
have fun,
Holger Krekel, merlinux GmbH
2.0.0
--------------
devpi-server:
- major revamp of the internal core of devpi to support
replication (both master and server code), a plugin architecture
with the new devpi-web plugin providing a new web interface.
Mostly done by Florian Schulze and Holger Krekel.
- moved all html views except for files and the simple index to new
devpi-web package. Thanks to Florian Schulze for the PR.
- implement issue103: By default if you register a package in an index,
no lookup on pypi is made for that package anymore. You have to add the
package to the pypi_whitelist of the index to let pypi releases be mixed in.
This is to prevent malicious uploads on pypi to overwrite private packages.
- change json api to get rid of the different meaning of URLs with and
without a trailing slash. "/{user}/" is now the same as "/user" and always
lists indices. "/{user}/{index}" and "/{user}/{index}/ now always
lists the index config and the contained per-stage projects
(not inherited ones).
- switch the wsgi app to use Pyramid and waitress for WSGI serving.
- don't refresh releaselinks from the mirroring thread but rather
rely on the next access to do it.
- fix issue98: deleting a project config or a project version now accepts
names which map to the canonical name of a project.
- fix issue82 and fix issue81: root/pypi now provides the same
attributes as normal indexes and results in a 409 MethodNotAllowed
http code when trying to change the config.
- fix issue91: make serverport available as well. Thanks David Bonner.
- fix issue100: support large file uploads. As we switched away from
bottle to pyramid, the body-size limit is gone.
- fix issue99: make "devpi-server --start" etc work when devpi-server
is not itself on PATH (by using sys.argv[0] for finding the binary)
- fix issue84: uploading of wheels where the registered package name
has an underscore works despite a wheel's metadata carrying
hyphens instead. At submit-file time we now lookup the registered
name and use that instead of assuming the one coming with the wheel
is the correct one.
- add refresh button on root/pypi project simple index pages which clears the
internal cache to force a refetch from PyPI.
- implement issue75: We use the custom X-Devpi-Auth header for authentication
now, instead of overwriting the Authentication header.
- added experimental support for using client certificates when running as a
replica of a server running behind a proxy
devpi-client:
- Compatibility with devpi-server >= 2.0.0
- introduce "patchjson PATH JSONFILE" command which allows to send
a request containing a json data structure to a specified path
- fix issue85: "devpi list -v" now shows package names with latest
versions.
- implement issue75: We use the custom X-Devpi-Auth header for authentication
now, instead of overwriting the Authentication header.
- added experimental support for basic authentication by parsing user and
password from the url given to the "devpi use" command.
- issue74: added experimental support for client side certificates via
"devpi use --client-cert"
devpi-web:
- initial release
Hello,
I have a central devpi-server instance deployed on a vps on the cloud,
which I'd like to mirror locally on my laptop.
Is it currently possible ? I would appreciate if someone could point me the
right docs. I already searched without any success.
Cheers, and thanks for the nice work done so far !
We upload our nightly builds to our devpi server. There are a lot and the
files are big so I like to delete files over 30 days old (plus they are no
longer useful at that age).
What is the best way to do this in an automated fashion with devpi? Using
the devpi client to do it manually it not appealing.
I could script something "devpi remove -y
pkgname>=2014.6.dev1;<2014.7.dev1" that would remove all 30 files
2014.6.dev1...2014.6.dev30?
Any other ideas?
Cheers
Craig
Hi folks,
We are about to release devpi-2.0 with tons of improvements,
probably Monday :)
As we are squashing some remaining bugs i'd be interested if you
succeed in importing your devpi-server-1.2 repositories.
You can try that like this:
# use the current old version of devpi-server (1.2.X)
devpi-server --serverdir YOURSERVERDIR --export EXPORT
Then test out if importing with devpi-2.0 works for you::
virtualenv devpi2
source devpi2/bin/activate
pip install -i https://devpi.net/hpk/dev/+simple devpi-server devpi-web
devpi-server --import EXPORT --serverdir NEWSERVERDIR
devpi-server --serverdir NEWSERVERDIR --port 4000 --debug
Now you can go to your browser at http://localhost:4000 to see if
things work and all your things are there.
have a nice weekend!
holger
P.S.: docs are already online at http://doc.devpi.net/2.0/
Preliminary release announcement is here:
http://doc.devpi.net/2.0/announce/2.0.0.html
I reported this
bug: https://bitbucket.org/hpk42/devpi/issue/116/extdb-instance-has-no-attribute…
which was closed because there was a fix in trunk. Can you tell me a
released version where pushing works?
It looks line that has the bug:
entry = stage.xom.filestore.getentry(relpath)
has been there since 2013-09-23, added as part of the "also push docs to
internal or external indexes" commit. And it looks like 1.0 is the only
release that doesn't have this line. Is that the version I need to use to
get pushing to work?
Hi, I was wondering what the difference between type=mirror and type=stage
is.
Is it that:
- a mirror gets its list of package exclusively from its 'bases' and you
can't upload to it
- a stage allows you to upload to it and never looks at its list of bases
for packages
Or is it something else? I'm seeing the following behavior:
$ devpi index --create one type=stage bases=root/pypi
http://localhost/pypi/root/one:
type=stage
bases=root/pypi
volatile=True
uploadtrigger_jenkins=None
acl_upload=root
$ devpi index --create one type=stage bases=root/pypi
http://localhost/pypi/root/one:
type=stage
bases=root/pypi
volatile=True
uploadtrigger_jenkins=None
acl_upload=root
$ devpi list
$ devpi index --create two type=mirror bases=
http://localhost/pypi/root/two:
type=mirror
bases=
volatile=False
Traceback (most recent call last):
File "/usr/local/bin/devpi", line 9, in <module>
load_entry_point('devpi-common==1.2', 'console_scripts', 'devpi')()
File "/usr/local/lib/python2.7/site-packages/devpi/main.py", line 29, in
main
return method(hub, hub.args)
File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 63, in
main
return index_create(hub, url, kvdict)
File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 5, in
index_create
index_show(hub, url)
File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 36, in
index_show
ixconfig["uploadtrigger_jenkins"],))
KeyError: 'uploadtrigger_jenkins'
$ devpi use root/two
current devpi index: http://localhost/pypi/root/two/ (logged in as root)
~/.pydistutils.cfg : no config file exists
~/.pip/pip.conf : no config file exists
~/.buildout/default.cfg: no config file exists
always-set-cfg: no
$ devpi list
0x10c-asm
1009558_nester
131228_pytest_1
17MonIP
18-e
1ee
1pass
1to001
2013007_pyh
2C.py
... [Removed huge list a pacakges presumably from root/pypi]
$ devpi use -l
root/pypi bases= volatile=False
root/two bases= volatile=True
root/one bases=root/pypi volatile=True
Why didn't 'root/one' list any packages when it had 'root/pypi' as a base.
And, why did 'root/two' list a whole bunch of packages when didn't have any
bases? Is a 'mirror' always a mirror of the central pypi server? Can I not
have 'root/one' and 'root/two' be stage indexes and 'root/three' be a
mirror of just those two and not the central public pypi server? And why
did I get that 'uploadtrigger_jenkins' error?
