Mailman 3 July 2014 - devpi-dev

Automatic building of wheels? (discuss ;-)
by Richard Jones 12 Oct '14

12 Oct '14

Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI. The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired. Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches the latest package and performs the build). Has anyone done this? How would it keep up to date? Richard

3 13

hg->git conversion of devpi
by holger krekel 01 Sep '14

01 Sep '14

Hi all, several people asked me to convert the devpi repository from hg/mercurial to git. Although i am mostly fine with hg/mercurial myself i'd like to use git in some more depth. Therefore i plan to convert the repository to git but continue to host it at bitbucket for the time being. Probably going to happen sometime this week. best, holger

3 3

devpi-2.0.0: web/search UI, replication, fixes
by holger krekel 16 Jul '14

16 Jul '14

devpi-2.0.0: web, search, replication for PyPI indexes ========================================================== The devpi system in version 2.0 brings tons of fixes and new features for the private github-style pypi caching server, most notably: - a new web interface featuring search of metadata and documentation as well as easier navigation and showing of test results per release file. - a new transactional storage system (based on sqlite) supporting real-time replication over http. - a new (experimental) hook system for extending server-side functionality. - ported to a solid web framework and wsgi-server: pyramid and waitress. Upgrade note: devpi-server-2.0 requires to ``--export`` your 1.2 server state and then using ``--import`` with the new version before you can serve your private packages through devpi-server-2.0.0. Also, please checkout the web plugin if you want to have a web interface:: http://doc.devpi.net/2.0/web.html Here is a Quickstart tutorial for efficient pypi-mirroring on your laptop:: http://doc.devpi.net/2.0/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/2.0/quickstart-releaseprocess.html If you want to host a devpi-server installation with nginx/supervisor and access it from clients from different hosts:: http://doc.devpi.net/2.0/quickstart-server.html More documentation here:: http://doc.devpi.net/2.0/ many many thanks to Florian Schulze who implemented the new ``devpi-web`` package and helped with many other improvements. have fun, Holger Krekel, merlinux GmbH 2.0.0 -------------- devpi-server: - major revamp of the internal core of devpi to support replication (both master and server code), a plugin architecture with the new devpi-web plugin providing a new web interface. Mostly done by Florian Schulze and Holger Krekel. - moved all html views except for files and the simple index to new devpi-web package. Thanks to Florian Schulze for the PR. - implement issue103: By default if you register a package in an index, no lookup on pypi is made for that package anymore. You have to add the package to the pypi_whitelist of the index to let pypi releases be mixed in. This is to prevent malicious uploads on pypi to overwrite private packages. - change json api to get rid of the different meaning of URLs with and without a trailing slash. "/{user}/" is now the same as "/user" and always lists indices. "/{user}/{index}" and "/{user}/{index}/ now always lists the index config and the contained per-stage projects (not inherited ones). - switch the wsgi app to use Pyramid and waitress for WSGI serving. - don't refresh releaselinks from the mirroring thread but rather rely on the next access to do it. - fix issue98: deleting a project config or a project version now accepts names which map to the canonical name of a project. - fix issue82 and fix issue81: root/pypi now provides the same attributes as normal indexes and results in a 409 MethodNotAllowed http code when trying to change the config. - fix issue91: make serverport available as well. Thanks David Bonner. - fix issue100: support large file uploads. As we switched away from bottle to pyramid, the body-size limit is gone. - fix issue99: make "devpi-server --start" etc work when devpi-server is not itself on PATH (by using sys.argv[0] for finding the binary) - fix issue84: uploading of wheels where the registered package name has an underscore works despite a wheel's metadata carrying hyphens instead. At submit-file time we now lookup the registered name and use that instead of assuming the one coming with the wheel is the correct one. - add refresh button on root/pypi project simple index pages which clears the internal cache to force a refetch from PyPI. - implement issue75: We use the custom X-Devpi-Auth header for authentication now, instead of overwriting the Authentication header. - added experimental support for using client certificates when running as a replica of a server running behind a proxy devpi-client: - Compatibility with devpi-server >= 2.0.0 - introduce "patchjson PATH JSONFILE" command which allows to send a request containing a json data structure to a specified path - fix issue85: "devpi list -v" now shows package names with latest versions. - implement issue75: We use the custom X-Devpi-Auth header for authentication now, instead of overwriting the Authentication header. - added experimental support for basic authentication by parsing user and password from the url given to the "devpi use" command. - issue74: added experimental support for client side certificates via "devpi use --client-cert" devpi-web: - initial release

1 1

Index inheritance across devpi-server instances
by Cristiano Cortezia 15 Jul '14

15 Jul '14

Hello, I have a central devpi-server instance deployed on a vps on the cloud, which I'd like to mirror locally on my laptop. Is it currently possible ? I would appreciate if someone could point me the right docs. I already searched without any success. Cheers, and thanks for the nice work done so far !

2 3

A scriptable approach to deleting many packages
by Craig Bruce 12 Jul '14

12 Jul '14

We upload our nightly builds to our devpi server. There are a lot and the files are big so I like to delete files over 30 days old (plus they are no longer useful at that age). What is the best way to do this in an automated fashion with devpi? Using the devpi client to do it manually it not appealing. I could script something "devpi remove -y pkgname>=2014.6.dev1;<2014.7.dev1" that would remove all 30 files 2014.6.dev1...2014.6.dev30? Any other ideas? Cheers Craig

2 1

2.0 about to be released / please try import with candidates
by holger krekel 11 Jul '14

11 Jul '14

Hi folks, We are about to release devpi-2.0 with tons of improvements, probably Monday :) As we are squashing some remaining bugs i'd be interested if you succeed in importing your devpi-server-1.2 repositories. You can try that like this: # use the current old version of devpi-server (1.2.X) devpi-server --serverdir YOURSERVERDIR --export EXPORT Then test out if importing with devpi-2.0 works for you:: virtualenv devpi2 source devpi2/bin/activate pip install -i https://devpi.net/hpk/dev/+simple devpi-server devpi-web devpi-server --import EXPORT --serverdir NEWSERVERDIR devpi-server --serverdir NEWSERVERDIR --port 4000 --debug Now you can go to your browser at http://localhost:4000 to see if things work and all your things are there. have a nice weekend! holger P.S.: docs are already online at http://doc.devpi.net/2.0/ Preliminary release announcement is here: http://doc.devpi.net/2.0/announce/2.0.0.html

1 0

version of devpi that works with pushing
by Steven Willis 09 Jul '14

09 Jul '14

I reported this bug: https://bitbucket.org/hpk42/devpi/issue/116/extdb-instance-has-no-attribute… which was closed because there was a fix in trunk. Can you tell me a released version where pushing works? It looks line that has the bug: entry = stage.xom.filestore.getentry(relpath) has been there since 2013-09-23, added as part of the "also push docs to internal or external indexes" commit. And it looks like 1.0 is the only release that doesn't have this line. Is that the version I need to use to get pushing to work?

2 1

Difference between index type stage and mirror
by Steven Willis 09 Jul '14

09 Jul '14

Hi, I was wondering what the difference between type=mirror and type=stage is. Is it that: - a mirror gets its list of package exclusively from its 'bases' and you can't upload to it - a stage allows you to upload to it and never looks at its list of bases for packages Or is it something else? I'm seeing the following behavior: $ devpi index --create one type=stage bases=root/pypi http://localhost/pypi/root/one: type=stage bases=root/pypi volatile=True uploadtrigger_jenkins=None acl_upload=root $ devpi index --create one type=stage bases=root/pypi http://localhost/pypi/root/one: type=stage bases=root/pypi volatile=True uploadtrigger_jenkins=None acl_upload=root $ devpi list $ devpi index --create two type=mirror bases= http://localhost/pypi/root/two: type=mirror bases= volatile=False Traceback (most recent call last): File "/usr/local/bin/devpi", line 9, in <module> load_entry_point('devpi-common==1.2', 'console_scripts', 'devpi')() File "/usr/local/lib/python2.7/site-packages/devpi/main.py", line 29, in main return method(hub, hub.args) File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 63, in main return index_create(hub, url, kvdict) File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 5, in index_create index_show(hub, url) File "/usr/local/lib/python2.7/site-packages/devpi/index.py", line 36, in index_show ixconfig["uploadtrigger_jenkins"],)) KeyError: 'uploadtrigger_jenkins' $ devpi use root/two current devpi index: http://localhost/pypi/root/two/ (logged in as root) ~/.pydistutils.cfg : no config file exists ~/.pip/pip.conf : no config file exists ~/.buildout/default.cfg: no config file exists always-set-cfg: no $ devpi list 0x10c-asm 1009558_nester 131228_pytest_1 17MonIP 18-e 1ee 1pass 1to001 2013007_pyh 2C.py ... [Removed huge list a pacakges presumably from root/pypi] $ devpi use -l root/pypi bases= volatile=False root/two bases= volatile=True root/one bases=root/pypi volatile=True Why didn't 'root/one' list any packages when it had 'root/pypi' as a base. And, why did 'root/two' list a whole bunch of packages when didn't have any bases? Is a 'mirror' always a mirror of the central pypi server? Can I not have 'root/one' and 'root/two' be stage indexes and 'root/three' be a mirror of just those two and not the central public pypi server? And why did I get that 'uploadtrigger_jenkins' error?

2 3