Mailman 3 - devpi-dev

devpi-1.1: improved pypi cache, data migration, wheels, ...
by holger krekel Sept. 25, 2013

Sept. 25, 2013

The devpi-{server,client}-1.1 releases bring a lot of refinements and improvements for serving standalone pypi mirrors or company/private indexes which seemlessly merge in PyPI releases, mainly: - improved speed and semantics of pypi cache. devpi-server has been tested to correctly mirror all ~34000 projects on pypi.python.org now, and minimizes upstream interactions by fully exploiting the existing mirroring protocols. - data migration: new --upgrade/import/export options. … [View More]versioning of serverstate and API calls so that future changes can be detected by clients. - better support for ".whl" files and many other little improvements and bug fixes, see the changelog below for more info. Note that devpi-1.1 will require to ``--upgrade`` your 1.0 server state before you can serve with devpi-server-1.1 release. Docs also were updated. Here is a Quickstart tutorial for efficient pypi-mirroring on your laptop:: http://doc.devpi.net/1.1/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/1.1/quickstart-releaseprocess.html If you want to permanently install devpi-server and potentially access it from many clients:: http://doc.devpi.net/1.1/quickstart-server.html More documentation and the beginning of an exhaustive user manual:: http://doc.devpi.net/latest/ have fun, holger CHANGELOG 1.1 ---------------------------- devpi-server: - systematically test pypi/mirror code against all 34K pypi projects so that we know that all http/https installable archive links that pypi offers are correctly recognized by devpi-server's root/pypi index. - if no pypi mirror state is known, devpi-server now calls pypi to obtain names/serials. It will fail to start if no such initial connection is possible. Once a first mirror state is known, subsequent devpi-server starts will not perform this initial query. - speed up and make more reliable all operations on private packages which have no pypi.python.org release: we can now determine if a project exists on pypi and under which name exactly without remote queries or redirects to pypi.python.org. - fix issue45: register/upload package names are now properly validated and redirects take place if e.g. a project was registered as "name-sub" and "+simple/name_sub" is queried. - new --upgrade-state command to allow for easy and safe in-place upgrading of server state. This is not guranteed to be possible for all future releases which might require using --export with an older version and --import with a newer version. - new --export/--import options to dump and import server contents: users, indexes, docs, release files and (test) attachments. Note that root/pypi (PyPI-caching information) will not be exported/imported. (maybe in the future if there is demand). - fix issue49: both push and import/export now support docfiles. Note, however, that docfiles relate to a project as a whole and are not tied to a particular version. This property is inherited from the PyPI standard upload_docs action and cannot be changed without interfering or replacing the upload_docs protocol of setuptools/sphinx. - fix issue51: return 200 code if release file is successfully uploaded but jenkins could not be triggered (previously returned 500) - reject simple/NAME if NAME contains non-ascii characters (PEP426 naming rules) - devpi-server now returns a X-DEVPI-API-VERSION and X-DEVPI-SERVER-VERSION header. For future incompatible changes these versions allow clients to reject interactions. - also add ".serverversion" file and write it if it does not exist, and make devpi-server use it to verify if operating on a compatible server data layout, otherwise bail out. - address issue43: --gendeploy now uses pip without --pre and explicitely instructs pip to install the exact same version of devpi-server with which --gendeploy is issued. - fix issue46 -- for GET /root/pypi/ only show a link to the simple page instead of computing "latest in-stage packages" which is only useful for devpi's user indices. - fix issue37: upload with expired login causes proper 401 devpi-client: - detect "X-DEVPI-API-VERSION" header and check for compatibility. devpi-client currently supports version "1" and warns if no version is known (defaulting to "1"). - getjson now prints http reply headers if "-v|--verbose" was supplied. - fix issue52: add "--no-vcs" option to force "devpi upload" to not vcs-export files before executing build/upload - fix issue50: introduce "--toxargs" to "devpi test" invocations in order to add arbitrary arguments to tox. - fix issue43: set PIP_PRE environment variable when executing "devpi install ..." so that the behaviour between pip < 1.4 and >= 1.4 is normalized to always install development versions. - fix issue47: set PIP_USE_WHEEL with "devpi install ..." so that it will automatically pick up wheel files if pip>1.4 is used. - fix issue42: allow to set empty bases for an index, so that it doesn't inherit anything. - fix issue44: "use -l" doesn't break when a user has no index - devpi test now invokes tox in-process (by importing tox) rather than a subprocess. [View Less]

1 0

import/export has landed, 1.1 plans
by holger krekel Sept. 10, 2013

Sept. 10, 2013

I have just committed my work on new devpi-server --export/--import options which allow to dump the server database and import it again. This includes users, indicies, release files and test reports. However, currently root/pypi and uploaded documentation is excluded from export. Future devpi-server versions might support it. Also, devpi-server now maintains a ``serverversion`` attribute and verifies it's running on a compatible version. Both changes help to care for schema migration in … [View More]

1 0

devpi-1.0: PyPI server and packaging/testing/release tool
by holger krekel Aug. 15, 2013

Aug. 15, 2013

devpi-1.0: PyPI server and packaging/testing/release tool ========================================================= devpi-1.0 brings an improved PyPI caching and internal index server as well as a new abilities for tox-testing and staging your Python release packages. For a (long) list of changes, see the below CHANGELOG. Documentation got revamped and extended and now contains three quickstart scenarios. First the Quickstart tutorial for pypi-mirroring on your laptop:: http://doc.… [View More]devpi.net/1.0/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/1.0/quickstart-releaseprocess.html If you want to permanently install devpi-server and potentially access it from many clients:: http://doc.devpi.net/1.0/quickstart-server.html More documentation and the beginning of an exhaustive user manual:: http://doc.devpi.net/latest/ Note that devpi-1.0 is not data-compatible to the previous 0.9.4 release: You need to start with a fresh devpi-1.0 installation and upload your packages again. Future releases of devpi should support data migration more directly. best and have fun, holger krekel Changelog 1.0 (-0.9.4) ---------------------------- devpi-server: - rename "--datadir" to "--serverdir" to better match the also picked up DEVPI_SERVERDIR environment variable. - fix a strange effect in that sometimes tools ask to receive a package url with a "#md5=..." arriving at the server side. We now strip that part out before trying to serve the file. - on startup don't create any initial indexes other than the "root/pypi" pypi caching mirror. - introduce ``--start``, ``--stop`` and ``--log`` commands for controling a background devpi-server run. (these commands previously were implemented with the devpi-client and the "server" sub command) - fix issue27: provide full list of pypi names in root/pypi's simple view (and simple pages from inheriting indices) - default to "eventlet" server when creating deployment with --gendeploy - fix issue25: return 403 Forbidden when trying to delete the root user. - fix name mangling issue for pypi-cache: "project_name*" is now matched correctly when a lookup for "project-name" happens. - fix issue22: don't bypass CDN by default, rather provide an "--bypass-cdn" option to do it (in case you have cache-invalidation troubles) - fix issue20 and fix issue23: normalize index specs internally ("/root/dev" -> "root/dev") and check if base indices exist. - add Jenkins build job triggering for running the tests for a package through tox. - inheritance cleanup: inherited versions for a project are now shadowed and not shown anymore with getreleaselinks() or in +simple pages if the "basename" is exactly shadowed. - fix issue16: enrich projectconfig json with a "+shadow" file which lists shadowed "versions" - initial wheel support: accept "whl" uploads and support caching of whl files from pypi.python.org - implemented internal push operation between devpi indexes - show "docs" link if documentation has been uploaded - pushing releases to pypi.python.org will now correctly report the filetype/pyversion in the metadata. - add setting of acl_upload for indexes. Only the owning user and acl_upload users may upload releases, files or documentation to an index. - add --passwd USER option for setting a user's password server-side - don't require email setting for creating users devpi-client: - removed ``server`` subcommand and options for controling background devpi-server processes to become options of ``devpi-server`` itself. - fix issue14: lookup "python" from PATH for upload/packaging activities instead of using "sys.executable" which comes from the interpreter executing the "devpi" script. This allows to alias "devpi" to come from a virtualenv which is separate from the one used to perform packaging. - fix issue35: "devpi index" cleanly errors out if no index is specified or in use. - remember authentication on a per-root basis and cleanup "devpi use" interactions. This makes switching between multiple devpi instances more seemless. - fix issue17: better reporting when "devpi use" does not operate on valid URL - test result upload and access: - "devpi test" invokes "tox --result-json ..." and uploads the test result log to devpi-server. - "devpi list [-f] PKG" shows test result information. - add "uploadtrigger_jenkins" configuration option through "devpi index". - fix issue19: devpi use now memorizes --venv setting properly. Thanks Laurent. - fix issue16: show files from shadowed versions - initial wheel support: "devpi upload --format=bdist_wheel" now uploads a wheel format file to the index. (XXX "devpi install" will trigger pip commands with option "--use-wheels".) - fix issue15: docs will now be built via "setup.py build_sphinx" using a internal build dir so that the upload succeeds if conf.py would otherwise specify a non-standard location. - implement and refine "devpi push" command. It now accepts two forms "user/name" for specifying an internal devpi index and "pypi:REPONAME" for specifying a repository which must be defined in a .pypirc file. - remove spurious pdb.set_trace() in devpi install command when no pip can be found. - show and allow to set "acl_upload" for uploading priviliges - add longer descriptions to each sub command, shown with "devpi COMMAND -h". - removed pytestplugin support for now (pytest reporting directly to devpi-server) [View Less]

1 0

1.0rc2 is out, with improved docs (and improving)
by holger krekel Aug. 14, 2013

Aug. 14, 2013

Hi all, just a quick heads up that devpi-1.0rc2 packages are out: pip install -U http://devpi.net/hpk/release/+simple/ devpi should bring it to your local env. The docs are at: http://doc.devpi.net/1.0rc2. Any and all feedback welcome. I am already getting some on IRC (#devpi on freenode) and incorporating changes for tomorrow's 1.0. cheers, holger Q29udGVudC1UeXBlOiBhcHBsaWNhdGlvbi9wZ3Atc2lnbmF0dXJlOyBuYW1lPSJzaWduYXR1cmUu … [View More]

1 0

devpi-1.0rc1 out / please give it a try
by holger krekel Aug. 13, 2013

Aug. 13, 2013

Hi all, i've put devpi-1.0rc1 out which installs devpi-{server,client}-1.0rc packages, respectively. It comes with a lot of bug fixes, refinements and new features. If some of you could install the package with:: pip install -U -i http://devpi.net/hpk/release/+simple/ devpi and try the main quickstart doc:: http://doc.devpi.net/1.0rc1/index.html that would be much appreciated! I am also on #devpi on irc.freenode.net for any feedback/questions. I want to put out a devpi-1.0 … [View More]

1 0

drop "automatic server" support?
by holger krekel Aug. 11, 2013

Aug. 11, 2013

Hi all, i am considering to drop auto-server support for 1.0 (due the coming week, btw, with lots of improvements, fixes and features!). Currently, if you install devpi, and then run "devpi install|upload|..." and are using the default http://localhost:3141 devpi instance, it will start and manage a background devpi-server process. The whole logic is hairy so i'd rather like to drop supporting it and require that you have a "devpi-server" running (for starters, just in another terminal, … [View More]

1 0