Mailman 3 September 2014 - devpi-dev

Automatic building of wheels? (discuss ;-)
by Richard Jones 12 Oct '14

12 Oct '14

Is it possible (or desirable) that devpi be able to trigger builds of wheels for packages that it caches? I realise it'd be possible as part of a Jenkins trigger for an upload, but this is for cached packages from PyPI. The actual building could be complex - in the case of current Linux wheels, we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu, SUSE, etc. since the wheels are typically not compatible. That complexity could be external to devpi - it's just a case of getting the trigger to indicate that a build is desired. Quite how this integrates with a user just doing a "pip install package" is a little beyond me for the moment - the delay between the cache event and build completing could be significant. In light of that it might be desirable to have a separate process which does all this work (fetches the latest package and performs the build). Has anyone done this? How would it keep up to date? Richard

3 13

testing wheel files / refinements
by holger krekel 01 Oct '14

01 Oct '14

Hi all, Currently you can't use "devpi test" with wheel files because they don't contain the tests unless you put all tests inside the software archive. To get the tests we could require that there is an "sdist" archive file containing the "tox.ini" file and the tests. Thus "devpi test X" would potentially download two files, a platform specific one and the sdist to configure and perform testing. Apart from running the tests against the sdist we probably also want to run the tests against the installed sdist. Not sure i grasp all implications yet (also UI wise) but does anyone of you have preliminary thoughts on the issue? best, holger

3 6

SSL cert handling writeup
by Jürgen Hermann 24 Sep '14

24 Sep '14

See https://github.com/jhermann/devpi-enterprisey#ssl-certificate-handling for the current state of the devpi / packaging universe in one place. All that stuff is tested, including SNI with Python2, going through an Apache2 named SSL vhost on one machine to another running devpi-server behind NginX. Yes, migration scenarios are strange. ☺

1 0

devpi-ldap plugin
by Florian Schulze 24 Sep '14

24 Sep '14

Hi! I'm currently writing a devpi-ldap plugin for devpi-server. At this time devpi-server hasn't been extended for external authentication plugins, so the plugin doesn't work with devpi-server yet. So far I've worked on the configuration for the LDAP connection. Since that is quite open and the specifics up to the LDAP administrator, I could use some real world testing and feedback. You can find the current state at https://github.com/devpi/devpi-ldap If you install that in a virtualenv, you will get a devpi-ldap script. With that script you can test your configuration and see if a user can be authenticated and it's groups be returned correctly. Details on the configuration are in the readme. Please add any bugs or feature requests to the issue tracker at https://github.com/devpi/devpi-ldap/issues Any problem with the documentation, like missing or unclear descriptions are considered bugs :) If you want to report issues, but don't want to publicly post your configuration or the output of the devpi-ldap script, then please feel free to contact me directly. Regards, Florian Schulze

2 4

devpi-ldap 1.0.0 released
by Florian Schulze 22 Sep '14

22 Sep '14

Hi! I just released devpi-ldap 1.0.0. It's an external authentication plugin for devpi-server >= 2.1.0. See https://pypi.python.org/pypi/devpi-ldap for details. Regards, Florian Schulze

1 0

devpi-{server-2.1,web-2.2}: upload history, deploy status, groups
by holger krekel 22 Sep '14

22 Sep '14

devpi-{server-2.1,web-2.2}: upload history, deploy status, groups ================================================================== With devpi-server-2.1 and devpi-web-2.2 you'll get a host of fixes and improvements as well as some major new features for the private pypi system: - upload history: release/tests/doc files now carry metadata for by whom, when and to which index something was uploaded - deployment status: the new json /+status gives detailed information about a replica or master's internal state - a new authentication hook supports arbitrary external authentication systems which can also return "group" membership information. An initial separately released "devpi-ldap" plugin implements verification accordingly. You can specify groups in ACLs with the ":GROUPNAME" syntax. - a new "--restrict-modify=ACL" option to start devpi-server such that only select accounts can create new or modify users or indexes For many more changes and fixes, please see the CHANGELOG information below. UPGRADE note: devpi-server-2.1 requires to ``--export`` your 2.0 server state and then using ``--import`` with the new version before you can serve your private packages through devpi-server-2.1. Please checkout the web plugin if you want to have a web interface:: http://doc.devpi.net/2.1/web.html Here is a Quickstart tutorial for efficient pypi-mirroring on your laptop:: http://doc.devpi.net/2.1/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/2.1/quickstart-releaseprocess.html If you want to host a devpi-server installation with nginx/supervisor and access it from clients from different hosts:: http://doc.devpi.net/2.1/quickstart-server.html More documentation here:: http://doc.devpi.net/2.1/ many thanks to Florian Schulze who co-implemented many of the new features. And special thanks go to the two companies who funded major parts of the above work. have fun, Holger Krekel, merlinux GmbH devpi-server-2.1.0 (compared to 2.0.6) ---------------------------------------- - make replication more precise: if a file cannot be replicated, fail with an error log and try again in a few seconds. This helps to maintain a consistent replica and discover the potential remaining bugs in the replication code. - add who/when metadata to release files, doczips and test results and preserve it during push operations so that any such file provides some history which can be visualized via the web-plugin. The metadata is also exposed via the json API (/USER/INDEX/PROJECTNAME[/VERSION]) - fix issue113: provide json status information at /+status including roles and replica polling status, UUIDs of the repository. See new server status docs for more info. - support for external authentication plugins: new devpiserver_auth_user hook which plugins can implement for user/password validation and for providing group membership. - support groups for acl_upload via the ":GROUPNAME" syntax. This requires an external authentication plugin that provides group information. - on replicas return auth status for "+api" requests by relaying to the master instead of using own key. - add "--restrict-modify" option to specify users/groups which can create, delete and modify users and indices. - make master/replica configuration more permanent and a bit safer against accidental errors: introduce "--role=auto" option, defaulting to determine the role from a previous invocation or the presence of the "--master-url" option if there was no previous invocation. Also verify that a replica talks to the same master UUID as with previous requests. - replaced hack from nginx template which abused "try_files" in "location /" with the recommended "error_page"/"return" combo. Thanks JÃ¼rgen Hermann - change command line option "--master" to "--master-url" - fix issue97: remove already deprecated --upgrade option in favor of just using --export/--import - actually store UTC in last_modified attribute of release files instead of the local time disguising as UTC. preserve last_modified when pushing a release. - fix exception when a static resource can't be found. - address issue152: return a proper 400 "not registered" message instead of 500 when a doczip is uploaded without prior registration. - add OSX/launchd example configuration when "--gen-config" is issued. thanks Sean Fisk. - fix replica proxying: don't pass original host header when relaying a modifying request from replica to master. - fix export error when a private project doesnt exist on pypi - fix pushing of a release when it contains multiple tox results. - fix "refresh" button on simple pages on replica sites - fix an internal link code issue possibly affecting strangeness or exceptions with test result links - be more tolerant when different indexes have different project names all mapping to the same canonical project name. - fix issue161: allow "{pkgversion}" to be part of a jenkins url devpi-web-2.2.0 (compared to 2.1.6) ---------------------------------------- - require devpi-server >= 2.1.0 - static resources now have a plus in front to avoid clashes with usernames and be consistent with how other urls work: "+static/..." and "+theme-static/..." - adjusted font-sizes and cut-off width of content. - only show underline on links when hovering. - make the "description hasn't been rendered" warning stand out. - version.pt: moved md5 sum from it's own column to the file column below the download link - version.pt: added "history" column showing last modified time and infos about uploads and pushes. - fix issue153: friendly error messages on upstream errors. - index.pt: show permissions on index page devpi-client-2.0.3 (compared to 2.0.2) ---------------------------------------- - use default "https://www.python.org/pypi" when no repository is set in .pypirc see https://docs.python.org/2/distutils/packageindex.html#the-pypirc-file - fix issue152: when --upload-docs is given, make sure to first register and upload the release file before attempting to upload docs (the latter requires prior registration) - fix issue75: add info about basic auth to "url" option help of "devpi use". - fix issue154: fix handling of vcs-exporting when unicode filenames are present. This is done by striking our own code in favor of Marius Gedminas' vcs exporting functions from his check-manifest project which devpi-client now depends on. This also adds in support for svn and bazaar in addition to the already supported git/hg. - devpi list: if a tox result does not contain basic information (probably a bug in tox) show a red error instead of crashing out with a traceback. - fix issue157: filtering of tox results started with the oldest ones and didn't show newer results if the host, platform and environment were the same.

1 0

devpi-server-2.0.7: various bug fixes
by holger krekel 19 Sep '14

19 Sep '14

devpi-server-2.0.7 is a pure bug fix release, a drop in replacement for earlier 2.0.X releases. See below for the CHANGELOG. many thanks for Florian Schulze, as always, for helping with the release. best, holger krekel, merlinux gmbH 2.0.7 ---------------- - fix wrong host header in proxy requests. - fix export error when a private project doesnt exist on pypi - fix pushing of a release when it contains multiple tox results. - be more tolerant when different indexes have different project names wrt to "normalization" - fix "refresh" button on simple pages on replica sites - fix an internal link code issue possibly affecting strangeness or exceptions with test result links - on export warn but otherwise skip missing toxresult files

1 1

devpi-web 2.1.1 released
by Florian Schulze 11 Sep '14

11 Sep '14

I released devpi-web 2.1.1 with bug fixes for indexing. If you see devpi_web exceptions in your log, then you can run devpi-server with the --recreate-search-index option once to reindex everything after upgrading devpi-web. Regards, Florian Schulze 2.1.1 --------------- - better exception handling/reporting while indexing. - renamed ``--index-projects`` to ``--recreate-search-index``.

1 0

Allow creating mirror type index
by Peter Gengler 11 Sep '14

11 Sep '14

Hi, Are there any plans to allow creating mirror type indexes in addition to stage types? That way if I have say a company pypi server somewhere with packages I could add an index in my local devpi to be a mirror of it, and then it could be base of other stage type indexes I create. This question seems to imply this is almost possible: https://groups.google.com/forum/#!searchin/devpi-dev/mirror/devpi-dev/tJp2X… But just needs to allow the upstream index-url to set for the mirror (and not crash of course). Thanks, \Peter

2 2

Looking for guidance on how to organize multiple devpi indices
by Marc Abramowitz 11 Sep '14

11 Sep '14

I am curious how people are using (or not using) devpi's capability of having multiple indices. When initially setting up devpi, I saw this cool new feature that was discussed and length in the docs and I felt compelled to use it. I set up 6 indexes to divide stuff up by OS and by company-internal vs. public. Now I wonder if we are getting much from value from this separation or if is complicating things without adding much value. I have different indices for different OSs -- e.g.: $ devpi index -l wheels/pure_python wheels/ubuntu_12.04 wheels/ubuntu_14.04 The latter two inherit from the first. The reasoning for this is that we have multiple platforms use at our company and wheels don't really have enough metadata in their filename to express what Linux and distro they were built for, so I need to maintain separate indexes. The other distinction I have is that all of these indices are for third-party stuff from PyPI and I have a separate index that has company-internal stuff It seemed useful to keep 3rd party and internal stuff separate, because the 3rd party is more critical to keep safe. How are other folks using indexes? Marc

3 2