Is it possible (or desirable) that devpi be able to trigger builds of
wheels for packages that it caches? I realise it'd be possible as part of a
Jenkins trigger for an upload, but this is for cached packages from PyPI.
The actual building could be complex - in the case of current Linux wheels,
we'd need to have separate indexes for the wheels for Debian, RHEL, Ubuntu,
SUSE, etc. since the wheels are typically not compatible. That complexity
could be external to devpi - it's just a case of getting the trigger to
indicate that a build is desired.
Quite how this integrates with a user just doing a "pip install package" is
a little beyond me for the moment - the delay between the cache event and
build completing could be significant. In light of that it might be
desirable to have a separate process which does all this work (fetches the
latest package and performs the build). Has anyone done this? How would it
keep up to date?
Richard
Hi all,
Currently you can't use "devpi test" with wheel files because they
don't contain the tests unless you put all tests inside the software
archive. To get the tests we could require that there is an "sdist"
archive file containing the "tox.ini" file and the tests. Thus
"devpi test X" would potentially download two files, a platform
specific one and the sdist to configure and perform testing.
Apart from running the tests against the sdist we probably also
want to run the tests against the installed sdist. Not sure
i grasp all implications yet (also UI wise) but does anyone
of you have preliminary thoughts on the issue?
best,
holger
See https://github.com/jhermann/devpi-enterprisey#ssl-certificate-handling
for the current state of the devpi / packaging universe in one place.
All that stuff is tested, including SNI with Python2, going through an
Apache2 named SSL vhost on one machine to another running devpi-server
behind NginX. Yes, migration scenarios are strange. ☺
Hi!
I'm currently writing a devpi-ldap plugin for devpi-server.
At this time devpi-server hasn't been extended for external
authentication plugins, so the plugin doesn't work with devpi-server
yet.
So far I've worked on the configuration for the LDAP connection. Since
that is quite open and the specifics up to the LDAP administrator, I
could use some real world testing and feedback.
You can find the current state at https://github.com/devpi/devpi-ldap
If you install that in a virtualenv, you will get a devpi-ldap script.
With that script you can test your configuration and see if a user can
be authenticated and it's groups be returned correctly.
Details on the configuration are in the readme.
Please add any bugs or feature requests to the issue tracker at
https://github.com/devpi/devpi-ldap/issues
Any problem with the documentation, like missing or unclear descriptions
are considered bugs :)
If you want to report issues, but don't want to publicly post your
configuration or the output of the devpi-ldap script, then please feel
free to contact me directly.
Regards,
Florian Schulze
Hi!
I just released devpi-ldap 1.0.0. It's an external authentication plugin
for devpi-server >= 2.1.0.
See https://pypi.python.org/pypi/devpi-ldap for details.
Regards,
Florian Schulze
devpi-{server-2.1,web-2.2}: upload history, deploy status, groups
==================================================================
With devpi-server-2.1 and devpi-web-2.2 you'll get a host of fixes and
improvements as well as some major new features for the private pypi
system:
- upload history: release/tests/doc files now carry metadata for
by whom, when and to which index something was uploaded
- deployment status: the new json /+status gives detailed information
about a replica or master's internal state
- a new authentication hook supports arbitrary external authentication
systems which can also return "group" membership information. An initial
separately released "devpi-ldap" plugin implements verification accordingly.
You can specify groups in ACLs with the ":GROUPNAME" syntax.
- a new "--restrict-modify=ACL" option to start devpi-server such that
only select accounts can create new or modify users or indexes
For many more changes and fixes, please see the CHANGELOG information below.
UPGRADE note: devpi-server-2.1 requires to ``--export`` your 2.0
server state and then using ``--import`` with the new version
before you can serve your private packages through devpi-server-2.1.
Please checkout the web plugin if you want to have a web interface::
http://doc.devpi.net/2.1/web.html
Here is a Quickstart tutorial for efficient pypi-mirroring
on your laptop::
http://doc.devpi.net/2.1/quickstart-pypimirror.html
And if you want to manage your releases or implement staging
as an individual or within an organisation::
http://doc.devpi.net/2.1/quickstart-releaseprocess.html
If you want to host a devpi-server installation with nginx/supervisor
and access it from clients from different hosts::
http://doc.devpi.net/2.1/quickstart-server.html
More documentation here::
http://doc.devpi.net/2.1/
many thanks to Florian Schulze who co-implemented many of the new features.
And special thanks go to the two companies who funded major parts
of the above work.
have fun,
Holger Krekel, merlinux GmbH
devpi-server-2.1.0 (compared to 2.0.6)
----------------------------------------
- make replication more precise: if a file cannot be replicated,
fail with an error log and try again in a few seconds.
This helps to maintain a consistent replica and discover
the potential remaining bugs in the replication code.
- add who/when metadata to release files, doczips and test results
and preserve it during push operations so that any such file provides
some history which can be visualized via the web-plugin. The metadata
is also exposed via the json API (/USER/INDEX/PROJECTNAME[/VERSION])
- fix issue113: provide json status information at /+status including roles
and replica polling status, UUIDs of the repository. See new
server status docs for more info.
- support for external authentication plugins: new devpiserver_auth_user
hook which plugins can implement for user/password validation and
for providing group membership.
- support groups for acl_upload via the ":GROUPNAME" syntax. This
requires an external authentication plugin that provides group
information.
- on replicas return auth status for "+api" requests
by relaying to the master instead of using own key.
- add "--restrict-modify" option to specify users/groups which can create,
delete and modify users and indices.
- make master/replica configuration more permanent and a bit safer
against accidental errors: introduce "--role=auto" option, defaulting
to determine the role from a previous invocation or the presence of the
"--master-url" option if there was no previous invocation. Also verify
that a replica talks to the same master UUID as with previous requests.
- replaced hack from nginx template which abused "try_files" in "location /"
with the recommended "error_page"/"return" combo.
Thanks Jürgen Hermann
- change command line option "--master" to "--master-url"
- fix issue97: remove already deprecated --upgrade
option in favor of just using --export/--import
- actually store UTC in last_modified attribute of release files instead of
the local time disguising as UTC. preserve last_modified when pushing
a release.
- fix exception when a static resource can't be found.
- address issue152: return a proper 400 "not registered" message instead
of 500 when a doczip is uploaded without prior registration.
- add OSX/launchd example configuration when "--gen-config" is issued.
thanks Sean Fisk.
- fix replica proxying: don't pass original host header when relaying a
modifying request from replica to master.
- fix export error when a private project doesnt exist on pypi
- fix pushing of a release when it contains multiple tox results.
- fix "refresh" button on simple pages on replica sites
- fix an internal link code issue possibly affecting strangeness
or exceptions with test result links
- be more tolerant when different indexes have different project names
all mapping to the same canonical project name.
- fix issue161: allow "{pkgversion}" to be part of a jenkins url
devpi-web-2.2.0 (compared to 2.1.6)
----------------------------------------
- require devpi-server >= 2.1.0
- static resources now have a plus in front to avoid clashes with usernames and
be consistent with how other urls work: "+static/..." and "+theme-static/..."
- adjusted font-sizes and cut-off width of content.
- only show underline on links when hovering.
- make the "description hasn't been rendered" warning stand out.
- version.pt: moved md5 sum from it's own column to the file column below the
download link
- version.pt: added "history" column showing last modified time and infos
about uploads and pushes.
- fix issue153: friendly error messages on upstream errors.
- index.pt: show permissions on index page
devpi-client-2.0.3 (compared to 2.0.2)
----------------------------------------
- use default "https://www.python.org/pypi" when no repository is set in .pypirc
see https://docs.python.org/2/distutils/packageindex.html#the-pypirc-file
- fix issue152: when --upload-docs is given, make sure to first register
and upload the release file before attempting to upload docs (the latter
requires prior registration)
- fix issue75: add info about basic auth to "url" option help of "devpi use".
- fix issue154: fix handling of vcs-exporting when unicode filenames are
present. This is done by striking our own code in favor of Marius Gedminas'
vcs exporting functions from his check-manifest project which devpi-client
now depends on. This also adds in support for svn and bazaar in addition
to the already supported git/hg.
- devpi list: if a tox result does not contain basic information (probably a bug in
tox) show a red error instead of crashing out with a traceback.
- fix issue157: filtering of tox results started with the oldest ones and
didn't show newer results if the host, platform and environment were the same.
devpi-server-2.0.7 is a pure bug fix release, a drop in replacement
for earlier 2.0.X releases. See below for the CHANGELOG.
many thanks for Florian Schulze, as always, for helping with the release.
best,
holger krekel, merlinux gmbH
2.0.7
----------------
- fix wrong host header in proxy requests.
- fix export error when a private project doesnt exist on pypi
- fix pushing of a release when it contains multiple tox results.
- be more tolerant when different indexes have different project names
wrt to "normalization"
- fix "refresh" button on simple pages on replica sites
- fix an internal link code issue possibly affecting strangeness
or exceptions with test result links
- on export warn but otherwise skip missing toxresult files
I released devpi-web 2.1.1 with bug fixes for indexing.
If you see devpi_web exceptions in your log, then you can run
devpi-server with the --recreate-search-index option once to reindex
everything after upgrading devpi-web.
Regards,
Florian Schulze
2.1.1
---------------
- better exception handling/reporting while indexing.
- renamed ``--index-projects`` to ``--recreate-search-index``.
Hi,
Are there any plans to allow creating mirror type indexes in addition to
stage types? That way if I have say a company pypi server somewhere with
packages I could add an index in my local devpi to be a mirror of it, and
then it could be base of other stage type indexes I create. This question
seems to imply this is almost possible:
https://groups.google.com/forum/#!searchin/devpi-dev/mirror/devpi-dev/tJp...
But just needs to allow the upstream index-url to set for the mirror (and
not crash of course).
Thanks,
\Peter
I am curious how people are using (or not using) devpi's capability of
having multiple indices.
When initially setting up devpi, I saw this cool new feature that was
discussed and length in the docs and I felt compelled to use it. I set up 6
indexes to divide stuff up by OS and by company-internal vs. public.
Now I wonder if we are getting much from value from this separation or if
is complicating things without adding much value.
I have different indices for different OSs -- e.g.:
$ devpi index -l
wheels/pure_python
wheels/ubuntu_12.04
wheels/ubuntu_14.04
The latter two inherit from the first.
The reasoning for this is that we have multiple platforms use at our
company and wheels don't really have enough metadata in their filename to
express what Linux and distro they were built for, so I need to maintain
separate indexes.
The other distinction I have is that all of these indices are for
third-party stuff from PyPI and I have a separate index that has
company-internal stuff
It seemed useful to keep 3rd party and internal stuff separate, because the
3rd party is more critical to keep safe.
How are other folks using indexes?
Marc
