Mailman 3 - devpi-dev

Question about PyPI full mirror (38,000+ packages) using devpi
by Richard Gomes 27 Jan '14

27 Jan '14

I've downloaded all packages from PyPI and I'd like to create a local mirror using devpi. I had the following idea: 1. create a new index say /root/pypimirror based on /root/pypi 2. upload the entire folder containing 38,000+ packages onto /root/pypimirror 3. eventually setting /root/pypimirrot to NotVolatile (if this can be done, somehow) /myuser/myindex +-- /root/pypimirror +-- /root/pypi Another idea would be creating a "parallel" index to /root/pypi and exposing a third index which derives from both. /myuser/myindex +-- /root/pypi +-- /root/pypimirror Does this idea make sense? Is there a better way of doing it, in particular without having to download everything again from PyPI? I have another concern: performance. Since 38,000+ packages implies on large directories in the file system... do you think that devpi will have troubles managing such amount of packages? Thanks a lot, -- Richard

2 15

Resolving of dependencies of updated packages
by Achim Domma 18 Jan '14

18 Jan '14

Hi, my machine is configured to use our local devpi server as pypi server. If there's a new package (newer pyramid version in our case), this package is fetched. But it looks like the old, cached versions of dependencies (WebOb in our case) are used by devpi/pip. I saw that some issues, which might be related to that, are fixed in the latest release. But according to "pip freeze", we are already using the latest version: Pygments==1.6 WebOb==1.3.1 argparse==1.2.1 bottle==0.11.6 devpi-common==1.2 devpi-server==1.2.1 docutils==0.11 eventlet==0.14.0 execnet==1.1 greenlet==0.4.1 itsdangerous==0.23 meld3==0.6.10 py==1.4.19 requests==2.1.0 supervisor==3.0 wsgiref==0.1.2 I checked the server logs, but did not see anything that would be helpful to me, to figure out what's going on. Is there a flag to make server side logging more verbose? Or is there any other hint what might be going on? Or how to figure it out? cheers, Achim

2 1

Hosted Devpi / SAAS?
by Erik 18 Jan '14

18 Jan '14

Hello- I tried googling for this on my own, but is anyone aware of an operational SAAS service which hosts devpi? For example, like AWS? I stumbled onto this project, from Defend Against Fruit, while looking for a respository system for python packages. The next best thing seems to be Artifactory Cloud. The price is reasonable but it seems like you have to jump through a lot of hoops to get it to work for python projects, as it is designed for Java. -Erik

2 1

"Shadowing" upstream repositories
by Mahmoud Abdelkader 06 Jan '14

06 Jan '14

Hi there, We're using devpi in production and we have some internal libraries that we don't really plan on releasing to the public. Some of them are named in a way that clashes with pypi's upstream packages (though they are in no way shape or form related). How can I achieve this with devpi? I wasn't able to really get a clear answer and the documentation on "shadowing" -- which is presumably what this feature that I'm inquiring about -- says "TODO." :) Any tips would be appreciated on how to workaround this issue! Cheers & thanks for devpi! Mahmoud

2 1

Anyone using Chef for deployment?
by daves...＠gmail.com 20 Dec '13

20 Dec '13

Hi everyone, We are looking at deploying devpi-server as a local python package index where I work. It will replace an aging Chishop instance. We manage our servers using (Opscode) Chef so I started a Chef cookbook to manage the devpi instance. If anyone else is using Chef, the cookbook is available as https://github.com/dave-shawley/devpi-cookbook. Comments, pull requests, and the like are always welcome. enjoy - dave

2 1

Fwd How do I know if I need to re-authenticate
by Laurent Brack 21 Nov '13

21 Nov '13

Just forwarding this email thread as it might benefit others as Holger suggested. On 11/20/13 11:19 AM, "holger krekel" <hol...(a)merlinux.eu> wrote: >hi Laurent, > >On Wed, Nov 20, 2013 at 19:07 +0000, Brack, Laurent P. wrote: >> Hi Holger, >> >> Thanks a lot for the clarification. Something strange though as even if >>I am logged in, the simple case shows that I am not logged in >> >> $ devpi login lpbrac >> password for user lpbrac: >> logged in 'lpbrac', credentials valid for 10.00 hours >> >> $ devpi use >> using index: http://devpi-eu.dolby.net/lpbrac/dev/ (logged in as lpbrac) >> >> $ curl http://devpi-eu.dolby.net/lpbrac/dev/+api >> { >> "type": "apiconfig", >> "result": { >> "index": "http://devpi-eu.dolby.net/lpbrac/dev/", >> "authstatus": [ >> "noauth", >> "" >> ], >> "simpleindex": "http://devpi-eu.dolby.net/lpbrac/dev/+simple/", >> "login": "http://devpi-eu.dolby.net/+login", >> "pypisubmit": "http://devpi-eu.dolby.net/lpbrac/dev/", >> "resultlog": "http://devpi-eu.dolby.net/+tests" >> } >> } > >Indeed, as you found out below, you need to send the auth information >along with the +api request for the server to tell you if it's >actual/valid. >(The "curl" request could come from any user as it stands). > >cheers, >holger > > >> Looking a little more into it (wrireshark), I found out that the major >>difference is that the local authentication token is embedded in the >>request. >> >> Basically here is the example >> >> In [7]: from requests import * >> >> In [8]: session = Session() >> >> In [10]: r = session.request("get", url = >>"http://devpi-eu.dolby.net/lpbrac/dev/+api", auth=None) >> >> In [11]: r.json() >> Out[11]: >> {u'result': {u'authstatus': [u'noauth', u''], >> u'index': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'login': u'http://devpi-eu.dolby.net/+login', >> u'pypisubmit': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'resultlog': u'http://devpi-eu.dolby.net/+tests', >> u'simpleindex': u'http://devpi-eu.dolby.net/lpbrac/dev/+simple/'}, >> u'type': u'apiconfig'} >> >> In [13]: auth = s._devpi_cfg.get_auth() >> >> In [14]: r = session.request("get", url = >>"http://devpi-eu.dolby.net/lpbrac/dev/+api", auth=auth) >> >> In [15]: r.json() >> Out[15]: >> {u'result': {u'authstatus': [u'ok', u'lpbrac'], >> u'index': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'login': u'http://devpi-eu.dolby.net/+login', >> u'pypisubmit': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'resultlog': u'http://devpi-eu.dolby.net/+tests', >> u'simpleindex': u'http://devpi-eu.dolby.net/lpbrac/dev/+simple/'}, >> u'type': u'apiconfig'} >> >> In [16]: >> >> Thanks for the hintŠ I learnt something in the process. >> >> /Laurent >> >> >> On 11/20/13 5:35 AM, "holger krekel" >><hol...(a)merlinux.eu<mailto:hol...@merlinux.eu>> wrote: >> >> Hi Laurent, >> >> On Wed, Nov 20, 2013 at 00:47 +0000, Brack, Laurent P. wrote: >> Hi Holger, >> I am trying to determine if I need to re-authenticate from time to >> time (that is, even though I have the password key in the devpi client >> store, it might be expired on the server side). >> >> When I perform a devpi use, I see the following >> >>> SREQ* * * * * * * * * * * * * * * * * * * * * * * * * >> method get >> url http://devpi-eu.dolby.net/lpbrac/dev/+api >> kvdict None >> >>> EREQ* * * * * * * * * * * * * * * * * * * * * * * * * >> <<< SRESP* * * * * * * * * * * * * * * * * * * * * * * * * >> STATUS CODE: 200 >> REASON: OK >> RESULT: >> {u'authstatus': [u'ok', u'lpbrac'], >> u'index': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'login': u'http://devpi-eu.dolby.net/+login', >> u'pypisubmit': u'http://devpi-eu.dolby.net/lpbrac/dev/', >> u'resultlog': u'http://devpi-eu.dolby.net/+tests', >> u'simpleindex': u'http://devpi-eu.dolby.net/lpbrac/dev/+simple/'} >> MESSAGE: >> <<< ERESP* * * * * * * * * * * * * * * * * * * * * * * * * >> which is processed by Current.configure_fromurl. >> So basically, if I perform a get the +api resource, how do I get the >>authentication status from the server side? >> >> In theory the "authstatus" above should tell if the login >> is "expired" (instead of "ok"), also according to >> devpi_server/view.py:117 and devpi_server/auth.py. >> >> So i guess you can do something like this:: >> >> r = requests.get("http://devpi-eu.dolby.net/lpbrac/dev/+api") >> status = r.json()["result"]["authstatus"] >> if status[0] != "ok": >> # need to re-login >> >> hope that helps. >> >> On a side note, i'd prefer to discuss such questions over the mailing >>list >> so that more people profit from the discussion. >> >> cheers, >> holger >> >> >> I hope my question is clear. >> Thanks in advance >> /Laurent >>

1 0

devpi-{server,client}-1.2.1: bug fixes + refinement to test
by holger krekel 18 Nov '13

18 Nov '13

devpi-1.2.1: bug fixes and improved test command ========================================================== The devpi-{server,client}-1.2.1 releases bring important bug fixes and refinements. See Changelog below. For getting started with deploying your own pypi server on a laptop or on company server and for using the "devpi" workflow tool (optional), see: http://doc.devpi.net If you want to upgrade an existing installation, you should be able to execute:: $ pip install -U devpi $ devpi-server --upgrade-state [--serverdir YOUR_SERVER_DIR] Have fun, holger krekel Changelog 1.2.1 ---------------------- devpi-server: - fix an import issue for doc files which were wrongly tied to a newer version of a base index. now version "auto" detection for storing doc files only works within a stage. Thanks Laurent Brack for bringing it up and providing the repo. - fix issue66: api endpoints now also respect --outside-url setting so that you can serve devpi from a subpath. Thanks for Fabian Snovna for reporting and analysis. - fix issue63: skip egg links that go to a directory (this requires doing a SVN checkout which devpi-server does not do). Thanks Ken Jung for analyzing the problem. - fix issue68: don't derive metadata from filename but instead look it up in metadata or submitted form. - fix cache-invalidation when normalized_project_name != real_name (e.g. for Django but also many others). addresses issue59. - add newline to simple list output for better human readability of the page (thanks Brandon Maister) - make xmlrpc calls to pypi's changelog API use "requests" sessions so that http proxies are respected there as well (fixes issue58). thanks to riehlm for identifying the problem and testing the fix. - internally refactor and consolidate mocking against requests library - --upgrade-state will upgrade now between major.minor/major.minor+1 changes. devpi-client: - fix "python -m devpi" invocation. Thanks Sebastian Ralph. - fix issue66: "devpi use user/index" can now switch between URLs if user/index is mounted on a subpath. - fix issue71: allow pip/setuptools like requirements specs with the test subcommand, e.g. "devpi test 'pkg>=1.0'". Thanks Sebastian Rahlf for the PR.

1 0

Inheritance and Security
by Bob Corsaro 15 Nov '13

15 Nov '13

I recently started migrating my organization from localshop to devpi, which seems to be a much more active project. Thank you to everyone who has and continues to contribute. I'm worried about what I consider a security risk when using devpi as a repository for private python packages, using pypi.python.org as the attack vector. I'd like to start a discussion with this community with the hopes of determining if my fear is well founded, and what possible solutions we can implement going forward. If a legitimate package *x* is uploaded to a private devpi index at version *n*, and an illegitimate package *x* is uploaded to pypi.python.org with a version >*n*, then it is possible that your deployment scripts, or setup.py of packages dependent on* x* will install the illegitimate package instead of the intended package. All the attacker would need to know is the package name, and possibly the version depending on how strict your dependencies are defined to pwn your system. I have thought of two possible solutions to this problem, the first being secure by default and the second insecure by default, but at least security is possible. *Secure By Default* Simply don't use inheritance on packages that exist in the child index. If *x* exists in child index then don't include *x* from the parent index. I haven't used a complicated inheritance scheme, and I don't know if others are, but this could be reduced to only excluded packages from *root/pypi*. This is also the simplest solution, but also could be disruptive depending on how you are using devpi. I can't think of a case when I personally would want both *root/pypi* and my private versions of a package included in the same index, but that's not to say it's not a valid use case for others. Valid permutations of this solution would be allowing inheritance of existing packages to be toggled per index. This is the solution I'd like to see implemented. I actually have a (probably insufficient) implementation of this solution at https://bitbucket.org/doki_pen/devpi. The change is very simple<https://bitbucket.org/doki_pen/devpi/commits/33630707a73a0aff422f757f462bf6…> . *Secure By Configuration* The other idea is less elegant and harder to implement, blacklisting inherited modules per index. This would involved extending the api to allow modules to be added to and removed from the blacklist on a per index bases. The advantage to this solution is that it would be less disruptive to any existing implementations of devpi.

2 1

SSLError: The read operation timed out
by Dave Dash 31 Oct '13

31 Oct '13

So we've been running devpi internall for a few weeks, 1.1 and now 1.2. One thing we've seen occasionally is pip running into this SSLError below. Apache Bench doesn't show any oddities when we try to hit devpi directly. Our SSL settings are the same we use with all our internal sites. It's fronted by nginx: ssl on; ssl_certificate ***; ssl_certificate_key ***; ssl_prefer_server_ciphers on; ssl_session_cache shared:SSL:10m; add_header Strict-Transport-Security max-age=86400; Exception: Traceback (most recent call last): File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main status = self.run(options, args) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files self.unpack_url(url, location, self.is_download) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/req.py", line 1238, in unpack_url retval = unpack_http_url(link, location, self.download_cache, self.download_dir) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/download.py", line 602, in unpack_http_url resp = _get_response_from_url(target_url, link) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/download.py", line 638, in _get_response_from_url resp = urlopen(target_url) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/download.py", line 176, in __call__ response = self.get_opener(scheme=scheme).open(url) File "/usr/lib/python2.7/urllib2.py", line 400, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 418, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain result = func(*args) File "/home/travis/virtualenv/python2.7/local/lib/python2.7/site-packages/pip/download.py", line 155, in https_open return self.do_open(self.specialized_conn_class, req) File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open r = h.getresponse(buffering=True) File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 407, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 365, in _read_status line = self.fp.readline() File "/usr/lib/python2.7/socket.py", line 447, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 241, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 160, in read return self._sslobj.read(len) SSLError: The read operation timed ou t

2 1

devpi 1.2 releases: many improvements + py33 support
by holger krekel 30 Oct '13

30 Oct '13

The devpi-{server,client}-1.2 releases bring a lot of refinements and improvements for serving and interacting with your own pypi indexes: - devpi-server serves release files from URLs containing a MD5 hash allowing safe serving of those files through nginx - devpi-server's USER/INDEX urls can now be used directly with pip/easy_install without the previously required (and still valid) ``+simple/`` suffix. - ``devpi use --set-cfg`` reads and writes pip/easy_install configuration files, making those installers pick up the in-use index seemlessly. You can even do ``devpi use --always-set-cfg`` to always set those config files when issuing a "devpi use" afterwards. - ``devpi upload`` got many improvements: - versioned files (git and hg) will be exported to a clean directory prior to the build step - distutils/setup.py is now only used for building a package - documentation upload is tied to a version now - you can directly upload distribution files, including wheel files - both devpi-server and devpi-client are python3.3 compatible now and depend on a new devpi-common package which consolidates various pypi-interaction aspects to avoid code duplication. Also, global http proxy settings are honoured. If you have an existing devpi-server-1.1 installation serving your own packages you can install devpi-server>=1.2 and migrate the data with:: devpi-server --upgrade-state [--serverdir your_server_dir] This upgrades your server state in-place. Please make sure you backup your serverdir ahead of doing the upgrade (default location is ~/.devpi/server). WARNING: ``devpi-server --gendeploy`` is deprecated and will be removed probably in favor of just generating example config files for nginx/supervisor/cron. Also ``devpi install`` is deprecated now in favor of using pip/easy_install directly (see also the --set-cfg and --always-set-cfg options). For more information please refer to the extensive documentation at: http://doc.devpi.net/ or check the CHANGELOG below. have fun, holger krekel 1.2 ---------------------------- devpi-server: - serve links to files on simple pages and index root as relative paths so that it works more nicely with proxy-pass server setups. fixes issue56. - make devpi-server and devpi-common python3.3 compatible, addresses issue57 - use system http/s proxy settings from devpi-server. fixes issue58. - refactor locations to allow nginx serving static files more directly. Also updated nginx template accordingly. - rework "--upgrade-state" to detect the state version of the server dir and create an appropriate virtualenv with a devpi-server install in order to export data, and then import that version. - allow to use /user/index as indexserver url for pip/easy_install by redirecting non-json queries to /user/index/PROJ[/] to /user/index/+simple/PROJ/ - fix submission of multi-value fields like "classifiers" or "platform" (previously they would be wrongly collapsed to become the last value of a list) - fix normalization import/export issue: pypi names take precendence for defining the "real" name of a project. - always store uploaded documentation with a version. While "devpi upload" will make sure to pass in the version, "setup.py upload_docs" will not pass in a version. In the latter case, devpi-server assumes the documentation belongs to the highest yet registered release. This change requires exporting with devpi-1.1 and importing with devpi-1.2 in order to properly store versioned docs internally. - use types/url/metadata/validation functionality of new depdency devpi_common - internal cleanup using pytest-flakes - make devpi-server use a proper UserAgent string devpi-client: - "devpi list" and "devpi remove" now accept a pip/setuptools style requirement like "pkg>=1.0" instead of the former for limited "pkg-1.0". - make devpi-client fully work with python3.3 and fix test bugs - use system http/s proxy settings. fixes issue58. - add "devpi test -c tox.ini package" to use a particular (external) tox.ini for running tox with the unpackaged package. also add "--fallback-ini tox.ini" option which will only be used if the download package has no tox.ini. - new "devpi use --set-cfg" option to set pip/easy_install configuration files when changing indexes. Also new "devpi use --always-set-cfg=yes" option if you want to imply "--set-cfg" on future "devpi use" invocations and "devpi use --always-st-cfg=no" to disable this implication. - support git and hg for exporting all versioned files of a directory before performing the build step when uploading - improve how upload works: setup.py is only used for building docs and release files but not for the remote upload part. This gets rid of a number of hacks that were done trying to get the Python shipped "distutils" to pick the proper devpi index and allows proper SSL verification on Python2.6 onwards. - upload: show response when uploading documentation failed - upload: allow to specify archive files as positional arguments (both files and directories can be specified but the latter additionally require a --upload-dirs option) - fix issue54: upload now works on wheel files as well. As pkginfo does not support wheels directly, we use the ``twine`` project which extends pkginfo for now. - only show highest version in "devpi list PROJECT" output, unless "--all" is specified. - on upload release files: skip rather than guess packages which contain no metadata - strike BeautifulSoup dependency and re-use vendored pip-link parser - use types/url/metadata/validation functionality of new depdency devpi_common - internal cleanup wrt pytest-flakes discoveries - remove "archive" dependency in favour of a small implementation in devpi_common - make devpi-client use a proper UserAgent string

1 0