devpi-dev February 2016

devpi-dev@python.org

5 participants
5 discussions

by Andrzej Ostrowski

Hi, I'm playing around with devpi themes. My goal is to create a nice theme for my company. I'd like to add extra information to some pages, like the index page or the main page. I added a snippet of code which produces an "index inheritance diagram" on each index page display. Because the company I work for is not that small, these diagrams can get strange. A sample diagram: index inheritance diagram <https://lh3.googleusercontent.com/--9yr5WSohOc/VZ0BL6xN3YI/AAAAAAAAAAs/8wZq…>. Each box is an index, I intentionally removed all user and index names, except for root/pypi. It's not my indices, please don't judge me for the design :) I'm concerned about performance here. First of all, currently I generate those on display action, which doesn't seem the way to go. Could devpi provide a hook so that I'm able to regenerate a diagram on an index update event? Additionally, I'd like to add more information to an index page and the root pages. So the question is: Could functions in devpi_web.views expose more data in get_index() and root()? Potentially other functions? I know I can always get the data with python snippets in the templates, but it all takes precious time. For instance, in my company, some indices are regular user indices, some of them are project indices. I'd like to sort the list of indices on the root page: display the project ones first, and user indices below. The way I would see this is the custom_data property of an index. The bottom line: I think people could easily create fancier themes if the devpi web views functions exposed more data to the templates, even if that data is not used in the default theme. It would be great to get the full indexconfig json structure for each index in both get_index() and root() functions from devpi_web.views module. Now I'm not sure how much more data others might want to get there. I hope these things make sense. Any thoughts?

5 years

devpi-server/web-3.0: generalized mirroring, speed, new backends

by holger krekel

We just released devpi-{server,web}-3.0 and devpi-client-2.5.0 with some major new features, see the announcement below. didn't want to cram too many mailing lists into it in case someone hits "reply all" :) have fun, holger ----- Forwarded message from holger krekel <hol...(a)merlinux.eu> ----- Date: Fri, 12 Feb 2016 11:13:36 +0000 From: holger krekel <hol...(a)merlinux.eu> To: python announce <python-ann...(a)python.org>, Testing in Python <testing-...(a)lists.idyll.org>, disutils-sig <distut...(a)python.org> Subject: devpi-server/web-3.0: generalized mirroring, speed, new backends The 3.0 releases of devpi-server and devpi-web, the python packaging and work flow system for handling release files, documentation, testing and staging, bring several major improvements: - Due to popular demand we now support generalized mirroring, i.e. you can create mirror indexes which proxy and cache release files from other pypi servers. Even if the mirror goes down, pip-installing will continue to work with your devpi-server instance. Previously we only supported mirroring of pypi.python.org. Using it is simple: http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index - For our enterprise clients we majorly worked on improving the speed of serving simple pages which is now several times faster with private indexes. We now also support multiple worker processes both on master and replica sites. http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances - For our enterprise clients we also introduced a new backend architecture which allows to store server state in sqlite or postgres (which is supported through a separately released plugin). The default remains to use the "sqlite" backend and store files on the filesystem. See http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection - we started a new "admin" manual for devpi-server which describes features relating to server configuration, replication and security aspects. It's a bit work-in-progress but should already be helpful. http://doc.devpi.net/3.0/adminman/ - A few option names changed and we also released devpi-client-2.5 where we took great care to keep it forward and backward compatible so it should run against devpi-server-2.1 and upwards all the way to 3.0. - The "3.0" major release number increase means that you will need to run through an export/import cycle to upgrade your devpi-2.X installation. For more details, see the changelog and the referenced documentation with the main entry point here: http://doc.devpi.net Many thanks to my partner Florian Schulze and to the several companies who funded parts of the work on 3.0. We are especially grateful for their support to not only cover their own direct issues but also support community driven demands. I'd also like to express my gratitude to Rackspace and Jesse Noller who provide VMs for our open source work and which help a lot with the testing of our releases. We are open towards entering more support contracts to make sure you get what you need out of devpi, tox and pytest which together provide a mature tool chain for professional python development. And speaking of showing support, if you or your company is interested to donate to or attend the largest python testing sprint in history with a particular focus to pytest or tox, please see https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/ have fun, holger krekel, http://merlinux.eu server-3.0.0 (2016-02-12) ------------------------- - dropped support for python2.6 - block most ascii symbols for user and index names except ``-.@_``. unicode characters are fine. - add ``--no-root-pypi`` option which prevents the creation of the ``root/pypi`` mirror instance on first startup. - added optional ``title`` and ``description`` options to users and indexes. - new indexes have no bases by default anymore. If you want to be able to install pypi packages, then you have to explicitly add ``root/pypi`` to the ``bases`` option of your index. - added optional ``custom_data`` option to users. - generalized mirroring to allow adding mirror indexes other than only PyPI - renamed ``pypi_whitelist`` to ``mirror_whitelist`` - speed up simple-page serving for private indexes. A private index with 200 release files should now be some 5 times faster. - internally use normalized project names everywhere, simplifying code and slightly speeding up some operations. - change {name} in route_urls to {project} to disambiguate. This is potentially incompatible for plugins which have registered on existing route_urls. - use "project" variable naming consistently in APIs - drop calling of devpi_pypi_initial hook in favor of the new "devpi_mirror_initialnames(stage, projectnames)" hook which is called when a mirror is initialized. - introduce new "devpiserver_stage_created(stage)" hook which is called for each index which is created. - simplify and unify internal mirroring code some more with "normal" stage handling. - don't persist the list of mirrored project names anymore but rely on a per-process RAM cache and the fact that neither the UI nor pip/easy_install typically need the projectnames list, anyway. - introduce new "devpiserver_storage_backend" hook which allows plugins to provide custom storage backends. When there is more than one backend available, the "--storage" option becomes required for startup. - introduce new "--requests-only" option to start devpi-server in "worker" mode. It can be used both for master and replica sites. It starts devpi-server without event processing and replication threads and thus depends on respective "main" instances (those not using "--request-only") to perform event and hook processing. Each worker instance needs to share the filesystem with a main instance. Worker instances can not serve the "/+status" URL which must always be routed to the main instance. - add more info when importing data. Thanks Marc Abramowitz for the PR. web-3.0.0 (2016-02-12) ---------------------- - dropped support for python2.6 - index.pt, root.pt, style.css: added title and description to users and indexes. - root.pt, style.css: more compact styling of user/index overview using flexbox, resulting in three columns at most sizes - cleanup previously unpacked documentation to remove obsolete files. - store hash of doczip with the unpacked data to avoid unpacking if the data already exists. - project.pt, version.pt: renamed ``pypi_whitelist`` related things to ``mirror_whitelist``. - require and adapt to devpi-server-3.0.0 which always uses normalized project names internally and offers new hooks. devpi-web-3.0.0 is incompatible to devpi-server-2.X. - doc.pt, macros.pt, style.css, docview.js: use scrollbar of documentation iframe, so documentation that contains dynamically resizing elements works correctly. For that to work, the search from and navigation was moved into a wrapping div with class ``header``, so it can overlap the top of the iframe. 2.5.0 (2016-02-08) ------------------ - the ``user`` command now behaves slightly more like ``index`` to show current user settings and modify them. - fix issue309: print server versions with ``devpi --version`` if available. This is only supported on Python 3.x because of shortcomings in older argparse versions for Python 2.x. - fix issue310: with --set-cfg the ``index`` setting in the ``[search]`` section would be set multiple times. - fix getjson to work when no index but a server is selected - allow full urls for getjson - "devpi quickstart" is not documented anymore and will be removed in a later release. -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu ----- End forwarded message ----- -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu

5 years, 2 months

restrict user/indexnames for 3.0!?

by holger krekel

Hey all, we are preparing a devpi-server-3.0 beta with a lot of interesting features and speedups. One thing we'd still like to do is to restrict user and index naming. Currently we are thinking about only allowing: [a-z0-9A-Z_-]* Could you tell/verify if this matches your current installations? What more characters would you like to see? We can mangle characters when we import 2.X state into 3.X and probably also do redirects in the READ web interface (as used by pip) so it should be seamless even if some indexes use odd characters. best, holger -- about me: http://holgerkrekel.net/about-me/ contracting: http://merlinux.eu

5 years, 2 months

Preloading the PyPI cache for offline use

by Paul Moore

I want to have a devpi instance I can use for offline use. The server will be in offline mode 99% of the time, and only online when I make a special effort. So I want a way to "preload" the PyPI cache so that packages I might need for offline use are available. I'd expect to maybe go online once a week or so (possibly less, depends on how much development activity I'm doing at the time), and refresh the cache with latest versions. What's the best way of doing this? One option would be to do something like "pip download -r requirements.txt" to refresh all the packages in the given requirements file. I guess that would probably work, but it feels a bit clunky. And it doesn't really give me a way of doing anything more complex (for example, removing older versions of packages that I no longer need to have cached). Is there a better way to manage the cache? Thanks, Paul

5 years, 2 months

devpi-client-2.4.1: index modify fix

by holger krekel

just a quick note that we released devpi-client-2.4.1 with a bugfix for "devpi index key=value" commands which could previously fail sometimes due to API usage. Thanks Jacob Geiger for pointing it out. holger

5 years, 2 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

devpi-dev February 2016