On Thu, Oct 16, 2014 at 13:21 +0200, Florian Schulze wrote:
I guess the dependencies would be stored with the same info as the test results, like platform, python version etc?
Indeed, enriching the tox result file and then parsing/reading it on the server side is probably the way to go. One related question is how to store the tox result files. Currently they are just tied to the sdist link (wheels cannot have tox.ini files to begin with) which was obtained from "devpi test" as the best matching link. But i think server-side we need to detach it (again) from the sdist link and rather store them indexed by the MD5 of the sdist content and also by the md5 of of all dependencies. The idea is to be able to efficiently: - query all test results where this release file (or release version) was a part of the (test) dependencies - query all "testing tasks" caused by changed dependencies (this could be polled from a new "devpi test-server" subcommand which executes "pending testing tasks" for the platform it runs on) - show and record dependency/test information even for root/pypi projects (like we used to) Changing the server storage and access methods for test results does not neccessarily change the client facing UI i think. We can still provide the same json but we produce it differently.
The only "unsafe" part I can think of is a general issue, malicious archives which exhaust server resources.
We can have an upper found because 640MB archives should be enough for anyone! But i think for wheels the reading of the metadata does not require full unpacking but just some lightweight iteration, anyway :) holger
Regards, Florian Schulze
On 16 Oct 2014, at 13:08, holger krekel wrote:
Hi all,
FYI i thought about how we make devpi-server know about which project depends on which other projects. With that information we could do all kinds of good things:
- display depency info on the per-project web page or along with release files (this project depends on ProjectY and ProjectZ)
- display if all recent versions of deps are properly working and tested with a project's latest release
- could trigger server-side "dependency changed" events so that for example a tox run could be triggered for the new test configuration
- create pin-versioned requirement files that could be used with "pip install -r tested-requirements.txt", and/or possibly a UI like "devpi rinstall X" where it would query the latest set of dependencies for which tests passed, download all according files and then run "pip install --no-index FILE1 FILE2 [...]" which wouldn't require any more network access.
Question is how to best get the (closure) set of dependencies for a project. I cam currently pondering the following possibilities to obtain the information at server side:
- if the project has release files as wheels, look at wheel metadata which lists deps (requires just virtually unzipping a wheel and looking at safe metadata files)
- "devpi test" could run "setup.py egg_info" and send the requirements it finds to the server (requires login), additionally it should probably "pip list" all test dependencies in the respective tox environments and add them as well because if test dependencies change, tests should be re-run as well.
These two methods would not require any change in client-facing UI and allow us to get and display the dependencies information.
Any comments or thoughts on the matter welcome.
best, holger
-- You received this message because you are subscribed to the Google Groups "devpi-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to devpi-dev+...@googlegroups.com. To post to this group, send email to devp...@googlegroups.com. Visit this group at http://groups.google.com/group/devpi-dev. For more options, visit https://groups.google.com/d/optout.