<div dir="ltr">* It is [currently [#PEP426JSONLD)] necessary to run setup.py with each given destination platform, because parameters are expanded within the scope of setup.py.<div> * Because of this, client side dependency resolution (with a given platform) is currently the only viable option for something like this</div><div><br></div><div>...</div><div><br></div><div>* Build: Docker, Tox (Dox) to build package(s)</div><div> * Each assembly of packages is / could be a package with a setup.py (and/or a requirements.txt)</div><div> * And tests:</div><div> * <a href="http://conda.pydata.org/docs/building/meta-yaml.html#test-section">http://conda.pydata.org/docs/building/meta-yaml.html#test-section</a></div><div>* Release: DevPi</div><div> * <a href="http://doc.devpi.net/latest/">http://doc.devpi.net/latest/</a></div><div>* conda env environment.yml YAML: <a href="http://conda.pydata.org/docs/using/envs.html">http://conda.pydata.org/docs/using/envs.html</a></div><div> * [x] conda packages</div><div> * [x] pip packages</div><div> * [ ] system packages (configuration management)</div><div><br></div><div>And then, really, Is there a stored version of this instance of a named Docker image?</div><div>#reproducibility #linkedreproducibility</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Nov 7, 2015 at 8:37 AM, Thomas Güttler <span dir="ltr"><<a href="mailto:guettliml@thomas-guettler.de" target="_blank">guettliml@thomas-guettler.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I wrote down a tought about Serverside Dependency Resolution and Virtualenv Build Server<br>
<br>
<br>
What do you think?<br>
<br>
Latest version: <a href="https://github.com/guettli/virtualenv-build-server" rel="noreferrer" target="_blank">https://github.com/guettli/virtualenv-build-server</a><br>
<br>
virtualenv-build-server<br>
#######################<br>
<br>
Rough roadmap how a server to build virtualenvs for the python programming language could be implemented.<br>
<br>
Highlevel goal<br>
--------------<br>
<br>
Make creating new virtual envionments for the Python programming language easy and fast.<br>
<br>
Input: fuzzy requirements like this: django>=1.8, requests=>2.7<br>
<br>
Output: virtualenv with packages installed.<br>
<br>
Two APIs<br>
------------<br>
<br>
#. Resolve fuzzy requirements to a fixed set of packages with exactly pinned versions.<br>
#. Read fixed set of packages. Build virtualenv according to given platform.<br>
<br>
<br>
Steps<br>
-----<br>
<br>
Steps:<br>
<br>
#. Client sends list of fuzzy requirements to server:<br>
<br>
* I need: django>=1.8, requests=>2.7, ...<br>
<br>
<br>
#. Server solves the fuzzy requirements to a fixed set of requirememnts: django==1.8.2, requests==2.8.1, ...<br>
<br>
#. Client reads the fixed set of requirements.<br>
<br>
#. Optional: Client sends fixed set of requirements to the server. Telling him the plattform<br>
<br>
* My platform: sys.version==2.7.6 and sys.platform=linux2<br>
<br>
#. Server builds a virtualenv according to the fixed set of requirements.<br>
<br>
#. Server sends the environment to the client<br>
<br>
#. Client unpacks the data and has a usable virtualenv<br>
<br>
Benefits<br>
--------<br>
<br>
Speed:<br>
<br>
* There is only one round-trip from client to server. If the dependencies get resolved on the client the client would need to download the available version information.<br>
* Caching: If the server gets input parameters (fuzzy requirements and platform information) which he has seen before, he can return the cached result from the previous request.<br>
<br>
Possible Implementations<br>
------------------------<br>
<br>
APIs<br>
====<br>
Both APIs could be implementated by a webservice/Rest interface passing json or yaml.<br>
<br>
Serverside<br>
==========<br>
<br>
<br>
Implementation Strategie "PostgreSQL"<br>
.....................................<br>
<br>
Since the API is de-coupled from the internals the implementation could be exchanged without the need for changes at the client side.<br>
<br>
I suggest using the PostgreSQL und resolving the dependcy graph using SQL (WITH RECURSIVE).<br>
<br>
The package and version data gets stored in PostgreSQL via ORM (Django or SQLAlchemy).<br>
<br>
The version numbers need to be normalized to ascii to allow fast comparision.<br>
<br>
Related: <a href="https://www.python.org/dev/peps/pep-0440/" rel="noreferrer" target="_blank">https://www.python.org/dev/peps/pep-0440/</a><br>
<br>
Implementation Strategie "Node.js"<br>
..................................<br>
<br>
I like python, but I am not married with it. Why not use a different tools that is already working? Maybe the node package manager: <a href="https://www.npmjs.com/" rel="noreferrer" target="_blank">https://www.npmjs.com/</a><br>
<br>
Questions<br>
---------<br>
Are virtualenv relocatable? AFAIK they are not.<br>
<br>
General Thoughts<br>
----------------<br>
<br>
* Ignore Updates. Focus on creating new virtualenvs. The server can do caching and that's why I prefer creating virtualenvs which never get updated. They get created and removed (immutable).<br>
<br>
<br>
I won't implement it<br>
--------------------<br>
<br>
This idea is in the public domain. If you are young and brave or old and wise: Go ahead, try to implement it. Please communicate early and often. Ask on mailing-lists or me for feedback. Good luck :-)<br>
<br>
I love feedback<br>
---------------<br>
<br>
Please tell me want you like or dislike:<br>
<br>
* typos and spelling stuff (I am not a native speaker)<br>
* alternative implementation strategies.<br>
* existing software which does this (even if implemented in a different programming language).<br>
* ...<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
<a href="http://www.thomas-guettler.de/" rel="noreferrer" target="_blank">http://www.thomas-guettler.de/</a><br>
_______________________________________________<br>
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org">Distutils-SIG@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br>
</font></span></blockquote></div><br></div>