Mailman 3 July 2013 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

error in windows 7 installation
by Manuel López 13 May '14

13 May '14

Hello Everybody: I am trying to install distribute-0.6.25 in a windows 7 machine. I have python 2.7.3 in 32 bits, although the machine is 64 bits. I installed the 32 bit python version because I want to install ipython and there is no 64 bit builds of the Windows installer for ipython. In any case, upon installing disuitils with python setup.py install I get the following error error: C:\manuel\prog_red\python\distribute-0.6.25\distribute-0.6.25\setuptools\cli-32.exe: No such file or directory Any help will be greatly appeciated. Thanks, Manuel -- Manuel López Mariscal Depto. de Oceanografía Física/CICESE malope(a)cicese.mx

2 1

503 errors from PyPI
by Paul Moore 23 Feb '14

23 Feb '14

Whenever I do a PyPI search at the moment, I get Error 503 backend read error backend read error Guru Meditation: XID: 477806858 ________________________________ Varnish cache server Paul

3 6

What to do about the PyPI mirrors
by Richard Jones 07 Aug '13

07 Aug '13

Hi all, I've just been contacted by someone who's set up a new public mirror of PyPI and would like it integrated into the mirror ecosystem. I think it's probably time we thought about how to demote the mirrors: - they cause problems with security (being under the python.org domain causes various issues including inability to use HTTPS and cookie issues) - they're no longer necessary thanks to the CDN work So, things to do: - links and information on PyPI itself can be removed - tools that use mirrors still need to be able to but mention of using public mirrors is probably something to demote These are just rough thoughts that occurred to me just now. Richard

13 58

Migrating Hashes from MD5 to SHA256
by Donald Stufft 02 Aug '13

02 Aug '13

PyPI has historically used MD5 in order to verify the downloads. However MD5 is severely broken and is generally regarded as something that should be migrated away from ASAP. From speaking with a number of cryptographers they've more or less said that the major reason they believe that MD5 hasn't had a published pre-image attack is just because it's so broken that most researchers have moved on to newer hashes. Since versions 1.2 pip has supported md5, sha1, and any of the sha2 family. Additionally it has only supported SSL verification since 1.3. This means there is no version of pip which both verifies SSL and only allows MD5. Since version 0.9 setuptools has supported md5, sha1, and any of the sha2 family and it has only supported SSL verification since 0.7. I propose we switch PyPI from using MD5 to using SHA256. There is no security lost from using a hash that pip prior to version 1.2 doesn't understand as it didn't verify SSL so an attacker could simply modify the hashes if they wanted. Additionally there is no security list from setuptools versions earlier than 0.7. However setuptools versions 0.7-0.8 will lose their hashes. I believe this is a ok thing to happen because the uptake on 0.7-0.8 is fairly low. Most people will use the setuptools bundled with virtualenv which has only ever bundled 0.6 or 0.9. So essentially moving from MD5 to SHA256 will only negatively affect the security of a handful of users while positively impacting the security of everyone else. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

16 81

Status report on PyPI+pip+TUF
by Trishank Karthik Kuppusamy 01 Aug '13

01 Aug '13

Hello Nick and the PyPI community, This is a brief status report on the integration of PyPI and pip with TUF. (A quick reminder: TUF is a general "plug-n-play" update framework designed to introduce usable security to community software repositories such as PyPI. If you think of PyPI as HTTP, then TUF is like adding SSL, and more, to HTTP. More information may be found at [https://www.updateframework.com/].) Firstly, thanks to the generous funding of the National Science Foundation, we are pleased to introduce the addition of a full-time developer, Vladimir Diaz, to our team. Vladimir has been instrumental to the development of TUF, and we are excited to have him join us full-time. (Now we do not just have one PhD student who works on TUF when he is not busy working on other projects!) We are also happy to have a few interns --- Zane Fisher, Tian Tian, John Ward, and Yuyu Zheng --- on board for the summer. Since the security attacks on the Python wiki infrastructure earlier this year, we have been closely following Distutils-SIG to see what we could do to help secure PyPI. We use Python heavily in all of our projects, and would love to help in any way we can. Here is what we have done: ========================== 1. At PyCon 2013, we showed that pip needs very little modification to work with a TUF-enabled PyPI mirror. 2. Soon after (during the spring break), we wrote automation to build a TUF-secured PyPI mirror (which is indistinguishable from any other PyPI mirror except that it has signed metadata about all of the files on PyPI). 3. At the same time, thanks to efforts of Konstantin Andrianov, we also wrote a lot of unit and integration tests to show the attacks that are possible without TUF and impossible with TUF. 4. After that, we started investigating the most efficient way to build TUF metadata for PyPI. We found that requiring a separate key for every package on PyPI may sound like a good idea, but besides generating too much metadata, this scheme also makes key management difficult. Here is what we are doing now: ============================== We are designing a usable key management scheme, coupled with efficient generation and download of metadata, which we think should make for a smooth integration of PyPI with TUF. We are actively working on this and think that we are almost there. As a conservative estimate, we do not believe that this should take longer than two weeks. Here is what we are going to do next: ===================================== In about a month, we will present to you a demonstration of a PyPI mirror and a pip client which are robust against entire classes of security attacks. We welcome you then to try our demo, be really critical of it and tell us what you think about what we could do better. Our goal with TUF is to provide a framework that works with as many software community repositories as possible and that secures as many users as possible. More details on our development are available at our mailing list: https://groups.google.com/forum/#!forum/theupdateframework We hope this gives you a good idea of the current status of integrating TUF with PyPI and pip. Let us know if you have questions. Thanks, The TUF team

2 4

Request to add a trove classifier for pelican plugins
by Alexis Métaireau 31 Jul '13

31 Jul '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I hope this is the right place to ask for this. I would like to have a trove classifier for pelican [0] plugins. We plan to release them on PyPI and having a classifier to distinguish them from all the other packages sounds the way to go. We're not really a framework, but following the already established pattern, I guess "Framework :: Pelican" makes sense. Thanks! Alexis [0] http://getpelican.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJR5CLuAAoJEAeIBdhn9W8SyqwH/3e64+X6KJ4WxX/zeO9isKqw F5XfeRieO2rthUd6ALREF+VYhgsIwrU6B6gwLjyDe5tFA8a0sPkXFIg7pCEXjNxt ufX7W8BhdjRcVOx//9/zP4v4+HeU9OUZBwpiNuMnE7N9jbq4iWtt3OQ0GtfVIl1h d9WJoxb+8aDGbes/jgNuTh4B/Jm9XjIm8fXP5mcyLkfj0vnyHJUTgm/GjnuYEF6K 9mtWMAMoZ6RgIS41JgI7yUt81pLBQrCJc5yVuG7lE3hdaFstKAdBZhOVSrIzj0eJ KN4TMABexVCsVfwklDPIifAneKQZEmONUJLWfzleU367kRGb5YNavrtpW/kQomY= =FomK -----END PGP SIGNATURE-----

2 2

Warehouse Migration Plan
by Donald Stufft 31 Jul '13

31 Jul '13

So, in the spirit of not treating distutils-sig like an adversary, here's the main thing I've been working on lately with regards to PyPI. None of this is set in stone but this is the general gist of the plan for moving things to be developed in a modern framework, as well as cleaning up the code and getting repeatable deployments. Warehouse Migration Plan ------------------------------------ Warehouse is currently primarily besides modeling for user accounts. It will be deployed alongside pypi-legacy at next.pypi.python.org in the near future. Initially it will have zero user facing value. As time goes on the database tables will be migrated from being "owned" by pypi-legacy to being "owned" by Warehouse. This primarily means that the schema definition and migration of those tables will be controlled by Warehouse. As tables are migrated to Warehouse ownership the PyPI code will be updated to reflect any changes in schema (without modification to what end users see). Once all tables that are going to be kept have been migrated, we will have a shared database that is accessible from both pypi-legacy and Warehouse. From this point Warehouse will begin evolving "legacy" views such as the simple and other pieces of API. The UX itself will continue to be ignored and focus will be on getting feature parity for automated tooling. Changes in behavior on these legacy views should be minimal and discussed on distutils-sig. Once the legacy views are finished people will be encouraged to test their real world workloads against those reimplemented legacy APIs. As changes in behaviors, missing functionality, or bugs are found they will be rectified or declared unsupported. During this time work on the UI of Warehouse will begin focusing on maintaing feature parity but with no promises of no changes to the url structure or UX. Once Warehouse gains feature parity with PyPI and has gotten enough testing against it's APIs then pypi-legacy will be retired and Warehouse will move from next.pypi.python.org to pypi.python.org. From there it will evolve on it's own without needing to keep pypi-legacy in sync. Specification & Acceptance Testing ------------------------------------------------ I do not want a packaging index server to be specified by implementation, so as the legacy API is ported over to Warehouse a specification will be drafted. This spec will represent the "promise" that PyPI makes about the API it presents to be consumed by machines. During the migration any behavior not codified inside of the spec is considered implementation defined behavior and backwards compatibility for that behavior will not be promised. In addition to a defined specification A repository of acceptance tests will be developed. These tests will be part of the test framework for any future changes to Warehouse but will be maintained separately alongside the specification. They will also allow any alternative implementation (such as DevPI) to test themselves against the spec. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

3 3