Mailman 3 November 2013 - Distutils-SIG

Download count for external packages on PyPI
by Stefan Krah 24 Nov '13

24 Nov '13

[I'm not subscribed, could you please cc me?] Hi, On the danger of sounding like a broken record: PyPI displays incorrect download counts (0) for external packages. Would it be possible to skip the download count entirely for external packages? Stefan Krah

2 3

PyPI v2 (Was: PyPI pull request #7)
by anatoly techtonik 23 Nov '13

23 Nov '13

On Thu, Oct 31, 2013 at 2:38 PM, Donald Stufft <donald(a)stufft.io> wrote: > > On Oct 31, 2013, at 7:32 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > >> Why I am skeptical that new site will replace old one soon? Just >> because I don't believe in rewrites by one man army. When you develop >> public resource, you need to rely on external feedback. > > Warehouse is live at https://preview-pypi.python.org/ (No landing page yet) Not very impressive - I don't enjoy huge box for one word query - guess Google is small for a reason. =) Updated https://wiki.python.org/moin/CheeseShopDev It still needs to fill up few historical details for historic preservation: 1. How many PyPI versions was there? 2. When the first one was deployed? (web.archive.org 1st reference is dated 2007) 3. Where it was hosted, who were authors? 4. Other interesting historic facts I like the initiative - there is something to learn from (if you're already proficient to read advanced web code). Still I am not yet ready to help with something I can't use. I guess I need to wait until beta, or until there is some two-week spring-planning process, where I can see if there is something worthy to add. Still I couldn't resist to ask about opinion on things that don't match my idealistic universe. First some background questions: 1. Everything for core development is in HG. Why Git now? Why Mercurial suxx (three major personal annoyances will do)? Why BitBucket suxx? 2. Why Apache 2.0? Is it because everybody is using that? Why not try CC0/CC-BY/MIT/ISC - code under these licenses is easier to copy/paste, and is a better investment for your time as a coder studying the stuff and filling up the space of your brain. 3. Why the raw SQL? It cripples the ability to scale, to host package indexes on GAE or OpenStack. And it is right at the core. Why not to "port NDB" to SQL or use HyperDB from Roundup? 4. Why not Django/TG/... framework? (+pinax, +openid, +other_stuff) >> You also need >> some designer guy in a team. > > Have one who’ll have time in a few weeks to go over everything and make it really great :) That's cool. Will it be an iterative process or just one shot? Do you care about public feedback? I mean of course you do, but is there a public channel to allow users set status if they liked the previous version more, and what should be changed for them to change their opinion? >> You also need a backlog for >> collaboration. My ETA for new PyPI is no earlier than PyCon 2014 if >> Donald and Richard will be working on it full time. > > Possibly! I’m unsure of how long it will take, it’s primarily Richard and I but we’ve a > few domain experts in particular pieces who have offered to help out as well when > we’re ready for their pieces. PostgreSQL is killing all the fun. It takes a few hours to get acquainted with its way of doing things - groups as users, per table grants and insufficient DB permissions. There should be an option to run on SQlite for development, like in Trac, Roundup etc. >> So, instead of >> all-or-nothing scenario I can try to find some help with incremental >> approach. > > Mostly the problem with improving the current base is every change is particularly > dangerous. The code base is large, it’s untested, and it’s not very nice. It’s extremely > easy to break things by accident with seemingly unrelated change. Richard and I > have both done this multiple times. This smells like a bad spaghetti. Do you think it will be maintainable if it is already so fragile on the early stages of development? > So every PR we accept has a danger of breaking > things. This incurs a high cognitive overload for accepting a PR because it means we > have to spin up a copy of the site and manually go through and test any feature we > think *might* be affected which typically catches most things but not always. It’s a time > and labor intensive process which none of us enjoy and which we’ve decided not to > prioritize unless the pay offs are large. If you're not doing this only for yourself and your own pleasure, then dedicating more time to improve the process and making it more inclusive will definitely pay off for the rest of pydotorg projects and active part of Python community, who is still here. Do you have a diagram of the system what are you trying to build, or is it just an experiment? It may worth to put IDE aside and try to de-couple some things first on the whiteboard. I am afraid that I will never be ready to help with reinventing Django. At least not without some research and art coverage. Most of this work about best practices, collaboration experiments and people communication should be made by PSF. When I donate to PSF I hope it thinks about and develops ways to encourage collaboration, help people grow in Python, share experience and learn from each other. Sprints, pydotorg projects, public discussions and code reviews - they should all work together. There are good examples of open source collaboration in Qt, Chromium and Android projects. The process can be streamlined and fun, but nobody can even meet each other to work on commonly interested things. Only random people who once a year can afford time and budget to visit PyCon. I dislike "motivational" blog posts about initiatives, without real visible research and actions with weekly retrospectives. For that I'd propose PSF to change to post-factum writing style from motivational, especially when the future mechanism of the good intent is unknown or unreliable. And the first thing it needs to do - is to start helping people inside pydotorg communicate with outside, removing the barriers and telling what help is needed, who is needed and where. -- anatoly t.

3 5

PEP 458: Surviving a Compromise of PyPI: Round 1
by Trishank Karthik Kuppusamy 23 Nov '13

23 Nov '13

Hello everyone, Donald, Justin and I have co-authored a PEP that recommends a comprehensive security solution to allow PyPI to secure its users against a wide array of compromises. The gist of the PEP is that the changes to PyPI are essentially invisible to users and developers unless an attack is underway. The key design ideas are as follows: * The main PyPI server will continue running as it is now, exposing HTTPS and legacy XML-RPC operations. * The next-generation PyPI server (Warehouse) will be exposing new API as well as TUF metadata to clients. * Developers do not have to opt-in to secure their projects with their own TUF metadata. In that case, PyPI will sign these "unclaimed" projects on their behalf. However, unclaimed projects will not be secure against a PyPI compromise. * To protect against a PyPI compromise, developers may choose to register their public keys with Warehouse and upload their own signed TUF metadata about their projects. * Therefore, developers do not have to concern themselves with key management in case they leave their projects as "unclaimed". When they do claim their projects, they simply have to register their keys once with Warehouse. After that, they may delegate signing for distributions as they wish without depending on Warehouse. * Clients will be instructed to first search for a project in the more secure claimed metadata (protected by offline keys) before looking for it in the less secure unclaimed metadata (protected by online keys). * Whether or not a project is claimed or unclaimed, all projects will be available through continuous delivery. * Consistent snapshots allow clients and mirrors to safely read metadata and data despite the addition of new files to PyPI. * It is efficient to securely install or update a project despite hundreds of thousands of files. The official PEP is here: http://www.python.org/dev/peps/pep-0458/ Whereas latest revisions to the PEP are here: https://github.com/theupdateframework/pep-on-pypi-with-tuf We welcome your feedback and suggestions. Thanks, The PEP 458 team

7 10

Distribution format for Python3 libraries
by Thomas Heller 19 Nov '13

19 Nov '13

What is the preferred format to distribute Python-3 libraries (for Windows) nowadays: wininst, egg, ...? It has been a long time that I have released something. Thanks, Thomas

6 14

sdist README.rst without MANIFEST
by anatoly techtonik 19 Nov '13

19 Nov '13

I use this in setup.py: .. long_description = open('README.rst', 'rb').read(), .. Which fails during installation, because README.rst is not shipped. The usual way to include is through MANIFEST.in http://docs.python.org/2/distutils/sourcedist.html Is it possible to include README.rst with source distribution, but without creating yet another file? Is it possible to specify includes in setup.py? -- anatoly t.

2 3

Warehouse and XMLRPC
by Donald Stufft 19 Nov '13

19 Nov '13

The bulk of XMLRPC has been implemented in Warehouse. It would be great if people who are using the XMLRPC could try it out using the https://preview-pypi.python.org/pypi url. Currently the search API does not work. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

6 8

pip feedback to user...
by Chris Barker 18 Nov '13

18 Nov '13

Is this the right place to discuss UX issues for pip? If not, point to the right place, if so, read on: I think pip's usability could be much improved with a little tweaking to the messages it prints to the console when it does its thing. For instance, when I do a: pip install some_package I get the message: Downloading/unpacking some_package When in fact, pip is not (yet) doing that -- what it is doing is looking for the package on pypi (or ???) Which is fine if the package is found, but if it's not, you then get some confusing messages, like: Could not find any downloads that satisfy the requirement some-package Cleaning up... not to bad -- it does say it wasn't found, but it seems it could be a bit odd to a new user -- "but I thought it was being downloaded?" I was just trying to upgrade a packe that I was told could now be pip installed: $ pip install --upgrade tracpy Could not find any downloads that satisfy the requirement tracpy in /Users/chris.barker/PythonStuff/TracPy/tracpy Downloading/unpacking tracpy Cleaning up... No distributions at all found for tracpy in /Users/chris.barker/PythonStuff/TracPy/tracpy Storing complete log in /Users/chris.barker/.pip/pip.log It turns out that they have not (yet?) registered it with pypi (and I have a n older copy in /Users/chris.barker/PythonStuff/TracPy/tracpy). But it does seem weird and confusing that it first says it can't find any downloads, then tells me its downloading, then cleaning up, then says it couldn't find anything... Anyway, small stuff, but these little things make a difference. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 1

devpi-{server, client}-1.2.1: bug fixes + refinement to test
by holger krekel 18 Nov '13

18 Nov '13

devpi-1.2.1: bug fixes and improved test command ========================================================== The devpi-{server,client}-1.2.1 releases bring important bug fixes and refinements. See Changelog below. For getting started with deploying your own pypi server on a laptop or on company server and for using the "devpi" workflow tool (optional), see: http://doc.devpi.net If you want to upgrade an existing installation, you should be able to execute:: $ pip install -U devpi $ devpi-server --upgrade-state [--serverdir YOUR_SERVER_DIR] Have fun, holger krekel Changelog 1.2.1 ---------------------- devpi-server: - fix an import issue for doc files which were wrongly tied to a newer version of a base index. now version "auto" detection for storing doc files only works within a stage. Thanks Laurent Brack for bringing it up and providing the repo. - fix issue66: api endpoints now also respect --outside-url setting so that you can serve devpi from a subpath. Thanks for Fabian Snovna for reporting and analysis. - fix issue63: skip egg links that go to a directory (this requires doing a SVN checkout which devpi-server does not do). Thanks Ken Jung for analyzing the problem. - fix issue68: don't derive metadata from filename but instead look it up in metadata or submitted form. - fix cache-invalidation when normalized_project_name != real_name (e.g. for Django but also many others). addresses issue59. - add newline to simple list output for better human readability of the page (thanks Brandon Maister) - make xmlrpc calls to pypi's changelog API use "requests" sessions so that http proxies are respected there as well (fixes issue58). thanks to riehlm for identifying the problem and testing the fix. - internally refactor and consolidate mocking against requests library - --upgrade-state will upgrade now between major.minor/major.minor+1 changes. devpi-client: - fix "python -m devpi" invocation. Thanks Sebastian Ralph. - fix issue66: "devpi use user/index" can now switch between URLs if user/index is mounted on a subpath. - fix issue71: allow pip/setuptools like requirements specs with the test subcommand, e.g. "devpi test 'pkg>=1.0'". Thanks Sebastian Rahlf for the PR.

1 0

venv and _ensurepip
by Paul Moore 17 Nov '13

17 Nov '13

I've not seen any documentation and/or code (other than the PEP) yet for the _ensurepip changes that will be going into Python 3.4, but can I check what the intention is for people using the extensibility API in venv? Specifically, if I want to install something as well as (or other than) pip/setuptools when I create a venv, will venv.EnvBuilder support that? Specifically, I'd need to be able to say whether or not I want _ensurepip to run in the venv. Ideally, I'd also like it if the temporary pip instance that _ensurepip uses was exposed somehow, so that I could use it to (for example) install extra packages in my venv build script. Obviously I could just run pip again after the venv is created but (1) only if I install pip (I might want a builder that uses pip to install another package, but I don't want pip in the final venv) and (2) that's an extra process run, which has a non-trivial runtime cost on Windows at least. Obviously I'll be looking at testing this type of thing once the beta comes out, but I wondered if the scripting API for venv was something that had been thought about in advance. Paul

2 3

"pip uninstall" and pip's lack of support for installing eggs
by Marcus Smith 15 Nov '13

15 Nov '13

Under the covers, pip uses "setup.py install --record" (which requires also using --single-version-externally-managed) in order to create an install log, that get's used for uninstalls. And due to using `--single-version-externally-managed`, the install is done "old-style", i.e., not installed as an egg, and not knowing how to install *from* an egg. the one caveat being that setuptools (not pip) does throw in an .egg-info directory with the project metadata. So is "pip uninstall" the primary historical reason for the lack of egg install support? or is there more to the story? Marcus

3 6