On 18 September 2014 18:01, Andrew Barnert
On Sep 17, 2014, at 23:15, Nick Coghlan
wrote: However, now that CPython ships with pip by default, we may want to consider providing more explicit pointers to such "If you want more advanced functionality than the standard library provides" libraries.
I love this idea, but there's one big potential problem, and one smaller one.
Many of the most popular and useful packages require C extensions. In itself, that doesn't have to be a problem; if you provide wheels for the official 3.4+ Win32, Win64, and Mac64 CPython builds, it can still be as simple as `pip install spam` for most users, including the ones with the least ability to figure it out for themselves.
OK, the key thing to look at here is the user experience for someone who has Python installed, and has a job to do, but needs to branch out into external packages because the stdlib doesn't provide enough functionality. To make this example concrete, I'll focus on a specific use case, which I believe is relatively common, although I can't back this up with hard data. Assume: * A user who is comfortable with Python, or with scripting languages in general * No licensing or connectivity issues to worry about * An existing manual process that the user wants to automate In my line of work, this constitutes the vast bulk of Python use - informal, simple automation scripts. So I'm writing this script, and I discover I need to do something that the stdlib doesn't cover, but I feel like it should be available "out there", and it's sufficiently fiddly that I'd prefer not to write it myself. Examples I've come across in the past: * A console progress bar * Scraping some data off a web page * Writing data into an Excel spreadsheet with formatting * Querying an Oracle database Every time an issue like this comes up, I know that I'm looking to do "pip install XXX". It's working out what XXX is that's the problem. So I go and ask Google. A quick check on the progress bar case gets me to a StackOverflow article that offers me a lot of "write it yourself" solutions, and pointers to a couple of libraries. Further down there are a few pointers to python-progressbar, which was mentioned in the StackOverflow article, which in turn leads me to the PyPI page for it. The latest version (2.3-dev) is not hosted on PyPI, so I hit all the fun of --allow-external. Ironically, "pip install tqdm" gives me what I want instantly. But it never came up via Google. The rest of the cases are similar, lots of Google searching, often combined with evaluating multiple options, followed by more or less pain installing the software. Things that aren't Python 3 or Windows compatible suck me into the "shall I patch it and submit a PR" minefield. For the last case (an Oracle driver), where I need a C extension and access to external libraries, ironically it's pretty easy. There's no real competition to cx_Oracle, and the PyPI page has what I need, although they ship wininst exes rather than wheels, which means I need to do a download then a wheel convert then a pip install, so it's not ideal, but doable.
From this example, I'd like to see the following improvements to the process:
1. Somewhere I can go to find useful modules, that's better than Google. 2. Someone else choosing the "best option" - I don't want to evaluate 3 different progressbar modules, I just want to write "57% complete" and a few dots! 3. C extensions aren't a huge problem to me on Windows, although I'm looking forward to the day when everyone distributes wheels (wheel convert is good enough for now though). [1] 4. Much more community pressure for projects to host their code on PyPI. Some projects have genuine issues with hosting on PyPI, and there are changes being looked at to support them, but for most projects it seems to just be history and inertia. [1] A Linux/OS X user might have more more issues with C extensions. Maybe this can't be solved in any meaningful sense, and maybe it's not something the "Python project" should take responsibility for, but without any doubt, it's the single most significant improvement that could be made to my experience with PyPI. Paul. PS I should also note that even in its current state, PyPI is streets ahead of the 3rd party module story I've experienced for any other language - C/C++, Lua, Powershell, and Java are all far worse. Perl/CPAN may be as good or better, it's so long since I used Perl that I don't really know these days.