On Sep 18, 2014, at 11:26, Paul Moore firstname.lastname@example.org wrote:
On 18 September 2014 18:01, Andrew Barnert email@example.com wrote:
On Sep 17, 2014, at 23:15, Nick Coghlan firstname.lastname@example.org wrote:
However, now that CPython ships with pip by default, we may want to consider providing more explicit pointers to such "If you want more advanced functionality than the standard library provides" libraries.
I love this idea, but there's one big potential problem, and one smaller one.
Many of the most popular and useful packages require C extensions. In itself, that doesn't have to be a problem; if you provide wheels for the official 3.4+ Win32, Win64, and Mac64 CPython builds, it can still be as simple as `pip install spam` for most users, including the ones with the least ability to figure it out for themselves.
OK, the key thing to look at here is the user experience for someone who has Python installed, and has a job to do, but needs to branch out into external packages because the stdlib doesn't provide enough functionality.
To make this example concrete, I'll focus on a specific use case, which I believe is relatively common, although I can't back this up with hard data.
- A user who is comfortable with Python, or with scripting languages in general
- No licensing or connectivity issues to worry about
- An existing manual process that the user wants to automate
In my line of work, this constitutes the vast bulk of Python use - informal, simple automation scripts.
So I'm writing this script, and I discover I need to do something that the stdlib doesn't cover, but I feel like it should be available "out there", and it's sufficiently fiddly that I'd prefer not to write it myself. Examples I've come across in the past:
- A console progress bar
- Scraping some data off a web page
- Writing data into an Excel spreadsheet with formatting
- Querying an Oracle database
Every time an issue like this comes up, I know that I'm looking to do "pip install XXX". It's working out what XXX is that's the problem.
So I go and ask Google.
Hold on. I'm pretty sure that the intended answer to this problem has, for years, been that you go and search PyPI. Is that too broken to use, or are people just not aware of it?
A quick check on the progress bar case gets me to a StackOverflow article that offers me a lot of "write it yourself" solutions, and pointers to a couple of libraries.
StackOverflow is the _last_ place you should be looking. It's part of their policy that "library-shopping" questions should be closed, and has been for quite some time. Anything you find is likely to be either out of date, or in some niche area that few of the active users see.
From this example, I'd like to see the following improvements to the process:
- Somewhere I can go to find useful modules, that's better than Google.
Again, isn't that PyPI?
- Someone else choosing the "best option" - I don't want to evaluate
3 different progressbar modules, I just want to write "57% complete" and a few dots!
I don't think anything like this can be "curated", unless it really is restricted to the dozen or two projects that are clearly "best in category" in areas of widespread demand, as (I think) Nick and Terry were discussing.
So you're looking for something crowd sourced, which I don't think exists yet. Maybe the newish Software Recs StackExchange site will eventually serve. Or maybe someone will build something specific to the Python community, possibly even built on top of PyPI. But whatever it is, I don't think it's going to be designed by a mailing list discussion; someone has to have a clever idea and hack it up until it works, or at least inspires further ideas. The same way we got easy_install and then pip.
There's also the problem that in many areas "the best" is different for different applications, and often you don't know the right question to ask. The best console-mode progress bar, OK, people can disagree with the answer, but they won't disagree much on the criteria. But for anything less trivial, that's not likely to be true. Which is the best XMPP client library, plugin framework, arbitrary-precision float library, sorted mapping, markdown converter, ...?
In other words, I agree that this is an important problem, but it's not going to be an easy one to solve, and I don't think solving it should be a prerequisite, or even really relevant, to Nick's stdlib++ idea.
- C extensions aren't a huge problem to me on Windows, although I'm
looking forward to the day when everyone distributes wheels (wheel convert is good enough for now though). 
I think this is pretty uncommon for the user group you're representing. Most Windows users who are comfortable with scripting languages don't have MSVC or MinGW set up, don't know how to do so (or even which one they should choose), etc. Many packages come with official Windows binaries, but an awful lot of them don't--and, even when they do, the "install" docs often start off with how to set up prereqs and build on *nix machines. Most of the Windows users I see tend to go straight to Christoph Gohlke's archive and use his installers, and for anything not there they just throw their hands up and panic.
Making wheels widespread is not just a nice-to-have; wheels are a great solution to a very real problem, and until they're pervasive the problem isn't solved.
But meanwhile, wheels don't solve everything. I don't want to repeat the whole second point from my previous email, but given that pip doesn't handle non-Python requirements, some of the most important packages on PyPI are still going to be out of reach for many people, and I don't know what the solution is. (Build a statically linked lxml? Include libxml2.dll in the wheel? Just give a better error message that includes a link to how to download it?)
- Much more community pressure for projects to host their code on
PyPI. Some projects have genuine issues with hosting on PyPI, and there are changes being looked at to support them, but for most projects it seems to just be history and inertia.
 A Linux/OS X user might have more more issues with C extensions.
In general, I think they have far fewer problems. Linux users have one thing to learn (install libxnl2-devel, not just libxml2). Same for Mac users (install Xcode and Homebrew). After that, they tend to have a lot fewer problems. (Except with scipy, since it needs a fortran compiler and sometimes needs you to force-rebuild numpy. And except for Mac users who end up with multiple copies of Python 2.7, which isn't going to be a problem for 3.x to worry about in the foreseeable future.)
Maybe this can't be solved in any meaningful sense, and maybe it's not something the "Python project" should take responsibility for, but without any doubt, it's the single most significant improvement that could be made to my experience with PyPI.
I agree that having some kind of
PS I should also note that even in its current state, PyPI is streets ahead of the 3rd party module story I've experienced for any other language - C/C++, Lua, Powershell, and Java are all far worse. Perl/CPAN may be as good or better, it's so long since I used Perl that I don't really know these days. _______________________________________________ Python-ideas mailing list Pythonemail@example.com https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/