[Python-ideas] The stdlib++ user experience (Was: Introduce `start=1` argument to `math.factorial`)

[Paul Moore]

On 18 September 2014 18:01, Andrew Barnert
<abarnert at yahoo.com.dmarc.invalid> wrote:
> On Sep 17, 2014, at 23:15, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> However, now that CPython ships with pip by default, we may want to
>> consider providing more explicit pointers to such "If you want more
>> advanced functionality than the standard library provides" libraries.
>
> I love this idea, but there's one big potential problem, and one smaller one.
>
> Many of the most popular and useful packages require C extensions. In itself,
> that doesn't have to be a problem; if you provide wheels for the official 3.4+ Win32,
> Win64, and Mac64 CPython builds, it can still be as simple as `pip install spam`
> for most users, including the ones with the least ability to figure it out for themselves.

OK, the key thing to look at here is the user experience for someone
who has Python installed, and has a job to do, but needs to branch out
into external packages because the stdlib doesn't provide enough
functionality.

To make this example concrete, I'll focus on a specific use case,
which I believe is relatively common, although I can't back this up
with hard data.

Assume:

* A user who is comfortable with Python, or with scripting languages in general
* No licensing or connectivity issues to worry about
* An existing manual process that the user wants to automate

In my line of work, this constitutes the vast bulk of Python use -
informal, simple automation scripts.

So I'm writing this script, and I discover I need to do something that
the stdlib doesn't cover, but I feel like it should be available "out
there", and it's sufficiently fiddly that I'd prefer not to write it
myself. Examples I've come across in the past:

* A console progress bar
* Scraping some data off a web page
* Writing data into an Excel spreadsheet with formatting
* Querying an Oracle database

Every time an issue like this comes up, I know that I'm looking to do
"pip install XXX". It's working out what XXX is that's the problem.

So I go and ask Google. A quick check on the progress bar case gets me
to a StackOverflow article that offers me a lot of "write it yourself"
solutions, and pointers to a couple of libraries. Further down there
are a few pointers to python-progressbar, which was mentioned in the
StackOverflow article, which in turn leads me to the PyPI page for it.
The latest version (2.3-dev) is not hosted on PyPI, so I hit all the
fun of --allow-external.

Ironically, "pip install tqdm" gives me what I want instantly. But it
never came up via Google.

The rest of the cases are similar, lots of Google searching, often
combined with evaluating multiple options, followed by more or less
pain installing the software. Things that aren't Python 3 or Windows
compatible suck me into the "shall I patch it and submit a PR"
minefield.

For the last case (an Oracle driver), where I need a C extension and
access to external libraries, ironically it's pretty easy. There's no
real competition to cx_Oracle, and the PyPI page has what I need,
although they ship wininst exes rather than wheels, which means I need
to do a download then a wheel convert then a pip install, so it's not
ideal, but doable.

>From this example, I'd like to see the following improvements to the process:

1. Somewhere I can go to find useful modules, that's better than Google.
2. Someone else choosing the "best option" - I don't want to evaluate
3 different progressbar modules, I just want to write "57% complete"
and a few dots!
3. C extensions aren't a huge problem to me on Windows, although I'm
looking forward to the day when everyone distributes wheels (wheel
convert is good enough for now though). [1]
4. Much more community pressure for projects to host their code on
PyPI. Some projects have genuine issues with hosting on PyPI, and
there are changes being looked at to support them, but for most
projects it seems to just be history and inertia.

[1] A Linux/OS X user might have more more issues with C extensions.

Maybe this can't be solved in any meaningful sense, and maybe it's not
something the "Python project" should take responsibility for, but
without any doubt, it's the single most significant improvement that
could be made to my experience with PyPI.

Paul.

PS I should also note that even in its current state, PyPI is streets
ahead of the 3rd party module story I've experienced for any other
language - C/C++, Lua, Powershell, and Java are all far worse.
Perl/CPAN may be as good or better, it's so long since I used Perl
that I don't really know these days.