On 19 Sep 2014 04:54, "Georg Brandl" wrote:

On 09/18/2014 08:26 PM, Paul Moore wrote:

...

Every time an issue like this comes up, I know that I'm looking to do "pip install XXX". It's working out what XXX is that's the problem.

So I go and ask Google. A quick check on the progress bar case gets me to a StackOverflow article that offers me a lot of "write it yourself" solutions, and pointers to a couple of libraries. Further down there are a few pointers to python-progressbar, which was mentioned in the StackOverflow article, which in turn leads me to the PyPI page for it. The latest version (2.3-dev) is not hosted on PyPI, so I hit all the fun of --allow-external.

Paul, this could make a good "What problem are we actually trying to fix?" summary on pypa.io. We have spent a lot of time so far on the "getting people the packages they ask for" side of things, but have barely scratched the surface of "helping people find the packages that can help them". At the moment "word of mouth" is one of our main discovery tools, and that's an issue for newcomers that may not have a big network of fellow developers yet. This is a problem I think the Django community actually addressed fairly well through https://www.djangopackages.com/ There are similar comparison sites for Pyramid & Plone, after Audrey & Danny broke out the back end of Django Packages to make it independently deployable (see http://opencomparison.readthedocs.org/en/latest/)

I'd recommend searching PyPI itself. This:

https://pypi.python.org/pypi?%3Aaction=search&term=progress+bar

gives about 20 top results that look highly relevant. A Google search with "site:pypi.python.org progress bar" also looks like it could have given you what you wanted.

...

Ironically, "pip install tqdm" gives me what I want instantly. But it never came up via Google.

It actually occurs to me: I wonder whether anyone at Google might be interested in enhancing it's usefulness as a Python packaging search tool by looking specifically at PyPI's own search index data. (And if they could do that for us, they might be willing to do it for CPAN, RubyGems, npm, CPAN, PEAR, etc) (on the other hand, being a vector for influencing Google search results would mean being a higher priority target for spam, so we may not actually want that)

It also doesn't come up in the PyPI search, because its PyPI page isn't exactly very full of information :)

"SEO for Python Packages" could be a good advanced topic for packaging.python.org :) Cheers, Nick.

cheers, Georg

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 18.09.2014 23:27, Nick Coghlan wrote:

It actually occurs to me: I wonder whether anyone at Google might be interested in enhancing it's usefulness as a Python packaging search tool by looking specifically at PyPI's own search index data. (And if they could do that for us, they might be willing to do it for CPAN, RubyGems, npm, CPAN, PEAR, etc)

(on the other hand, being a vector for influencing Google search results would mean being a higher priority target for spam, so we may not actually want that)

I'm not sure, but isn't Google doing this already? At least package updates uploaded to PyPI usually appear in Google searches quite fast (sometimes instantaneously) so I don't think this is only the result of regular crawls.

On 18 September 2014 23:20, Andrew Barnert wrote:

Hold on. I'm pretty sure that the intended answer to this problem has, for years, been that you go and search PyPI. Is that too broken to use, or are people just not aware of it?

Sorry, I should have mentioned that. I've found that PyPI search is great for getting packages named like you expect, but a lot worse when you have something vague ("website scraping" didn't bring up html5lib, BeautifulSoup or lxml, all of which are tools I've used in the past). So I tend to not bother these days on the assumption that it'll just be another dead end. I know, I'm old and cynical, sorry ;-) Paul

I like stdlib++, but I also want to say it should remain as non-Python-dev endorsement kind of thing. As a user and a library developer, I see pros and cons. Official endorsement can lead to people to abandon whatever they are working or make them feel excluded or unappreciated, which is not a very positive thing to do. On the other hand, there may only be a very limited number of stdlib replacement people can vouch for easily. For HTTP request, it is obvious that at the moment, Requests is the most widely used library in modern Python codebase in the past several years. For asyn-networking and asyn-task, Twisted and Tornado are probably the best. You might celery in the asyn-task just because many people who choose to run asyn task will end up queuing. For cryptography and security, what do you suggest? There are APIs cryptography doesn't have yet but pycrypto or M2Crypto does. It isn't that neither are bad library, I've used pycrypto and I am careful with using the API, but do we endorse cryptography over the other two (the latter one is pretty dead based on its commit activity, I don't know).

On Fri, Sep 19, 2014 at 01:39:12AM -0400, John Wong wrote:

I like stdlib++, but I also want to say it should remain as non-Python-dev endorsement kind of thing. As a user and a library developer, I see pros and cons.

Official endorsement can lead to people to abandon whatever they are working or make them feel excluded or unappreciated, which is not a very positive thing to do.

Yes. Also, it can discourage innovation and encourage monoculture. As has often been said, the standard library is where good packages go to die. In the standard library, a package is ubiquitous, but it's also pretty much in stasis, unlikely to change much. "We biologists have a word for stable: 'dead'." - some biologist So if you want something under active development, likely to gain new features, you normally look outside the std lib, where there is plenty of innovation, experimentation and competition. Now imagine that one package gets recommended as "best of breed" for some particular task in this hypothetical stdlib++. It will combine the ubiquity of the stdlib (because everyone uses it) without the disadvantage of stasis. That could make it much harder for a new package in the same field to become well-known enough to compete for users and developers. Now obviously there are "best of breed" third party libraries. I don't see many people trying to build a better BeautifulSoup. (Maybe they are, and I just don't know about them, which demonstrates the problem.) That's not necessarily a bad thing. But I think we should be careful of putting the thumb on the scales too much. -- Steven

On Sep 19, 2014, at 5:26, Paul Moore wrote:

On 19 September 2014 12:59, Steven D'Aprano wrote:

...

Now obviously there are "best of breed" third party libraries. I don't see many people trying to build a better BeautifulSoup. (Maybe they are, and I just don't know about them, which demonstrates the problem.) That's not necessarily a bad thing. But I think we should be careful of putting the thumb on the scales too much.

Agreed. I hadn't considered the point about discouraging innovation. So it seems to me that the key things needed are:

1. Discoverability - making it easy for package authors to ensure people find their packages. There's a social aspect here as well - quirky project names like celery or freezegun are a part of the Python culture, but they do make things less discoverable (what do those two projects do?). We need a way to balance that.

There's more to it. Many of the most important packages are doing something that didn't have a category until they invented it. For example, what is BeautifulSoup exactly? Originally, IIRC, it was intended to be a more lenient HTML parser. But nowadays it doesn't even do the parsing itself; it drives a stdlib or other third-party parser. It's sort-of a better (than what?) DOM interface for HTML and XML, which is something no one realized they needed until it existed. If the Python community had the kind of less-fanciful, easier-to-find, specific names that, say, perl has, what would it be called? HTML::Parsers::TagSoupParser? Also, the boring names tend to mean that, when there _is_ an obvious category, the first entrant stakes out the best name, and everyone else ends up with just less-felicitous variants of the same name. The fact that SleekXMPP and Wokkel replaced Jabberpy as the most popular XMPP libraries means that it's pretty easy for me to recommend SleekXMPP over Jabberpy. In another language, you have to tell people to use libXMPPClient or XMPPLib instead of libXMPP.

2. A clearer way for projects to declare what they support. Python 2 or 3 or both? Is Windows supported? And it's not just "yes or no", often it's somewhere in the middle - for example I know projects that want to work on Windows, but don't have the expertise, so they'll accept patches but you have to expect to do some of the work for them... Classifiers should do this, but use is patchy (particularly around the supported OS). 3. Most controversially, some way of rating otherwise-equal packages. Ultimately, I don't want a choice, I just want to install something and move on. Download counts may be the easiest compromise here (at least lots of people use the project...)

As well as the technical aspect of making it possible to provide this information (much of it, projects *can* provide right now, although maybe not always as easily as might be ideal) there is a social aspect - making it so that it's the norm for projects to do so, and not doing so is viewed as an indication of overall attention to detail on the project.

Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sep 19, 2014, at 8:25, Paul Moore wrote:

On 19 September 2014 16:15, Andrew Barnert wrote:

...

There's more to it. Many of the most important packages are doing something that didn't have a category until they invented it. For example, what is BeautifulSoup exactly? Originally, IIRC, it was intended to be a more lenient HTML parser. But nowadays it doesn't even do the parsing itself; it drives a stdlib or other third-party parser. It's sort-of a better (than what?) DOM interface for HTML and XML, which is something no one realized they needed until it existed. If the Python community had the kind of less-fanciful, easier-to-find, specific names that, say, perl has, what would it be called? HTML::Parsers::TagSoupParser?

Also, the boring names tend to mean that, when there _is_ an obvious category, the first entrant stakes out the best name, and everyone else ends up with just less-felicitous variants of the same name. The fact that SleekXMPP and Wokkel replaced Jabberpy as the most popular XMPP libraries means that it's pretty easy for me to recommend SleekXMPP over Jabberpy. In another language, you have to tell people to use libXMPPClient or XMPPLib instead of libXMPP.

I wasn't trying to say that "boring" names were better - far from it, quirky names are part of the Python culture. But there isn't anything stopping Beautiful Soup adding keywords like "html parser scraping" or something like that. The trick is picking keywords that people will search for, which is often hard (and that's probably why the keyword metadata is underused). Having a registry of commonly-used keywords, and a user interface that means you don't have to go online and search when you're writing your app, might help. Think of something like Stack Overflow's auto-completed tags.

I don't have any good solutions here, I just think it *should* be possible for the author of Beautiful Soup to make it clear to potential users that it might be what they are looking for when they think of <insert relevant task here>. As somebody said earlier in this thread, think of it as search engine optimisation for projects.

That's a good point. BeautifulSoup, lxml, requests, Scrapy, Mechanize, and pyv8 certainly don't belong in any sane category together, but they also clearly share in common the fact that they're useful for (different but overlapping kinds of) scraping projects, and a keyword-influenced search (as opposed to keyword browsing) seems like the right way to get that across. The problem is that search tends to work best when, even if you don't know what you're looking for, you'll easily recognize it when you see it. But if I'm searching for scraping, is Mechanize or BeautifulSoup better? That question doesn't just not have an answer, it doesn't even make sense. I pretty much have to look at what they both do to figure out which one fills the hole in my scraping project. Which means that "search is the answer" tends to conflict with the earlier goal from this discussion of "just tell me the best answer, don't make me research it". That may just be because that goal is impossible to do well, and making it a lot easier to research by giving you pointers at a few different things that you are likely to want to look into. Or it may mean that "find the most stable implementation of frequently-implemented simple thing X" (like a console-mode progress bar) and "find me good tools to help with complex thing Y" (like scraping web sites) are inherently different problems, and there might be a solution that's _helpful_ for both, but anything that's perfect for one will be useless for the other.

On 19 September 2014 21:59, Steven D'Aprano wrote:

Now obviously there are "best of breed" third party libraries. I don't see many people trying to build a better BeautifulSoup. (Maybe they are, and I just don't know about them, which demonstrates the problem.) That's not necessarily a bad thing. But I think we should be careful of putting the thumb on the scales too much.

The problem is that *failing* to provide recommendations can induce analysis paralysis in newcomers, such that they decide a less mature, more centralised ecosystem with fewer choices is "easier", or even "more powerful". While the "easier" can be accurate (since there are fewer choices to make), the "more powerful" is generally. It's just that the discoverability problem is *much* easier to solve when a language community is dominated by a single use case or a single major vendor, so providing opinionated guidance becomes less controversial. This can lead to comparing one language's entire ecosystem to another (less familiar) language's standard library, rather than comparing the full strength of both ecosystems. As an example of how a default recommendation can help address the discoverability problem, these days, if someone is completely new to Python web development, I will tell them "start with Django, and use djangopackages.com for package recommendations". However, I will also tell them that Django isn't necessarily the best fit for everything, so sometimes something like Flask or Pyramid (et al) might be a better choice. The reason I do it that way is that Django is far more opinionated than most other Python web frameworks and makes a lot more decisions *for* you. Experts have legitimate reasons for debating several of Django's choices, but newcomers don't yet know enough to understand when and why you might want to do something differently. Recommending a framework that makes those choices on their behalf is actually helpful at that point, even if it means they'll be missing out on some other cool libraries (at least for the time being). This approach provides an experience far closer to the modern Ruby web development model where Rails is the default choice (by a long way), and folks only later start considering the use of something less opinionated like Sinatra for problems where an all-inclusive solution like Rails is less appropriate. Providing "pip" by default is the same way - there are actually times when it isn't the best choice for solving particular packaging related problems, but it's always enough to get you started, *and* it lets you more easily bootstrap other tools like conda and zc.buildout when they're a better fit. The goal in providing "default recommendations" is thus to minimise the "time to awesome" for newcomers, while gently nudging them in the direction of good development practices, rather than raising barriers in front of them that say "you must first learn how to make this apparently irrelevant decision before you can continue". And if someone never needs to tackle problems that require moving beyond the default recommendations? That's fine - there are an awful lot of people happily solving problems without moving beyond what the standard library or their redistributor provide. You also see this pattern with some of the design guidance we give newcomers like "don't use metaclasses" and "don't use monkeypatching (except as part of a test mocking library)". What we actually mean is "these are power tools, and genuinely hard to use well. By the time you're ready to wield them, you'll likely also have realised that the implied caveat on the standard advice is 'unless it's the only way to solve a specific problem'". That's a complex mouthful to inflict on someone that is still learning though, so we just oversimplify the situation instead (even though the end result is so oversimplified as to technically be a lie). My experience is also that the scientific community appear to be *far* more pragmatic about library choices than the professional development community (or anyone that is interested in programming for its own sake, rather than what it lets us do). We're a bit more prone to looking under the hood and deciding we don't like a library because of how it's made - competing libraries may arise based on differences in opinion regarding good design and development practices, rather than fundamental limitations in what a library makes possible. Scientists and data analysts are far more likely to just grab a library because of what it lets them *do* (or communicate), without looking too closely at how its put together. That greater level of pragmatism then seems to make it easier for category killers to arise at the library level. However, I have to admit this particular idea is a speculative hypothesis, rather than something that has been subjected to rigorous objective analysis. Anyone looking for a PhD topic in sociology? :) Regards, Nick. P.S. As far as I can tell, the relative ease with which dominance can be asserted is also why VCs tend to favour newer languages with more centralised ecosystems - it's easier for them to assert control, and attempt to "punch the ticket" if the language achieves future success. It's much harder to do that for a massive decentralised sprawl like the Python community. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Paul Moore writes:

On 19 September 2014 13:15, Akira Li <4kir4.1i@gmail.com> wrote:

...

Paul Moore writes:

...

...

3. C extensions aren't a huge problem to me on Windows, although I'm looking forward to the day when everyone distributes wheels (wheel convert is good enough for now though). [1]

I have the opposite impression. http://pythonforengineers.com/stop-struggling-with-python-on-windows/

Well yes, but that article is clearly focused on scientific use (a known "worst case" area) and ends up recommending conda (which is a perfectly fair solution, and as the author states, works well).

My comment was a bit unfair, though. I have a C compiler installed (which, although it's not hard to set up VS Express, isn't normal). And I also treat "find a wininst installer or egg and run wheel convert on it" as trivial and acceptable, which it isn't for people who aren't packaging specialists.

But we're working on this, and I stand by the statement that when projects routinely distribute wheels if they include C extensions, binary dependencies will be a minor issue.

...

...

PS I should also note that even in its current state, PyPI is streets ahead of the 3rd party module story I've experienced for any other language - C/C++, Lua, Powershell, and Java are all far worse. Perl/CPAN may be as good or better, it's so long since I used Perl that I don't really know these days.

Opinions may vary: http://www.reddit.com/r/Python/comments/1ew4l5/im_giving_a_demo_of_python_to...

The discussion here is about packaging, not about finding 3rd party packages to solve a problem. I know of no better way to find a package to parse an ini file in Java than google, which is no better than Python. And what I found was packages on sourceforge and other generic hosting sites.

The main idea of my message [1] is that Google is very good at "finding 3rd party packages to solve a problem." If you know a better solution for any programming language; do tell. [1]: https://mail.python.org/pipermail/python-ideas/2014-September/029417.html The rest is just some references to balance your statements.

Maven may be a central repository - I've never used it myself as the complexity has always scared me off (you could say that about most of Java, though ;-))

...

Or: "The artifact approach is unambiguously better for any production deployment. The source-based approach found in Ruby, Perl, and Python is a problem for me more often than a solution." https://news.ycombinator.com/item?id=7070464

Again, that's deployment rather than discovery.

Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Akira

On Sep 19, 2014, at 11:39 AM, Andrew Barnert wrote:

On Sep 19, 2014, at 5:42, Paul Moore wrote:

...

Maven may be a central repository - I've never used it myself as the complexity has always scared me off (you could say that about most of Java, though ;-))

Don't you just go to the authority factory, ask it to give you an authority, then ask the authority for repository recommender factor, and so on through 8 more steps until you get a general package object that you can cast to the type you need? And the beauty is that, thanks to the magic of static type checking with a rigid but weak type system, if it turns out not to be able to do what you want, instead of a TypeError: BeautifulSoup has no SGML parser that you've never seen before? you get the same convenient NullPointerException from your ISGMLParser as in every other bug you've had so far. :)

But seriously, I think it is worth looking at what other languages' package systems have that we might want to be jealous of as far as discovery, especially those (Cargo, go pkg, Node, gems, CocoaPods, etc.) that were designed after CPAN and the Cheeseshop and the related communities existed. I think Java, C++, and C may be the _least_ relevant places to look (although I could definitely be wrong about that).

For example, some of the newer languages' systems make it easier for you to fork an existing package, and to tie packaging to DVCS repos, with the consequence that if abarnert/spamify falls by the wayside, it might be more likely pmoore/spamify that replaces it than in Python, where it would more likely be a punny PIL->Pillow type of thing. That seems like a nice feature. Do we want that? Can we get from here to there without radical changes?

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

I think at the point we’re well into the territory of what should move to distutils-sig. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA