On Dec 29, 2014, at 18:53, Antoine Pitrou wrote:

On Mon, 29 Dec 2014 18:54:46 +0200 Ram Rachum wrote:

...

Hi guys,

I want to have a method of `pathlib.Path` that would send a file to the recycle bin. (i.e. soft delete.)

What do you think about adding this?

I see there's a PyPI package `Send2Trash` that does this, but it would be nice if this was in the standard library.

I think the corresponding low-level function to the os module would be a good start (the hard part is probably to write the cross-platform code, anyway).

There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there. On OS X, unless you want to use higher-level CoreFoundation APIs that require a runloop, you just look up the name of the trash directory (which I believe is theoretically localized but actually always ~/.Trash on the drive with your user directory, $mnt/.Trash otherwise) and call rename. On Linux, and some other *nix platforms, you assume XDG directory layout and call rename. On other platforms, I don't know if there even is a way to do it. And I don't know what the right thing to do on OS X would be for a file on a different filesystem than the user's home that doesn't have its own .Trash or is mounted read-only or no-trash. Finder gives you a popup in each of those cases asking if you want to delete it immediately because it can't be trashed. And similarly for Linux, but the details are different (especially since you might not even have a home trash). I'd guess Send2Trash is a good place to at least start, but if this is going in the stdlib someone needs to make sure it meets Apple HIG rules and XDG standards.

Regards

Antoine.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Andrew Barnert wrote:

you just look up the name of the trash directory (which I believe is theoretically localized but actually always ~/.Trash on the drive with your user directory, $mnt/.Trash otherwise) and call rename.

You may also need to invent a unique name if there's already a file in the trash with the same name.

And I don't know what the right thing to do on OS X would be for a file on a different filesystem than the user's home that doesn't have its own .Trash or is mounted read-only or no-trash.

Raising an exception would seem to be a reasonable thing to do in that case. -- Greg

On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote:

I believe all of those issues could just raise the normal exception, because Finder treats them as normal errors--that is, rather than giving you the special "can't trash; delete?" message, it tries, fails, asks you to authenticate as an administrator, and tries again.

That doesn't make sense - the problem is with the state of the trash directory (it may be in a state that you can access, but the "wrong" one, e.g. one that would allow other people to delete your files - I don't know if this is a problem for OSX but it is for XDG), not your access to the file being deleted.

On 30 Dec 2014, at 21:33, random832@fastmail.us wrote:

On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote:

...

I believe all of those issues could just raise the normal exception, because Finder treats them as normal errors--that is, rather than giving you the special "can't trash; delete?" message, it tries, fails, asks you to authenticate as an administrator, and tries again.

That doesn't make sense - the problem is with the state of the trash directory (it may be in a state that you can access, but the "wrong" one, e.g. one that would allow other people to delete your files - I don't know if this is a problem for OSX but it is for XDG), not your access to the file being deleted.

On OSX there are at least two reasons why the Finder can prompt you when you try to move a file to the trash. The first is a source location that doesn’t support the trash (such as a fileserver), the Finder will ask if should just permanently remove the file. The second is when you don’t have permissions to remove the file, the Finder will then ask if it should retry with Administrator privileges (and will prompt you for a username and password if you confirm). Ronald

On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote:

By "suitable" I mean "suitable for widespread-enough use that it should be in the stdlib". As I said earlier in the thread, I think it's perfectly reasonable to have a third-party library that uses a deprecated API, requires 10.8, requires a runloop, directly or indirectly asks the Finder to do the work for you, or doesn't quite meet Apple's HIG rules, because all of those could be acceptable in _many_ applications; I just don't think any of those should be in the stdlib, because none of those are acceptable in _most_ applications.

What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is? http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-... Having to call out to an external program via Applescript feels rather icky, but given the difficulty of simultaneously supporting multiple OS X versions, satisfying the rules of Apple Store, and using Apple's APIs, I don't see any alternative. -- Steve

On Dec 31, 2014, at 0:47, Steven D'Aprano wrote:

On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote:

...

By "suitable" I mean "suitable for widespread-enough use that it should be in the stdlib". As I said earlier in the thread, I think it's perfectly reasonable to have a third-party library that uses a deprecated API, requires 10.8, requires a runloop, directly or indirectly asks the Finder to do the work for you, or doesn't quite meet Apple's HIG rules, because all of those could be acceptable in _many_ applications; I just don't think any of those should be in the stdlib, because none of those are acceptable in _most_ applications.

What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is?

I already answered this briefly earlier in the thread, but in more detail: 1. If you don't have a GUI login session (e.g., because you've ssh'd into the box, or are running as a LaunchDaemon or a traditional detached script), you have no Finder in your session to script. 2. If you're running in a sandbox that doesn't have scripting rights for Finder added in--as will be the case in most Mac App Store apps--you can't script Finder. Of course you can include a sandbox exception to allow for this, but the only way to do this that's allowed in the App Store requires 10.8.0 to run and, IIRC, 10.8.2 to run without serious bugs. And the Python docs would have to explain that any app that may call shutil.trash must use a custom sandbox. 3. If you're running in a sandbox, you can't execute osascript; if you need pre-10.8 compat, you can't use the modern APIs; this means the only thing you can do is include a compiled AppleScript as a resource and use the 10.7 APIs to execute it--which still requires 10.7, and it seems to have problems with the new 10.9.5/10.10.1 code signing rules (although it may be easy to work around that; I haven't tried). 4. If you're using a third-party Finder replacement like PathFinder, it won't respond to messages sent to Finder. 5. If you've killed the Finder to save resources (pretty rare nowadays, but still legal) you'll relaunch it. 6. The Finder may pop up a progress bar--or, worse, an auth dialog, or a "delete immediately?" dialog, on behalf of your app. This could be surprising from a console program. 7. Depending on how you do the scripting, your request may time out after 5 or 30 seconds, leaving you no way to know whether the trashing succeeded. 8. If Finder is hidden, it may unhide. If another app is fullscreen in front of Finder, it may slide offscreen, or Finder's dialogs may just be invisible. (You can guarantee a fix for the latter by forcing the former.) 9. If your session or system is screwed up and you're trying to shut down cleanly or repair things, Finder could be unresponsive or even hung. For example, on 10.6, if you have for separate moves blocked on an NFS mount, a trash will block forever, but (assuming you aren't trying to trash something on the same mount) there's nothing stopping you from trashing on your own. In some cases, this may all be fine. For example, if you're writing a trash script that you intend to use on your specific computer in local Terminal sessions only, you may _want_ the Finder to take care of auth, no-trash mounts, and other issues for you, and show you progress on long tasks. Fortunately, you can already write that script with about 5 lines of ScriptingBridge code that will run on the Python 2.7 with PyObjC and other extras that Apple has pre-installed as /usr/bin/python on all 10.6-10.10 versions, so you're already covered.

http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-...

Even if you want to use osascript, you don't want to do it that way. Build a single AppleScript command that trashes all of your files in a single call. One of the other blog posts someone linked earlier in this thread shows how to do that. Also, for the usual reasons, os.system is not the best way to call osascript (you'll end up with a success retcode even on failure, quoting for both the shell and AppleScript is double the fun, etc.).

Having to call out to an external program via Applescript feels rather icky, but given the difficulty of simultaneously supporting multiple OS X versions, satisfying the rules of Apple Store, and using Apple's APIs, I don't see any alternative.

Since this doesn't satisfy the rules of the App Store, it doesn't solve the problem it's intended to. But see my previous email. For almost any app, there is a much better alternative. The problem is that there's no _single_ alternative that's good for most apps; there are different ones for different use cases. In particular, for just writing a little trash script for use on the command line on your own 10.8+ computer, the NSFileManager version is even simpler than the scripting version (again using Apple's pre-installed 2.7 with PyObjC), and without any of its problems. For older computers, it's a _bit_ more work with the FS function, but still very simple, and it's a known problem that multiple people have solved years ago (and linked to earlier in this thread), so Python 3.5 doesn't need to retroactively solve it.

On Wed, Dec 31, 2014 at 11:50:04AM +0100, Andrew Barnert wrote:

On Dec 31, 2014, at 0:47, Steven D'Aprano wrote:

...

What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is?

I already answered this briefly earlier in the thread, but in more detail: [...]

Thank you for the detailed and extensive answer! -- Steven

On Thu, Jan 1, 2015 at 9:37 PM, Ram Rachum wrote:

As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it. (And of course, Python has lots of OS-specific modules.)

Most OS-specific stuff is in the os module, but path handling is inherently platform-specific to some extent, so either place would be reasonable.

So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this:

if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file)

What do you think?

Or: try: delete = os.recycle except AttributeError: delete = os.remove delete(my_file) ChrisA

I take your point about it not being in the `os` module, I have no issue with it being in shutil or pathlib. I also agree about the name, "recycle" doesn't really make sense. Regarding your question: "But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?" Well, I can use a third-party solution for everything, but it would be nicer for me if it was in the standard library because then I could count on it always being there and the API not changing. I could say the same thing you said about the `webbrowser` module, that opening a new browser tab is quite a specialist need, but I do enjoy having it available in the standard library and I feel the same about sending a file to trash, which I think is a basic feature that's we've all been taking for granted for the last decade or two. Happy new year, Ram. On Thu, Jan 1, 2015 at 2:35 PM, Steven D'Aprano wrote:

On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote:

...

So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this:

The os module is for low-level operating-system functions. "Send to trash" is neither low-level nor part of the OS per se, it is part of the desktop environment.

I'm not convinced that this needs to be in the standard library, but if it is, I think that the os module is completely the wrong place for it. I think, in order of preference:

1) shutil 2) pathlib

is the right place.

(The actual implementation for the move_to_trash function could come from another, platform-specific, module.)

...

if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file)

"Recycle" is not a good name for the function, because it doesn't recycle the file. It does the opposite of recycle: it prevents the file from being deleted and over-written. I think an explicit name like move_to_trash is better than something misleading like "recycle".

No need for a special flag to check if the function exists, just check for the function:

try: f = shutil.send_to_trash except AttributeError: f = os.remove f(my_file)

But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Jan 1, 2015, at 11:37, Ram Rachum wrote:

Hi everyone,

I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.)

On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional. Really, on all three platforms, the answer is the same: you call an API from your desktop manager. The only difference is that Windows merges desktop and kernel together to the point where they can't be separated--so Python is already using Win32, unlike Cocoa or GObject.

As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it.

No one is "depriving" Windows users of anything. Python comes with pip, send2trash installs without needing a compiler (at least I hope so; if not, someone can write a new third-party module that does). Not everything needs to be in the stdlib. So far, there aren't many third-party modules mentioned in the stdlib docs, but that's only because we didn't have preinstalled pip, wheels, etc. until 3.4.

(And of course, Python has lots of OS-specific modules.)

So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this:

if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file)

What do you think?

There's no such flag for anything else platform-contingent. EAFP.

Thanks, Ram.

On Thu, Jan 1, 2015 at 1:14 AM, Steven D'Aprano wrote:

...

On Wed, Dec 31, 2014 at 11:50:04AM +0100, Andrew Barnert wrote:

...

On Dec 31, 2014, at 0:47, Steven D'Aprano wrote:

...

...

What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is?

I already answered this briefly earlier in the thread, but in more detail: [...]

Thank you for the detailed and extensive answer!

-- Steven _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Jan 01, 2015 at 04:37:52PM +0100, Andrew Barnert wrote:

On Jan 1, 2015, at 11:37, Ram Rachum wrote:

...

Hi everyone,

I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.)

On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional.

Actually it seems that the trash location is standardized by the XDG Desktop Specification: https://github.com/kevva/xdg-trash http://www.ramendik.ru/docs/trashspec.html

On Thu, Jan 01, 2015 at 02:43:09PM +0200, Ram Rachum wrote:

Regarding your question: "But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?"

Well, I can use a third-party solution for everything, but it would be nicer for me if it was in the standard library because then I could count on it always being there and the API not changing.

Which is exactly the point. It has been said that the standard library is where good modules go to die. Do you think that the "move to trash" API is ready to die? I don't think so. I think that there is a lot more than just moving files to the trash. I've spent a little bit of time over the last day or so thinking about what sort of trash-related functionality I would like, as a Linux user: - move to trash, of course, but which trash? - return the name of the trash directory in my home directory - list trash directories on other partitions and disks - list trash directories for all users (if I have read permission) - empty one or more of those trash directories - optionally only deleting sufficiently old and/or large files - validate and/or repair the trash metadata - report how much space is used by the trash, with or without metadata - support legacy .Trash directories as well as .local/share/Trash For some operations, I would want to specify either a specific trash directory, or all trash directories for a given user (defaulting to the current user), or all trash directories for all users. (The later would only work if I was running as the root/Administrator user, of course.) The point being that the API for all this functionality isn't obvious, and it is very unlikely to be so stable that it is ready to move into the standard library. -- Steven

On Thu, Jan 1, 2015, at 07:35, Steven D'Aprano wrote:

The os module is for low-level operating-system functions. "Send to trash" is neither low-level nor part of the OS per se, it is part of the desktop environment.

What is the correct place for high-level functions that need to be implemented in a different way on each OS? I think the problem is that some people view os as "low-level functions, like the ones found in unistd.h" and other people view it as "where to put stuff that needs a separate implementation per platform". The actual description in the documentation is "This module provides a portable way of using operating system dependent functionality." Nothing about low-level.

On Thu, Jan 1, 2015, at 17:59, Andrew Barnert wrote:

Of course you _could_ implement that full spec yourself in Python, right? Well, read it. There are references to things "the implementation" may do, like providing a way for the administrator to disable sticky-bit checking for filesystems that don't support it. Do you know how to check whether the administrator has disabled sticky-bit checking on a filesystem?

A python implementation would be an independent implementation, and not bound by another implementation's (even on the same machine) implementation-specific decisions. There is therefore no reason the python implementation couldn't simply not provide such a method, and therefore any .Trash directory without the sticky bit set does not pass the checks, and all that will happen is .Trash-$uid gets used instead - no harm done.

On Jan 2, 2015, at 14:36, random832@fastmail.us wrote:

On Thu, Jan 1, 2015, at 10:37, Andrew Barnert wrote:

...

On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional.

There is an actual standard for the Trash system that all three major desktop environments implement, so your "complicated approximation based on rename" can actually be correct rather than an approximation. If that's not exactly like how GNOME does it, then GNOME is incorrect.

First, XDG specifications are explicitly not standards (and XDG Trash is a draft extension proposal to those specs) so there is not an actual standard. But, far more importantly, there are large areas that the spec leaves open to the implementation, or open for future expansion, that GNOME (and KDE) fills in with appropriate behavior. I gave one example earlier in the thread, which I picked by pointing my finger at one random paragraph from the spec. If you read through it, there are plenty of others. If you call the APIs via GObject or Qt on a GNOME or KDE system, you will get the exact same behavior as everything else on the system. If you implement it yourself and ignore what GNOME and KDE do, nobody's going to care that you're 100% compliant with a draft spec proposal, they're going to care that your app refuses to trash something that everything else can trash (or, worse, that your app trashes it by copying 8GB over the network).

On 3 Jan 2015 06:06, "Andrew Barnert" wrote:

Except that if you can't create a .Trash-$uid, then you can't trash files

on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.) It sounds to me that what is needed before changes to pathlib can be discussed is a reference third party module that appropriately respects desktop conventions (whether Windows, Mac OS X, GNOME, KDE, or other XDG based Linux window manager), and associated clear documentation on how to delete files in a way that integrates well with underlying platforms. The standard library isn't the place to work through that standardisation activity, although a PEP is certainly a reasonable place to track it. Guido's work on PEP 3156 and the asnycio module is a good point of reference here, as is Antoine's work on pathlib itself. Once a pathlib independent reference implementation of imperative desktop trash management exists, and there is broad acknowledgement that it is a suitable candidate for inclusion in the standard library (perhaps as part of shutil, perhaps as a new independent module, depending on API complexity), *then* it becomes feasible to discuss adding trash management support to the pathlib abstraction layer. Regards, Nick.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano wrote:

Otherwise, consider that the great bulk of scripts that are written in Python don't do any file management at all, other than perhaps reading or writing a few files. So even if Python gained this functionality, it wouldn't have any impact on most scripts and applications.[1]

I've always thought of a trash can as an inherently human-controlled feature. When you go in using a GUI file manager and delete a file, it lands in trash. When you use another application, it's always felt surprising - for instance, BitTorrent on Windows deletes things by moving them to trash. (Which has abysmal performance if you select a huge bunch of things and wipe them all out at once; trashing gobs of stuff across thousands of files seems to trigger an O(N*N) repeated search for which files "fall out" of trash - not to mention that this probably means a fairly arbitrary subset of the freshly-deleted is in trash, and little or nothing from previous deletions remains.) Most apps should do file management at the file system level, unless they're specifically replicating/replacing the file manager. So a "move to trash" feature is something that (a) isn't going to be needed by most apps, and especially (b) would be an attractive nuisance. A third-party module has that extra little hump of having to be consciously sought, so it can be there for those who want it but not waving itself around at those who really should be looking at os.remove(). I'm -0.5 on adding a "move to trash" function to the stdlib, ever, and a definite -1 on adding one now. ChrisA

On Sat, Jan 3, 2015 at 7:07 PM, Mark Lawrence wrote:

On 03/01/2015 02:18, Chris Angelico wrote:

...

On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano wrote:

...

Otherwise, consider that the great bulk of scripts that are written in Python don't do any file management at all, other than perhaps reading or writing a few files. So even if Python gained this functionality, it wouldn't have any impact on most scripts and applications.[1]

I've always thought of a trash can as an inherently human-controlled feature.

As, for example, holding down the SHIFT key on Windows and skipping the recycle bin completely?

That's a human deciding _not_ to use it; but yes, it should be under human control. ChrisA

On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert wrote:

The Mac version also does the move-to-trash; it has perfectly acceptable performance.

Good. The note about performance was a mere parenthesis, and the last time I did anything like that on Windows was on XP, so it's quite possibly different on Win7/8.

I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib.

Maybe. Is there a mention of other third-party packages in stdlib docs? It's a slippery slope; which ones deserve mention? Maybe just something like "Note that moving objects to the trash can is possible in OS-dependent ways; packages are available on PyPI to make this easier", without naming any.

Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem.

As an explicit "trash" tool, sure. Typing "trash some_file" is under human control just as much as hitting Delete without holding Shift is. I just don't think it should be the obvious way for an application to remove files. ChrisA

On 3 January 2015 at 21:04, Andrew Barnert wrote:

On Jan 3, 2015, at 11:39, Chris Angelico wrote:

...

On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert wrote:

...

I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib.

Maybe. Is there a mention of other third-party packages in stdlib docs? It's a slippery slope; which ones deserve mention? Maybe just something like "Note that moving objects to the trash can is possible in OS-dependent ways; packages are available on PyPI to make this easier", without naming any.

The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths.

Your parenthetical qualification is correct - while I personally think providing authoritative recommendations of third party modules is a good idea, this is far from a universally held opinion amongst the core development team. My own caveat on the idea is that any such explicit recommendations should only cover projects that potentially *could* be included in the standard library from an API design perspective, but we don't want to include them (yet, or perhaps ever) for project governance reasons, such as their update cycles for feature releases needing to be faster than those of the standard library. The "pro" case for this approach is primarily about lowering barriers to entry: for folks just learning Python, "not in the standard library" is currently functionally equivalent in many cases to "Python can't do this". New users often won't yet have the skills needed to do their own research and evaluate competing solutions to their problem, and teaching someone how to find a module on the internet, evaluate the trustworthiness of the people publishing it, get it onto their computer, and update it appropriately over time is genuinely difficult (even with pip provided by default - that helps with the last two steps, but the first two are still genuinely difficult, in a way experienced open source developers often take for granted). Providing authoritative recommendations of third party projects in the standard library documentation helps resolve that situation without needing to resort to making the standard library larger (with the additional long term maintenance complications the latter approach entails). You see a similar dynamic operating in a slightly different way with a project like Django - when the Django developers are significantly more experienced in web development than you are, their opinionated, all-inclusive approach to web framework design is a *feature*, not a defect. It's only when you become more experienced with web service development yourself, and need to deal with problems that don't fit neatly into the SQL-backed three-tier web application model, or need to integrate with other systems which weren't built with Django in mind, that less opinionated frameworks like Flask or Pyramid become interesting - those frameworks leave many more decisions to the developers using them, which is exactly what you want in many situations as an experienced developer, but constantly asking people to make decisions they aren't yet qualified to make poses a huge barrier to entry. It's a potential problem when less experienced folks are put off by the request to make those decisions (since not understanding the questions you're being asked by your tools makes for a terrible user experience), and decide to go learn something else that places fewer demands on them, but it's even worse for everyone trying to deploy and manage the resulting software systems when folks happily make the decisions these advanced DIY integration frameworks ask of them without even realising they're not yet qualified to make them. djangopackages.com is also incredibly useful as a tool for finding peer evaluations of projects in the Django ecosystem. I'm not yet aware of any successful attempts to expand that model to other plugin ecosystems, but the core grid concept of assembling a set of criteria, providing answers across a set of related projects, and publishing the results to provide a starting point for anyone doing their own research in that topic area is a wonderful one. The "con" case is that providing default recommendations can be controversial - when there are multiple competing projects in an area, being the default recommendation of the CPython core development team can be a huge popularity boost, since reputation is to some degree transitive. If someone trusts CPython, and we trust another project enough to recommend it, then a lot of folks will use that recommendation as a substitute for doing their own research (even when they *can* do that research themselves, the default recommendation is likely to be good enough, letting them defer investigation of alternatives until after they've tried the default option). I suspect that objection could potentially be dealt with the same way we deal with any other standard library design decisions: if a module maintainer is comfortable making a recommendation based on their own expertise they can, but broader recommendations spanning several modules would require at least a discussion on python-dev, and potentially even a PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 4 January 2015 at 20:32, Andrew Barnert wrote:

The question here, then, is how this applies to desktop/shell functionality like moving to trash, administering trash in all the ways someone (Steven?) wanted, other features that someone else raised (aliases/bookmarks/shortcuts/desktop files), and maybe additional stuff (accessing the desktop's extension/file-type mapping). Some of that stuff is already in the stdlib (or might be confusing to novices--e.g., IIRC, on XDG desktops, some things that are done with aliases on Mac are done with desktop files, but in some cases you just use symlinks, so a Linux novice might not even realize he's not writing cross-platform code...).

For a GUI app, it's not at all clear that choosing Qt or Wx or one of the platform-specific libraries automatically solves all of these problems for you, while choosing Tk doesn't. For a command-line app, it's even worse. Most Windows users probably have no idea they're asking for desktop integration rather than OS support when they ask how to move a file to the trash or create a shortcut, because Windows intentionally buries that distinction. So when they see that neither os nor shutil nor anything else in the stdlib can help them, they're not going to think, "Oh, I need to go find a platform-specific or cross-platform desktop integration library." (And they're also not going to think "Well, Perl and Ruby and Node don't have these either, so why should I expect Python to?") So, I think in this case it would be useful to recommend something. The question is _what_ to recommend (and where).

I think for platform-specific code there are pretty clear winners (PyWin32, PyObjC, GObject, and PyQt/PySide, for the big four). Some might argue for including the XDG package (e.g., if you want to do the same thing as one of the XDG-utils tools, you don't need Gtk or Qt for that), and Mac users might also want a link to the ScriptingBridge guide or something (because some things that are done by API on Windows are done by scripting System Events or Finder or something else), and people using a third-tier desktop on Linux may not have Gtk or Qt libraries to bind to, etc., but I think those are minor questions compared to whether it would be useful to have pointers to these packages for novices. (I think it goes without saying that none of these packages should be added to the stdlib, right?)

Right, a useful first step might be a "Tools for writing rich client applications in Python" overview page. That's out of scope for any of the current *.python.org documentation set (it doesn't fit under docs.python.org *or* packaging.python.org, nor anywhere on the main site), but if it's written in an appropriate tone (i.e. making it clear that these are default starting points, and recommendations for further evaluation, rather than universally applicable definitive conclusions), then a future "ecosystem.python.org" guide could start the same way p.p.o did, as a separate Sphinx project hosted on ReadTheDocs that was later blessed with a python.org subdomain. https://packaging.python.org/en/latest/deployment.html is an example skeleton of such a page for package deployment tools. We haven't filled that one *in* yet, but hopefully it conveys the general idea. https://packaging.python.org/en/latest/extensions.html is another one reviewing binary extensions, and some of the tools available in that space. For those curious as to "Why not the wiki?", a Sphinx project hosted on a pull request capable service like GitHub, BitBucket or Kallithea offers a much nicer workflow for reviewing of proposed changes, together with an integrated issue tracker for submitting proposals for updates (https://github.com/pypa/python-packaging-user-guide/ is the project behind packaging.python.org, for example).

But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither.

Just describing the problem space can be incredibly valuable for newcomers - one of the hardest things to learn as a beginner is the difference between "this is hard because I don't know what I'm doing" and "this is hard because I'm working on an unsolved problem" (with "this is deceptively easy because I'm setting myself up for problems later on" being an interesting variant that's hard to detect no matter how experienced you are). I see this is a *lot* when working with the scientific community - they actually have much higher expectations of cross-platform tooling than professional developers do. From the outside, the political, personal and commercial meta-issues that combine to make cross-platform standards development an unholy nightmare aren't immediately obvious, and nor are the trade-offs between "lowest common denominator" (or Java-style "reinvent all the wheels") functionality that works everywhere, and providing access to advanced platform specific functionality.

And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong.

While I think it's better to leave the question open rather than try to pre-empt an answer, I do believe we can at least say *now* isn't the right time to be adding "move to trash" functionality to the standard library - the functionality isn't mature enough within the broader ecosystem to determine an appropriate answer. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5 Jan 2015 05:25, wrote:

On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:

...

You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support.

I don't see why you need to call an API to the desktop enviroment. The entire point of the spec is to provide compatibility _between_ implementations on the same filesystem - you can quit gnome and log into KDE and see the same trash.

That compatibility is implemented *by* the desktop environments. Yes, you could write your own, but why would you, when the whole point of providing a standard API would be to integrate more cleanly with the underlying OS.

Python would just be a separate implementation that stands by itself.

The word "just" does not belong in that sentence. Reliable cross-platform development & testing for operations that are tightly coupled to operating system services is hard, period - the associated test matrix of platforms and permission levels ensures that. Regards, Nick.

On Jan 4, 2015, at 20:24, random832@fastmail.us wrote:

On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:

...

Getting OS X right is very complicated, as Andrew has so ably explained.

In Linux and other Unixes, the situation is like OS X, only more so since there's no one single authority.

However, what authority there is does not allegedly demand that you present dialog boxes and admin elevation prompts to the user (this has been asserted, but no citation has been given).

Where has that, or anything remotely similar, been alleged or asserted? What _has_ been asserted is that an app has to fit in with the choices made by the platform implementation. And the only example given was given with a direct quote from the spec linked in the preceding email. I'm not sure how much simpler a citation you need in order to be able to follow it. The example was about platform-specific ways to allow admins to disable sticky-bit checking on network mounts. There are more such things that are left incompletely specified, or explicitly left unspecified for future development. I don't think any of them have anything to do with dialog boxes or other user interaction, and I don't recall anyone saying otherwise. (I suppose if you didn't read carefully, you might take something like "SHOULD warn the administrator" as implying a prompt saying "Go get your sys admin and drag him over to the screen" rather than, say, a syslog warning with a string the admin can see later and look up in help or google).

...

You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support.

I don't see why you need to call an API to the desktop enviroment. The entire point of the spec is to provide compatibility _between_ implementations on the same filesystem - you can quit gnome and log into KDE and see the same trash. Python would just be a separate implementation that stands by itself.

The last sentence is exactly the same thing you already got two detailed answers to that you haven't responded to in any way. So I suspect the reason you don't understand why calling an API is a good idea is that you skimmed over the explanations that would explain exactly that. Go back and read those emails.

On Jan 4, 2015, at 21:35, Antoine Pitrou wrote:

On Mon, 5 Jan 2015 06:25:01 +1000 Nick Coghlan wrote:

...

On 5 Jan 2015 05:25, wrote:

...

On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:

...

...

You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support.

I don't see why you need to call an API to the desktop enviroment. The entire point of the spec is to provide compatibility _between_ implementations on the same filesystem - you can quit gnome and log into KDE and see the same trash.

That compatibility is implemented *by* the desktop environments. Yes, you could write your own, but why would you, when the whole point of providing a standard API would be to integrate more cleanly with the underlying OS.

For the same reason that we ship a HTTP server while we could integrate more cleanly with Apache? :-)

That's an interesting point--but note that Python _also_ has WSGI and various other ways to integrate more cleanly with Apache (and various other web servers). If you just want to put up a simple web service to handle a few users (e.g., as a user interface to your application), SimpleHTTPServer is sometimes the answer. But if you want to add a web service to a major Apache deployment, you generally don't want to do that by deploying SimpleHTTPServer alongside it. It's not _impossible_ to share ports, SSL certs, configs, logs, deployment scripts, monitoring tools, etc. between the two, but it's not the way you'd want to do things. (There's also the fact that SimpleHTTPServer won't be able to handle the same load as Apache…)

When there's a well-defined spec,

I suppose it depends on your definition of "well-defined spec". As Nick said, XFCE only partly supports the spec, and he's not sure how completely GNOME does. As I pointed out, there are gaps in the spec for both implementation-defined and future-expansion behavior. That isn't the same as the case with HTTP/1.1. Look at http://www.freedesktop.org/wiki/Software. Notice that xdg-utils does not yet have a trash command, and there is no separate trash program or library. (As far as I know, there are at least two XDG trash programs on github which have not yet been submitted to be part of the XDG software collection, which don't appear to be 100% compatible with each other. (For example, one of them requires top trash directories to have exactly mode 41777, the other just requires them to be executable for the current user and have the sticky bit on.) Does creating a third one--and putting it in the Python stdlib forever--really sound like a good idea? (As a side note, one of the two above-mentioned trash programs is written in Node, and I don't see anyone suggesting that the functionality be moved into the Node stdlib.)

On Sun, Jan 4, 2015, at 16:02, Ethan Furman wrote:

If by "stands by itself" you mean "desktops can't interact with it", how would that be useful?

They interact with it _through the filesystem_, because it does the same things with the filesystem as all the desktop environments do and as the spec says. Just like Gnome doesn't have to interact directly with KDE - programs from each environment don't have to call the other's library APIs - for both Nautilus and Dolphin to display the same trash contents and to do the same thing when you trash a file.

On Mon, Jan 5, 2015 at 8:02 AM, Ethan Furman wrote:

On 01/04/2015 11:24 AM, random832@fastmail.us wrote:

...

If being cross-platform isn't easy, it won't happen. You see it now with the lack of any support for "call glob on arguments on windows and not on unix" [because the shell handles it on unix] whether directly, in argparse, or in fileinput

Could you elaborate on this point?

On Unix, a command like "python some_script.py *.txt" is parsed by the shell[1] into a bunch of extra arguments, eg "python some_script.py file_one.txt file_two.txt file_three.txt". On Windows[2], this expansion isn't done, so the script gets a literal "*.txt" as an argument. So a Unix program is usually written to accept multiple args, but a Windows program usually expects to do its own globbing. Now, what happens when you want to reference a file whose name actually includes U+002A (or \x2A, given that file names on most Linux systems consist of bytes)? On Unix, you put a backslash in front of the name, to prevent the shell from expanding it ("\*.txt"), or use some other means of calling up the program (subprocess.call(["python","some_script.py","*.txt"])). Your expectation then is that the script will NOT expand globs; but the script will see the exact same thing in sys.argv as it does on Windows, where the expectation is that it WILL expand globs (and, if I'm not mistaken, asterisks are simply disallowed in file names - though that might be a filesystem-specific rule). So somehow, your script has to know whether it's supposed to call glob or not. Or, more likely, it'll do whatever is appropriate for the platform its developer uses, and then misbehaves on the other one. ChrisA [1] Assuming you're using one of the standard shells, and assuming you haven't configured this feature away, etc, etc. Normal user expectations. [2] Again, assuming you're using the default shell; but again, normal user expectation, ergo replacement shells often replicate this.

On Mon, Jan 5, 2015 at 11:57 AM, Steven D'Aprano wrote:

On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832@fastmail.us wrote: [...]

...

Right now, none of this is done. If you pass *.txt on the command line to a python script, it will attempt to open a file called "*.txt".

On Windows, you will get Python's definition of globbing. On POSIX systems, the globbing module will do nothing, because the shell will most likely have already expanded the wild-cards

You're assuming that there are no wildcard characters actually included in the file names, which on Unix systems is perfectly possible. How can you reliably manipulate a file called *.txt when there are foo.txt and bar.txt in the same directory? With most Unix programs, you escape the asterisk to get it past the shell, the program does no globbing of its own, and you're safe. If your Python script unconditionally globs its file names, you're stuck.

...

Another separate but related issue is the fact that windows wildcards do not behave in the same way as python glob patterns.

I don't understand why you say this. As I understand it, there is no such thing as "Windows wildcards" as every application which wants to support wildcards has to implement their own. If you want to know what kinds of globbing wildcards the application supports, you have to read the application documentation. (Or guess.)

I am not a Windows expert, so I may have been misinformed. Anyone care to comment?

There's a very standard interpretation, which I think is codified in the FindFirstFile API, although I'm not sure of the details.

...

and left bracket is a valid character in filenames unlike ? and *.

And on POSIX systems, *all* wildcards are valid characters in file names. If you want to specify a file called literally "*.txt", or "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no different.

No, Windows *is* different, because you simply aren't allowed to have those characters in file names:

...

...

open("*.txt","w") Traceback (most recent call last): File "", line 1, in <module> open("*.txt","w") OSError: [Errno 22] Invalid argument: '*.txt'

Unix allows anything other than a slash or NUL, so all wildcards have to be escaped to get them past the shell; or, alternatively, you can just use a non-shell way of starting a program, like Python's own subprocess module.

The glob module supports escaping of wildcards, doesn't it? If so, we have no problem. If not, that's a bug, or at least an obvious and important piece of missing functionality.

I'm not sure that's the solution, because then Unix users would have to double-escape everything. It can be done, of course, but this is the exact sort of complication that single-platform developers often don't even think of. Suppose you develop on Windows, and just unconditionally glob all your arguments... your program will work fine on Unix, until it's asked to deal with a file with an asterisk in it, and your user complains very loudly about how it misbehaved... and maybe destroyed a bunch of files. Oops. Not good. ChrisA

On Jan 4, 2015, at 8:13 PM, Steven D'Aprano wrote:

On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote:

...

For those curious as to "Why not the wiki?", a Sphinx project hosted on a pull request capable service like GitHub, BitBucket or Kallithea offers a much nicer workflow for reviewing of proposed changes, together with an integrated issue tracker for submitting proposals for updates (https://github.com/pypa/python-packaging-user-guide/ is the project behind packaging.python.org, for example).

The concept of "proposed changes" goes completely against the grain of community-managed content. Imagine if Wikipedia required you to make pull requests.

The question to be asked is not whether "pull requests" are "nicer" than a wiki, but whether this information needs to be owned by a single person (or a small group thereof), and hence use the packaging.python.org module, or owned by the whole community, and hence use the wiki model.

At the moment, the wiki is horribly underused. I'm not blaming anyone for that, I'm just as much to blame as anyone else. I have made far more contributions to the ActiveState recipes than I have to the Python wiki. But I'd really like to see the wiki become the second place people look for information, after the official docs. And hopefully there won't be a need for a third place :-)

Although the quality of contrabutions is variable, the PHP community has made a very interesting decision to integrate community-supplied information with official docs:

http://php.net/manual/en/function.dechex.php

I wonder how we might do something similar?

On the other hand, a number of pages on the Wiki are either completely out of date because nobody cares about them or need to be locked because too many people care about it and they fight over which project should be named as recommended. Wiki content can be good for more factual based information but I think the format is relatively poor for things which are more opinion based. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sun, Jan 04, 2015 at 02:24:36PM -0500, random832@fastmail.us wrote:

On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:

...

...

Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind

Really? I would expect that Windows is probably the *easiest* platform to implement this functionality, as it has a single, standard, stable API for moving files to trash. (Or so I am lead to believe.) You can probably use that API via ctypes, or via pywin32.

Why would someone who is developing on Linux and has no pre-made library function to call bother with that?

Because they want to write cross-platform code. For my own personal scripts, I wouldn't bother. If I had a Windows system which I intended to use the script on, I would, or if I intended to make the script available to others, I might.

If being cross-platform isn't easy, it won't happen. You see it now with the lack of any support for "call glob on arguments on windows and not on unix" [because the shell handles it on unix] whether directly, in argparse, or in fileinput - today, nothing ever calls glob, and so people calling such scripts on windows can't use wildcards (glob is also different from windows wildcards in subtle ways)

I think you are mistaken that nothing uses glob: https://searchcode.com/?q=import+glob+lang%3Apython [...]

...

You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support.

I don't see why you need to call an API to the desktop enviroment.

You don't *need* to do so, but doing so guarantees that your code will match the expected behaviour as provided by the API, and avoids needing to re-invent the wheel. I would expect that for something which eventually makes it into the standard library, if it ever does, it will have multiple implementations: - call the platform API, if one exists and it is available - fall back to a Python implementation, if no such API exists That is exactly what the third party Send2Trash package does. If Gnome is installed and the GIO library is installed, it uses that, otherwise it falls back on its own (possibly inferior?) implementation. -- Steven

On Sun, Jan 4, 2015, at 20:31, Steven D'Aprano wrote:

I think you are mistaken that nothing uses glob:

I am very confident in saying that programs that do not use glob _to process lists of filenames given on the command line_ outnumber those that do at least 1000:1, and that the vast majority of your search consists of uses for processing things other than command line arguments.

https://searchcode.com/?q=import+glob+lang%3Apython

The vast majority of results (I used https://searchcode.com/?q=glob.glob+lang%3Apython so I could see the actual call in context) are using it for hardcoded strings, not command line arguments (and to do it correctly they should do even that conditionally on os.name == 'nt' [or os2 or ce, not sure about the other non-posix values])

On Mon, Jan 05, 2015 at 12:08:06PM +1100, Chris Angelico wrote:

On Mon, Jan 5, 2015 at 11:57 AM, Steven D'Aprano wrote:

...

On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832@fastmail.us wrote: [...]

...

Right now, none of this is done. If you pass *.txt on the command line to a python script, it will attempt to open a file called "*.txt".

On Windows, you will get Python's definition of globbing. On POSIX systems, the globbing module will do nothing, because the shell will most likely have already expanded the wild-cards

You're assuming that there are no wildcard characters actually included in the file names, which on Unix systems is perfectly possible. How can you reliably manipulate a file called *.txt when there are foo.txt and bar.txt in the same directory? With most Unix programs, you escape the asterisk to get it past the shell, the program does no globbing of its own, and you're safe. If your Python script unconditionally globs its file names, you're stuck.

You seem to be right. With double wildcard expansion (by the shell and by the python script) there doesn't seem to be a straightforward way to get just the file name with the wildcard in it: [steve@ando junk]$ cat testglob.py import sys import glob for arg in sys.argv[1:]: print(arg + ":-") for name in glob.glob(arg): print(" " + name) [steve@ando junk]$ ls eggs.jpg *.jpg testglob.py [steve@ando junk]$ python3.3 testglob.py *.jpg eggs.jpg:- eggs.jpg *.jpg:- eggs.jpg *.jpg [steve@ando junk]$ python3.3 testglob.py \*.jpg *.jpg:- eggs.jpg *.jpg [steve@ando junk]$ python3.3 testglob.py \\*.jpg \*.jpg:- I guess the simple solutions are to either do a platform check first, or to provide a --no-glob command-line switch to turn globbing off. But I guess people won't think of that until its reported as a bug :-)

...

...

Another separate but related issue is the fact that windows wildcards do not behave in the same way as python glob patterns.

I don't understand why you say this. As I understand it, there is no such thing as "Windows wildcards" as every application which wants to support wildcards has to implement their own. If you want to know what kinds of globbing wildcards the application supports, you have to read the application documentation. (Or guess.)

I am not a Windows expert, so I may have been misinformed. Anyone care to comment?

There's a very standard interpretation, which I think is codified in the FindFirstFile API, although I'm not sure of the details.

Do all Windows apps call FindFirstFile?

...

...

and left bracket is a valid character in filenames unlike ? and *.

And on POSIX systems, *all* wildcards are valid characters in file names. If you want to specify a file called literally "*.txt", or "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no different.

No, Windows *is* different, because you simply aren't allowed to have those characters in file names:

I know that. The point that Random raised is that some wildcards are legal in Windows file names. Yes, and *all* wildcards are legal in POSIX filenames, so whatever problems we have with escaping are going to occur on both platforms, as you have already pointed out above. In any case, having a platform specific globbing module is looking more and more useful. Does this need a PEP? -- Steven

Steven D'Aprano writes:

In any case, having a platform specific globbing module

By "platform-specific", do you mean one module that handles all platforms appropriately (what this thread is about), or do you mean per-platform modules (what "platform-specific" means to me)?

is looking more and more useful. Does this need a PEP?

If you're really serious about putting it in the stdlib, I would certainly think so. It's absolutely clear that the solution is going to be complicated and hard to understand. Without a PEP, nobody will know what is and what is not a bug. Likely even with a PEP a lot of people will claim the PEP is buggy no matter what it says! I would hope the different specs will partition the complainers, but I'm not even sure of that. ;-)

On Jan 5, 2015, at 2:41, Ethan Furman wrote:

On 01/04/2015 05:13 PM, Steven D'Aprano wrote:

...

Although the quality of contributions is variable, the PHP community has made a very interesting decision to integrate community-supplied information with official docs:

http://php.net/manual/en/function.dechex.php

I wonder how we might do something similar?

I hope that we do not. A separate wiki is fine, but the official docs should just have the official stuff in them.

MSDN has something similar. They've gone back and forth between a half-dozen different models, and I think at present it's just (formatted) comments on each documentation page. At any rate, it's very clear what's official documentation and what's some guy saying "the docs say you can't do Foo, but here's some code that simulates Foo in most cases". I've found useful stuff there. Then again, I've also found live code (and StackOverflow questions, etc.) that's broken because someone took some sample fragment from a comment and used it inappropriately (often ignoring the very next comment that explains why you shouldn't do that if X, Y, or Z). Both PHP and MSDN have a separate page for each function/method/attribute, while Python has a page per module (broken down into sections that are usually very nicely organized, so I wouldn't want to lose that...), so I'm not sure a similar comment format would work in the first place. Maybe allowing inline comments at every section break, entry, or even paragraph would work (like the way PyGame docs have inline example-search and other links)? But that could easily lead to docs that are so cluttered with comments (or with disclosure triangles, if they start off hidden) that the actual documentation is hard to read.

If the docs were a wiki, we'd have scads of entries on "how to program <language-x> in Python",

In other fora (python-list, python-tutor, StackOverflow, comments on widely-followed blogs, etc.) when someone suggests writing Java-style code instead of Pythonic code they usually get quickly shouted down, often even with good explanations, so I'm not sure things would be any worse on a wiki attached to the Python docs. Also, the Python wiki is already a wiki, and isn't dominated by that kind of cruft. So I'm not sure that problem is as much of a threat. That being said, I agree with your conclusion. Docs seem to work best information that's managed like code; things that aren't quite docs but are useful may work best in other ways, but people have already found the wiki, StackOverflow, ActiveState, blogs, etc. and aren't having any problem using them. And, of all the alternate places to find elaborated documentation, the wiki seems to be among the least popular, which doesn't seem to imply that more wikification is the answer to any relevant question.

On Sun, Jan 4, 2015 at 7:22 PM, Ben Finney wrote:

In recent years (I noticed this since about 2013), the Python wiki does not offer an “Edit” function when I am logged in; every page presents “Immutable Page” instead.

Ben (and others), I just noticed this small paragraph on the FrontPage: If you want to edit a page and have just signed up, or you find that you can no longer edit a page that you could edit before, please write to the pydotorg-www mailing list, stating your account name and your intended edits and we'll add you to the EditorsGroup. I suspect that has something to do with your inability to edit. Somewhere along the way, someone must have made the executive decision that open editing no longer worked (too much spam?) and that editing needed to be a by-invitation or by-request sort of thing. I'd add you, but am getting nothing but Gateway Timeout responses at the moment. Skip

On Mon, Jan 5, 2015 at 9:27 AM, M.-A. Lemburg wrote:

...

I'd add you, but am getting nothing but Gateway Timeout responses at the moment.

If you get those, simply try to resend/reload. The wiki will notice duplicate editing requests. In my experience, the editing requests typically make it through to the wiki the first time and you get a duplicate editing warning on the second one.

Unfortunately, I couldn't even get to the EditorsGroup page to edit it. Now, about an hour later, I finally succeeded. Ben, send me your wiki login and I'll add you. MAL, I thought the spam problem had mostly been solved by Martin's modification several years ago to require a certain time to pass between a new login being created and their first attempt to edit. Skip

On Mon, Jan 05, 2015 at 01:19:16PM +0900, Stephen J. Turnbull wrote:

Steven D'Aprano writes:

...

In any case, having a platform specific globbing module

By "platform-specific", do you mean one module that handles all platforms appropriately (what this thread is about), or do you mean per-platform modules (what "platform-specific" means to me)?

Yes to both. The model I am proposing is that of the os and os.path modules: we import os and it does the right thing for the current platform. In particular, os.path will do the right thing for the current platform, but you can import ntpath, posixpath, genericpath etc. to get behaviour for a specific platform. I think we have to leave the glob module as-is, for backwards compatibility, and add a new package, say globbing (or osglob, the name can be bike-shedded to death later *wink*) with the new behaviour. globbing will look something like this: globbing/ __init__.py posix.py nt.py generic.py (plus any other platforms that may need special treatment, if any) and the globbing/__init__.py implementation could potentially be as simple as "from foo import *" from the appropriate sub-module. The globbing.generic implementation might be as simple as: from glob import * Or perhaps the relationship should go the other way? Move the current glob.py implementation into globbing.generic, and replace it with: from globbing.generic import * These little details are to be decided later, but the aim is: * Current code that imports glob will continue to work the same as it does now, warts and all. * Code that imports globbing will do the right thing for the platform. * If you want to apply globs to something other than file names, the right module to use would probably be globbing.generic (or possible fnmatch directly). -- Steve

Steven D'Aprano writes:

The model I am proposing is that of the os and os.path modules: we import os and it does the right thing for the current platform.

Which is normally "nothing" on POSIX, since the shell does it for you. Or, if you're talking about what the shell does for you, while I suppose there is a basic globbing defined by POSIX, but bash and zsh go well beyond that, and they behave somewhat differently in corner cases IIRC. If the program is going to invoke a globbing function specifically, I suppose it makes sense to default to the same behavior as on the current platform, but it's not obvious that that's what the user wants. I use bash on Windows; why settle for cmd.exe if you want the power of the command line? It could even cause problems (is there a reliable way to determine what the shell is, let alone whether it implements globbing or not, on Windows?)

In particular, os.path will do the right thing for the current platform, but you can import ntpath, posixpath, genericpath etc. to get behaviour for a specific platform.

But there *is* a RightThang for paths; the OS decides. Globbing is a user preference thing.

On Sun, Jan 4, 2015, at 23:19, Stephen J. Turnbull wrote:

By "platform-specific", do you mean one module that handles all platforms appropriately (what this thread is about), or do you mean per-platform modules (what "platform-specific" means to me)?

Windows technically requires calls to the actual filesystem driver for correct behavior - different filesystems may have different case folding, theoretically other different rules, etc. Plus you've got to examine both the long and short filenames - if you have foobar.html, *.htm will match it because the short filename is FOOBAR~1.HTM (it, however, _returns_ the long filename.) This means that the equivalent function to glob can't simply be the equivalent of listdir filtered by fnmatch. On Mon, Jan 5, 2015, at 05:10, Andrew Barnert wrote:

It seems to me that if you want auto-globbing support on Windows, the "right" way to do it is to link in the fancy setargv instead of the default one. This handles wildcards exactly the same way the command.com/cmd.exe builtin commands do, which I suspect is what Windows users would actually be expecting of they passed a wildcard on the command line. (I think the docs no longer guarantee that this is true, but it's probably still true, and certainly closer to true than if you try to do it manually.)

The problem with that is that if you do that you can no longer pass in _non_-filename arguments that contain a question mark or asterisk (and happen to match a file). Better to do it inside the program, when you actually know the argument you're looking at is intended to be a filename spec. Which, I assume, is _why_ it's not done by default when you compile a C program. A program like grep "a.*b" "Some Directory\*.txt" should handle wildcards in the filename and not in the regex. That is the user expectation on windows, and that is something that windows programs (and not unix programs) can and do deliver. Setargv doesn't, which is why you have to opt in to it (if you know your program doesn't need non-filename arguments that look like wildcards). It should be possible to write Python programs that can do the same, in a way that is relatively painless. A setargv-like solution is insufficient for that. On Mon, Jan 5, 2015, at 13:45, Skip Montanaro wrote:

I only discovered this "shortcoming" (or "Bourne Shell dependency") relatively recently. I've been using bash for so long it never even occurred to me that {...} notation wasn't available in all shells.

Brace expansion is technically not globbing. yen{2,3}.txt will return yen2.txt yen3.txt even if one or both of those files don't exist. The reason it works with globs is that yen{2,3}.* literally expands to yen2.* yen3.* which is then globbed.

On 5 January 2015 at 20:33, wrote:

On Mon, Jan 5, 2015, at 05:10, Andrew Barnert wrote:

...

It seems to me that if you want auto-globbing support on Windows, the "right" way to do it is to link in the fancy setargv instead of the default one. This handles wildcards exactly the same way the command.com/cmd.exe builtin commands do, which I suspect is what Windows users would actually be expecting of they passed a wildcard on the command line. (I think the docs no longer guarantee that this is true, but it's probably still true, and certainly closer to true than if you try to do it manually.)

The problem with that is that if you do that you can no longer pass in _non_-filename arguments that contain a question mark or asterisk (and happen to match a file). Better to do it inside the program, when you actually know the argument you're looking at is intended to be a filename spec. Which, I assume, is _why_ it's not done by default when you compile a C program.

Just as a note - I find the fact that Python *doesn't* allow the C runtime to do its glob-mangling of the supplied command arguments a very useful feature, and I would not want it to be "fixed". The setargv behaviour in the MSVC runtime is full of arcane corner cases and quoting gotchas, and I wouldn't want it imposed on me by default. Paul

On Tue, Jan 06, 2015 at 04:14:38AM +1100, Chris Angelico wrote:

On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano wrote:

...

* If you want to apply globs to something other than file names, the right module to use would probably be globbing.generic (or possible fnmatch directly).

To clarify: Do you mean "something other than names of currently-existing files", or "something other than valid names for the local file system", or something else? For instance, suppose you write an unarchiver (in case we don't have enough of them already, of course), and you can "python unarchive.py archivename *.txt" to extract all files matching *.txt. The globbing would be done against some sort of internal index, but the names would have to be valid for the local file system, or you wouldn't be able to create them. Which module should you use?

fnmatch. I've had a look inside glob.py and it calls os.listdir directly, so you cannot use glob to match things other than actual existing files. (Well, I suppose you could if you monkey-patched the module, but lets not go there.) fnmatch, on the other hand, provides the right tool for matching names other than actual file names: fnmatch.filter(names, pattern). (Well, almost the right tool -- it has a few issues too.) To summarise: - the glob module returns only actual files in the file system; - the new globbing package will do the same, except it will use platform-specific wildcards where possible; - fnmatch continues as Python's generic "match globs against arbitrary strings" module (despite the name). -- Steven

On Tue, Jan 6, 2015 at 10:57 AM, Steven D'Aprano wrote:

On Tue, Jan 06, 2015 at 04:14:38AM +1100, Chris Angelico wrote:

...

On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano wrote:

...

* If you want to apply globs to something other than file names, the right module to use would probably be globbing.generic (or possible fnmatch directly).

Which module should you use [to match non-existing files]?

fnmatch. I've had a look inside glob.py and it calls os.listdir directly, so you cannot use glob to match things other than actual existing files. (Well, I suppose you could if you monkey-patched the module, but lets not go there.)

fnmatch, on the other hand, provides the right tool for matching names other than actual file names: fnmatch.filter(names, pattern). (Well, almost the right tool -- it has a few issues too.)

Ugh. That's poor naming, then. If this new globbing module happens, I would be a strong +1 on having 'globbing.generic' (or somesuch) to do this kind of thing, if only to fix the name. ChrisA

On 5 January 2015 at 20:38, David Wilson wrote:

I have no problem with this being included somehow in the standard library, but pathlib to me is all about the filesystem, whereas (as others have pointed out), the Recycle Bin and associated concepts on other OS relate pretty uniformly to the OS shell.

There are many shell-related concepts that could be nicely exposed, but again I don't think any of them belong in pathlib. For example, labels/tags, owner application, icon, visibility flags, etc.

Better tools for accessing typical extended file metadata would indeed be highly desirable (thumbnails/previews are another one that comes to mind). As with the recycle bin though, the generally preferred approach for standard library additions is incorporating existing modules that have been available through PyPI for some time, or else creating a derivative of such a project that meets other constraints of standard library inclusion (e.g. ipaddress vs the project it was based on, ipaddr). Unfortunately, most of the folks that need these cross-platform integration features are likely to be using a C/C++ toolkit that provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces the incentive for anyone to create a Python-specific abstraction that would be suitable for eventual inclusion in the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 01/05/2015 10:26 PM, Ben Finney wrote:

"M.-A. Lemburg" writes:

...

[editing the Python wiki] requiring asking for editing rights on the pydotorg-www mailing list. Which is a little annoying and some work for the admins, but has resulted in the amount of spam to go down to zero. We usually grant the editing requests within a day.

It does need to be recognised that this will dramatically reduce the number of pople who will pass that hurdle to edit the wiki, and thus the wiki loses a lot of edits (both helpful and unhelpful) it might otherwise have.

At least the edits being made now are by folks who care. Better fewer edits than a site full of little but spam.

Thanks to those doing the work to maintain Python community infrastructure.

Seconded. -- ~Ethan~

On Jan 6, 2015, at 5:43, Nick Coghlan wrote:

On 5 January 2015 at 20:38, David Wilson wrote:

...

I have no problem with this being included somehow in the standard library, but pathlib to me is all about the filesystem, whereas (as others have pointed out), the Recycle Bin and associated concepts on other OS relate pretty uniformly to the OS shell.

There are many shell-related concepts that could be nicely exposed, but again I don't think any of them belong in pathlib. For example, labels/tags, owner application, icon, visibility flags, etc.

Better tools for accessing typical extended file metadata would indeed be highly desirable (thumbnails/previews are another one that comes to mind). As with the recycle bin though, the generally preferred approach for standard library additions is incorporating existing modules that have been available through PyPI for some time, or else creating a derivative of such a project that meets other constraints of standard library inclusion (e.g. ipaddress vs the project it was based on, ipaddr).

Unfortunately, most of the folks that need these cross-platform integration features are likely to be using a C/C++ toolkit that provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces the incentive for anyone to create a Python-specific abstraction that would be suitable for eventual inclusion in the standard library.

I'm not sure that's really unfortunate. First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did? Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework… but many do. And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.).

On Jan 6, 2015, at 11:25, Nick Coghlan wrote:

On 6 January 2015 at 19:29, Andrew Barnert wrote:

...

On Jan 6, 2015, at 5:43, Nick Coghlan wrote:

...

Unfortunately, most of the folks that need these cross-platform integration features are likely to be using a C/C++ toolkit that provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces the incentive for anyone to create a Python-specific abstraction that would be suitable for eventual inclusion in the standard library.

I'm not sure that's really unfortunate.

First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did?

Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework… but many do.

And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.).

Yes, there's certainly a lot of value in adopting an existing specialist toolkit when writing rich client applications. The key problem is that it isn't easy to start with the provided Tcl/Tk support in the standard library (to avoid introducing a large external dependency) and then later migrate to one of the other third party toolkits (if the basic functionality proves insufficient).

While an asyncio-style pluggable solution may be one possible option for that (i.e. provide default implementations of features which may be fairly basic, but allow them to be replaced by full featured implementations backed by Gtk/Qt/wxWidgets/etc), that kind of abstraction layer design is extraordinarily difficult to design well. It's hard enough even in cases like wsgiref and asyncio that are relatively simple compared to the services a full rich client application will expect to have available.

Yes, and even besides the difference in complexity, even the most basic elements--the API for the runloop and message maps (wx), signals/slots (Qt), event handlers (Tk), etc.--are pretty hard to abstract away. For servers, a reactor is a reactor, but for GUIs, a signal/slot system is not a message pump/message map system. And whichever abstraction you pick, adapting it to work with the other abstractions is no fun. (Look at how long the TkAqua Cocoa port took, and how much Cocoa functionality is still missing.)

... I had created an awesome-python-android page, but I seem to have `push -f`'ed over it and the reflog is too hard. On Tue, Jan 6, 2015 at 1:25 PM, Wes Turner wrote:

Fair disclosure: I'm not a big fan of Moin syntax (can Pandoc convert this?). I don't like having to login to share with the Python community. It would be great to gain maximum synergy from the community on this.

Some interesting perspectives on wikis, and where wikis are going:

[1] https://github.com/blog/1939-how-github-uses-github-to-document-github [2] https://github.com/westurner/wiki/blob/master/Makefile

[2] is a Sphinx Makefile for a GitHub ReStructuredText wiki. It seems to work okay for my purposes. What could prevent this from scaling to handle the Python wiki?

From https://github.com/westurner/wiki/issues/5 , in Markdown (which works on both BitBucket and GitHub):

# Advantages

...

* Beautiful Sphinx output:

...

https://github.com/westurner/wiki/blob/master/conf.py * Search * Context-focused pages * Navbar * **Responsive** CSS Freedom; for my cell phone * Pull Requests # Challenges / Disadvantages * New wiki pages created through the GitHub wiki interface add the `.rest` suffix, rather than `.rst` * Sort of an advantage (`.rest` files are docutils-only) * This precludes inclusion of the wiki pages into a standard Sphinx docset as a submodule or subtree. * Sphinx ReStructuredText is a superset of GitHub ReStructuredText (docutils), so Sphinx markup is, in the best case not supported, and in the worst case breaks the wiki render * Notably: `:ref:`, `.. index:`, and `.. toctree::` are only supported by Sphinx * To test this: `make rst2html_all` * Double Tables of Contents (#4) * Double pulls and pushes (`make pull push`): https://github.com/westurner/wiki/blob/master/Makefile * Sphinx requires a manual build step in order to publish in-progress changes

On Tue, Jan 6, 2015 at 2:31 AM, M.-A. Lemburg wrote:

...

On 06.01.2015 07:26, Ben Finney wrote:

...

"M.-A. Lemburg" writes:

...

[editing the Python wiki] requiring asking for editing rights on the pydotorg-www mailing list. Which is a little annoying and some work for the admins, but has resulted in the amount of spam to go down to zero. We usually grant the editing requests within a day.

I quite understand the motivation for this.

It does need to be recognised that this will dramatically reduce the number of pople who will pass that hurdle to edit the wiki, and thus the wiki loses a lot of edits (both helpful and unhelpful) it might otherwise have.

We might lose some occasional quick typo fixes, but it seems that those genuinely interested in helping with the wiki don't have a problem asking on the list.

To reduce the problem I had created a list of user names who had made good edits in the months before the change.

The only nit I still have with the setup is that it's not necessarily apparent for new users how to get editing rights. There's a section on the front page, but you don't see this when looking at a random page.

...

Thanks to those doing the work to maintain Python community infrastructure.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jan 06 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum wrote:

it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant.

Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn. Skip

On Tue, Jan 6, 2015 at 1:25 PM, Wes Turner wrote:

I'm not a big fan of Moin syntax (can Pandoc convert this?).

I haven't followed wiki markup at all in the past several years. Back when Moin came along, it was (arguably?) the correct choice for wiki.python.org. I think most of us active at the time were very much in the "eat your own dog food" camp at the time. I've never heard of Pandoc (I grow old, can't follow every trend), but I thought there were some tools which could convert Moin into other formats. I can't find much now, though this https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ looks like it might be a reasonable starting point. Skip

On Jan 6, 2015, at 2:44 PM, Guido van Rossum wrote:

On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro mailto:skip.montanaro@gmail.com> wrote:

On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum mailto:guido@python.org> wrote: it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant.

Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn.

Nice one, and agreed. I don't see anyone with a serious wish to be a master gardener for wiki.python.org http://wiki.python.org/ in this sense though. :-( Perhaps we should advertise the position? It's a volunteer role, but will require a lot of motivation. Ideally the master gardener team should be allowed to select the tool suite and be given permission to switch to a new suite pretty aggressively. The team should also be responsible for deciding on the policy for edit access. This seems more workable than having an open-ended discussion on python-ideas.

I have very little opinion on what the wiki is or does but I wanted to just say if at some point we do switch software then with my infrastructure team hat on I would ask for two properties to exist in the software (not really specific to the wiki tbh): 1. Configuration that is able to be handled via config management like salt/chef/puppet etc. Typically this just means that the configuration is handled via a text file and the software itself doesn’t attempt to write to the configuration as part of it’s normal execution. 2. State does not need to be stored on the local file system and instead can be stored in something like a database (preferably PostgreSQL) or an object store (preferably cloudfiles). This allows us to treat the servers running the software as emphereal with all of the state stored elsewhere. This makes it easier to manage the servers, makes it easier to recover from security issues on a server, makes it easier to upgrade the server, and makes it easier to scale the service in general. Of the two, the first property is the most important, but the second one is a really good idea too. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 06.01.2015 20:44, Guido van Rossum wrote:

On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro wrote:

...

On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum wrote:

...

it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant.

Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn.

Nice one, and agreed. I don't see anyone with a serious wish to be a master gardener for wiki.python.org in this sense though. :-( Perhaps we should advertise the position? It's a volunteer role, but will require a lot of motivation. Ideally the master gardener team should be allowed to select the tool suite and be given permission to switch to a new suite pretty aggressively. The team should also be responsible for deciding on the policy for edit access. This seems more workable than having an open-ended discussion on python-ideas.

Such discussions should really happen on pydotorg-www where that team already works. We do have several people who maintain pages or page sets on the wiki, but more editorial help is always welcome. We need people who have experience in technical writing and a passion to maintain informational resources. FWIW: A change of tools won't magically give us better content. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 01/06/2015 02:11 PM, Wes Turner wrote:

On Tue, Jan 6, 2015 at 4:01 PM, M.-A. Lemburg wrote:

...

Such discussions should really happen on pydotorg-www where that team already works. We do have several people who maintain pages or page sets on the wiki, but more editorial help is always welcome.

https://www.python.org/dev/pydotorg/website/

Subversion, huh?

A boost in familiar usability could encourage effortless contributions.

Not even breathing is effortless. -- ~Ethan~

Hi, This idea was already proposed as part of a larger PEP: https://www.python.org/dev/peps/pep-0471/#wildcard-support The PEP explains why the Windows wildcard idea was rejected. Victor

On Tue, Jan 6, 2015 at 5:04 PM, M.-A. Lemburg wrote:

...

https://mail.python.org/mailman/listinfo/pydotorg-www

Yep, that's the list.

...

https://www.python.org/dev/pydotorg/website/

That page explains the old python.org website system. It doesn't have anything to do with wiki.python.org.

My mistake. That page does seem outdated. I must've been looking for one level up: https://www.python.org/dev/pydotorg/ What content should be version controlled with the website?

...

...

We need people who have experience in technical writing and a passion to maintain informational resources.

FWIW: A change of tools won't magically give us better content.

A boost in familiar usability could encourage effortless contributions.

The wiki does support ReST syntax if you don't like the wiki markup:

http://moinmo.in/HelpOnParsers/ReStructuredText

Ah, thanks!

On Wed, Jan 07, 2015 at 12:11:51AM +0100, Victor Stinner wrote:

Hi,

This idea was already proposed as part of a larger PEP: https://www.python.org/dev/peps/pep-0471/#wildcard-support

The PEP explains why the Windows wildcard idea was rejected.

The Windows wildcard idea was rejected in the context of a faster os.walk, not globbing. The PEP is inaccurate: it gives the wrong Python-Ideas thread for discussions on the wildcard idea. The wildcard idea is discussed in this thread: https://mail.python.org/pipermail/python-ideas/2012-November/017965.html not in the thread given in the PEP. Well, I say "discussed", but it's mostly ignored, with only a couple of comments. There's somewhat more discussion on Python-Dev: https://mail.python.org/pipermail/python-dev/2014-June/135217.html but it's mostly people just saying "-1 on wildcards". I agree with this decision, for PEP 471. But I don't believe this is relevant to the question of explicit globbing. -- Steve

On 01/06/2015 05:16 PM, Stephen J. Turnbull wrote:

Ethan Furman writes:

...

On 01/06/2015 02:11 PM, Wes Turner wrote:

...

A boost in familiar usability could encourage effortless contributions.

Not even breathing is effortless.

True, but that's the wrong fallacy. [...]

The fallacy in applying wiki-think to Python is that [an] *increase in quantity* of documentation is [not necessarily an] *improvement in quality* of documentation. [...]

It isn't even clear that filling an obvious hole with an accurate-as- far-as-it-goes drive-by contribution is a good thing. Not only may the residual inaccuracy mislead the reader, it may also contribute to an impression that the Python docs are lower quality on average than they actually are.

Thanks -- that's worded much better than my own feeble attempt. :) -- ~Ethan~

On Wed, Jan 7, 2015 at 2:49 AM, M.-A. Lemburg wrote:

...

What content should be version controlled with the website?

The current python.org website is a mix of code and templates in github repo, content/data stored in a database and a (for various reasons) homegrown CMS to manage pages such as the ones you found.

There's work underway to replace the homegrown CMS with a standard more feature rich one, which will hopefully allow us to open up site editing to way more volunteers than we currently have.

So there are no topical guidelines?

wiki.python.org is a separate VM, which doesn't have anything to do with www.python.org. The VM runs a more or less standard moin installation, which uses its own version control system (you can access this via the "info" link on the pages).

https://github.com/dwf/moin2rst Formatter plugin

Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's ``plugin/formatter`` directory.

Action plugin

Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's ``plugin/action`` directory.

...

...

...

...

We need people who have experience in technical writing and a passion to maintain informational resources.

FWIW: A change of tools won't magically give us better content.

A boost in familiar usability could encourage effortless contributions.

The wiki does support ReST syntax if you don't like the wiki markup:

http://moinmo.in/HelpOnParsers/ReStructuredText

Ah, thanks!

On Tue, Jan 6, 2015, at 19:42, Steven D'Aprano wrote:

The PEP is inaccurate: it gives the wrong Python-Ideas thread for discussions on the wildcard idea. The wildcard idea is discussed in this thread:

https://mail.python.org/pipermail/python-ideas/2012-November/017965.html

I think the fundamental difference between these cases is in handling user-entered strings, especially command-line arguments (where the user will expect them to behave a certain way, in particular "*.*" matching all files even those with no dot - most of the other quirks are obscure anyway) vs hardcoded (where expectations are determined by the programmer, not the user).

On Wed, Jan 7, 2015 at 7:27 AM, Skip Montanaro wrote:

I'm going to add one more bit here about wiki gardening.

In order to test parsing code in order to produce Linked Data http://5stardata.info , it could be helpful to produce a periodic archive/dump sort of like http://wiki.dbpedia.org/Datasets . A github repository could be useful for hosting these types of releases. python/wiki___ ?

BITD, the main website had many pages which today would be called listicles. The problem was that those lists were dynamic. They weren't just the "top ten scientific Python modules of all time" or the "seven best packaging tools." They were whatever was at the front of the author's brain at some point in the past. Having these static pages given the imprimatur of the PSF when in fact none of the site maintainers were obviously responsible for (or interested in) keeping them up-to-date did a disservice to the community and to authors of other packages which weren't represented in those lists. In addition, there were more barriers to update than necessary (essentially, figure out how to report the problem, offer suggested fix(es), then have them swallowed up into the site update mechanism).

Are the links out of date, or the descriptions and exclusive clusters? * http://schema.org/SoftwareApplication (RDFa in HTML5, Microdata) * http://json-ld.org/ * URIs are tags

One of the main uses envisioned for the wiki was as a place where these listicles could be maintained by the greater Python community. For some things, that's worked out pretty well. The most obvious thing that comes to mind is the PythonTraining page. That works because the people whose skills are represented on that page have a very good reason for keeping things up-to-date: it's free advertising for their businesses.

Other listicle type pages haven't been keep as up-to-date. For example, the PythonEditors page was last updated in Feb 2014, is huge (and might benefit from being split into multiple pages), and probably no longer accurately represents the available editors or IDEs which support Python.

The takeaway in my mind is that we could probably use "gardeners" to take over active maintenance of these listicle pages. That, coupled with a couple "master gardeners" to develop some suitable structure, and some landscape crews to prune dead/outdated/no-longer-useful pages, would likely go a long way to improving the quality of the wiki.

Skip

Are there tools for working with a regular editor (such as vim) and MoinMoin pages? With ReStructuredText, the Riv.vim syntax helpers are great for things like tables; Voom is great for outlines (:Voom rest).

On 07.01.2015 17:55, Skip Montanaro wrote:

On Wed, Jan 7, 2015 at 10:23 AM, Wes Turner mailto:wes.turner@gmail.com wrote:

...

Are there tools for working with a regular editor (such as vim) and MoinMoin pages?

I'm not at all concerned with this, as I see the editing step being pretty much orthogonal to the markup language and the version control tools. Editing through a web browser pretty much sucks no matter what you're trying to edit, especially if you are used to using highly functional, mature editors like vim and Emacs. If you want your editing to happen outside a glorified <textarea> widget, find an "offline" editing plugin. (I happen to use Edit with Emacs from Chrome, but there are other extensions available.)

I assume that MoinMoin will eventually be supplanted by some other (wiki?) technology. I do think it's worth noting though, that while Github and other version control services are all the rage for all sorts of collaborative editing, wikis in general predate that sort of technology by a long way. If I read correctly, the original C2 Wiki was launched in 1995, MoinMoin in 2000, Git in 2005, and Github not until 2007. See:

* http://c2.com/cgi/wiki?WikiHistory * http://en.wikipedia.org/wiki/MoinMoin * http://en.wikipedia.org/wiki/Git_(software) * http://en.wikipedia.org/wiki/GitHub

It's only more recently developed wiki systems (or perhaps a few wiki systems whose authors were prescient and saw DVCSs coming) which support generic version control backends.

I think I posted a link yesterday which looked like a reasonable starting point for converting MoinMoin markup to Markdown. Run with that if you want. If you can parse MoinMoin markup and generate Markdown, it shouldn't be too difficult to iterate over all the pages and their iterations in wiki.python.org, convert to Markdown, then commit to a Github repo.

... and then you've changed the technology, lost the dynamic features of wikis, the permission controls, full text search, but have not solved the real issues, only created more work that's not relevant to the content.

I think MoinMoin has served the Python community pretty well. Despite its apparent unpopularity, I think structural and content issues are more important than the markup syntax used for that content or the version control scheme implemented in the backend.

In the end, it all boils down to people curating content. The tools, the markup and version control are all secondary. My recommendation: look through the wiki pages, grab some that are older and some care, and improve them. That's going help much more than this tools discussion. Here's starting point: https://wiki.python.org/moin/WikiGuidelines -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 07.01.2015 21:55, Skip Montanaro wrote:

On Wed, Jan 7, 2015 at 2:45 PM, M.-A. Lemburg wrote:

...

... and then you've changed the technology, lost the dynamic features of wikis, the permission controls, full text search, but have not solved the real issues, only created more work that's not relevant to the content.

I'm not sure what the disagreement is here. I wasn't advocating moving away from wiki technology generally, just observing that the current tool (MoinMoin) has different markup syntax and version control than people appear to like (Markdown and GitHub). I imagine you could find a wiki system which uses those two frontend and backend technologies. I was only pointing out that converting from one to the other shouldn't be terribly onerous, as long as you could iterate over the versions and parse the existing syntax into the new form.

Ah, sorry. I was obviously misunderstanding your emails :-) Moin does support ReST out of the box and there's also an addon parser for Markdown, we could install, if it's stable enough.

In any event, I agree that the content and structure is more important than the implementation technology. If you can't solve those problems, changing technology winds up being just a bunch of useless effort.

Yep. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/