Adding `pathlib.Path` method that would send file to recycle bin

Hi guys, I want to have a method of `pathlib.Path` that would send a file to the recycle bin. (i.e. soft delete.) What do you think about adding this? I see there's a PyPI package `Send2Trash` that does this, but it would be nice if this was in the standard library. Thanks, Ram.

On Mon, 29 Dec 2014 18:54:46 +0200 Ram Rachum <ram@rachum.com> wrote:
I think the corresponding low-level function to the os module would be a good start (the hard part is probably to write the cross-platform code, anyway). Regards Antoine.

On Dec 29, 2014, at 18:53, Antoine Pitrou <solipsis@pitrou.net> wrote:
There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there. On OS X, unless you want to use higher-level CoreFoundation APIs that require a runloop, you just look up the name of the trash directory (which I believe is theoretically localized but actually always ~/.Trash on the drive with your user directory, $mnt/.Trash otherwise) and call rename. On Linux, and some other *nix platforms, you assume XDG directory layout and call rename. On other platforms, I don't know if there even is a way to do it. And I don't know what the right thing to do on OS X would be for a file on a different filesystem than the user's home that doesn't have its own .Trash or is mounted read-only or no-trash. Finder gives you a popup in each of those cases asking if you want to delete it immediately because it can't be trashed. And similarly for Linux, but the details are different (especially since you might not even have a home trash). I'd guess Send2Trash is a good place to at least start, but if this is going in the stdlib someone needs to make sure it meets Apple HIG rules and XDG standards.

On Mon, Dec 29, 2014, at 13:58, Andrew Barnert wrote:
There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there.
The Windows function for this is SHFileOperation, which isn't really low-level (but doesn't require a message loop).
Is it not uid-specific? Windows and XDG trash directories are uid-specific. A google search shows that it's $mnt/.Trashes/$uid.
If it's mounted read-only, you have a bigger problem. Another possible issue could be: bad permission on trash directory, bad permission or nonexistent parent directory that trash directory must be created in (home or .Trashes). These are also all issues that must be faced for XDG. XDG has a database of original location and deleted date for files in the trash - does OSX? (according to the tool I found, it does, at least for original location, but only works if you use Finder to trash the files)
Another option would be to just script Finder with osascript. The tool http://hasseg.org/trash/ has objc code to "directly" script Finder without actually going through osascript. When it does not use Finder, it uses FSMoveObjectToTrashSync, which seems to be depreciated.

On Dec 29, 2014, at 22:15, random832@fastmail.us wrote:
Well, yes, but the point is that Finder gives you a special message for this rather than trying to move it and giving you a permissions error. And more importantly, if it's mounted no-trash or doesn't have a trash can, you don't have a bigger problem, but you still have this problem, so this problem needs to be solved.
I believe all of those issues could just raise the normal exception, because Finder treats them as normal errors--that is, rather than giving you the special "can't trash; delete?" message, it tries, fails, asks you to authenticate as an administrator, and tries again.
That's a whole other issue. I believe that if you use the higher-level APIs instead of POSIX functions they take care of the desktop database and alias/bookmark info and all that stuff. If not... It may happen automatically (at least if you're on an HFS+ drive) but it may not.
If you want to script Finder, the easy way to do it (well, as easy as it gets without Appscript) is ScriptingBridge and/or the lower-level Cocoa AppleScript APIs (via PyObjC, but none of that is going to go into the stdlib, which means shutil needs to be written partly in ObjC), but that will raise the minimum OS X version, because 10.6 didn't have NSAppleScript and friends yet. (And the Carbon AEM calls are deprecated, of course.) This could also make it more difficult for Python apps to be sandboxed and AppStore'd. But the biggest problem is that you're asking Finder to do it. If you don't have permission to trash a file, instead if getting an error, you'll get an auth dialog from Finder. If you trash a bunch of files you'll get a progress bar in Finder (and your script will get a timeout from the synchronous AppleEvent and not know if the trashing succeeded). Worse, if you don't have a Finder to talk to (because you're running in an ssh shell, or in a LaunchDaemon or a detached script, or because you're using a third-party Finder replacement like PathFinder) you won't be able to trash anything. There are plenty of applications that's reasonable for, but I don't think it's reasonable stdlib behavior.
When it does not use Finder, it uses FSMoveObjectToTrashSync, which seems to be depreciated.
Yeah, that's Carbon; you're supposed to use the Cocoa/CF APIs instead. NSWorkspace had one method that's guaranteed to do exactly the same thing as trashing in Finder--which it does by asking Finder--and a more general "perform operations on files" method that can be used for trashing without the Finder, but there's no way to control whether it's synchronous or asynchronous, and requires a runloop in case it's async. Anyway, I don't think these details change the point: There is no low-level function to be added to os. There may be a high-level function to be added to shutil, but it's not trivial.

On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote:
That doesn't make sense - the problem is with the state of the trash directory (it may be in a state that you can access, but the "wrong" one, e.g. one that would allow other people to delete your files - I don't know if this is a problem for OSX but it is for XDG), not your access to the file being deleted.

On OSX there are at least two reasons why the Finder can prompt you when you try to move a file to the trash. The first is a source location that doesn’t support the trash (such as a fileserver), the Finder will ask if should just permanently remove the file. The second is when you don’t have permissions to remove the file, the Finder will then ask if it should retry with Administrator privileges (and will prompt you for a username and password if you confirm). Ronald

Andrew Barnert wrote:
You may also need to invent a unique name if there's already a file in the trash with the same name.
Raising an exception would seem to be a reasonable thing to do in that case. -- Greg

On Dec 30, 2014, at 22:36, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
At this point too much context has been stripped from the message for the replies to make sense. The original message was pointing out that there is no separate "low-level API" for trashing that could go in os along with a "high-level API" in shutil--unless you consider rename a low-level API for trashing, in which case it's already there.
And whatever else the api does for you… IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync.
Except that API is deprecated (meaning, among other things, that if shutil linked to it, any py2app-ed or embedding app that imports shutil can't be sold in the Mac App Store), and there are no suitable replacements. By "suitable" I mean "suitable for widespread-enough use that it should be in the stdlib". As I said earlier in the thread, I think it's perfectly reasonable to have a third-party library that uses a deprecated API, requires 10.8, requires a runloop, directly or indirectly asks the Finder to do the work for you, or doesn't quite meet Apple's HIG rules, because all of those could be acceptable in _many_ applications; I just don't think any of those should be in the stdlib, because none of those are acceptable in _most_ applications. If I were designing an app right now, here's what I'd do: If you don't need 10.7 or earlier, use NSFileManager (via PyObjC if possible, via a tiny custom (Obj)C extension module if for some reason it's not, and same goes for all the following). If you need 10.5-10.7 but don't care about the App Store, check NSFileManager dynamically, and fall back to the FS function if the method is missing. If you need 10.6-10.7 and also need the App Store, you've probably got a runloop, so use the async function in NSWorkspace (recycleURLs:completionHandler:). If you need 10.6-10.7 in a helper app in the App Store, use the other function in NSWorkspace (performFileOperation:source:destination:files:tag:) and deal with all of its problems because you have no choice in the matter.

On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote:
What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is? http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-... Having to call out to an external program via Applescript feels rather icky, but given the difficulty of simultaneously supporting multiple OS X versions, satisfying the rules of Apple Store, and using Apple's APIs, I don't see any alternative. -- Steve

On Dec 31, 2014, at 0:47, Steven D'Aprano <steve@pearwood.info> wrote:
I already answered this briefly earlier in the thread, but in more detail: 1. If you don't have a GUI login session (e.g., because you've ssh'd into the box, or are running as a LaunchDaemon or a traditional detached script), you have no Finder in your session to script. 2. If you're running in a sandbox that doesn't have scripting rights for Finder added in--as will be the case in most Mac App Store apps--you can't script Finder. Of course you can include a sandbox exception to allow for this, but the only way to do this that's allowed in the App Store requires 10.8.0 to run and, IIRC, 10.8.2 to run without serious bugs. And the Python docs would have to explain that any app that may call shutil.trash must use a custom sandbox. 3. If you're running in a sandbox, you can't execute osascript; if you need pre-10.8 compat, you can't use the modern APIs; this means the only thing you can do is include a compiled AppleScript as a resource and use the 10.7 APIs to execute it--which still requires 10.7, and it seems to have problems with the new 10.9.5/10.10.1 code signing rules (although it may be easy to work around that; I haven't tried). 4. If you're using a third-party Finder replacement like PathFinder, it won't respond to messages sent to Finder. 5. If you've killed the Finder to save resources (pretty rare nowadays, but still legal) you'll relaunch it. 6. The Finder may pop up a progress bar--or, worse, an auth dialog, or a "delete immediately?" dialog, on behalf of your app. This could be surprising from a console program. 7. Depending on how you do the scripting, your request may time out after 5 or 30 seconds, leaving you no way to know whether the trashing succeeded. 8. If Finder is hidden, it may unhide. If another app is fullscreen in front of Finder, it may slide offscreen, or Finder's dialogs may just be invisible. (You can guarantee a fix for the latter by forcing the former.) 9. If your session or system is screwed up and you're trying to shut down cleanly or repair things, Finder could be unresponsive or even hung. For example, on 10.6, if you have for separate moves blocked on an NFS mount, a trash will block forever, but (assuming you aren't trying to trash something on the same mount) there's nothing stopping you from trashing on your own. In some cases, this may all be fine. For example, if you're writing a trash script that you intend to use on your specific computer in local Terminal sessions only, you may _want_ the Finder to take care of auth, no-trash mounts, and other issues for you, and show you progress on long tasks. Fortunately, you can already write that script with about 5 lines of ScriptingBridge code that will run on the Python 2.7 with PyObjC and other extras that Apple has pre-installed as /usr/bin/python on all 10.6-10.10 versions, so you're already covered.
http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-...
Even if you want to use osascript, you don't want to do it that way. Build a single AppleScript command that trashes all of your files in a single call. One of the other blog posts someone linked earlier in this thread shows how to do that. Also, for the usual reasons, os.system is not the best way to call osascript (you'll end up with a success retcode even on failure, quoting for both the shell and AppleScript is double the fun, etc.).
Since this doesn't satisfy the rules of the App Store, it doesn't solve the problem it's intended to. But see my previous email. For almost any app, there is a much better alternative. The problem is that there's no _single_ alternative that's good for most apps; there are different ones for different use cases. In particular, for just writing a little trash script for use on the command line on your own 10.8+ computer, the NSFileManager version is even simpler than the scripting version (again using Apple's pre-installed 2.7 with PyObjC), and without any of its problems. For older computers, it's a _bit_ more work with the FS function, but still very simple, and it's a known problem that multiple people have solved years ago (and linked to earlier in this thread), so Python 3.5 doesn't need to retroactively solve it.

Hi everyone, I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.) As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it. (And of course, Python has lots of OS-specific modules.) So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this: if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file) What do you think? Thanks, Ram. On Thu, Jan 1, 2015 at 1:14 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Jan 1, 2015 at 9:37 PM, Ram Rachum <ram@rachum.com> wrote:
Most OS-specific stuff is in the os module, but path handling is inherently platform-specific to some extent, so either place would be reasonable.
Or: try: delete = os.recycle except AttributeError: delete = os.remove delete(my_file) ChrisA

On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote:
The os module is for low-level operating-system functions. "Send to trash" is neither low-level nor part of the OS per se, it is part of the desktop environment. I'm not convinced that this needs to be in the standard library, but if it is, I think that the os module is completely the wrong place for it. I think, in order of preference: 1) shutil 2) pathlib is the right place. (The actual implementation for the move_to_trash function could come from another, platform-specific, module.)
"Recycle" is not a good name for the function, because it doesn't recycle the file. It does the opposite of recycle: it prevents the file from being deleted and over-written. I think an explicit name like move_to_trash is better than something misleading like "recycle". No need for a special flag to check if the function exists, just check for the function: try: f = shutil.send_to_trash except AttributeError: f = os.remove f(my_file) But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution? -- Steven

I take your point about it not being in the `os` module, I have no issue with it being in shutil or pathlib. I also agree about the name, "recycle" doesn't really make sense. Regarding your question: "But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?" Well, I can use a third-party solution for everything, but it would be nicer for me if it was in the standard library because then I could count on it always being there and the API not changing. I could say the same thing you said about the `webbrowser` module, that opening a new browser tab is quite a specialist need, but I do enjoy having it available in the standard library and I feel the same about sending a file to trash, which I think is a basic feature that's we've all been taking for granted for the last decade or two. Happy new year, Ram. On Thu, Jan 1, 2015 at 2:35 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Jan 01, 2015 at 02:43:09PM +0200, Ram Rachum wrote:
Which is exactly the point. It has been said that the standard library is where good modules go to die. Do you think that the "move to trash" API is ready to die? I don't think so. I think that there is a lot more than just moving files to the trash. I've spent a little bit of time over the last day or so thinking about what sort of trash-related functionality I would like, as a Linux user: - move to trash, of course, but which trash? - return the name of the trash directory in my home directory - list trash directories on other partitions and disks - list trash directories for all users (if I have read permission) - empty one or more of those trash directories - optionally only deleting sufficiently old and/or large files - validate and/or repair the trash metadata - report how much space is used by the trash, with or without metadata - support legacy .Trash directories as well as .local/share/Trash For some operations, I would want to specify either a specific trash directory, or all trash directories for a given user (defaulting to the current user), or all trash directories for all users. (The later would only work if I was running as the root/Administrator user, of course.) The point being that the API for all this functionality isn't obvious, and it is very unlikely to be so stable that it is ready to move into the standard library. -- Steven

On Thu, Jan 01, 2015 at 11:35:26PM +1100, Steven D'Aprano wrote:
Maybe the right way to do this is to create a new module, stdlib or not, for desktop-related APIs. Besides a `recycle` function this might, for example, include a subset of pyxdg's functionality. -- Markus

On Jan 1, 2015, at 13:47, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
That's a great idea. To do something as simple as getting special directory names or creating a shortcut/alias/bookmark (as opposed to a symlink) or, yes, trashing a file in a cross-platform way requires something way too heavy-duty (like Qt or JUCE)... or copying code off some blog (that doesn't actually work on localized Windows or in a sandboxed OS X app or on non-Debianish Linux, but you won't know that until you think to test it). A third-party module can get away with 90% support. Most Mac users are on 10.8+, most apps don't fork, most Linux users who care about the desktop have a GNOME-compatible one; etc.

On Thu, Jan 1, 2015, at 07:35, Steven D'Aprano wrote:
What is the correct place for high-level functions that need to be implemented in a different way on each OS? I think the problem is that some people view os as "low-level functions, like the ones found in unistd.h" and other people view it as "where to put stuff that needs a separate implementation per platform". The actual description in the documentation is "This module provides a portable way of using operating system dependent functionality." Nothing about low-level.

On Jan 1, 2015, at 11:37, Ram Rachum <ram@rachum.com> wrote:
Hi everyone,
I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.)
On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional. Really, on all three platforms, the answer is the same: you call an API from your desktop manager. The only difference is that Windows merges desktop and kernel together to the point where they can't be separated--so Python is already using Win32, unlike Cocoa or GObject.
As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it.
No one is "depriving" Windows users of anything. Python comes with pip, send2trash installs without needing a compiler (at least I hope so; if not, someone can write a new third-party module that does). Not everything needs to be in the stdlib. So far, there aren't many third-party modules mentioned in the stdlib docs, but that's only because we didn't have preinstalled pip, wheels, etc. until 3.4.
There's no such flag for anything else platform-contingent. EAFP.

On Thu, Jan 01, 2015 at 04:37:52PM +0100, Andrew Barnert wrote:
Actually it seems that the trash location is standardized by the XDG Desktop Specification: https://github.com/kevva/xdg-trash http://www.ramendik.ru/docs/trashspec.html

On Jan 1, 2015, at 17:00, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
Yes, as I just said: Linux has no notion of trash; desktop environments do. The FreeDesktop standard is not Linux; it's a standard that at least two desktop environments (which also work on multiple non-Linux systems) support. On most Linux systems, you will have Gtk libraries; on most Linux systems without Gtk libraries, you will not have a FreeDesktop-compliant trash. So, using GObject is a good 90% solution (like using NSFileManager is a good 90% solution on Mac). Of course you _could_ implement that full spec yourself in Python, right? Well, read it. There are references to things "the implementation" may do, like providing a way for the administrator to disable sticky-bit checking for filesystems that don't support it. Do you know how to check whether the administrator has disabled sticky-bit checking on a filesystem? Or do you just want your code to say a file can't be trashed, or trash by moving it across the network to ~, even when the desktop and other apps have no problem trashing it properly? For that matter, do you know how every FreeDesktop-compatible system wants to report a problem to the administrator? You may meet the XDG Trash spec and still not satisfy user expectations. And it'll be a lot of work. In other words, "you have to implement a complicated approximation based on rename that won't be fully functional". Is that really a better solution than using GObject? If you're writing your own portability library like Qt or JUCE, probably; otherwise, I doubt it.

On Thu, Jan 1, 2015, at 17:59, Andrew Barnert wrote:
A python implementation would be an independent implementation, and not bound by another implementation's (even on the same machine) implementation-specific decisions. There is therefore no reason the python implementation couldn't simply not provide such a method, and therefore any .Trash directory without the sticky bit set does not pass the checks, and all that will happen is .Trash-$uid gets used instead - no harm done.

On Jan 2, 2015, at 14:43, random832@fastmail.us wrote:
Then a Python implementation would be a bad implementation. The whole point of XDG is to make it easier to write desktop software that's consistent with other desktop software. So if you write some Qt software and I run it on a GNOME system, Qt follows (as best it can--which is better each version) the GNOME rules, and your GNOME-based settings, rather than ignoring them and pretending it's on a KDE system.
Except that if you can't create a .Trash-$uid, then you can't trash files on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.)

On 3 Jan 2015 06:06, "Andrew Barnert" <abarnert@yahoo.com.dmarc.invalid> wrote:
Except that if you can't create a .Trash-$uid, then you can't trash files
on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.) It sounds to me that what is needed before changes to pathlib can be discussed is a reference third party module that appropriately respects desktop conventions (whether Windows, Mac OS X, GNOME, KDE, or other XDG based Linux window manager), and associated clear documentation on how to delete files in a way that integrates well with underlying platforms. The standard library isn't the place to work through that standardisation activity, although a PEP is certainly a reasonable place to track it. Guido's work on PEP 3156 and the asnycio module is a good point of reference here, as is Antoine's work on pathlib itself. Once a pathlib independent reference implementation of imperative desktop trash management exists, and there is broad acknowledgement that it is a suitable candidate for inclusion in the standard library (perhaps as part of shutil, perhaps as a new independent module, depending on API complexity), *then* it becomes feasible to discuss adding trash management support to the pathlib abstraction layer. Regards, Nick.

On Thu, Jan 1, 2015, at 10:37, Andrew Barnert wrote:
There is an actual standard for the Trash system that all three major desktop environments implement, so your "complicated approximation based on rename" can actually be correct rather than an approximation. If that's not exactly like how GNOME does it, then GNOME is incorrect.

On Jan 2, 2015, at 14:36, random832@fastmail.us wrote:
First, XDG specifications are explicitly not standards (and XDG Trash is a draft extension proposal to those specs) so there is not an actual standard. But, far more importantly, there are large areas that the spec leaves open to the implementation, or open for future expansion, that GNOME (and KDE) fills in with appropriate behavior. I gave one example earlier in the thread, which I picked by pointing my finger at one random paragraph from the spec. If you read through it, there are plenty of others. If you call the APIs via GObject or Qt on a GNOME or KDE system, you will get the exact same behavior as everything else on the system. If you implement it yourself and ignore what GNOME and KDE do, nobody's going to care that you're 100% compliant with a draft spec proposal, they're going to care that your app refuses to trash something that everything else can trash (or, worse, that your app trashes it by copying 8GB over the network).

There’s always the Cocoa APIs for moving files to the Trash, if its fine to depend on OSX 10.6 or later (as you mentioned below). Whether or not you’d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea. BTW. I’m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we’ve had with proxy support in the _scproxy extension (basically resulting in “random” crashes when using urllib in a child proces started with os.fork because Apple’s system frameworks don’t support that). Ronald

On Dec 31, 2014, at 9:20, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Yes, but the 10.6 Cocoa APIs both have problems; the useful one requires 10.8. So as long as Python standard builds are 10.6… (I suppose you could have it in the stdlib but only available at runtime on 10.8+, but I think that would be weird.)
Whether or not you’d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea.
Yes, and therefore there just _is_ no low-level, suitable-for-os, function that would help for a higher-level shutil/pathlib function.
BTW. I’m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we’ve had with proxy support in the _scproxy extension (basically resulting in “random” crashes when using urllib in a child proces started with os.fork because Apple’s system frameworks don’t support that).
I forgot about that, but you're right, that's another potential reason not to put this in the stdlib. Anyway, maybe what we need is something in the docs explaining the various ways to do it (with links to PyPI projects as appropriate) with the pros and cons of each. What I wrote before isn't right, because it doesn't take into account the forking issue. It could be easily extended, although that might get a bit long for stdlib documentation.

On Tue, Dec 30, 2014, at 18:12, Andrew Barnert wrote:
Not having one isn't acceptable either. Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind by applications with badly-written direct implementations of the way you do it on Unix. Can you define "doesn't quite meet Apple's HIG" rules and link the relevant section of the HIG?

On Jan 2, 2015, at 14:24, random832@fastmail.us wrote:
I'm not sure what "the way you do it on Unix" means. There is no Unix/POSIX/etc. notion of trashing a file. Of course some Unix desktop environments (going back at least as far as NeXTStep) have had such a thing, but that can't be what you mean. If you're referring to XDG, I don't know of a single application that uses a badly-written direct implementation of the XDG trash spec on Mac or Windows. Do you? More importantly, which of those other options do you think is better than continuing to have no trash API? Drop 10.6-10.7 compat? Use a deprecated API and break Mac App Store compatibility? Require all scripts to have a runloop?
Can you define "doesn't quite meet Apple's HIG" rules and link the relevant section of the HIG?
Of course not, because I have no idea what exactly you'd write in place of calling the appropriate API. Give me a design or some code, and I'll tell you what it does wrong.

On Fri, Jan 02, 2015 at 08:24:26AM -0500, random832@fastmail.us wrote:
Not having one [move to trash function] isn't acceptable either.
I'm not sure about that. Do other languages offer a move-to-trash functionality? Say, PHP, Javascript, Java, Ruby, Boo, Go, Lua? If Python is the odd one out that DOESN'T offer such functionality, and all the others do, then you might have a strong case. Otherwise, consider that the great bulk of scripts that are written in Python don't do any file management at all, other than perhaps reading or writing a few files. So even if Python gained this functionality, it wouldn't have any impact on most scripts and applications.[1]
Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind
Really? I would expect that Windows is probably the *easiest* platform to implement this functionality, as it has a single, standard, stable API for moving files to trash. (Or so I am lead to believe.) You can probably use that API via ctypes, or via pywin32. Getting OS X right is very complicated, as Andrew has so ably explained. In Linux and other Unixes, the situation is like OS X, only more so since there's no one single authority. You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support. It seems to me that "move to trash" is at least straight forward on Windows, but tricky and complicated everywhere else. Or, there is always the option of using a third-party module: https://pypi.python.org/pypi/Send2Trash [1] With one possible exception. If Andrew is correct, then the wrong implementation of move-to-trash might effect in a bad way anyone trying to put their app on the Apple Store, whether they use that move-to-trash functionality or not. -- Steve

On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I've always thought of a trash can as an inherently human-controlled feature. When you go in using a GUI file manager and delete a file, it lands in trash. When you use another application, it's always felt surprising - for instance, BitTorrent on Windows deletes things by moving them to trash. (Which has abysmal performance if you select a huge bunch of things and wipe them all out at once; trashing gobs of stuff across thousands of files seems to trigger an O(N*N) repeated search for which files "fall out" of trash - not to mention that this probably means a fairly arbitrary subset of the freshly-deleted is in trash, and little or nothing from previous deletions remains.) Most apps should do file management at the file system level, unless they're specifically replicating/replacing the file manager. So a "move to trash" feature is something that (a) isn't going to be needed by most apps, and especially (b) would be an attractive nuisance. A third-party module has that extra little hump of having to be consciously sought, so it can be there for those who want it but not waving itself around at those who really should be looking at os.remove(). I'm -0.5 on adding a "move to trash" function to the stdlib, ever, and a definite -1 on adding one now. ChrisA

On 03/01/2015 02:18, Chris Angelico wrote:
As, for example, holding down the SHIFT key on Windows and skipping the recycle bin completely? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Jan 3, 2015, at 3:18, Chris Angelico <rosuav@gmail.com> wrote:
I think this may be an accident of the fact that cmd.exe didn't include a trash command, neither did OS X, and the idea was dropped from xdg-trash; we've all been trained for a couple decades now that you can't trash from the command line. I'm not sure that's a good thing. But it is nevertheless a thing that can't be ignored.
In some GUI applications, it does make sense to trash things via a menu or other user interaction. (Except maybe when you're using Apple's versions/history thingy or some similar mechanism.) One of the things Aquamacs adds to the standard emacs distribution is a move-to-trash command that you can use in diredit mode, for example. Although I guess you could argue that counts as "unless you're replacing the file manager", and maybe the same is true of most other such examples... But not all. When I worked on TuneUp, we had a feature to move selected duplicate tracks from quarantine to trash, which I thought was stupid and unnecessary (the quarantine folder is already effectively an app-level trash can...). But when we removed it, we got hundreds of user complaints, and a nasty blog post saying we were trying to deliberately destroy out users' music collections on behalf of the RIAA or something. And:
The Mac version also does the move-to-trash; it has perfectly acceptable performance. (Actually, old versions were terribly slow at trashing a torrent with a lot of top-level files or dirs, but newer versions seem to do a single operation per torrent instead, which works fine.) And I'm actually glad it does this. At least twice, I've downloaded an ISO, mounted it, then trashed the torrent with all files; if they were deleted instead of trashed, I would have had to repeat the download to update my VMs... You could argue that I'm an idiot for doing this, and you'd be right, but I'm probably not the only such idiot; in fact, that's kind of what trash is for in the first place.
I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. Whether that's send2trash or a complete trash management module as Steven (I think?) envisioned or a reference to the platform APIs and the various ways to get at them, it would be useful. (In fact, some prominent reference to PyWin32, PyObjC, GObject, and PyQt/PySide for non-GUI-but-still-desktop apps in the docs might be helpful for lots of reasons.) Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem.

On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert@yahoo.com> wrote:
The Mac version also does the move-to-trash; it has perfectly acceptable performance.
Good. The note about performance was a mere parenthesis, and the last time I did anything like that on Windows was on XP, so it's quite possibly different on Win7/8.
I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib.
Maybe. Is there a mention of other third-party packages in stdlib docs? It's a slippery slope; which ones deserve mention? Maybe just something like "Note that moving objects to the trash can is possible in OS-dependent ways; packages are available on PyPI to make this easier", without naming any.
Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem.
As an explicit "trash" tool, sure. Typing "trash some_file" is under human control just as much as hitting Delete without holding Shift is. I just don't think it should be the obvious way for an application to remove files. ChrisA

On Jan 3, 2015, at 11:39, Chris Angelico <rosuav@gmail.com> wrote:
The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths.

On 3 January 2015 at 21:04, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
Your parenthetical qualification is correct - while I personally think providing authoritative recommendations of third party modules is a good idea, this is far from a universally held opinion amongst the core development team. My own caveat on the idea is that any such explicit recommendations should only cover projects that potentially *could* be included in the standard library from an API design perspective, but we don't want to include them (yet, or perhaps ever) for project governance reasons, such as their update cycles for feature releases needing to be faster than those of the standard library. The "pro" case for this approach is primarily about lowering barriers to entry: for folks just learning Python, "not in the standard library" is currently functionally equivalent in many cases to "Python can't do this". New users often won't yet have the skills needed to do their own research and evaluate competing solutions to their problem, and teaching someone how to find a module on the internet, evaluate the trustworthiness of the people publishing it, get it onto their computer, and update it appropriately over time is genuinely difficult (even with pip provided by default - that helps with the last two steps, but the first two are still genuinely difficult, in a way experienced open source developers often take for granted). Providing authoritative recommendations of third party projects in the standard library documentation helps resolve that situation without needing to resort to making the standard library larger (with the additional long term maintenance complications the latter approach entails). You see a similar dynamic operating in a slightly different way with a project like Django - when the Django developers are significantly more experienced in web development than you are, their opinionated, all-inclusive approach to web framework design is a *feature*, not a defect. It's only when you become more experienced with web service development yourself, and need to deal with problems that don't fit neatly into the SQL-backed three-tier web application model, or need to integrate with other systems which weren't built with Django in mind, that less opinionated frameworks like Flask or Pyramid become interesting - those frameworks leave many more decisions to the developers using them, which is exactly what you want in many situations as an experienced developer, but constantly asking people to make decisions they aren't yet qualified to make poses a huge barrier to entry. It's a potential problem when less experienced folks are put off by the request to make those decisions (since not understanding the questions you're being asked by your tools makes for a terrible user experience), and decide to go learn something else that places fewer demands on them, but it's even worse for everyone trying to deploy and manage the resulting software systems when folks happily make the decisions these advanced DIY integration frameworks ask of them without even realising they're not yet qualified to make them. djangopackages.com is also incredibly useful as a tool for finding peer evaluations of projects in the Django ecosystem. I'm not yet aware of any successful attempts to expand that model to other plugin ecosystems, but the core grid concept of assembling a set of criteria, providing answers across a set of related projects, and publishing the results to provide a starting point for anyone doing their own research in that topic area is a wonderful one. The "con" case is that providing default recommendations can be controversial - when there are multiple competing projects in an area, being the default recommendation of the CPython core development team can be a huge popularity boost, since reputation is to some degree transitive. If someone trusts CPython, and we trust another project enough to recommend it, then a lot of folks will use that recommendation as a substitute for doing their own research (even when they *can* do that research themselves, the default recommendation is likely to be good enough, letting them defer investigation of alternatives until after they've tried the default option). I suspect that objection could potentially be dealt with the same way we deal with any other standard library design decisions: if a module maintainer is comfortable making a recommendation based on their own expertise they can, but broader recommendations spanning several modules would require at least a discussion on python-dev, and potentially even a PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jan 4, 2015, at 7:05, Nick Coghlan <ncoghlan@gmail.com> wrote:
The question here, then, is how this applies to desktop/shell functionality like moving to trash, administering trash in all the ways someone (Steven?) wanted, other features that someone else raised (aliases/bookmarks/shortcuts/desktop files), and maybe additional stuff (accessing the desktop's extension/file-type mapping). Some of that stuff is already in the stdlib (or might be confusing to novices--e.g., IIRC, on XDG desktops, some things that are done with aliases on Mac are done with desktop files, but in some cases you just use symlinks, so a Linux novice might not even realize he's not writing cross-platform code...). For a GUI app, it's not at all clear that choosing Qt or Wx or one of the platform-specific libraries automatically solves all of these problems for you, while choosing Tk doesn't. For a command-line app, it's even worse. Most Windows users probably have no idea they're asking for desktop integration rather than OS support when they ask how to move a file to the trash or create a shortcut, because Windows intentionally buries that distinction. So when they see that neither os nor shutil nor anything else in the stdlib can help them, they're not going to think, "Oh, I need to go find a platform-specific or cross-platform desktop integration library." (And they're also not going to think "Well, Perl and Ruby and Node don't have these either, so why should I expect Python to?") So, I think in this case it would be useful to recommend something. The question is _what_ to recommend (and where). I think for platform-specific code there are pretty clear winners (PyWin32, PyObjC, GObject, and PyQt/PySide, for the big four). Some might argue for including the XDG package (e.g., if you want to do the same thing as one of the XDG-utils tools, you don't need Gtk or Qt for that), and Mac users might also want a link to the ScriptingBridge guide or something (because some things that are done by API on Windows are done by scripting System Events or Finder or something else), and people using a third-tier desktop on Linux may not have Gtk or Qt libraries to bind to, etc., but I think those are minor questions compared to whether it would be useful to have pointers to these packages for novices. (I think it goes without saying that none of these packages should be added to the stdlib, right?) But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither. And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong.

On 4 January 2015 at 20:32, Andrew Barnert <abarnert@yahoo.com> wrote:
Right, a useful first step might be a "Tools for writing rich client applications in Python" overview page. That's out of scope for any of the current *.python.org documentation set (it doesn't fit under docs.python.org *or* packaging.python.org, nor anywhere on the main site), but if it's written in an appropriate tone (i.e. making it clear that these are default starting points, and recommendations for further evaluation, rather than universally applicable definitive conclusions), then a future "ecosystem.python.org" guide could start the same way p.p.o did, as a separate Sphinx project hosted on ReadTheDocs that was later blessed with a python.org subdomain. https://packaging.python.org/en/latest/deployment.html is an example skeleton of such a page for package deployment tools. We haven't filled that one *in* yet, but hopefully it conveys the general idea. https://packaging.python.org/en/latest/extensions.html is another one reviewing binary extensions, and some of the tools available in that space. For those curious as to "Why not the wiki?", a Sphinx project hosted on a pull request capable service like GitHub, BitBucket or Kallithea offers a much nicer workflow for reviewing of proposed changes, together with an integrated issue tracker for submitting proposals for updates (https://github.com/pypa/python-packaging-user-guide/ is the project behind packaging.python.org, for example).
But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither.
Just describing the problem space can be incredibly valuable for newcomers - one of the hardest things to learn as a beginner is the difference between "this is hard because I don't know what I'm doing" and "this is hard because I'm working on an unsolved problem" (with "this is deceptively easy because I'm setting myself up for problems later on" being an interesting variant that's hard to detect no matter how experienced you are). I see this is a *lot* when working with the scientific community - they actually have much higher expectations of cross-platform tooling than professional developers do. From the outside, the political, personal and commercial meta-issues that combine to make cross-platform standards development an unholy nightmare aren't immediately obvious, and nor are the trade-offs between "lowest common denominator" (or Java-style "reinvent all the wheels") functionality that works everywhere, and providing access to advanced platform specific functionality.
And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong.
While I think it's better to leave the question open rather than try to pre-empt an answer, I do believe we can at least say *now* isn't the right time to be adding "move to trash" functionality to the standard library - the functionality isn't mature enough within the broader ecosystem to determine an appropriate answer. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote:
The concept of "proposed changes" goes completely against the grain of community-managed content. Imagine if Wikipedia required you to make pull requests. The question to be asked is not whether "pull requests" are "nicer" than a wiki, but whether this information needs to be owned by a single person (or a small group thereof), and hence use the packaging.python.org module, or owned by the whole community, and hence use the wiki model. At the moment, the wiki is horribly underused. I'm not blaming anyone for that, I'm just as much to blame as anyone else. I have made far more contributions to the ActiveState recipes than I have to the Python wiki. But I'd really like to see the wiki become the second place people look for information, after the official docs. And hopefully there won't be a need for a third place :-) Although the quality of contrabutions is variable, the PHP community has made a very interesting decision to integrate community-supplied information with official docs: http://php.net/manual/en/function.dechex.php I wonder how we might do something similar? -- Steve

On the other hand, a number of pages on the Wiki are either completely out of date because nobody cares about them or need to be locked because too many people care about it and they fight over which project should be named as recommended. Wiki content can be good for more factual based information but I think the format is relatively poor for things which are more opinion based. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Donald Stufft <donald@stufft.io> writes:
In recent years (I noticed this since about 2013), the Python wiki does not offer an “Edit” function when I am logged in; every page presents “Immutable Page” instead. That has prevented me from improving pages I've wanted to. And no, I haven't gone to the effort of constructing a bug report for this. Perhaps the same is true for many other logged-in users as well? -- \ “It is well to remember that the entire universe, with one | `\ trifling exception, is composed of others.” —John Andrew Holmes | _o__) | Ben Finney

On Sun, Jan 4, 2015 at 7:22 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Ben (and others), I just noticed this small paragraph on the FrontPage: If you want to edit a page and have just signed up, or you find that you can no longer edit a page that you could edit before, please write to the pydotorg-www mailing list, stating your account name and your intended edits and we'll add you to the EditorsGroup. I suspect that has something to do with your inability to edit. Somewhere along the way, someone must have made the executive decision that open editing no longer worked (too much spam?) and that editing needed to be a by-invitation or by-request sort of thing. I'd add you, but am getting nothing but Gateway Timeout responses at the moment. Skip

On 05.01.2015 15:49, Skip Montanaro wrote:
This was discussed and announced on the pydotorg-www list at the time. The step was necessary, because of excessive spam and vandalim - editors were spending more time cleaning up than editing pages and were losing interest in the wiki altogether because of this. The first step was to require logins for editing. This worked for a (short) while. The second was adding more complicated textchas, the third step was requiring asking for editing rights on the pydotorg-www mailing list. Which is a little annoying and some work for the admins, but has resulted in the amount of spam to go down to zero. We usually grant the editing requests within a day. FWIW: I've maintained the wiki VM ever since it came under attack in 2013.
I'd add you, but am getting nothing but Gateway Timeout responses at the moment.
If you get those, simply try to resend/reload. The wiki will notice duplicate editing requests. In my experience, the editing requests typically make it through to the wiki the first time and you get a duplicate editing warning on the second one. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Mon, Jan 5, 2015 at 9:27 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Unfortunately, I couldn't even get to the EditorsGroup page to edit it. Now, about an hour later, I finally succeeded. Ben, send me your wiki login and I'll add you. MAL, I thought the spam problem had mostly been solved by Martin's modification several years ago to require a certain time to pass between a new login being created and their first attempt to edit. Skip

On 05.01.2015 16:42, Skip Montanaro wrote:
Hmm, looks like the VM is overloaded again :-(
I'm not aware of such a modification in the installation. In any case, the problem was real and the editors group solved it for good now :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

"M.-A. Lemburg" <mal@egenix.com> writes:
I quite understand the motivation for this. It does need to be recognised that this will dramatically reduce the number of pople who will pass that hurdle to edit the wiki, and thus the wiki loses a lot of edits (both helpful and unhelpful) it might otherwise have. Thanks to those doing the work to maintain Python community infrastructure. -- \ “If you can do no good, at least do no harm.” —_Slapstick_, | `\ Kurt Vonnegut | _o__) | Ben Finney

On 01/05/2015 10:26 PM, Ben Finney wrote:
"M.-A. Lemburg" writes:
At least the edits being made now are by folks who care. Better fewer edits than a site full of little but spam.
Thanks to those doing the work to maintain Python community infrastructure.
Seconded. -- ~Ethan~

On 06.01.2015 07:26, Ben Finney wrote:
We might lose some occasional quick typo fixes, but it seems that those genuinely interested in helping with the wiki don't have a problem asking on the list. To reduce the problem I had created a list of user names who had made good edits in the months before the change. The only nit I still have with the setup is that it's not necessarily apparent for new users how to get editing rights. There's a section on the front page, but you don't see this when looking at a random page.
Thanks to those doing the work to maintain Python community infrastructure.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Fair disclosure: I'm not a big fan of Moin syntax (can Pandoc convert this?). I don't like having to login to share with the Python community. It would be great to gain maximum synergy from the community on this. Some interesting perspectives on wikis, and where wikis are going: [1] https://github.com/blog/1939-how-github-uses-github-to-document-github [2] https://github.com/westurner/wiki/blob/master/Makefile [2] is a Sphinx Makefile for a GitHub ReStructuredText wiki. It seems to work okay for my purposes. What could prevent this from scaling to handle the Python wiki?
From https://github.com/westurner/wiki/issues/5 , in Markdown (which works on both BitBucket and GitHub):
# Advantages
* Beautiful Sphinx output:
On Tue, Jan 6, 2015 at 2:31 AM, M.-A. Lemburg <mal@egenix.com> wrote:

On Tue, Jan 6, 2015 at 1:25 PM, Wes Turner <wes.turner@gmail.com> wrote:
I'm not a big fan of Moin syntax (can Pandoc convert this?).
I haven't followed wiki markup at all in the past several years. Back when Moin came along, it was (arguably?) the correct choice for wiki.python.org. I think most of us active at the time were very much in the "eat your own dog food" camp at the time. I've never heard of Pandoc (I grow old, can't follow every trend), but I thought there were some tools which could convert Moin into other formats. I can't find much now, though this https://ikiwiki.info/tips/convert_moinmoin_to_ikiwiki/ looks like it might be a reasonable starting point. Skip

If you ask me the bigger problem with the Wiki is not how to encourage or manage small edits, it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant. (A random example: https://wiki.python.org/moin/MostPopularPythonProjects is 10 years old, which makes it actively harmful given its lofty title and linkage from other pages.) An early Python contributor, Ken Manheimer, described the necessary activity as "wiki gardening". We need more wiki gardeners, people who have an eye for the big picture and actively edit content, not just guards who passively judge contribute proposed changes (even that's also needed -- Python is sufficiently well-known that the wiki would attract a lot of spam and occasional vandalism if it was completely open). -- --Guido van Rossum (python.org/~guido)

On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido@python.org> wrote:
it is how to deal with the entire wiki gradually getting out of date due to page "owners" losing interest or topics becoming irrelevant.
Even if you can recruit lots of gardeners, you need a few master gardeners to lay things out (define the structure of the garden). Otherwise you wind up with just a bag of pages. Pushing the gardening metaphor to its limit: no planning means the lettuce is always shaded by the corn. Skip

On Tue, Jan 6, 2015 at 11:39 AM, Skip Montanaro <skip.montanaro@gmail.com> wrote:
Nice one, and agreed. I don't see anyone with a serious wish to be a master gardener for wiki.python.org in this sense though. :-( Perhaps we should advertise the position? It's a volunteer role, but will require a lot of motivation. Ideally the master gardener team should be allowed to select the tool suite and be given permission to switch to a new suite pretty aggressively. The team should also be responsible for deciding on the policy for edit access. This seems more workable than having an open-ended discussion on python-ideas. -- --Guido van Rossum (python.org/~guido)

I have very little opinion on what the wiki is or does but I wanted to just say if at some point we do switch software then with my infrastructure team hat on I would ask for two properties to exist in the software (not really specific to the wiki tbh): 1. Configuration that is able to be handled via config management like salt/chef/puppet etc. Typically this just means that the configuration is handled via a text file and the software itself doesn’t attempt to write to the configuration as part of it’s normal execution. 2. State does not need to be stored on the local file system and instead can be stored in something like a database (preferably PostgreSQL) or an object store (preferably cloudfiles). This allows us to treat the servers running the software as emphereal with all of the state stored elsewhere. This makes it easier to manage the servers, makes it easier to recover from security issues on a server, makes it easier to upgrade the server, and makes it easier to scale the service in general. Of the two, the first property is the most important, but the second one is a really good idea too. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Tue, Jan 6, 2015 at 2:24 PM, Donald Stufft <donald@stufft.io> wrote:
+1 * http://psf-salt.readthedocs.org/en/latest/ * https://github.com/python/psf-salt
re: persistence * difflib * hg diff * git diff structured data * http://www.w3.org/2001/sw/wiki/Tools (RDF; FreeBase is moving to WikiData) * https://westurner.github.io/wiki/tools * https://westurner.github.io/dotfiles/tools.html#python * https://westurner.github.io/dotfiles/_sources/tools.txt (this is the simplest ReST markup I could think of, a sphinxcontrib-rdf directive/role could be really useful, but wouldn't work w/ GitHub wikis without a Sphinx compile step) : .. index:: Python
.. _python:
Python
Python is a dynamically-typed, :ref:`C`-based third-generation
programming language.
[...]

On 06.01.2015 20:44, Guido van Rossum wrote:
Such discussions should really happen on pydotorg-www where that team already works. We do have several people who maintain pages or page sets on the wiki, but more editorial help is always welcome. We need people who have experience in technical writing and a passion to maintain informational resources. FWIW: A change of tools won't magically give us better content. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, Jan 6, 2015 at 4:01 PM, M.-A. Lemburg <mal@egenix.com> wrote:
https://mail.python.org/mailman/listinfo/pydotorg-www https://www.python.org/dev/pydotorg/website/
A boost in familiar usability could encourage effortless contributions.

Ethan Furman writes:
On 01/06/2015 02:11 PM, Wes Turner wrote:
A boost in familiar usability could encourage effortless contributions.
Not even breathing is effortless.
True, but that's the wrong fallacy. The energy required is not the point, it's the thinking. Wes's point is that he'd like it to be possible to contribute to Python without being aware of the mechanics of contribution, just as it's possible to breath without being aware of your lungs most of the time. That's reasonable, and wikis actually do work in some contexts. The fallacy in applying wiki-think to Python is that for new and small projects with SEI "Level Zero" processes, *increase in quantity* of documentation is *improvement in quality* of documentation. That's no longer true of Python, and hasn't been for two decades (at least). It isn't even clear that filling an obvious hole with an accurate-as- far-as-it-goes drive-by contribution is a good thing. Not only may the residual inaccuracy mislead the reader, it may also contribute to an impression that the Python docs are lower quality on average than they actually are.

On 06.01.2015 23:11, Wes Turner wrote:
Yep, that's the list.
That page explains the old python.org website system. It doesn't have anything to do with wiki.python.org.
The wiki does support ReST syntax if you don't like the wiki markup: http://moinmo.in/HelpOnParsers/ReStructuredText and it also supports WYSIWYG editing using FCKeditor (even though the generated markup code doesn't look all that nice). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 06 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

While we're in wishlist mode: A contribution we'd really appreciate is a moin 1.9 theme adapted to the new python.org style :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, Jan 6, 2015 at 5:04 PM, M.-A. Lemburg <mal@egenix.com> wrote:
My mistake. That page does seem outdated. I must've been looking for one level up: https://www.python.org/dev/pydotorg/ What content should be version controlled with the website?
Ah, thanks!

On 07.01.2015 00:21, Wes Turner wrote:
Sorry to say, but that content is outdated as well :-(
What content should be version controlled with the website?
The current python.org website is a mix of code and templates in github repo, content/data stored in a database and a (for various reasons) homegrown CMS to manage pages such as the ones you found. There's work underway to replace the homegrown CMS with a standard more feature rich one, which will hopefully allow us to open up site editing to way more volunteers than we currently have. wiki.python.org is a separate VM, which doesn't have anything to do with www.python.org. The VM runs a more or less standard moin installation, which uses its own version control system (you can access this via the "info" link on the pages).
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, Jan 7, 2015 at 2:49 AM, M.-A. Lemburg <mal@egenix.com> wrote:
So there are no topical guidelines?
https://github.com/dwf/moin2rst Formatter plugin
Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's ``plugin/formatter`` directory.
Action plugin
Simply put ``RenderAsRestructuredtext.py`` to MoinMoin's ``plugin/action`` directory.

I'm going to add one more bit here about wiki gardening. BITD, the main website had many pages which today would be called listicles. The problem was that those lists were dynamic. They weren't just the "top ten scientific Python modules of all time" or the "seven best packaging tools." They were whatever was at the front of the author's brain at some point in the past. Having these static pages given the imprimatur of the PSF when in fact none of the site maintainers were obviously responsible for (or interested in) keeping them up-to-date did a disservice to the community and to authors of other packages which weren't represented in those lists. In addition, there were more barriers to update than necessary (essentially, figure out how to report the problem, offer suggested fix(es), then have them swallowed up into the site update mechanism). One of the main uses envisioned for the wiki was as a place where these listicles could be maintained by the greater Python community. For some things, that's worked out pretty well. The most obvious thing that comes to mind is the PythonTraining page. That works because the people whose skills are represented on that page have a very good reason for keeping things up-to-date: it's free advertising for their businesses. Other listicle type pages haven't been keep as up-to-date. For example, the PythonEditors page was last updated in Feb 2014, is huge (and might benefit from being split into multiple pages), and probably no longer accurately represents the available editors or IDEs which support Python. The takeaway in my mind is that we could probably use "gardeners" to take over active maintenance of these listicle pages. That, coupled with a couple "master gardeners" to develop some suitable structure, and some landscape crews to prune dead/outdated/no-longer-useful pages, would likely go a long way to improving the quality of the wiki. Skip

On Wed, Jan 7, 2015 at 7:27 AM, Skip Montanaro <skip.montanaro@gmail.com> wrote:
I'm going to add one more bit here about wiki gardening.
In order to test parsing code in order to produce Linked Data http://5stardata.info , it could be helpful to produce a periodic archive/dump sort of like http://wiki.dbpedia.org/Datasets . A github repository could be useful for hosting these types of releases. python/wiki___ ?
Are the links out of date, or the descriptions and exclusive clusters? * http://schema.org/SoftwareApplication (RDFa in HTML5, Microdata) * http://json-ld.org/ * URIs are tags
Are there tools for working with a regular editor (such as vim) and MoinMoin pages? With ReStructuredText, the Riv.vim syntax helpers are great for things like tables; Voom is great for outlines (:Voom rest).

On Wed, Jan 7, 2015 at 10:23 AM, Wes Turner <mailto:wes.turner@gmail.com> wrote:
Are there tools for working with a regular editor (such as vim) and MoinMoin pages?
I'm not at all concerned with this, as I see the editing step being pretty much orthogonal to the markup language and the version control tools. Editing through a web browser pretty much sucks no matter what you're trying to edit, especially if you are used to using highly functional, mature editors like vim and Emacs. If you want your editing to happen outside a glorified <textarea> widget, find an "offline" editing plugin. (I happen to use Edit with Emacs from Chrome, but there are other extensions available.) I assume that MoinMoin will eventually be supplanted by some other (wiki?) technology. I do think it's worth noting though, that while Github and other version control services are all the rage for all sorts of collaborative editing, wikis in general predate that sort of technology by a long way. If I read correctly, the original C2 Wiki was launched in 1995, MoinMoin in 2000, Git in 2005, and Github not until 2007. See: * http://c2.com/cgi/wiki?WikiHistory * http://en.wikipedia.org/wiki/MoinMoin * http://en.wikipedia.org/wiki/Git_(software) * http://en.wikipedia.org/wiki/GitHub It's only more recently developed wiki systems (or perhaps a few wiki systems whose authors were prescient and saw DVCSs coming) which support generic version control backends. I think I posted a link yesterday which looked like a reasonable starting point for converting MoinMoin markup to Markdown. Run with that if you want. If you can parse MoinMoin markup and generate Markdown, it shouldn't be too difficult to iterate over all the pages and their iterations in wiki.python.org, convert to Markdown, then commit to a Github repo. I think MoinMoin has served the Python community pretty well. Despite its apparent unpopularity, I think structural and content issues are more important than the markup syntax used for that content or the version control scheme implemented in the backend. Skip

On 07.01.2015 17:55, Skip Montanaro wrote:
... and then you've changed the technology, lost the dynamic features of wikis, the permission controls, full text search, but have not solved the real issues, only created more work that's not relevant to the content.
In the end, it all boils down to people curating content. The tools, the markup and version control are all secondary. My recommendation: look through the wiki pages, grab some that are older and some care, and improve them. That's going help much more than this tools discussion. Here's starting point: https://wiki.python.org/moin/WikiGuidelines -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, Jan 7, 2015 at 2:45 PM, M.-A. Lemburg <mal@egenix.com> wrote:
I'm not sure what the disagreement is here. I wasn't advocating moving away from wiki technology generally, just observing that the current tool (MoinMoin) has different markup syntax and version control than people appear to like (Markdown and GitHub). I imagine you could find a wiki system which uses those two frontend and backend technologies. I was only pointing out that converting from one to the other shouldn't be terribly onerous, as long as you could iterate over the versions and parse the existing syntax into the new form. In any event, I agree that the content and structure is more important than the implementation technology. If you can't solve those problems, changing technology winds up being just a bunch of useless effort. Skip

On 07.01.2015 21:55, Skip Montanaro wrote:
Ah, sorry. I was obviously misunderstanding your emails :-) Moin does support ReST out of the box and there's also an addon parser for Markdown, we could install, if it's stable enough.
Yep. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 07 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Tue, Jan 6, 2015 at 1:31 PM, Guido van Rossum <guido@python.org> wrote:
The edx wiki seems to avoid many of these challenges. https://github.com/edx/edx-platform/wiki/How-to-Rebase-a-Pull-Request Other than versioning docs/ with the code and encouraging helpful community members to provide a valuable service, IDK of any other good solutions. As far as change control, pull requests (of the wiki branch) seem to work great for just me.

On 01/04/2015 05:13 PM, Steven D'Aprano wrote:
I hope that we do not. A separate wiki is fine, but the official docs should just have the official stuff in them. If the docs were a wiki, we'd have scads of entries on "how to program <language-x> in Python", and IMNSHO the official docs should just be about programming Python in Python. -- ~Ethan~

On Jan 5, 2015, at 2:41, Ethan Furman <ethan@stoneleaf.us> wrote:
MSDN has something similar. They've gone back and forth between a half-dozen different models, and I think at present it's just (formatted) comments on each documentation page. At any rate, it's very clear what's official documentation and what's some guy saying "the docs say you can't do Foo, but here's some code that simulates Foo in most cases". I've found useful stuff there. Then again, I've also found live code (and StackOverflow questions, etc.) that's broken because someone took some sample fragment from a comment and used it inappropriately (often ignoring the very next comment that explains why you shouldn't do that if X, Y, or Z). Both PHP and MSDN have a separate page for each function/method/attribute, while Python has a page per module (broken down into sections that are usually very nicely organized, so I wouldn't want to lose that...), so I'm not sure a similar comment format would work in the first place. Maybe allowing inline comments at every section break, entry, or even paragraph would work (like the way PyGame docs have inline example-search and other links)? But that could easily lead to docs that are so cluttered with comments (or with disclosure triangles, if they start off hidden) that the actual documentation is hard to read.
If the docs were a wiki, we'd have scads of entries on "how to program <language-x> in Python",
In other fora (python-list, python-tutor, StackOverflow, comments on widely-followed blogs, etc.) when someone suggests writing Java-style code instead of Pythonic code they usually get quickly shouted down, often even with good explanations, so I'm not sure things would be any worse on a wiki attached to the Python docs. Also, the Python wiki is already a wiki, and isn't dominated by that kind of cruft. So I'm not sure that problem is as much of a threat. That being said, I agree with your conclusion. Docs seem to work best information that's managed like code; things that aren't quite docs but are useful may work best in other ways, but people have already found the wiki, StackOverflow, ActiveState, blogs, etc. and aren't having any problem using them. And, of all the alternate places to find elaborated documentation, the wiki seems to be among the least popular, which doesn't seem to imply that more wikification is the answer to any relevant question.

On 5 January 2015 at 11:13, Steven D'Aprano <steve@pearwood.info> wrote:
Wikipedia's typical equivalent of pull requests is when editors revert a page to an earlier version and move the discussion of the proposed change to an earlier version. This approach can be escalated to *actual* pull requests when the editors lock a page to disrupt an ongoing edit war. The "anyone can publish by default" approach has the advantage of significantly increasing editing throughput by streamlining the handling of non-controversial cases. The downside as a reader is that it achieves this by allowing a brief window where controversial changes can be published without first establishing consensus, which means you may be presented with inaccurate information on controversial topics depending on when you check a page. I think that model works well for the task of creating a collaborative encyclopaedia (where "eventually accurate" is good enough for almost all purposes, and lowering barriers to entry for casual contribution of corrections is a high priority), but I don't believe it's appropriate for the task of delegating the CPython core development team's reputation and authority to other groups. For that, a pre-commit review process, or a more explicit topic area delegation (like the Python Packaging Authority handling packaging.python.org), makes more sense to me: if we trust folks to maintain a module or topic area *in* the standard library, we should be able to trust them to provide reasonable and balanced recommendations regarding applications and libraries that are maintained *outside* the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:
Why would someone who is developing on Linux and has no pre-made library function to call bother with that? If being cross-platform isn't easy, it won't happen. You see it now with the lack of any support for "call glob on arguments on windows and not on unix" [because the shell handles it on unix] whether directly, in argparse, or in fileinput - today, nothing ever calls glob, and so people calling such scripts on windows can't use wildcards (glob is also different from windows wildcards in subtle ways)
However, what authority there is does not allegedly demand that you present dialog boxes and admin elevation prompts to the user (this has been asserted, but no citation has been given).
I don't see why you need to call an API to the desktop enviroment. The entire point of the spec is to provide compatibility _between_ implementations on the same filesystem - you can quit gnome and log into KDE and see the same trash. Python would just be a separate implementation that stands by itself.

On 5 Jan 2015 05:25, <random832@fastmail.us> wrote:
On Fri, Jan 2, 2015, at 20:50, Steven D'Aprano wrote:
That compatibility is implemented *by* the desktop environments. Yes, you could write your own, but why would you, when the whole point of providing a standard API would be to integrate more cleanly with the underlying OS.
Python would just be a separate implementation that stands by itself.
The word "just" does not belong in that sentence. Reliable cross-platform development & testing for operations that are tightly coupled to operating system services is hard, period - the associated test matrix of platforms and permission levels ensures that. Regards, Nick.

On Mon, 5 Jan 2015 06:25:01 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
For the same reason that we ship a HTTP server while we could integrate more cleanly with Apache? :-) When there's a well-defined spec, it can be simpler to implement the spec yourself rather than pull third-party dependencies, especially when those happen to be huge as may be the case with desktop environments. Regards Antoine.

On Jan 4, 2015, at 21:35, Antoine Pitrou <solipsis@pitrou.net> wrote:
That's an interesting point--but note that Python _also_ has WSGI and various other ways to integrate more cleanly with Apache (and various other web servers). If you just want to put up a simple web service to handle a few users (e.g., as a user interface to your application), SimpleHTTPServer is sometimes the answer. But if you want to add a web service to a major Apache deployment, you generally don't want to do that by deploying SimpleHTTPServer alongside it. It's not _impossible_ to share ports, SSL certs, configs, logs, deployment scripts, monitoring tools, etc. between the two, but it's not the way you'd want to do things. (There's also the fact that SimpleHTTPServer won't be able to handle the same load as Apache…)
When there's a well-defined spec,
I suppose it depends on your definition of "well-defined spec". As Nick said, XFCE only partly supports the spec, and he's not sure how completely GNOME does. As I pointed out, there are gaps in the spec for both implementation-defined and future-expansion behavior. That isn't the same as the case with HTTP/1.1. Look at http://www.freedesktop.org/wiki/Software. Notice that xdg-utils does not yet have a trash command, and there is no separate trash program or library. (As far as I know, there are at least two XDG trash programs on github which have not yet been submitted to be part of the XDG software collection, which don't appear to be 100% compatible with each other. (For example, one of them requires top trash directories to have exactly mode 41777, the other just requires them to be executable for the current user and have the sticky bit on.) Does creating a third one--and putting it in the Python stdlib forever--really sound like a good idea? (As a side note, one of the two above-mentioned trash programs is written in Node, and I don't see anyone suggesting that the functionality be moved into the Node stdlib.)

On Jan 4, 2015, at 20:24, random832@fastmail.us wrote:
Where has that, or anything remotely similar, been alleged or asserted? What _has_ been asserted is that an app has to fit in with the choices made by the platform implementation. And the only example given was given with a direct quote from the spec linked in the preceding email. I'm not sure how much simpler a citation you need in order to be able to follow it. The example was about platform-specific ways to allow admins to disable sticky-bit checking on network mounts. There are more such things that are left incompletely specified, or explicitly left unspecified for future development. I don't think any of them have anything to do with dialog boxes or other user interaction, and I don't recall anyone saying otherwise. (I suppose if you didn't read carefully, you might take something like "SHOULD warn the administrator" as implying a prompt saying "Go get your sys admin and drag him over to the screen" rather than, say, a syslog warning with a string the admin can see later and look up in help or google).
The last sentence is exactly the same thing you already got two detailed answers to that you haven't responded to in any way. So I suspect the reason you don't understand why calling an API is a good idea is that you skimmed over the explanations that would explain exactly that. Go back and read those emails.

On Sun, Jan 4, 2015, at 16:02, Ethan Furman wrote:
If by "stands by itself" you mean "desktops can't interact with it", how would that be useful?
They interact with it _through the filesystem_, because it does the same things with the filesystem as all the desktop environments do and as the spec says. Just like Gnome doesn't have to interact directly with KDE - programs from each environment don't have to call the other's library APIs - for both Nautilus and Dolphin to display the same trash contents and to do the same thing when you trash a file.

On Sun, Jan 4, 2015, at 16:02, Ethan Furman wrote:
On Unix, as I assume you know, the shell is responsible for interpreting the kind of wildcard patterns that glob uses (plus shell-specific extensions), and passing a list of proper filenames as the child process's argv. On Windows, this does not happen - the child process is simply passed a single string with the whole command line. Python (or the C Runtime Library that the python interpreter is linked against) converts this to a list of strings for individual arguments based on spaces and quotes, but does not interpret wildcard patterns in any of the arguments. The C Runtime Library _can_ do this automatically - this is done on MSVC by linking the "setargv.obj" library routine, which replaces a standard internal routine that does not expand wildcards, but it does a poor job because it does not know which arguments are intended to be filenames vs other strings, and there is traditionally no way to escape them [since * and ? aren't allowed in filenames, there's no reason not to allow "some directory\*.txt", all in quotes, as an argument that will be handled as a wildcard] The appropriate place to expand them would be after you know you intend to treat a list of arguments as a list of filenames, rather than at program start - after options are parsed, for example (so an option with an argument with an asterisk in it doesn't get turned into multiple arguments), or if a list is being passed in to the fileinput module. This should also only be done on windows, and not on other platforms (since on other platforms this is supposed to be handled by the shell rather than the child process). Right now, none of this is done. If you pass *.txt on the command line to a python script, it will attempt to open a file called "*.txt". ---- Another separate but related issue is the fact that windows wildcards do not behave in the same way as python glob patterns. Bracketed character classes are not supported, and left bracket is a valid character in filenames unlike ? and *. There are some subtleties around dots ("*.*" will match all filenames, even with no dot. "*." matches filenames without any dot.), they're case-insensitive (I think glob does handle this part, but not in the same way as the platform in some cases), and they can match the short-form alternate filenames [8 characters, dot, 3 characters], so "*.htm" will typically match most files ending in ".html" as well as those ending in ".htm". It might be useful to provide a way to make glob behave in the windows-specific way (using the platform-specific functions FindFirstFileEx and RtlIsNameInExpression on windows.)

On Sun, Jan 04, 2015 at 06:10:01PM -0500, random832@fastmail.us wrote: [...]
Right now, none of this is done. If you pass *.txt on the command line to a python script, it will attempt to open a file called "*.txt".
The important thing is that shells have different behaviour. A way, perhaps not the best way, but a way, of getting close to platform independent behaviour when it comes to globbing is to use the glob module. On Windows, you will get Python's definition of globbing. On POSIX systems, the globbing module will do nothing, because the shell will most likely have already expanded the wild-cards. (I don't know of any Unix shells which behave like Windows.) Windows users are unlikely to try to use Unix-shell-specific wildcards, because they are Windows users and they won't have any expectation that they will work. Unix users will, and they will work because the shell interprets the wildcards before Python sees them.
Another separate but related issue is the fact that windows wildcards do not behave in the same way as python glob patterns.
I don't understand why you say this. As I understand it, there is no such thing as "Windows wildcards" as every application which wants to support wildcards has to implement their own. If you want to know what kinds of globbing wildcards the application supports, you have to read the application documentation. (Or guess.) I am not a Windows expert, so I may have been misinformed. Anyone care to comment?
Bracketed character classes are not supported,
Applications which use the Python glob module do support them. Other applications may support them, if they so choose to implement it.
and left bracket is a valid character in filenames unlike ? and *.
And on POSIX systems, *all* wildcards are valid characters in file names. If you want to specify a file called literally "*.txt", or "spam[and eggs?].jpg", you have to escape the wildcards. Windows is no different. The glob module supports escaping of wildcards, doesn't it? If so, we have no problem. If not, that's a bug, or at least an obvious and important piece of missing functionality.
As I said, I don't believe that there is any standard Windows filename wildcard handling, so the behaviour you describe may apply to some applications but not all. Anyone like to comment?
This may be a good idea. Maybe glob needs to stay as it is, for backwards compatibility, and a new module osglob be created that aims to implement globbing according to the expected rules of the operating system. osglob could work like the os module and delegate to platform-specific modules posix_glob, windows_glob, osx_glob etc., with the current glob module remaining for applications which want to present a more-or-less equivalent globbing behaviour. -- Steven

On Mon, Jan 5, 2015 at 11:57 AM, Steven D'Aprano <steve@pearwood.info> wrote:
You're assuming that there are no wildcard characters actually included in the file names, which on Unix systems is perfectly possible. How can you reliably manipulate a file called *.txt when there are foo.txt and bar.txt in the same directory? With most Unix programs, you escape the asterisk to get it past the shell, the program does no globbing of its own, and you're safe. If your Python script unconditionally globs its file names, you're stuck.
There's a very standard interpretation, which I think is codified in the FindFirstFile API, although I'm not sure of the details.
No, Windows *is* different, because you simply aren't allowed to have those characters in file names:
Unix allows anything other than a slash or NUL, so all wildcards have to be escaped to get them past the shell; or, alternatively, you can just use a non-shell way of starting a program, like Python's own subprocess module.
I'm not sure that's the solution, because then Unix users would have to double-escape everything. It can be done, of course, but this is the exact sort of complication that single-platform developers often don't even think of. Suppose you develop on Windows, and just unconditionally glob all your arguments... your program will work fine on Unix, until it's asked to deal with a file with an asterisk in it, and your user complains very loudly about how it misbehaved... and maybe destroyed a bunch of files. Oops. Not good. ChrisA

On Mon, Jan 05, 2015 at 12:08:06PM +1100, Chris Angelico wrote:
You seem to be right. With double wildcard expansion (by the shell and by the python script) there doesn't seem to be a straightforward way to get just the file name with the wildcard in it: [steve@ando junk]$ cat testglob.py import sys import glob for arg in sys.argv[1:]: print(arg + ":-") for name in glob.glob(arg): print(" " + name) [steve@ando junk]$ ls eggs.jpg *.jpg testglob.py [steve@ando junk]$ python3.3 testglob.py *.jpg eggs.jpg:- eggs.jpg *.jpg:- eggs.jpg *.jpg [steve@ando junk]$ python3.3 testglob.py \*.jpg *.jpg:- eggs.jpg *.jpg [steve@ando junk]$ python3.3 testglob.py \\*.jpg \*.jpg:- I guess the simple solutions are to either do a platform check first, or to provide a --no-glob command-line switch to turn globbing off. But I guess people won't think of that until its reported as a bug :-)
Do all Windows apps call FindFirstFile?
I know that. The point that Random raised is that some wildcards are legal in Windows file names. Yes, and *all* wildcards are legal in POSIX filenames, so whatever problems we have with escaping are going to occur on both platforms, as you have already pointed out above. In any case, having a platform specific globbing module is looking more and more useful. Does this need a PEP? -- Steven

On Sun, Jan 4, 2015, at 21:07, Steven D'Aprano wrote:
Do all Windows apps call FindFirstFile?
Most Windows apps' only interaction with wildcards is via the system file selection dialog. Ultimately, FindFirstFile/FindNextFile is the only way to enumerate files in a directory - it's the equivalent of opendir/readdir. You're expected to pass in directory\* or *.* if you want to enumerate all files in a directory with no filtering. I think it's rare for programs not deliberately written to emulate some other behavior or ported from unix (with canned implementations of opendir, readdir, glob, and fnmatch being easier than rewriting everything to use the windows functions) to implement their own wildcard handling rather than using the Windows API - the C Runtime functions findfirst/findnext date back to early DOS C compilers (the OS functions were int 21,4E and 4F.)
I know that. The point that Random raised is that some wildcards are legal in Windows file names.
Well, more specifically, that the square bracket is both allowed in Windows filenames and not expected to be used as a wildcard, so there is no escape mechanism (since the only standard wildcards are ? and *, which are not allowed in filenames)
Well, yes, except for the fact that escaping and quoting is handled by the shell on POSIX. My _main_ reason for mentioning it was to forestall objections involving applying wildcards to double-quoted strings, since double quote does _not_ normally escape wildcards on Windows.

Steven D'Aprano writes:
In any case, having a platform specific globbing module
By "platform-specific", do you mean one module that handles all platforms appropriately (what this thread is about), or do you mean per-platform modules (what "platform-specific" means to me)?
is looking more and more useful. Does this need a PEP?
If you're really serious about putting it in the stdlib, I would certainly think so. It's absolutely clear that the solution is going to be complicated and hard to understand. Without a PEP, nobody will know what is and what is not a bug. Likely even with a PEP a lot of people will claim the PEP is buggy no matter what it says! I would hope the different specs will partition the complainers, but I'm not even sure of that. ;-)

It seems to me that if you want auto-globbing support on Windows, the "right" way to do it is to link in the fancy setargv instead of the default one. This handles wildcards exactly the same way the command.com/cmd.exe builtin commands do, which I suspect is what Windows users would actually be expecting of they passed a wildcard on the command line. (I think the docs no longer guarantee that this is true, but it's probably still true, and certainly closer to true than if you try to do it manually.) The only problem is that you can't add a command-line switch that controls at runtime which implementation of setargv got linked into the msvcrt prequel at build time. There used to be a way around this for DOS/Win16 programs, but I don't think there is for NT/Win32 (short of ignoring the argv and asking for and parsing the command line instead), so this would be an all-or-nothing change. Are there programs that depend on _not_ having auto-globbing on Windows? There might be... At any rate, I can't imagine many programs want POSIX (sh-style) globbing but without other POSIX shell features (at least the different split/quoting/escape rules, if not the various kinds of expansion, etc.); the only reason anyone calls glob.glob for Windows is because it's relatively easy (compared to using win32api to call FindFirstFile, or using win32api to get the whole command line and then using shlex and os to process it, depending on which one they actually wanted) and not _too_ terrible for simple cases. If there's a way to give people something that's not only easier but better, I don't think they'd complain. Sent from a random iPhone On Jan 5, 2015, at 5:19, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:

On Mon, Jan 05, 2015 at 01:19:16PM +0900, Stephen J. Turnbull wrote:
Yes to both. The model I am proposing is that of the os and os.path modules: we import os and it does the right thing for the current platform. In particular, os.path will do the right thing for the current platform, but you can import ntpath, posixpath, genericpath etc. to get behaviour for a specific platform. I think we have to leave the glob module as-is, for backwards compatibility, and add a new package, say globbing (or osglob, the name can be bike-shedded to death later *wink*) with the new behaviour. globbing will look something like this: globbing/ __init__.py posix.py nt.py generic.py (plus any other platforms that may need special treatment, if any) and the globbing/__init__.py implementation could potentially be as simple as "from foo import *" from the appropriate sub-module. The globbing.generic implementation might be as simple as: from glob import * Or perhaps the relationship should go the other way? Move the current glob.py implementation into globbing.generic, and replace it with: from globbing.generic import * These little details are to be decided later, but the aim is: * Current code that imports glob will continue to work the same as it does now, warts and all. * Code that imports globbing will do the right thing for the platform. * If you want to apply globs to something other than file names, the right module to use would probably be globbing.generic (or possible fnmatch directly). -- Steve

On Tue, Jan 6, 2015 at 4:10 AM, Steven D'Aprano <steve@pearwood.info> wrote:
To clarify: Do you mean "something other than names of currently-existing files", or "something other than valid names for the local file system", or something else? For instance, suppose you write an unarchiver (in case we don't have enough of them already, of course), and you can "python unarchive.py archivename *.txt" to extract all files matching *.txt. The globbing would be done against some sort of internal index, but the names would have to be valid for the local file system, or you wouldn't be able to create them. Which module should you use? ChrisA

On Tue, Jan 06, 2015 at 04:14:38AM +1100, Chris Angelico wrote:
fnmatch. I've had a look inside glob.py and it calls os.listdir directly, so you cannot use glob to match things other than actual existing files. (Well, I suppose you could if you monkey-patched the module, but lets not go there.) fnmatch, on the other hand, provides the right tool for matching names other than actual file names: fnmatch.filter(names, pattern). (Well, almost the right tool -- it has a few issues too.) To summarise: - the glob module returns only actual files in the file system; - the new globbing package will do the same, except it will use platform-specific wildcards where possible; - fnmatch continues as Python's generic "match globs against arbitrary strings" module (despite the name). -- Steven

On Tue, Jan 6, 2015 at 10:57 AM, Steven D'Aprano <steve@pearwood.info> wrote:
Ugh. That's poor naming, then. If this new globbing module happens, I would be a strong +1 on having 'globbing.generic' (or somesuch) to do this kind of thing, if only to fix the name. ChrisA

Steven D'Aprano writes:
The model I am proposing is that of the os and os.path modules: we import os and it does the right thing for the current platform.
Which is normally "nothing" on POSIX, since the shell does it for you. Or, if you're talking about what the shell does for you, while I suppose there is a basic globbing defined by POSIX, but bash and zsh go well beyond that, and they behave somewhat differently in corner cases IIRC. If the program is going to invoke a globbing function specifically, I suppose it makes sense to default to the same behavior as on the current platform, but it's not obvious that that's what the user wants. I use bash on Windows; why settle for cmd.exe if you want the power of the command line? It could even cause problems (is there a reliable way to determine what the shell is, let alone whether it implements globbing or not, on Windows?)
But there *is* a RightThang for paths; the OS decides. Globbing is a user preference thing.

On Mon, Jan 5, 2015 at 12:09 PM, Stephen J. Turnbull < turnbull@sk.tsukuba.ac.jp> wrote:
Indeed. I only have 2.7.2 available here at work. Here's what bash tells me on a Linux box: % ls yen{2,3}.* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade % ls yen[23].* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade Here's what /bin/sh tells me on a Solaris 10 box: $ ls yen{2,3}.* yen{2,3}.*: No such file or directory $ ls yen[23].* yen2.out yen2.png yen2.trade yen3.out yen3.png yen3.trade Here's what the glob module tells me: % python Python 2.7.2 (default, Nov 14 2012, 05:07:35) [GCC 4.4.6 [TWW]] on linux3 Type "help", "copyright", "credits" or "license" for more information.
I only discovered this "shortcoming" (or "Bourne Shell dependency") relatively recently. I've been using bash for so long it never even occurred to me that {...} notation wasn't available in all shells. Skip

On 01/05/2015 07:45 PM, Skip Montanaro wrote:
[...]
Note that the {...} notation is not a part of globbing, it's a different mechanism (bash calls it brace expansion IIRC). With brace expansion, the different choices are *always* expanded regardless of the existence of matching filenames. Your pattern first gets expanded to "yen2.* yen3.*" and then globbing ensues (with the standard rule that if there is nothing matching e.g. yen2.* it is either given literally to the program or the command line is rejected, depending on the shell). => I wouldn't expect the Python glob module to perform brace expansion. cheers, Georg

On Tue, Jan 06, 2015 at 03:09:39AM +0900, Stephen J. Turnbull wrote:
All correct.
I see that as an application issue, not a Python issue. As the author of the application, I have some flexibility: - Currently, I can not use glob.py at all, and my Unix users will be able to use globs, but most of my Windows users won't. - If I unconditionally use glob.py, my Unix users can not reliably specify filenames containing wildcards but my Windows users may (assuming that glob supports escaping, which it currently doesn't). - I can conditionally use glob or not, depending on the platform. - Or I can provide an application switch/preference which allows the user to decide whether to use wildcards or not. I don't know of any reliable way for a Python script to determine the current shell (there may not even be a shell!). If such a way exists, that would be good functionality to have.
Not entirely user-preference. As Random's discussion has pointed out, there are issues with the way globbing is currently handled. For example, Windows doesn't support [...] wildcards and so most Windows users will expect to be able to specify [ and ] as ordinary characters. You can't do that with glob.py. Windows users who have bash installed are a very small minority :-) -- Steven

On Sun, Jan 4, 2015, at 23:19, Stephen J. Turnbull wrote:
Windows technically requires calls to the actual filesystem driver for correct behavior - different filesystems may have different case folding, theoretically other different rules, etc. Plus you've got to examine both the long and short filenames - if you have foobar.html, *.htm will match it because the short filename is FOOBAR~1.HTM (it, however, _returns_ the long filename.) This means that the equivalent function to glob can't simply be the equivalent of listdir filtered by fnmatch. On Mon, Jan 5, 2015, at 05:10, Andrew Barnert wrote:
The problem with that is that if you do that you can no longer pass in _non_-filename arguments that contain a question mark or asterisk (and happen to match a file). Better to do it inside the program, when you actually know the argument you're looking at is intended to be a filename spec. Which, I assume, is _why_ it's not done by default when you compile a C program. A program like grep "a.*b" "Some Directory\*.txt" should handle wildcards in the filename and not in the regex. That is the user expectation on windows, and that is something that windows programs (and not unix programs) can and do deliver. Setargv doesn't, which is why you have to opt in to it (if you know your program doesn't need non-filename arguments that look like wildcards). It should be possible to write Python programs that can do the same, in a way that is relatively painless. A setargv-like solution is insufficient for that. On Mon, Jan 5, 2015, at 13:45, Skip Montanaro wrote:
Brace expansion is technically not globbing. yen{2,3}.txt will return yen2.txt yen3.txt even if one or both of those files don't exist. The reason it works with globs is that yen{2,3}.* literally expands to yen2.* yen3.* which is then globbed.

On Tue, Jan 6, 2015 at 7:33 AM, <random832@fastmail.us> wrote:
Which, by the way, is the main reason I use brace expansion. Type "mv" and then fill in a file name using tab completion, then insert or delete something by editing it to have braces and a comma: mv some-long-file-name-blah-blah-blah mv some-{long-,}file-name-blah-blah-blah Et voila! Remove one word from a tab-completed file name. There's no way that a generic glob expansion tool can do everything that bash does, and I doubt anyone expects it. ChrisA

On 5 January 2015 at 20:33, <random832@fastmail.us> wrote:
Just as a note - I find the fact that Python *doesn't* allow the C runtime to do its glob-mangling of the supplied command arguments a very useful feature, and I would not want it to be "fixed". The setargv behaviour in the MSVC runtime is full of arcane corner cases and quoting gotchas, and I wouldn't want it imposed on me by default. Paul

On Sun, Jan 4, 2015, at 19:57, Steven D'Aprano wrote:
Wildcards are built into the FindFirstFile/FindNextFile (and thereby the _tfindfirst/next) API used for enumerating directories, and users do not generally expect to be able to apply wildcards to path components other than the last one. These functions are used by applications, and generally by libraries provided in higher-level languages.
I am not a Windows expert, so I may have been misinformed. Anyone care to comment?

Hi, This idea was already proposed as part of a larger PEP: https://www.python.org/dev/peps/pep-0471/#wildcard-support The PEP explains why the Windows wildcard idea was rejected. Victor

On Tue, Jan 6, 2015, at 18:11, Victor Stinner wrote:
Not the same idea, really. The _main_ thrust of this idea is to have a way to have a single function (OS-specific algorithms or otherwise) which acts (roughly) like itertools.chain(map(glob,lst)) on windows and simply uses the original list unmodified on unix, and to use this in places like fileinput. The point is that command line argument wildcard processing belongs in a different place on each platform. The windows-specific matching algorithm is just an aside.

On Wed, Jan 07, 2015 at 12:11:51AM +0100, Victor Stinner wrote:
The Windows wildcard idea was rejected in the context of a faster os.walk, not globbing. The PEP is inaccurate: it gives the wrong Python-Ideas thread for discussions on the wildcard idea. The wildcard idea is discussed in this thread: https://mail.python.org/pipermail/python-ideas/2012-November/017965.html not in the thread given in the PEP. Well, I say "discussed", but it's mostly ignored, with only a couple of comments. There's somewhat more discussion on Python-Dev: https://mail.python.org/pipermail/python-dev/2014-June/135217.html but it's mostly people just saying "-1 on wildcards". I agree with this decision, for PEP 471. But I don't believe this is relevant to the question of explicit globbing. -- Steve

On Tue, Jan 6, 2015, at 19:42, Steven D'Aprano wrote:
I think the fundamental difference between these cases is in handling user-entered strings, especially command-line arguments (where the user will expect them to behave a certain way, in particular "*.*" matching all files even those with no dot - most of the other quirks are obscure anyway) vs hardcoded (where expectations are determined by the programmer, not the user).

On Mon, Jan 5, 2015 at 8:02 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
On Unix, a command like "python some_script.py *.txt" is parsed by the shell[1] into a bunch of extra arguments, eg "python some_script.py file_one.txt file_two.txt file_three.txt". On Windows[2], this expansion isn't done, so the script gets a literal "*.txt" as an argument. So a Unix program is usually written to accept multiple args, but a Windows program usually expects to do its own globbing. Now, what happens when you want to reference a file whose name actually includes U+002A (or \x2A, given that file names on most Linux systems consist of bytes)? On Unix, you put a backslash in front of the name, to prevent the shell from expanding it ("\*.txt"), or use some other means of calling up the program (subprocess.call(["python","some_script.py","*.txt"])). Your expectation then is that the script will NOT expand globs; but the script will see the exact same thing in sys.argv as it does on Windows, where the expectation is that it WILL expand globs (and, if I'm not mistaken, asterisks are simply disallowed in file names - though that might be a filesystem-specific rule). So somehow, your script has to know whether it's supposed to call glob or not. Or, more likely, it'll do whatever is appropriate for the platform its developer uses, and then misbehaves on the other one. ChrisA [1] Assuming you're using one of the standard shells, and assuming you haven't configured this feature away, etc, etc. Normal user expectations. [2] Again, assuming you're using the default shell; but again, normal user expectation, ergo replacement shells often replicate this.

random832@fastmail.us writes:
But Python doesn't much care about similarities or differences between GNOME and KDE AFAICS -- the relationships to Mac and Windows are much more interesting. Emacs has had a policy of implementing everything itself for decades, and its mindshare has been decreasing for decades.[1] Many Emacs developers are cross-platform users, and the common cross-platform implementation is great for them. Most potential users are not, and the small differences between Emacs's behavior and the platform API mean that Emacs is not a separate implementation that stands by itself -- it's a bad implementation that spends a lot of time standing in a corner.[2] This is not the only reason for Emacs's relative loss of popularity, but it's definitely a common complaint -- and the complainers often never return to Emacs channels when it becomes clear that the Emacs way is the only way. In any case, the small platform differences and the odd problem with the spec mean that this is definitely suited for out-of-stdlib experimentation. Footnotes: [1] The former is a consequence of a policy that (official) Emacs on non-GNU systems can't have any feature that Emacs on the GNU System doesn't have -- if you want a move-to-trash command on Mac, you have to implement for GNU. [2] A traditional punishment for mischievous children, especially those who don't finish their chores or homework, where I grew up.

On Sun, Jan 04, 2015 at 02:24:36PM -0500, random832@fastmail.us wrote:
Because they want to write cross-platform code. For my own personal scripts, I wouldn't bother. If I had a Windows system which I intended to use the script on, I would, or if I intended to make the script available to others, I might.
I think you are mistaken that nothing uses glob: https://searchcode.com/?q=import+glob+lang%3Apython [...]
You don't *need* to do so, but doing so guarantees that your code will match the expected behaviour as provided by the API, and avoids needing to re-invent the wheel. I would expect that for something which eventually makes it into the standard library, if it ever does, it will have multiple implementations: - call the platform API, if one exists and it is available - fall back to a Python implementation, if no such API exists That is exactly what the third party Send2Trash package does. If Gnome is installed and the GIO library is installed, it uses that, otherwise it falls back on its own (possibly inferior?) implementation. -- Steven

On Sun, Jan 4, 2015, at 20:31, Steven D'Aprano wrote:
I think you are mistaken that nothing uses glob:
I am very confident in saying that programs that do not use glob _to process lists of filenames given on the command line_ outnumber those that do at least 1000:1, and that the vast majority of your search consists of uses for processing things other than command line arguments.
The vast majority of results (I used https://searchcode.com/?q=glob.glob+lang%3Apython so I could see the actual call in context) are using it for hardcoded strings, not command line arguments (and to do it correctly they should do even that conditionally on os.name == 'nt' [or os2 or ce, not sure about the other non-posix values])

I have no problem with this being included somehow in the standard library, but pathlib to me is all about the filesystem, whereas (as others have pointed out), the Recycle Bin and associated concepts on other OS relate pretty uniformly to the OS shell. There are many shell-related concepts that could be nicely exposed, but again I don't think any of them belong in pathlib. For example, labels/tags, owner application, icon, visibility flags, etc. David On Mon, Dec 29, 2014 at 06:54:46PM +0200, Ram Rachum wrote:

On 5 January 2015 at 20:38, David Wilson <dw+python-ideas@hmmz.org> wrote:
Better tools for accessing typical extended file metadata would indeed be highly desirable (thumbnails/previews are another one that comes to mind). As with the recycle bin though, the generally preferred approach for standard library additions is incorporating existing modules that have been available through PyPI for some time, or else creating a derivative of such a project that meets other constraints of standard library inclusion (e.g. ipaddress vs the project it was based on, ipaddr). Unfortunately, most of the folks that need these cross-platform integration features are likely to be using a C/C++ toolkit that provides them (e.g. Gtk, Qt, wxWidgets), which significantly reduces the incentive for anyone to create a Python-specific abstraction that would be suitable for eventual inclusion in the standard library. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jan 6, 2015, at 5:43, Nick Coghlan <ncoghlan@gmail.com> wrote:
I'm not sure that's really unfortunate. First, a framework like Qt or wx is maintained by people who think about nothing but desktop and GUI integration on the major platforms. They track changes pretty quickly (especially compared to the Python stdlib). A few years back, when Apple got rid of legacy-style preference files, I had a Qt app that I was able to fix just by updating to the next version of Qt, and a non-Qt app that I had to fix by reseatching, writing, and testing, a wrapper that "temporarily" (of course it's still there in 2015...) stashes the entire legacy file as an encoded string in the new-style prefs database. What are the odds a Python stdlib solution (or even a third-party Python lib with less than 1/20th the user base, community, and corporate support) would have fixed it for me automatically in time for my next release the way Qt did? Also, the fact that Qt and wx are complete frameworks means they can make assumptions that allow for simpler solutions. A Qt library knows how to check whether there's a runloop and, if there is, knows it'll be a Qt runloop, and, if not, knows how to create a short-term blocking runloop and background thread, and it also knows how to pass signals from background threads to the runloop, and so on; this means Qt could just use the (10.5+, and still not deprecated as of 10.10) NSWorkspace API for trashing on Mac, because none of its deficiencies apply to any Qt app. A library meant to be used generally across all kinds of apps can't do that. Obviously this is also a weakness, because not all cross-platform desktop-integration apps really need a full framework… but many do. And again, many simple tools don't need to be cross-platform (like my completely separate 3-line trash tools for each platform that I use), or don't need full generality (my Mac trash tool doesn't need to work on 10.6, or to be App Store compliant; my Ubuntu trash tool doesn't have to know how Unity deals with trashing over CIFS because I only have two CIFS mounts and they both do server-side automatic recycling; etc.).

On 6 January 2015 at 19:29, Andrew Barnert <abarnert@yahoo.com> wrote:
Yes, there's certainly a lot of value in adopting an existing specialist toolkit when writing rich client applications. The key problem is that it isn't easy to start with the provided Tcl/Tk support in the standard library (to avoid introducing a large external dependency) and then later migrate to one of the other third party toolkits (if the basic functionality proves insufficient). While an asyncio-style pluggable solution may be one possible option for that (i.e. provide default implementations of features which may be fairly basic, but allow them to be replaced by full featured implementations backed by Gtk/Qt/wxWidgets/etc), that kind of abstraction layer design is extraordinarily difficult to design well. It's hard enough even in cases like wsgiref and asyncio that are relatively simple compared to the services a full rich client application will expect to have available. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jan 6, 2015, at 11:25, Nick Coghlan <ncoghlan@gmail.com> wrote:
Yes, and even besides the difference in complexity, even the most basic elements--the API for the runloop and message maps (wx), signals/slots (Qt), event handlers (Tk), etc.--are pretty hard to abstract away. For servers, a reactor is a reactor, but for GUIs, a signal/slot system is not a message pump/message map system. And whichever abstraction you pick, adapting it to work with the other abstractions is no fun. (Look at how long the TkAqua Cocoa port took, and how much Cocoa functionality is still missing.)

On Mon, 29 Dec 2014 18:54:46 +0200 Ram Rachum <ram@rachum.com> wrote:
I think the corresponding low-level function to the os module would be a good start (the hard part is probably to write the cross-platform code, anyway). Regards Antoine.

On Dec 29, 2014, at 18:53, Antoine Pitrou <solipsis@pitrou.net> wrote:
There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there. On OS X, unless you want to use higher-level CoreFoundation APIs that require a runloop, you just look up the name of the trash directory (which I believe is theoretically localized but actually always ~/.Trash on the drive with your user directory, $mnt/.Trash otherwise) and call rename. On Linux, and some other *nix platforms, you assume XDG directory layout and call rename. On other platforms, I don't know if there even is a way to do it. And I don't know what the right thing to do on OS X would be for a file on a different filesystem than the user's home that doesn't have its own .Trash or is mounted read-only or no-trash. Finder gives you a popup in each of those cases asking if you want to delete it immediately because it can't be trashed. And similarly for Linux, but the details are different (especially since you might not even have a home trash). I'd guess Send2Trash is a good place to at least start, but if this is going in the stdlib someone needs to make sure it meets Apple HIG rules and XDG standards.

On Mon, Dec 29, 2014, at 13:58, Andrew Barnert wrote:
There isn't really a low-level function for it on any platform but maybe Windows, except rename, which is already there.
The Windows function for this is SHFileOperation, which isn't really low-level (but doesn't require a message loop).
Is it not uid-specific? Windows and XDG trash directories are uid-specific. A google search shows that it's $mnt/.Trashes/$uid.
If it's mounted read-only, you have a bigger problem. Another possible issue could be: bad permission on trash directory, bad permission or nonexistent parent directory that trash directory must be created in (home or .Trashes). These are also all issues that must be faced for XDG. XDG has a database of original location and deleted date for files in the trash - does OSX? (according to the tool I found, it does, at least for original location, but only works if you use Finder to trash the files)
Another option would be to just script Finder with osascript. The tool http://hasseg.org/trash/ has objc code to "directly" script Finder without actually going through osascript. When it does not use Finder, it uses FSMoveObjectToTrashSync, which seems to be depreciated.

On Dec 29, 2014, at 22:15, random832@fastmail.us wrote:
Well, yes, but the point is that Finder gives you a special message for this rather than trying to move it and giving you a permissions error. And more importantly, if it's mounted no-trash or doesn't have a trash can, you don't have a bigger problem, but you still have this problem, so this problem needs to be solved.
I believe all of those issues could just raise the normal exception, because Finder treats them as normal errors--that is, rather than giving you the special "can't trash; delete?" message, it tries, fails, asks you to authenticate as an administrator, and tries again.
That's a whole other issue. I believe that if you use the higher-level APIs instead of POSIX functions they take care of the desktop database and alias/bookmark info and all that stuff. If not... It may happen automatically (at least if you're on an HFS+ drive) but it may not.
If you want to script Finder, the easy way to do it (well, as easy as it gets without Appscript) is ScriptingBridge and/or the lower-level Cocoa AppleScript APIs (via PyObjC, but none of that is going to go into the stdlib, which means shutil needs to be written partly in ObjC), but that will raise the minimum OS X version, because 10.6 didn't have NSAppleScript and friends yet. (And the Carbon AEM calls are deprecated, of course.) This could also make it more difficult for Python apps to be sandboxed and AppStore'd. But the biggest problem is that you're asking Finder to do it. If you don't have permission to trash a file, instead if getting an error, you'll get an auth dialog from Finder. If you trash a bunch of files you'll get a progress bar in Finder (and your script will get a timeout from the synchronous AppleEvent and not know if the trashing succeeded). Worse, if you don't have a Finder to talk to (because you're running in an ssh shell, or in a LaunchDaemon or a detached script, or because you're using a third-party Finder replacement like PathFinder) you won't be able to trash anything. There are plenty of applications that's reasonable for, but I don't think it's reasonable stdlib behavior.
When it does not use Finder, it uses FSMoveObjectToTrashSync, which seems to be depreciated.
Yeah, that's Carbon; you're supposed to use the Cocoa/CF APIs instead. NSWorkspace had one method that's guaranteed to do exactly the same thing as trashing in Finder--which it does by asking Finder--and a more general "perform operations on files" method that can be used for trashing without the Finder, but there's no way to control whether it's synchronous or asynchronous, and requires a runloop in case it's async. Anyway, I don't think these details change the point: There is no low-level function to be added to os. There may be a high-level function to be added to shutil, but it's not trivial.

On Mon, Dec 29, 2014, at 18:46, Andrew Barnert wrote:
That doesn't make sense - the problem is with the state of the trash directory (it may be in a state that you can access, but the "wrong" one, e.g. one that would allow other people to delete your files - I don't know if this is a problem for OSX but it is for XDG), not your access to the file being deleted.

On OSX there are at least two reasons why the Finder can prompt you when you try to move a file to the trash. The first is a source location that doesn’t support the trash (such as a fileserver), the Finder will ask if should just permanently remove the file. The second is when you don’t have permissions to remove the file, the Finder will then ask if it should retry with Administrator privileges (and will prompt you for a username and password if you confirm). Ronald

Andrew Barnert wrote:
You may also need to invent a unique name if there's already a file in the trash with the same name.
Raising an exception would seem to be a reasonable thing to do in that case. -- Greg

On Dec 30, 2014, at 22:36, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
At this point too much context has been stripped from the message for the replies to make sense. The original message was pointing out that there is no separate "low-level API" for trashing that could go in os along with a "high-level API" in shutil--unless you consider rename a low-level API for trashing, in which case it's already there.
And whatever else the api does for you… IMHO its better to use the API for this on OSX, such as FSMoveObjectToTrashSync.
Except that API is deprecated (meaning, among other things, that if shutil linked to it, any py2app-ed or embedding app that imports shutil can't be sold in the Mac App Store), and there are no suitable replacements. By "suitable" I mean "suitable for widespread-enough use that it should be in the stdlib". As I said earlier in the thread, I think it's perfectly reasonable to have a third-party library that uses a deprecated API, requires 10.8, requires a runloop, directly or indirectly asks the Finder to do the work for you, or doesn't quite meet Apple's HIG rules, because all of those could be acceptable in _many_ applications; I just don't think any of those should be in the stdlib, because none of those are acceptable in _most_ applications. If I were designing an app right now, here's what I'd do: If you don't need 10.7 or earlier, use NSFileManager (via PyObjC if possible, via a tiny custom (Obj)C extension module if for some reason it's not, and same goes for all the following). If you need 10.5-10.7 but don't care about the App Store, check NSFileManager dynamically, and fall back to the FS function if the method is missing. If you need 10.6-10.7 and also need the App Store, you've probably got a runloop, so use the async function in NSWorkspace (recycleURLs:completionHandler:). If you need 10.6-10.7 in a helper app in the App Store, use the other function in NSWorkspace (performFileOperation:source:destination:files:tag:) and deal with all of its problems because you have no choice in the matter.

On Wed, Dec 31, 2014 at 12:12:01AM +0100, Andrew Barnert wrote:
What's wrong with asking the Finder to move the file? Under what circumstances would that be unacceptable? Are there scenarios on OS X where the Finder isn't available but the trash is? http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-... Having to call out to an external program via Applescript feels rather icky, but given the difficulty of simultaneously supporting multiple OS X versions, satisfying the rules of Apple Store, and using Apple's APIs, I don't see any alternative. -- Steve

On Dec 31, 2014, at 0:47, Steven D'Aprano <steve@pearwood.info> wrote:
I already answered this briefly earlier in the thread, but in more detail: 1. If you don't have a GUI login session (e.g., because you've ssh'd into the box, or are running as a LaunchDaemon or a traditional detached script), you have no Finder in your session to script. 2. If you're running in a sandbox that doesn't have scripting rights for Finder added in--as will be the case in most Mac App Store apps--you can't script Finder. Of course you can include a sandbox exception to allow for this, but the only way to do this that's allowed in the App Store requires 10.8.0 to run and, IIRC, 10.8.2 to run without serious bugs. And the Python docs would have to explain that any app that may call shutil.trash must use a custom sandbox. 3. If you're running in a sandbox, you can't execute osascript; if you need pre-10.8 compat, you can't use the modern APIs; this means the only thing you can do is include a compiled AppleScript as a resource and use the 10.7 APIs to execute it--which still requires 10.7, and it seems to have problems with the new 10.9.5/10.10.1 code signing rules (although it may be easy to work around that; I haven't tried). 4. If you're using a third-party Finder replacement like PathFinder, it won't respond to messages sent to Finder. 5. If you've killed the Finder to save resources (pretty rare nowadays, but still legal) you'll relaunch it. 6. The Finder may pop up a progress bar--or, worse, an auth dialog, or a "delete immediately?" dialog, on behalf of your app. This could be surprising from a console program. 7. Depending on how you do the scripting, your request may time out after 5 or 30 seconds, leaving you no way to know whether the trashing succeeded. 8. If Finder is hidden, it may unhide. If another app is fullscreen in front of Finder, it may slide offscreen, or Finder's dialogs may just be invisible. (You can guarantee a fix for the latter by forcing the former.) 9. If your session or system is screwed up and you're trying to shut down cleanly or repair things, Finder could be unresponsive or even hung. For example, on 10.6, if you have for separate moves blocked on an NFS mount, a trash will block forever, but (assuming you aren't trying to trash something on the same mount) there's nothing stopping you from trashing on your own. In some cases, this may all be fine. For example, if you're writing a trash script that you intend to use on your specific computer in local Terminal sessions only, you may _want_ the Finder to take care of auth, no-trash mounts, and other issues for you, and show you progress on long tasks. Fortunately, you can already write that script with about 5 lines of ScriptingBridge code that will run on the Python 2.7 with PyObjC and other extras that Apple has pre-installed as /usr/bin/python on all 10.6-10.10 versions, so you're already covered.
http://www.anthonysmith.me.uk/2008/01/08/moving-files-to-trash-from-the-mac-...
Even if you want to use osascript, you don't want to do it that way. Build a single AppleScript command that trashes all of your files in a single call. One of the other blog posts someone linked earlier in this thread shows how to do that. Also, for the usual reasons, os.system is not the best way to call osascript (you'll end up with a success retcode even on failure, quoting for both the shell and AppleScript is double the fun, etc.).
Since this doesn't satisfy the rules of the App Store, it doesn't solve the problem it's intended to. But see my previous email. For almost any app, there is a much better alternative. The problem is that there's no _single_ alternative that's good for most apps; there are different ones for different use cases. In particular, for just writing a little trash script for use on the command line on your own 10.8+ computer, the NSFileManager version is even simpler than the scripting version (again using Apple's pre-installed 2.7 with PyObjC), and without any of its problems. For older computers, it's a _bit_ more work with the FS function, but still very simple, and it's a known problem that multiple people have solved years ago (and linked to earlier in this thread), so Python 3.5 doesn't need to retroactively solve it.

Hi everyone, I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.) As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it. (And of course, Python has lots of OS-specific modules.) So maybe we can add a function to the `os` module that sends a file to the recycle bin, and a constant that will say whether the current OS supports this? Then we could have code like this: if os.RECYCLE_BIN_SUPPORT: os.recycle(my_file) else: os.remove(my_file) What do you think? Thanks, Ram. On Thu, Jan 1, 2015 at 1:14 AM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Jan 1, 2015 at 9:37 PM, Ram Rachum <ram@rachum.com> wrote:
Most OS-specific stuff is in the os module, but path handling is inherently platform-specific to some extent, so either place would be reasonable.
Or: try: delete = os.recycle except AttributeError: delete = os.remove delete(my_file) ChrisA

On Thu, Jan 01, 2015 at 12:37:55PM +0200, Ram Rachum wrote:
The os module is for low-level operating-system functions. "Send to trash" is neither low-level nor part of the OS per se, it is part of the desktop environment. I'm not convinced that this needs to be in the standard library, but if it is, I think that the os module is completely the wrong place for it. I think, in order of preference: 1) shutil 2) pathlib is the right place. (The actual implementation for the move_to_trash function could come from another, platform-specific, module.)
"Recycle" is not a good name for the function, because it doesn't recycle the file. It does the opposite of recycle: it prevents the file from being deleted and over-written. I think an explicit name like move_to_trash is better than something misleading like "recycle". No need for a special flag to check if the function exists, just check for the function: try: f = shutil.send_to_trash except AttributeError: f = os.remove f(my_file) But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution? -- Steven

I take your point about it not being in the `os` module, I have no issue with it being in shutil or pathlib. I also agree about the name, "recycle" doesn't really make sense. Regarding your question: "But I'm still not sure that this needs to be in the standard library. For such a specialist need, what's wrong with using a third party solution?" Well, I can use a third-party solution for everything, but it would be nicer for me if it was in the standard library because then I could count on it always being there and the API not changing. I could say the same thing you said about the `webbrowser` module, that opening a new browser tab is quite a specialist need, but I do enjoy having it available in the standard library and I feel the same about sending a file to trash, which I think is a basic feature that's we've all been taking for granted for the last decade or two. Happy new year, Ram. On Thu, Jan 1, 2015 at 2:35 PM, Steven D'Aprano <steve@pearwood.info> wrote:

On Thu, Jan 01, 2015 at 02:43:09PM +0200, Ram Rachum wrote:
Which is exactly the point. It has been said that the standard library is where good modules go to die. Do you think that the "move to trash" API is ready to die? I don't think so. I think that there is a lot more than just moving files to the trash. I've spent a little bit of time over the last day or so thinking about what sort of trash-related functionality I would like, as a Linux user: - move to trash, of course, but which trash? - return the name of the trash directory in my home directory - list trash directories on other partitions and disks - list trash directories for all users (if I have read permission) - empty one or more of those trash directories - optionally only deleting sufficiently old and/or large files - validate and/or repair the trash metadata - report how much space is used by the trash, with or without metadata - support legacy .Trash directories as well as .local/share/Trash For some operations, I would want to specify either a specific trash directory, or all trash directories for a given user (defaulting to the current user), or all trash directories for all users. (The later would only work if I was running as the root/Administrator user, of course.) The point being that the API for all this functionality isn't obvious, and it is very unlikely to be so stable that it is ready to move into the standard library. -- Steven

On Thu, Jan 01, 2015 at 11:35:26PM +1100, Steven D'Aprano wrote:
Maybe the right way to do this is to create a new module, stdlib or not, for desktop-related APIs. Besides a `recycle` function this might, for example, include a subset of pyxdg's functionality. -- Markus

On Jan 1, 2015, at 13:47, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
That's a great idea. To do something as simple as getting special directory names or creating a shortcut/alias/bookmark (as opposed to a symlink) or, yes, trashing a file in a cross-platform way requires something way too heavy-duty (like Qt or JUCE)... or copying code off some blog (that doesn't actually work on localized Windows or in a sandboxed OS X app or on non-Debianish Linux, but you won't know that until you think to test it). A third-party module can get away with 90% support. Most Mac users are on 10.8+, most apps don't fork, most Linux users who care about the desktop have a GNOME-compatible one; etc.

On Thu, Jan 1, 2015, at 07:35, Steven D'Aprano wrote:
What is the correct place for high-level functions that need to be implemented in a different way on each OS? I think the problem is that some people view os as "low-level functions, like the ones found in unistd.h" and other people view it as "where to put stuff that needs a separate implementation per platform". The actual description in the documentation is "This module provides a portable way of using operating system dependent functionality." Nothing about low-level.

On Jan 1, 2015, at 11:37, Ram Rachum <ram@rachum.com> wrote:
Hi everyone,
I'm a complete ignoramus when it comes to Mac, so I can't say anything about that, but as I understand the discussion, there are problems with implementing this on Mac. Okay. So maybe this should be available only on Windows and Linux? (If it's not a problem to do on Linux.)
On Linux, it's not exactly like Mac, but it's similar. Linux has no notion of trash at all; desktop environments like GNOME do. And if you want it to work exactly like the way GNOME does, you call the API via GObject; otherwise you have to implement a complicated approximation based on rename that won't be fully functional. Really, on all three platforms, the answer is the same: you call an API from your desktop manager. The only difference is that Windows merges desktop and kernel together to the point where they can't be separated--so Python is already using Win32, unlike Cocoa or GObject.
As much as I don't like having functionality that's only available on certain OSs, it makes more sense than depriving Windows users of functionality that they could be using just because Mac doesn't support it.
No one is "depriving" Windows users of anything. Python comes with pip, send2trash installs without needing a compiler (at least I hope so; if not, someone can write a new third-party module that does). Not everything needs to be in the stdlib. So far, there aren't many third-party modules mentioned in the stdlib docs, but that's only because we didn't have preinstalled pip, wheels, etc. until 3.4.
There's no such flag for anything else platform-contingent. EAFP.

On Thu, Jan 01, 2015 at 04:37:52PM +0100, Andrew Barnert wrote:
Actually it seems that the trash location is standardized by the XDG Desktop Specification: https://github.com/kevva/xdg-trash http://www.ramendik.ru/docs/trashspec.html

On Jan 1, 2015, at 17:00, Markus Unterwaditzer <markus@unterwaditzer.net> wrote:
Yes, as I just said: Linux has no notion of trash; desktop environments do. The FreeDesktop standard is not Linux; it's a standard that at least two desktop environments (which also work on multiple non-Linux systems) support. On most Linux systems, you will have Gtk libraries; on most Linux systems without Gtk libraries, you will not have a FreeDesktop-compliant trash. So, using GObject is a good 90% solution (like using NSFileManager is a good 90% solution on Mac). Of course you _could_ implement that full spec yourself in Python, right? Well, read it. There are references to things "the implementation" may do, like providing a way for the administrator to disable sticky-bit checking for filesystems that don't support it. Do you know how to check whether the administrator has disabled sticky-bit checking on a filesystem? Or do you just want your code to say a file can't be trashed, or trash by moving it across the network to ~, even when the desktop and other apps have no problem trashing it properly? For that matter, do you know how every FreeDesktop-compatible system wants to report a problem to the administrator? You may meet the XDG Trash spec and still not satisfy user expectations. And it'll be a lot of work. In other words, "you have to implement a complicated approximation based on rename that won't be fully functional". Is that really a better solution than using GObject? If you're writing your own portability library like Qt or JUCE, probably; otherwise, I doubt it.

On Thu, Jan 1, 2015, at 17:59, Andrew Barnert wrote:
A python implementation would be an independent implementation, and not bound by another implementation's (even on the same machine) implementation-specific decisions. There is therefore no reason the python implementation couldn't simply not provide such a method, and therefore any .Trash directory without the sticky bit set does not pass the checks, and all that will happen is .Trash-$uid gets used instead - no harm done.

On Jan 2, 2015, at 14:43, random832@fastmail.us wrote:
Then a Python implementation would be a bad implementation. The whole point of XDG is to make it easier to write desktop software that's consistent with other desktop software. So if you write some Qt software and I run it on a GNOME system, Qt follows (as best it can--which is better each version) the GNOME rules, and your GNOME-based settings, rather than ignoring them and pretending it's on a KDE system.
Except that if you can't create a .Trash-$uid, then you can't trash files on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.)

On 3 Jan 2015 06:06, "Andrew Barnert" <abarnert@yahoo.com.dmarc.invalid> wrote:
Except that if you can't create a .Trash-$uid, then you can't trash files
on that filesystem even though Nautilus, etc. can. (And, needless to say, if you try to implement any other trash functionality, like listing trash contents or undeleting, you're going to be wrong there too.) It sounds to me that what is needed before changes to pathlib can be discussed is a reference third party module that appropriately respects desktop conventions (whether Windows, Mac OS X, GNOME, KDE, or other XDG based Linux window manager), and associated clear documentation on how to delete files in a way that integrates well with underlying platforms. The standard library isn't the place to work through that standardisation activity, although a PEP is certainly a reasonable place to track it. Guido's work on PEP 3156 and the asnycio module is a good point of reference here, as is Antoine's work on pathlib itself. Once a pathlib independent reference implementation of imperative desktop trash management exists, and there is broad acknowledgement that it is a suitable candidate for inclusion in the standard library (perhaps as part of shutil, perhaps as a new independent module, depending on API complexity), *then* it becomes feasible to discuss adding trash management support to the pathlib abstraction layer. Regards, Nick.

On Thu, Jan 1, 2015, at 10:37, Andrew Barnert wrote:
There is an actual standard for the Trash system that all three major desktop environments implement, so your "complicated approximation based on rename" can actually be correct rather than an approximation. If that's not exactly like how GNOME does it, then GNOME is incorrect.

On Jan 2, 2015, at 14:36, random832@fastmail.us wrote:
First, XDG specifications are explicitly not standards (and XDG Trash is a draft extension proposal to those specs) so there is not an actual standard. But, far more importantly, there are large areas that the spec leaves open to the implementation, or open for future expansion, that GNOME (and KDE) fills in with appropriate behavior. I gave one example earlier in the thread, which I picked by pointing my finger at one random paragraph from the spec. If you read through it, there are plenty of others. If you call the APIs via GObject or Qt on a GNOME or KDE system, you will get the exact same behavior as everything else on the system. If you implement it yourself and ignore what GNOME and KDE do, nobody's going to care that you're 100% compliant with a draft spec proposal, they're going to care that your app refuses to trash something that everything else can trash (or, worse, that your app trashes it by copying 8GB over the network).

There’s always the Cocoa APIs for moving files to the Trash, if its fine to depend on OSX 10.6 or later (as you mentioned below). Whether or not you’d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea. BTW. I’m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we’ve had with proxy support in the _scproxy extension (basically resulting in “random” crashes when using urllib in a child proces started with os.fork because Apple’s system frameworks don’t support that). Ronald

On Dec 31, 2014, at 9:20, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Yes, but the 10.6 Cocoa APIs both have problems; the useful one requires 10.8. So as long as Python standard builds are 10.6… (I suppose you could have it in the stdlib but only available at runtime on 10.8+, but I think that would be weird.)
Whether or not you’d want to do that in the stdlib is a different question. My point was that just using os.rename move a file to the trash is a bad idea.
Yes, and therefore there just _is_ no low-level, suitable-for-os, function that would help for a higher-level shutil/pathlib function.
BTW. I’m not really in favour of using either FSMoveObjectToTrash or the Cocoa APIs in the stdlib, given the problems we’ve had with proxy support in the _scproxy extension (basically resulting in “random” crashes when using urllib in a child proces started with os.fork because Apple’s system frameworks don’t support that).
I forgot about that, but you're right, that's another potential reason not to put this in the stdlib. Anyway, maybe what we need is something in the docs explaining the various ways to do it (with links to PyPI projects as appropriate) with the pros and cons of each. What I wrote before isn't right, because it doesn't take into account the forking issue. It could be easily extended, although that might get a bit long for stdlib documentation.

On Tue, Dec 30, 2014, at 18:12, Andrew Barnert wrote:
Not having one isn't acceptable either. Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind by applications with badly-written direct implementations of the way you do it on Unix. Can you define "doesn't quite meet Apple's HIG" rules and link the relevant section of the HIG?

On Jan 2, 2015, at 14:24, random832@fastmail.us wrote:
I'm not sure what "the way you do it on Unix" means. There is no Unix/POSIX/etc. notion of trashing a file. Of course some Unix desktop environments (going back at least as far as NeXTStep) have had such a thing, but that can't be what you mean. If you're referring to XDG, I don't know of a single application that uses a badly-written direct implementation of the XDG trash spec on Mac or Windows. Do you? More importantly, which of those other options do you think is better than continuing to have no trash API? Drop 10.6-10.7 compat? Use a deprecated API and break Mac App Store compatibility? Require all scripts to have a runloop?
Can you define "doesn't quite meet Apple's HIG" rules and link the relevant section of the HIG?
Of course not, because I have no idea what exactly you'd write in place of calling the appropriate API. Give me a design or some code, and I'll tell you what it does wrong.

On Fri, Jan 02, 2015 at 08:24:26AM -0500, random832@fastmail.us wrote:
Not having one [move to trash function] isn't acceptable either.
I'm not sure about that. Do other languages offer a move-to-trash functionality? Say, PHP, Javascript, Java, Ruby, Boo, Go, Lua? If Python is the odd one out that DOESN'T offer such functionality, and all the others do, then you might have a strong case. Otherwise, consider that the great bulk of scripts that are written in Python don't do any file management at all, other than perhaps reading or writing a few files. So even if Python gained this functionality, it wouldn't have any impact on most scripts and applications.[1]
Because what _not_ having a cross-platform wrapper gets you is windows and mac left behind
Really? I would expect that Windows is probably the *easiest* platform to implement this functionality, as it has a single, standard, stable API for moving files to trash. (Or so I am lead to believe.) You can probably use that API via ctypes, or via pywin32. Getting OS X right is very complicated, as Andrew has so ably explained. In Linux and other Unixes, the situation is like OS X, only more so since there's no one single authority. You have a choice of desktop environments, which may or may not provide a move-to-trash API, including no desktop environment at all. Gnome provides an API for moving to trash, but I don't know how well it supports the freedesktop standard; KDE supports the freedesktop standard, but I don't know if it provides an API that can be called. XFCE has partial support. It seems to me that "move to trash" is at least straight forward on Windows, but tricky and complicated everywhere else. Or, there is always the option of using a third-party module: https://pypi.python.org/pypi/Send2Trash [1] With one possible exception. If Andrew is correct, then the wrong implementation of move-to-trash might effect in a bad way anyone trying to put their app on the Apple Store, whether they use that move-to-trash functionality or not. -- Steve

On Sat, Jan 3, 2015 at 12:50 PM, Steven D'Aprano <steve@pearwood.info> wrote:
I've always thought of a trash can as an inherently human-controlled feature. When you go in using a GUI file manager and delete a file, it lands in trash. When you use another application, it's always felt surprising - for instance, BitTorrent on Windows deletes things by moving them to trash. (Which has abysmal performance if you select a huge bunch of things and wipe them all out at once; trashing gobs of stuff across thousands of files seems to trigger an O(N*N) repeated search for which files "fall out" of trash - not to mention that this probably means a fairly arbitrary subset of the freshly-deleted is in trash, and little or nothing from previous deletions remains.) Most apps should do file management at the file system level, unless they're specifically replicating/replacing the file manager. So a "move to trash" feature is something that (a) isn't going to be needed by most apps, and especially (b) would be an attractive nuisance. A third-party module has that extra little hump of having to be consciously sought, so it can be there for those who want it but not waving itself around at those who really should be looking at os.remove(). I'm -0.5 on adding a "move to trash" function to the stdlib, ever, and a definite -1 on adding one now. ChrisA

On 03/01/2015 02:18, Chris Angelico wrote:
As, for example, holding down the SHIFT key on Windows and skipping the recycle bin completely? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Jan 3, 2015, at 3:18, Chris Angelico <rosuav@gmail.com> wrote:
I think this may be an accident of the fact that cmd.exe didn't include a trash command, neither did OS X, and the idea was dropped from xdg-trash; we've all been trained for a couple decades now that you can't trash from the command line. I'm not sure that's a good thing. But it is nevertheless a thing that can't be ignored.
In some GUI applications, it does make sense to trash things via a menu or other user interaction. (Except maybe when you're using Apple's versions/history thingy or some similar mechanism.) One of the things Aquamacs adds to the standard emacs distribution is a move-to-trash command that you can use in diredit mode, for example. Although I guess you could argue that counts as "unless you're replacing the file manager", and maybe the same is true of most other such examples... But not all. When I worked on TuneUp, we had a feature to move selected duplicate tracks from quarantine to trash, which I thought was stupid and unnecessary (the quarantine folder is already effectively an app-level trash can...). But when we removed it, we got hundreds of user complaints, and a nasty blog post saying we were trying to deliberately destroy out users' music collections on behalf of the RIAA or something. And:
The Mac version also does the move-to-trash; it has perfectly acceptable performance. (Actually, old versions were terribly slow at trashing a torrent with a lot of top-level files or dirs, but newer versions seem to do a single operation per torrent instead, which works fine.) And I'm actually glad it does this. At least twice, I've downloaded an ISO, mounted it, then trashed the torrent with all files; if they were deleted instead of trashed, I would have had to repeat the download to update my VMs... You could argue that I'm an idiot for doing this, and you'd be right, but I'm probably not the only such idiot; in fact, that's kind of what trash is for in the first place.
I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib. Whether that's send2trash or a complete trash management module as Steven (I think?) envisioned or a reference to the platform APIs and the various ways to get at them, it would be useful. (In fact, some prominent reference to PyWin32, PyObjC, GObject, and PyQt/PySide for non-GUI-but-still-desktop apps in the docs might be helpful for lots of reasons.) Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem.

On Sat, Jan 3, 2015 at 9:28 PM, Andrew Barnert <abarnert@yahoo.com> wrote:
The Mac version also does the move-to-trash; it has perfectly acceptable performance.
Good. The note about performance was a mere parenthesis, and the last time I did anything like that on Windows was on XP, so it's quite possibly different on Win7/8.
I agree with this, but it still might be nice to mention that third-party module in shutil and/or pathlib.
Maybe. Is there a mention of other third-party packages in stdlib docs? It's a slippery slope; which ones deserve mention? Maybe just something like "Note that moving objects to the trash can is possible in OS-dependent ways; packages are available on PyPI to make this easier", without naming any.
Honestly, I think one of the big things people want is the ability to write the "trash" tool for use on the command line that all of the major desktop systems are missing. It feels like it should be a 3-line Python script, and the fact that it isn't possible looks like a weakness. But really, I don't think that's a problem. OS X comes with a Python with PyObjC; a GNOME or KDE system can install python-gobject or PySide in one apt-get/urpmi/port/etc. command; Windows' function is accessible via the built-in ctypes. I have such scripts on my Mac and Fedora boxes; the fact that they're _different_ 3-line scripts is hardly a serious problem.
As an explicit "trash" tool, sure. Typing "trash some_file" is under human control just as much as hitting Delete without holding Shift is. I just don't think it should be the obvious way for an application to remove files. ChrisA

On Jan 3, 2015, at 11:39, Chris Angelico <rosuav@gmail.com> wrote:
The impression I get is that this is a slope that Python (or at least Nick Coghlan) wants to start sliding down--with the advent of wheels and pre-installed pip, keeping things out of the stdlib but still accessible for most users is now possible, and often desirable. But I could be putting words (or complete ideas) in people's mouths.

On 3 January 2015 at 21:04, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
Your parenthetical qualification is correct - while I personally think providing authoritative recommendations of third party modules is a good idea, this is far from a universally held opinion amongst the core development team. My own caveat on the idea is that any such explicit recommendations should only cover projects that potentially *could* be included in the standard library from an API design perspective, but we don't want to include them (yet, or perhaps ever) for project governance reasons, such as their update cycles for feature releases needing to be faster than those of the standard library. The "pro" case for this approach is primarily about lowering barriers to entry: for folks just learning Python, "not in the standard library" is currently functionally equivalent in many cases to "Python can't do this". New users often won't yet have the skills needed to do their own research and evaluate competing solutions to their problem, and teaching someone how to find a module on the internet, evaluate the trustworthiness of the people publishing it, get it onto their computer, and update it appropriately over time is genuinely difficult (even with pip provided by default - that helps with the last two steps, but the first two are still genuinely difficult, in a way experienced open source developers often take for granted). Providing authoritative recommendations of third party projects in the standard library documentation helps resolve that situation without needing to resort to making the standard library larger (with the additional long term maintenance complications the latter approach entails). You see a similar dynamic operating in a slightly different way with a project like Django - when the Django developers are significantly more experienced in web development than you are, their opinionated, all-inclusive approach to web framework design is a *feature*, not a defect. It's only when you become more experienced with web service development yourself, and need to deal with problems that don't fit neatly into the SQL-backed three-tier web application model, or need to integrate with other systems which weren't built with Django in mind, that less opinionated frameworks like Flask or Pyramid become interesting - those frameworks leave many more decisions to the developers using them, which is exactly what you want in many situations as an experienced developer, but constantly asking people to make decisions they aren't yet qualified to make poses a huge barrier to entry. It's a potential problem when less experienced folks are put off by the request to make those decisions (since not understanding the questions you're being asked by your tools makes for a terrible user experience), and decide to go learn something else that places fewer demands on them, but it's even worse for everyone trying to deploy and manage the resulting software systems when folks happily make the decisions these advanced DIY integration frameworks ask of them without even realising they're not yet qualified to make them. djangopackages.com is also incredibly useful as a tool for finding peer evaluations of projects in the Django ecosystem. I'm not yet aware of any successful attempts to expand that model to other plugin ecosystems, but the core grid concept of assembling a set of criteria, providing answers across a set of related projects, and publishing the results to provide a starting point for anyone doing their own research in that topic area is a wonderful one. The "con" case is that providing default recommendations can be controversial - when there are multiple competing projects in an area, being the default recommendation of the CPython core development team can be a huge popularity boost, since reputation is to some degree transitive. If someone trusts CPython, and we trust another project enough to recommend it, then a lot of folks will use that recommendation as a substitute for doing their own research (even when they *can* do that research themselves, the default recommendation is likely to be good enough, letting them defer investigation of alternatives until after they've tried the default option). I suspect that objection could potentially be dealt with the same way we deal with any other standard library design decisions: if a module maintainer is comfortable making a recommendation based on their own expertise they can, but broader recommendations spanning several modules would require at least a discussion on python-dev, and potentially even a PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Jan 4, 2015, at 7:05, Nick Coghlan <ncoghlan@gmail.com> wrote:
The question here, then, is how this applies to desktop/shell functionality like moving to trash, administering trash in all the ways someone (Steven?) wanted, other features that someone else raised (aliases/bookmarks/shortcuts/desktop files), and maybe additional stuff (accessing the desktop's extension/file-type mapping). Some of that stuff is already in the stdlib (or might be confusing to novices--e.g., IIRC, on XDG desktops, some things that are done with aliases on Mac are done with desktop files, but in some cases you just use symlinks, so a Linux novice might not even realize he's not writing cross-platform code...). For a GUI app, it's not at all clear that choosing Qt or Wx or one of the platform-specific libraries automatically solves all of these problems for you, while choosing Tk doesn't. For a command-line app, it's even worse. Most Windows users probably have no idea they're asking for desktop integration rather than OS support when they ask how to move a file to the trash or create a shortcut, because Windows intentionally buries that distinction. So when they see that neither os nor shutil nor anything else in the stdlib can help them, they're not going to think, "Oh, I need to go find a platform-specific or cross-platform desktop integration library." (And they're also not going to think "Well, Perl and Ruby and Node don't have these either, so why should I expect Python to?") So, I think in this case it would be useful to recommend something. The question is _what_ to recommend (and where). I think for platform-specific code there are pretty clear winners (PyWin32, PyObjC, GObject, and PyQt/PySide, for the big four). Some might argue for including the XDG package (e.g., if you want to do the same thing as one of the XDG-utils tools, you don't need Gtk or Qt for that), and Mac users might also want a link to the ScriptingBridge guide or something (because some things that are done by API on Windows are done by scripting System Events or Finder or something else), and people using a third-tier desktop on Linux may not have Gtk or Qt libraries to bind to, etc., but I think those are minor questions compared to whether it would be useful to have pointers to these packages for novices. (I think it goes without saying that none of these packages should be added to the stdlib, right?) But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither. And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong.

On 4 January 2015 at 20:32, Andrew Barnert <abarnert@yahoo.com> wrote:
Right, a useful first step might be a "Tools for writing rich client applications in Python" overview page. That's out of scope for any of the current *.python.org documentation set (it doesn't fit under docs.python.org *or* packaging.python.org, nor anywhere on the main site), but if it's written in an appropriate tone (i.e. making it clear that these are default starting points, and recommendations for further evaluation, rather than universally applicable definitive conclusions), then a future "ecosystem.python.org" guide could start the same way p.p.o did, as a separate Sphinx project hosted on ReadTheDocs that was later blessed with a python.org subdomain. https://packaging.python.org/en/latest/deployment.html is an example skeleton of such a page for package deployment tools. We haven't filled that one *in* yet, but hopefully it conveys the general idea. https://packaging.python.org/en/latest/extensions.html is another one reviewing binary extensions, and some of the tools available in that space. For those curious as to "Why not the wiki?", a Sphinx project hosted on a pull request capable service like GitHub, BitBucket or Kallithea offers a much nicer workflow for reviewing of proposed changes, together with an integrated issue tracker for submitting proposals for updates (https://github.com/pypa/python-packaging-user-guide/ is the project behind packaging.python.org, for example).
But for cross-platform code there really isn't an answer. (Well, PyQt/PySide would work, but it's still non-trivial to install and configure on Mac and Windows, and it's overkill.) So, maybe what we need here is for someone to first create the desktop package, and then we can worry about whether it should be added to the stdlib, recommended in the stdlib, or neither.
Just describing the problem space can be incredibly valuable for newcomers - one of the hardest things to learn as a beginner is the difference between "this is hard because I don't know what I'm doing" and "this is hard because I'm working on an unsolved problem" (with "this is deceptively easy because I'm setting myself up for problems later on" being an interesting variant that's hard to detect no matter how experienced you are). I see this is a *lot* when working with the scientific community - they actually have much higher expectations of cross-platform tooling than professional developers do. From the outside, the political, personal and commercial meta-issues that combine to make cross-platform standards development an unholy nightmare aren't immediately obvious, and nor are the trade-offs between "lowest common denominator" (or Java-style "reinvent all the wheels") functionality that works everywhere, and providing access to advanced platform specific functionality.
And that still leaves open the possibility that some small subset of this functionality might be separable enough and useful enough to put in the stdlib, or in a small package linked from the stdlib. After all, the equivalent of start/open/xdg-open is already there for some platforms, and webbrowser.open and some of the stuff in shutil. I don't think moving to trash is part of such a subset (for the reasons I gave earlier), but I could be wrong.
While I think it's better to leave the question open rather than try to pre-empt an answer, I do believe we can at least say *now* isn't the right time to be adding "move to trash" functionality to the standard library - the functionality isn't mature enough within the broader ecosystem to determine an appropriate answer. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sun, Jan 04, 2015 at 11:01:08PM +1000, Nick Coghlan wrote:
The concept of "proposed changes" goes completely against the grain of community-managed content. Imagine if Wikipedia required you to make pull requests. The question to be asked is not whether "pull requests" are "nicer" than a wiki, but whether this information needs to be owned by a single person (or a small group thereof), and hence use the packaging.python.org module, or owned by the whole community, and hence use the wiki model. At the moment, the wiki is horribly underused. I'm not blaming anyone for that, I'm just as much to blame as anyone else. I have made far more contributions to the ActiveState recipes than I have to the Python wiki. But I'd really like to see the wiki become the second place people look for information, after the official docs. And hopefully there won't be a need for a third place :-) Although the quality of contrabutions is variable, the PHP community has made a very interesting decision to integrate community-supplied information with official docs: http://php.net/manual/en/function.dechex.php I wonder how we might do something similar? -- Steve

On the other hand, a number of pages on the Wiki are either completely out of date because nobody cares about them or need to be locked because too many people care about it and they fight over which project should be named as recommended. Wiki content can be good for more factual based information but I think the format is relatively poor for things which are more opinion based. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Donald Stufft <donald@stufft.io> writes:
In recent years (I noticed this since about 2013), the Python wiki does not offer an “Edit” function when I am logged in; every page presents “Immutable Page” instead. That has prevented me from improving pages I've wanted to. And no, I haven't gone to the effort of constructing a bug report for this. Perhaps the same is true for many other logged-in users as well? -- \ “It is well to remember that the entire universe, with one | `\ trifling exception, is composed of others.” —John Andrew Holmes | _o__) | Ben Finney

On Sun, Jan 4, 2015 at 7:22 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Ben (and others), I just noticed this small paragraph on the FrontPage: If you want to edit a page and have just signed up, or you find that you can no longer edit a page that you could edit before, please write to the pydotorg-www mailing list, stating your account name and your intended edits and we'll add you to the EditorsGroup. I suspect that has something to do with your inability to edit. Somewhere along the way, someone must have made the executive decision that open editing no longer worked (too much spam?) and that editing needed to be a by-invitation or by-request sort of thing. I'd add you, but am getting nothing but Gateway Timeout responses at the moment. Skip

On 05.01.2015 15:49, Skip Montanaro wrote:
This was discussed and announced on the pydotorg-www list at the time. The step was necessary, because of excessive spam and vandalim - editors were spending more time cleaning up than editing pages and were losing interest in the wiki altogether because of this. The first step was to require logins for editing. This worked for a (short) while. The second was adding more complicated textchas, the third step was requiring asking for editing rights on the pydotorg-www mailing list. Which is a little annoying and some work for the admins, but has resulted in the amount of spam to go down to zero. We usually grant the editing requests within a day. FWIW: I've maintained the wiki VM ever since it came under attack in 2013.
I'd add you, but am getting nothing but Gateway Timeout responses at the moment.
If you get those, simply try to resend/reload. The wiki will notice duplicate editing requests. In my experience, the editing requests typically make it through to the wiki the first time and you get a duplicate editing warning on the second one. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2015)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Mon, Jan 5, 2015 at 9:27 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Unfortunately, I couldn't even get to the EditorsGroup page to edit it. Now, about an hour later, I finally succeeded. Ben, send me your wiki login and I'll add you. MAL, I thought the spam problem had mostly been solved by Martin's modification several years ago to require a certain time to pass between a new login being created and their first attempt to edit. Skip
participants (24)
-
Andrew Barnert
-
Antoine Pitrou
-
Ben Finney
-
Chris Angelico
-
David Wilson
-
Donald Stufft
-
Ethan Furman
-
Georg Brandl
-
Greg Ewing
-
Guido van Rossum
-
M.-A. Lemburg
-
Mark Lawrence
-
Markus Unterwaditzer
-
Nick Coghlan
-
Paul Moore
-
Ram Rachum
-
random832@fastmail.us
-
Ronald Oussoren
-
Skip Montanaro
-
Stephen J. Turnbull
-
Stephen J. Turnbull
-
Steven D'Aprano
-
Victor Stinner
-
Wes Turner