Have os.unlink remove junction points on Windows

Hi all, I posted this bug a while back, but haven't had any feedback on it, so I figured I'd open discussion here: http://bugs.python.org/issue18314 I want to have os.unlink recognize NTFS junction points (a prequel to proper symbolic links) and remove them transparently. Currently, os.unlink on a path to a junction point will fail with a WindowsError and the error code 5, access denied. Does this sound controversial, or would anybody be interested in reviewing a patch to this effect? Thanks, - Kim

On Sun, Aug 25, 2013, at 14:26, Kim Gräsman wrote:
What happens if you call os.rmdir? And just out of curiosity, what happens if you call msvcrt's _wremove and _wrmdir functions? While we're on the subject of os.remove, can someone explain to me why it doesn't work on directories in Unix? That's the main difference between the C function of that name vs unlink in POSIX, and there doesn't seem to be a "remove a file or directory" function in os at all on unix systems as it stands (whereas both of them seem to be able to remove directories on windows). I'm almost more bothered by the fact that it works on Windows and not on Unix (and a bit by the fact that the "remove" name was used without actually implementing the behavior or calling the POSIX "remove" function) than by the functionality not existing in the first place. But it makes much more sense to add the functionality on Unix than to remove it on windows. Alternately, we could create a distinction between unlink and remove, and only do this in remove.

Hi random, On Mon, Aug 26, 2013 at 6:19 AM, <random832@fastmail.us> wrote:
os.rmdir just delegates to RemoveDirectoryW and so successfully removes junction points too. This seems slightly against the spirit of POSIX: " The rmdir() function shall remove a directory whose name is given by path. The directory shall be removed only if it is an empty directory. [...] If path names a symbolic link, then rmdir() shall fail and set errno to [ENOTDIR]. " - http://pubs.opengroup.org/onlinepubs/009695399/functions/rmdir.html The junction point is removed irrespective of whether the target is empty or not. Junction points are sort of symbolic links, but are removed without error. I can't speak for _wremove and _wrmdir, but I assume they're POSIX compat shims, so they probably follow remove and rmdir, possibly with less understanding of links in general. I wouldn't switch to using os.rmdir, however -- if the path names an actual directory rather than a symlink or a junction point, os.rmdir will delete it, whereas os.unlink will fail with access denied (as I believe it should.)
While we're on the subject of os.remove, can someone explain to me why it doesn't work on directories in Unix? [...]
I have nothing to offer here, sorry. It seems a little dangerous to muck about with the details of os file management, there must be millions of lines of code relying on the current behavior in one way or another. Thanks, - Kim

On Mon, Aug 26, 2013, at 7:42, Kim Gräsman wrote:
Only if it's empty. You could at least replace whatever your _delete_junction_point function is with it.
I just don't like the fact that it's called "remove" but doesn't behave the same as the remove function from POSIX. Of course, remove/_wremove do not remove directories on Windows, and I've found evidence that this was true for some early Unixes as well. And the fact that windows unlink() allows you to remove some (but not all) things that windows considers to be directories is already violating the principle of being thin wrappers around system calls.

On Mon, Aug 26, 2013 at 2:25 PM, <random832@fastmail.us> wrote:
I don't want to remove empty dirs, only links.
Do you mean Python's os.unlink() here, or the Microsoft C-runtime's unlink()? - Kim

On Tue, Aug 27, 2013, at 11:47, Kim Gräsman wrote:
I don't want to remove empty dirs, only links.
So only call it on links.
I'm talking about Python's unlink. The fact that it works to remove directory symlinks is what I was talking about here.

I'm not sure this has anything to do with my original question anymore.
I'm talking about Python's unlink. The fact that it works to remove directory symlinks is what I was talking about here.
I don't know enough about Unix to discuss the merits of changing os.unlink(), but I do believe that it should treat junction points as symlinks on Windows. I'll try to continue that thread on my original subject. - Kim

Hi all, On Sun, Aug 25, 2013 at 8:26 PM, Kim Gräsman <kim.grasman@gmail.com> wrote:
Is there a better place to look for opinions? I'm happy to see Python getting more link-aware on Windows, and I think this could help getting further in that direction. Thanks, - Kim

On 24 September 2013 19:41, Kim Gräsman <kim.grasman@gmail.com> wrote:
Since no one has responded to this for some time I would estimate that not many people particularly dislike your idea. So feel free to open an issue about it on the tracker (after checking that there isn't already an open issue and that your problem is not already solved in the most recent release): http://bugs.python.org/ On the other hand evidently not many people are very enthusiastic about this idea so it's possible that the tracker issue will not go anywhere unless you write the patch yourself. Oscar

Hi Oscar, On Wed, Sep 25, 2013 at 12:06 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Thanks for responding! I opened an issue before posting here: http://bugs.python.org/issue18314 I'd be happy to provide a patch, but I only want to put time into it if there's a reasonable chance it gets committed. That's why I wanted to hear if there were any objections, so I don't end up writing, testing and posting a patch only to end up in quibbles around the general idea. I'm new to Python development; would a concrete patch help move this forward? Thanks, - Kim

On 25 September 2013 12:01, Kim Gräsman <kim.grasman@gmail.com> wrote:
Sorry, I've just looked back over this thread and I see that now.
It doesn't look like anyone else will write a patch so I don't think much will happen if you don't either. I don't know anything about junction points though so I have no idea how likely it is that a patch would be accepted. Oscar

On 25 Sep 2013 21:51, "Oscar Benjamin" <oscar.j.benjamin@gmail.com> wrote:
forward?
My recollection is that permissions around junction points are a little weird at the Windows OS level (so the access denied might be genuine for a regular user account), but if a patch can make os.unlink handle them more like *nix symlinks, that sounds reasonable to me. Cheers, Nick.

Hi Nick, On Wed, Sep 25, 2013 at 2:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks for the heads-up! I haven't observed any differences on XP or Windows 7. Now I'm stuck in an organization where they force all command prompts to be elevated, so it's been a while since I was able to test the more normal cases. - Kim

On 26 September 2013 06:37, Kim Gräsman <kim.grasman@gmail.com> wrote: > I haven't observed any differences on XP or Windows 7. Now I'm stuck > in an organization where they force all command prompts to be > elevated, so it's been a while since I was able to test the more > normal cases. Er, does this not already work? >From an elevated Powershell prompt: PS 08:20 C:\Work\Scratch >new-symlink symps .\ps.vim Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 26/09/2013 08:20 <SYMLINK> symps [C:\Work\Scratch\ps.vim] PS 08:20 C:\Work\Scratch >type symps set shell=powershell set shellcmdflag=-c set shellquote=\" set shellxquote= >From a non-elevated prompt: PS 08:20 C:\Work\Scratch >py Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.unlink('symps') >>> ^Z PS 08:21 C:\Work\Scratch >type symps type : Cannot find path 'C:\Work\Scratch\symps' because it does not exist. At line:1 char:1 + type symps + ~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (C:\Work\Scratch\symps:String) [Get-Content], ItemNotFoundException + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand Paul

Nick, On Wed, Sep 25, 2013 at 2:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Just to follow up on this: I found a lot of Win32-specific code in Lib/test/symlink_support.py to detect whether the current user has the SeCreateSymbolicLink privilege (privileges in Windows are permissions not bound to a resource, rather like global access flags). So I think the "little weird" applies more to symbolic links than to junction points. At least I can't find any privileges that apply to junction points as such. There is a blurb on MSDN about the system-provided junction points from C:\Documents and Settings\... -> C:\Users\..., but that seems to concern actual file system object permissions for those specific paths rather than something general around junction points. http://msdn.microsoft.com/en-us/library/windows/desktop/bb968829(v=vs.85).as... Cheers, - Kim

On 9/25/2013 7:01 AM, Kim Gräsman wrote:
I opened an issue before posting here: http://bugs.python.org/issue18314
I added as nosy the two Windows experts listed in http://docs.python.org/devguide/experts.html#experts I suspect that at least one of them knows enough about junction points to review a patch *were you to write one*.
It is possible that one of the two might have an opinion to the contrary, but after reading the Wikipedia article, https://en.wikipedia.org/wiki/NTFS_junction_point It seems that you ought to be able to delete junction points from Python. That is no guarantee that any particular patch will be accepted.
I'm new to Python development; would a concrete patch help move this forward?
Definitely. I added a note to the issue about testing and Windows versions. -- Terry Jan Reedy

Hi Terry, On Wed, Sep 25, 2013 at 10:51 PM, Terry Reedy <tjreedy@udel.edu> wrote:
OK, I'll get to it when I find time. Last time I looked at it, it seemed pretty trivial, but I need to get the development environment for Python up.
Definitely. I added a note to the issue about testing and Windows versions.
Thanks for your help! - Kim

On Sun, Aug 25, 2013, at 14:26, Kim Gräsman wrote:
What happens if you call os.rmdir? And just out of curiosity, what happens if you call msvcrt's _wremove and _wrmdir functions? While we're on the subject of os.remove, can someone explain to me why it doesn't work on directories in Unix? That's the main difference between the C function of that name vs unlink in POSIX, and there doesn't seem to be a "remove a file or directory" function in os at all on unix systems as it stands (whereas both of them seem to be able to remove directories on windows). I'm almost more bothered by the fact that it works on Windows and not on Unix (and a bit by the fact that the "remove" name was used without actually implementing the behavior or calling the POSIX "remove" function) than by the functionality not existing in the first place. But it makes much more sense to add the functionality on Unix than to remove it on windows. Alternately, we could create a distinction between unlink and remove, and only do this in remove.

Hi random, On Mon, Aug 26, 2013 at 6:19 AM, <random832@fastmail.us> wrote:
os.rmdir just delegates to RemoveDirectoryW and so successfully removes junction points too. This seems slightly against the spirit of POSIX: " The rmdir() function shall remove a directory whose name is given by path. The directory shall be removed only if it is an empty directory. [...] If path names a symbolic link, then rmdir() shall fail and set errno to [ENOTDIR]. " - http://pubs.opengroup.org/onlinepubs/009695399/functions/rmdir.html The junction point is removed irrespective of whether the target is empty or not. Junction points are sort of symbolic links, but are removed without error. I can't speak for _wremove and _wrmdir, but I assume they're POSIX compat shims, so they probably follow remove and rmdir, possibly with less understanding of links in general. I wouldn't switch to using os.rmdir, however -- if the path names an actual directory rather than a symlink or a junction point, os.rmdir will delete it, whereas os.unlink will fail with access denied (as I believe it should.)
While we're on the subject of os.remove, can someone explain to me why it doesn't work on directories in Unix? [...]
I have nothing to offer here, sorry. It seems a little dangerous to muck about with the details of os file management, there must be millions of lines of code relying on the current behavior in one way or another. Thanks, - Kim

On Mon, Aug 26, 2013, at 7:42, Kim Gräsman wrote:
Only if it's empty. You could at least replace whatever your _delete_junction_point function is with it.
I just don't like the fact that it's called "remove" but doesn't behave the same as the remove function from POSIX. Of course, remove/_wremove do not remove directories on Windows, and I've found evidence that this was true for some early Unixes as well. And the fact that windows unlink() allows you to remove some (but not all) things that windows considers to be directories is already violating the principle of being thin wrappers around system calls.

On Mon, Aug 26, 2013 at 2:25 PM, <random832@fastmail.us> wrote:
I don't want to remove empty dirs, only links.
Do you mean Python's os.unlink() here, or the Microsoft C-runtime's unlink()? - Kim

On Tue, Aug 27, 2013, at 11:47, Kim Gräsman wrote:
I don't want to remove empty dirs, only links.
So only call it on links.
I'm talking about Python's unlink. The fact that it works to remove directory symlinks is what I was talking about here.

I'm not sure this has anything to do with my original question anymore.
I'm talking about Python's unlink. The fact that it works to remove directory symlinks is what I was talking about here.
I don't know enough about Unix to discuss the merits of changing os.unlink(), but I do believe that it should treat junction points as symlinks on Windows. I'll try to continue that thread on my original subject. - Kim

Hi all, On Sun, Aug 25, 2013 at 8:26 PM, Kim Gräsman <kim.grasman@gmail.com> wrote:
Is there a better place to look for opinions? I'm happy to see Python getting more link-aware on Windows, and I think this could help getting further in that direction. Thanks, - Kim

On 24 September 2013 19:41, Kim Gräsman <kim.grasman@gmail.com> wrote:
Since no one has responded to this for some time I would estimate that not many people particularly dislike your idea. So feel free to open an issue about it on the tracker (after checking that there isn't already an open issue and that your problem is not already solved in the most recent release): http://bugs.python.org/ On the other hand evidently not many people are very enthusiastic about this idea so it's possible that the tracker issue will not go anywhere unless you write the patch yourself. Oscar

Hi Oscar, On Wed, Sep 25, 2013 at 12:06 PM, Oscar Benjamin <oscar.j.benjamin@gmail.com> wrote:
Thanks for responding! I opened an issue before posting here: http://bugs.python.org/issue18314 I'd be happy to provide a patch, but I only want to put time into it if there's a reasonable chance it gets committed. That's why I wanted to hear if there were any objections, so I don't end up writing, testing and posting a patch only to end up in quibbles around the general idea. I'm new to Python development; would a concrete patch help move this forward? Thanks, - Kim

On 25 September 2013 12:01, Kim Gräsman <kim.grasman@gmail.com> wrote:
Sorry, I've just looked back over this thread and I see that now.
It doesn't look like anyone else will write a patch so I don't think much will happen if you don't either. I don't know anything about junction points though so I have no idea how likely it is that a patch would be accepted. Oscar

On 25 Sep 2013 21:51, "Oscar Benjamin" <oscar.j.benjamin@gmail.com> wrote:
forward?
My recollection is that permissions around junction points are a little weird at the Windows OS level (so the access denied might be genuine for a regular user account), but if a patch can make os.unlink handle them more like *nix symlinks, that sounds reasonable to me. Cheers, Nick.

Hi Nick, On Wed, Sep 25, 2013 at 2:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks for the heads-up! I haven't observed any differences on XP or Windows 7. Now I'm stuck in an organization where they force all command prompts to be elevated, so it's been a while since I was able to test the more normal cases. - Kim

On 26 September 2013 06:37, Kim Gräsman <kim.grasman@gmail.com> wrote: > I haven't observed any differences on XP or Windows 7. Now I'm stuck > in an organization where they force all command prompts to be > elevated, so it's been a while since I was able to test the more > normal cases. Er, does this not already work? >From an elevated Powershell prompt: PS 08:20 C:\Work\Scratch >new-symlink symps .\ps.vim Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 26/09/2013 08:20 <SYMLINK> symps [C:\Work\Scratch\ps.vim] PS 08:20 C:\Work\Scratch >type symps set shell=powershell set shellcmdflag=-c set shellquote=\" set shellxquote= >From a non-elevated prompt: PS 08:20 C:\Work\Scratch >py Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:06:53) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.unlink('symps') >>> ^Z PS 08:21 C:\Work\Scratch >type symps type : Cannot find path 'C:\Work\Scratch\symps' because it does not exist. At line:1 char:1 + type symps + ~~~~~~~~~~ + CategoryInfo : ObjectNotFound: (C:\Work\Scratch\symps:String) [Get-Content], ItemNotFoundException + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand Paul

Nick, On Wed, Sep 25, 2013 at 2:04 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Just to follow up on this: I found a lot of Win32-specific code in Lib/test/symlink_support.py to detect whether the current user has the SeCreateSymbolicLink privilege (privileges in Windows are permissions not bound to a resource, rather like global access flags). So I think the "little weird" applies more to symbolic links than to junction points. At least I can't find any privileges that apply to junction points as such. There is a blurb on MSDN about the system-provided junction points from C:\Documents and Settings\... -> C:\Users\..., but that seems to concern actual file system object permissions for those specific paths rather than something general around junction points. http://msdn.microsoft.com/en-us/library/windows/desktop/bb968829(v=vs.85).as... Cheers, - Kim

On 9/25/2013 7:01 AM, Kim Gräsman wrote:
I opened an issue before posting here: http://bugs.python.org/issue18314
I added as nosy the two Windows experts listed in http://docs.python.org/devguide/experts.html#experts I suspect that at least one of them knows enough about junction points to review a patch *were you to write one*.
It is possible that one of the two might have an opinion to the contrary, but after reading the Wikipedia article, https://en.wikipedia.org/wiki/NTFS_junction_point It seems that you ought to be able to delete junction points from Python. That is no guarantee that any particular patch will be accepted.
I'm new to Python development; would a concrete patch help move this forward?
Definitely. I added a note to the issue about testing and Windows versions. -- Terry Jan Reedy

Hi Terry, On Wed, Sep 25, 2013 at 10:51 PM, Terry Reedy <tjreedy@udel.edu> wrote:
OK, I'll get to it when I find time. Last time I looked at it, it seemed pretty trivial, but I need to get the development environment for Python up.
Definitely. I added a note to the issue about testing and Windows versions.
Thanks for your help! - Kim
participants (6)
-
Kim Gräsman
-
Nick Coghlan
-
Oscar Benjamin
-
Paul Moore
-
random832@fastmail.us
-
Terry Reedy