Hello all, I sometimes write Python scripts which need to work in specific work directories. When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of: class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False Would there be interest in adding something like this to the os module as os.workdir() ? Example: def backup_home_dir(account): with os.workdir(os.path.join('/home', account)): # Create a backup of the account dir, rooted at the account's # home dir restic.backup(repo, '.') Notes: - The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal. - The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes. - The same could be added to pathlib's objects as .workdir() method returning a context manager. Thoughts ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 14 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Tue, 14 Sept 2021 at 20:44, Marc-Andre Lemburg <mal@egenix.com> wrote:
I sometimes write Python scripts which need to work in specific work directories.
When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of:
class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False
Would there be interest in adding something like this to the os module as os.workdir() ?
I've needed (and implemented my own version of) this often enough that I'd be in favour of it. Of course, as you show, it's not hard to write it yourself, so I can live with it not getting implemented, too :-) Paul
I also really like this idea. Though I wonder if it might be a better fit for `contextlib` rather than `os`? Can see arguments both ways (and it's a minor detail, anyway). Alex
On 14 Sep 2021, at 20:55, Paul Moore <p.f.moore@gmail.com> wrote:
On Tue, 14 Sept 2021 at 20:44, Marc-Andre Lemburg <mal@egenix.com> wrote:
I sometimes write Python scripts which need to work in specific work directories.
When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of:
class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False
Would there be interest in adding something like this to the os module as os.workdir() ?
I've needed (and implemented my own version of) this often enough that I'd be in favour of it. Of course, as you show, it's not hard to write it yourself, so I can live with it not getting implemented, too :-)
Paul _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/I6YFWY... Code of Conduct: http://python.org/psf/codeofconduct/
On 14Sep2021 21:43, M.-A. Lemburg <mal@egenix.com> wrote:
- The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal.
This is the source of my concerns. Though of course it applies to any process global state. It would need this stated up front in big letters (as of course does chdir itself). My real concern is that this can leak into other functions whose use then makes whatever uses them to become inherently and unrepairably not thread safe. I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user. I think what would be more useful is a context manager which worked on a threading.local which pushed/popped a reference directory path, and had an open() method which used that (and friends for other pathname based functions). In my own code I'd write this like (sketch, untested): from cs.threads import State as ThreadState class RefDir(ThreadState) def __init__(self, refpath=None): if refpath is None: refpath = os.getcwd() self.refpath = abspath(refpath) def open(self, path, *a, **kw): if not isabs(path): path = os.path.join(self.refpath, path) return open(path, *a, **kw) # calls the builtin open() ... listdir, mkdir, etc etc ... # on reflection, __enter__ and __exit__ would make the below even # more convenient @contextmanager def dirpath(newrefpath): ''' Push `newrefpath` for the duration of the context manager. ''' with self(refpath=newrefpath): yield self.refpath and then use it like this: R = RefDir() .... with R.dirpath('/some/new/place') as newpath: ... with R.open("something.txt") as f: ... work on /some/new/place/something.txt ... In the above, cs.threads.State is a threading.lcoal subclass which is also a context manager whose operation pushes arbitrary attribute values. Great for thread safe execution scoped state. Like this example. All that said, I wrote pretty much exactly what you describe just the other week for umask(). Cheers, Cameron Simpson <cs@cskk.id.au>
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
On 14Sep2021 21:43, M.-A. Lemburg <mal@egenix.com> wrote:
- The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal.
This is the source of my concerns. Though of course it applies to any process global state. It would need this stated up front in big letters (as of course does chdir itself). My real concern is that this can leak into other functions whose use then makes whatever uses them to become inherently and unrepairably not thread safe.
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
I think what would be more useful is a context manager which worked on a threading.local which pushed/popped a reference directory path, and had an open() method which used that (and friends for other pathname based functions).
Hmm. If it's going to push/pop directories, it should probably work with file descriptors rather than path names, and effectively turn all open calls into openat calls. I'm currently not sure what the best way to do that in Python is. ChrisA
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree. I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't goingg to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I think what would be more useful is a context manager which worked on a threading.local which pushed/popped a reference directory path, and had an open() method which used that (and friends for other pathname based functions).
Hmm. If it's going to push/pop directories, it should probably work with file descriptors rather than path names, and effectively turn all open calls into openat calls.
Yah. I even thought about that during my sketch. But (a) I've possibly never used openat() so I'm unfamiliar with it (other than its existence) and (b) I don't think that many other OS calls accept a file descriptor. Which still leaves one with the os.path.join() approach, and a mix might be inconsistent. Cheers, Cameron Simpson <cs@cskk.id.au>
On Wed, Sep 15, 2021 at 9:36 AM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't goingg to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I think what would be more useful is a context manager which worked on a threading.local which pushed/popped a reference directory path, and had an open() method which used that (and friends for other pathname based functions).
Hmm. If it's going to push/pop directories, it should probably work with file descriptors rather than path names, and effectively turn all open calls into openat calls.
Yah. I even thought about that during my sketch. But (a) I've possibly never used openat() so I'm unfamiliar with it (other than its existence) and (b) I don't think that many other OS calls accept a file descriptor. Which still leaves one with the os.path.join() approach, and a mix might be inconsistent.
I believe that, at the OS level, there are *at() versions of everything relevant. But the question is more whether it would work at the Python level. For instance, you can use os.open() with a dir_fd parameter, but I don't think there's a good way to make the builtin open() use a reference directory. Making that happen would probably be a good thing, but a decidedly nontrivial change. ChrisA
On Tue, Sep 14, 2021 at 4:36 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't goingg to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I know where I'd file a bug. :-) "Bug magnet" is an extremely subjective pejorative term. When the *better* way to do things (os.workdir()) is harder than the *easy* way to do (os.chdir()), which is the real bug magnet? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 15/09/2021 01.55, Guido van Rossum wrote:
I know where I'd file a bug. :-)
"Bug magnet" is an extremely subjective pejorative term. When the *better* way to do things (os.workdir()) is harder than the *easy* way to do (os.chdir()), which is the real bug magnet?
The "better way" to handle current working directory is to use the modern *at() variants of syscalls, e.g. openat() instead open(). The variants take an additional file descriptor dirfd that is used as the current working directory for the syscall. Christian
On 9/15/2021 3:02 AM, Christian Heimes wrote:
On 15/09/2021 01.55, Guido van Rossum wrote:
I know where I'd file a bug. :-)
"Bug magnet" is an extremely subjective pejorative term. When the *better* way to do things (os.workdir()) is harder than the *easy* way to do (os.chdir()), which is the real bug magnet? The "better way" to handle current working directory is to use the modern *at() variants of syscalls, e.g. openat() instead open(). The variants take an additional file descriptor dirfd that is used as the current working directory for the syscall.
While I generally agree, the only times I've written a context manager like os.workdir() is when running an executable with subprocess.call(), and the executable requires that its current directory be set to some specific directory. So while I don't use this functionality very often, there are times when nothing else will do. I realize I could handle this temporary working directory with yet another executable (including a shell), but using a context manager is just easier, and I only use this in single-threaded programs. And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either. Eric
On 15/09/2021 09.21, Eric V. Smith wrote:
On 9/15/2021 3:02 AM, Christian Heimes wrote:
On 15/09/2021 01.55, Guido van Rossum wrote:
I know where I'd file a bug. :-)
"Bug magnet" is an extremely subjective pejorative term. When the *better* way to do things (os.workdir()) is harder than the *easy* way to do (os.chdir()), which is the real bug magnet? The "better way" to handle current working directory is to use the modern *at() variants of syscalls, e.g. openat() instead open(). The variants take an additional file descriptor dirfd that is used as the current working directory for the syscall.
While I generally agree, the only times I've written a context manager like os.workdir() is when running an executable with subprocess.call(), and the executable requires that its current directory be set to some specific directory. So while I don't use this functionality very often, there are times when nothing else will do. I realize I could handle this temporary working directory with yet another executable (including a shell), but using a context manager is just easier, and I only use this in single-threaded programs.
You don't have to change the current working directory of your process in order to run a child process in a different working directory. subprocess.call() and other functions in the subprocess module accept a "cwd" argument that lets you run a process with a different working directory. The "cwd" argument is thread safe.
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Speaking with almost 25 years experience in Unix system scripting: Changing the current working directory during the runtime of a process is problematic and can lead to bugs. In general applications should only change their working directory once right after start and not rely on the cwd. It's better to normalize paths, use absolute paths, or *at() syscalls with dirfd. In my opinion a workdir() context manager would only be useful for small quick-n-dirty scripts. Christian
On Sep 15, 2021, at 4:11 AM, Christian Heimes <christian@python.org> wrote:
On 15/09/2021 09.21, Eric V. Smith wrote:
On 9/15/2021 3:02 AM, Christian Heimes wrote: On 15/09/2021 01.55, Guido van Rossum wrote:
I know where I'd file a bug. :-)
"Bug magnet" is an extremely subjective pejorative term. When the *better* way to do things (os.workdir()) is harder than the *easy* way to do (os.chdir()), which is the real bug magnet? The "better way" to handle current working directory is to use the modern *at() variants of syscalls, e.g. openat() instead open(). The variants take an additional file descriptor dirfd that is used as the current working directory for the syscall.
While I generally agree, the only times I've written a context manager like os.workdir() is when running an executable with subprocess.call(), and the executable requires that its current directory be set to some specific directory. So while I don't use this functionality very often, there are times when nothing else will do. I realize I could handle this temporary working directory with yet another executable (including a shell), but using a context manager is just easier, and I only use this in single-threaded programs.
You don't have to change the current working directory of your process in order to run a child process in a different working directory. subprocess.call() and other functions in the subprocess module accept a "cwd" argument that lets you run a process with a different working directory. The "cwd" argument is thread safe.
I learned something today. Thanks! Eric
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith <eric@trueblade.com> wrote:
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Our version is named "pushdir", modeled after shell's pushd (even though pushdir is a context manager and does auto-pop). Everyone figures out what it does at a glance.
On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Our version is named "pushdir", modeled after shell's pushd (even though pushdir is a context manager and does auto-pop). Everyone figures out what it does at a glance.
That's a great name! Although I think having os.chdir() return a context manager is a better design (assuming it can work, but at first blush it would seem so). Eric
To make chdir() return a context manager *and* keep it working without calling `__enter__`, it would have to call `getcwd()`, which I've heard is expensive. So I don't think that would work, alas. On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith <eric@trueblade.com> wrote:
On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith <eric@trueblade.com> wrote:
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Our version is named "pushdir", modeled after shell's pushd (even though pushdir is a context manager and does auto-pop). Everyone figures out what it does at a glance.
That's a great name!
Although I think having os.chdir() return a context manager is a better design (assuming it can work, but at first blush it would seem so).
Eric
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/U5UQFD... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 15.09.2021 21:02, Guido van Rossum wrote:
To make chdir() return a context manager *and* keep it working without calling `__enter__`, it would have to call `getcwd()`, which I've heard is expensive.
So I don't think that would work, alas.
At least on Linux, the cost for os.getcwd() is similar to the cost of os.chdir(), but yes, since we can't have os.chdir() not change the dir when called, the logic would need the extra os.getcwd() call: # python3 -m timeit -s 'import os' "os.getcwd()" 500000 loops, best of 5: 619 nsec per loop # python3 -m timeit -s 'import os' "os.chdir('.')" 500000 loops, best of 5: 726 nsec per loop Here's simple implementation of the chdir() context manager: import os import pathlib # chdir context manager PlatformPath = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath class chdir(PlatformPath): def __init__(self, dir): self.dir = dir self.olddir = os.getcwd() os.chdir(dir) def __enter__(self): return self def __exit__(self, *exc): os.chdir(self.olddir) return False # Normal chdir() path = chdir('abc/') print (os.getcwd()) print (path.olddir) # chdir() context manager with chdir('def/') as wd: print (repr(wd)) print (os.getcwd()) print (os.listdir('.')) For extra perks, I made os.chdir() return a pathlib Path object and you get to see the old directory, so you can backtrack if needed, even without a context manager.
On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Our version is named "pushdir", modeled after shell's pushd (even though pushdir is a context manager and does auto-pop). Everyone figures out what it does at a glance.
That's a great name!
Although I think having os.chdir() return a context manager is a better design (assuming it can work, but at first blush it would seem so).
Eric -- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Sep 15 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
Just as an update to this thread: The workdir context manager got implemented for Python 3.11 and now lives in contextlib as contextlib.chdir(): https://github.com/python/cpython/commit/3592980f9122ab0d9ed93711347742d110b... Thanks to everyone who contributed to the thread, the SC for approving it, Filipe Laíns who implemented it and others who are still helping to sort out some smaller issues: https://bugs.python.org/issue45545 Cheers, -- Marc-Andre Lemburg On 15.09.2021 21:41, Marc-Andre Lemburg wrote:
On 15.09.2021 21:02, Guido van Rossum wrote:
To make chdir() return a context manager *and* keep it working without calling `__enter__`, it would have to call `getcwd()`, which I've heard is expensive.
So I don't think that would work, alas.
At least on Linux, the cost for os.getcwd() is similar to the cost of os.chdir(), but yes, since we can't have os.chdir() not change the dir when called, the logic would need the extra os.getcwd() call:
# python3 -m timeit -s 'import os' "os.getcwd()" 500000 loops, best of 5: 619 nsec per loop # python3 -m timeit -s 'import os' "os.chdir('.')" 500000 loops, best of 5: 726 nsec per loop
Here's simple implementation of the chdir() context manager:
import os import pathlib
# chdir context manager PlatformPath = pathlib.WindowsPath if os.name == 'nt' else pathlib.PosixPath class chdir(PlatformPath): def __init__(self, dir): self.dir = dir self.olddir = os.getcwd() os.chdir(dir) def __enter__(self): return self def __exit__(self, *exc): os.chdir(self.olddir) return False
# Normal chdir() path = chdir('abc/') print (os.getcwd()) print (path.olddir)
# chdir() context manager with chdir('def/') as wd: print (repr(wd)) print (os.getcwd()) print (os.listdir('.'))
For extra perks, I made os.chdir() return a pathlib Path object and you get to see the old directory, so you can backtrack if needed, even without a context manager.
On Wed, Sep 15, 2021 at 11:55 AM Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
On 9/15/2021 2:48 PM, Eric Fahlgren wrote:
On Wed, Sep 15, 2021 at 12:21 AM Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>> wrote:
And I'm not crazy about the name "workdir". To me, it sounds like it returns something, not sets and resets something. But I don't have a particularly great alternative in mind: in my own code I've used "change_dir", which isn't awesome either.
Our version is named "pushdir", modeled after shell's pushd (even though pushdir is a context manager and does auto-pop). Everyone figures out what it does at a glance.
That's a great name!
Although I think having os.chdir() return a context manager is a better design (assuming it can work, but at first blush it would seem so).
Eric
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Oct 22 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On Wed, Sep 15, 2021 at 12:02:04PM -0700, Guido van Rossum wrote:
To make chdir() return a context manager *and* keep it working without calling `__enter__`, it would have to call `getcwd()`, which I've heard is expensive.
So I don't think that would work, alas.
How expensive would it have to be to rule it out? Calling os.getcwd() is about 80% as costly as chdir on my machine so it is clearly not free. [steve ~]$ python3.9 -m timeit -s "import os" "os.getcwd()" 500000 loops, best of 5: 687 nsec per loop [steve ~]$ python3.9 -m timeit -s "import os" "os.chdir('.')" 500000 loops, best of 5: 842 nsec per loop But I don't think that chdir is likely to be the bottleneck in many scripts. -- Steve
On Wed, 15 Sept 2021 at 08:02, Christian Heimes <christian@python.org> wrote:
The "better way" to handle current working directory is to use the modern *at() variants of syscalls, e.g. openat() instead open(). The variants take an additional file descriptor dirfd that is used as the current working directory for the syscall.
Just a somewhat off-topic note, but dir_fd arguments are only supported on Unix, and the functionality only appears to be present at the NT Kernel level on Windows, not in the Windows API. So this is only a "better" approach if you are writing Unix-only code. Conversely, the most significant cases where I've wanted a "set the current directory" context manager are to run some code with a temporary directory as CWD, but then make sure I don't leave that as the CWD, because Windows won't let me delete a directory that's the CWD of a running process, so my cleanup will fail if I don't reset the CWD. So the proposed context manager is probably more beneficial for Windows users. Paul
On 9/15/21, Paul Moore <p.f.moore@gmail.com> wrote:
Just a somewhat off-topic note, but dir_fd arguments are only supported on Unix, and the functionality only appears to be present at the NT Kernel level on Windows, not in the Windows API.
Handle-relative paths are supported by all NT system calls that access object paths, but NT doesn't support ".." components. Normal user-mode programs can make system calls directly (e.g. call NtCreateFile instead of CreateFile), but even if Python bypassed the Windows API to support dir_fd, the lack of support for ".." components in relative paths would be an annoying inconsistency with POSIX dir_fd support.
On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't going to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I don't think we should worry about it "creeping into library code." The thread-unsafety is not a cause of this context manager. It comes from the preexisting `os.chdir()`. If the library is changing the CWD, it's already thread-unsafe. It's not because of the new context manager. All `os.workdir()` does is make things easier. However, if it's implemented (which I personally support), there should still of course be a warning in the documentation. But I'd like to emphasize that *it is not because of `workdir()` itself, but the underlying `chdir()`!* On 14Sep2021 15:16, Guido van Rossum <guido@python.org> wrote:
Here I think we need to drop our perfectionist attitude.
I completely agree.
BTW, should it be `WorkDir` instead of `workdir` because it's a class, or would that be too inconsistent? On Tue, Sep 14, 2021, 6:47 PM Finn Mason <finnjavier08@gmail.com> wrote:
On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't going to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I don't think we should worry about it "creeping into library code." The thread-unsafety is not a cause of this context manager. It comes from the preexisting `os.chdir()`. If the library is changing the CWD, it's already thread-unsafe. It's not because of the new context manager. All `os.workdir()` does is make things easier. However, if it's implemented (which I personally support), there should still of course be a warning in the documentation. But I'd like to emphasize that *it is not because of `workdir()` itself, but the underlying `chdir()`!*
On 14Sep2021 15:16, Guido van Rossum <guido@python.org> wrote:
Here I think we need to drop our perfectionist attitude.
I completely agree.
For the stdlib context managers that I know of, all-lowercase seems the convention. Would it even be a class? The simplest implementation would use contextlib.contextmanager (unless that’s a undesirable dependency for os.py). —Guido On Tue, Sep 14, 2021 at 17:53 Finn Mason <finnjavier08@gmail.com> wrote:
BTW, should it be `WorkDir` instead of `workdir` because it's a class, or would that be too inconsistent?
On Tue, Sep 14, 2021, 6:47 PM Finn Mason <finnjavier08@gmail.com> wrote:
On Tue, Sep 14, 2021, 5:36 PM Cameron Simpson <cs@cskk.id.au> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote: pypi,
quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't going to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
I don't think we should worry about it "creeping into library code." The thread-unsafety is not a cause of this context manager. It comes from the preexisting `os.chdir()`. If the library is changing the CWD, it's already thread-unsafe. It's not because of the new context manager. All `os.workdir()` does is make things easier. However, if it's implemented (which I personally support), there should still of course be a warning in the documentation. But I'd like to emphasize that *it is not because of `workdir()` itself, but the underlying `chdir()`!*
On 14Sep2021 15:16, Guido van Rossum <guido@python.org> wrote:
Here I think we need to drop our perfectionist attitude.
I completely agree.
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/J4L3I3... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
I think this would be convenient. And yes, it's not thread safe. But neither is os.chdir() to start with. Someone whose script, or library, wants to chdir can already shoot themselves in the foot. This makes that slightly less likely, not more. In terms of the bikeshed color, I think putting this in `pathlib` is the best approach for "modern Python." On Tue, Sep 14, 2021, 7:37 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't goingg to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
On Tue, Sep 14, 2021, 7:58 PM David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
I think this would be convenient.
And yes, it's not thread safe. But neither is os.chdir() to start with. Someone whose script, or library, wants to chdir can already shoot themselves in the foot. This makes that slightly less likely, not more.
I agree. In terms of the bikeshed color, I think putting this in `pathlib` is the
best approach for "modern Python."
I disagree. The way I see it, `pathlib` is home of *Path(). No functions. Just *Path(). Then again, I didn't write pathlib. I could be misunderstanding the intent. However, workdir() doesn't deal with Paths. Sure, it takes a PathLike as an argument, but it doesn't perform operations on a Path or return a Path. I think changing the CWD belongs squarely in the os module.
On Tue, Sep 14, 2021, 11:17 PM Finn Mason <finnjavier08@gmail.com> wrote:
I disagree. The way I see it, `pathlib` is home of *Path(). No functions. Just *Path(). Then again, I didn't write pathlib. I could be misunderstanding the intent.
When I wrote "in" I was thinking more of a method than a function. E.g. with Path("/new/dir/").workdir() as wd: # ... do stuff ...
On Tue, 14 Sept 2021 at 22:59, David Mertz, Ph.D. <david.mertz@gmail.com> wrote:
I think this would be convenient.
And yes, it's not thread safe. But neither is os.chdir() to start with. Someone whose script, or library, wants to chdir can already shoot themselves in the foot. This makes that slightly less likely, not more.
The problem is someone thinking that by using this they would not be shooting themselves on the foot because "wow, a 2021 adition to the stdlib will certainly be thread and async safe". I am with Chris Angelico in that this would be dangerous.
In terms of the bikeshed color, I think putting this in `pathlib` is the best approach for "modern Python."
OTOH, on Pathlib we could have a proper implementation that could take care of _all_ in context file operations, and internally use absolute paths (instead of calling chdir). Such an implementation would work. ``` with Path("/tmp").workdir as mydir: with mydir.open("myfile"): ... ``` and maybe could be practical enough to stirr people away from "chdir".
On Tue, Sep 14, 2021, 7:37 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 15Sep2021 07:50, Chris Angelico <rosuav@gmail.com> wrote:
On Wed, Sep 15, 2021 at 7:43 AM Cameron Simpson <cs@cskk.id.au> wrote:
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
Library code shouldn't be changing the working directory, context manager or not. That belongs to the application.
Entirely agree.
I'm concerned that convenient stackable chdir is a bug magnet, and would creep into library code. Maybe not in the stdlib, but there's no point writing such a context manager if it isn't goingg to be used, and therefore it could get used in library code. Imagine when a popular pypi module starts using it internally and breaks a multithreaded app previously relying on it?
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/KM6NEM... Code of Conduct: http://python.org/psf/codeofconduct/
Here I think we need to drop our perfectionist attitude. When I saw Marc-Andre's proposal my first response was also "but what about threads." But really, os.chdir() is the culprit here, and since it's a syscall we can't fix it. If we can live with that, we can live with the proposed os.workdir(). The docs just need a warning about threads. Yes, you could create a thread-safe version of this by overriding open() -- and several hundred other functions that refer to pathnames natively. Such a project will inevitably have serious backwards compatibility concerns (any use of pathnames from C extensions will not work right) so it's just not going to be done. If we don't offer this in the stdlib, users will just implement this themselves, poorly (for example, by not restoring the original when done). On Tue, Sep 14, 2021 at 2:43 PM Cameron Simpson <cs@cskk.id.au> wrote:
On 14Sep2021 21:43, M.-A. Lemburg <mal@egenix.com> wrote:
- The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal.
This is the source of my concerns. Though of course it applies to any process global state. It would need this stated up front in big letters (as of course does chdir itself). My real concern is that this can leak into other functions whose use then makes whatever uses them to become inherently and unrepairably not thread safe.
I know I'm atypical, but I have quite a lot of multithreaded stuff, including command line code. So while it'd be ok to avoid this context manager for my own code, I fear library modules, either stdlib or pypi, quietly using this in their code, making them unuseable in the general case. Unrepairably unuseable, for the user.
I think what would be more useful is a context manager which worked on a threading.local which pushed/popped a reference directory path, and had an open() method which used that (and friends for other pathname based functions).
In my own code I'd write this like (sketch, untested):
from cs.threads import State as ThreadState
class RefDir(ThreadState)
def __init__(self, refpath=None): if refpath is None: refpath = os.getcwd() self.refpath = abspath(refpath)
def open(self, path, *a, **kw): if not isabs(path): path = os.path.join(self.refpath, path) return open(path, *a, **kw) # calls the builtin open()
... listdir, mkdir, etc etc ...
# on reflection, __enter__ and __exit__ would make the below even # more convenient @contextmanager def dirpath(newrefpath): ''' Push `newrefpath` for the duration of the context manager. ''' with self(refpath=newrefpath): yield self.refpath
and then use it like this:
R = RefDir() .... with R.dirpath('/some/new/place') as newpath: ... with R.open("something.txt") as f: ... work on /some/new/place/something.txt ...
In the above, cs.threads.State is a threading.lcoal subclass which is also a context manager whose operation pushes arbitrary attribute values. Great for thread safe execution scoped state. Like this example.
All that said, I wrote pretty much exactly what you describe just the other week for umask().
Cheers, Cameron Simpson <cs@cskk.id.au> _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/EJETKX... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
On 14Sep2021 15:16, Guido van Rossum <guido@python.org> wrote:
Here I think we need to drop our perfectionist attitude. When I saw Marc-Andre's proposal my first response was also "but what about threads." But really, os.chdir() is the culprit here, and since it's a syscall we can't fix it. If we can live with that, we can live with the proposed os.workdir(). The docs just need a warning about threads. [...] If we don't offer this in the stdlib, users will just implement this themselves, poorly (for example, by not restoring the original when done).
Agreed here. As long as it has a big clear warning about process global state. That is the beauty of libraries, to me: not just reuse, but a correct implementation is correct everywhere, and likewise a bugfix fixes all the things. Cheers, Cameron Simpson <cs@cskk.id.au>
On 15.09.2021 00:16, Guido van Rossum wrote:
Here I think we need to drop our perfectionist attitude. When I saw Marc-Andre's proposal my first response was also "but what about threads." But really, os.chdir() is the culprit here, and since it's a syscall we can't fix it. If we can live with that, we can live with the proposed os.workdir(). The docs just need a warning about threads.
Yes, you could create a thread-safe version of this by overriding open() -- and several hundred other functions that refer to pathnames natively. Such a project will inevitably have serious backwards compatibility concerns (any use of pathnames from C extensions will not work right) so it's just not going to be done.
If we don't offer this in the stdlib, users will just implement this themselves, poorly (for example, by not restoring the original when done).
Thanks for all the feedback. I'm just going to reply here in a single email. First, let me repeat the notes I had added to the proposal, since these seem to have gotten lost in the thread. I'll some more context:
Notes: - The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal.
The CWD (current working directory) concept is a very old concept where the OS manages a pointer into the file system on a per process basis in order to handle requests for relative paths and map them to absolute paths. https://en.wikipedia.org/wiki/Working_directory Since this is per process and managed by the OS, the concept does not extend to threads (possibly) running in the process. While Python could theoretically try to emulate parts of this for threads as well, this is not within the scope of os.workdir(). It's supposed to be a simple convenient way of saying: please chdir here, do something and then return to the previously active work dir. The problem it's trying to solve is that last part... functions forgetting to restore the previous active work dir can easily break other parts of the script, since the CWD is used for resolving relative paths in Python as well. If thread-safety is seen as a big problem, we could add a concept of managing a single CWD across all threads using lock protected globals. Not sure whether this is worth the trouble, but perhaps making it optional would ease some concerns. In this case, os.chdir() would have to be adapted as well, in order to update the Python CWD state in the same way (and possibly block or warn when os.chdir() is called inside an active with os.workdir() block).
- The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes.
We could add a logger parameter to the context manager for this to customize the logger, or simply use the default one from logging.
- The same could be added to pathlib's objects as .workdir() method returning a context manager.
Since people are starting to use pathlib more, this would be a natural thing to do, as others have noted already :-) The context manager will also accept a path-like object as argument (simply because it passes this directly to os.chdir() which already does). As for having the context manager method on a Path object return the Path object itself (suggested by David), this could be made to work by returning a subclass of Path, which then has the necessary extras needed for CWD processing. I'll add some more notes: - Why in the os module ? Because this is where you'd search for a way to set the CWD and indeed, we have os.chdir() and os.getcwd() in this module. Also see the note on pathlib above. - There are other global settings which could benefit from such context managers as well, such as umask (mentioned by Cameron) or the locale settings, which you sometimes want to temporarily modify, but those are not used as often. - Why the lowercase name ? Simply to make it fit the other os module APIs. Classes should normally have SnakeCase names, but since we're not interested in the class aspect here, but rather the context manager API of the resulting object, I think it's fair to stick with lower case. Many other context managers use a lower case name as well. - Chris mentioned that library code should not be changing the CWD. In an ideal world, I'd agree, but there are cases where you don't have an option and you need to change the CWD in order make certain things work, e.g. you don't control the code you want to run, but need to make it work in a specific directory by changing the CWD and then passing relative paths to the code. If you use subprocesses for this, you can make this work, since the module forks for you and then sets the CWD in the forked process on Unix, but this still leaves you with a gap in Python, since you may well want to code in the same block which e.g. relies on os.abspath() working with your modified version of the CWD. - Main use case ? Shell scripts written in Python instead of bash. - Adding a big warning to the docs. Yes, absolutely, and os.chdir() should receive one in the docs as well. - openat() et al. (mentioned by Cameron and Christian). These could be used to implement a thread-safe way of working with CWDs, essentially placing the CWD under control of Python. However, Python extensions and the OS would now know about this per-thread CWD, so you'd most likely run into problems with only part of your code working with these per-thread CWDs. This is definitely out of scope for os.workdir(). - Bike shedding on the name. "workdir" is what I use for the context manager, but e.g. os.cwd() would also do and is more in line with os.getcwd(). We could even have os.chdir() return the context manager, without introducing any new API in the os module. Still, I find "with os.workdir()" quite intuitive :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 15 2021)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 15/09/2021 11.56, Marc-Andre Lemburg wrote:
- Chris mentioned that library code should not be changing the CWD. In an ideal world, I'd agree, but there are cases where you don't have an option and you need to change the CWD in order make certain things work, e.g. you don't control the code you want to run, but need to make it work in a specific directory by changing the CWD and then passing relative paths to the code.
This seems rather hypothetical to me. Can you provide a real-world example where you cannot use absolute paths? Christian
On Wed, Sep 15, 2021 at 8:41 PM Christian Heimes <christian@python.org> wrote:
On 15/09/2021 11.56, Marc-Andre Lemburg wrote:
- Chris mentioned that library code should not be changing the CWD. In an ideal world, I'd agree, but there are cases where you don't have an option and you need to change the CWD in order make certain things work, e.g. you don't control the code you want to run, but need to make it work in a specific directory by changing the CWD and then passing relative paths to the code.
This seems rather hypothetical to me. Can you provide a real-world example where you cannot use absolute paths?
Sometimes there is no valid path to something, or you need to be immune to directory renames. The normal solution would be to use openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure what happens on Windows. Does Windows let you open a directory and retain a handle to it? If not, it probably just won't allow the concept in question, and absolutification would be a solution. This would be another good reason to wrap this up into a library - on POSIX systems, it can open a directory and make everything use dir_fd, but on Windows, it remembers the drive letter and path, and absolutes out all path names. Additional wrinkle: Can this be made asyncio compatible? Is it possible to make it, instead of thread-local, aware in some way of which task it's running? ChrisA
On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:
Sometimes there is no valid path to something,
I don't understand what you mean by that. If there is no valid path to something, where exactly are you cd'ing to, and how does it help you with the thing that has no valid path to it?
or you need to be immune to directory renames. The normal solution would be to use openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure what happens on Windows.
Normal for whom? Let's not forget that there are still a bazillion people who write scripts in bash or some other shell, or Python, that just do the simplest thing that can possibly work. Namely cd into the directory they want to be in. This is a legitimate use-case. Not everyone needs to care about threads.
Additional wrinkle: Can this be made asyncio compatible? Is it possible to make it, instead of thread-local, aware in some way of which task it's running?
Are you suggesting that you might want to run multiple async coroutines in different working directories? I think that designing this for threads or async is over-engineering. -- Steve
On Wed, Sep 15, 2021 at 9:22 PM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:
Sometimes there is no valid path to something,
I don't understand what you mean by that. If there is no valid path to something, where exactly are you cd'ing to, and how does it help you with the thing that has no valid path to it?
You retain it from when it DID have a path.
or you need to be immune to directory renames. The normal solution would be to use openat() etc (or, in Python, os.open() with a dir_fd). I'm not sure what happens on Windows.
Normal for whom?
Let's not forget that there are still a bazillion people who write scripts in bash or some other shell, or Python, that just do the simplest thing that can possibly work. Namely cd into the directory they want to be in.
This is a legitimate use-case. Not everyone needs to care about threads.
When you cd into something (on POSIX, at least), your process retains a reference to it. Not a file descriptor as such, I think, but it has some sort of reference. It is entirely possible to be in a directory that no longer has a valid path name to it, or when the path to that directory has changed. (Renames are easiest to demo, and can be seen in bash without even entering Python.) The way I see it, this is exactly the same use-case, just implemented slightly differently.
Additional wrinkle: Can this be made asyncio compatible? Is it possible to make it, instead of thread-local, aware in some way of which task it's running?
Are you suggesting that you might want to run multiple async coroutines in different working directories?
I think that designing this for threads or async is over-engineering.
If you don't care about going back to the original working directory, just use os.chdir() as it already is. For a lot of scripts, ANY context manager is over-engineering. Why is this being proposed? Because there are use-cases where there needs to be a reset. So if there needs to be a reset, that means you need to isolate the action of "change working directory" to a subset of code. And while it's of some value to be able to do that even if it's not thread-safe or asyncio-safe, it would be of distinctly more value if it can be aware of those use-cases. Depending on how that's implemented, it might not even be that difficult - for instance, if there's a stack of file descriptors, and all open() calls are rerouted via os.open(dir_fd=top_of_stack), then it would be only a small amount more work to make it per-task instead of per-thread. Having some proper language support may be necessary here, though. I'm not sure how deeply this would affect things. Of course, it would be pretty straight-forward if it's all done cooperatively, with a dedicated "open relative to top of stack" API. ChrisA
On Wed, Sep 15, 2021 at 4:20 AM Steven D'Aprano <steve@pearwood.info> wrote:
On Wed, Sep 15, 2021 at 08:53:21PM +1000, Chris Angelico wrote:
Sometimes there is no valid path to something,
I don't understand what you mean by that. If there is no valid path to something, where exactly are you cd'ing to, and how does it help you with the thing that has no valid path to it?
No one has mentioned the other case where cd'ing somewhere makes paths suddenly valid, and that is for long path names. Once you pass PATH_MAX length your only options are to cd as deep as possible and continue relatively, or to pass dir fds. The second is perhaps the better option, but harder to implement as you have to handle extra fds and remember to clean them up and I at least never remember how to thread fds through python code without going and looking it up. This also brings up the issue of how to restore the prior working directory. Just remembering its path and calling a new chdir will fail, both in the long path case and in the case that the old dir was deleted since last visit. We could hold an open fd before leaving the old cwd, and that should allow us to return in both cases. However I don't know how others feel about holding on to dir fds, especially essentially hidden ones; it sounds ripe for FS deadlock. Regards, ~Jeremiah
On Wed, Sep 15, 2021 at 11:56:17AM +0200, Marc-Andre Lemburg wrote:
- The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes.
We could add a logger parameter to the context manager for this to customize the logger, or simply use the default one from logging.
That seems unnecessary. We don't put logging in other context managers. with open(filename, logging=...) as f: Keep workdir() nice and simple, let it do one thing (change the working directory and restore it on exit) and not overload it with "elaborate" functionality. If people want logging, they can handle logging themselves. Or maybe they can use a logging context manager?
We could even have os.chdir() return the context manager, without introducing any new API in the os module.
That's an interesting suggestion. I think I like it. No, I'm *sure* that I really like it. Let's do this! # The old way: os.chdir(dirname) ... os.chdir(prevdir) # The new way: with os.chdir(dirname): ... Of course the old way would still work. -- Steve
14.09.21 22:43, Marc-Andre Lemburg пише:
Would there be interest in adding something like this to the os module as os.workdir() ?
There is already an open issue and PR for this: https://bugs.python.org/issue25625 https://github.com/python/cpython/pull/28271 My main concern is that it is not thread-safe (nor generator-safe, nor coroutine-safe), but on other hand, redirect_stdout() and redirect_stderr() have same limitations.
On Tue, 2021-09-14 at 21:43 +0200, Marc-Andre Lemburg wrote:
Hello all,
I sometimes write Python scripts which need to work in specific work directories.
When putting such code into functions, the outer function typically does not expect the current work dir (CWD) to be changed, so wrap the code which need the (possibly) modified CWD using a simply context manager along the lines of:
class workdir: def __init__(self, dir): self.dir = dir def __enter__(self): self.curdir = os.getcwd() os.chdir(self.dir) def __exit__(self, *exc): os.chdir(self.curdir) return False
Would there be interest in adding something like this to the os module as os.workdir() ?
Example:
def backup_home_dir(account): with os.workdir(os.path.join('/home', account)): # Create a backup of the account dir, rooted at the account's # home dir restic.backup(repo, '.')
Notes: - The context manager is not thread safe. There's no thread safe model for the current work dir. OTOH, scripts usually don't use threads, so not a big deal. - The context manager could be made more elaborate, e.g. adding optional logging for debugging purposes. - The same could be added to pathlib's objects as .workdir() method returning a context manager.
Thoughts ?
Thanks,
bpo: https://bugs.python.org/issue25625 PR: https://github.com/python/cpython/pull/28271 Filipe Laíns
participants (17)
-
Alex Waygood -
Cameron Simpson -
Chris Angelico -
Christian Heimes -
David Mertz, Ph.D. -
Eric Fahlgren -
Eric V. Smith -
Eryk Sun -
Filipe Laíns -
Finn Mason -
Guido van Rossum -
Jeremiah Paige -
Joao S. O. Bueno -
Marc-Andre Lemburg -
Paul Moore -
Serhiy Storchaka -
Steven D'Aprano