Can someone look at dummy_thread (#622537)?
There is a slight chance some of you remember the discussion that Zack Weinberg brought up here back in September about his experiences in rewriting ``tempfile``. One of the things he brought up was having to write his own fake lock so that the code would work when ``thread`` was not available. Guido suggested writing a module called ``dummy_thread`` that was API compatible with ``thread`` but was available on all platforms; he said he would check it in if it was written. Well, I wrote it. But Guido has told me that he is too busy to check it in and so I am emailing you guys to see if someone can look at it and check it in. I would like to get it in before 2.3a so as to get the extra testing the alpha will get. The patch is #622537 ( http://www.python.org/sf/622537 ). It is complete to the best of my abilities. I have docs and the module and the tests and such. Should be pretty straight-forward to apply assuming I didn't muck up the docs (first attempt at doing any docs from scratch). The only thing beyond quality of code and documentation that might have to be dealt with is how far to integrate ``dummy_thread`` into the stdlib. I have a patch to have ``Queue``, ``threading``, and ``tempfile`` all use ``dummy_thread`` when ``thread`` is not available. Now I tested all of them with and without ``thread`` available and they all pass. But of course I wonder if this will break any code. The problem is that since ``dummy_thread`` basically just executes thread calls serially there is a problem when sommething like blocking I/O is used that blocks waiting for another thread (test_asynchat and test_socket do that, but they require ``thread`` so there is no issue there outright). So there is a chance that some code, when trying to run using ``threading`` will lock up. But I don't think this is an issue. I say in the docs that if you want to guarantee true threading to run ``import thread; del thread`` before importing ``threading``. But the main reason I don't think it will be an issue is that all other code that runs fine (but slowly) using ``dummy_thread`` now can be run where threads are not available. And those programs where deadlock would occur do not lose or gain anything since they couldn't run on a platform without ``thread`` anyway. The only issue is that if they don't check for ``thread`` someone trying to run the program will have a deadlocked program (although they shouldn't have been trying to run it in the first place). This is the only real questionable thing. Obviously the module can be checked in with no problem without touching the other modules (although ``Queue`` and ``tempfile`` should be patched regardless) if this seems to dicey to do. -Brett
try: import threading except ImportError: threading = None # use non-threaded behaviour
But then you'd have to sprinkle the rest of the code with "if threading is not None" tests. Brett's dummy_thread is an attempt to reduce this clutter. I've been known to write a mini-version of dummy_thread.py in code to minimize the clutter; it seems useful to generalize this. --Guido van Rossum (home page: http://www.python.org/~guido/)
guido wrote:
try: import threading except ImportError: threading = None # use non-threaded behaviour
But then you'd have to sprinkle the rest of the code with "if threading is not None" tests. Brett's dummy_thread is an attempt to reduce this clutter.
the problem (if there is one) is that the proposed integration may break code that uses the above pattern. deadlocks doesn't always happen when you want them to, so it may not be obvious to developers and testers that their program isn't as reliable as it was before the upgrade. </F>
try: import threading except ImportError: threading = None # use non-threaded behaviour
But then you'd have to sprinkle the rest of the code with "if threading is not None" tests. Brett's dummy_thread is an attempt to reduce this clutter.
the problem (if there is one) is that the proposed integration may break code that uses the above pattern.
I don't propose to automate this. I propose this: try: import threading except ImportError: import dummy_threading as threading
deadlocks doesn't always happen when you want them to, so it may not be obvious to developers and testers that their program isn't as reliable as it was before the upgrade.
So require testing on a system that has threads. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido van Rossum]
try: import threading except ImportError: threading = None # use non-threaded behaviour
But then you'd have to sprinkle the rest of the code with "if threading is not None" tests. Brett's dummy_thread is an attempt to reduce this clutter.
the problem (if there is one) is that the proposed integration may break code that uses the above pattern.
But if it does it will be on platforms it couldn't run on previously. ``threading`` uses ``thread`` when possible and only falls back on ``dummy_thread`` when it isn't available. That means this could only be a problem where the code was not even runnable before. But yes, there are chances of deadlock in certain situations. It basically comes down to whether we want to protect users from themselves by not having them get into a possible deadlock situation.
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
So is this an indirect request for me to write ``dumy_threading``? If it is it won't be hard: I would do an ``import *`` on ``threading`` and then override the functions it uses from ``thread`` with ``import ... from ... as ...`` from ``dummy_thread``. Also would mean rewriting the ``threading`` testing suite. Prefectly happy to write it, just need you to say you want it. -Brett
Brett Cannon wrote:
try: import threading except ImportError: threading = None # use non-threaded behaviour
But if it does it will be on platforms it couldn't run on previously.
Sigh. Let's try one more time: Under 2.2 and earlier, on a platform that doesn't support threads, the above code will set threading to None, and the application can then use "if threading is None" to provide application-specific work- arounds. (catching import errors is a very common pattern when you're writing portable Python code). If you change Python so that "import threading" succeeds also on plat- forms that don't support threads, the application-specific workarounds will no longer be used, and the application may no longer work properly. And since we're talking deadlocks, chances are that nobody will notice before it's too late. </F>
[Fredrik Lundh]
Brett Cannon wrote:
try: import threading except ImportError: threading = None # use non-threaded behaviour
But if it does it will be on platforms it couldn't run on previously.
Sigh. Let's try one more time:
Under 2.2 and earlier, on a platform that doesn't support threads, the above code will set threading to None, and the application can then use "if threading is None" to provide application-specific work- arounds.
I get this and always have. I just have not thought of it as huge of an issue as you do. If it is that big of a worry, add a ``import thread; del thread`` statement to make sure ``thread`` is there and warn users not to use the code. But I know you are going to be against the idea of modifying existing code for this, so I am going to let it go. Your point is valid, we just differ on its importance. Is Guido's suggested ``dummy_threading`` okay with you?
(catching import errors is a very common pattern when you're writing portable Python code).
I know, I use it myself.
If you change Python so that "import threading" succeeds also on plat- forms that don't support threads, the application-specific workarounds will no longer be used, and the application may no longer work properly.
And since we're talking deadlocks, chances are that nobody will notice before it's too late.
OK, I am going to concede on this and stop pushing for it. Let me know what you think of Guido's ``dummy_threading`` idea. I assume you don't object to changing ``Queue``, though, right? -Brett
Under 2.2 and earlier, on a platform that doesn't support threads, the above code will set threading to None, and the application can then use "if threading is None" to provide application-specific work- arounds.
I get this and always have. I just have not thought of it as huge of an issue as you do.
It is a big deal, Brett. --Guido van Rossum (home page: http://www.python.org/~guido/)
Brett Cannon
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
So is this an indirect request for me to write ``dumy_threading``?
I would suggest to make this import dummy_thread as threading i.e. have dummy_thread expose the union of the thread and threading APIs.
If it is it won't be hard: I would do an ``import *`` on ``threading``
That wont't work: "from threading import *" will raise an ImportError. I suggest to add dummy_thread to sys.modules, then execfile threading.py in dummy_thread, then remove it from sys.modules. Regards, Martin
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
So is this an indirect request for me to write ``dumy_threading``? If it is it won't be hard: I would do an ``import *`` on ``threading`` and then override the functions it uses from ``thread`` with ``import ... from ... as ...`` from ``dummy_thread``. Also would mean rewriting the ``threading`` testing suite. Prefectly happy to write it, just need you to say you want it.
Yes. I don't want threading to automatically substitute dummy_thread for thread. Multiple imports of threading must all fail when thread cannot be imported. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido van Rossum]
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
So is this an indirect request for me to write ``dumy_threading``? If it is it won't be hard: I would do an ``import *`` on ``threading`` and then override the functions it uses from ``thread`` with ``import ... from ... as ...`` from ``dummy_thread``. Also would mean rewriting the ``threading`` testing suite. Prefectly happy to write it, just need you to say you want it.
Yes. I don't want threading to automatically substitute dummy_thread for thread. Multiple imports of threading must all fail when thread cannot be imported.
Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself? -Brett
[Guido]
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
... [Brett Cannon]
Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself?
Looking at his code, I think it's clear he wants a separate dummy_threading.py and nothing fancier than that. The idiomatic use would be as shown, where a user who knows what they're doing can import dummy_threading (if desired) and *name* it threading locally themself (if desired). No existing code could break, except for users who already have a file named dummy_threading.py on their PYTHONPATH somewhere after the std library.
[Tim Peters]
[Guido]
I don't propose to automate this. I propose this:
try: import threading except ImportError: import dummy_threading as threading
...
[Brett Cannon]
Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself?
Looking at his code, I think it's clear he wants a separate
dummy_threading.py
and nothing fancier than that. The idiomatic use would be as shown, where a user who knows what they're doing can import dummy_threading (if desired) and *name* it threading locally themself (if desired). No existing code could break, except for users who already have a file named dummy_threading.py on their PYTHONPATH somewhere after the std library.
OK. Then I will just whip together the module (most likely following Martin's correction of my previous from-the-cuff idea on how to do it) and add it to the patch. So if anyone was planning on reviewing the patch, please hold off. I will email the list again once I have the module and the patches regenerated (although someone can go ahead and delete all the files in the patch; I will just put all the files up anew since practically all of them will change). -Brett
Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself?
Two separate modules. They have different APIs and it wouldn't be wise to export a different API in the dummy case than in the real case. (I'm sure you can figure out why.) --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido van Rossum]
Do you prefer I do a separate ``dummy_threading`` module or follow Martin's suggestions and just turn ``dummy_thread`` into a ``threading`` substitute itself?
Two separate modules. They have different APIs and it wouldn't be wise to export a different API in the dummy case than in the real case. (I'm sure you can figure out why.)
I would hope I could. =) OK, so as I said in the response to Tim's email, anyone who was actually going to look at the patch can hold off until I email the list again when I have ``dummy_threading`` done. And if someone can delete the old files from the patch I would appreciate it (almost all of them will have to be regenerated anyway). -Brett
[Brett Cannon]
OK, so as I said in the response to Tim's email, anyone who was actually going to look at the patch can hold off until I email the list again when I have ``dummy_threading`` done.
I intend to review it, but can't promise to get to it in a timely fashion.
And if someone can delete the old files from the patch I would appreciate it (almost all of them will have to be regenerated anyway).
OK, they're deleted. I would have done it the first time you asked, except the "almost all" bit (which was there the first time too) seems to imply that you don't want *all* of the deleted. So applying wisdom worthy of Solomon, I just deleted all of them <wink>.
[Tim Peters]
[Brett Cannon] I intend to review it, but can't promise to get to it in a timely fashion.
Cool. Thanks, Tim.
And if someone can delete the old files from the patch I would appreciate it (almost all of them will have to be regenerated anyway).
OK, they're deleted. I would have done it the first time you asked, except the "almost all" bit (which was there the first time too) seems to imply that you don't want *all* of the deleted. So applying wisdom worthy of Solomon, I just deleted all of them <wink>.
I only hesitated because I don't see ``dummy_thread`` from changing, but because I had a previous patch have a lot of issues because old files were not deleted, I am glad you just went ahead and used your wisdom. -Brett
participants (5)
-
Brett Cannon -
Fredrik Lundh -
Guido van Rossum -
martin＠v.loewis.de -
Tim Peters