posix_spawn would really speed up spawnProcess if we used it
I am writing some code that was spawning a lot of processes with spawnProcess, and it was going a bit slower than I expected; it was taking about 51 seconds on my M1 Max mac. So I hopped over to my local copy of Twisted and applied this patch: 1 file changed, 9 insertions(+) src/twisted/internet/process.py | 9 +++++++++ modified src/twisted/internet/process.py @@ -367,6 +367,15 @@ class _BaseProcess(BaseProcess): @type environment: L{dict}. @param kwargs: keyword arguments to L{_setupChild} method. """ + from os import posix_spawn, POSIX_SPAWN_DUP2, POSIX_SPAWN_CLOSE + fdmap = kwargs.get("fdmap", {}) + actions = [ + (POSIX_SPAWN_DUP2, childFD, parentFD) + for (parentFD, childFD) in fdmap.items() + ] + self.pid = posix_spawn(executable, args, environment, file_actions=actions) + return + collectorEnabled = gc.isenabled() gc.disable() try: This is obviously not a complete reimplementation and might have some slight performance advantage due to its incompleteness, but the task in question then took 16 seconds; i.e. a 319% speedup. Anyone want to do this implementation for real? Or want to commit to a review, if I did it? -g
Oh hey it looks like… I already did this and it already got reviewed and already got approved and is waiting to get merged: https://github.com/twisted/twisted/pull/1675 but it predated the review robot so it fell through the cracks. Looks like I'll land this as soon as tests can pass :) -g
On Mar 14, 2023, at 1:15 PM, Glyph <glyph@twistedmatrix.com> wrote:
I am writing some code that was spawning a lot of processes with spawnProcess, and it was going a bit slower than I expected; it was taking about 51 seconds on my M1 Max mac. So I hopped over to my local copy of Twisted and applied this patch:
1 file changed, 9 insertions(+) src/twisted/internet/process.py | 9 +++++++++
modified src/twisted/internet/process.py @@ -367,6 +367,15 @@ class _BaseProcess(BaseProcess): @type environment: L{dict}. @param kwargs: keyword arguments to L{_setupChild} method. """ + from os import posix_spawn, POSIX_SPAWN_DUP2, POSIX_SPAWN_CLOSE + fdmap = kwargs.get("fdmap", {}) + actions = [ + (POSIX_SPAWN_DUP2, childFD, parentFD) + for (parentFD, childFD) in fdmap.items() + ] + self.pid = posix_spawn(executable, args, environment, file_actions=actions) + return + collectorEnabled = gc.isenabled() gc.disable() try:
This is obviously not a complete reimplementation and might have some slight performance advantage due to its incompleteness, but the task in question then took 16 seconds; i.e. a 319% speedup.
Anyone want to do this implementation for real? Or want to commit to a review, if I did it?
-g
Aaaaaand it's landed. Fastest feature on Twisted ever ;-) -g
On Mar 14, 2023, at 1:26 PM, Glyph <glyph@twistedmatrix.com> wrote:
Oh hey it looks like… I already did this and it already got reviewed and already got approved and is waiting to get merged: https://github.com/twisted/twisted/pull/1675
but it predated the review robot so it fell through the cracks.
Looks like I'll land this as soon as tests can pass :)
-g
On Mar 14, 2023, at 1:15 PM, Glyph <glyph@twistedmatrix.com> wrote:
I am writing some code that was spawning a lot of processes with spawnProcess, and it was going a bit slower than I expected; it was taking about 51 seconds on my M1 Max mac. So I hopped over to my local copy of Twisted and applied this patch:
1 file changed, 9 insertions(+) src/twisted/internet/process.py | 9 +++++++++
modified src/twisted/internet/process.py @@ -367,6 +367,15 @@ class _BaseProcess(BaseProcess): @type environment: L{dict}. @param kwargs: keyword arguments to L{_setupChild} method. """ + from os import posix_spawn, POSIX_SPAWN_DUP2, POSIX_SPAWN_CLOSE + fdmap = kwargs.get("fdmap", {}) + actions = [ + (POSIX_SPAWN_DUP2, childFD, parentFD) + for (parentFD, childFD) in fdmap.items() + ] + self.pid = posix_spawn(executable, args, environment, file_actions=actions) + return + collectorEnabled = gc.isenabled() gc.disable() try:
This is obviously not a complete reimplementation and might have some slight performance advantage due to its incompleteness, but the task in question then took 16 seconds; i.e. a 319% speedup.
Anyone want to do this implementation for real? Or want to commit to a review, if I did it?
-g
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/HT7S4IXLH3X... Code of Conduct: https://twisted.org/conduct
On 14 Mar 2023, at 22:38, Glyph <glyph@twistedmatrix.com> wrote:
Aaaaaand it's landed. Fastest feature on Twisted ever ;-)
How does it perform on linux system? Barry
-g
On Mar 14, 2023, at 1:26 PM, Glyph <glyph@twistedmatrix.com> wrote:
Oh hey it looks like… I already did this and it already got reviewed and already got approved and is waiting to get merged: https://github.com/twisted/twisted/pull/1675
but it predated the review robot so it fell through the cracks.
Looks like I'll land this as soon as tests can pass :)
-g
On Mar 14, 2023, at 1:15 PM, Glyph <glyph@twistedmatrix.com> wrote:
I am writing some code that was spawning a lot of processes with spawnProcess, and it was going a bit slower than I expected; it was taking about 51 seconds on my M1 Max mac. So I hopped over to my local copy of Twisted and applied this patch:
1 file changed, 9 insertions(+) src/twisted/internet/process.py | 9 +++++++++
modified src/twisted/internet/process.py @@ -367,6 +367,15 @@ class _BaseProcess(BaseProcess): @type environment: L{dict}. @param kwargs: keyword arguments to L{_setupChild} method. """ + from os import posix_spawn, POSIX_SPAWN_DUP2, POSIX_SPAWN_CLOSE + fdmap = kwargs.get("fdmap", {}) + actions = [ + (POSIX_SPAWN_DUP2, childFD, parentFD) + for (parentFD, childFD) in fdmap.items() + ] + self.pid = posix_spawn(executable, args, environment, file_actions=actions) + return + collectorEnabled = gc.isenabled() gc.disable() try:
This is obviously not a complete reimplementation and might have some slight performance advantage due to its incompleteness, but the task in question then took 16 seconds; i.e. a 319% speedup.
Anyone want to do this implementation for real? Or want to commit to a review, if I did it?
-g
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/HT7S4IXLH3X... Code of Conduct: https://twisted.org/conduct
_______________________________________________ Twisted mailing list -- twisted@python.org To unsubscribe send an email to twisted-leave@python.org https://mail.python.org/mailman3/lists/twisted.python.org/ Message archived at https://mail.python.org/archives/list/twisted@python.org/message/7JFFYLSN2V2... Code of Conduct: https://twisted.org/conduct
participants (2)
-
Barry -
Glyph