Hi all, Is anyone working on adding synchronous sockets support in pyodide yet? If not, does anyone have opinions on how it should be done? I can see with support for shared memory across browsers now it would be easy enough to do it with that in JavaScript with a web worker and atomics.wait. The obvious alternative would be to make threading and synchronisation primitives work in python and do fetch calls in a thread. Has anyone built threaded pyodide yet? Does webassembly threading work in WebKit now? Any ideas how that would interact with JavaScript? Threading is obviously a heavier weight thing, but on the other hand I guess it would fix lots of problems in a way that a dedicated JavaScript solution wouldn't. Joe This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.
Hi Joe, On 09/04/2022 11:56, Joe Marshall wrote:
Is anyone working on adding synchronous sockets support in pyodide yet? > If not, does anyone have opinions on how it should be done? I can see with support for shared memory across browsers now it would be easy enough to do it with that in JavaScript with a web worker and atomics.wait.
I think the latest discussion about this was in https://github.com/pyodide/pyodide/issues/1503 and Hood also did some work on it in https://github.com/hoodmane/synclink I agree that using a web worker seems that most direct approach.
The obvious alternative would be to make threading and synchronisation primitives work in python and do fetch calls in a thread. Has anyone built threaded pyodide yet?
Threading is obviously a heavier weight thing, but on the other hand I guess it would fix lots of problems in a way that a dedicated JavaScript solution wouldn't.
Yes, I think we could eventually get treading enabled default, as discussed in https://github.com/pyodide/pyodide/issues/237. But it's a more long term subject.
Any ideas how that would interact with JavaScript?
Good question. Currently, JS access from Python works in web-workers as we initialize a separate Pyodide instance there. If it is created via Emscripten pthreads https://emscripten.org/docs/porting/pthreads.html, I guess it wouldn't work without changes to the FFI code and doing some of that initialization manually. Also, I have no idea whether passing JS objects wrapped into JsProxy between threads is even something we want to attempt. Maybe not. cc @Hood. But anyway, someone would need to try to build Pyodide with threading enabled first and seeing what happens :) Roman
Hi, Perhaps it would be better to have this conversation on the issue tracker? Or maybe we can copy the relevant parts over.
synchronous sockets support
As Roman said, I have https://github.com/hoodmane/synclink, which is a proof of concept of a strategy of using Atomics.wait, a Comlink-like API, and a hand implemented polling event loop on the blocking thread. It needs some work on CI, testing, and specifying the design/behavior of the proxies a bit more carefully. Could also use some ease-of-use improvements. But it works as a prototype. For example I have code here: https://github.com/hoodmane/worker-pyodide-console/tree/nativefs that implements an Emscripten file system based on the native file system API using synclink. This is an example of the synchronous C FS API consuming the asynchronous Javascript FS API by proxying to a different thread. You can try it out with the following code (only in chrome): import pathlib from native_fs import mount_native_fs # this next line will open a directory picker, then ask # first for read permissions and then for write permissions mount_native_fs("/native") # This next line creates a file called "blah.txt" in the folder you just selected. pathlib.Path("/native/blah.txt").write_text("hi from browser?") The code for the Pyodide thread is here: https://github.com/hoodmane/worker-pyodide-console/blob/nativefs/nativefs_py... The code for the helper thread is here: https://github.com/hoodmane/worker-pyodide-console/blob/nativefs/nativefs_ma... These have to be loaded on the appropriate sides and hooked up, but the setup code is very simple. Basically all of the complexity is hidden into synclink. I haven't thought about sockets at all (is the idea to translate reads/writes into fetch POST and GET?) but the problem of consuming an async API from a sync API is totally solvable.
Has anyone built threaded pyodide yet? Does webassembly threading work in WebKit now? Any ideas how that would interact with JavaScript?
Threading is obviously a heavier weight thing, but on the other hand I guess it would fix lots of problems in a way that a dedicated JavaScript solution wouldn't.
Yeah so Safari 15 supports Atomics and SharedArrayBuffer and everything. Unfortunately more people are currently using Safari <=14 than Safari 15 since the version of Safari is tied to the device. This pattern is different from Chrome, Edge, and Firefox where almost all users use the most recent version. As soon as we are comfortable dropping "support" for Safari v14 we should switch to WASM_BIGINT. Though I think https://github.com/emscripten-core/emscripten/pull/16693 might end our issues with MAIN_MODULE=1 and not WASM_BIGINT. Also, we should seriously consider adding Safari to CI. I think we get the worst of both worlds by allowing Safari limitations to prevent us from using browser features but also not testing in Safari so we might break it anyways. I am concerned that if we try to combine MAIN_MODULE=1 with PTHREADS we will see a lot of bugs. MAIN_MODULE=1 by itself has almost no test coverage in the Emscripten test suite -- all the tests use MAIN_MODULE=2. One good project would be to modify the Emscripten test suite to automatically run all dynlink tests with both MAIN_MODULE=1 and MAIN_MODULE=2. I am sure we will find more bugs. So my thought is first try to catch up with upstream Emscripten (only blocker is a Firefox bug https://github.com/emscripten-core/emscripten/issues/16538). Also add test coverage upstream for MAIN_MODULE=1. Once we get to this point, then we could start to look more seriously at enabling threads.
I think we could eventually get threading enabled default, as discussed in https://github.com/pyodide/pyodide/issues/237. But it's a more long term subject.
I suspect we will want to ship a threading-enabled and a threading-disabled version for a long time. The version without threading will be lighter weight, more stable, and easier to debug. Unfortunately, this may be hard. I worry that PROXY_TO_PTHREAD might require significant changes to the design of the Javascript API in src/js. We would also have to look into Emscripten's threading runtimes. Presumably there are parts of the runtime that are main thread only and parts of the runtime that are duplicated in each thread? How does our code interact with this? Presumably we can inject extra stuff like `pyproxy.ts` into the per-thread runtime? Currently we use Emscripten's bundler *and* rollup separately. Maybe we can rollup the code and then ask Emscripten to inject the rolled-up bundle into Emscripten? Now that I think about this, this rollup-then-inject-into-emscripten strategy might lead to a much more elegant solution to https://github.com/pyodide/pyodide/issues/2255
Also, I have no idea whether passing JS objects wrapped into JsProxy between threads is even something we want to attempt. Maybe not. cc @Hood.
From an interface perspective I think it would be possible to do this without any change in interface between the thread-native proxies and the cross-thread proxies. It would even be possible to abstract the differences in such a way as to avoid changing much in jsproxy.c. We probably could store in Hiwire a tag for the thread of origin. I think synclink by itself can be designed to handle the simplest deadlocks -- raise an error if we try to block on a thread when we know it is already blocking on us. On Sun, Apr 10, 2022 at 7:42 AM Roman Yurchak <rth.yurchak@gmail.com> wrote:
Hi Joe,
On 09/04/2022 11:56, Joe Marshall wrote:
Is anyone working on adding synchronous sockets support in pyodide yet? > If not, does anyone have opinions on how it should be done? I can see with support for shared memory across browsers now it would be easy enough to do it with that in JavaScript with a web worker and atomics.wait.
I think the latest discussion about this was in https://github.com/pyodide/pyodide/issues/1503 and Hood also did some work on it in https://github.com/hoodmane/synclink
I agree that using a web worker seems that most direct approach.
The obvious alternative would be to make threading and synchronisation primitives work in python and do fetch calls in a thread. Has anyone built threaded pyodide yet?
Threading is obviously a heavier weight thing, but on the other hand I guess it would fix lots of problems in a way that a dedicated JavaScript solution wouldn't.
Yes, I think we could eventually get treading enabled default, as discussed in https://github.com/pyodide/pyodide/issues/237. But it's a more long term subject.
Any ideas how that would interact with JavaScript?
Good question. Currently, JS access from Python works in web-workers as we initialize a separate Pyodide instance there. If it is created via Emscripten pthreads https://emscripten.org/docs/porting/pthreads.html, I guess it wouldn't work without changes to the FFI code and doing some of that initialization manually. Also, I have no idea whether passing JS objects wrapped into JsProxy between threads is even something we want to attempt. Maybe not. cc @Hood.
But anyway, someone would need to try to build Pyodide with threading enabled first and seeing what happens :)
Roman _______________________________________________ Pyodide mailing list -- pyodide@python.org To unsubscribe send an email to pyodide-leave@python.org https://mail.python.org/mailman3/lists/pyodide.python.org/ Member address: robert.hood.chatham@gmail.com
participants (3)
-
Hood Chatham -
Joe Marshall -
Roman Yurchak