Hi,
Perhaps it would be better to have this conversation on the issue tracker? Or maybe we can copy the relevant parts over.
> synchronous sockets support
As Roman said, I have
https://github.com/hoodmane/synclink, which is a proof of concept of a strategy of using Atomics.wait, a Comlink-like API, and a hand implemented polling event loop on the blocking thread. It needs some work on CI, testing, and specifying the design/behavior of the proxies a bit more carefully. Could also use some ease-of-use improvements. But it works as a prototype. For example I have code here:
https://github.com/hoodmane/worker-pyodide-console/tree/nativefsthat implements an Emscripten file system based on the native file system API using synclink. This is an example of the synchronous C FS API consuming the asynchronous Javascript FS API by proxying to a different thread. You can try it out with the following code (only in chrome):
import pathlib
from native_fs import mount_native_fs
# this next line will open a directory picker, then ask
# first for read permissions and then for write permissions
mount_native_fs("/native")
# This next line creates a file called "blah.txt" in the folder you just selected.
pathlib.Path("/native/blah.txt").write_text("hi from browser?")
These have to be loaded on the appropriate sides and hooked up, but the setup code is very simple. Basically all of the complexity is hidden into synclink.
I haven't thought about sockets at all (is the idea to translate reads/writes into fetch POST and GET?) but the problem of consuming an async API from a sync API is totally solvable.
> Has anyone built threaded pyodide yet? Does webassembly threading work in WebKit now? Any ideas how that would interact with
JavaScript?
>
> Threading is obviously a heavier weight thing, but on the other
hand I guess it would fix lots of problems in a way that a dedicated
JavaScript solution wouldn't.
Yeah so Safari 15 supports Atomics and SharedArrayBuffer and everything. Unfortunately more people are currently using Safari <=14 than Safari 15 since the version of Safari is tied to the device. This pattern is different from Chrome, Edge, and Firefox where almost all users use the most recent version. As soon as we are comfortable dropping "support" for Safari v14 we should switch to WASM_BIGINT. Though I think
https://github.com/emscripten-core/emscripten/pull/16693 might end our issues with MAIN_MODULE=1 and not WASM_BIGINT.
Also, we should seriously consider adding Safari to CI. I think we get the worst of both worlds by allowing Safari limitations to prevent us from using browser features but also not testing in Safari so we might break it anyways.
I am concerned that if we try to combine MAIN_MODULE=1 with PTHREADS we will see a lot of bugs. MAIN_MODULE=1 by itself has almost no test coverage in the Emscripten test suite -- all the tests use MAIN_MODULE=2. One good project would be to modify the Emscripten test suite to automatically run all dynlink tests with both MAIN_MODULE=1 and MAIN_MODULE=2. I am sure we will find more bugs.
So my thought is first try to catch up with upstream Emscripten (only blocker is a Firefox bug
https://github.com/emscripten-core/emscripten/issues/16538). Also add test coverage upstream for MAIN_MODULE=1. Once we get to this point, then we could start to look more seriously at enabling threads.
I suspect we will want to ship a threading-enabled and a threading-disabled version for a long time. The version without threading will be lighter weight, more stable, and easier to debug. Unfortunately, this may be hard. I worry that PROXY_TO_PTHREAD might require significant changes to the design of the Javascript API in src/js. We would also have to look into Emscripten's threading runtimes. Presumably there are parts of the runtime that are main thread only and parts of the runtime that are duplicated in each thread? How does our code interact with this? Presumably we can inject extra stuff like `pyproxy.ts` into the per-thread runtime? Currently we use Emscripten's bundler *and* rollup separately. Maybe we can rollup the code and then ask Emscripten to inject the rolled-up bundle into Emscripten?
Now that I think about this, this rollup-then-inject-into-emscripten strategy might lead to a much more elegant solution to
> Also, I have no idea whether passing JS
> objects wrapped into JsProxy between threads is even something we want
> to attempt. Maybe not. cc @Hood.
From an interface perspective I think it would be possible to do this without any change in interface between the thread-native proxies and the cross-thread proxies. It would even be possible to abstract the differences in such a way as to avoid changing much in jsproxy.c. We probably could store in Hiwire a tag for the thread of origin. I think synclink by itself can be designed to handle the simplest deadlocks -- raise an error if we try to block on a thread when we know it is already blocking on us.