[Twisted-Python] verilog simulation calling out to twisted
Hi, I've been writing Verilog simulations for years, and tinkering with Python for years. In the last couple years I have been calling out to Python from Verilog simulations, which has been awesome. Just this week I tried twisted out for the first time to write a simple custom TCP server. It was a great experience and even though I still know next to nothing about twisted, I think I'm in love :-) Now I have a crazy idea. I'd like that Python code that my Verilog calls out to to use twisted. OK, actually I've already done that, my Verilog calls out to my that TCP server that I wrote. I had to spawn a thread and run the server in that thread so that the Verilog could continue to do stuff in parallel with the server. It's working great. The real crazy thing I want is for the Verilog to call out to a twisted TCP client as well. Again, I need the client and server to not block the Verilog. My first attempt was to run client and server in two separate threads. I start the server thread, then I start the client thread and only in the client thread do I call reactor.run(). It seems to not be working. Before I try to figure out why, I thought I might ask here if it even should work. Here's a different way to explain it, in case that helps: main thread is verilog which spawns server_thread def server_thread(): reactor.listenTCP(ServerFactory()) main thread spawns client_thread def client_thread() reactor.connectTCP(ClientFactory()) reactor.run() I read the page on threads in twisted and I'm guessing Would it be better to have the main thread just spawn one thread that does this? def client_server_thread(): reactor.listenTCP(...) reactor.connectTCP(...) reactor.run() Hmm, as I finish typing this all out, I'm realizing that surely someone has written at TCP proxy (essentially what this verilog simulation is doing) in twisted before. A quick internet search tells me yes...and of course they are not using threads. The way the whole verilog mess is written right now it would be much easier to use separate threads for client and server. Is that possible? If not I'll have to find a way to justify re-architecting the verilog mess. Thanks, Bryan
On 2017-09-20, at 22:33, Bryan Murdock <bmurdock@gmail.com> wrote:
I've been writing Verilog simulations for years, and tinkering with Python for years. In the last couple years I have been calling out to Python from Verilog simulations, which has been awesome. Just this week I tried twisted out for the first time to write a simple custom TCP server. It was a great experience and even though I still know next to nothing about twisted, I think I'm in love :-)
Welcome to the club! :)
Now I have a crazy idea. I'd like that Python code that my Verilog calls out to to use twisted. OK, actually I've already done that, my Verilog calls out to my that TCP server that I wrote. I had to spawn a thread and run the server in that thread so that the Verilog could continue to do stuff in parallel with the server. It's working great.
Can you clarify what "your Verilog" is? Python code running a Verilog simulation, maybe? Something else?
The real crazy thing I want is for the Verilog to call out to a twisted TCP client as well. Again, I need the client and server to not block the Verilog. My first attempt was to run client and server in two separate threads. I start the server thread, then I start the client thread and only in the client thread do I call reactor.run(). It seems to not be working. Before I try to figure out why, I thought I might ask here if it even should work.
I do not fully understand this: if "Verilog calls out to TCP server" means it establishes a TCP connection (does it?) what does "Verilog calls out to a TCP client" mean? Does it call a function/method with async Twisted code? How does it handle the fact that the result is async? In general you are able to run concurrent (not parallel) code for TCP clients and servers within the same thread - after all, that's the key idea behind async programming with Twisted, with the reactor driving concurrent handling of multiple connections and events. Of course, again, in general, you'll want your code not to block and to do its things as fast as possible so that control can be returned to the reactor (so that it is ready to quickly handle other events). A few notes about threads and Twisted: - IIRC the reactor should (must?) be running on the main thread. - Threads are a good fit to call out to "I/O blocking code" from Twisted. (talking to a database or doing heavy disk I/O, for example) A few notes about threads and Python: - Python (CPython that is) does not run two CPU-bound threads in parallel. (IOW, it can't simultaneously use two CPUs/CPU-cores in the system) - That's due to the GIL: there's lot's of information about it on the internet. - You will get lower I/O responsiveness even with a single CPU-bound + one I/O bound thread. If "Verilog" needs to continue working and is, I assume, CPU-bound, you will probably be better off by having that run in a process and your Twisted code in a single-threaded separate process. Benefits: - Better separation of concerns. - No mixing of sync/async code which at times can lead to complex design/code. - No shared memory between "Verilog" and Twisted code. - "Verilog" can use as much CPU as it needs. - "Twisted" will be as responsive as possible. - Easier to test. Costs: - Need to define how to start both processes. - Need to define how those processes communicate. - All communication between processes needs to be serialized (no shared memory). These are a few general ideas you may want to consider. Like I said, I couldn't fully grasp where you're currently at and what exactly you're trying to achieve but hopefully these comments will serve as guidance. Cheers, -- exvito
On Thu, Sep 21, 2017 at 4:07 AM, ex vito <ex.vitorino@gmail.com> wrote:
On 2017-09-20, at 22:33, Bryan Murdock <bmurdock@gmail.com> wrote:
Welcome to the club! :)
Thanks!
Now I have a crazy idea. I'd like that Python code that my Verilog calls out to to use twisted. OK, actually I've already done that, my Verilog calls out to my that TCP server that I wrote. I had to spawn a thread and run the server in that thread so that the Verilog could continue to do stuff in parallel with the server. It's working great.
Can you clarify what "your Verilog" is? Python code running a Verilog simulation, maybe? Something else? <snip/> I do not fully understand this: if "Verilog calls out to TCP server" means it establishes a TCP connection (does it?) what does "Verilog calls out to a TCP client" mean? Does it call a function/method with async Twisted code? How does it handle the fact that the result is async?
Sorry, I was worried that this would be too hard to explain and/or I may be giving too many details and confusing the issue. Let me try again. It's probably simplest to think of the whole simulation as a TCP proxy. It looks like this: some TCP client process <--> Python TCP Server <--> Verilog <--> Python TCP Client <--> some TCP server The Verilog, Python TCP server, and Python TCP client are all running in the same process (some TCP client and some TCP server can be a web browser and web server or whatever). The Verilog code starts up, calls a python function to start the Python TCP Server as a thread, calls another python function to start the Python TCP Client as a thread, and then essentially just loops asking (polling) the Python TCP Server if it has any data, if so, it calls a function to send that data to the Python TCP Client. It does the same in reverse to flow data from client to server. I originally wrote the Python TCP Client and Server as low-level socket code, handling only one connection at a time and not automatically reconnecting if a connection was lost. All the downsides of using multiple processes instead of threads that you listed are exactly the reasons I used threads. Verilog has no support for spawning subprocesses or doing interprocess communication. It can't spawn threads either, to be fair though[1]. I have to do "real programming" things like that by calling python functions. Threads just seemed easier at the time because I can just use a Queue to get data to/from the server and client threads from/to the Verilog. And it all worked great. Then, when I went to add support for multiple connections and auto-reconnects I discovered twisted. I rewrote the Python TCP server and client with twisted but now data doesn't flow. I assume that's because the client and server are making thread-unsafe calls to the reactor. Is that true? (By the way, I can run the client and server standalone as separate processes, without verilog involved, and they work fine that way. I have also replaced the twisted Python TCP Client with my original socket-based TCP client and everything works fine. It's just when I try to use twisted for both client and server that it fails).
In general you are able to run concurrent (not parallel) code for TCP clients and servers within the same thread - after all, that's the key idea behind async programming with Twisted, with the reactor driving concurrent handling of multiple connections and events.
Right, I think the solution is going to have to be: have verilog spawn one thread that runs the Python TCP Server and Client. Or start them both as subprocesses and figure out how to communicate with them, I guess. Bryan 1. Aside from it's "green threads" that it uses to model concurrent digital logic
participants (2)
-
Bryan Murdock -
ex vito