Very confused? -can someone at least tell me..? (was naive vm question)

Ok -let me try again. If someone could answer 1. below, then I would be very grateful. Better yet, tell me where I leave the road thereafter. If I still sound hopelessly confused then I won't bother the list further, but I will be looking for someone to pay money to, for some skype-consulting to explain where I am going wrong. 1. Suppose you ran the PyPy-C interpreter as a CPython extension. Could the extension release the CPython GIL? 2. My naive bet is that it could -because the PyPy interpreter is not using CPython "in any essential way". 3. Leaving aside how useful this would be -can we agree that if indeed you could release the CPython GIL, then you could have two threads interpreting Python code, in one process, simultaneously. 4. This is possible because you have two of everything. Two code bases, two memory managers, two GILS etc etc etc. 5. The separation is so extreme that opportunities for cooperation between the two interpreters is severly curtailed. The type implementations aren't interchangeable for example. 6. Would this arrangement even be useful? This point is tangential. Personally I'm interested in any opportunity to utilize a second core, from one CPython process, simply using an extension -even with the severe constraints of messaging between two totally separate, incompatible VMs. But that is tangential. If this proved popular then that would be great for PyPy i think. But it is tangential. 7. The question I'm really getting at is, are there opportunities for a middle ground? Ok, so two threads running in one interpreter with no GIL is too intimate. Two distinct interpreters (a CPython and a PyPyC) in one process are too isolated (as are two threads in two processes). What is in the middle (other than locking)? 8. Yes, I am confusing designs and instances Boris. I'm trying to understand exactly what are the design differences, that create the implementation diferences, that conspire to make the CPython with PyPy-C extension mix, two separate instances. I'm speculating about whether you could contrive to create *just enough difference* to sustain two "instances" of essentialy the same interpreter, in one process. 9. I'm just talking about sufficient partitioning to achieve 2 instances in one process -where "instance" means that a thread running in either will never touch the data of the other. 10. A further wild hunch is that this partitioning might in part be achieved by wholesale code duplication, and the munging of names -as opposed to smarter code (viz structurally more sophisticated code to support the partition). I wouldn't be surprised to be totally wrong on this one and the concept doesn't hinge on it. It does scream "translation aspect" to me however. 11. I assume that if its possible, this partitioned "dual core" (as it were) interpreter would be far more easily achieved as a PyPy translation aspect, than by hand. This, the PyPyC extension GIL question, and the aim of PyPy to support different features is why I'm asking the question here. Also, much as I don't like fandom as a phenomena -I allow myself to be a PyPy fan. 12. But how useful would this be? Why do I think this would be better than simply running two processes with some kind of inter-process messaging? Simply speed. Suffice to say that I have a "cross process bad, in process good" mindset. 13. I'm betting on the identical types to give you far better cooperative efficiency than the CPython + PyPy-C approach, even if we don't try to actually share objects (but hey, that might be possible, a la posh, but in one process and on windows). 14. Finally -the key idea that niggles at me is that while the expensive, dense locking required for free threading presumably gives you an unbounded(ish) number of threads -the number of cores on modern machines is very bounded. Hence my grasping for fast, in process, messaging based concurrency that takes advantage of only having to support a bounded number of threads (2 for now) with their own instances -ie "dual core Python" (a good metaphor that) as a PyPy translation aspect.
Putting aside the relevance to pypy<<
Well I thought it was, but if someone could just answer 1. above, I concede that I could go away. If there is a more appropriate PyPy list for this kind of question -please tell me. Twins? I'm after viable Siamese twins. Co-joined but not too co-joined. PyPy really is aboue genetic engineering isn't it? Tweak the DNA and grow a new interpreter. regards Pete F

Hi Peter, On Mon, Nov 19, 2007 at 04:29:00AM +1300, Peter Fraser wrote:
1. Suppose you ran the PyPy-C interpreter as a CPython extension. Could the extension release the CPython GIL?
Yes. The part of your argument which seems confusing, though, is that you don't really need to have one CPython and one PyPy in order to run Python code on two CPUs in parallel. You don't gain much (except confusion) by running one CPython and one PyPy for this purpose. (There are other reasons why you might want to do that, like using some special PyPy feature in a subpart of your CPython application.) In fact, the only restriction is that putting two CPythons in the same process is not really possible. Aside from that, you could do any of the following: * run several CPythons in two different processes and communicate using pipes, or shared memory, etc. * run several PyPys in either the same or different processes An advantage of PyPy over CPython in this domain is that because multiple PyPys can run in the same process you could experiment with communication models based on in-process shared data and locks. A bientot, Armin
participants (2)
-
Armin Rigo
-
Peter Fraser