[pypy-dev] Very confused? -can someone at least tell me..? (was naive vm question)

[Peter Fraser]

Ok  -let me try again. If someone could answer 1. below,  then I would 
be very grateful. Better yet, tell me where I leave the road thereafter.

If I still sound hopelessly confused then I won't bother the list 
further, but I will be looking for someone to pay money to, for some 
skype-consulting to explain where I am going wrong.


1. Suppose you ran the PyPy-C interpreter as a CPython extension. Could 
the extension release the CPython GIL?

2. My naive bet is that it could  -because the PyPy interpreter is not 
using CPython "in any essential way".

3. Leaving aside how useful this would be  -can we agree that if indeed 
you could release the CPython GIL, then you could have two threads 
interpreting Python code, in one process, simultaneously.

4. This is possible because you have two of everything. Two code bases, 
two memory managers, two GILS etc etc etc.

5. The separation is so extreme that opportunities for cooperation 
between the two interpreters is severly curtailed. The type 
implementations aren't interchangeable for example.

6. Would this arrangement even be useful? This point is tangential. 
Personally I'm interested in any opportunity to utilize a second core, 
from one CPython process, simply using an extension  -even with the 
severe constraints of messaging between two totally separate, 
incompatible VMs. But that is tangential. If this proved popular then 
that would be great for PyPy i think. But it is tangential.

7. The question I'm really getting at is, are there opportunities for a 
middle ground? Ok, so two threads running in one interpreter with no GIL 
is too intimate. Two distinct interpreters (a CPython and a PyPyC) in 
one process are too isolated (as are two threads in two processes). What 
is in the middle (other than locking)?

8. Yes, I am confusing designs and instances Boris. I'm trying to 
understand exactly what are the design differences, that create the 
implementation diferences,  that conspire to make the CPython with 
PyPy-C extension mix, two separate instances.  I'm speculating about 
whether you could contrive to create *just enough difference* to sustain 
two "instances" of essentialy the same interpreter, in one process. 

9. I'm just talking about sufficient partitioning to achieve 2 instances 
in one process  -where "instance" means that a thread running in either 
will never touch the data of the other.

10. A further wild hunch is that this partitioning might in part be 
achieved by wholesale code duplication, and the munging of names  -as 
opposed to smarter code (viz structurally more sophisticated code to 
support the partition). I wouldn't be surprised to be totally wrong on 
this one and the concept doesn't hinge on it. It does scream 
"translation aspect" to me however.

11. I assume that if its possible, this partitioned "dual core" (as it 
were) interpreter would be far more easily achieved as a PyPy 
translation aspect, than by hand. This, the PyPyC extension GIL 
question, and the aim of PyPy to support different features is why I'm 
asking the question here. Also, much as I don't like fandom as a 
phenomena  -I allow myself to be a PyPy fan.

12. But how useful would this be? Why do I think this would be better 
than simply running two processes with some kind of inter-process 
messaging? Simply speed. Suffice to say that I have a "cross process 
bad, in process good" mindset.

13. I'm betting on the identical types to give you far better 
cooperative efficiency than the CPython + PyPy-C approach, even if we 
don't try to actually share objects  (but hey, that might be possible, a 
la posh, but in one process and on windows).

14. Finally  -the key idea that niggles at me is that while the 
expensive, dense locking required for free threading presumably gives 
you an unbounded(ish) number of threads  -the number of cores on modern 
machines is very bounded. Hence my grasping for fast, in process, 
messaging based concurrency that takes advantage of only having to 
support a bounded number of threads (2 for now) with their own 
instances  -ie "dual core Python" (a good metaphor that) as a PyPy 
translation aspect.


 >>Putting aside the relevance to pypy<<

Well I thought it was, but if someone could just answer 1. above, I 
concede that I could go away.  If there is a more appropriate PyPy list 
for this kind of question  -please tell me.

Twins? I'm after viable Siamese twins. Co-joined but not too co-joined. 
PyPy really is aboue genetic engineering isn't it? Tweak the DNA and 
grow a new interpreter.


regards
Pete F