[pypy-dev] Reference counting

[Marko Tasic]

Maciej,

Thank you for an express response.

> Hi Marko, we're very glad to hear that!

Off topic, but I have to mention that I've been an ambassador of PyPy
for last few years.
In beginning, it was hard, but now results that you've achieved are
obvious and promising,
and everyone starts to trust your implementation. However, I have to
say that there
are many notoriously xenophobic developers. Funny thing is that they
trust how their
source code is written but not how its executed.

> * refcounting is implemented as a transformation of flow graphs. See
> pypy/rpython/memory/gctransform/refcounting.py for details. This
> approach works very well in the sense we don't have to maintain
> refcounts by hand in the interpreter (we'll never do that). This is
> all well and good, however it also means there is a significant
> redundancy in references. You would need to either implement a smarter
> policy or a refcount-removing optimization (typically run after
> inlining) in order for that to be viable.

Is there a way to always have explicit reference counting field for all objects?
I know this is bad, but I want to go even further. I would like to have even
more fields in object structure besides refcount field. Reason for this is a
way to test correctness of alternative RC implementations that handle cycles.
I guess that I should turn off inlining?

> * we generally never cared about avoiding cycles in the python
> interpreter itself. I don't think pypy's refcounting implementation
> comes with cycle detector included, but don't quote me on that

I've developed an alternative way of handling cycles related to reference
counting without using tracing GC, or traversing items in container objects
such are list, dict, set, etc. Still checking algorithm and testing correctness,
but everything has been fine until now. It is not my intention to make anything
spectacular but just to research this field.

If anyone has interest in my reference counting mechanism can check:
http://code.google.com/p/cosmos-lang/wiki/CosmosRefCounter
http://code.google.com/p/cosmos-lang/source/browse/doc/corc.py

Instance of "Reference" class can be embedded in "Abstract"
class/instance, so no
need for separate object.

> * there is not JIT support. Typically JIT requires a bit of support
> from the GC to cooperate. In order to get good results, you would
> probably also need to implement an optimization that removes
> unnecessary refcounts in traces.

I don't mind if there is no JIT support, but I want to have
"unnecessary refcounts".
Again, my main interest is testing correctness and not speed. Later
on, I will try
to optimize interaction between GC and JIT if possible.

Cheers,
Marko Tasic


On Sat, Jan 7, 2012 at 1:15 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
>
> On Sat, Jan 7, 2012 at 1:46 PM, Marko Tasic <mtasic85 at gmail.com> wrote:
> > Hi,
> >
> > I've been carefully following your mailing list for years, and using PyPy
> > for
> > different kind of projects, mostly for highly distributed and decentralized
> > systems,
> > and everything is just great compared to CPython, Jython and IronPython.
> > I've even written partial ctypes wrapper for GObject Introspection, so I
> > could
> > use Gtk as GUI toolkit on top of PyPy. Today, I cannot imagine writing
> > Python
> > code without executing it on PyPy because it just feels natural to use it as
> > default
> > implementation instead of an alternative CPython :)
>
> Hi Marko, we're very glad to hear that!
>
> >
> > I know that you don't feel that using reference counting as GC mechanism is
> > good
> > idea because of overhead of maintaining reference counts to each object and
> > non concurrent nature of it, but can you give me any idea where to start and
> > how
> > I can implement reference counting for PyPy. Have in mind that I am not new
> > to
> > Python, and low level stuff, just want to measure performance and possibly
> > implement alternative reference counting strategies.
>
> it is a very valid question. In fact, we already do have a refcounting
> implementation which you get by passing --gc=ref to the translate.py
> script. Refcounting has several ups and downs and it's definitely
> valid to experiment and see overheads. However, at least current
> approach and pypy in general has several problems with that:
>
> * refcounting is implemented as a transformation of flow graphs. See
> pypy/rpython/memory/gctransform/refcounting.py for details. This
> approach works very well in the sense we don't have to maintain
> refcounts by hand in the interpreter (we'll never do that). This is
> all well and good, however it also means there is a significant
> redundancy in references. You would need to either implement a smarter
> policy or a refcount-removing optimization (typically run after
> inlining) in order for that to be viable.
>
> * we generally never cared about avoiding cycles in the python
> interpreter itself. I don't think pypy's refcounting implementation
> comes with cycle detector included, but don't quote me on that
>
> * there is not JIT support. Typically JIT requires a bit of support
> from the GC to cooperate. In order to get good results, you would
> probably also need to implement an optimization that removes
> unnecessary refcounts in traces.
>
> I hope that helps clarify some things, feel free to ask more if not!
>
> Cheers,
> fijal