seeking pypy gc configuring guideline
I was trying to run pypy using semispace gc. But as it stands currently I can only compile using incminimark and boehm. What changes I have to make so that I would be able to test pypy with any existing gc implementation? Moreover I want to write my own version of gc and test with pypy. In future. Can anyone point me where do I have to make changes to make my implementation of any arbitrary gc work? So far I was able to compile an edited version of incminimark gc. Any suggestions or link to any perticular documentation would be appreciated. p.s. I ran pypy with the following command and got this error. I can run same command just replacing incminimark in place of semispace. [image: image.png]
Hi Raihan, On Sat, 22 May 2021 at 09:05, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
This screenshot contains the whole explanation in bold red text: some components of PyPy have evolved over time to depend on special features of the GC. These features have been implemented only in the 'incminimark' and the 'boehm' GC. For experimentation with other GCs, you can disable these components---like the 'cpyext' module of PyPy---by following the instructions in the bold red text. Note also that if you want to play with the GC, it makes little sense to translate a complete PyPy every time. You should at least have a small RPython program, much simpler than the whole PyPy, and use that for testing. But ideally you should look at the way we write various tests in rpython/memory/. A bientôt, Armin
Hello Armin, Thanks for helping me out. I was actually looking for a generalized solution. The last paragraph of your answer covers that. What I understand from that is, if I want to understand how pypy gc works and if i want to write my own version of GC at first I have to understand all the tests written on rpython/memory. I will now look extensively into that. I have tried to understand some of the test codes earlier but there is a problem that I faced. Suppose gc_test_base.py written in rpython/memory/test not only uses modules written in rpython.memory but also uses modules like llinterp from rpython.rtyper. As I don't know how those modules work how do I figure out their function written in the test codes? Do you have any suggestions for someone who is facing this issue? On Sat, May 22, 2021 at 2:00 PM Armin Rigo <armin.rigo@gmail.com> wrote:
Hi Raihan, On Sat, 22 May 2021 at 14:40, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
Thanks for helping me out. I was actually looking for a generalized solution. The last paragraph of your answer covers that. What I understand from that is, if I want to understand how pypy gc works and if i want to write my own version of GC at first I have to understand all the tests written on rpython/memory. I will now look extensively into that.
I have tried to understand some of the test codes earlier but there is a problem that I faced. Suppose gc_test_base.py written in rpython/memory/test not only uses modules written in rpython.memory but also uses modules like llinterp from rpython.rtyper. As I don't know how those modules work how do I figure out their function written in the test codes? Do you have any suggestions for someone who is facing this issue?
Modules from rpython.rtyper.lltypesystem are essential. You need to learn about them to the extent that they are used. They have test files in "test" subdirectories, like almost every file in PyPy and RPython. Armin
Hello Carl, Thanks for your response. I am currently trying to do my thesis. My long term goal is to create a binding of pypy with mmtk ( https://github.com/mmtk/mmtk-core) so that I can test some of the memory manager written in this mmtk platform on pypy. In order to do that at first I have to able to write a pypy memory manager like incminimark that can communicate with the rest of the compiler and extract all information that is needed for a memory manager to function well. Then I will write an api for mmtk and pass those information to mmtk platform so that it can use its memory manager to perform garbage collection for pypy. In that way I would be able to test how new garbage collection algorithms behave in pypy and if I am lucky then I will find some possible imporvements. Any suggestion given by anyone will be much appreciated. Thanks, Raihan On Sun, May 23, 2021 at 1:27 PM Carl Friedrich Bolz-Tereick <cfbolz@gmx.de> wrote:
Hello Armin, I have read the test codes of test_llinterp.py and llinterp.py. I am writing a summary below. I just want you to point out my mistakes if I am wrong. Please help me to understand this. Basically, llinterp is an object that can interpret given a python function and function's arguments. It is used to test any component of pypy. If we plug any component (that we want to test) into llinterp such as if we plug it with a heap that has semispace gc the interpreter will use semispace gc when it needs to use gc functionality. So if I get desired output with semispace gc when I plug it into llinterp and do my test then I can assume that if I build pypy properly using semispace gc that newly build pypy would behave the same given the same test input. Please do me a favor by commenting on this summary. This will help to know whether my understanding is right on this matter. Moreover, can you tell me why some of the member functions written in class LLFrame (inside lliterp.py) only raise NotImplmentedError exception. Methods such as op_gc_heap_stats, op_gc_obtain_free_space, and many more just only raise this exception without doing anything. Thanks a lot! -Raihan On Sun, May 23, 2021 at 9:00 PM Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
Hi Raihan, On Thu, 27 May 2021 at 17:44, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
I have read the test codes of test_llinterp.py and llinterp.py. I am writing a summary below. I just want you to point out my mistakes if I am wrong. Please help me to understand this.
There are many tests doing various checks at various levels. I think you should focus on the "final" tests that try to translate the GC to C and run it: they are in rpython/translator/c/test/test_newgc.py. The reason I'm suggesting to focus on these tests instead of looking at all intermediate levels is that, if I'm understanding it correctly, you mostly want to plug in an existing library to do the GC. You're thus going to fight problems of integration more than problems of algorithmically correct code. It's OK if you ignore the other tests because they are likely to not work out of the box in your case. For example they're half-emulating the actual memory layout of the objects, but your external library cannot work with such emulation. So I'd recommend looking at test_newgc instead of, say, rpython/rtyper/llinterp.py, which is here only to make this emulation work. test_newgc contains tests of the form of a top-level RPython function (typically called f()) that does random RPython things that need memory allocation (which is almost anything that makes lists, tuples, instances of classes, etc.). These RPython things are automatically translated to calls to lltype.malloc(). Some of the tests in test_newgc are directly calling lltype.malloc() too. For the purpose of this test, the two kinds of approaches are equivalent. Note also that in the past, during development of the branch "stmgc-c8", we made a different GC in C directly and integrated it a bit differently, in a way that is simpler and makes more sense for external GCs. Maybe you want to look at this old branch instead. But it can also be confusing because this STM GC added many kinds of unusual barriers (rpython/memory/gctransform/stmframework.py in this branch). A bientôt, Armin.
Hi Raihan, On Sat, 22 May 2021 at 09:05, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
This screenshot contains the whole explanation in bold red text: some components of PyPy have evolved over time to depend on special features of the GC. These features have been implemented only in the 'incminimark' and the 'boehm' GC. For experimentation with other GCs, you can disable these components---like the 'cpyext' module of PyPy---by following the instructions in the bold red text. Note also that if you want to play with the GC, it makes little sense to translate a complete PyPy every time. You should at least have a small RPython program, much simpler than the whole PyPy, and use that for testing. But ideally you should look at the way we write various tests in rpython/memory/. A bientôt, Armin
Hello Armin, Thanks for helping me out. I was actually looking for a generalized solution. The last paragraph of your answer covers that. What I understand from that is, if I want to understand how pypy gc works and if i want to write my own version of GC at first I have to understand all the tests written on rpython/memory. I will now look extensively into that. I have tried to understand some of the test codes earlier but there is a problem that I faced. Suppose gc_test_base.py written in rpython/memory/test not only uses modules written in rpython.memory but also uses modules like llinterp from rpython.rtyper. As I don't know how those modules work how do I figure out their function written in the test codes? Do you have any suggestions for someone who is facing this issue? On Sat, May 22, 2021 at 2:00 PM Armin Rigo <armin.rigo@gmail.com> wrote:
Hi Raihan, On Sat, 22 May 2021 at 14:40, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
Thanks for helping me out. I was actually looking for a generalized solution. The last paragraph of your answer covers that. What I understand from that is, if I want to understand how pypy gc works and if i want to write my own version of GC at first I have to understand all the tests written on rpython/memory. I will now look extensively into that.
I have tried to understand some of the test codes earlier but there is a problem that I faced. Suppose gc_test_base.py written in rpython/memory/test not only uses modules written in rpython.memory but also uses modules like llinterp from rpython.rtyper. As I don't know how those modules work how do I figure out their function written in the test codes? Do you have any suggestions for someone who is facing this issue?
Modules from rpython.rtyper.lltypesystem are essential. You need to learn about them to the extent that they are used. They have test files in "test" subdirectories, like almost every file in PyPy and RPython. Armin
Hello Carl, Thanks for your response. I am currently trying to do my thesis. My long term goal is to create a binding of pypy with mmtk ( https://github.com/mmtk/mmtk-core) so that I can test some of the memory manager written in this mmtk platform on pypy. In order to do that at first I have to able to write a pypy memory manager like incminimark that can communicate with the rest of the compiler and extract all information that is needed for a memory manager to function well. Then I will write an api for mmtk and pass those information to mmtk platform so that it can use its memory manager to perform garbage collection for pypy. In that way I would be able to test how new garbage collection algorithms behave in pypy and if I am lucky then I will find some possible imporvements. Any suggestion given by anyone will be much appreciated. Thanks, Raihan On Sun, May 23, 2021 at 1:27 PM Carl Friedrich Bolz-Tereick <cfbolz@gmx.de> wrote:
Hello Armin, I have read the test codes of test_llinterp.py and llinterp.py. I am writing a summary below. I just want you to point out my mistakes if I am wrong. Please help me to understand this. Basically, llinterp is an object that can interpret given a python function and function's arguments. It is used to test any component of pypy. If we plug any component (that we want to test) into llinterp such as if we plug it with a heap that has semispace gc the interpreter will use semispace gc when it needs to use gc functionality. So if I get desired output with semispace gc when I plug it into llinterp and do my test then I can assume that if I build pypy properly using semispace gc that newly build pypy would behave the same given the same test input. Please do me a favor by commenting on this summary. This will help to know whether my understanding is right on this matter. Moreover, can you tell me why some of the member functions written in class LLFrame (inside lliterp.py) only raise NotImplmentedError exception. Methods such as op_gc_heap_stats, op_gc_obtain_free_space, and many more just only raise this exception without doing anything. Thanks a lot! -Raihan On Sun, May 23, 2021 at 9:00 PM Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
Hi Raihan, On Thu, 27 May 2021 at 17:44, Raihan Rasheed Apurbo <apurbo97@gmail.com> wrote:
I have read the test codes of test_llinterp.py and llinterp.py. I am writing a summary below. I just want you to point out my mistakes if I am wrong. Please help me to understand this.
There are many tests doing various checks at various levels. I think you should focus on the "final" tests that try to translate the GC to C and run it: they are in rpython/translator/c/test/test_newgc.py. The reason I'm suggesting to focus on these tests instead of looking at all intermediate levels is that, if I'm understanding it correctly, you mostly want to plug in an existing library to do the GC. You're thus going to fight problems of integration more than problems of algorithmically correct code. It's OK if you ignore the other tests because they are likely to not work out of the box in your case. For example they're half-emulating the actual memory layout of the objects, but your external library cannot work with such emulation. So I'd recommend looking at test_newgc instead of, say, rpython/rtyper/llinterp.py, which is here only to make this emulation work. test_newgc contains tests of the form of a top-level RPython function (typically called f()) that does random RPython things that need memory allocation (which is almost anything that makes lists, tuples, instances of classes, etc.). These RPython things are automatically translated to calls to lltype.malloc(). Some of the tests in test_newgc are directly calling lltype.malloc() too. For the purpose of this test, the two kinds of approaches are equivalent. Note also that in the past, during development of the branch "stmgc-c8", we made a different GC in C directly and integrated it a bit differently, in a way that is simpler and makes more sense for external GCs. Maybe you want to look at this old branch instead. But it can also be confusing because this STM GC added many kinds of unusual barriers (rpython/memory/gctransform/stmframework.py in this branch). A bientôt, Armin.
participants (3)
-
Armin Rigo -
Carl Friedrich Bolz-Tereick -
Raihan Rasheed Apurbo