How to make a builtin pypy module visible?
Hi, I tried to write a simple test module in pypy: 1. Make a new directory in pypy/modules, and put the code there. 2. Add the new module name in pypyoption.py and built the new libpypy-c.so The reason that I put the module in pypy/modules is I tried to make the module build into libpypy-c.so. :) After that, in pypy console, I am able to import the module and run module's function. However, when I tried to import the module in python script, I met the errors: pypy test.py Traceback (most recent call last): File "<builtin>/app_main.py", line 75, in run_toplevel File "test.py", line 1, in <module> import pypytest ImportError: No module named pypytest Any steps I missed to make the buitin module visible?
Oh, I found out I should use the correct pypy(and .so) to run the python script. In previous test, "pypy test.py" point to the old one without the new module. Using the new built pypy, the result is good. On Thu, May 28, 2015 at 6:31 PM, Yicong Huang <hengha.mao@gmail.com> wrote:
Great thanks for you advices! I tried to use the module for the lib in pypy/modules, but observed the following errors from the tests: Exception: object with a __call__ is not RPython: <built-in function test_func> The function test_func is the function from the new built module.
On Thu, May 28, 2015 at 7:50 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Yicong, On 28 May 2015 at 14:50, Yicong Huang <hengha.mao@gmail.com> wrote:
What is the possible casue to the error? And where we could find some syntax introductions on RPython.
http://rpython.readthedocs.org/en/latest/rpython.html Note that this method of adding new modules is generally discouraged: http://pypy.readthedocs.org/en/latest/extending.html A bientôt, Armin.
Thanks for the document! To my current understanding, RPython does not allow a class member function call. Is this correct? The module I wrote looks like this: pytest module: class a def func1: ... s = a() def func: s.func() And in another module to use pytest module, the code is import pytest .pytest.func() I think I should modify pytest module and remove class a: def func: ... Is this correct? On Thu, May 28, 2015 at 9:11 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong, If you really want to add an RPython module in PyPy, you should follow closely the example of another module. The first step is to add tests that run untranslated. Your module must be correct and tested *before* you attempt to translate it. Being correct RPython is not the first concern, as long as you follow the general guidelines shown by the other existing module. The last test to add (after all other tests) is a "test_ztranslation.py", which tests that your module is correct RPython code. We can discuss specific RPython issues you'd have at this point. If you only want to learn RPython, you could also try to write a small interpreter for a custom language. This is the path that other people take, and it allows them to choose which approach they prefer, e.g. if they don't care about untranslated tests. However, if you're working with PyPy, then you have to follow the rules and write tests, otherwise you're unlikely to get much help from any of us. (Also, it would help us get some motivation if you could successfully explain why you need an RPython module instead of, say, using CFFI.) A bientôt, Armin.
Hi Armin, For the RPython module I added, I had added and passed test_ztranslation.py, and sucessfully built libpypy-c.so. And in pypy console, it worked correctly. The problem happened when I tried to use this RPython module for the existed RPython module in pypy/module. The test I failed was test_ztranslation.py. It reported the error: ... else: if hasattr(pyobj, '__call__'): msg = "object with a __call__ is not RPython" else: msg = "unexpected prebuilt constant"
raise Exception("%s: %r" % (msg, pyobj)
E Exception: object with a __call__ is not RPython: <built-in function testFunc> The corresponding line is where I used my RPython module: myModule.testFunc() --------------------------------------------------------------------- And here are my thoughts why I need to use RPython module: I tried to implement a special lightweight sandbox or critical section to block certain builtin function. One user scenario might look like this: criticalSection.start() criticalSection.block('select.epoll') ...some code in critical section... criticcalSection.exit() Different from "usual sandbox", we would like to only block some customized builtin functions for a segment of the code. To achieve this purpose, here are my plans: 1. Write a builtin RPython module 'criticalsection', because I thought only builtin RPython module could be used for existed builtin RPython module 2. For the list of builtin functions that we might block, add the code in the begining of those functions, e.g. def epoll: if criticalsection.isInCriticalSection() and criticalsection.block('select.epoll') return None ... the original code... On Thu, May 28, 2015 at 9:57 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong, On 28 May 2015 at 17:09, Yicong Huang <hengha.mao@gmail.com> wrote:
How about this pure Python solution, which would give the equivalent level of security (i.e., okish against naive code, but no hard security against a motivated attacker): def just_returns_none(*args, **kwds): return None def enter_critical_section(): socket.epoll = just_returns_none original_socket_epoll = socket.epoll def leave_criticial_section(): socket.epoll = original_socket_epoll A bientôt, Armin.
Hi Armin, Thanks for the solution! I tired it, and found out a command "reload(socket)" could break it. So I might consider my previous method is more safer. The protection is at the function entry point. In addition, 'criticalsection' module and customized 'select' module are both build into libpypy-c.so. It might be more difficult to break. And for the mentioned RPython error: "object with a __call__ is not RPython", any advices? On Thu, May 28, 2015 at 11:41 PM, Armin Rigo <arigo@tunes.org> wrote:
Le 28/05/15 17:11, Yicong Huang a écrit :
And for the mentioned RPython error: "object with a __call__ is not RPython", any advices?
In this case, it means that the translator sees a built-in function that it knows nothing about. My guess is that you're using an interpreter that you built yourself to run the translation. You should rather just use CPython to run the translation, it'll be less confusing.
Hi again, I'm suggesting to patch "reload" simply in order to fix the next obvious hole you found. By no means will you ever be able to obtain a truly secure environment in this way, but you can patch things as you find them. Depending on your goal it might be secure enough. For reference, you may be interested in reading more about this topic on the web. A starting point is: http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-... , but use Google to find more. A bientôt, Armin.
Oh, I found out I should use the correct pypy(and .so) to run the python script. In previous test, "pypy test.py" point to the old one without the new module. Using the new built pypy, the result is good. On Thu, May 28, 2015 at 6:31 PM, Yicong Huang <hengha.mao@gmail.com> wrote:
Great thanks for you advices! I tried to use the module for the lib in pypy/modules, but observed the following errors from the tests: Exception: object with a __call__ is not RPython: <built-in function test_func> The function test_func is the function from the new built module.
On Thu, May 28, 2015 at 7:50 PM, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi Yicong, On 28 May 2015 at 14:50, Yicong Huang <hengha.mao@gmail.com> wrote:
What is the possible casue to the error? And where we could find some syntax introductions on RPython.
http://rpython.readthedocs.org/en/latest/rpython.html Note that this method of adding new modules is generally discouraged: http://pypy.readthedocs.org/en/latest/extending.html A bientôt, Armin.
Thanks for the document! To my current understanding, RPython does not allow a class member function call. Is this correct? The module I wrote looks like this: pytest module: class a def func1: ... s = a() def func: s.func() And in another module to use pytest module, the code is import pytest .pytest.func() I think I should modify pytest module and remove class a: def func: ... Is this correct? On Thu, May 28, 2015 at 9:11 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong, If you really want to add an RPython module in PyPy, you should follow closely the example of another module. The first step is to add tests that run untranslated. Your module must be correct and tested *before* you attempt to translate it. Being correct RPython is not the first concern, as long as you follow the general guidelines shown by the other existing module. The last test to add (after all other tests) is a "test_ztranslation.py", which tests that your module is correct RPython code. We can discuss specific RPython issues you'd have at this point. If you only want to learn RPython, you could also try to write a small interpreter for a custom language. This is the path that other people take, and it allows them to choose which approach they prefer, e.g. if they don't care about untranslated tests. However, if you're working with PyPy, then you have to follow the rules and write tests, otherwise you're unlikely to get much help from any of us. (Also, it would help us get some motivation if you could successfully explain why you need an RPython module instead of, say, using CFFI.) A bientôt, Armin.
Hi Armin, For the RPython module I added, I had added and passed test_ztranslation.py, and sucessfully built libpypy-c.so. And in pypy console, it worked correctly. The problem happened when I tried to use this RPython module for the existed RPython module in pypy/module. The test I failed was test_ztranslation.py. It reported the error: ... else: if hasattr(pyobj, '__call__'): msg = "object with a __call__ is not RPython" else: msg = "unexpected prebuilt constant"
raise Exception("%s: %r" % (msg, pyobj)
E Exception: object with a __call__ is not RPython: <built-in function testFunc> The corresponding line is where I used my RPython module: myModule.testFunc() --------------------------------------------------------------------- And here are my thoughts why I need to use RPython module: I tried to implement a special lightweight sandbox or critical section to block certain builtin function. One user scenario might look like this: criticalSection.start() criticalSection.block('select.epoll') ...some code in critical section... criticcalSection.exit() Different from "usual sandbox", we would like to only block some customized builtin functions for a segment of the code. To achieve this purpose, here are my plans: 1. Write a builtin RPython module 'criticalsection', because I thought only builtin RPython module could be used for existed builtin RPython module 2. For the list of builtin functions that we might block, add the code in the begining of those functions, e.g. def epoll: if criticalsection.isInCriticalSection() and criticalsection.block('select.epoll') return None ... the original code... On Thu, May 28, 2015 at 9:57 PM, Armin Rigo <arigo@tunes.org> wrote:
Hi Yicong, On 28 May 2015 at 17:09, Yicong Huang <hengha.mao@gmail.com> wrote:
How about this pure Python solution, which would give the equivalent level of security (i.e., okish against naive code, but no hard security against a motivated attacker): def just_returns_none(*args, **kwds): return None def enter_critical_section(): socket.epoll = just_returns_none original_socket_epoll = socket.epoll def leave_criticial_section(): socket.epoll = original_socket_epoll A bientôt, Armin.
Hi Armin, Thanks for the solution! I tired it, and found out a command "reload(socket)" could break it. So I might consider my previous method is more safer. The protection is at the function entry point. In addition, 'criticalsection' module and customized 'select' module are both build into libpypy-c.so. It might be more difficult to break. And for the mentioned RPython error: "object with a __call__ is not RPython", any advices? On Thu, May 28, 2015 at 11:41 PM, Armin Rigo <arigo@tunes.org> wrote:
Le 28/05/15 17:11, Yicong Huang a écrit :
And for the mentioned RPython error: "object with a __call__ is not RPython", any advices?
In this case, it means that the translator sees a built-in function that it knows nothing about. My guess is that you're using an interpreter that you built yourself to run the translation. You should rather just use CPython to run the translation, it'll be less confusing.
Hi again, I'm suggesting to patch "reload" simply in order to fix the next obvious hole you found. By no means will you ever be able to obtain a truly secure environment in this way, but you can patch things as you find them. Depending on your goal it might be secure enough. For reference, you may be interested in reading more about this topic on the web. A starting point is: http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-... , but use Google to find more. A bientôt, Armin.
participants (4)
-
Armin Rigo -
Maciej Fijalkowski -
Ronan Lamy -
Yicong Huang