Mailman 3 July 2008 - pypy-dev

numpy and pypy : info required
by rahul garg 31 Jul '08

31 Jul '08

Hi. I am writing a compiler called unPython (http://code.google.com/p/unpython) a very young and immature compiler for compiling Python to C. I am presenting the compiler at SciPy conference and need to write a paper about it. Under related work, I want to describe PyPy and also the status of NumPy support in PyPy. Can someone comment on whats the status of NumPy support in PyPy? Btw, unPython is a compiler I am writing for my Masters thesis. It compiles Python to C code with particular focus on NumPy. In fact currently, not much of Python is supported. One particular focus of unPython is parallel computing .. in particular OpenMP like parallel-for loops. I will release a new (much better) version over the next 1-2 weeks and will let people here know as well if its not considered spam :) So if you want to try out the compiler, I would request you to wait 1-2 weeks. thanks, rahul

2 1

Re: [pypy-dev] Zope2-RPython implementation help wanted..
by Chris Withers 31 Jul '08

31 Jul '08

Martijn Faassen wrote: >>> * try to patch up RestrictedPython by closing any new Python 2.5 >>> related holes. This would require figuring out what these are. >> This has been "the plan", it has, unfortunately, met prettymuch universal >> derision from the PyPy guys and silence from Jim :-( > > Well, the PyPy people aren't familiar with this code base. It's been > proven enough technology for the last 8 years or so, so I suspect this > plan is at least feasible, if hardly perfect. Well, I dunno. Part of me is trying to get to the bottom of why the PyPy guys hate RestrictedPython so much. It seems a robust way of providing a secured (as opposed to sandboxed) pthon environment and while it requires work when the AST changes, I've yet to see someone suggest a suitable alternative, as this discussion has kindof showed ;-) >> It's a shame we can't form a better "blood/brain barrier" that proxies on >> the way in and de-proxies on the way out... > > Uh, Chris, that was exactly what I was talking about. You could do so, > but you'd need to alter the way proxies work. I'm not sure where you could generically plug in the un-proxying code... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 2

Re: [pypy-dev] Zope2-RPython implementation help wanted..
by holger krekel 31 Jul '08

31 Jul '08

Hi Martijn, Chris, On Mon, Jul 28, 2008 at 11:09 +0200, Martijn Faassen wrote: > On Fri, Jul 25, 2008 at 8:46 PM, holger krekel <holger(a)merlinux.eu> wrote: > [description of how this might work] > > Cool, thanks for the description. you are welcome! > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. > > If I were to write a new version of sandboxed Python in a new > framework, this sounds potentially useful. I know Chris is interested > in such technology aside from the interest from the Zope 2 > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Indeed, what i described is not likely to come to happen in the next weeks. > The pay-off would be a more secure version of Python Script in Zope 2 > that is hopefully more easy to port to new Python versions. That would > be nice, but the risks involved in this project are also rather huge, > and unfortunately I don't think the pay-off is huge enough to warrant > a lot of investment in time by (rare) Zope 2 hackers. I'd be happy if > one of them proved me wrong, but it doesn't seem to be a good idea to > let the new Python 2.5 compatible version of Zope 2 be dependent on > unfinished PyPy technology.... Makes sense to me as it stands. I am sure that when the time comes we'll find good use uses for a robust virtualized embeddable Python Interpreter. Having people or parties already asking for it (or better yet, investing into it :-) would certainly accelerate it. cheers, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org

2 1

Unable to translate locks
by Marek Paška 30 Jul '08

30 Jul '08

Hi, PyPy hackers, I am playing with Python translation to C, especially with threads. I am using start_new_thread function provided by pypy.module.thread.ll_thread. My example (see mytasks.py in attachment) works fine. But when I tried to use locks (provided by the same module), I got an exception during the translation (see mytasks_err.py for the source code and translation_trace.txt). End of the trace: [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/database.py", line 156, in getcontainernode [translation:ERROR] node = nodefactory(self, T, container, **buildkwds) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/node.py", line 860, in opaquenode_factory [translation:ERROR] raise Exception("don't know about %r" % (T,)) [translation:ERROR] Exception: don't know about <struct RPyOpaque_ThreadLock (opaque)> I have no idea whats the problem, translation of pypy interpreter with thread module enabled works perfectly. Maybe it is caused by the fact that pypy interpreter uses some special annotation policy, is it possible? Please, I would be thankful for any suggestions/comments what to do to make the locks work. Marek Paška

3 2

Re: [pypy-dev] Zope2-RPython implementation help wanted..
by Chris Withers 28 Jul '08

28 Jul '08

Martijn Faassen wrote: > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. Agreed. > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Agreed. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

1 0

Re: [pypy-dev] Zope2-RPython implementation help wanted..
by Chris Withers 25 Jul '08

25 Jul '08

Hi All, Martijn Faassen wrote: > I think it's unlikely path to try to make pypy-sandbox, a special > version of PyPy, work in a Zope 2 context right now. I'm inclined to agree, but I'd avoid writing it off until the PyPy guys have agreed. I would be surprised if they can pull something out of their box of tricks here, we may just need to find the right way to phrase the problem ;-) At the very least, we should be trying to find the "right" way for realms beyond Zope 2, which it sounds like both myself and Stefan are encountering... > I think there are two possible paths: > > * try to patch up RestrictedPython by closing any new Python 2.5 > related holes. This would require figuring out what these are. This has been "the plan", it has, unfortunately, met prettymuch universal derision from the PyPy guys and silence from Jim :-( > * reimplement RestrictedPython using zope 3 security proxies This is only half the problem, though... > Zope 3 security proxies right now do the following: > > 1) control any attribute access/setting by asking a security policy > whether it's allowed > > 2) if you call a method on a proxied object, the result will be proxied as well > > 3) if you pass something into a method on a proxied object, the thing > passed as an argument will be proxied as well. > > I think to implement something like Restricted Python we would need 1 > (with a Zope 2 specific custom security policy) and 2, but not 3. 3 > causes proxies to start spreading through Zope 2 code Why would they cause havoc? They'd only result from stuff in a python script calling back into Zope, surely? It's a shame we can't form a better "blood/brain barrier" that proxies on the way in and de-proxies on the way out... > The hope of this strategy is that it's easier to maintain in the face > of new Python versions than the current RestrictedPython approach. Right, but there's still the language problems, of which importing is the most serious, but I'm sure there are more lurking there... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 1

Re: [pypy-dev] I have a problem interpreting pypy functionality.
by Diego Jacobi 23 Jul '08

23 Jul '08

2008/7/23 Simon Burton <simon(a)arrowtheory.com>: > On Wed, 23 Jul 2008 02:37:20 -0300 > "Diego Jacobi" <jacobidiego(a)gmail.com> wrote: > >> >> Is something about a negative slice, i replaced all my negative slices >> but the error still saying the same. > > try adding assert(index>=0) statements. > >> >> Is it possible an error of pypy? or it is sure that it is my code. Why >> doesnt point me to the line being parsed? > > It is a "global" error. Very common when compiling rpython. > > Simon. > Thanks that solved the problem, i will have to write lots of asserts. I think that this tells me that i can not continue with this: File "/home/data/diego/Desktop/Comparativa/PyPy/pypy-dist/pypy/annotation/annrpython.py", line 317, in ondegenerated raise AnnotatorError(msgstr) pypy.tool.error.AnnotatorError: annotation of v0 degenerated to SomeObject() v0 = getattr(terms_0, ('sort')) Happened at file /home/data/diego/Desktop/Comparativa/PyPy/QuineMcClusky.py line 51 ==> terms.sort( __sort_condition ) marks = [ False for x in range(len(ones)) ] .. SomeObject() origin: <FunctionGraph of (QuineMcClusky:19)reduce at 0x889eedc> block@240 op=0 Previous annotation: (none) There was previusly a lambda function on sort, i changed that for a normal function but the error is the same. If i am not wrong that means that sorting is not supported. Cheers. PD: i recently noted that i am sending thoose mails directly to you, sorry.

1 0

I have a problem interpreting pypy functionality.
by Diego Jacobi 20 Jul '08

20 Jul '08

Hi. I am trying to decifer what exactly is the result that i got from using pypy. As little as i understand, it can translate code from python to C. I have followed the "Getting started" guide, but it makes it too long to see the pypy features that i am confused. First at all, i have had some issues from trying to compile pypy into pypy-c, it claims for some error with cc, but cc is on my system. And i am using debian lenny, which is still under development. Im not sure if that can cause the problem. So, i couldnt see the code generated or the optimiced program running. I have found pypy, by looking for a translator from python to C. But what i was looking for is a translator which gives me a code without a python dependency. Maybe using stdlibs, or glib, or something, and libffi, so the python wrappers are converted too. In some of the tests in the guide i see that after compiling with t.compile_c() i can directly use the function in python. So, it is a python-module compiler? Also the guide doesnt mention where is saved the resultant module. I just want to translate a simple algoritm which i am using to test the differences of speed between multiple languages: http://code.google.com/p/languagebenchmarks/ By now i only have the implementation with only Python (without external optimizations) and standard C, showing that python is only between 20 and 40 times slower than C and a promedium of 35, which is not much. I want to add the code generated by pypy to compare how good is optimiced in relation to C and Python, and in the future to other languages. The algorithm is very linear and doesnt make use of threads or anything too much complicated. It does extensive use of lists and strings. But when running the translator with the main.py file i have an error in an initial line: #python pypy-dist/pypy/translator/goal/translate.py main.py File "main.py", line 6, in <module> N_entradas = int(sys.argv[1]) ValueError: invalid literal for int() with base 10: 'main.py' The program is called with 2 integers determinning the number of bits of input and the number of bits of output in an electronic true table which the algoritm should optimice with the QuineMcClusky algorithm. The true table is passed through stdin. By eample: ./main.py 3 2 00010 00101 01011 0111- 100-- 101-1 11000 11100 That is a random table of example. An electronic should understand that, but the fact is that the algoritm have to convert that in a set of lists of string and work with them iterativelly, until it gets the best possible optimization. What i understand of the error line, is that it cant convert 'main.py' to integer. So seems like argv is like ['python pypy-dist/pypy/.....','main.py'] Does the translator haves to run the code to translate it? The 'main.py' should be the first argument received by the program not the second one. How can i pass any argument in the right order? Thanks in advance. Diego.

2 1

Re: [pypy-dev] Zope2-RPython implementation help wanted..
by holger krekel 18 Jul '08

18 Jul '08

Hey Chris, all, On Thu, Jul 17, 2008 at 08:43 +0100, Chris Withers wrote: > holger krekel wrote: >> names are confusing here. > > Indeed, I think Ranjith can be forgiven for getting confused given the > proliferation of similar names in this area ;-) > > For clarity, shall we agree to refer to things as: > > RestrictedPython - the current Zope 2 implementation found at > http://pypi.python.org/pypi/RestrictedPython > > Sandboxing - the technique of getting a secure and restricted python > environment running. > > RPython - some special in the pypy world ;-) "some special", aha :) i just added a FAQ entry: http://codespeak.net/pypy/dist/pypy/doc/faq.html#does-rpython-have-anything… >> Indeed, PyPy provides a robust way of sandboxing its Python >> Interpreter, see >> >> http://codespeak.net/pypy/dist/pypy/doc/sandbox.html > > Indeed, although as it is, I'm not sure how good a fit it is. > >> I am not sure how well this fits in your "porting zope2 to >> python2.5" project - but to me it looks like a good idea to go for >> using pypy-sandbox in the future instead of trying to get Zope2's >> "RestrictedPython" secure and ported to 2.5. > > Well, the rough feature request list is: > > - something that behaves like a python function > - can be pickled this is basically about storing user-provided scripts in the zodb, right? > - prevents people stripping security proxies from objects that are > passed in to the function hum, pypy-sandbox currently virtualizes at a low level (access to the Operating Systems files, io, etc.) and provides a *very* trustable way of running untrusted code. For providing higher and python level virtualization more work is needed so it might indeed not directly fit some use cases that RestrictedPython tries to satisfy. I am interested in this as are others, i think. Could you or someone else point to some concrete usages and use cases of the current RestrictedPython? > - prevents people importing things they shouldn't > - prevents people from getting hold of un-security-proxies-objects > - executes as fast as possible > - each function plays in its own sandbox and can be called many many > times, often simultaneously in different threads > - limits the amount of memory and processor they can hog (nice to have) the rest, i think, is quite doable, particularly limiting RAM/CPU resources. > I'm sure others will pitch in if I've forgotten anything... best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org

2 1

Zope2-RPython implementation help wanted..
by ranjith kannikara 17 Jul '08

17 Jul '08

I have taken the gsoc 08 project of porting zope2 to python2.5. Through my way to the successful completetion of the project I have to implement Restricted python in Zope2. Chris Withers, consultant of simplistix have volunteered to give guidance and help in the RPython implementation I was also suggested by him to engage with the PyPy guys on their mailing list. and work with their guidance to re-implement RestrictedPython in a way that doesn't use AST hacks. Being a student I am not much familiar with the RPython yet need some help to get the RPython implementaion started.. How can I get started with the RPython implementation..

2 1