Mailman 3 October 2007 - pypy-dev

rffi and rpython layer consistency
by Christian Tismer Oct. 16, 2007

Oct. 16, 2007

[a lengthy brain dump from last night which was meant for pypy-dev] [today I would be much more concise, but well, it is like it is] Dear Friends, I am talking about different layers of RPython and how coherently they respond. ATM, they don't. --------------------------------------------------- I am writing because I feel the need for it. We (you) have reached quite some state with PyPy and RPython. At the current state, I digged in and worked with the windows branch of Maciey's rmmap implementation. It went quite smooth, until I followed some XXX saying that the native windows types should be used. Not sure if they should be used, but I wanted to be consistent, and to try a show-case. And this caused me grief! Internal reasons for the trouble --------------------------------- Unfortunately, there are different flavors of the windows API. Most of them are consistent, having an agreement on what a HANDLE should be, for instance. But there are lower-level, also public functions with no general equivalent, which simply define a Handle as an int. This is not nice, but given by the gods of MS. Now I wanted to use exactly those inconsistent declarations with RFFI. Exactly! Why did I want this? --------------------- Well, ideally an extension writer should not bother about any types, just use them and pass them around as necessary. But there is this inconsistency. In order to make functions compatible, I tried to insert casts, a bad thing, I know. But how are casts worse that modifying an original definition? I wanted to provide a clean interface, as clean as can be, with no modifications. But rffi stopped me doing this. Putting it short: ------------------ I don't want to have cast restrictions in llinterp, which don't exist in ctypes and low level modes. After trying this for hours and already days, I came up with the following limitations of our type system: 1) things are running fine on top of cpython, with the ctypes emulation in place 2) things do not run well with llinterp in place. llinterp has grown into a large handler of different cases, no objection, but it does not do what it should do in this particular case. 3) things in a full translation are behaving like expected. We should move away from a full trabslation requirement as a proof of concept. What I can agree with would be a strong hierarchy, getting more restrictive as we become more specific. But the current situation just feels badly inconsistent and unsorted. rffi has a cast operation, which is implemented by force_cast. This should be unconditional. For every mode of interpretation or translatition. But it is not. Good but then bad: Using ctypes, everything works great. This should be the point of trust: it ctypes takes it, there should be no more testing needed. Worse, much worse: llinterp barfs at me, moaning that it is only able to cast an odd int. Hey why that!!! Crazy, in the end: I can produce a perfect extension to PyPy. Yes, I know, this was a contribution for testing tagged pointers. I do hate this argument. This is not PyPy. We do not implement something that is needed in some place. It is either explicitly enabled and used in a controlled manner, or it is disabled. So what I'm saying is this: ---------------------------- We should not support casting to any odd pointers at all, unless we are doing a tagged pointer implementation. By default, these operations should be hidden. If we do so, then we should emit a special operation for all pointers in that domain, and it should not be value dependent, but type dependent, only for that domain (whatever a domain should be, in this context). We also should not support casting to any pointer at all but for good reasons. If we have to cast, then for inconsistent, given annotations. That is just the case for rffi! I just want it there -- rffi.cast does a cast. Maybe restricted to raw, no idea. But it should be there, and only there! ------------------ Conclusion: This is what I'd like to do. - make pointer coercion functions as restrictive as possible - drop the llinterp odd pointer rule; instead - implement tagging exactly for those kinds of pointers where we want to support pointer tagging - make casting rules consistent through all layers of abstraction. If interpreted pypy allows/forbids casts, llinterp must adhere to it, and backends must follow the rules, too. - testing should have a defined hierarchy of things that may break when moving from one layer to the other. Ideally, they all would be the same. But they should be monotone: If a passes level 3, it also should pass level 1 and 2. - I would like to add tests for these meta-test levels, if seems appropriate. Yes, this was what I wanted to communicate since days. I'm open to work on these things, while I hope that these issues are not just my problem, after some thought ouf your's. You will know. G'nite -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

1 0

pypy with sympy
by Ondrej Certik Oct. 16, 2007

Oct. 16, 2007

Hi, since pypy is in Debian for some time, I sometimes test it with our project SymPy[1], written in pure Python 2.4 and newer. Generally, it works out of the box and it's very nice and quite fast. You did an awesome job! However, sometimes, it fails, because of the bug[2], that we reported two months ago. Do you think you could please apply the patch provided in there, or just fix it somehow? Also another question - what is your plan with your RPython to javascript converter - are you planning to create something like the Google Web Toolkit, or just leave it as is, if someone becomes interested? I think it would be a very cool feature to have something like this in Python, just a little more documented and officially released. Thanks a lot, Ondrej Certik [1] http://code.google.com/p/sympy/ [2] https://codespeak.net/issue/pypy-dev/issue316

3 10

PyPy JVM builds "out of the box"
by Niko Matsakis Oct. 15, 2007

Oct. 15, 2007

It is no longer necessary to apply any patches to get PyPy JVM to build. It should work "out of the box" now (does for me, anyhow). There are a still a few external functions unimplemented, though I have some partial implementations hanging around. Niko

3 2

some GC statistics
by Carl Friedrich Bolz Oct. 14, 2007

Oct. 14, 2007

Hi all! I did some statistics of the memory pypy-c uses when running pystone. Some results below: - objects allocated: 14228349 - size allocated: 314.25MB - average object size: 23.16 bytes - average object life time: 1.17MB allocated - 0.3615% of objects lifes longer die before 8MB are allocated (which is the default heap size (it's very similar for number of objects) in this setting, a collection happened every 50KB allocated, so this is about the error on life time numbers. top ten allocated types: GcStruct W_IntObject { super, inst_intval } 104.6MB GcStruct StdObjSpaceFrame { super } 77.9MB GcArray of * GcStruct object { typeptr } 62.9MB GcStruct list { length, items } 27.5MB GcArray of dictentry { key: * GcStruct W_Root { super }, f_everused: Bool, value: * GcStruct W_Root { super }, f_hash: Signed } 6.7MB GcStruct LoopBlock { super } 5.7MB GcStruct rpy_string { hash, chars } 3.7MB GcStruct ArgumentsPrepended { super, inst_args, inst_w_firstarg } 3.5MB GcArray of Char 3.2MB GcStruct W_StringObject { super, inst__value } 3.1 MB there are also some more or less useful graphs here: http://codespeak.net/~cfbolz/gcstats/pystone.pdf This is not easily repeatable with richards, just because there is too much data :-). Cheers, Carl Friedrich

1 0

Pypy-c error on windows
by Raj Bandyopadhyay Oct. 14, 2007

Oct. 14, 2007

Hi there I'm new to trying out pypy, and I hope this is the right list to ask questions. I just installed the pypy version for Windows on a Windows XP/Intel platform. However, I get errors when I start it up > pypy-c -------------------- Python 2.4.1.... Type "help".... Error calling sys.excepthook: debug: OperationError debug: operrortype: ImportError debug: operror-value: No module named traceback --------------------- Any idea what's going on? I'd appreciate getting this working. Thanks Raj

3 5

Re: [pypy-dev] Pypy-c error on windows
by Laura Creighton Oct. 14, 2007

Oct. 14, 2007

In a message of Sun, 14 Oct 2007 08:18:41 CDT, Raj Bandyopadhyay writes: >Hi there > >I'm new to trying out pypy, and I hope this is the right list to ask >questions. >I just installed the pypy version for Windows on a Windows XP/Intel >platform. > >However, I get errors when I start it up > > > pypy-c > >-------------------- >Python 2.4.1.... >Type "help".... > >Error calling sys.excepthook: >debug: OperationError >debug: operrortype: ImportError >debug: operror-value: No module named traceback >--------------------- > > >Any idea what's going on? I'd appreciate getting this working. > >Thanks >Raj This is the right place to ask questions. #pypy on freenode.net is another good place. I don't have a windows installation, but this looks to me as if you need to set your PYTHONPATH to point to wherever the modules are. But without a windows to install, I cannot check and see if I am completely wrong. Sorry about that, Laura

1 0

Re: [pypy-dev] pypy-jvm translating magic?
by Antonio Cuni Oct. 12, 2007

Oct. 12, 2007

Niko Matsakis wrote: > So, I see the bug, it's a bit odd. It seems to have to do with the > constant 9223372036854775808L, which gets serialized incorrectly into > the jasmin code. However, the Python code looks ok, so I am not sure > where the problem is. > > I tried to create a unit test to reproduce the error, but I am running > into a seemingly different problem that has to do with a (builtin?) > function PyLong_FromUnsignedLongLong. The oosupport direct_call opcode > fails to handle it properly, whatever it is. Hi Niko! Thanks to Carl Friedrich, I found that the problem with that test is that you must to annotate it explicitly with r_ulonglong, else the annotation degenerates to SomeObject and the rtyper assign it the PyObject* type. I checked in the test in oosupport.test_template.operations.test_compare_big_ullongs; now the test passes in gencli but fails in genjvm with the same error as the full translation. Moreover, I also addedd a new failing test to oosupport.test_template.runtest.test_big_ullong. I don't know if I will find the time to look at them today, though. ciao Anto

2 1

Re: [pypy-dev] [pypy-svn] r47380 - pypy/dist/pypy/translator/c/test
by Amaury Forgeot d'Arc Oct. 10, 2007

Oct. 10, 2007

Hello, > Modified: > pypy/dist/pypy/translator/c/test/test_typed.py > Log: > this test breaks the c backend in a curious way I think I found the reason: in gen_assignments(), variable names are compared by reference, but source and destination names are are constructed in different ways. I propose the following patch: Index: pypy/translator/c/support.py =================================================================== --- pypy/translator/c/support.py (revision 47380) +++ pypy/translator/c/support.py (working copy) @@ -205,7 +205,7 @@ startingpoint = dst tmpdecl = cdecl(typename, 'tmp') code = ['{ %s = %s;' % (tmpdecl, dst)] - while src is not startingpoint: + while src != startingpoint: code.append('%s = %s;' % (dst, src)) dst = src src, typename = dest2src.pop(dst) -- Amaury Forgeot d'Arc

1 0

Customized parser
by Franck Pommereau Oct. 9, 2007

Oct. 9, 2007

Dear PyPy developpers, I'd like to build a CPython program that would include a parser for a Python-like language. My idea was to start from PyPy's parser that is already written in pure Python. All I need is a parser that builds an AST or something equivalent. This is the idea. But from now on, I'm lost in PyPy's sources. I did not even figure out how to call the parser correctly... Does any one has an example that could guide me? The changes to the languages I need are removing some statements and adding few ones like: parallel: suite and: suite Thank you in advance for any help. Franck

2 1

Anybody been following the Open Mono project?
by Laura Creighton Oct. 8, 2007

Oct. 8, 2007

It is here: http://wiki.openmoko.org/wiki/Main_Page On their wishlist: http://wiki.openmoko.org/wiki/Wish_List They say that they are looking for a scripting language. Python is liked, but they fear it is too big. see: http://wiki.openmoko.org/wiki/Wish_List#Built-in_Scripting_Language It seems to me that they would like PyPy if we built a small version for them. It also seems as if a really old friend of mine (Ian Darwin) is involved enough in the project to be presenting it to linux Ontario this October. I wonder if some of the ideas we had 20 years ago about lightweight processes and tinier-than-v7 os's have made it into open moko? I wonder if it would be worth it to get a few of the experimental phones? Laura

2 1