Mailman 3 August 2010 - pypy-dev

Wrapping C, void pointer
by Hart's Antler 13 Aug '10

13 Aug '10

I am wrapping PortAudio for RPython. Following the source code in RSDL as a guide i can clearly see what to do with constants, structs, functions and so on. So far so good until i reached something new in PortAudio i am not sure how to deal with, a void typedef and a void pointer. In the RSDL example pointers were defined by Ptr = lltype.Ptr(lltype.ForwardReference()), then in the CConfig class the struct was defined and rffi_platform.configure parses, finally the Ptr is told TO become the type - given the output of platform.configure. How do we deal with this situation from PortAudio? # from portaudio.h typedef void PaStream; PaError Pa_OpenStream( PaStream** stream, const PaStreamParameters *inputParameters, const PaStreamParameters *outputParameters, double sampleRate, unsigned long framesPerBuffer, PaStreamFlags streamFlags, PaStreamCallback *streamCallback, void *userData ); ############################### I have tried the following but it fails when i try to malloc the void pointer. OpenDefaultStream = external( 'Pa_OpenDefaultStream', [ rffi.VOIDPP, # PaStream** rffi.INT, # numInputChannels rffi.INT, # numOutputChannels rffi.INT, # sampleFormat rffi.INT, # sampleRate rffi.INT, # framesPerBuffer rffi.INT, #streamcallback rffi.VOIDP, #userData ], rffi.INT ) Stream = lltype.Void #rffi.VOIDP def test(): print 'portaudio version %s' %GetVersion() assert Initialize() == 0 # paNoError = 0, error code is returned on init fail. stream = lltype.malloc(Stream, flavor='raw') try: ok = OpenDefaultStream( stream, 1, 1, Int16, 22050, FramesPerBufferUnspecified, 0 ) finally: lltype.free(stream, flavor='raw') Terminate() -brett

2 2

pre-emptive micro-threads utilizing shared memory message passing?
by Kevin Ar18 12 Aug '10

12 Aug '10

Might as well warn you: This is going to be a rather long post. I'm not sure if this is appropriate to post here or if would fit right in with the mailing list. Sorry, if it is the wrong place to post about this. I've looked through the documenation (http://codespeak.net/pypy/dist/pypy/doc/stackless.html) and didn't really see what I was looking for. I've also investigated several options in the default CPython. What I'm trying to accomplish: I am trying to write a particular threading scenario that follows these rules. It is partly an experiment and partly for actual production code. 1. Hundreds or thousands of micro-threads that are essentially small self-contained programs (not really, but you can think of them that way). 2. No shared state - data is passed around from one micro-thread to another; only one micro-thread has access to the data at a time. (although the programmer gets the impression there is no shared state, in reality, the underlying implementation uses shared memory / shared state for speed; the data does not move; you just pass around a reference/pointer to some shared memory) 3. The micro-threads can run in parallel on different cpu cores, get moved to a different core, etc.... 4. The micro-threads are truly pre-emptive (uses hardware interrupt pre-emption). 5. It is my intention to write my own scheduler that will suspend the micro-threads, start them, control the sharing of data, assign them to different CPU cores etc.... In fact, for my purposes, I MUST write my own scheduler as I have very specific requirements on when they should and should not run. Now, I have spent some time trying to find a way to achieve this ... and I can implement a rather poor version using default Python. However, I don't see any way to implement my ideal version. Maybe someone here might have some pointers for me. Shared Memory between parallel processes ---------------------------------------- Quick Question: Do queues from the multiprocessing module use shared memory? If the answer is YES, you can just skip this section, because that would solve this particular problem. (For simplicity, let's assume a quad core CPU) It is my intent to create 4 threads/processs (one per core) and use the scheduler to assign a micro-thread (of which there may be hundreds) to one of the 4 threads/processes. However, the micro-threads need to exchange data quickly; to do that I need shared memory -- and that is where I'm having some trouble. Normally, 4 threads would be the ideal solution -- as they can run in parallel and use shared memory. However, because of the Python GIL, I can't use threads in this way; thus, I have to use 4 processes, which are not setup to share memory. Question: How can I share Python Objects between processes USING SHARED MEMORY? I do not want to have to copy or "pass" data back and forth between processes or have to use a proxy "server" process. These are both too much of a performance hit for my needs; shared memory is what I need. The multiprocessing module offers me 4 options: "queues", "pipes", "shared memory map", and a "server process". "Shared memory map" won't work as it only handles C values and arrays (not Python objects or variables). "Server Process" sounds like a bad idea. Am I correct in that this option requires extra processing power and does not even use shared memory? If so, that would be a very bad choice for me. The big question then... do "queues" and "pipes" used shared memory or do they pass data back and forth between processes? (if they used shared memory, then that would be perfect) Does PyPy have any other options for me? True Pre-emptive scheduling? ---------------------------- Any way to get pre-emptive micro-threads? Stackless (the real Stackless, not the one in PyPy) has the ability to suspend them after a certain number of interpreter instructions; however, this is prone to problems because it can run much longer than expected. Ideally, I would like to have true pre-emptive scheduling using hardware interrupts based on timing or CPU cycles (like the OS does for real threads). I am currently not aware of any way to achieve this in CPython, PyPy, Unladen Swallow, Stackless, etc.... Are there detailed docs on why the Python GIL exists? ----------------------------------------------------- I don't mean trivial statements like "because of C extensions" or "because the interpreter can't handle it". It may be possible that my particular usage would not require the GIL. However, I won't know this until I can understand what threading problems the Python interpreter has that the GIL was meant to protect against. Is there detailed documentation about this anywhere that covers all the threading issues that the GIL was meant to solve? Thanks, Kevin _________________________________________________________________ Hotmail has tools for the New Busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:…

10 39

Re: [pypy-dev] [pypy-svn] r76608 - in pypy/branch/jit-bounds/pypy/jit/metainterp: . test
by Antonio Cuni 12 Aug '10

12 Aug '10

On 12/08/10 19:02, hakanardo(a)codespeak.net wrote: > + def boundint_gt(self, val): > + if val is None: return > + self.minint = val + 1 what happens if val == sys.maxint? ciao, Anto

1 0

What can Cython do for PyPy?
by Stefan Behnel 12 Aug '10

12 Aug '10

Hi, there has recently been a move towards a .NET/IronPython port of Cython, mostly driven by the need for a fast NumPy port. During the related discussion, the question came up how much it would take to let Cython also target other runtimes, including PyPy. Given that PyPy already has a CPython C-API compatibility layer, I doubt that it would be hard to enable that. With my limited knowledge about the internals of that layer, I guess the question thus becomes: is there anything Cython could do to the C code it generates that would make the Cython generated extension modules run faster/better/safer on PyPy than they would currently? I never tried to make a Cython module actually run on PyPy (simply because I don't use PyPy), but I have my doubts that they'd run perfectly out of the box. While generally portable, I'm pretty sure the C code relies on some specific internals of CPython that PyPy can't easily (or efficiently) provide. Stefan

5 5

Re: [pypy-dev] Percentage Python as RPython.
by Hart's Antler 11 Aug '10

11 Aug '10

Todd, I think you want a frontend, not a backend. The frontend would take in normal Python and convert it to RPython. RPython seems to have infinite meta-programming possibilities, so its just a matter of how hard would it be to make the meta frontend. Probably too hard since python is so dynamic, but maybe its possible with a new subset of Python halfway to RPython, developers would then only have to port to Not-So-Restricted-Python, and then the frontend does the final job of converting to RPython. -brett

2 1

I broke stackless
by Alex Gaynor 11 Aug '10

11 Aug '10

The work I did changing CALL_METHOD to support keyword arguments moved some rstack.resume_point calls around, and it seems to have inadvertantly broken stackless on trunk. The latest translation fail can be found here: http://buildbot.pypy.org/builders/pypy-c-stackless-app-level-linux-x86-32/b…. Anyone have a suggestion as to what exactly I need to do to get this working? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me

3 2

rstruct where is pack?
by Hart's Antler 10 Aug '10

10 Aug '10

Seems like struct.pack is not RPython? I see the examples for unpack in the tests folder, but not for packing.

3 3

demoting method, cannot follow, call result degenerated
by Hart's Antler 04 Aug '10

04 Aug '10

I'm still struggling to learn all the rules of RPython, i have read the coding guide, and the PDF's PyGirl and Ancona's RPython paper, but still i feel i'm not fully grasping everything. I have a function that returns different classes that all share a common base class. It works until i introduce a new subclass that has some methods of the same name. Then i get the demotion, can not follow, degenerated error. I googled, but all i can find is an IRC log where Fijal seems to taking talking about my problem. http://www.tismer.com/pypy/irc-logs/pypy/%23pypy.log.20070125 <fijal> pedronis: if function can return (in rpython) set of classes with common superclass, than all methods that I call later must be defined on that superclass, right? [11:30] <fijal> [15:01] <pedronis> yes, unless you assert a specific subclass So i just need to use an assert statement before the function return, and assert the class i am returning? I am blogging about my progress while learning RPython, i have posted about meta-programming in Rpython which is a new concept to me. http://pyppet.blogspot.com/2010/08/meta-programming-in-rpython.html -brett

2 2

Percentage Python as RPython.
by Anderson, Todd A 03 Aug '10

03 Aug '10

Sorry if this has been asked before. I did some searching of the archive and didn't see anything but I might have missed it. I am curious what percentage of real-world Python programs in use are also RPython programs. I know that the FAQ says that the translator is not intended for Python programs in general but only for the PyPy interpreter itself but I've also seen a few mentions (on other sites) of attempting to translate Python to C. I've been thinking about adding a backend to the translator but would only want to do so if a significant amount of Python programs could use it. thanks, Todd

3 2

py.test/debian and pypy issue
by holger krekel 01 Aug '10

01 Aug '10

Hi all, just for you information: if you are running Debian (e.g. Ubuntu 10.04) and install "py.test" (codespeak-python-lib) from there you get the 9-month old py.test-1.1 which cannot run PyPy's trunk-test suite. Solutions: * uninstall the debian version. install 'py' from PyPI with e.g. "pip install py" or "easy_install py" - this should get you the 1.3.3 version which should work fine. * uninstall the debian version, don't install any other and then alias "py.test" to "trunk/pypy/py/bin/py.test" which means you use the pypy-included py version, currently version 1.3.1 which is also the version used in nightly test runs etc. sidenote: Fedora 13 ships 1.3.2 and Gentoo ships 1.3.3 so you mostly only get the issues on debian-based systems, i guess. best, holger

1 0