Mailman 3 January 2012 - pypy-dev

Undeliverable: [VIRUS FOUND]Important
by postmaster＠meratusline.com Jan. 10, 2012

Jan. 10, 2012

Delivery has failed to these recipients or groups: salesjkt(a)meratusline.com<mailto:salesjkt@meratusline.com> The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk. Diagnostic information for administrators: Generating server: meratusline.com salesjkt(a)meratusline.com #550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ## Original message headers: … [View More]

1 0

STM
by Armin Rigo Jan. 9, 2012

Jan. 9, 2012

Hi all, (Andrew: I'll answer your mail too, but this is independent) Sorry if this mail is more general than just pypy-dev, but I'll post it here at least at first. When thinking more about STM (and also after reading a paper that cfbolz pointed me to), I would like to argue that the currently accepted way to add STM to languages is slightly bogus. So far, the approach is to take an existing language, like Python or C, and add a keyword like "transaction" that delimits a nested scope. You … [View More]end up with syntax like this (example in Python): def foo(): before() with transaction: during1() during2() after() In this example, "during1(); during2()" is the content of the transaction. But the issue with this approach is that there is no way to structure the transactions differently. What if I want a transaction that starts somewhere, and ends at some unrelated place? This is of course related to the way "we" (I, but others before me) described how transactions could be applied to the CPython interpreter. When you think about the C source of CPython, there is a precise point at which the previous transaction should stop and the next one immediately start (namely, between bytecodes). But there is no way to express this using the syntax above. What would be needed is something like (example in C): void mainloop() { while (1) { dispatch_next_bytecode(); call_stuff_between_bytecodes(); } } void call_stuff_between_bytecodes() { ...; /* here, we would like the previous transaction to stop and the next one to start */ ...; } The above issue occurs when trying to apply STM to CPython, but I believe the issue to be much more general. The solution proposed with the "transaction" keyword assumes a generally-non-STM program into which we carefully insert STM here and there. This is reasonable from the point of view of performance, because running a STM transaction is costly. But if we ignore this point of performance (for example, by assuming we have a hybrid HTM-STM-based approach that was carefully tweaked for years, with the GC and the JIT that are aware of it), then this argument no longer applies. The solution that we really need for CPython requires a different point of view: a generally-STM program, in which we carefully add here and there a "yield", i.e. a point at which it's ok to end the current transaction and start the next one. While this is obvious in the case of the CPython interpreter, I'd argue that it is a useful point of view in general. The paper mentioned above says that the current approach to multithreading is completely non-deterministic, with a large collection of tools to help us "patch" it here and there to make it more deterministic --- i.e. multithreaded programs behave randomly, and you need to carefully use locks, semaphores, and so on, in order to get back something that behaves as you want, hopefully in all cases. That approach is, according to this paper, the reason for which programming with multiple threads is so hard (and I tend to agree). The paper argues in favour of a would-be system that would be the other way around: deterministic with explicit non-determinism added. So, taking this point of view and making it more concrete, I'd like to argue that what we really need is not a nested "transaction" keyword, in C or in Python or in any language. That would be just another way to add some amount of determinism to a inherently non-deterministic approach. Instead, we need by default all threads to run always in a coarse-grained transaction; and using a "yield" keyword or function call, we would as needed make the transactions more fine-grained. (I ignore here the issue of I/O.) In other words, this would be a step back to the old world of cooperative multithreading: each thread needs to call "yield" regularly, otherwise it never allows other threads to run. The difference with the old world of cooperative multithreading is that nowadays we have STM --- "never allows other threads to run" has become a figure of speach, a way to reason about programs *as if* they were really switching between threads only at the points where we call "yield". What *really* occurs is that each thread runs a transaction between two yield points, so each thread can actually run concurrently with the others, and the only issue is that if the threads actually do conflict, then the performance degrades as some threads need to repeat their action. This gives a model where we can write a multithreaded application by starting with almost no "yield" points, and then, *if performance requires it*, we can carefully add extra yield points to reduce conflicts. In other words, it is a model in which we start writing multithreaded applications using just plain old (rather deterministic) cooperative multithreading, and then, as needs dictate, we insert some extra points of non-determinism in order to boost performance. Following that reasoning, I no longer want to go for a PyPy version exposing a "with transaction" context manager. Instead, it would rather be something like a function to call at any point to "yield" control --- or, in the idea that it's better to avoid having yields at random unexpected points, maybe an alternative solution that looks more promising: to start a new thread, the programmer calls a function "start_transactional_thread(g)" where g is a generator; when g yields, it marks the end of a transaction and the start of the next one. (The exact yielded value should be None.) The difference with the previous solution is that you cannot yield from a random function; your function has to have some structure, namely be a generator, in order to use this yield. Also, it turns out that this model is easy to write a non-STM implementation for, for debugging and testing: _running = [] def start_transactional_thread(g): _running.append(g) def run(): "Call this in the main thread to run and wait for all started sub-threads" while _running: i = random.choice(len(_running)) g = _running.pop(i) try: g.next() _running.append(g) except StopIteration: pass (Interestingly, this run() function is similar to stackless.run(). The exact relationships between stackless and this model are yet to investigate, but it seems that one point of view on what I'm proposing might be essentially "use stackless, remove some determinism from the already-unclear tasklet switching order, and add STM under the hood".) A bientôt, Armin. [View Less]

6 14

Re: [pypy-dev] when will pypy support psycopg2?
by Yan Chunlu Jan. 8, 2012

Jan. 8, 2012

sorry I somehow replied to your personally, sorry for disturbing. just didn't get used to gmail's new UI.... about quora's adoption, brought me a lot confidence. thanks a lot! will test it! I am curious that does it hard for pypy to add those data type? like PyByteArray_Type, PyMemoryView_Type and PyInterpreterState On Sat, Jan 7, 2012 at 6:44 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Sat, Jan 7, 2012 at 12:39 PM, Yan Chunlu <springrider(a)gmail.com> wrote: >&… [View More]

3 4

Reference counting
by Marko Tasic Jan. 7, 2012

Jan. 7, 2012

Hi, I've been carefully following your mailing list for years, and using PyPy for different kind of projects, mostly for highly distributed and decentralized systems, and everything is just great compared to CPython, Jython and IronPython. I've even written partial ctypes wrapper for GObject Introspection, so I could use Gtk as GUI toolkit on top of PyPy. Today, I cannot imagine writing Python code without executing it on PyPy because it just feels natural to use it as default implementation … [View More]

2 4

when will pypy support psycopg2?
by Yan Chunlu Jan. 7, 2012

Jan. 7, 2012

based on the irc chat here: http://www.tismer.com/pypy/irc-logs/pypy/pypy.2011-11-02.log.html PyByteArray_Type, PyMemoryView_Type and PyInterpreterState are missing from the headers. http://codepad.org/FYkhcZKf just wonder is there any schedule about this? I think psycopg2 is crucial for many developers to adopt pypy. thanks!

3 2

C interpreter written in Python, and running CPython on top of PyPy
by Celil Jan. 6, 2012

Jan. 6, 2012

I have been thinking about the possibility of creating a C interpreter in Python. Is anybody already working on that? With PyPy this would presumably be quite easy to do. The interpreter will load the C code, create an AST (presumably using pyparsing and the EBNF spec of the C-language), and then populate the Flow Object Space with all the C objects, and create a control flow graph of the application logic. This graph will containe low level lltype objects, and can then be directly … [View More]

8 10

Problems Installing STM
by Andrew Francis Jan. 6, 2012

Jan. 6, 2012

Hi Folks: I want to experiment with the STM module. I read the "Comprehensive Strategy for Contention Management in STM" paper and compiled the rstm module. I figured learning to write C/C++ programmes with the module would be a good way to get a feel for STM. On the PyPy side: I have downloaded pypy from the repository, overlaid it with a pre-compiled (I don't have a big enough machine to compile from source) and use virtualenv, altering PYTHONPATH to point to the top of the pypy … [View More]

2 5

micronumpy 'fromnumeric' patch
by mike c Jan. 4, 2012

Jan. 4, 2012

Hi All, Numpy is somewhat derived from another array library called 'Numeric'. What were originally functions in 'Numeric' have been promoted to ndarray methods in numpy. The implementation of this code in numpy is in * numpy/core/fromnumeric.py*. The list of functions that were promoted is listed at the end of this email. However, these functions must be retained for compatibility, but all they should do is call the ndarray method. e.g. "numpy.ravel(a)" is just a wrapper which returns "a.… [View More]

1 1

How to turn a crawling caterpillar of a VM into a graceful butterfly
by Laurence Tratt Jan. 1, 2012

Jan. 1, 2012

Hi all, As many of you know, over the past few months I've been creating an RPython VM for the Converge language <http://convergepl.org>. This is now mostly complete at a basic level - enough to run pretty much all of my Converge programs at least on Linux and OpenBSD. [Major remaining issues are: no floating point numbers; 32 bit support is not finished; may not compile on OS X.] First, the good news. The RPython VM (~3 months effort) is currently close to 3 times faster than the old C-… [View More]based VM (~18 months effort). I think the Converge VM is the first medium-scale VM to be created by someone outside the core PyPy group, so those numbers are a testament to the power of the RPython approach. I'd like to thank (alphabetically) Carl Friedrich Bolz, Maciej Fijalkowski, and Armin Rigo who've offered help, encouragement and (in Armin's case) big changes to RPython to help the Converge VM. I wouldn't have got this far without their help, or of others on the PyPy IRC channel. Now, the bad news: the RPython VM should be a lot faster than it currently is as the old VM is (to say the least) not very good. In large part this is because I simply don't know how best to optimise an RPython VM, particularly the JIT (in fact, at the moment the JITted VM seems to be sometimes slower than the non-JIT VM: I don't have an explanation for why). I'm therefore soliciting advice / code from those more knowledgeable than myself on how to optimise an RPython VM. Note that I'm intentionally being more general than the Converge VM: I hope that some of the ideas generated will prove useful to the many VMs that I hope will come to be implemented in RPython. Some of my ignorance is simply that many parts of RPython have little or no documentation: by playing the part of a clueless outsider (it comes naturally to me!), I hope I may help pinpoint areas where further documentation is most needed. If you want to test out the VM it's here: https://github.com/ltratt/converge Building should mostly be: $ export PYPY_SRC=<location of PyPy source directory> $ cd converge $ ./configure $ make That will build a JITted VM. If you want a lower level of optimisation, specify it to configure e.g. "./configure --opt=3". Please note that, for the time being, on Linux the JIT won't work with the default GC root finder, so you'll need to manually specify --gcrootfinder=shadowstack in vm/Makefile. Apart from that, building on 64 bit Unix should hopefully be reasonably simple. A simple performance benchmark is along the lines of "make clean ; cd vm ; make ; cd .. ; time make regress" which builds the compiler, various examples, and the handful of tests that Converge comes with. [Please note, Converge wasn't built with a TDD philosophy; while I welcome contributions of tests, I am unable to make any major efforts in that regard myself in the short-term. I know this sits uneasily with many in the PyPy community, but I hope you are able to overlook this difference in development philosophy.] Here are some examples of questions I'd love to know answers to: * Why is the JITted VM that's built sometimes 2x slower than --opt=3, but other times a few percent faster *on the same benchmark*?! * What are virtualrefs? * What is a more precise semantics of elidable? * What is 'specialize'? * Is it worth manually inlining functions in the main VM loop? * Can I avoid the (many) calls to rffi.charp[size]2str, given that they're mostly taken from the never-free'd mod_bc? Would this work well on some GCs but not others? No doubt there are many other things I would do well to know, but plainly do not - please educate me! If you have any questions or comments about Converge or the VM, please don't hesitate to ask me - and thank you in advance for your help. Yours, Laurie -- Personal http://tratt.net/laurie/ The Converge programming language http://convergepl.org/ https://github.com/ltratt http://twitter.com/laurencetratt [View Less]

4 11