Mailman 3 January 2012 - pypy-dev

Undeliverable: [VIRUS FOUND]Important
by postmaster＠meratusline.com Jan. 10, 2012

Jan. 10, 2012

Delivery has failed to these recipients or groups: salesjkt(a)meratusline.com<mailto:salesjkt@meratusline.com> The e-mail address you entered couldn't be found. Please check the recipient's e-mail address and try to resend the message. If the problem continues, please contact your helpdesk. Diagnostic information for administrators: Generating server: meratusline.com salesjkt(a)meratusline.com #550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ## Original message headers: Received: from MRTS-VMXC5.meratusline.com (192.168.16.11) by MRTS-VMXC2.meratus.com (172.16.0.116) with Microsoft SMTP Server (TLS) id 14.1.218.12; Tue, 10 Jan 2012 15:47:50 +0700 Received: from mail2.meratusline.com (192.168.16.6) by MRTS-VMXC5.meratusline.com (192.168.16.11) with Microsoft SMTP Server (TLS) id 14.1.218.12; Tue, 10 Jan 2012 15:49:48 +0700 Received: from meratusline.com (155.subnet125-164-77.speedy.telkom.net.id [125.164.77.155] (may be forged)) by mail2.meratusline.com with ESMTP id q0A8logo025062-q0A8logp025062 for <salesjkt(a)meratusline.com>; Tue, 10 Jan 2012 15:47:50 +0700 Message-ID: <201201100847.q0A8logo025062-q0A8logp025062(a)mail2.meratusline.com> From: <pypy-dev(a)codespeak.net> To: <salesjkt(a)meratusline.com> Subject: [VIRUS FOUND]Important Date: Tue, 10 Jan 2012 15:53:39 -0800 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0009_00006D04.00003E07" X-Priority: 1 X-MSMail-Priority: High Return-Path: pypy-dev(a)codespeak.net Received-SPF: None (MRTS-VMXC5.meratusline.com: pypy-dev(a)codespeak.net does not designate permitted sender hosts) X-Brightmail-Tracker: AAAAAgAAAUAAAAFS

1 0

STM
by Armin Rigo Jan. 9, 2012

Jan. 9, 2012

Hi all, (Andrew: I'll answer your mail too, but this is independent) Sorry if this mail is more general than just pypy-dev, but I'll post it here at least at first. When thinking more about STM (and also after reading a paper that cfbolz pointed me to), I would like to argue that the currently accepted way to add STM to languages is slightly bogus. So far, the approach is to take an existing language, like Python or C, and add a keyword like "transaction" that delimits a nested scope. You end up with syntax like this (example in Python): def foo(): before() with transaction: during1() during2() after() In this example, "during1(); during2()" is the content of the transaction. But the issue with this approach is that there is no way to structure the transactions differently. What if I want a transaction that starts somewhere, and ends at some unrelated place? This is of course related to the way "we" (I, but others before me) described how transactions could be applied to the CPython interpreter. When you think about the C source of CPython, there is a precise point at which the previous transaction should stop and the next one immediately start (namely, between bytecodes). But there is no way to express this using the syntax above. What would be needed is something like (example in C): void mainloop() { while (1) { dispatch_next_bytecode(); call_stuff_between_bytecodes(); } } void call_stuff_between_bytecodes() { ...; /* here, we would like the previous transaction to stop and the next one to start */ ...; } The above issue occurs when trying to apply STM to CPython, but I believe the issue to be much more general. The solution proposed with the "transaction" keyword assumes a generally-non-STM program into which we carefully insert STM here and there. This is reasonable from the point of view of performance, because running a STM transaction is costly. But if we ignore this point of performance (for example, by assuming we have a hybrid HTM-STM-based approach that was carefully tweaked for years, with the GC and the JIT that are aware of it), then this argument no longer applies. The solution that we really need for CPython requires a different point of view: a generally-STM program, in which we carefully add here and there a "yield", i.e. a point at which it's ok to end the current transaction and start the next one. While this is obvious in the case of the CPython interpreter, I'd argue that it is a useful point of view in general. The paper mentioned above says that the current approach to multithreading is completely non-deterministic, with a large collection of tools to help us "patch" it here and there to make it more deterministic --- i.e. multithreaded programs behave randomly, and you need to carefully use locks, semaphores, and so on, in order to get back something that behaves as you want, hopefully in all cases. That approach is, according to this paper, the reason for which programming with multiple threads is so hard (and I tend to agree). The paper argues in favour of a would-be system that would be the other way around: deterministic with explicit non-determinism added. So, taking this point of view and making it more concrete, I'd like to argue that what we really need is not a nested "transaction" keyword, in C or in Python or in any language. That would be just another way to add some amount of determinism to a inherently non-deterministic approach. Instead, we need by default all threads to run always in a coarse-grained transaction; and using a "yield" keyword or function call, we would as needed make the transactions more fine-grained. (I ignore here the issue of I/O.) In other words, this would be a step back to the old world of cooperative multithreading: each thread needs to call "yield" regularly, otherwise it never allows other threads to run. The difference with the old world of cooperative multithreading is that nowadays we have STM --- "never allows other threads to run" has become a figure of speach, a way to reason about programs *as if* they were really switching between threads only at the points where we call "yield". What *really* occurs is that each thread runs a transaction between two yield points, so each thread can actually run concurrently with the others, and the only issue is that if the threads actually do conflict, then the performance degrades as some threads need to repeat their action. This gives a model where we can write a multithreaded application by starting with almost no "yield" points, and then, *if performance requires it*, we can carefully add extra yield points to reduce conflicts. In other words, it is a model in which we start writing multithreaded applications using just plain old (rather deterministic) cooperative multithreading, and then, as needs dictate, we insert some extra points of non-determinism in order to boost performance. Following that reasoning, I no longer want to go for a PyPy version exposing a "with transaction" context manager. Instead, it would rather be something like a function to call at any point to "yield" control --- or, in the idea that it's better to avoid having yields at random unexpected points, maybe an alternative solution that looks more promising: to start a new thread, the programmer calls a function "start_transactional_thread(g)" where g is a generator; when g yields, it marks the end of a transaction and the start of the next one. (The exact yielded value should be None.) The difference with the previous solution is that you cannot yield from a random function; your function has to have some structure, namely be a generator, in order to use this yield. Also, it turns out that this model is easy to write a non-STM implementation for, for debugging and testing: _running = [] def start_transactional_thread(g): _running.append(g) def run(): "Call this in the main thread to run and wait for all started sub-threads" while _running: i = random.choice(len(_running)) g = _running.pop(i) try: g.next() _running.append(g) except StopIteration: pass (Interestingly, this run() function is similar to stackless.run(). The exact relationships between stackless and this model are yet to investigate, but it seems that one point of view on what I'm proposing might be essentially "use stackless, remove some determinism from the already-unclear tasklet switching order, and add STM under the hood".) A bientôt, Armin.

6 14

Re: [pypy-dev] when will pypy support psycopg2?
by Yan Chunlu Jan. 8, 2012

Jan. 8, 2012

sorry I somehow replied to your personally, sorry for disturbing. just didn't get used to gmail's new UI.... about quora's adoption, brought me a lot confidence. thanks a lot! will test it! I am curious that does it hard for pypy to add those data type? like PyByteArray_Type, PyMemoryView_Type and PyInterpreterState On Sat, Jan 7, 2012 at 6:44 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: > On Sat, Jan 7, 2012 at 12:39 PM, Yan Chunlu <springrider(a)gmail.com> wrote: >> okay, thanks! I have installed psycopg2ct and will give a try. >> >> but since my app is serving several millions requests per day, I am >> really worrying if it suitable for production usage. > > You mean because it's relatively new? you should test it and see how > it works, FYI quora moved to the ctypes based new DB binding and they > never complained, seems to be working just fine for them. > >> >> On Sat, Jan 7, 2012 at 6:22 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >>> On Sat, Jan 7, 2012 at 12:20 PM, Yan Chunlu <springrider(a)gmail.com> wrote: >>>> thanks for the quick reply. seems psycopg2 is "An (experimental) >>>> implementation" based on http://pypi.python.org/pypi/psycopg2ct >>> >>> the other way around >>> >>>> >>>> and also, both lib need to make many changes to the code... >>>> >>> >>> they should not, they all support DB API >>> >>>> >>>> >>>> On Sat, Jan 7, 2012 at 6:10 PM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >>>>> On Sat, Jan 7, 2012 at 12:07 PM, Yan Chunlu <springrider(a)gmail.com> wrote: >>>>>> based on the irc chat here: >>>>>> >>>>>> http://www.tismer.com/pypy/irc-logs/pypy/pypy.2011-11-02.log.html >>>>>> >>>>>> >>>>>> PyByteArray_Type, PyMemoryView_Type and PyInterpreterState are missing >>>>>> from the headers. http://codepad.org/FYkhcZKf >>>>>> >>>>>> >>>>>> just wonder is there any schedule about this? I think psycopg2 is >>>>>> crucial for many developers to adopt pypy. thanks! >>>>>> _______________________________________________ >>>>>> pypy-dev mailing list >>>>>> pypy-dev(a)python.org >>>>>> http://mail.python.org/mailman/listinfo/pypy-dev >>>>> >>>>> Hi >>>>> >>>>> Use psycopg2-ct or pg8000 instead, those work fine in PyPy. >>>>> >>>>> Cheers, >>>>> fijal

3 4

Reference counting
by Marko Tasic Jan. 7, 2012

Jan. 7, 2012

Hi, I've been carefully following your mailing list for years, and using PyPy for different kind of projects, mostly for highly distributed and decentralized systems, and everything is just great compared to CPython, Jython and IronPython. I've even written partial ctypes wrapper for GObject Introspection, so I could use Gtk as GUI toolkit on top of PyPy. Today, I cannot imagine writing Python code without executing it on PyPy because it just feels natural to use it as default implementation instead of an alternative CPython :) I know that you don't feel that using reference counting as GC mechanism is good idea because of overhead of maintaining reference counts to each object and non concurrent nature of it, but can you give me any idea where to start and how I can implement reference counting for PyPy. Have in mind that I am not new to Python, and low level stuff, just want to measure performance and possibly implement alternative reference counting strategies. Cheers, Marko Tasic

2 4

when will pypy support psycopg2?
by Yan Chunlu Jan. 7, 2012

Jan. 7, 2012

based on the irc chat here: http://www.tismer.com/pypy/irc-logs/pypy/pypy.2011-11-02.log.html PyByteArray_Type, PyMemoryView_Type and PyInterpreterState are missing from the headers. http://codepad.org/FYkhcZKf just wonder is there any schedule about this? I think psycopg2 is crucial for many developers to adopt pypy. thanks!

3 2

C interpreter written in Python, and running CPython on top of PyPy
by Celil Jan. 6, 2012

Jan. 6, 2012

I have been thinking about the possibility of creating a C interpreter in Python. Is anybody already working on that? With PyPy this would presumably be quite easy to do. The interpreter will load the C code, create an AST (presumably using pyparsing and the EBNF spec of the C-language), and then populate the Flow Object Space with all the C objects, and create a control flow graph of the application logic. This graph will containe low level lltype objects, and can then be directly connected to the RPython flow-graph generated after the RTyper step. This would allow for seamless interoperability between C and PyPy, and would also greatly simplify the task of porting existing CPython extensions such as numpy. Rather than going through the error prone task of translating the whole code base into RPython, one will be able to simply load the exiting C source code and integrate it directly into the RPython flow graph. It will be possible to import *.h and *.c files directly without any compilation, and they will run nearly as fast thanks to PyPy's JIT technology. This would also allow us to do things like running CPython on top of PyPy. Right now it is possible to run PyPy on top of CPython, but the reverse is not. If CPython could be run on top of PyPy by interpreting its C source code that would be truly amazing. Interpreting C code would greatly help CPython developers by freeing them from the task of having to repeatedly compile their code. Celil

8 10

Problems Installing STM
by Andrew Francis Jan. 6, 2012

Jan. 6, 2012

Hi Folks: I want to experiment with the STM module. I read the "Comprehensive Strategy for Contention Management in STM" paper and compiled the rstm module. I figured learning to write C/C++ programmes with the module would be a good way to get a feel for STM. On the PyPy side: I have downloaded pypy from the repository, overlaid it with a pre-compiled (I don't have a big enough machine to compile from source) and use virtualenv, altering PYTHONPATH to point to the top of the pypy directory. When I execute python test_rstm.py the programme starts to compile but ends with no errors. I decided to test some other programmes to see if my environment is functioning correctly. I decided to run bpnn ../pypy/translator/goal/translate.py --stm bpnn.py (I added the --stm because the translator complained) translation:ERROR] File "/home/andrew/pypy-stm/pypy/translator/c/funcgen.py", line 697, in OP_CAST_PTR_TO_ADR [translation:ERROR] "in %r" % (self.graph,)) [translation:ERROR] AssertionError: cast_ptr_to_adr(gcref) is a bad idea with STM. Consider checking config.stm in <FunctionGraph of (pypy.rpython.rlist:490)ll_alloc_and_set__GcStruct_listLlT_Signed_rpy_stringPtr at 0xd62a0b0> [translation] start debugger... > /home/andrew/pypy-stm/pypy/translator/c/funcgen.py(697)OP_CAST_PTR_TO_ADR() -> "in %r" % (self.graph,)) (Pdb+) What am I doing wrong? How should I be going about this? Cheers, Andrew

2 5

micronumpy 'fromnumeric' patch
by mike c Jan. 4, 2012

Jan. 4, 2012

Hi All, Numpy is somewhat derived from another array library called 'Numeric'. What were originally functions in 'Numeric' have been promoted to ndarray methods in numpy. The implementation of this code in numpy is in * numpy/core/fromnumeric.py*. The list of functions that were promoted is listed at the end of this email. However, these functions must be retained for compatibility, but all they should do is call the ndarray method. e.g. "numpy.ravel(a)" is just a wrapper which returns "a.ravel()" I have: * copied all docstrings from numpy1.6 numpy/core/fromnumeric.py into micronumpy/app_numpy.py * for methods which we currently have (e.g. max, argmax), I have * made the function just call the relavant method and return the value * written a test_fromnumeric_* using the docstring examples * for methods which we don't have I have raised a NotImplemented Error I realise that this is a lot of boring boilerplate code & docstrings, but I think it's to everyone's advantage to flesh out the micronumpy implementation a bit more so that we can see what has/hasn't been done. This will also be useful to newcomers - like me - who can now find obvious functions that need implementation. Could someone please review, fix if necessary and commit. Thanks. Mike ====================== # __all__ = ['take', 'reshape', 'choose', 'repeat', 'put', # 'swapaxes', 'transpose', 'sort', 'argsort', 'argmax', 'argmin', # 'searchsorted', 'alen', # 'resize', 'diagonal', 'trace', 'ravel', 'nonzero', 'shape', # 'compress', 'clip', 'sum', 'product', 'prod', 'sometrue', 'alltrue', # 'any', 'all', 'cumsum', 'cumproduct', 'cumprod', 'ptp', 'ndim', # 'rank', 'size', 'around', 'round_', 'mean', 'std', 'var', 'squeeze', # 'amax', 'amin', # ]

1 1

How to turn a crawling caterpillar of a VM into a graceful butterfly
by Laurence Tratt Jan. 1, 2012

Jan. 1, 2012

Hi all, As many of you know, over the past few months I've been creating an RPython VM for the Converge language <http://convergepl.org>. This is now mostly complete at a basic level - enough to run pretty much all of my Converge programs at least on Linux and OpenBSD. [Major remaining issues are: no floating point numbers; 32 bit support is not finished; may not compile on OS X.] First, the good news. The RPython VM (~3 months effort) is currently close to 3 times faster than the old C-based VM (~18 months effort). I think the Converge VM is the first medium-scale VM to be created by someone outside the core PyPy group, so those numbers are a testament to the power of the RPython approach. I'd like to thank (alphabetically) Carl Friedrich Bolz, Maciej Fijalkowski, and Armin Rigo who've offered help, encouragement and (in Armin's case) big changes to RPython to help the Converge VM. I wouldn't have got this far without their help, or of others on the PyPy IRC channel. Now, the bad news: the RPython VM should be a lot faster than it currently is as the old VM is (to say the least) not very good. In large part this is because I simply don't know how best to optimise an RPython VM, particularly the JIT (in fact, at the moment the JITted VM seems to be sometimes slower than the non-JIT VM: I don't have an explanation for why). I'm therefore soliciting advice / code from those more knowledgeable than myself on how to optimise an RPython VM. Note that I'm intentionally being more general than the Converge VM: I hope that some of the ideas generated will prove useful to the many VMs that I hope will come to be implemented in RPython. Some of my ignorance is simply that many parts of RPython have little or no documentation: by playing the part of a clueless outsider (it comes naturally to me!), I hope I may help pinpoint areas where further documentation is most needed. If you want to test out the VM it's here: https://github.com/ltratt/converge Building should mostly be: $ export PYPY_SRC=<location of PyPy source directory> $ cd converge $ ./configure $ make That will build a JITted VM. If you want a lower level of optimisation, specify it to configure e.g. "./configure --opt=3". Please note that, for the time being, on Linux the JIT won't work with the default GC root finder, so you'll need to manually specify --gcrootfinder=shadowstack in vm/Makefile. Apart from that, building on 64 bit Unix should hopefully be reasonably simple. A simple performance benchmark is along the lines of "make clean ; cd vm ; make ; cd .. ; time make regress" which builds the compiler, various examples, and the handful of tests that Converge comes with. [Please note, Converge wasn't built with a TDD philosophy; while I welcome contributions of tests, I am unable to make any major efforts in that regard myself in the short-term. I know this sits uneasily with many in the PyPy community, but I hope you are able to overlook this difference in development philosophy.] Here are some examples of questions I'd love to know answers to: * Why is the JITted VM that's built sometimes 2x slower than --opt=3, but other times a few percent faster *on the same benchmark*?! * What are virtualrefs? * What is a more precise semantics of elidable? * What is 'specialize'? * Is it worth manually inlining functions in the main VM loop? * Can I avoid the (many) calls to rffi.charp[size]2str, given that they're mostly taken from the never-free'd mod_bc? Would this work well on some GCs but not others? No doubt there are many other things I would do well to know, but plainly do not - please educate me! If you have any questions or comments about Converge or the VM, please don't hesitate to ask me - and thank you in advance for your help. Yours, Laurie -- Personal http://tratt.net/laurie/ The Converge programming language http://convergepl.org/ https://github.com/ltratt http://twitter.com/laurencetratt

4 11