Mailman 3 May 2008 - pypy-dev

video recordings S3
by Carl Friedrich Bolz 26 May '08

26 May '08

Hi all, seems that the video recordings of S3 are available: http://www.swa.hpi.uni-potsdam.de/s3/program/ The talk slides of the talk I gave there can be found here: http://codespeak.net/svn/pypy/extradoc/talk/s3-2008/talk.pdf The paper that we submitted can be found here: http://codespeak.net/svn/pypy/extradoc/paper/spy-s3.pdf Cheers, Carl Friedrich

3 3

Development philosophy
by Jacob Hallén 25 May '08

25 May '08

Hi everyone, I have been following several discussions that have been going on in the past couple of weeks concering the speed of certain parts of the code and the size of the the code base. While I think the sentiments about the different things are absolutely right, I think that the limited resources that we have require us to make choices in what we do and when we do it. My views are that: - Apart from a few libraries, like socket() and I/O, it doesn't matter that the library code is extremely slow, for the time being. We want to find some application where we can make a big difference early, rather than try to be everything to everyone from the start. Things like zlib and cryptography can be very slow, as long as they are correct. - For core language functionality, speed matters a lot. Apart from the JIT work, which still is a long term project, I can see a couple of useful avenues for improving speed. One is optimising calls, which Samuele has more thoughts about. The other one is to dig into the list operation microbenchmarks, where our performance is a lot worse than CPython 2.4. List operations are a very large part of most real Python applications (like all the zillion web frameworks) and improving them may increase our chances of adoption before the JIT work is usable. - I agree that the size of the project in lines of code is getting problematic, but I think it would be a mistake to start doing anything about it at this point in time. We are still not doing well enough for completeness and correctness to start simplifying the code. we need to tackle one aspect at a time. Otherwise we won't be able to see if we are making progress or just making things worse. Our planned work to make Django run is very much about completeness and correctness. Let's keep our focus there for a while. Jacob

1 0

Re: [pypy-dev] r55005 - in pypy/branch/win32port/pypy: lib lib/_ctypes module/_rawffi module/_rawffi/test rlib rlib/test
by Thomas Heller 21 May '08

21 May '08

afa(a)codespeak.net schrieb: > Author: afa > Date: Tue May 20 16:17:22 2008 > New Revision: 55005 > > Modified: > pypy/branch/win32port/pypy/lib/_ctypes/builtin.py > pypy/branch/win32port/pypy/lib/msvcrt.py > pypy/branch/win32port/pypy/module/_rawffi/__init__.py > pypy/branch/win32port/pypy/module/_rawffi/interp_rawffi.py > pypy/branch/win32port/pypy/module/_rawffi/test/test__rawffi.py > pypy/branch/win32port/pypy/rlib/libffi.py > pypy/branch/win32port/pypy/rlib/rwin32.py > pypy/branch/win32port/pypy/rlib/test/test_libffi.py > Log: > Maciej was right: it's better to dynamically get the name of the msvcrt library: > the same generated sources may be recompiled, e.g. in debug mode. > > I hope I got it right: in pypy-c.exe, we compile a function that > retrieves the dll containing the "fopen" pointer address. > > Hardcoded "libc.so.6" on posix platforms. > There must be a better way. May be that dl_iterate_phdr(3) can help? Thomas

2 1

Europython 2008 - talks?
by Beatrice During 21 May '08

21 May '08

Hi there Thursday 22nd of May is the submission deadline for EuroPython 2008 (http://www.europython.org/FrontPage, 7-12th of July). Question 1: Are any pypy people going to submit talks? Question 2: Is there a sprint interest? (3 days after the conference) Just checking.... Cheers Bea

3 6

Re: [pypy-dev] r55005 - in pypy/branch/win32port/pypy: lib lib/_ctypes module/_rawffi module/_rawffi/test rlib rlib/test
by Thomas Heller 21 May '08

21 May '08

afa(a)codespeak.net schrieb: > > Author: afa > > Date: Tue May 20 16:17:22 2008 > > New Revision: 55005 > > > > Modified: > > pypy/branch/win32port/pypy/lib/_ctypes/builtin.py > > pypy/branch/win32port/pypy/lib/msvcrt.py > > pypy/branch/win32port/pypy/module/_rawffi/__init__.py > > pypy/branch/win32port/pypy/module/_rawffi/interp_rawffi.py > > pypy/branch/win32port/pypy/module/_rawffi/test/test__rawffi.py > > pypy/branch/win32port/pypy/rlib/libffi.py > > pypy/branch/win32port/pypy/rlib/rwin32.py > > pypy/branch/win32port/pypy/rlib/test/test_libffi.py > > Log: > > Maciej was right: it's better to dynamically get the name of the msvcrt library: > > the same generated sources may be recompiled, e.g. in debug mode. > > > > I hope I got it right: in pypy-c.exe, we compile a function that > > retrieves the dll containing the "fopen" pointer address. > > + #include <stdio.h> > > + > > + /* Get the module where the "fopen" function resides in */ > > + HANDLE get_libc_handle() { > > + MEMORY_BASIC_INFORMATION mi; > > + memset(&mi, 0, sizeof(mi)); > > + > > + if( !VirtualQueryEx(GetCurrentProcess(), &fopen, &mi, sizeof(mi)) ) > > + return 0; > > + > > + return (HMODULE)mi.AllocationBase; > > + } > > + '''], Ha, that's the best approach I have seen so far to get the name if the msvcrt runtime library. Hope you don't mind if I steal it for 'official' ctypes (I will check if it works in win64 also before) ? -- Thanks, Thomas

2 5

Performance Plots moved
by Armin Rigo 21 May '08

21 May '08

Hi all, I moved the entry point url of the performance plots of tuatara on tuatara itself, which makes some sense (they were on wyvern, and wyvern is down currently (but should come back on-line one of these days with a fresh new hard drive)). The url is: http://tuatara.cs.uni-duesseldorf.de/plots.html A bientot, Armin.

1 0

Running pysqlite on pypy
by Antonio Cuni 19 May '08

19 May '08

Hi Gerhard, hi all, in the last days, we have been trying to run django on pypy, using the ctypes based implementation of pysqlite. In doing this, we encountered a problem; we have exchanged a bit of private mails, so I try to sum up here: This snippet explodes when run with pysqlite-ctypes, either on cpython or pypy-c: >>>> from pysqlite2.dbapi2 import connect >>>> db = connect(':memory:') >>>> db.execute('BEGIN IMMEDIATE TRANSACTION') <pysqlite2.dbapi2.Cursor object at 0xb7dc4f6c> >>>> db.execute('COMMIT') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/exarkun/Scratch/Source/pysqlite3/pysqlite2/dbapi2.py", line 315, in execute return cur.execute(*args) File "/home/exarkun/Scratch/Source/pysqlite3/pysqlite2/dbapi2.py", line 483, in execute raise self.connection._get_exception() pysqlite2.dbapi2.OperationalError: SQL logic error or missing database The very same thing happens on python 2.4 + pysqlite2 (non-ctypes version): >>> from pysqlite2.dbapi2 import connect >>> db = connect(':memory:') >>> db.execute('BEGIN IMMEDIATE TRANSACTION') <pysqlite2.dbapi2.Cursor object at 0xb7cb0860> >>> db.execute('COMMIT') Traceback (most recent call last): File "<stdin>", line 1, in ? pysqlite2.dbapi2.OperationalError: cannot commit - no transaction is active However, it works perfectly on cpython 2.5 + sqlite3: >>> from sqlite3.dbapi2 import connect >>> db = connect(':memory:') >>> db.execute('BEGIN IMMEDIATE') <sqlite3.Cursor object at 0xf7cff050> >>> db.execute('COMMIT') <sqlite3.Cursor object at 0xf7cff020> Samuele pointed out that maybe it's just a difference between pysqlite2 and pysqlite3; after more digging, he changed pysqlite-ctypes to print every SQL statement before sending it to sqlite; what he got is this: Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from pysqlite2 import dbapi2 >>> db = dbapi2.connect(':memory:') >>> db.execute('BEGIN IMMEDIATE TRANSACTION') BEGIN IMMEDIATE TRANSACTION <pysqlite2.dbapi2.Cursor object at 0x55e90> >>> db.execute('COMMIT') COMMIT COMMIT Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pysqlite2/dbapi2.py", line 318, in execute return cur.execute(*args) File "pysqlite2/dbapi2.py", line 489, in execute raise self.connection._get_exception() pysqlite2.dbapi2.OperationalError: SQL logic error or missing database The double COMMIT is probably causing the problem; not sure if it's a bug in pysqlite-ctypes or an expected behavior. Gerhard, what do you think about all of this? What's the best way to solve? ciao, Anto

3 8

Re: [pypy-dev] [pypy-svn] r54903 - in pypy/branch/win32port/pypy: config rlib rlib/test
by Maciej Fijalkowski 18 May '08

18 May '08

> +def test_translate(): > + from pypy.translator.interactive import Translation > + > + def func(): > + poll({}) > + > + t = Translation(func) > + fc = t.compile_c([]) wuaa. Please use from pypy.translator.c.test.test_genc import compile and use this one. Cheers, fijal

1 0

3rd party modules / fixing
by holger krekel 16 May '08

16 May '08

Hi all, after some discussion with maciej and elsewhere i'd suggest to setup a pypy/dist/lib-python/3rdparty directory where we temporarily put our own versions of packages that we aim to support and from which we can build upstream patches. If we evolve into maintaining packages or if the packages get too big we could shift them to a different place so that they don't come along with a standard checkout). In any case, let's note down exact source information including urls, revision etc. in the commit log messages so that things remain easily trackable. I guess for convenience we could also add that directory above to PyPy's app level module lookup paths. best, holger -- Holger Krekel - freelance manager and programmer pylib py.test/greenlets/svn APIs: http://pylib.org PyPy Python/Compiler tool chain: http://codespeak.net/pypy merlinux collaborative contracting: http://merlinux.de

1 0

Bad news about pysqlite-ctypes
by Maciej Fijalkowski 15 May '08

15 May '08

I'm not sure we should use this. It's seems to be broken. Tests of divmod's axiom (ORM) on top of cpython + sqlite-ctypes show: FAILED (skips=5, expectedFailures=1, errors=400, successes=47) JP Calderone also provided some simple code snippets that explode, ie: >>> from pysqlite2.dbapi2 import connect >>> db = connect(':memory:') >>> db.execute('BEGIN IMMEDIATE TRANSACTION') <pysqlite2.dbapi2.Cursor object at 0xb7dc4f6c> >>> db.execute('COMMIT') Traceback (most recent call last): File "<stdin>", line 1, in ? File "/home/exarkun/Scratch/Source/pysqlite3/pysqlite2/dbapi2.py", line 315, in execute return cur.execute(*args) File "/home/exarkun/Scratch/Source/pysqlite3/pysqlite2/dbapi2.py", line 483, in execute raise self.connection._get_exception() pysqlite2.dbapi2.OperationalError: SQL logic error or missing database >>> >>> from pysqlite2.dbapi2 import connect >>> db = connect(':memory:') >>> db.execute('create table foo ( x int )') <pysqlite2.dbapi2.Cursor object at 0xb7d63f6c> >>> list(_) [infinite rec] Not sure where to go from now.... Cheers, fijal

1 0