Mailman 3 March 2018 - pypy-dev

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

Re: [pypy-dev] Mysterious IndexError in service running with PyPy
by William ML Leslie 22 May '18

22 May '18

Where is the code that changes the size of self.heap? How do we know that size(self.heap) is constant? My guess is that some thread changes this; but l is not recomputed. On 18 Dec 2017 6:59 PM, "hubo" <hubo(a)jiedaibao.com> wrote: > I'm reporting this issue in this mail group, though I don't know if it is > related with PyPy, because it is really strange, and is not able to > reproduce stably. But I hope someone may know the reason or have some > points. > > I'm running some SDN services written in Python with PyPy 5.3.0. The > related code is here: > > https://github.com/hubo1016/vlcp/blob/8022e3a3c67cf4305af503d507640a730ca39… > > The full code is also in the repo, but may be too complex to describe. But > this related piece is quite simple: > > def _siftdown(self, pos): > temp = self.heap[pos] > l = len(self.heap) > while pos * 2 + 1 < l: > cindex = pos * 2 + 1 > pt = self.heap[cindex] > if cindex + 1 < l and self.heap[cindex+1][0] < pt[0]: > cindex = cindex + 1 > pt = self.heap[cindex] > if pt[0] < temp[0]: > self.heap[pos] = pt > self.index[pt[1]] = pos > else: > break > pos = cindex > self.heap[pos] = temp > self.index[temp[1]] = pos > It is a simple heap operation. The service uses a heap to process timers. > When the service is not busy, it usually runs this piece of code several > times per minute. > > I have 32 servers running this service. They are quite stable in about > three months, but one day one of the services crashes on line 100 reporting > IndexError: > pt = self.heap[cindex] > > As you can see, cindex = pos * 2 + 1, which is tested by the while > pre-conditon just two lines before. And there is not any multi-threading > issues here because this piece of code always runs in the same thread. So > it is not possible in theory for this to happen. > > Only the following facts are known about this issue: > > 1. It reproduces - through quite rarely. I've met this kind of > crashes 4 times, each with different machine, so it should not be related > to hardware issuess. Since I've got 32 servers, it might take more than one > year to reproduce with a single server. > 2. It seems not to be related to pressures. All of the crashes happens > at night, when there are little requests. Only some cleanup tasks are > running in fixed interval. > > The services are running with PyPy 5.3.0. I've upgraded a few of them to > 5.9, but it will take a long time to validate whether this still happens. > And It is not validated on CPython too. I'm also trying to collect more > debugging information for this issue, but it is very hard since it rarely > reproduces. > > It is not a serious issue. It could be workarounded with a auto-restart, > but I'm searching the cause. > > > 2017-12-18 > ------------------------------ > hubo > > _______________________________________________ > pypy-dev mailing list > pypy-dev(a)python.org > https://mail.python.org/mailman/listinfo/pypy-dev > >

2 3

PhD Positions in programming language runtimes (Python)
by David Bremner 03 May '18

03 May '18

1 Position I: Python on OMR =========================== The candidate will explore the benefits and challenges of using OMR [1] components (e.g. JIT, GC) in Python [2,3] runtimes. This may involve extending OMR in novel ways to better support Python, or vice-versa. If questions like how to balance fast startup time for scripting applications against good performance for compute-intensive workloads sound interesting, this might an opportunity for you. 2 Position II: Python and PGAS ============================== The candidate will implement support for parallel programming with Python, using the PGAS [4] (parallel global address space) model. The PGAS model can significantly simplify the implementation of parallel algorithms on distributed and shared memory computers. Challenges include enabling parallelism while dealing with Python's Global Interpreter Lock (GIL), and implementing distributed garbage collection efficiently. 3 Qualifications ================ Normally applicants to the Computer Science PhD program at UNB require a Master's degree in computer science or a closely related area. This project will require substantial hands-on and experimental work, and being comfortable with Python and C++. Familiarity with Linux and Open Source Software development would be a plus. For position I, exposure to programming language concepts like GC, JIT, and bytecode would be helpful. For position II, experience with parallel programming frameworks and/or languages such as MPI, OpenMP, X10 is an asset. 4 Environment ============= This project is supported by the IBM Centre for Advanced Studies (Atlantic) at the University of New Brunswick [5]. We have a lively group of students, postdocs, and research assistants working on related problems. Students work closely with both UNB researchers and technical experts from IBM, gaining valuable perspective and experience. 5 Support ========= These positions are "fully funded"; the exact funding package is subject to finalization but will be consistent with other fully funded students in Computer Science at UNB. You can find some information about expenses on the UNB web site [6]. 6 Contact ========= Send your resumé and unofficial transcripts by email to Stephen MacKay <smackay3(a)unb.ca> Please specify which position(s) you are interested in. [1]: [http://www.eclipse.org/omr/] [2]: [http://www.python.org] [3]: [http://www.pypy.org] [4]: [http://www.pgas.org/] [5]: [http://www.unb.ca/research/casatlantic/] [6]: [http://www.unb.ca/fredericton/studentservices/international/new_internation…]

1 1

Problem with cgi module and pypy handling POST request
by Etienne Robillard 27 Mar '18

27 Mar '18

Hi, I got the following error playing with pypy 5.9.0 under my linux32 development server: Traceback (most recent call last): File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/controllers/base.py", line 195, in get_response response = callback(request, *args, **kwargs) File "/home/erob/src/django-hotsauce-stable/extras/libauthkit/authkit/authorize/decorators.py", line 67, in _wrapper return view_func(request, **kwargs) File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/dbapi/orm/decorators.py", line 35, in _wrapper return view_func(req, **kwargs) File "/home/erob/src/blogengine2-devel/lib/blogengine2/views/category.py", line 57, in add new_data = request.POST.copy() File "/usr/local/pypy/pypy2-v5.9.0-linux32/site-packages/werkzeug/local.py", line 347, in __getattr__ return getattr(self._get_current_object(), name) File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", line 96, in POST return self.get_POST() File "/home/erob/src/django-hotsauce-0.9.1/lib/notmm/utils/wsgilib/request.py", line 69, in get_POST keep_blank_values=True) File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 507, in __init__ self.read_multi(environ, keep_blank_values, strict_parsing) File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 631, in read_multi environ, keep_blank_values, strict_parsing) File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 509, in __init__ self.read_single() File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 646, in read_single self.read_lines() File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 668, in read_lines self.read_lines_to_outerboundary() File "/usr/local/pypy/pypy2-v5.9.0-linux32/lib-python/2.7/cgi.py", line 696, in read_lines_to_outerboundary line = self.fp.readline(1<<16) TypeError: readline() takes exactly 1 argument (2 given) Any ideas how to fix this issue? Thank you in advance, Etienne https://www.isotopesoftware.ca

2 5

Pip fails to work
by Grant Shepherd 25 Mar '18

25 Mar '18

PyPy seems like a great python version Except the extremely painful effort trying to get pip to work Can't connect to HTTPS URL because the SSL module is not available. - skipping

3 3

Preheating pypy
by Nagy, Attila 22 Mar '18

22 Mar '18

Hi, I use pypy to run an application in gunicorn. Gunicorn (as well) has a "preload" capability in multiprocess workers, which means it loads the application in one interpreter/process and instead of starting new interpreters/processes for additional workers, it just forks from the first one. This means the OS can share memory pages between the processes, which makes the app use less memory and start faster. This works nicely with pypy too and the memory savings are significant (taking into account that pypy uses much more memory than cpython, this is even more true). The problem is that each pypy process before the fork is "cold" and the JIT starts to compile code in the forked processes, independently from the original process, which makes a good deal of additonal memory (and CPU) go wasted. It would be nice to make this happen in the original process, so the operating system's COW semantics could work for the JIT-ed code too. Is there a way to achieve this? Thanks,

2 3

Ordered **kw officially suported?
by Arkadiusz Bulski 12 Mar '18

12 Mar '18

Hi, For the record, I am the developer of Construct library. https://construct.readthedocs.io/en/latest/# I noticed that test cases using ordered **kw arguments, which is a feature of 3.6, also pass on pypy (both 2.7 and 3.5). I need to ask, is this feature officially supported by pypy, or just passes by chance? -- ~ Arkadiusz Bulski ~ tel 503 357 111 lub WhatsApp/Telegram

2 1

Not sure what lltype to use in this case
by Timothy Baldridge 10 Mar '18

10 Mar '18

I have a rather strange use-case for a data type in a rpython interpreter. What I need is a raw malloc, but with a custom GC hook. I understand how to setup the GC hook, but what I can't figure out is what lltype to use with malloc. This type will inherit directly from object so I don't need any polymorphism. However I do need the struct to be of a variable size. So something like the C struct hack: struct MyObject { Type* header; char data[0]; } Thanks, Timothy

2 4

Sprint Leysin: rooms full, but we can try to get one more
by Armin Rigo 06 Mar '18

06 Mar '18

Hi all, Just a quick note: the Leysin sprint rooms are fully booked so far. That's a room with 4 people and a room with 5 people, in addition to a separate booking for a single room and a room for two. That will be quite a busy sprint! There is an option to get another room for 3 people. We can book this extra room either now or only if there is yet another person joining. The advantage is more confort, as the 5 people room is pretty basic and will be full. If there are two or three persons that would prefer the nicer 3 people room, I can book it now. (The prices should be around 50-55 CHF in the big room and maybe 65-75 CHF in the smaller room; I can check the details if needed.) A bientôt, Armin.

3 3

Status of "remove the gil"
by Timothy Baldridge 04 Mar '18

04 Mar '18

There was some talk awhile back of using fine-grained locks in PyPy to remove the gil. From the bitbucket repo it looks like the experiments never made it into master. Is there any work that was done here that could be used by other RPython interpreters, especially for interpreters that default to immutable or share-nothing semantics? Thanks, Timothy Baldridge

2 1