Mailman 3 May 2018 - pypy-dev

[PATCH] FreeBSD build fix
by David CARLIER 18 Jul '18

18 Jul '18

Here a new version of the patch. Kind regards.

2 1

About gurobipy
by Luis José Novoa 31 May '18

31 May '18

Hi All, My name is Luis Novoa. Im working on a large scale optimization problem using python/gurobipy for my implementation. I was wondering if any of you in this list has used gurobipy with pypy, or if it is even possible at this time. Thanks in advance, Luis.

9 9

Re: [pypy-dev] Mysterious IndexError in service running with PyPy
by Armin Rigo 22 May '18

22 May '18

Hi, On 22 May 2018 at 04:46, hubo <hubo(a)jiedaibao.com> wrote: > I think I found the cause, happily it is not a PyPy bug (although it is > PyPy-related). Some generators are calling some cleanup code in the > "finally" clause, but they are not correctly collected in PyPy until GC > (because PyPy does not use reference counting). Apparently when they are > collected by GC module, the "finally" clause is executed in a separated > thread. Unlike Java, we're not purposefully running finalizers in its own threads: instead, if your program has got more than one thread running, finalizers will run in the one where the GC happens to run. There isn't much we can do about that. All these hard GC issues also exist on CPython, but they are just more hidden. Typically, if there is a cycle of objects which itself holds, indirectly, a reference to the generator-iterator, then you get the same effect: the cycle will be broken by the GC module of CPython at a random point in time and from a random thread, too. So we can argue that you potentially have a bug even on CPython, in the sense that an apparently unrelated change somewhere else can make the same bug appear on CPython. A bientôt, Armin.

2 1

Re: [pypy-dev] Mysterious IndexError in service running with PyPy
by William ML Leslie 22 May '18

22 May '18

Where is the code that changes the size of self.heap? How do we know that size(self.heap) is constant? My guess is that some thread changes this; but l is not recomputed. On 18 Dec 2017 6:59 PM, "hubo" <hubo(a)jiedaibao.com> wrote: > I'm reporting this issue in this mail group, though I don't know if it is > related with PyPy, because it is really strange, and is not able to > reproduce stably. But I hope someone may know the reason or have some > points. > > I'm running some SDN services written in Python with PyPy 5.3.0. The > related code is here: > > https://github.com/hubo1016/vlcp/blob/8022e3a3c67cf4305af503d507640a730ca39… > > The full code is also in the repo, but may be too complex to describe. But > this related piece is quite simple: > > def _siftdown(self, pos): > temp = self.heap[pos] > l = len(self.heap) > while pos * 2 + 1 < l: > cindex = pos * 2 + 1 > pt = self.heap[cindex] > if cindex + 1 < l and self.heap[cindex+1][0] < pt[0]: > cindex = cindex + 1 > pt = self.heap[cindex] > if pt[0] < temp[0]: > self.heap[pos] = pt > self.index[pt[1]] = pos > else: > break > pos = cindex > self.heap[pos] = temp > self.index[temp[1]] = pos > It is a simple heap operation. The service uses a heap to process timers. > When the service is not busy, it usually runs this piece of code several > times per minute. > > I have 32 servers running this service. They are quite stable in about > three months, but one day one of the services crashes on line 100 reporting > IndexError: > pt = self.heap[cindex] > > As you can see, cindex = pos * 2 + 1, which is tested by the while > pre-conditon just two lines before. And there is not any multi-threading > issues here because this piece of code always runs in the same thread. So > it is not possible in theory for this to happen. > > Only the following facts are known about this issue: > > 1. It reproduces - through quite rarely. I've met this kind of > crashes 4 times, each with different machine, so it should not be related > to hardware issuess. Since I've got 32 servers, it might take more than one > year to reproduce with a single server. > 2. It seems not to be related to pressures. All of the crashes happens > at night, when there are little requests. Only some cleanup tasks are > running in fixed interval. > > The services are running with PyPy 5.3.0. I've upgraded a few of them to > 5.9, but it will take a long time to validate whether this still happens. > And It is not validated on CPython too. I'm also trying to collect more > debugging information for this issue, but it is very hard since it rarely > reproduces. > > It is not a serious issue. It could be workarounded with a auto-restart, > but I'm searching the cause. > > > 2017-12-18 > ------------------------------ > hubo > > _______________________________________________ > pypy-dev mailing list > pypy-dev(a)python.org > https://mail.python.org/mailman/listinfo/pypy-dev > >

2 3

[PATCH] FreeBSD build fix
by David CARLIER 19 May '18

19 May '18

Hi dear list, I have hard time to fork the project thus I m sendng this little patch to fix FreeBSD build. Hope it is helpful. Kind regards.

3 3

EuroScipy
by Carl Friedrich Bolz-Tereick 16 May '18

16 May '18

Hi all, Somebody on Twitter asked whether there would be a pypy talk at EuroScipy which is from August 28 to September 1 in Trento, Italy. Here is the Call for Presentations: https://pretalx.com/euroscipy18/cfp NB: the deadline is tomorrow! Carl Friedrich

4 3

More memory allocated for GC than arenas+rawmalloc+nursery (bug?)
by Oxylibrium Person 16 May '18

16 May '18

So, I’ve been testing moving some of my projects to using PyPy as an interpreter from CPython, but PyPy 6.0.0/ Python 3.5.3 seems to memory bloat and trigger OOM protections in about 6 hours on a server with 2GB of RAM, while CPython 3.6.2 ran for 24 days with a more or less constant memory footprint of ~670MB (+/- 40MB) on my discord bot, Birb. Output of gc.get_stats() about 2-3 hours before the crash are as follows: Total memory consumed: GC used: 641.5MB (peak: 775.5MB) in arenas: 536.5MB rawmalloced: 102.9MB nursery: 2.0MB raw assembler used: 5.5MB —————————————— Total: 647.0MB Total memory allocated: GC allocated: 1235.9MB (peak: 1303.8MB) in arenas: 602.8MB rawmalloced: 170.8MB nursery: 2.0MB raw assembler allocated: 6.0MB —————————————— Total: 1241.9MB This behaviour can be successfully replicated on brew’s macOS build of PyPy3 6.0.0 and the official PyPy3 6.0.0 release for 64-bit Ubuntu. To replicate this behaviour, I can use the following code, after installing the rewrite branch on discord.py from git+https://github.com/Rapptz/discord.py@rewrite#egg=discord.py; ``` import gc from discord.ext import commands bot = commands.AutoShardedBot(“>>”) # Variable passed is prefix, changing will not affect result @bot.command() async def get_gc_state(ctx): await ctx.send(gc.get_stats()) bot.run(TOKEN) # Replace TOKEN with authentication token, number of guilds/users visible will affect result ``` Unfortunately, I can only reproduce this issue on my production instance, with ~8.6k guilds, ~143.7k channels, and ~305k visible user accounts. On my test instance with 1 guild, 7 channels and 12 users, there are no measurable memory issues. Of note is that under total memory allocated, the sum of the parts of the GC allocation (602.8+170.8+2.0=775.6) is just a little over 62.75% of the total memory reported as being allocated for the GC. Is this a PyPy bug? Thanks for reading, Oxylibrium.

1 0

py 3.6 bytecode changes, state of py3.6
by Carl Friedrich Bolz 12 May '18

12 May '18

Hi all, Today I found out why on the py3.6 branch dis does not work properly. It's because cpython 3.6 switched to a slightly different bytecode format, where every instruction is 16 bit long, see these issues: https://bugs.python.org/issue26647 https://bugs.python.org/issue28050 Do we want to implement that too, so that we stay closer to what CPython is doing? Or should I partially undo the 3.6 changes to dis.py to fix it? Also, does anybody have a high-level overview which big things are missing on the py3.6 branch? Or what the current blockers are? Cheers, Carl Friedrich

1 0

module _cppyy disabled in py3.5 release?
by wlavrijsen＠lbl.gov 08 May '18

08 May '18

Hi, in the release-pypy3.5-v6.0.0 branch (and hence in the 3.5 release), module _cppyy was disabled: "_csv", "_pypyjson", "_posixsubprocess", # "_cppyy", "micronumpy" The commit history doesn't say? Anything I should fix? Thanks! Best regards, Wim -- WLavrijsen(a)lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net

3 8

The Biggest EU Observation Challenge by DigitalGlobe & ESA
by Lauri Aarnio 03 May '18

03 May '18

A call for all Python devs: the Biggest EU Observation Challenge by DigitalGlobe & ESA! *DigitalGlobe has teamed up with the European Space Agency (ESA) in order to organize The Earth Observation Challenge <http://bit.ly/DGCommsMtup>. This online challenge turns massive data into groundbreaking solutions that will serve to support disaster relief in developing areas.Participants will get to Identify & map water resources in developing countries and Identify urban change to help mitigate social and environmental risk using DigitalGlobe Library & ESA Sentinel 2 Data. The prizes include 10k€ in cash and a possibility to develop your project with ESA.You can import your own Python data to complement the existing sets and it is possible to use Jupyter Notebook.More about the challenge on the website: http://bit.ly/eochallengedg <http://bit.ly/eochallengedg> * -- *Lauri Aarnio* Communications Manager *@* BeMyApp <http://www.bemyapp.com/> 86 rue de Charonne Paris 75011 <https://maps.google.com/?q=86+rue+de+Charonne+Paris+75011&entry=gmail&sourc…> +33 66 83 93 12 88 | <https://www.facebook.com/BeMyAppUK/> <https://www.linkedin.com/in/aarniolauri/> <https://twitter.com/BeMyAppUK> ᐧ

1 0