Mailman 3 August 2019 - pypy-dev

PyPy Winter Sprint + cpyext
by Stefan Beyer 20 Dec '22

20 Dec '22

Hey everyone! Those of you who attended or followed one of the last two winter sprints in Leysin might know me, the others probably won't. I'm (still) a Master's student writing my thesis on PyPy's current issue with cross-heap cycles when using cpyext. The main point is, that they are not collected and stay around as floating garbage, even if they are not reachable any more. Correct me if I'm wrong, but I didn't notice anyone else working on that topic (and/or committing something) since I've picked it up two years ago (yeah, I know, that long ago...). I saw that you would be working on "cpyext performance and completeness" during the current winter sprint this week and thought that this might also concern my thesis. So I thought I'll give you an update. I recently pushed my current (non-optimized, breaking-some-tests, but more or less working) implementation of the "CPython-style" GC-extension for cpyext. It is still in an early stage and not all cases (legacy and non-legacy-finalizers, weakrefs) are handled. However, I picked up some pace during the last couple of weeks and I'm determined to finish this implementation during the following weeks (some quirks with the tests have been haunting me, but I think I figured most of them out by now). After that, I will implement a second alternative implementation (to compare to, mostly for the sake of my thesis), which will take some more time. I also added a couple of fancy test cases (the so called "dot-tests"), so that we can test complex object graphs a little bit easier (and also because it was kind of cool to have another language inside PyPy/RPython, even if it was only for the tests...), with more test cases to come (the current ones are a bit messy). None of the new changes concerning low-latency applications are currently integrated, but that should not be too hard. I guess it won't make much sense for me to join you at this year's winter sprint, especially as I won't be able to get there before Thursday, but I might be able to join you over the IRC channel (or some other form of communication if you like). If there is anything that is worth discussing please let me know! Also feel free to comment on my code, but beware that I might change some things once I try to do some optimizations (probably not many, but at least fix the worst issues) and make it a little bit more readable. You can find my code on the cpyext-gc-cycle branch.<https://bitbucket.org/pypy/pypy/commits/branch/cpyext-gc-cycle> Looking forward to hearing from you! Greetings, Stefan

2 2

Installying pycrypto in 7.1.1?
by Robert Whitcher 22 Aug '19

22 Aug '19

So 7.1.1 fixed my issue with string size *BUT* pip install pycrypto does not work... I get the following --- any thoughs? └─ $ pip install pycrypto Collecting pycrypto Cache entry deserialization failed, entry ignored Cache entry deserialization failed, entry ignored Downloading https://files.pythonhosted.org/packages/60/db/645aa9af249f059cc3a368b118de3… (446kB) 100% |████████████████████████████████| 450kB 1.9MB/s Installing collected packages: pycrypto Running setup.py install for pycrypto ... error Complete output from command /home/datos/.pyenv/versions/pypy2.7-7.1.1/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-rGgV3v/pycrypto/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-UfVkX8-record/install-record.txt --single-version-externally-managed --compile: running install running build running build_py creating build creating build/lib.linux-x86_64-2.7 creating build/lib.linux-x86_64-2.7/Crypto copying lib/Crypto/__init__.py -> build/lib.linux-x86_64-2.7/Crypto copying lib/Crypto/pct_warnings.py -> build/lib.linux-x86_64-2.7/Crypto creating build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/SHA224.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/SHA384.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/SHA.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/MD4.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/RIPEMD.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/MD2.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/HMAC.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/SHA512.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/hashalgo.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/SHA256.py -> build/lib.linux-x86_64-2.7/Crypto/Hash copying lib/Crypto/Hash/MD5.py -> build/lib.linux-x86_64-2.7/Crypto/Hash creating build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/PKCS1_OAEP.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/DES.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/ARC4.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/XOR.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/CAST.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/PKCS1_v1_5.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/blockalgo.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/AES.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/ARC2.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/DES3.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher copying lib/Crypto/Cipher/Blowfish.py -> build/lib.linux-x86_64-2.7/Crypto/Cipher creating build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/asn1.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/_number_new.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/winrandom.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/Counter.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/RFC1751.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/py21compat.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/randpool.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/py3compat.py -> build/lib.linux-x86_64-2.7/Crypto/Util copying lib/Crypto/Util/number.py -> build/lib.linux-x86_64-2.7/Crypto/Util creating build/lib.linux-x86_64-2.7/Crypto/Random copying lib/Crypto/Random/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Random copying lib/Crypto/Random/_UserFriendlyRNG.py -> build/lib.linux-x86_64-2.7/Crypto/Random copying lib/Crypto/Random/random.py -> build/lib.linux-x86_64-2.7/Crypto/Random creating build/lib.linux-x86_64-2.7/Crypto/Random/Fortuna copying lib/Crypto/Random/Fortuna/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Random/Fortuna copying lib/Crypto/Random/Fortuna/SHAd256.py -> build/lib.linux-x86_64-2.7/Crypto/Random/Fortuna copying lib/Crypto/Random/Fortuna/FortunaGenerator.py -> build/lib.linux-x86_64-2.7/Crypto/Random/Fortuna copying lib/Crypto/Random/Fortuna/FortunaAccumulator.py -> build/lib.linux-x86_64-2.7/Crypto/Random/Fortuna creating build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG copying lib/Crypto/Random/OSRNG/posix.py -> build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG copying lib/Crypto/Random/OSRNG/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG copying lib/Crypto/Random/OSRNG/fallback.py -> build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG copying lib/Crypto/Random/OSRNG/nt.py -> build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG copying lib/Crypto/Random/OSRNG/rng_base.py -> build/lib.linux-x86_64-2.7/Crypto/Random/OSRNG creating build/lib.linux-x86_64-2.7/Crypto/SelfTest copying lib/Crypto/SelfTest/st_common.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest copying lib/Crypto/SelfTest/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_pkcs1_15.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_ARC4.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_AES.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_ARC2.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_Blowfish.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_XOR.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_DES.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/common.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_pkcs1_oaep.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_CAST.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher copying lib/Crypto/SelfTest/Cipher/test_DES3.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Cipher creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_SHA256.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_MD2.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_HMAC.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_SHA384.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_RIPEMD.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_MD5.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_SHA.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/common.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_MD4.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_SHA224.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash copying lib/Crypto/SelfTest/Hash/test_SHA512.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Hash creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol copying lib/Crypto/SelfTest/Protocol/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol copying lib/Crypto/SelfTest/Protocol/test_rfc1751.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol copying lib/Crypto/SelfTest/Protocol/test_chaffing.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol copying lib/Crypto/SelfTest/Protocol/test_KDF.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol copying lib/Crypto/SelfTest/Protocol/test_AllOrNothing.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Protocol creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey copying lib/Crypto/SelfTest/PublicKey/test_ElGamal.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey copying lib/Crypto/SelfTest/PublicKey/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey copying lib/Crypto/SelfTest/PublicKey/test_DSA.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey copying lib/Crypto/SelfTest/PublicKey/test_RSA.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey copying lib/Crypto/SelfTest/PublicKey/test_importKey.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/PublicKey creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random copying lib/Crypto/SelfTest/Random/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random copying lib/Crypto/SelfTest/Random/test_random.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random copying lib/Crypto/SelfTest/Random/test_rpoolcompat.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random copying lib/Crypto/SelfTest/Random/test__UserFriendlyRNG.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/Fortuna copying lib/Crypto/SelfTest/Random/Fortuna/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/Fortuna copying lib/Crypto/SelfTest/Random/Fortuna/test_FortunaAccumulator.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/Fortuna copying lib/Crypto/SelfTest/Random/Fortuna/test_FortunaGenerator.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/Fortuna copying lib/Crypto/SelfTest/Random/Fortuna/test_SHAd256.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/Fortuna creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/test_generic.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/test_nt.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/test_fallback.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/test_winrandom.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG copying lib/Crypto/SelfTest/Random/OSRNG/test_posix.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Random/OSRNG creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util copying lib/Crypto/SelfTest/Util/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util copying lib/Crypto/SelfTest/Util/test_asn1.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util copying lib/Crypto/SelfTest/Util/test_number.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util copying lib/Crypto/SelfTest/Util/test_winrandom.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util copying lib/Crypto/SelfTest/Util/test_Counter.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Util creating build/lib.linux-x86_64-2.7/Crypto/SelfTest/Signature copying lib/Crypto/SelfTest/Signature/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Signature copying lib/Crypto/SelfTest/Signature/test_pkcs1_15.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Signature copying lib/Crypto/SelfTest/Signature/test_pkcs1_pss.py -> build/lib.linux-x86_64-2.7/Crypto/SelfTest/Signature creating build/lib.linux-x86_64-2.7/Crypto/Protocol copying lib/Crypto/Protocol/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Protocol copying lib/Crypto/Protocol/KDF.py -> build/lib.linux-x86_64-2.7/Crypto/Protocol copying lib/Crypto/Protocol/Chaffing.py -> build/lib.linux-x86_64-2.7/Crypto/Protocol copying lib/Crypto/Protocol/AllOrNothing.py -> build/lib.linux-x86_64-2.7/Crypto/Protocol creating build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/_RSA.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/ElGamal.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/_DSA.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/DSA.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/RSA.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/pubkey.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey copying lib/Crypto/PublicKey/_slowmath.py -> build/lib.linux-x86_64-2.7/Crypto/PublicKey creating build/lib.linux-x86_64-2.7/Crypto/Signature copying lib/Crypto/Signature/__init__.py -> build/lib.linux-x86_64-2.7/Crypto/Signature copying lib/Crypto/Signature/PKCS1_v1_5.py -> build/lib.linux-x86_64-2.7/Crypto/Signature copying lib/Crypto/Signature/PKCS1_PSS.py -> build/lib.linux-x86_64-2.7/Crypto/Signature running build_ext running build_configure checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for __gmpz_init in -lgmp... yes checking for __gmpz_init in -lmpir... no checking whether mpz_powm is declared... yes checking whether mpz_powm_sec is declared... yes checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking for inttypes.h... (cached) yes checking limits.h usability... yes checking limits.h presence... yes checking for limits.h... yes checking stddef.h usability... yes checking stddef.h presence... yes checking for stddef.h... yes checking for stdint.h... (cached) yes checking for stdlib.h... (cached) yes checking for string.h... (cached) yes checking wchar.h usability... yes checking wchar.h presence... yes checking for wchar.h... yes checking for inline... inline checking for int16_t... yes checking for int32_t... yes checking for int64_t... yes checking for int8_t... yes checking for size_t... yes checking for uint16_t... yes checking for uint32_t... yes checking for uint64_t... yes checking for uint8_t... yes checking for stdlib.h... (cached) yes checking for GNU libc compatible malloc... yes checking for memmove... yes checking for memset... yes configure: creating ./config.status config.status: creating src/config.h building 'Crypto.PublicKey._fastmath' extension creating build/temp.linux-x86_64-2.7 creating build/temp.linux-x86_64-2.7/src cc -pthread -fPIC -std=c99 -O3 -fomit-frame-pointer -Isrc/ -I/usr/include/ -I/home/datos/.pyenv/versions/pypy2.7-7.1.1/include -c src/_fastmath.c -o build/temp.linux-x86_64-2.7/src/_fastmath.o src/_fastmath.c: In function ‘longObjToMPZ’: src/_fastmath.c:82:7: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_size’ if (p->ob_size > 0) { ^ src/_fastmath.c:83:11: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_size’ size = p->ob_size; ^ src/_fastmath.c:86:12: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_size’ size = -p->ob_size; ^ src/_fastmath.c:93:22: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_digit’ mpz_set_ui (temp, p->ob_digit[i]); ^ src/_fastmath.c:97:30: error: ‘SHIFT’ undeclared (first use in this function) mpz_mul_2exp (temp2, temp, SHIFT * i); ^ src/_fastmath.c:97:30: note: each undeclared identifier is reported only once for each function it appears in src/_fastmath.c: In function ‘mpzToLongObj’: src/_fastmath.c:113:38: error: ‘SHIFT’ undeclared (first use in this function) int size = (mpz_sizeinbase (m, 2) + SHIFT - 1) / SHIFT; ^ src/_fastmath.c:118:20: warning: implicit declaration of function ‘_PyLong_New’ [-Wimplicit-function-declaration] PyLongObject *l = _PyLong_New (size); ^ src/_fastmath.c:118:20: warning: initialization makes pointer from integer without a cast [-Wint-conversion] src/_fastmath.c:130:4: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_digit’ l->ob_digit[i] = (digit) (mpz_get_ui (temp) & MASK); ^ src/_fastmath.c:130:21: error: ‘digit’ undeclared (first use in this function) l->ob_digit[i] = (digit) (mpz_get_ui (temp) & MASK); ^ src/_fastmath.c:130:49: error: ‘MASK’ undeclared (first use in this function) l->ob_digit[i] = (digit) (mpz_get_ui (temp) & MASK); ^ src/_fastmath.c:135:22: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_digit’ while ((i > 0) && (l->ob_digit[i - 1] == 0)) ^ src/_fastmath.c:140:3: error: ‘PyLongObject {aka struct <anonymous>}’ has no member named ‘ob_size’ l->ob_size = i * sgn; ^ error: command 'cc' failed with exit status 1 -- [image: photo] Robert Whitcher Member of Technical Staff at Rubrik M 512-633-1771 <512-633-1771> E robert.whitcher(a)rubrik.com <robert.whitcher(a)rubrik.com> W www.rubrik.com <http://www.rubrik.com?utm_source=WiseStamp&utm_medium=email&utm_term=&utm_c…>

1 1

Help Understanding Memory Consumption
by Robert Whitcher 22 Aug '19

22 Aug '19

Hi, I am running a very simple test case (as we are hitting OOM on our larger PyPy deployments) and I'd love some help understanding what is happening here.... We have a lot of processes that send messages to each other. These can be large JSON serializations of objects. But the memory being consumed seems out of order and hard to manage across processes. I have this loop running: import time import json def main(): with open("/tmp/test12334.1234", "r") as f: json_msg = f.read() while True: j = json.loads(json_msg) time.sleep(10) if __name__ == "__main__": main() I have tried 3 separate general runs across both pypy and cpython. The first does nothing but the sleep (it doesn't load or json the message) The second just loaded the json_str from a file The third is the full loop. If I run this in cpython I get (80MB, 92MB and 136MB) respectively This makes sense as the file is 11MB json serialization of a dictionary and json.loads takes up some memory However if I run this in pypy I get 120MB, 153MB and between 360-405MB when it settles out. I get the JIT and startup memory being higher, spending a little more loading the string but WOW does json loading the string chew up a bunch. Multiplying that memory across processes is eating a bunch. What easy things am I missing? Thanks, Rob

2 3

RPython by Example
by Mingshen Sun 18 Aug '19

18 Aug '19

Hi all, I think RPython is a very promising language in some scenarios other than implementing PyPy itself. However, the usages (specs) of RPython are not very intuitive. Therefore, after reading the documentations and source code of PyPy/RPython, I started to write a "book" titled RPython by Example. Some chapters are still missing and not complete, and I'll continue improve it. Feel free to ask me any questions about the "book". Thanks. Link: https://github.com/mesalock-linux/rpython-by-example Rendered: - HTML: https://mesapy.org/rpython-by-example/ - PDF: https://mesapy.org/rpython-by-example/RPythonByExample.pdf -- Best, Mingshen Sun

1 0

Sandbox 2
by Armin Rigo 15 Aug '19

15 Aug '19

Hi all, I've got an early but probably usable PyPy and PyPy3 with sandboxing. Like the old sandboxing, these new versions are made out of a special version of PyPy (or PyPy3, which is new), running in a mode where no direct I/O should be possible; and a separate controller process. The communication between these two processes has been rewritten. Now, it actually looks similar to the communication between a process and a OS kernel. For example, when a regular process wants to write data to a file, it calls the OS function "write(int fd, char *buf, size_t count)" (actually it calls the standard C library, which is itself a relatively thin wrapper around the OS, but we'll ignore that). Then the OS proceeds to read directly the part of the memory owned by the process, from addresses "buf" to "buf + count". In the sandboxed version of PyPy, this is replaced by the sandboxed process saying to the controller process "I'm trying to call write() with these three arguments: two numbers and one pointer". Assuming that the controller supports "write()", it contains logic that will then ask the sandboxed process for the content of some of its memory, from "buf" to "buf + count". In other system calls like "read(fd, buf, count)", the controller would write (instead of reading) data into the sandboxed process' raw memory, between "buf" and "buf + count". This approach means that a single sandboxed PyPy (or PyPy3) is needed, and it supports out of the box all "system calls". It's up to the controller process to either support them or not, and when it does, to do things like (if necessary) reading and writing raw memory from/to the sandboxed subprocess. It puts more responsibility into the hands of the controller process, but is also far more flexible. It is the same security approach as for the OS, basically: the sandboxed process can do whatever it wants but all its I/O are tightly controlled. Sources: * The sandboxed PyPy including the necessary RPython changes is in PyPy's standard repo, branch "sandbox-2". Translate with ``rpython/bin/rpython -O2 --sandbox``. * The PyPy3 version is in the branch "py3.6-sandbox-2". Same command to translate. * The controller process, or at least one possible version of it, has been split into its own repo: http://bitbucket.org/pypy/sandboxlib . This is CPython-or-PyPy-2-or-3 source code. You need to run ``setup.py build_ext -f -i``. To run the tests (with pytest), make a symlink from ``test/pypy-c-sandbox`` to the sandboxed version built above. Try also ``interact.py``. Right now there is no way to limit the amount of RAM of CPU time consumed by the sandboxed process, but I think it should be done with standard platform tools (e.g. ``ulimit``). Please try it out and give me any feedback ! A bientôt, Armin.

2 2

Profiling another thread with VMProf real_time
by Timothy Palpant 09 Aug '19

09 Aug '19

Hi team, I would like to use VMProf to perform wall-clock profiling (real_time=True) on another thread in my application. I made a proposal for enabling this in https://github.com/vmprof/vmprof-python/issues/201 and a corresponding patch in https://github.com/vmprof/vmprof-python/pull/205. I was hoping someone could help review it. I'm also happy to have input (and have some time to work on it) if someone thinks there is a better approach. I've a handful of other small patches proposed as well to try to help fix the build on OS X and the real_time_threaded tests, but I'm mostly interested in this one. Thanks! Tim

1 0

better line number information in PyPy3
by Carl Friedrich Bolz 08 Aug '19

08 Aug '19

Hi all, This is a discussion mail about improving the traceback line number quality in PyPy3. Some context: CPython has started supporting negative line number offset from one bytecode to the next in its lnotab encoding, see here: https://bugs.python.org/issue26107 https://github.com/python/cpython/commit/f3914eb16d2#diff-cb296cc5109f5640f… (Please don't ask me why not a single test was added for this new feature.) However, the support is so far only theoretically possible, the Python 3.7/3.8 bytecode compiler does not seem to make use of the feature. Negative line number jumps occur quite regularly with continuation lines, eg like this: def foobar(a, b): return "" + (1, a, b, 2 ) If you call this function, the traceback will contain this line: File "x.py", line 5, in foobar 2 TypeError: can only concatenate str (not "tuple") to str Which is obviously misleading. A slightly more realistic example are function decorators. There, the execution order is regularly "backwards": first the function definition is executed, then the individual decorators are run. There is currently a hack to sometimes leave the line number on the line of the first decorator, leading to confusing tracebacks. Example: def works(f): return f def broken(f): assert 0 @works @broken @works def g(): pass Running this, you'll get the following traceback: Traceback (most recent call last): File "y.py", line 9, in <module> @works File "y.py", line 5, in broken assert 0 AssertionError Pointing you again to the wrong line. We could fix this problem rather easily by removing the code that prevents the bytecode compiler from emitting negative line number offsets. Should I implement this? Advantages: - A lot less confusing tracebacks in real-world cases Disadvantages: - Different behavior from CPython (but arguably more correct) - Tracing tools could potentially be confused - Potentially larger lnotab strings due to more jumping around Cheers, Carl Friedrich

2 1

Python 2 vs Python 3 again, and a 2.x-only dependency
by Dan Stromberg 01 Aug '19

01 Aug '19

What is Pypy planning to do with Python 2.x, now that many packages on Pypi are dropping Python 2.x support? EG, I believe Pypy caters to Scientific Computing, but with Numpy dropping Python 2.x support, will Pypy3 become the more relevant version of Pypy? Will Numpypy be revived? Will there be a pypypi or something similar-to-but-different-from pypi just for pypy? Will 2.x in Pypy become mostly relevant for RPython, and not most developer projects written in Python that use Pypy? I'm asking mostly because I have a large, complex dependency that isn't planning to move to Python 3.x, and I'm wondering if Pypy 2.x might be what we need to keep that code alive. But with Numpy and other dependencies dropping 2.x, that might not be enough. Thanks for Pypy, BTW. I really like it a lot.

8 9