[Python-Dev] Mysterious Python pyc file corruption problems

[Barry Warsaw]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On May 15, 2013, at 06:06 PM, Tres Seaver wrote:

>On 05/15/2013 04:58 PM, Barry Warsaw wrote:
>> This leads me to hypothesize that the bug is due to an as yet
>> unidentified race condition during installation of Python source code
>> on Ubuntu, which is normally when we automatically byte compile the
>> source to .pyc files.
>
>Any chance you are using 'detox' or the equivalent to run tests on
>mutliple interpreters in parallel?  The only "bad marshall data" errors I
>have seen lately seemed to be provoked by that kind of practice.

Nope.  PyPI's detox isn't even available in Ubuntu currently.  (The detox
package in Ubuntu is something else.)

Tests should only be run at package build time, not installation time, and the
byte compiling of source files at installation time *should* be single
threaded and single process.

We've since found a few cases where Python 3.3 pyc files are probably
corrupted, so that shoots down my theory about a race condition on
reading/writing pyc files, since 3.3 implements atomic-rename and *should* be
immune to that kind of thing.

It's still a mystery though.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iQIcBAEBCAAGBQJRlP3SAAoJEBJutWOnSwa/Xy8QAI6Ul/s0nF+rac8nGy6fieLB
FHHEfmjIgj6MkyUUw/zcbR48ELiOPkkV+GM4HJnY/H2ZG9vZqwuQYWFI0cgIBmxj
EmHfKK7OTdaaHZeNTt83RDwGRLUS/gYXQ7JVikGyFnSbftfmUoN/y/ndlX5DX1hT
ecDHVtXCH/ti/kcOWe2OlMABONZQPW0qYB7/0PiCCmaOxulqUsz20Ofy8SfWmSPd
Rbig5i8fSnI98dkLVUzyy1tbUkdRkLBro/hawu1V9y7qVkoYx1Jz6p8XkQLp3jES
m22m+6CLrnD39HxvJGGNkIaYmu5xTW+rK/Li8OrfOKx6QVIZ+XQRJFkiXnKmiezk
sMYv/psySWJ/BSImsQOSt/sLHJWAGh8fkMIBpx9tI3BWMvyMkI0Hs9l7JyQn0moo
oSTNb9AbgRSrkh0rVv4fhOhd1Ir3LXYTGwwYE5+o7tB/Pp0AKi2tX/XTBctDpy86
xqNHOaCV0hRA2Y+/C2QAAA7LRruP0yv10DfkciVUHR7UzbXgViICEEUizGmnkni7
utGg9EDk5VcSeg2ySxhX9Uj3E2M/ijOuYpXUJ2Gwd4UNUT5XGK1+6i2JTO2pEQOM
HqhsGqk4WJsfEBTIrAt4NSxZyEuQ2nRV3MIsNaVCDp1FDySZWt3Cckq8hkZ/6vOM
7ncE6aG1cJgq4WKErvCk
=BMFW
-----END PGP SIGNATURE-----