[Python-Dev] Deterministic builds of the interpreter

[Freddy Rietdijk]

For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items
still seem indeterministic.
To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out`
where $out is the path where I installed Python.

Do you have an idea why in [3], this is Python 2.7, the timestamps are
still incorrect? I think they're all required for `compileall` and somehow
it doesn't seem capable of taking into account DETERMINISTIC_BUILD.
Explicitly removing those pyc and pyo files and recompiling them to
bytecode still results in timestamp issues for these 4 files.

On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki <songofacandy at gmail.com> wrote:

> On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk
> <freddyrietdijk at fridh.nl> wrote:
> > Correct, that was an older version from before I patched
> > `_bootstrap_external.py`. A more recent diff can be found at
> >
> > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa
> >
> > These all seem to be sets.
>
> Maybe, PYTHONHASHSEED help you.
> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED
>
> >
> > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki <songofacandy at gmail.com>
> wrote:
> >>
> >> As reading [4], mtime is not 0.
> >>
> >>     data = bytearray(MAGIC_NUMBER)
> >>     data.extend(_w_long(mtime))
> >>     data.extend(_w_long(source_size))
> >>     data.extend(marshal.dumps(code))
> >>
> >> First 4 bytes are magic.
> >> Next 4 bytes are mtime.
> >>
> >> │ │ │ │ -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000
> >> ....k..Xl!......
> >> │ │ │ │ +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000
> >> .....1.Xl!......
> >>
> >> mtime is 6b2e9c58 vs e6319c53 (little endian)
> >>
> >> maybe, you failed to use customized py_compile when building?
> >>
> >>
> >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk <
> freddyrietdijk at fridh.nl>
> >> wrote:
> >> > Hi,
> >> >
> >> > I'm attempting to make the builds of the Python interpreters for
> Nixpkgs
> >> > [1]
> >> > deterministic.
> >> >
> >> > In the case of Python 2.7 we have a patch [2] that fixes the timestamp
> >> > used
> >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We
> also
> >> > remove `wininst*.exe`. This works fine, although there are 4 small
> >> > issues
> >> > left [3]. Do you have any idea what is going on in these files that
> >> > could
> >> > make them indeterministic?
> >> >
> >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and
> >> > modified
> >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds
> >> > are
> >> > not yet deterministic [4]. Any suggestions what could be fixed here?
> >> >
> >> > Kind regards,
> >> >
> >> > Freddy
> >> >
> >> >
> >> > [1] https://github.com/NixOS/nixpkgs
> >> > [2]
> >> >
> >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/
> development/interpreters/python/cpython/2.7/deterministic-build.patch
> >> > [3] https://github.com/NixOS/nixpkgs/issues/22570#
> issuecomment-278474082
> >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e
> 6b
> >> >
> >> > _______________________________________________
> >> > Python-Dev mailing list
> >> > Python-Dev at python.org
> >> > https://mail.python.org/mailman/listinfo/python-dev
> >> > Unsubscribe:
> >> >
> >> > https://mail.python.org/mailman/options/python-dev/
> songofacandy%40gmail.com
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170210/4d13e0f0/attachment-0001.html>