From status at bugs.python.org  Fri Jan  1 12:08:34 2016
From: status at bugs.python.org (Python tracker)
Date: Fri,  1 Jan 2016 18:08:34 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160101170834.CF291566E9@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2015-12-25 - 2016-01-01)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5341 ( -4)
  closed 32402 (+37)
  total  37743 (+33)

Open issues with patches: 2351 


Issues opened (18)
==================

#25953: re fails to identify invalid numeric group references in repla
http://bugs.python.org/issue25953  opened by bazwal

#25954: Python 3.5.1 installer fails on Windows 7
http://bugs.python.org/issue25954  opened by nedbat

#25957: sockaddr_l2 lacks CID, address type (AF_BLUETOOTH sockets)
http://bugs.python.org/issue25957  opened by mikeryan

#25958: Implicit ABCs have no means of "anti-registration"
http://bugs.python.org/issue25958  opened by abarnert

#25959: tkinter - PhotoImage.zoom() causes segfault
http://bugs.python.org/issue25959  opened by hcbd

#25962: Ctrl+C Can't Exit Script with Pool on Windows
http://bugs.python.org/issue25962  opened by Meng-Yuan Huang

#25963: strptime not parsing some timezones
http://bugs.python.org/issue25963  opened by utkonos

#25966: Bug in  asyncio.corotuines._format_coroutine
http://bugs.python.org/issue25966  opened by Brett Rosen

#25967: Devguide: add 2to3 to the "Changing CPython's Grammar" checkli
http://bugs.python.org/issue25967  opened by ezio.melotti

#25968: Cannot import modules out of zip file with extended-length pat
http://bugs.python.org/issue25968  opened by Chris Moore

#25969: Update lib2to3 grammar to include missing unpacking generaliza
http://bugs.python.org/issue25969  opened by ezio.melotti

#25970: py_compile.compile fails if existing bytecode file is unwritab
http://bugs.python.org/issue25970  opened by byrnes

#25974: Fix statistics.py after the Decimal.as_integer_ratio() change
http://bugs.python.org/issue25974  opened by skrah

#25981: Intern namedtuple field names
http://bugs.python.org/issue25981  opened by serhiy.storchaka

#25982: multiprocessing docs for Namespace lacks class definition
http://bugs.python.org/issue25982  opened by davin

#25983: Add tests for multi-argument type()
http://bugs.python.org/issue25983  opened by serhiy.storchaka

#25984: Expose a simple "is IEEE 754" flag in sys.float_info
http://bugs.python.org/issue25984  opened by random832

#25985: Use sys.version_info instead of sys.version
http://bugs.python.org/issue25985  opened by serhiy.storchaka



Most recent 15 issues with no replies (15)
==========================================

#25985: Use sys.version_info instead of sys.version
http://bugs.python.org/issue25985

#25984: Expose a simple "is IEEE 754" flag in sys.float_info
http://bugs.python.org/issue25984

#25982: multiprocessing docs for Namespace lacks class definition
http://bugs.python.org/issue25982

#25970: py_compile.compile fails if existing bytecode file is unwritab
http://bugs.python.org/issue25970

#25969: Update lib2to3 grammar to include missing unpacking generaliza
http://bugs.python.org/issue25969

#25968: Cannot import modules out of zip file with extended-length pat
http://bugs.python.org/issue25968

#25967: Devguide: add 2to3 to the "Changing CPython's Grammar" checkli
http://bugs.python.org/issue25967

#25966: Bug in  asyncio.corotuines._format_coroutine
http://bugs.python.org/issue25966

#25962: Ctrl+C Can't Exit Script with Pool on Windows
http://bugs.python.org/issue25962

#25952: code_context not available in exec()
http://bugs.python.org/issue25952

#25951: SSLSocket.sendall() does not return None on success like socke
http://bugs.python.org/issue25951

#25948: Invalid MIME encoding generated by email.mime (line too long)
http://bugs.python.org/issue25948

#25946: configure should pick /usr/bin/g++ automatically if present
http://bugs.python.org/issue25946

#25943: Integer overflow in _bsddb leads to heap corruption
http://bugs.python.org/issue25943

#25934: ICC compiler: ICC treats denormal floating point numbers as 0.
http://bugs.python.org/issue25934



Most recent 15 issues waiting for review (15)
=============================================

#25985: Use sys.version_info instead of sys.version
http://bugs.python.org/issue25985

#25983: Add tests for multi-argument type()
http://bugs.python.org/issue25983

#25981: Intern namedtuple field names
http://bugs.python.org/issue25981

#25974: Fix statistics.py after the Decimal.as_integer_ratio() change
http://bugs.python.org/issue25974

#25953: re fails to identify invalid numeric group references in repla
http://bugs.python.org/issue25953

#25949: Lazy creation of __dict__ in OrderedDict
http://bugs.python.org/issue25949

#25945: Type confusion in partial_setstate and partial_call leads to m
http://bugs.python.org/issue25945

#25942: subprocess.call SIGKILLs too liberally
http://bugs.python.org/issue25942

#25941: Add 'How to Review a Patch' section to devguide
http://bugs.python.org/issue25941

#25939: _ssl.enum_certificates() fails with ERROR_ACCESS_DENIED if pyt
http://bugs.python.org/issue25939

#25937: DIfference between utf8 and utf-8 when i define python source 
http://bugs.python.org/issue25937

#25935: OrderedDict prevents garbage collection if a circulary referen
http://bugs.python.org/issue25935

#25933: Unhandled exception (TypeError) with ftplib in function retrbi
http://bugs.python.org/issue25933

#25925: Coverage support for CPython 2
http://bugs.python.org/issue25925

#25919: http.client PUT method ignores error responses sent immediatly
http://bugs.python.org/issue25919



Top 10 most discussed issues (10)
=================================

#25937: DIfference between utf8 and utf-8 when i define python source 
http://bugs.python.org/issue25937  12 msgs

#25864: collections.abc.Mapping should include a __reversed__ that rai
http://bugs.python.org/issue25864  10 msgs

#25959: tkinter - PhotoImage.zoom() causes segfault
http://bugs.python.org/issue25959   6 msgs

#25981: Intern namedtuple field names
http://bugs.python.org/issue25981   6 msgs

#19475: Add timespec optional flag to datetime isoformat() to choose t
http://bugs.python.org/issue19475   5 msgs

#25954: Python 3.5.1 installer fails on Windows 7
http://bugs.python.org/issue25954   5 msgs

#25939: _ssl.enum_certificates() fails with ERROR_ACCESS_DENIED if pyt
http://bugs.python.org/issue25939   4 msgs

#25942: subprocess.call SIGKILLs too liberally
http://bugs.python.org/issue25942   4 msgs

#25958: Implicit ABCs have no means of "anti-registration"
http://bugs.python.org/issue25958   4 msgs

#25974: Fix statistics.py after the Decimal.as_integer_ratio() change
http://bugs.python.org/issue25974   4 msgs



Issues closed (30)
==================

#12484: The Py_InitModule functions no longer exist, but remain in the
http://bugs.python.org/issue12484  closed by brett.cannon

#19511: lib2to3 Grammar file is no longer a Python 3 superset
http://bugs.python.org/issue19511  closed by ezio.melotti

#19873: There is a duplicate function in Lib/test/test_pathlib.py
http://bugs.python.org/issue19873  closed by ezio.melotti

#20607: multiprocessing cx_Freeze windows GUI bug (& easy fixes)
http://bugs.python.org/issue20607  closed by davin

#21579: Python 3.4: tempfile.close attribute does not work
http://bugs.python.org/issue21579  closed by mmarkk

#22995: Restrict default pickleability
http://bugs.python.org/issue22995  closed by serhiy.storchaka

#23166: urllib2 ignores opener configuration under certain circumstanc
http://bugs.python.org/issue23166  closed by martin.panter

#24725: test_socket testFDPassEmpty fails on OS X 10.11 DP with "Canno
http://bugs.python.org/issue24725  closed by brett.cannon

#25157: Installing Python 3.5.0 32bit on Windows 8.1 64bit system give
http://bugs.python.org/issue25157  closed by terry.reedy

#25360: pyw should search for pythonw to implement #!/usr/bin/env  pyt
http://bugs.python.org/issue25360  closed by python-dev

#25664: Logging cannot handle Unicode logger names
http://bugs.python.org/issue25664  closed by python-dev

#25685: Inefficiency with SocketHandler - may send log record message 
http://bugs.python.org/issue25685  closed by python-dev

#25789: py launcher stderr is not piped to subprocess.Popen.stderr
http://bugs.python.org/issue25789  closed by python-dev

#25923: More const char
http://bugs.python.org/issue25923  closed by serhiy.storchaka

#25928: Add Decimal.as_integer_ratio()
http://bugs.python.org/issue25928  closed by skrah

#25955: email.utils.formataddr does not support RFC 6532
http://bugs.python.org/issue25955  closed by r.david.murray

#25956: Unambiguous representation of recursive objects
http://bugs.python.org/issue25956  closed by serhiy.storchaka

#25960: Popen.wait() hangs when called from a signal handler when os.w
http://bugs.python.org/issue25960  closed by gregory.p.smith

#25961: Disallow the null character in type name
http://bugs.python.org/issue25961  closed by serhiy.storchaka

#25964: optparse.py:1668: (file) shadows builtin
http://bugs.python.org/issue25964  closed by SilentGhost

#25965: decimal.py: issues reported by pychecker
http://bugs.python.org/issue25965  closed by r.david.murray

#25971: Optimize converting float and Decimal to Fraction
http://bugs.python.org/issue25971  closed by serhiy.storchaka

#25972: All Windows buildbots fail compile/clean
http://bugs.python.org/issue25972  closed by zach.ware

#25973: Segmentation fault with nonlocal and two underscores
http://bugs.python.org/issue25973  closed by python-dev

#25975: Weird multiplication
http://bugs.python.org/issue25975  closed by skrah

#25976: telnetlib SyntaxError: invalid syntax
http://bugs.python.org/issue25976  closed by SilentGhost

#25977: Typo fixes in Lib/tokenize.py
http://bugs.python.org/issue25977  closed by berker.peksag

#25978: escape sequence r'\' giving compilation error
http://bugs.python.org/issue25978  closed by r.david.murray

#25979: String functions lstrip are not working properly when you have
http://bugs.python.org/issue25979  closed by ethan.furman

#25980: not able to find module request in lib urllib - Python35-32
http://bugs.python.org/issue25980  closed by brett.cannon

From mike.romberg at comcast.net  Sat Jan  2 14:32:01 2016
From: mike.romberg at comcast.net (mike.romberg at comcast.net)
Date: Sat, 2 Jan 2016 12:32:01 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
Message-ID: <22152.9649.202823.766027@lrd.home.lan>


  BRIEF INTRODUCTION:  I've been using python since the early 1.X
releases.  Mostly for application development.  On occasion I've
contributed bits to the core:

> grep Romberg Misc/ACKS 
Mike Romberg

  I've recently ported a large application to python3 (it started life
as using 1.1 so it has been a long road for this codebase).  The one
big killer feature of python3 I'm attempting to use is implicit
namespace packages.  But they are broken with the zipimport.c module.

  
  It seems that zipimport.c never worked with these as it is
comparing paths in the form 'a.b.c' to paths in the form 'a/b/c'.  I
created a patch that fixes this and makes zipimport work exactly the
same as a standard filesystem import.  I was getting my patch ready to
submit when I found that this problem has already been reported:

https://bugs.python.org/issue17633

  Is there anything I can do to help fix this issue?  I could polish
up my patch create test cases and submit them.  But it looks like the
above patch does the same thing and is in "the process".  But it has
been "in the process" for three years.  What else needs to be done?
I'll help if I can.

Mike

From brett at python.org  Sat Jan  2 15:36:23 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 02 Jan 2016 20:36:23 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <22152.9649.202823.766027@lrd.home.lan>
References: <22152.9649.202823.766027@lrd.home.lan>
Message-ID: <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>

FIrst off, Mike, sorry about the bug. We have unfortunately let zipimport
get into a sorry state that has made no one want to work on the code
anymore. That being said, I opened https://bugs.python.org/issue25711 to
specifically try to fix this issue once and for all and along the way
modernize zipimport by rewriting it from scratch to be more maintainable
(or whatever the module is named in case we break backwards-compatibility).

At this point the best option might be, Mike, if you do a code review for
https://bugs.python.org/issue17633, even if it is simply a LGTM. I will
then personally make sure the approved patch gets checked in for Python 3.6
in case the rewrite of zipimport misses the release.

On Sat, 2 Jan 2016 at 11:35 <mike.romberg at comcast.net> wrote:

>
>   BRIEF INTRODUCTION:  I've been using python since the early 1.X
> releases.  Mostly for application development.  On occasion I've
> contributed bits to the core:
>
> > grep Romberg Misc/ACKS
> Mike Romberg
>
>   I've recently ported a large application to python3 (it started life
> as using 1.1 so it has been a long road for this codebase).  The one
> big killer feature of python3 I'm attempting to use is implicit
> namespace packages.  But they are broken with the zipimport.c module.
>
>
>   It seems that zipimport.c never worked with these as it is
> comparing paths in the form 'a.b.c' to paths in the form 'a/b/c'.  I
> created a patch that fixes this and makes zipimport work exactly the
> same as a standard filesystem import.  I was getting my patch ready to
> submit when I found that this problem has already been reported:
>
> https://bugs.python.org/issue17633
>
>   Is there anything I can do to help fix this issue?  I could polish
> up my patch create test cases and submit them.  But it looks like the
> above patch does the same thing and is in "the process".  But it has
> been "in the process" for three years.  What else needs to be done?
> I'll help if I can.
>
> Mike
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160102/429bb4b6/attachment.html>

From mike.romberg at comcast.net  Sat Jan  2 17:26:33 2016
From: mike.romberg at comcast.net (mike.romberg at comcast.net)
Date: Sat, 2 Jan 2016 15:26:33 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
Message-ID: <22152.20121.966791.318438@lrd.home.lan>


-- 
>>>>> "Brett" == Brett Cannon <brett at python.org> writes:

    > I opened
    > https://bugs.python.org/issue25711 to specifically try to
    > fix this issue once and for all and along the way modernize
    > zipimport by rewriting it from scratch to be more
    > maintainable

  Every time I read about impementing a custom loader:

https://docs.python.org/3/library/importlib.html

  I've wondered why python does not have some sort of virtual
filesystem layer to deal with locating modules/packages/support
files.   Virtual file systems seem like a good way to store data on a
wide range of storage devices.

  A VFSLoader object would interface with importlib and deal with:

  - implementing a finder and loader

  - Determine the actual type of file to load (.py, .pyc, .pyo,
    __pycache__, etc).

  - Do all of it's work by calling virtual functions such as:
     * listdir(path)
     * read(path)
     * stat(path)  # for things like mtime, size, etc
     * write(path, data)  # not all VFS implement this

  Then things like a ziploader would just inherit from VFSLoader
implement the straight forward methods and everything should "just
work" :).  I see no reason why every type of loader (real filesystem,
http, ssh, sql database, etc) would not do this as well.  Leave all
the details such as implicit namespace packages, presence of
__init__.py[oc] files, .pth files, etc in one single
location and put the details on how to interact with the actual
storage device in leaf classes which don't know or care about the high
level details.  They would not even know they are loading python
modules.  It is just blobs of data to them.

  I may try my hand at creating a prototype of this for just the
zipimporter and see how it goes.

  
    > At this point the best option might be, Mike, if you do a
    > code review for https://bugs.python.org/issue17633, even if
    > it is simply a LGTM. I will then personally make sure the
    > approved patch gets checked in for Python 3.6 in case the
    > rewrite of zipimport misses the release.

  Cool.  I'll see what I can do.  I was having a bit of trouble with
the register/login part of the bug tracker.  Which is why I came
here.  I'll battle with it one more time and see if I can get it to
log me in.

  The patch should be fairly simple.  In a nutshell it just does a:

  path.replace('.', '/') + '/' in two locations.  One where it checks
for the path being a directory entry in the zip file and the second to
return an implicit namespace path (instead of not found) if it is a
match.   I'll check the patch on the tracker and see if it still works
with 3.5.1.  If not I'll attach mine.

Mike

From guido at python.org  Sat Jan  2 22:41:28 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 2 Jan 2016 20:41:28 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <22152.20121.966791.318438@lrd.home.lan>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
Message-ID: <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>

On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:

>
> --
> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
>
>     > I opened
>     > https://bugs.python.org/issue25711 to specifically try to
>     > fix this issue once and for all and along the way modernize
>     > zipimport by rewriting it from scratch to be more
>     > maintainable
>
>   Every time I read about impementing a custom loader:
>
> https://docs.python.org/3/library/importlib.html
>
>   I've wondered why python does not have some sort of virtual
> filesystem layer to deal with locating modules/packages/support
> files.   Virtual file systems seem like a good way to store data on a
> wide range of storage devices.
>

Yeah, but most devices already implement a *real* filesystem, so the only
time the VFS would come in handy would be for zipfiles, where we already
have a solution.


>   A VFSLoader object would interface with importlib and deal with:
>
>   - implementing a finder and loader
>
>   - Determine the actual type of file to load (.py, .pyc, .pyo,
>     __pycache__, etc).
>
>   - Do all of it's work by calling virtual functions such as:
>      * listdir(path)
>      * read(path)
>      * stat(path)  # for things like mtime, size, etc
>      * write(path, data)  # not all VFS implement this
>

Emulating a decent filesystem API requires you to implement functionality
that would never be used by an import loader (write() is an example -- many
of the stat() fields are another example). So it would just be overkill.


>   Then things like a ziploader would just inherit from VFSLoader
> implement the straight forward methods and everything should "just
> work" :).  I see no reason why every type of loader (real filesystem,
> http, ssh, sql database, etc) would not do this as well.


All those examples except "real filesystem" are of very limited practical
value.


> Leave all
> the details such as implicit namespace packages, presence of
> __init__.py[oc] files, .pth files, etc in one single
> location and put the details on how to interact with the actual
> storage device in leaf classes which don't know or care about the high
> level details.  They would not even know they are loading python
> modules.  It is just blobs of data to them.
>

Actually the import loader API is much more suitable and less work to
implement than a VFS.


>   I may try my hand at creating a prototype of this for just the
> zipimporter and see how it goes.
>

That would nevertheless be an interesting exercise -- I hope you do it and
report back.


>     > At this point the best option might be, Mike, if you do a
>     > code review for https://bugs.python.org/issue17633, even if
>     > it is simply a LGTM. I will then personally make sure the
>     > approved patch gets checked in for Python 3.6 in case the
>     > rewrite of zipimport misses the release.
>
>   Cool.  I'll see what I can do.  I was having a bit of trouble with
> the register/login part of the bug tracker.  Which is why I came
> here.  I'll battle with it one more time and see if I can get it to
> log me in.
>

If you really can't manage to comment in the tracker (which sounds unlikely
-- many people have succeeded :-) you can post your LGTM on the specific
patch here.


>   The patch should be fairly simple.  In a nutshell it just does a:
>
>   path.replace('.', '/') + '/' in two locations.  One where it checks
> for the path being a directory entry in the zip file and the second to
> return an implicit namespace path (instead of not found) if it is a
> match.   I'll check the patch on the tracker and see if it still works
> with 3.5.1.  If not I'll attach mine.


Well, Brett would like to see your feedback on the specific patch. Does it
work for you?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160102/64ba5d83/attachment.html>

From brett at python.org  Sat Jan  2 23:42:08 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 03 Jan 2016 04:42:08 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
Message-ID: <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>

I just wanted to quickly say that Guido's observation as to how a VFS is
overkill is right. Imagine implementing a loader using sqlite and you
quickly realize that doing a dull VFS is more than necessary to implement
what import needs to function.

On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido at python.org> wrote:

> On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
>
>>
>> --
>> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
>>
>>     > I opened
>>     > https://bugs.python.org/issue25711 to specifically try to
>>     > fix this issue once and for all and along the way modernize
>>     > zipimport by rewriting it from scratch to be more
>>     > maintainable
>>
>>   Every time I read about impementing a custom loader:
>>
>> https://docs.python.org/3/library/importlib.html
>>
>>   I've wondered why python does not have some sort of virtual
>> filesystem layer to deal with locating modules/packages/support
>> files.   Virtual file systems seem like a good way to store data on a
>> wide range of storage devices.
>>
>
> Yeah, but most devices already implement a *real* filesystem, so the only
> time the VFS would come in handy would be for zipfiles, where we already
> have a solution.
>
>
>>   A VFSLoader object would interface with importlib and deal with:
>>
>>   - implementing a finder and loader
>>
>>   - Determine the actual type of file to load (.py, .pyc, .pyo,
>>     __pycache__, etc).
>>
>>   - Do all of it's work by calling virtual functions such as:
>>      * listdir(path)
>>      * read(path)
>>      * stat(path)  # for things like mtime, size, etc
>>      * write(path, data)  # not all VFS implement this
>>
>
> Emulating a decent filesystem API requires you to implement functionality
> that would never be used by an import loader (write() is an example -- many
> of the stat() fields are another example). So it would just be overkill.
>
>
>>   Then things like a ziploader would just inherit from VFSLoader
>> implement the straight forward methods and everything should "just
>> work" :).  I see no reason why every type of loader (real filesystem,
>> http, ssh, sql database, etc) would not do this as well.
>
>
> All those examples except "real filesystem" are of very limited practical
> value.
>
>
>> Leave all
>> the details such as implicit namespace packages, presence of
>> __init__.py[oc] files, .pth files, etc in one single
>> location and put the details on how to interact with the actual
>> storage device in leaf classes which don't know or care about the high
>> level details.  They would not even know they are loading python
>> modules.  It is just blobs of data to them.
>>
>
> Actually the import loader API is much more suitable and less work to
> implement than a VFS.
>
>
>>   I may try my hand at creating a prototype of this for just the
>> zipimporter and see how it goes.
>>
>
> That would nevertheless be an interesting exercise -- I hope you do it and
> report back.
>
>
>>     > At this point the best option might be, Mike, if you do a
>>     > code review for https://bugs.python.org/issue17633, even if
>>     > it is simply a LGTM. I will then personally make sure the
>>     > approved patch gets checked in for Python 3.6 in case the
>>     > rewrite of zipimport misses the release.
>>
>>   Cool.  I'll see what I can do.  I was having a bit of trouble with
>> the register/login part of the bug tracker.  Which is why I came
>> here.  I'll battle with it one more time and see if I can get it to
>> log me in.
>>
>
> If you really can't manage to comment in the tracker (which sounds
> unlikely -- many people have succeeded :-) you can post your LGTM on the
> specific patch here.
>
>
>>   The patch should be fairly simple.  In a nutshell it just does a:
>>
>>   path.replace('.', '/') + '/' in two locations.  One where it checks
>> for the path being a directory entry in the zip file and the second to
>> return an implicit namespace path (instead of not found) if it is a
>> match.   I'll check the patch on the tracker and see if it still works
>> with 3.5.1.  If not I'll attach mine.
>
>
> Well, Brett would like to see your feedback on the specific patch. Does it
> work for you?
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/a22a7b31/attachment.html>

From brett at python.org  Sat Jan  2 23:43:13 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 03 Jan 2016 04:43:13 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
Message-ID: <CAP1=2W64zkE_hrHK0=L4ZLh61u+xkKX9cOFjzKbnonvs5wokFQ@mail.gmail.com>

On Sat, 2 Jan 2016, 20:42 Brett Cannon <brett at python.org> wrote:

> I just wanted to quickly say that Guido's observation as to how a VFS is
> overkill is right. Imagine implementing a loader using sqlite and you
> quickly realize that doing a dull VFS is more
>
"dull" -> "full"

than necessary to implement what import needs to function.
>
> On Sat, 2 Jan 2016, 19:42 Guido van Rossum <guido at python.org> wrote:
>
>> On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
>>
>>>
>>> --
>>> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
>>>
>>>     > I opened
>>>     > https://bugs.python.org/issue25711 to specifically try to
>>>     > fix this issue once and for all and along the way modernize
>>>     > zipimport by rewriting it from scratch to be more
>>>     > maintainable
>>>
>>>   Every time I read about impementing a custom loader:
>>>
>>> https://docs.python.org/3/library/importlib.html
>>>
>>>   I've wondered why python does not have some sort of virtual
>>> filesystem layer to deal with locating modules/packages/support
>>> files.   Virtual file systems seem like a good way to store data on a
>>> wide range of storage devices.
>>>
>>
>> Yeah, but most devices already implement a *real* filesystem, so the only
>> time the VFS would come in handy would be for zipfiles, where we already
>> have a solution.
>>
>>
>>>   A VFSLoader object would interface with importlib and deal with:
>>>
>>>   - implementing a finder and loader
>>>
>>>   - Determine the actual type of file to load (.py, .pyc, .pyo,
>>>     __pycache__, etc).
>>>
>>>   - Do all of it's work by calling virtual functions such as:
>>>      * listdir(path)
>>>      * read(path)
>>>      * stat(path)  # for things like mtime, size, etc
>>>      * write(path, data)  # not all VFS implement this
>>>
>>
>> Emulating a decent filesystem API requires you to implement functionality
>> that would never be used by an import loader (write() is an example -- many
>> of the stat() fields are another example). So it would just be overkill.
>>
>>
>>>   Then things like a ziploader would just inherit from VFSLoader
>>> implement the straight forward methods and everything should "just
>>> work" :).  I see no reason why every type of loader (real filesystem,
>>> http, ssh, sql database, etc) would not do this as well.
>>
>>
>> All those examples except "real filesystem" are of very limited practical
>> value.
>>
>>
>>> Leave all
>>> the details such as implicit namespace packages, presence of
>>> __init__.py[oc] files, .pth files, etc in one single
>>> location and put the details on how to interact with the actual
>>> storage device in leaf classes which don't know or care about the high
>>> level details.  They would not even know they are loading python
>>> modules.  It is just blobs of data to them.
>>>
>>
>> Actually the import loader API is much more suitable and less work to
>> implement than a VFS.
>>
>>
>>>   I may try my hand at creating a prototype of this for just the
>>> zipimporter and see how it goes.
>>>
>>
>> That would nevertheless be an interesting exercise -- I hope you do it
>> and report back.
>>
>>
>>>     > At this point the best option might be, Mike, if you do a
>>>     > code review for https://bugs.python.org/issue17633, even if
>>>     > it is simply a LGTM. I will then personally make sure the
>>>     > approved patch gets checked in for Python 3.6 in case the
>>>     > rewrite of zipimport misses the release.
>>>
>>>   Cool.  I'll see what I can do.  I was having a bit of trouble with
>>> the register/login part of the bug tracker.  Which is why I came
>>> here.  I'll battle with it one more time and see if I can get it to
>>> log me in.
>>>
>>
>> If you really can't manage to comment in the tracker (which sounds
>> unlikely -- many people have succeeded :-) you can post your LGTM on the
>> specific patch here.
>>
>>
>>>   The patch should be fairly simple.  In a nutshell it just does a:
>>>
>>>   path.replace('.', '/') + '/' in two locations.  One where it checks
>>> for the path being a directory entry in the zip file and the second to
>>> return an implicit namespace path (instead of not found) if it is a
>>> match.   I'll check the patch on the tracker and see if it still works
>>> with 3.5.1.  If not I'll attach mine.
>>
>>
>> Well, Brett would like to see your feedback on the specific patch. Does
>> it work for you?
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/6f65a3c3/attachment-0001.html>

From mike.romberg at comcast.net  Sun Jan  3 00:29:52 2016
From: mike.romberg at comcast.net (mike.romberg at comcast.net)
Date: Sat, 2 Jan 2016 22:29:52 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
Message-ID: <22152.45520.973803.741563@lrd.home.lan>

>>>>> " " == Brett Cannon <brett at python.org> writes:

     > I just wanted to quickly say that Guido's observation as to how
     > a VFS is overkill is right. Imagine implementing a loader using
     > sqlite and you quickly realize that doing a dull VFS is more
     > than necessary to implement what import needs to function.

  I fear I've made a poor choice in calling this abstract class a VFS
(I'm terrible with names).  I'm not thinking of anything along the
lines of a full file system that supports open(), seek(), read() and
everything else.   That for sure would be overkill and way more
complicated than it needs to be.

  All I'm really thinking about is a simple abstract interface that is
used by an importer to actually locate and retrieve the binary objects
that will be loaded.  For the simple case I think just two methods
would/could server a read only "blob/byte database":

  listdir(path)  # returns an iterable container of "files"/"dirs" found
                 # at path
                 
  get(path)      # returns a bytes object for the given path

  I think with those two virtual calls a more high level import layer
can locate and retrieve modules to be loaded without even knowing
where they came from.

  The higher level would know about things such as the difference
between .py and .pyc "files" or the possible existance of __pycache__
directories and what may be found in them.  Right now the zipimporter
contains a list of file extensions to try and load (and in what
order).  It also lacks any knowledge of __pycache__ directories (which
is one of the outstanding bugs with it).   It just seems to me that
this sorta logic would be better moved to a higher layer and the zip
layer just translates paths into reads of byte blobs.

  I mentioned write()/put() for two reasons:

  1)  When I import a .py file then a .pyc file is created on my
      filesystem.  I don't really know what piece of code created it.
      But a write to the filesystem (assuming it is writeable and
      permissions set etc) occurs.   It might be nice for other
      storage systems (zip, sql, etc) could optionally support this.
      They could if the code that crated the .pyc simply did a put()
      to the object that pulled in the .py file.  The interface is
      expanded by two calls (put() and delete()).

  2)  Integration with package data.  I know there are
      modules/packages out there that help a module try and locate
      data files that may be associated with a package.  I think it
      would be kinda cool for a module to instead be able to get a
      handle to the abstract class that loaded it.  it could then use
      the same listdir() get() and possibly write methods the importer
      did.  The writing bit of this may or may not be a good idea :)


  Anyway, hope I did not muddy the waters.  I was just thinking a bit
out loud and none of this may live past my own experiments.   I was/am
just trying to think of why the importers like the zipimporter don't
work like a filesystem importer and how they would be cleaner if they
just dealt with paths and byte blobs to store/get based on those paths.

Mike

From ncoghlan at gmail.com  Sun Jan  3 01:34:21 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Jan 2016 16:34:21 +1000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CADiSq7doS-2Rr9bWfT=DtCv0D9pVb1J1-tQ8m++zJ3Y0dxmP+A@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
 <22152.45520.973803.741563@lrd.home.lan>
 <CADiSq7doS-2Rr9bWfT=DtCv0D9pVb1J1-tQ8m++zJ3Y0dxmP+A@mail.gmail.com>
Message-ID: <CADiSq7dMYYdo9+FCLXP-nqXfnSTC7KMnUqDupH0etiHcymvgWQ@mail.gmail.com>

On 3 January 2016 at 16:32, Nick Coghlan <ncoghlan at gmail.com> wrote:
> For folks that are interested in that, folks that aren't following
> import-sig in addition to python-dev may want to take a look at
> Brett's design for the importlib.resources API:
> http://nbviewer.jupyter.org/gist/brettcannon/9c4681a77a7fa09c5347

Sorry, I meant to include a link to the import-sig thread as well:
https://mail.python.org/pipermail/import-sig/2015-November/001041.html

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan  3 01:32:24 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 3 Jan 2016 16:32:24 +1000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <22152.45520.973803.741563@lrd.home.lan>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
 <22152.45520.973803.741563@lrd.home.lan>
Message-ID: <CADiSq7doS-2Rr9bWfT=DtCv0D9pVb1J1-tQ8m++zJ3Y0dxmP+A@mail.gmail.com>

On 3 January 2016 at 15:29,  <mike.romberg at comcast.net> wrote:
>>>>>> " " == Brett Cannon <brett at python.org> writes:
>
>      > I just wanted to quickly say that Guido's observation as to how
>      > a VFS is overkill is right. Imagine implementing a loader using
>      > sqlite and you quickly realize that doing a dull VFS is more
>      > than necessary to implement what import needs to function.
>
>   I fear I've made a poor choice in calling this abstract class a VFS
> (I'm terrible with names).  I'm not thinking of anything along the
> lines of a full file system that supports open(), seek(), read() and
> everything else.   That for sure would be overkill and way more
> complicated than it needs to be.
>
>   All I'm really thinking about is a simple abstract interface that is
> used by an importer to actually locate and retrieve the binary objects
> that will be loaded.  For the simple case I think just two methods
> would/could server a read only "blob/byte database":
>
>   listdir(path)  # returns an iterable container of "files"/"dirs" found
>                  # at path
>
>   get(path)      # returns a bytes object for the given path

We already have the latter:
https://docs.python.org/3/library/importlib.html#importlib.abc.ResourceLoader.get_data

It's the former that has taken a while to get to, as the 3rd party
pkg_resources module (part of setuptools) already provides a pragmatic
API that also has the virtue of being compatible with both Python 2 &
3, and there a few subtleties related to the possible use of temporary
files that make a robust API design trickier than it first appears to
be.

For folks that are interested in that, folks that aren't following
import-sig in addition to python-dev may want to take a look at
Brett's design for the importlib.resources API:
http://nbviewer.jupyter.org/gist/brettcannon/9c4681a77a7fa09c5347

Cheers,
Nick.

P.S. If anyone actually *does* want a full "virtual file system layer"
API for non-filesystem storage locations:
http://docs.pyfilesystem.org/en/latest/filesystems.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From phil at riverbankcomputing.com  Sun Jan  3 05:43:48 2016
From: phil at riverbankcomputing.com (Phil Thompson)
Date: Sun, 3 Jan 2016 10:43:48 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
Message-ID: <0DC6BE63-6816-4288-B2D3-11ADDD661BFA@riverbankcomputing.com>

On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido at python.org> wrote:
> 
> On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
> 
> --
> >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
> 
>     > I opened
>     > https://bugs.python.org/issue25711 to specifically try to
>     > fix this issue once and for all and along the way modernize
>     > zipimport by rewriting it from scratch to be more
>     > maintainable
> 
>   Every time I read about impementing a custom loader:
> 
> https://docs.python.org/3/library/importlib.html
> 
>   I've wondered why python does not have some sort of virtual
> filesystem layer to deal with locating modules/packages/support
> files.   Virtual file systems seem like a good way to store data on a
> wide range of storage devices.
> 
> Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.

Just to point out that it would be nice to have an easier way to use something other that zipfiles. I have a need to exploit a different solution and have to patch the bootstrap code (because the zipfile support is handled as a special case). BTW the need is to create iOS and Android executables from frozen Python code.

Phil


From brett at python.org  Sun Jan  3 12:31:28 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 03 Jan 2016 17:31:28 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <22152.45520.973803.741563@lrd.home.lan>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
 <22152.45520.973803.741563@lrd.home.lan>
Message-ID: <CAP1=2W78KPWoPbKUZt+HpVgNF7k-9KgFrTWB4G9364cmpjjOkQ@mail.gmail.com>

On Sat, 2 Jan 2016 at 21:31 <mike.romberg at comcast.net> wrote:

> >>>>> " " == Brett Cannon <brett at python.org> writes:
>
>      > I just wanted to quickly say that Guido's observation as to how
>      > a VFS is overkill is right. Imagine implementing a loader using
>      > sqlite and you quickly realize that doing a dull VFS is more
>      > than necessary to implement what import needs to function.
>
>   I fear I've made a poor choice in calling this abstract class a VFS
> (I'm terrible with names).  I'm not thinking of anything along the
> lines of a full file system that supports open(), seek(), read() and
> everything else.   That for sure would be overkill and way more
> complicated than it needs to be.
>
>   All I'm really thinking about is a simple abstract interface that is
> used by an importer to actually locate and retrieve the binary objects
> that will be loaded.  For the simple case I think just two methods
> would/could server a read only "blob/byte database":
>
>   listdir(path)  # returns an iterable container of "files"/"dirs" found
>                  # at path
>
>   get(path)      # returns a bytes object for the given path
>
>   I think with those two virtual calls a more high level import layer
> can locate and retrieve modules to be loaded without even knowing
> where they came from.
>
>   The higher level would know about things such as the difference
> between .py and .pyc "files" or the possible existance of __pycache__
> directories and what may be found in them.  Right now the zipimporter
> contains a list of file extensions to try and load (and in what
> order).  It also lacks any knowledge of __pycache__ directories (which
> is one of the outstanding bugs with it).   It just seems to me that
> this sorta logic would be better moved to a higher layer and the zip
> layer just translates paths into reads of byte blobs.
>
>   I mentioned write()/put() for two reasons:
>
>   1)  When I import a .py file then a .pyc file is created on my
>       filesystem.  I don't really know what piece of code created it.
>       But a write to the filesystem (assuming it is writeable and
>       permissions set etc) occurs.   It might be nice for other
>       storage systems (zip, sql, etc) could optionally support this.
>       They could if the code that crated the .pyc simply did a put()
>       to the object that pulled in the .py file.  The interface is
>       expanded by two calls (put() and delete()).
>
>   2)  Integration with package data.  I know there are
>       modules/packages out there that help a module try and locate
>       data files that may be associated with a package.  I think it
>       would be kinda cool for a module to instead be able to get a
>       handle to the abstract class that loaded it.  it could then use
>       the same listdir() get() and possibly write methods the importer
>       did.  The writing bit of this may or may not be a good idea :)
>
>
So it's possible that some abstraction might be possible, but up to this
point there hasn't been a motivating factor for importlib to do this as
zipimport is written in C and thus won't benefit from any abstraction that
importlib uses in its Python code (hence why zipimport needs to be
rewritten). Maybe after we have a zip-backed importer written in Python a
common API will fall through and we can abstract that out, but I wouldn't
want to guess until the code is written and someone can stare at the two
implementations.

It should also be said that there is nothing requiring that an importer
support any concept of a file. Going back to the sqlite database example,
you could make it nothing more than a table that maps module name to source
code and bytecode:

CREATE TABLE code (
    name TEXT PRIMARY KEY NOT NULL,
    source BLOB,
    bytecode BLOB
);
/* A trigger that re-generates `bytecode` when `source` is set would be
nice in this example, but that's beyond my SQL abilities. */

That's enough to implement an importer and there is no mention of file
paths anywhere. If you really wanted to could do a second table that acted
as a database-backed file system for reading package data files, but that
is not required to meet the minimum functionality -- and then some thanks
to bytecode support -- for an importer.


>
>   Anyway, hope I did not muddy the waters.  I was just thinking a bit
> out loud and none of this may live past my own experiments.   I was/am
> just trying to think of why the importers like the zipimporter don't
> work like a filesystem importer and how they would be cleaner if they
> just dealt with paths and byte blobs to store/get based on those paths.
>

It's a reasonable thing to consider, but it would be better to get
zipimport fixed for you, then rewritten, and then look at code for the
rewritten zipimport and what importlib.machinery.SourceFileLoader has to
see if there is a common API to abstract out and build upon.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/ac7b0927/attachment.html>

From brett at python.org  Sun Jan  3 12:33:14 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 03 Jan 2016 17:33:14 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <0DC6BE63-6816-4288-B2D3-11ADDD661BFA@riverbankcomputing.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <0DC6BE63-6816-4288-B2D3-11ADDD661BFA@riverbankcomputing.com>
Message-ID: <CAP1=2W4bqYLU99SU2CxDZUKRPxDgsoNYoXTTH-FrLwfy+5AWuQ@mail.gmail.com>

On Sun, 3 Jan 2016 at 02:55 Phil Thompson <phil at riverbankcomputing.com>
wrote:

> On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido at python.org> wrote:
> >
> > On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
> >
> > --
> > >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
> >
> >     > I opened
> >     > https://bugs.python.org/issue25711 to specifically try to
> >     > fix this issue once and for all and along the way modernize
> >     > zipimport by rewriting it from scratch to be more
> >     > maintainable
> >
> >   Every time I read about impementing a custom loader:
> >
> > https://docs.python.org/3/library/importlib.html
> >
> >   I've wondered why python does not have some sort of virtual
> > filesystem layer to deal with locating modules/packages/support
> > files.   Virtual file systems seem like a good way to store data on a
> > wide range of storage devices.
> >
> > Yeah, but most devices already implement a *real* filesystem, so the
> only time the VFS would come in handy would be for zipfiles, where we
> already have a solution.
>
> Just to point out that it would be nice to have an easier way to use
> something other that zipfiles. I have a need to exploit a different
> solution and have to patch the bootstrap code (because the zipfile support
> is handled as a special case). BTW the need is to create iOS and Android
> executables from frozen Python code.
>

Not quite sure about how zip files are a special-case beyond just being put
in sys.meta_path automatically. You can get the same results with a .pth
file or a sitecustomize.py depending on how pervasive your need is.
Otherwise feel free to file an issue at bugs.python.org and we can talk
over there about what you specifically need and if it's reasonable to try
and support.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/e018c308/attachment.html>

From mike.romberg at comcast.net  Sun Jan  3 17:37:05 2016
From: mike.romberg at comcast.net (mike.romberg at comcast.net)
Date: Sun, 3 Jan 2016 15:37:05 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W78KPWoPbKUZt+HpVgNF7k-9KgFrTWB4G9364cmpjjOkQ@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
 <22152.45520.973803.741563@lrd.home.lan>
 <CAP1=2W78KPWoPbKUZt+HpVgNF7k-9KgFrTWB4G9364cmpjjOkQ@mail.gmail.com>
Message-ID: <22153.41617.619847.318185@lrd.home.lan>

>>>>> " " == Brett Cannon <brett at python.org> writes:

...

     > So it's possible that some abstraction might be possible, but
     > up to this point there hasn't been a motivating factor for
     > importlib to do this as zipimport is written in C and thus
     > won't benefit from any abstraction that importlib uses in its
     > Python code (hence why zipimport needs to be rewritten). Maybe
     > after we have a zip-backed importer written in Python a common
     > API will fall through and we can abstract that out, but I
     > wouldn't want to guess until the code is written and someone
     > can stare at the two implementations.

  Fair enough.  I too think it is a good idea to make base classes
after their is a need for them and not before.   Some argument could
be made that there is a need now as zipimported modules/packages don't
work exactly the same way as "normal" ones.  But since you plan a
rewrite of zipimport collecting the common stuff after that makes sense.

     > It should also be said that there is nothing requiring that an
     > importer support any concept of a file. Going back to the
     > sqlite database example, you could make it nothing more than a
     > table that maps module name to source code and bytecode:

  Yep.  I was not thinking file either.  I may have said file (again
I'm terrible with names) but I'm thinking an array of bytes with a
string id (which I call a path).  The actual storage could simply be
storing the byte array using the stringID/path.

...

     > That's enough to implement an importer and there is no mention
     > of file paths anywhere. If you really wanted to could do a
     > second table that acted as a database-backed file system for
     > reading package data files, but that is not required to meet
     > the minimum functionality -- and then some thanks to bytecode
     > support -- for an importer.

  Yea the python module name could be viewed as a path
(package.subpackage.module) and stored in a hierarchical way.  Or it
could simply be viewed as a string and stored in some flat database.


  Anyway, back at the ranch...  I've got an extended version of the
patch for issue17633 working on my system.  I've added to the test
case to check for the proper functioning of implicit namespace
packages spread between two zip files.  I still need to add tests for
a namespace package spread between a normal filesystem and a zip
file.  I expect I'll have that done in a day or two.  I'll post it to
the bug tracker.

Mike



From guido at python.org  Sun Jan  3 18:21:32 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 3 Jan 2016 15:21:32 -0800
Subject: [Python-Dev] PEP 257 and __init__
In-Reply-To: <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
References: <CAM09pzTEb62AqR_APxMBaywScigVggtS5iv=wKrpCp41z907Dw@mail.gmail.com>
 <B29616C5-5E09-4814-9510-F52187D413E4@yahoo.com>
 <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
Message-ID: <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>

On Tue, Dec 29, 2015 at 1:03 PM, Facundo Batista <facundobatista at gmail.com>
wrote:

> On Tue, Dec 29, 2015 at 4:38 PM, Andrew Barnert <abarnert at yahoo.com>
> wrote:
>
> > Isn't the same thing true for every special method? There are lots of
> classes where __add___ just says "a.__add__(b) = a + b" or (better
> following the PEP) "Return self + value." But, in the rare case where the
> semantics of "a + b" are a little tricky (think of "a / b" for
> pathlib.Path), where else could you put it but __add__?
> >
> > Similarly, for most classes, there's only one of __init__ or __new__,
> and the construction/initialization semantics are simple enough to describe
> in one line of the class docstring--but when things are more complicated
> and need to be documented, where else would you put it?
>
> Yeap. Note that I'm ok to include a docstring when the actual
> behaviour would deviate from the expected one as per Reference Docs.
> My point is to not make it mandatory.
>
>
> > I usually just don't bother. You can violate PEP 257 when it makes
> sense, just like PEP 8. They're just guidelines, not iron-clad rules.
>
> Yeap, but pep257 (the tool [0]) complains for __init__, and wanted to
> know how serious was it.
>
>
> [0] https://pypi.python.org/pypi/pep257


That is the tool's fault. I personally hate with a vengeance that there are
tools named after style guide PEPs that claim to enforce the guidelines
from those PEPs. The tools' rigidity and simplicity reflects badly on the
PEPs, which try hard not to be rigid or simplistic.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160103/1a47a3a4/attachment.html>

From tjreedy at udel.edu  Mon Jan  4 02:01:15 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 4 Jan 2016 02:01:15 -0500
Subject: [Python-Dev] PEP 257 and __init__
In-Reply-To: <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>
References: <CAM09pzTEb62AqR_APxMBaywScigVggtS5iv=wKrpCp41z907Dw@mail.gmail.com>
 <B29616C5-5E09-4814-9510-F52187D413E4@yahoo.com>
 <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
 <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>
Message-ID: <n6d5c2$5sm$1@ger.gmane.org>

On 1/3/2016 6:21 PM, Guido van Rossum wrote:
> On Tue, Dec 29, 2015 at 1:03 PM, Facundo Batista
> <facundobatista at gmail.com <mailto:facundobatista at gmail.com>> wrote:
>
>     On Tue, Dec 29, 2015 at 4:38 PM, Andrew Barnert <abarnert at yahoo.com
>     <mailto:abarnert at yahoo.com>> wrote:
>
>     > Isn't the same thing true for every special method? There are lots of classes where __add___ just says "a.__add__(b) = a + b" or (better following the PEP) "Return self + value." But, in the rare case where the semantics of "a + b" are a little tricky (think of "a / b" for pathlib.Path), where else could you put it but __add__?
>     >
>     > Similarly, for most classes, there's only one of __init__ or __new__, and the construction/initialization semantics are simple enough to describe in one line of the class docstring--but when things are more complicated and need to be documented, where else would you put it?
>
>     Yeap. Note that I'm ok to include a docstring when the actual
>     behaviour would deviate from the expected one as per Reference Docs.
>     My point is to not make it mandatory.
>
>
>     > I usually just don't bother. You can violate PEP 257 when it makes sense, just like PEP 8. They're just guidelines, not iron-clad rules.
>
>     Yeap, but pep257 (the tool [0]) complains for __init__, and wanted to
>     know how serious was it.
>
>
>     [0] https://pypi.python.org/pypi/pep257
>
>
> That is the tool's fault. I personally hate with a vengeance that there
> are tools named after style guide PEPs that claim to enforce the
> guidelines from those PEPs. The tools' rigidity and simplicity reflects
> badly on the PEPs, which try hard not to be rigid or simplistic.

Ask the PSF/pypi people to either prohibit such names or require a 
disclaimer of some sort.  They are inherently confusing: "I took a look 
at pep008" does not mean that one even looked at the PEP.  Even when the 
context makes clear that the referent is the module, there is confusion 
as to its authoritativeness.  That Facudo would post here about the 
module's output illustrates that. To me, the name copying violates our 
informal trademark within Pythonland on 'PEP####'.

-- 
Terry Jan Reedy



From ncoghlan at gmail.com  Mon Jan  4 02:49:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 4 Jan 2016 17:49:32 +1000
Subject: [Python-Dev] PEP 257 and __init__
In-Reply-To: <n6d5c2$5sm$1@ger.gmane.org>
References: <CAM09pzTEb62AqR_APxMBaywScigVggtS5iv=wKrpCp41z907Dw@mail.gmail.com>
 <B29616C5-5E09-4814-9510-F52187D413E4@yahoo.com>
 <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
 <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>
 <n6d5c2$5sm$1@ger.gmane.org>
Message-ID: <CADiSq7fMb0u0kWz93B5Ee24rtiTXgZuep8aYSGKSgfDTLEkXWA@mail.gmail.com>

On 4 January 2016 at 17:01, Terry Reedy <tjreedy at udel.edu> wrote:
> Ask the PSF/pypi people to either prohibit such names or require a
> disclaimer of some sort.  They are inherently confusing: "I took a look at
> pep008" does not mean that one even looked at the PEP.  Even when the
> context makes clear that the referent is the module, there is confusion as
> to its authoritativeness.  That Facudo would post here about the module's
> output illustrates that. To me, the name copying violates our informal
> trademark within Pythonland on 'PEP####'.

I don't think that's the right answer, as opinionated tools do serve a
useful purpose in preventing bikeshedding during code review (people
*expect* computers to be annoyingly pedantic, which frees up the human
reviewers to focus on higher level concerns). As projects evolve over
time, they may develop their own tweaks and customisations in their
style guide and switch to a more configurable tool, or they may not.

When some of the default settings for the pep8 utility became a
problem, I was able to talk to the developers and persuade them to
tune their defaults to be more in line with the actual PEP text, and
keep their extensions to optional settings.

A similar approach may work for PEP 257, by clarifying which aspects
tools should be leaving to human judgement (beyond the question of
whether or not to opt in to following PEP 257 at all - it's far less
universal than PEP 8).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From facundobatista at gmail.com  Mon Jan  4 06:32:08 2016
From: facundobatista at gmail.com (Facundo Batista)
Date: Mon, 4 Jan 2016 08:32:08 -0300
Subject: [Python-Dev] PEP 257 and __init__
In-Reply-To: <CADiSq7fMb0u0kWz93B5Ee24rtiTXgZuep8aYSGKSgfDTLEkXWA@mail.gmail.com>
References: <CAM09pzTEb62AqR_APxMBaywScigVggtS5iv=wKrpCp41z907Dw@mail.gmail.com>
 <B29616C5-5E09-4814-9510-F52187D413E4@yahoo.com>
 <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
 <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>
 <n6d5c2$5sm$1@ger.gmane.org>
 <CADiSq7fMb0u0kWz93B5Ee24rtiTXgZuep8aYSGKSgfDTLEkXWA@mail.gmail.com>
Message-ID: <CAM09pzTxCFMpQ99aEPwZwgSFUkL9bW1hALLTkc+OYPFY4LGuew@mail.gmail.com>

On Mon, Jan 4, 2016 at 4:49 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> When some of the default settings for the pep8 utility became a
> problem, I was able to talk to the developers and persuade them to
> tune their defaults to be more in line with the actual PEP text, and
> keep their extensions to optional settings.

In that spirit, I opened an issue [0] in the pep257 project to be able
to configure that and bypass the default behaviour, so the tool can be
used in a wider set of projects.

Thanks everybody for all the info.

Regards,

[0] https://github.com/GreenSteam/pep257/issues/171

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/
Twitter: @facundobatista

From phil at riverbankcomputing.com  Mon Jan  4 11:16:06 2016
From: phil at riverbankcomputing.com (Phil Thompson)
Date: Mon, 4 Jan 2016 16:16:06 +0000
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W4bqYLU99SU2CxDZUKRPxDgsoNYoXTTH-FrLwfy+5AWuQ@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <0DC6BE63-6816-4288-B2D3-11ADDD661BFA@riverbankcomputing.com>
 <CAP1=2W4bqYLU99SU2CxDZUKRPxDgsoNYoXTTH-FrLwfy+5AWuQ@mail.gmail.com>
Message-ID: <5AF2F684-CBD1-4D88-AE60-1CA2E6357C38@riverbankcomputing.com>

On 3 Jan 2016, at 5:33 pm, Brett Cannon <brett at python.org> wrote:
> 
> 
> 
> On Sun, 3 Jan 2016 at 02:55 Phil Thompson <phil at riverbankcomputing.com> wrote:
> On 3 Jan 2016, at 3:41 am, Guido van Rossum <guido at python.org> wrote:
> >
> > On Sat, Jan 2, 2016 at 3:26 PM, <mike.romberg at comcast.net> wrote:
> >
> > --
> > >>>>> "Brett" == Brett Cannon <brett at python.org> writes:
> >
> >     > I opened
> >     > https://bugs.python.org/issue25711 to specifically try to
> >     > fix this issue once and for all and along the way modernize
> >     > zipimport by rewriting it from scratch to be more
> >     > maintainable
> >
> >   Every time I read about impementing a custom loader:
> >
> > https://docs.python.org/3/library/importlib.html
> >
> >   I've wondered why python does not have some sort of virtual
> > filesystem layer to deal with locating modules/packages/support
> > files.   Virtual file systems seem like a good way to store data on a
> > wide range of storage devices.
> >
> > Yeah, but most devices already implement a *real* filesystem, so the only time the VFS would come in handy would be for zipfiles, where we already have a solution.
> 
> Just to point out that it would be nice to have an easier way to use something other that zipfiles. I have a need to exploit a different solution and have to patch the bootstrap code (because the zipfile support is handled as a special case). BTW the need is to create iOS and Android executables from frozen Python code.
> 
> Not quite sure about how zip files are a special-case beyond just being put in sys.meta_path automatically. You can get the same results with a .pth file or a sitecustomize.py depending on how pervasive your need is. Otherwise feel free to file an issue at bugs.python.org and we can talk over there about what you specifically need and if it's reasonable to try and support. 

I've created http://bugs.python.org/issue26007 and hope it's clear enough what the issue is.

Thanks,
Phil

From guido at python.org  Mon Jan  4 12:18:01 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 4 Jan 2016 09:18:01 -0800
Subject: [Python-Dev] PEP 257 and __init__
In-Reply-To: <CADiSq7fMb0u0kWz93B5Ee24rtiTXgZuep8aYSGKSgfDTLEkXWA@mail.gmail.com>
References: <CAM09pzTEb62AqR_APxMBaywScigVggtS5iv=wKrpCp41z907Dw@mail.gmail.com>
 <B29616C5-5E09-4814-9510-F52187D413E4@yahoo.com>
 <CAM09pzS9=Szfkfpdgx=Dpp=dUPNzkrFGT03Ue0KHD088JssxTg@mail.gmail.com>
 <CAP7+vJLb85fF+WPPZ=GeB+bdKEjoEDHBFs3vL-+RrU7pMe9RJA@mail.gmail.com>
 <n6d5c2$5sm$1@ger.gmane.org>
 <CADiSq7fMb0u0kWz93B5Ee24rtiTXgZuep8aYSGKSgfDTLEkXWA@mail.gmail.com>
Message-ID: <CAP7+vJJF92JTrA8k5MGAr52KBOH5WaL1SbkEdjFzKUhCkW-69g@mail.gmail.com>

On Sun, Jan 3, 2016 at 11:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 4 January 2016 at 17:01, Terry Reedy <tjreedy at udel.edu> wrote:
> > Ask the PSF/pypi people to either prohibit such names or require a
> > disclaimer of some sort.  They are inherently confusing: "I took a look
> at
> > pep008" does not mean that one even looked at the PEP.  Even when the
> > context makes clear that the referent is the module, there is confusion
> as
> > to its authoritativeness.  That Facudo would post here about the module's
> > output illustrates that. To me, the name copying violates our informal
> > trademark within Pythonland on 'PEP####'.
>
> I don't think that's the right answer, as opinionated tools do serve a
> useful purpose in preventing bikeshedding during code review (people
> *expect* computers to be annoyingly pedantic, which frees up the human
> reviewers to focus on higher level concerns). As projects evolve over
> time, they may develop their own tweaks and customisations in their
> style guide and switch to a more configurable tool, or they may not.
>
> When some of the default settings for the pep8 utility became a
> problem, I was able to talk to the developers and persuade them to
> tune their defaults to be more in line with the actual PEP text, and
> keep their extensions to optional settings.
>
> A similar approach may work for PEP 257, by clarifying which aspects
> tools should be leaving to human judgement (beyond the question of
> whether or not to opt in to following PEP 257 at all - it's far less
> universal than PEP 8).
>

Hm. I don't want the PSF to flex its muscles about trademarks, but I still
don't like that there are tools named after PEPs (especially since the
tools are not written by the same people that wrote the PEPs). I still
recall the first time someone emailed me about a "pep8" issue (I had never
heard of the tool by that name) and I was thoroughly confused for a long
time.

That said I expect it's too late to try and get the pep8 authors to rename
it; but I filed an issue with the pep257 project and they are going to
change the name: https://github.com/GreenSteam/pep257/issues/172 .

FWIW I am happy that the tools exist! They can be very useful and I use
pep8 myself. But I always let it know who's boss. :-)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160104/7a6fe821/attachment.html>

From mike.romberg at comcast.net  Mon Jan  4 13:27:42 2016
From: mike.romberg at comcast.net (mike.romberg at comcast.net)
Date: Mon, 4 Jan 2016 11:27:42 -0700
Subject: [Python-Dev] zipimport.c broken with implicit namespace packages
In-Reply-To: <CAP1=2W78KPWoPbKUZt+HpVgNF7k-9KgFrTWB4G9364cmpjjOkQ@mail.gmail.com>
References: <22152.9649.202823.766027@lrd.home.lan>
 <CAP1=2W7eDQNP8j3sxr=dqopRrwm2_rEQfgpn8iqGag-GiQY3fg@mail.gmail.com>
 <22152.20121.966791.318438@lrd.home.lan>
 <CAP7+vJJPxUObMxRKFCmkELq6FwUgid+NokgWhrWLmGK4Vbnaig@mail.gmail.com>
 <CAP1=2W7SB5JmyNB2Yj23iii80Y2td82npJTC0rBbQ94n6-1LHg@mail.gmail.com>
 <22152.45520.973803.741563@lrd.home.lan>
 <CAP1=2W78KPWoPbKUZt+HpVgNF7k-9KgFrTWB4G9364cmpjjOkQ@mail.gmail.com>
Message-ID: <22154.47518.523647.735239@lrd.home.lan>

>>>>> " " == Brett Cannon <brett at python.org> writes:

...

     > It's a reasonable thing to consider, but it would be better to
     > get zipimport fixed for you, then rewritten

  To that end, I've added a patch to the issue tracker:

https://bugs.python.org/issue17633

  My patch is issue17633-3.diff which builds upon issue17633-2.diff in
that it fixes an issue with the enumerated value used by find_loader
(find_loader was returning -1 which was not even a valid enumerated
value).

  I also expanded the test cases for zipimport to cover namespace
packages spread between multiple zip archives and zip archives and
real filesystems.

  Please let me know if there is anything else I can do.

Thanks,

Mike


From barry at python.org  Tue Jan  5 18:49:21 2016
From: barry at python.org (Barry Warsaw)
Date: Tue, 5 Jan 2016 18:49:21 -0500
Subject: [Python-Dev] PEP 9 - plaintext PEP format - is officially deprecated
Message-ID: <20160105184921.317ac5ec@limelight.wooz.org>

I don't think this will be at all controversial.  Brett suggested, and there
was no disagreement from the PEP editors, that plain text PEPs be deprecated.
reStructuredText is clearly a better format, and all recent PEP submissions
have been in reST for a while now anyway.

I am therefore withdrawing[*] PEP 9 and have made other appropriate changes to
make it clear that only PEP 12 format is acceptable going forward.  The PEP
editors will not be converting the legacy PEPs to reST, nor will we currently
be renaming the relevant PEP source files to end with ".rst" since there's too
much tooling that would have to change to do so.  However, if either task
really interests you, please get in touch with the PEP editors.

it-only-took-15-years-ly y'rs,
-Barry (on behalf of the PEP editors)

[*] Status: Withdrawn being about the only currently appropriate resolution
status for process PEPs.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160105/dd7901d3/attachment.sig>

From vadmium+py at gmail.com  Wed Jan  6 22:06:13 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 7 Jan 2016 03:06:13 +0000
Subject: [Python-Dev] Branches in which to fix the SSL tests
Message-ID: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>

Currently some SSL tests in the test suite are broken by a recent
certificate change at https://svn.python.org/; see
<https://bugs.python.org/issue25940> for the bug report. The tests are
broken when the test suite is run with the ?-unetwork? option enabled,
and most of the buildbots appear to be affected. (In 3.6 the tests
have temporarily been disabled as a workaround.) I have a simple patch
that subsitutes the old root certificate for the new which I would
like to commit, but I?m not sure which branches to apply it to, or
even which branches are open to normal maintainence and bug fixes.

According to Larry
<https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
branch should now be in security-fixes-only mode. However this branch
still seems to get a lot of non-security action, for example the most
recent bunch of changes were some work on the provisional ?pathlib?
module. So firstly I would like some clarification on the status of
3.4 and what its future is.

Secondly, I would normally say a fix for the test suite isn?t really
appropriate for the older security branches. But in the bug report,
Koobs specifically requested this be fixed in 3.4 and possibly earlier
branches as well. What do others think about this?

From guido at python.org  Wed Jan  6 23:17:10 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 6 Jan 2016 20:17:10 -0800
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
Message-ID: <CAP7+vJLB2KEU4_q3gMrjOt17SHF07_EcxdAVY-ochRg778YDMw@mail.gmail.com>

On Wed, Jan 6, 2016 at 7:06 PM, Martin Panter <vadmium+py at gmail.com> wrote:

> Currently some SSL tests in the test suite are broken by a recent
> certificate change at https://svn.python.org/; see
> <https://bugs.python.org/issue25940> for the bug report. The tests are
> broken when the test suite is run with the ?-unetwork? option enabled,
> and most of the buildbots appear to be affected. (In 3.6 the tests
> have temporarily been disabled as a workaround.) I have a simple patch
> that subsitutes the old root certificate for the new which I would
> like to commit, but I?m not sure which branches to apply it to, or
> even which branches are open to normal maintainence and bug fixes.
>
> According to Larry
> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> branch should now be in security-fixes-only mode. However this branch
> still seems to get a lot of non-security action, for example the most
> recent bunch of changes were some work on the provisional ?pathlib?
> module. So firstly I would like some clarification on the status of
> 3.4 and what its future is.
>

To me Larry's email mainly indicates that we're not going to do more binary
releases in the 3.4 branch. The work I did on pathlib is probably never
going to be released in that branch -- but since I merged it into 3.5 it's
not going to waste, and the effort was pretty minimal. (And people *could*
still pick it up from the source.)


> Secondly, I would normally say a fix for the test suite isn?t really
> appropriate for the older security branches. But in the bug report,
> Koobs specifically requested this be fixed in 3.4 and possibly earlier
> branches as well. What do others think about this?
>

It should definitely be fixed in 2.7, 3.5 and 3.6. If you want to do it in
3.4 too that sounds totally fine (it's just one extra merge).

Are there still any active buildbots for 3.4? The question is, why? I guess
one reason might be to ensure that if we *do* have to do an emergency
release of 3.4 (it's always a possibility), it would be easier if the tests
were all known to be passing, otherwise doing a release (even a source-only
release) would be a big hassle. OTOH if we don't make changes the tests
will generally not start failing -- the current issue notwithstanding. :-)
So if buildbots are a scarce resource it's fine to repurpose them for more
recent releases.

Oh, finally. I'm eager to answer this but I'm not actually the best
resource here -- I'm pretty rusty where it comes to our build and test
practices.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160106/cd9deab6/attachment.html>

From benjamin at python.org  Thu Jan  7 01:02:04 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 06 Jan 2016 22:02:04 -0800
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
Message-ID: <1452146524.2360668.485185474.2107A656@webmail.messagingengine.com>

We should fix hard-failing tests in all branches to which releases might
be made in the future. It's important to be able to run the rests
successfully before releasing.

On Wed, Jan 6, 2016, at 19:06, Martin Panter wrote:
> Currently some SSL tests in the test suite are broken by a recent
> certificate change at https://svn.python.org/; see
> <https://bugs.python.org/issue25940> for the bug report. The tests are
> broken when the test suite is run with the ?-unetwork? option enabled,
> and most of the buildbots appear to be affected. (In 3.6 the tests
> have temporarily been disabled as a workaround.) I have a simple patch
> that subsitutes the old root certificate for the new which I would
> like to commit, but I?m not sure which branches to apply it to, or
> even which branches are open to normal maintainence and bug fixes.
> 
> According to Larry
> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> branch should now be in security-fixes-only mode. However this branch
> still seems to get a lot of non-security action, for example the most
> recent bunch of changes were some work on the provisional ?pathlib?
> module. So firstly I would like some clarification on the status of
> 3.4 and what its future is.
> 
> Secondly, I would normally say a fix for the test suite isn?t really
> appropriate for the older security branches. But in the bug report,
> Koobs specifically requested this be fixed in 3.4 and possibly earlier
> branches as well. What do others think about this?
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

From nad at python.org  Thu Jan  7 01:52:17 2016
From: nad at python.org (Ned Deily)
Date: Thu, 7 Jan 2016 01:52:17 -0500
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CAP7+vJLB2KEU4_q3gMrjOt17SHF07_EcxdAVY-ochRg778YDMw@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <CAP7+vJLB2KEU4_q3gMrjOt17SHF07_EcxdAVY-ochRg778YDMw@mail.gmail.com>
Message-ID: <95858B31-AFC9-48FF-8735-55A13FE6CAD3@python.org>

On Jan 6, 2016, at 23:17, Guido van Rossum <guido at python.org> wrote:
> On Wed, Jan 6, 2016 at 7:06 PM, Martin Panter <vadmium+py at gmail.com> wrote:
> Currently some SSL tests in the test suite are broken by a recent
> certificate change at https://svn.python.org/; see
> <https://bugs.python.org/issue25940> for the bug report. The tests are
> broken when the test suite is run with the ?-unetwork? option enabled,
> and most of the buildbots appear to be affected. (In 3.6 the tests
> have temporarily been disabled as a workaround.) I have a simple patch
> that subsitutes the old root certificate for the new which I would
> like to commit, but I?m not sure which branches to apply it to, or
> even which branches are open to normal maintainence and bug fixes.
> 
> According to Larry
> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> branch should now be in security-fixes-only mode. However this branch
> still seems to get a lot of non-security action, for example the most
> recent bunch of changes were some work on the provisional ?pathlib?
> module. So firstly I would like some clarification on the status of
> 3.4 and what its future is.
> 
> To me Larry's email mainly indicates that we're not going to do more binary releases in the 3.4 branch. The work I did on pathlib is probably never going to be released in that branch -- but since I merged it into 3.5 it's not going to waste, and the effort was pretty minimal. (And people *could* still pick it up from the source.)
>  
> Secondly, I would normally say a fix for the test suite isn?t really
> appropriate for the older security branches. But in the bug report,
> Koobs specifically requested this be fixed in 3.4 and possibly earlier
> branches as well. What do others think about this?
> 
> It should definitely be fixed in 2.7, 3.5 and 3.6. If you want to do it in 3.4 too that sounds totally fine (it's just one extra merge).

The Developer's Guide describes current practices for backporting of fixes to security-fix branches (https://docs.python.org/devguide/devcycle.html#security-branches):

"The only changes made to a security branch are those fixing issues exploitable
by attackers such as crashes, privilege escalation and, optionally, other
issues such as denial of service attacks.  Any other changes are
**not** considered a security risk and thus not backported to a security branch."

Benjamin brings up a good point, though, about the importance of fixing hard-failures in the test suite.  I've added the following to the above paragraph in the guide:

"You should also consider fixing hard-failing tests in open security branches
since it is important to be able to run the tests successfully before releasing."

Also, I've tried to update the information in the Developer's Guide regarding branches and releases to match the current state of the world, e.g. 3.6 is the feature release under development, 3.5 and 2.7 are the current maintenance branches, and the current security-fix-only branches are 3.4, 3.3, and (for one more month) 3.2.  (The web site should update within the next day.)

Hope that helps!

--Ned

--
  Ned Deily
  nad at python.org -- []



From mal at egenix.com  Thu Jan  7 04:57:08 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 7 Jan 2016 10:57:08 +0100
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
Message-ID: <568E3674.3010203@egenix.com>

On 07.01.2016 04:06, Martin Panter wrote:
> Currently some SSL tests in the test suite are broken by a recent
> certificate change at https://svn.python.org/; see
> <https://bugs.python.org/issue25940> for the bug report. The tests are
> broken when the test suite is run with the ?-unetwork? option enabled,
> and most of the buildbots appear to be affected. (In 3.6 the tests
> have temporarily been disabled as a workaround.) I have a simple patch
> that subsitutes the old root certificate for the new which I would
> like to commit, but I?m not sure which branches to apply it to, or
> even which branches are open to normal maintainence and bug fixes.

As mentioned on the issue tracker I'm not convinced that your
patch is a good solution going forward. Rather than basing
the test on svn.python.org, which can change again in the
future, we should use the domain we have specifically
reserved for stdlib tests, namely pythontest.net and get this
set up in a way so that we can run SSL tests again.

I can help with getting SSL certificates for pythontest.net,
since I'm managing the PSF SSL certificates (well, trying to
keep those managed anyway ;-) - I was not involved in the
choice of new CA for svn.python.org):

https://wiki.python.org/psf/PSF%20SSL%20Certificates

Regarding which branches to apply the final fix, others have
already chimed in. Essentially, any branch which we'll need
to run tests on in the foreseeable future will need to be fixed,
since the change in svn.python.org's SSL setup (the expired
certificate which caused the tests to fail was replaced with
a new one from a different CA, again causing tests to fail)
affects all released test suites.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 07 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From vadmium+py at gmail.com  Thu Jan  7 06:36:54 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 7 Jan 2016 11:36:54 +0000
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <568E3674.3010203@egenix.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E3674.3010203@egenix.com>
Message-ID: <CA+eR4cGphOO8jEVQWRXLQGeUL+GNFj_A_RK8MWtv=mCadn7r4Q@mail.gmail.com>

On 7 January 2016 at 09:57, M.-A. Lemburg <mal at egenix.com> wrote:
> As mentioned on the issue tracker I'm not convinced that your
> patch is a good solution going forward. Rather than basing
> the test on svn.python.org, which can change again in the
> future, we should use the domain we have specifically
> reserved for stdlib tests, namely pythontest.net and get this
> set up in a way so that we can run SSL tests again.

I agree my first patch is not really sustainable; I only thought of it
as a temporary workaround.

If we want to change to pythontest.net, it seems there is already an
SSL certificate set up with https://self-signed.pythontest.net,
already used by other tests. If people want, I should be able to make
a straightforward patch to switch to that.

But I would prefer to go with something like my newer
local-server.v2.patch, which essentially switches to a local server
run for the duration of each test. It is just a question of finding
someone to review my changes.

> Regarding which branches to apply the final fix, others have
> already chimed in. Essentially, any branch which we'll need
> to run tests on in the foreseeable future will need to be fixed

Yes, it seems that is a reasonable conclusion. In this case that would
indeed mean all branches including 3.2. Thanks everyone for the input.

From dario670 at gmail.com  Thu Jan  7 03:14:38 2016
From: dario670 at gmail.com (Dario Roman)
Date: Thu, 7 Jan 2016 03:14:38 -0500
Subject: [Python-Dev] Build bots situations
Message-ID: <CAC86xE2zt4JSftezxUKodGs3H4oPzjjhYD8why+ipMddRkzXdw@mail.gmail.com>

I have develop few years ago a small net bit that could help , I was giving
my self into it but unpleasant to my life , that they want java I never
stood from them . It was develop to run that bit in python strictly but a
combination with servers and I net stream sockets I could fix the problem .

Nice to meet with you I consider your explanation as mentor you know the
industry I built the bot to work with the Alice project and the dictionary
recipients of sparkov  , it it function right know but feds and Police try
to hold the main program , when I find out I try to get online with the
webmaster and he could not pass the permit he neglected so fast , that for
me was conducting right .
Anyways take care I you want copy of my work send me email to
serocom13 at gmail.com,  I born in this code an still understanding what was
before , me my self being human and loves beers .

I would like to overcome your language and try it ti put your self in
another way of life , you are my mentor ....

Thanks for every thing ...

Serocom13 at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/8ce3edc0/attachment.html>

From larry at hastings.org  Thu Jan  7 11:32:44 2016
From: larry at hastings.org (Larry Hastings)
Date: Thu, 7 Jan 2016 11:32:44 -0500
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
Message-ID: <568E932C.4080509@hastings.org>



On 01/06/2016 10:06 PM, Martin Panter wrote:
> According to Larry
> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> branch should now be in security-fixes-only mode.

You assume correctly.

> However this branch
> still seems to get a lot of non-security action, for example the most
> recent bunch of changes were some work on the provisional ?pathlib?
> module.

I haven't looked at the changes you mention (I'm on a trip) but... what 
am I to do?  Beyond tagging all future 3.4 releases from a repository 
only I have write access to and never pulling from hg.python.org, I 
don't really have any power to enforce the condition.  I'm left trusting 
the better judgment of the Python core dev community.

I feel like a parent who's caught his kid reading after bedtime using a 
flashlight under the covers.  I mean, no, that's not what you should be 
doing right now.  But fixing bugs in Python is hardly the worst thing in 
the world.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/33ff8b4a/attachment.html>

From brett at python.org  Thu Jan  7 11:52:09 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 07 Jan 2016 16:52:09 +0000
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CA+eR4cGphOO8jEVQWRXLQGeUL+GNFj_A_RK8MWtv=mCadn7r4Q@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E3674.3010203@egenix.com>
 <CA+eR4cGphOO8jEVQWRXLQGeUL+GNFj_A_RK8MWtv=mCadn7r4Q@mail.gmail.com>
Message-ID: <CAP1=2W4W2Smu6LWJziXNEcLixhepnzJ7fsfbN_bX9yUY9xf_Gg@mail.gmail.com>

On Thu, 7 Jan 2016 at 03:37 Martin Panter <vadmium+py at gmail.com> wrote:

> On 7 January 2016 at 09:57, M.-A. Lemburg <mal at egenix.com> wrote:
> > As mentioned on the issue tracker I'm not convinced that your
> > patch is a good solution going forward. Rather than basing
> > the test on svn.python.org, which can change again in the
> > future, we should use the domain we have specifically
> > reserved for stdlib tests, namely pythontest.net and get this
> > set up in a way so that we can run SSL tests again.
>
> I agree my first patch is not really sustainable; I only thought of it
> as a temporary workaround.
>
> If we want to change to pythontest.net, it seems there is already an
> SSL certificate set up with https://self-signed.pythontest.net,
> already used by other tests. If people want, I should be able to make
> a straightforward patch to switch to that.
>
> But I would prefer to go with something like my newer
> local-server.v2.patch, which essentially switches to a local server
> run for the duration of each test. It is just a question of finding
> someone to review my changes.
>

I say update bugfix branches with simple pythontest.net and consider
updating default w/ your local server solution as there we can watch the
tests to make sure the test is stable, etc.

-Brett


>
> > Regarding which branches to apply the final fix, others have
> > already chimed in. Essentially, any branch which we'll need
> > to run tests on in the foreseeable future will need to be fixed
>
> Yes, it seems that is a reasonable conclusion. In this case that would
> indeed mean all branches including 3.2. Thanks everyone for the input.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/7fc8ddf8/attachment-0001.html>

From victor.stinner at gmail.com  Thu Jan  7 12:04:12 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 7 Jan 2016 18:04:12 +0100
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <568E932C.4080509@hastings.org>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E932C.4080509@hastings.org>
Message-ID: <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>

Hi,

2016-01-07 17:32 GMT+01:00 Larry Hastings <larry at hastings.org>:
> On 01/06/2016 10:06 PM, Martin Panter wrote:
>
> According to Larry
> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> branch should now be in security-fixes-only mode.

Would it be possible to have a (clear and up to date) table like
http://docs.openstack.org/releases/ in the Developer Guide? List of
Python versions with their status (end of life, security supported,
current stable release, under development).

Victor

From brett at python.org  Thu Jan  7 12:10:33 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 07 Jan 2016 17:10:33 +0000
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E932C.4080509@hastings.org>
 <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>
Message-ID: <CAP1=2W7L2fs=jKU3y-zifQxvJRax3Vddj-wqQ1Gy=E_Sn3ZcXw@mail.gmail.com>

On Thu, 7 Jan 2016 at 09:06 Victor Stinner <victor.stinner at gmail.com> wrote:

> Hi,
>
> 2016-01-07 17:32 GMT+01:00 Larry Hastings <larry at hastings.org>:
> > On 01/06/2016 10:06 PM, Martin Panter wrote:
> >
> > According to Larry
> > <https://mail.python.org/pipermail/python-dev/2015-December/142566.html
> >,
> > 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
> > branch should now be in security-fixes-only mode.
>
> Would it be possible to have a (clear and up to date) table like
> http://docs.openstack.org/releases/ in the Developer Guide? List of
> Python versions with their status (end of life, security supported,
> current stable release, under development).
>

Sure, someone just has to write it and then either make sure it stays
up-to-date or add it as another release step for RMs to follow.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/e54bafa5/attachment.html>

From mal at egenix.com  Thu Jan  7 12:56:41 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 7 Jan 2016 18:56:41 +0100
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CAP1=2W4W2Smu6LWJziXNEcLixhepnzJ7fsfbN_bX9yUY9xf_Gg@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E3674.3010203@egenix.com>
 <CA+eR4cGphOO8jEVQWRXLQGeUL+GNFj_A_RK8MWtv=mCadn7r4Q@mail.gmail.com>
 <CAP1=2W4W2Smu6LWJziXNEcLixhepnzJ7fsfbN_bX9yUY9xf_Gg@mail.gmail.com>
Message-ID: <568EA6D9.3070703@egenix.com>

On 07.01.2016 17:52, Brett Cannon wrote:
> On Thu, 7 Jan 2016 at 03:37 Martin Panter <vadmium+py at gmail.com> wrote:
> 
>> On 7 January 2016 at 09:57, M.-A. Lemburg <mal at egenix.com> wrote:
>>> As mentioned on the issue tracker I'm not convinced that your
>>> patch is a good solution going forward. Rather than basing
>>> the test on svn.python.org, which can change again in the
>>> future, we should use the domain we have specifically
>>> reserved for stdlib tests, namely pythontest.net and get this
>>> set up in a way so that we can run SSL tests again.
>>
>> I agree my first patch is not really sustainable; I only thought of it
>> as a temporary workaround.
>>
>> If we want to change to pythontest.net, it seems there is already an
>> SSL certificate set up with https://self-signed.pythontest.net,
>> already used by other tests. If people want, I should be able to make
>> a straightforward patch to switch to that.
>>
>> But I would prefer to go with something like my newer
>> local-server.v2.patch, which essentially switches to a local server
>> run for the duration of each test. It is just a question of finding
>> someone to review my changes.
>>
> 
> I say update bugfix branches with simple pythontest.net and consider
> updating default w/ your local server solution as there we can watch the
> tests to make sure the test is stable, etc.

+1

> -Brett
> 
> 
>>
>>> Regarding which branches to apply the final fix, others have
>>> already chimed in. Essentially, any branch which we'll need
>>> to run tests on in the foreseeable future will need to be fixed
>>
>> Yes, it seems that is a reasonable conclusion. In this case that would
>> indeed mean all branches including 3.2. Thanks everyone for the input.
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 07 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From nad at python.org  Thu Jan  7 14:38:43 2016
From: nad at python.org (Ned Deily)
Date: Thu, 7 Jan 2016 14:38:43 -0500
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E932C.4080509@hastings.org>
 <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>
Message-ID: <4D3394EC-9E9E-45B8-8C55-79C6965D2AE0@python.org>

On Jan 7, 2016, at 12:04, Victor Stinner <victor.stinner at gmail.com> wrote:
> 2016-01-07 17:32 GMT+01:00 Larry Hastings <larry at hastings.org>:
>> On 01/06/2016 10:06 PM, Martin Panter wrote:
>> 
>> According to Larry
>> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
>> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
>> branch should now be in security-fixes-only mode.
> 
> Would it be possible to have a (clear and up to date) table like
> http://docs.openstack.org/releases/ in the Developer Guide? List of
> Python versions with their status (end of life, security supported,
> current stable release, under development).

Yes, it's a good idea.  It's on my to-do list.

--
  Ned Deily
  nad at python.org -- []


From blake.a.griffith at gmail.com  Thu Jan  7 17:26:23 2016
From: blake.a.griffith at gmail.com (Blake Griffith)
Date: Thu, 7 Jan 2016 23:26:23 +0100
Subject: [Python-Dev] bitwise operations for bytes and bytearray
Message-ID: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>

Hi!

I'm interested in adding the functionality to do something like:

>>>  b'a' ^ b'b'
b'\x03'


Instead of the good ol' TypeError.

I think both bytes and bytearray should support all the bitwise operations.

I've never hacked on cpython before. I'm starting by just trying to add xor
to bytearray. I have a ByteArray_Xor function that I think should do this
here
https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f

But I'm not sure how to hook this in to the rest of cypython. I tried
adding it where bytearray_as_sequence is declared in this bytearrayobject.c
file. But that gave me compiler warnings and broke things.

So now that I have this ByteArray_Xor function, how do I make it be
bytearray.__xor___?

Thanks!

Blake
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/86243ac9/attachment.html>

From brett at python.org  Thu Jan  7 17:38:29 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 07 Jan 2016 22:38:29 +0000
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
References: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
Message-ID: <CAP1=2W5pp92ou58J=yBK=e7M3uux55XiSVSWEu=5+cFx2wVxMg@mail.gmail.com>

On Thu, 7 Jan 2016 at 14:29 Blake Griffith <blake.a.griffith at gmail.com>
wrote:

> Hi!
>
> I'm interested in adding the functionality to do something like:
>
> >>>  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.
>
> I've never hacked on cpython before. I'm starting by just trying to add
> xor to bytearray. I have a ByteArray_Xor function that I think should do
> this here
> https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f
>
> But I'm not sure how to hook this in to the rest of cypython. I tried
> adding it where bytearray_as_sequence is declared in this
> bytearrayobject.c file. But that gave me compiler warnings and broke things.
>
> So now that I have this ByteArray_Xor function, how do I make it be
> bytearray.__xor___?
>

You need to set the PyNumberMethods struct with the appropriate function
and then set that on the PyTypeObject. Look at
https://hg.python.org/cpython/file/tip/Include/object.h#l237 and
https://hg.python.org/cpython/file/tip/Objects/longobject.c#l5238 for an
idea of what it takes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/92a37da2/attachment.html>

From jjevnik at quantopian.com  Thu Jan  7 17:39:12 2016
From: jjevnik at quantopian.com (Joe Jevnik)
Date: Thu, 7 Jan 2016 17:39:12 -0500
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
References: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
Message-ID: <CAFo6DPZaf5zSuFiaW14p58F82k3OoJ=hd5b+SxpeXwDD8hwABg@mail.gmail.com>

You want to put the `xor` method in the `nb_xor` field of the
`PyNumberMethods` structure that lives in the `tp_as_number` field of the
bytes type object. Two things I noticed in a quick pass: you might want to
add some type checking around the case where `a` or `b` is not a
`PyByteArray` object. Also, variable length arrays are added in C99, so you
will need to manually allocate the `raw_*` arrays in the heap. This seems
like a cool feature!

On Thu, Jan 7, 2016 at 5:26 PM, Blake Griffith <blake.a.griffith at gmail.com>
wrote:

> Hi!
>
> I'm interested in adding the functionality to do something like:
>
> >>>  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.
>
> I've never hacked on cpython before. I'm starting by just trying to add
> xor to bytearray. I have a ByteArray_Xor function that I think should do
> this here
> https://github.com/cowlicks/cpython/commit/d6dddb11cdb33032b39dcb9dfdaa7b10d4377b5f
>
> But I'm not sure how to hook this in to the rest of cypython. I tried
> adding it where bytearray_as_sequence is declared in this
> bytearrayobject.c file. But that gave me compiler warnings and broke things.
>
> So now that I have this ByteArray_Xor function, how do I make it be
> bytearray.__xor___?
>
> Thanks!
>
> Blake
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/joe%40quantopian.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160107/377aaa7f/attachment.html>

From vadmium+py at gmail.com  Thu Jan  7 18:57:24 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 7 Jan 2016 23:57:24 +0000
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
References: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
Message-ID: <CA+eR4cFiB9cT=Lz3jHjo9jtg_0efmBKc1oCs==qGA3zx1xAerQ@mail.gmail.com>

On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com> wrote:
> I'm interested in adding the functionality to do something like:
>
>>>>  b'a' ^ b'b'
> b'\x03'
>
>
> Instead of the good ol' TypeError.
>
> I think both bytes and bytearray should support all the bitwise operations.

There is a bug open about adding this kind of functionality:
<https://bugs.python.org/issue19251>.

From abarnert at yahoo.com  Thu Jan  7 19:12:06 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 7 Jan 2016 16:12:06 -0800
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CA+eR4cFiB9cT=Lz3jHjo9jtg_0efmBKc1oCs==qGA3zx1xAerQ@mail.gmail.com>
References: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
 <CA+eR4cFiB9cT=Lz3jHjo9jtg_0efmBKc1oCs==qGA3zx1xAerQ@mail.gmail.com>
Message-ID: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>

On Jan 7, 2016, at 15:57, Martin Panter <vadmium+py at gmail.com> wrote:
> 
>> On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com> wrote:
>> I'm interested in adding the functionality to do something like:
>> 
>>>>> b'a' ^ b'b'
>> b'\x03'
>> 
>> 
>> Instead of the good ol' TypeError.
>> 
>> I think both bytes and bytearray should support all the bitwise operations.
> 
> There is a bug open about adding this kind of functionality:
> <https://bugs.python.org/issue19251>.

And it's in the needs patch stage, which makes it perfect for the OP: in addition to learning how to hack on builtin types, he can also learn the other parts of the dev process. (Even if the bug is eventually rejected, as seems likely given that it sat around for three years with no compelling use case  and then Guido added a "very skeptical" comment.)

From blake.a.griffith at gmail.com  Thu Jan  7 20:06:06 2016
From: blake.a.griffith at gmail.com (Blake Griffith)
Date: Fri, 8 Jan 2016 02:06:06 +0100
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
References: <CAOiFMpwZq76UgEJCANOgZsAfybrH5J4DkEkZ5ciS0VwQuYDa9A@mail.gmail.com>
 <CA+eR4cFiB9cT=Lz3jHjo9jtg_0efmBKc1oCs==qGA3zx1xAerQ@mail.gmail.com>
 <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
Message-ID: <CAOiFMpxMCHZZN5+cARFnX_UgPa298h-otCpOrW0U4Yy2d2Pg1A@mail.gmail.com>

Thanks for the quick responses y'all. I have something compiling on my
branch, which is enough for me tonight.

I asked a question about this on stackoverflow a while ago, it wasn't very
popular
https://stackoverflow.com/questions/32658420/why-cant-you-xor-bytes-objects-in-python

Someone there pointed out this feature was suggested on the mailing list a
while back (2006)
https://mail.python.org/pipermail/python-dev/2006-March/061980.html

On Fri, Jan 8, 2016 at 1:12 AM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Jan 7, 2016, at 15:57, Martin Panter <vadmium+py at gmail.com> wrote:
> >
> >> On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com>
> wrote:
> >> I'm interested in adding the functionality to do something like:
> >>
> >>>>> b'a' ^ b'b'
> >> b'\x03'
> >>
> >>
> >> Instead of the good ol' TypeError.
> >>
> >> I think both bytes and bytearray should support all the bitwise
> operations.
> >
> > There is a bug open about adding this kind of functionality:
> > <https://bugs.python.org/issue19251>.
>
> And it's in the needs patch stage, which makes it perfect for the OP: in
> addition to learning how to hack on builtin types, he can also learn the
> other parts of the dev process. (Even if the bug is eventually rejected, as
> seems likely given that it sat around for three years with no compelling
> use case  and then Guido added a "very skeptical" comment.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160108/37631f75/attachment-0001.html>

From cs at zip.com.au  Thu Jan  7 20:08:20 2016
From: cs at zip.com.au (Cameron Simpson)
Date: Fri, 8 Jan 2016 12:08:20 +1100
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
References: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
Message-ID: <20160108010820.GA6099@cskk.homeip.net>

On 07Jan2016 16:12, Python-Dev <python-dev at python.org> wrote:
>On Jan 7, 2016, at 15:57, Martin Panter <vadmium+py at gmail.com> wrote:
>>> On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com> wrote:
>>> I'm interested in adding the functionality to do something like:
>>>>>> b'a' ^ b'b'
>>> b'\x03'
>>> Instead of the good ol' TypeError.
>>>
>>> I think both bytes and bytearray should support all the bitwise operations.
>>
>> There is a bug open about adding this kind of functionality:
>> <https://bugs.python.org/issue19251>.
>
>And it's in the needs patch stage, which makes it perfect for the OP: in 
>addition to learning how to hack on builtin types, he can also learn the other 
>parts of the dev process. (Even if the bug is eventually rejected, as seems 
>likely given that it sat around for three years with no compelling use case  
>and then Guido added a "very skeptical" comment.)

The use case which springs immediately to my mind is cryptography. To encrypt a 
stream symmetrically you can go:

  cleartext-bytes ^ cryptographicly-random-bytes-from-cipher

so with this one could write:

  def crypted(byteses, crypto_source):
    ''' Accept an iterable source of bytes objects and a preprimed source of 
    crypto bytes, yield encrypted versions of the bytes objects.
    '''
    for bs in byteses:
      cbs = crypto_source.next_bytes(len(bs))
      yield bs ^ cbs

Cheers,
Cameron Simpson <cs at zip.com.au>

From ncoghlan at gmail.com  Fri Jan  8 07:45:39 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 8 Jan 2016 22:45:39 +1000
Subject: [Python-Dev] Branches in which to fix the SSL tests
In-Reply-To: <4D3394EC-9E9E-45B8-8C55-79C6965D2AE0@python.org>
References: <CA+eR4cFKw8B=2F=VrAf6ti-MorPpJs4F2irsUXYZBhvJ9sbOBQ@mail.gmail.com>
 <568E932C.4080509@hastings.org>
 <CAMpsgwZSSrLNpCVKqPhYhR-7pyofcna4n+or738e_vH2RHyLjQ@mail.gmail.com>
 <4D3394EC-9E9E-45B8-8C55-79C6965D2AE0@python.org>
Message-ID: <CADiSq7eDvhV2H+9war8z6P3Ki7ouw4DO_Vj2qvRKqnMyvV4sNQ@mail.gmail.com>

On 8 January 2016 at 05:38, Ned Deily <nad at python.org> wrote:
> On Jan 7, 2016, at 12:04, Victor Stinner <victor.stinner at gmail.com> wrote:
>> 2016-01-07 17:32 GMT+01:00 Larry Hastings <larry at hastings.org>:
>>> On 01/06/2016 10:06 PM, Martin Panter wrote:
>>>
>>> According to Larry
>>> <https://mail.python.org/pipermail/python-dev/2015-December/142566.html>,
>>> 3.4.4 was the last bug fix release for 3.4, so I assumed the 3.4
>>> branch should now be in security-fixes-only mode.
>>
>> Would it be possible to have a (clear and up to date) table like
>> http://docs.openstack.org/releases/ in the Developer Guide? List of
>> Python versions with their status (end of life, security supported,
>> current stable release, under development).
>
> Yes, it's a good idea.  It's on my to-do list.

PHP's support status page is one of the nicest examples of this I've
seen (someone mentioned it last time this question came up):
http://php.net/supported-versions.php

Like a lot of things though, "yes, that's a good idea" is easy, having
it actually get to the top of someone's todo list is a different
question :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From status at bugs.python.org  Fri Jan  8 12:08:35 2016
From: status at bugs.python.org (Python tracker)
Date: Fri,  8 Jan 2016 18:08:35 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160108170835.83A12566EA@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-01-01 - 2016-01-08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5371 (+30)
  closed 32436 (+34)
  total  37807 (+64)

Open issues with patches: 2362 


Issues opened (50)
==================

#6500: urllib2 maximum recursion depth exceeded
http://bugs.python.org/issue6500  reopened by berker.peksag

#22995: Restrict default pickleability
http://bugs.python.org/issue22995  reopened by barry

#25986: Collections.deque maxlen: added in 2.6 or 2.7?
http://bugs.python.org/issue25986  opened by terry.reedy

#25987: collections.abc.Reversible
http://bugs.python.org/issue25987  opened by abarnert

#25988: collections.abc.Indexable
http://bugs.python.org/issue25988  opened by abarnert

#25989: documentation version switcher is broken fro 2.6, 3.2, 3.3
http://bugs.python.org/issue25989  opened by Vincentdavis

#25991: readline example eventually consumes all memory
http://bugs.python.org/issue25991  opened by dangyogi

#25992: test_gdb fails
http://bugs.python.org/issue25992  opened by Bryce Miller

#25993: Crashed when call time.time() after using _mm_xor_si64
http://bugs.python.org/issue25993  opened by Xiongzhi Gao

#25994: File descriptor leaks in os.scandir()
http://bugs.python.org/issue25994  opened by serhiy.storchaka

#25995: os.walk() consumes a lot of file descriptors
http://bugs.python.org/issue25995  opened by serhiy.storchaka

#25996: Add support of file descriptor in os.scandir()
http://bugs.python.org/issue25996  opened by serhiy.storchaka

#25998: doctest terminates when accessing __wrapped__ raises an error
http://bugs.python.org/issue25998  opened by Lasse Schuirmann

#26000: Crash in Tokenizer - Heap-use-after-free
http://bugs.python.org/issue26000  opened by William Bowling

#26001: Tutorial: write() does not expect string in binary mode
http://bugs.python.org/issue26001  opened by Dimitri Papadopoulos Orfanos

#26002: make statistics.median_grouped more efficient
http://bugs.python.org/issue26002  opened by upendra-k14

#26003: Issues with PyEval_InitThreads and PyGILState_Ensure
http://bugs.python.org/issue26003  opened by tzickel

#26004: pip install lifetimes - throwing error and unable to install p
http://bugs.python.org/issue26004  opened by dudestc

#26005: Denial of Service in SimpleHTTPServer and BaseHTTPServer
http://bugs.python.org/issue26005  opened by Richard Clifford

#26007: Request for Support for Embedding the Standard Library in an E
http://bugs.python.org/issue26007  opened by philthompson10

#26009: HTMLParser lacking a few features to reconstruct input exactly
http://bugs.python.org/issue26009  opened by jason_s

#26010: document CO_* constants
http://bugs.python.org/issue26010  opened by yselivanov

#26011: Document necesities for cmp argument of sorted
http://bugs.python.org/issue26011  opened by krichter

#26013: Pickle protocol 2.0 not loading in python 3.5
http://bugs.python.org/issue26013  opened by anilredshift

#26014: Guide users to the newer package install instructions
http://bugs.python.org/issue26014  opened by ncoghlan

#26015: Add new tests for pickling iterators of mutable sequences
http://bugs.python.org/issue26015  opened by serhiy.storchaka

#26016: io.TextIOWrapper.tell() report 65bit number when mix readline(
http://bugs.python.org/issue26016  opened by EcmaXp_

#26017: Update https://docs.python.org/3/installing/index.html to alwa
http://bugs.python.org/issue26017  opened by brett.cannon

#26018: documentation of ZipFile file name encoding
http://bugs.python.org/issue26018  opened by gagern

#26019: collections.abc documentation incomplete
http://bugs.python.org/issue26019  opened by abarnert

#26020: set_display evaluation order doesn't match documented behaviou
http://bugs.python.org/issue26020  opened by Hamish Campbell

#26023: Missing signatures operator module
http://bugs.python.org/issue26023  opened by Freddy Rietdijk

#26024: Non-ascii Windows locale names
http://bugs.python.org/issue26024  opened by vidartf

#26025: Document pathlib.Path.__truediv__()
http://bugs.python.org/issue26025  opened by brett.cannon

#26027: Support Path objects in the posix module
http://bugs.python.org/issue26027  opened by serhiy.storchaka

#26029: Broken sentence in extending documentation
http://bugs.python.org/issue26029  opened by sizeof

#26030: Use PEP8 in documentation examples
http://bugs.python.org/issue26030  opened by sizeof

#26031: Add stat caching option to pathlib
http://bugs.python.org/issue26031  opened by gvanrossum

#26032: Use scandir() to speed up pathlib globbing
http://bugs.python.org/issue26032  opened by gvanrossum

#26033: distutils default compiler API is incomplete
http://bugs.python.org/issue26033  opened by stefan

#26034: venv documentation out of date
http://bugs.python.org/issue26034  opened by dsadowski

#26035: traceback.print_tb() takes `tb`, not `traceback` as a keyword 
http://bugs.python.org/issue26035  opened by Nicholas Chammas

#26036: Unnecessary arguments on smtpd.SMTPServer
http://bugs.python.org/issue26036  opened by sleepycal

#26037: Crash when reading sys.stdin.buffer in a daemon thread
http://bugs.python.org/issue26037  opened by eph ???

#26038: zipfile cannot handle zip files where the archive size for a f
http://bugs.python.org/issue26038  opened by Brett Rosen

#26039: More flexibility in zipfile interface
http://bugs.python.org/issue26039  opened by takluyver

#26040: Improve coverage and rigour of test.test_math
http://bugs.python.org/issue26040  opened by jeff.allen

#26041: Update deprecation messages of platform.dist() and platform.li
http://bugs.python.org/issue26041  opened by berker.peksag

#26045: Improve error message for http.client when posting unicode str
http://bugs.python.org/issue26045  opened by Emil Stenstr??m

#26049: Poor performance when reading large xmlrpc data
http://bugs.python.org/issue26049  opened by pokoli



Most recent 15 issues with no replies (15)
==========================================

#26041: Update deprecation messages of platform.dist() and platform.li
http://bugs.python.org/issue26041

#26040: Improve coverage and rigour of test.test_math
http://bugs.python.org/issue26040

#26038: zipfile cannot handle zip files where the archive size for a f
http://bugs.python.org/issue26038

#26034: venv documentation out of date
http://bugs.python.org/issue26034

#26033: distutils default compiler API is incomplete
http://bugs.python.org/issue26033

#26027: Support Path objects in the posix module
http://bugs.python.org/issue26027

#26025: Document pathlib.Path.__truediv__()
http://bugs.python.org/issue26025

#26018: documentation of ZipFile file name encoding
http://bugs.python.org/issue26018

#26015: Add new tests for pickling iterators of mutable sequences
http://bugs.python.org/issue26015

#26014: Guide users to the newer package install instructions
http://bugs.python.org/issue26014

#26003: Issues with PyEval_InitThreads and PyGILState_Ensure
http://bugs.python.org/issue26003

#26002: make statistics.median_grouped more efficient
http://bugs.python.org/issue26002

#25993: Crashed when call time.time() after using _mm_xor_si64
http://bugs.python.org/issue25993

#25991: readline example eventually consumes all memory
http://bugs.python.org/issue25991

#25989: documentation version switcher is broken fro 2.6, 3.2, 3.3
http://bugs.python.org/issue25989



Most recent 15 issues waiting for review (15)
=============================================

#26049: Poor performance when reading large xmlrpc data
http://bugs.python.org/issue26049

#26045: Improve error message for http.client when posting unicode str
http://bugs.python.org/issue26045

#26039: More flexibility in zipfile interface
http://bugs.python.org/issue26039

#26038: zipfile cannot handle zip files where the archive size for a f
http://bugs.python.org/issue26038

#26035: traceback.print_tb() takes `tb`, not `traceback` as a keyword 
http://bugs.python.org/issue26035

#26031: Add stat caching option to pathlib
http://bugs.python.org/issue26031

#26029: Broken sentence in extending documentation
http://bugs.python.org/issue26029

#26020: set_display evaluation order doesn't match documented behaviou
http://bugs.python.org/issue26020

#26019: collections.abc documentation incomplete
http://bugs.python.org/issue26019

#26017: Update https://docs.python.org/3/installing/index.html to alwa
http://bugs.python.org/issue26017

#26015: Add new tests for pickling iterators of mutable sequences
http://bugs.python.org/issue26015

#26013: Pickle protocol 2.0 not loading in python 3.5
http://bugs.python.org/issue26013

#26010: document CO_* constants
http://bugs.python.org/issue26010

#26002: make statistics.median_grouped more efficient
http://bugs.python.org/issue26002

#26001: Tutorial: write() does not expect string in binary mode
http://bugs.python.org/issue26001



Top 10 most discussed issues (10)
=================================

#25958: Implicit ABCs have no means of "anti-registration"
http://bugs.python.org/issue25958  25 msgs

#22570: Better stdlib support for Path objects
http://bugs.python.org/issue22570  17 msgs

#25596: regular files handled as directories in the glob module
http://bugs.python.org/issue25596  10 msgs

#26007: Request for Support for Embedding the Standard Library in an E
http://bugs.python.org/issue26007  10 msgs

#19251: bitwise ops for bytes of equal length
http://bugs.python.org/issue19251   9 msgs

#22995: Restrict default pickleability
http://bugs.python.org/issue22995   9 msgs

#25940: SSL tests failed due to expired svn.python.org SSL certificate
http://bugs.python.org/issue25940   9 msgs

#25864: collections.abc.Mapping should include a __reversed__ that rai
http://bugs.python.org/issue25864   7 msgs

#26032: Use scandir() to speed up pathlib globbing
http://bugs.python.org/issue26032   7 msgs

#6500: urllib2 maximum recursion depth exceeded
http://bugs.python.org/issue6500   6 msgs



Issues closed (33)
==================

#5501: Update multiprocessing docs re: freeze_support
http://bugs.python.org/issue5501  closed by berker.peksag

#16544: Add external link to ast docs
http://bugs.python.org/issue16544  closed by orsenthil

#18918: help('FILES') finds no documentation
http://bugs.python.org/issue18918  closed by orsenthil

#20440: Use the Py_SETREF macro
http://bugs.python.org/issue20440  closed by serhiy.storchaka

#20969: Author of EPUB version of Python docs is set to Unknown instea
http://bugs.python.org/issue20969  closed by orsenthil

#21221: Minor struct_time documentation bug
http://bugs.python.org/issue21221  closed by orsenthil

#21815: imaplib truncates some untagged responses
http://bugs.python.org/issue21815  closed by r.david.murray

#22709: restore accepting detached stdin in fileinput binary mode
http://bugs.python.org/issue22709  closed by r.david.murray

#24036: GB2312 codec is using a wrong covert table
http://bugs.python.org/issue24036  closed by lemburg

#24104: Use after free in xmlparser_setevents (2)
http://bugs.python.org/issue24104  closed by serhiy.storchaka

#24120: pathlib.(r)glob stops on PermissionDenied exception
http://bugs.python.org/issue24120  closed by gvanrossum

#24733: Logically Dead Code
http://bugs.python.org/issue24733  closed by orsenthil

#24898: Documentation for str.find() is confusing
http://bugs.python.org/issue24898  closed by orsenthil

#25637: Move non-collections-related ABCs out of collections.abc
http://bugs.python.org/issue25637  closed by brett.cannon

#25672: set SSL_MODE_RELEASE_BUFFERS
http://bugs.python.org/issue25672  closed by python-dev

#25813: co_flags section of inspect module docs out of date
http://bugs.python.org/issue25813  closed by BreamoreBoy

#25917: Fixing howto links in docs
http://bugs.python.org/issue25917  closed by orsenthil

#25990: Pydoc fails on Python file with nonlocal
http://bugs.python.org/issue25990  closed by serhiy.storchaka

#25997: Tarfile.add with bytes path is failing
http://bugs.python.org/issue25997  closed by Patrik Dufresne

#25999: Add support of negative number in bin()
http://bugs.python.org/issue25999  closed by larry

#26006: 32 bits python ctypes creates 64 bits process from 32 bits exe
http://bugs.python.org/issue26006  closed by Artur Korobeynyk

#26008: Different behaviour platform.linux_distribution() on Python2.7
http://bugs.python.org/issue26008  closed by eric.smith

#26012: pathlib.Path().rglob() is fooled by symlink loops
http://bugs.python.org/issue26012  closed by gvanrossum

#26021: Missing IPv6 support for pypi.python.org
http://bugs.python.org/issue26021  closed by dstufft

#26022: string.replace('  ',' ') has to be called 2 times before it wo
http://bugs.python.org/issue26022  closed by orsenthil

#26026: True%2 is True
http://bugs.python.org/issue26026  closed by SilentGhost

#26028: .sort() Causing Strings to Be Listed on the same line
http://bugs.python.org/issue26028  closed by SilentGhost

#26042: Consider dropping magic number for more detailed .pyc file nam
http://bugs.python.org/issue26042  closed by brett.cannon

#26043: ON DELETE CASCADE does not work when using sqlite3 library
http://bugs.python.org/issue26043  closed by Vitaminus Maximus

#26044: Name mangling overrides externally defined names
http://bugs.python.org/issue26044  closed by ethan.furman

#26046: Typo in documentation of unittest
http://bugs.python.org/issue26046  closed by python-dev

#26047: argparse.ArgumentError documentation wrong
http://bugs.python.org/issue26047  closed by SilentGhost

#26048: New user in community
http://bugs.python.org/issue26048  closed by ezio.melotti

From blake.a.griffith at gmail.com  Sat Jan  9 19:17:49 2016
From: blake.a.griffith at gmail.com (Blake Griffith)
Date: Sun, 10 Jan 2016 01:17:49 +0100
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <20160108010820.GA6099@cskk.homeip.net>
References: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
 <20160108010820.GA6099@cskk.homeip.net>
Message-ID: <CAOiFMpxN_yp7YK+-jdBxFPzgOSc1H5OcAS_Pa3X6WzLfLFaN8w@mail.gmail.com>

A little update, I got ^, &, and | working for bytearrays. You can view the
diff here:
https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1

How does it look?
Joe, is this how I should allocate the arrays? Am I freeing them properly?
Am I checking the input enough?

After some feedback, I'll probably add bitshifting and negation. Then work
on bytes objects.

Does this warrant a pep?

On Fri, Jan 8, 2016 at 2:08 AM, Cameron Simpson <cs at zip.com.au> wrote:

> On 07Jan2016 16:12, Python-Dev <python-dev at python.org> wrote:
>
>> On Jan 7, 2016, at 15:57, Martin Panter <vadmium+py at gmail.com> wrote:
>>
>>> On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com>
>>>> wrote:
>>>> I'm interested in adding the functionality to do something like:
>>>>
>>>>> b'a' ^ b'b'
>>>>>>>
>>>>>> b'\x03'
>>>> Instead of the good ol' TypeError.
>>>>
>>>> I think both bytes and bytearray should support all the bitwise
>>>> operations.
>>>>
>>>
>>> There is a bug open about adding this kind of functionality:
>>> <https://bugs.python.org/issue19251>.
>>>
>>
>> And it's in the needs patch stage, which makes it perfect for the OP: in
>> addition to learning how to hack on builtin types, he can also learn the
>> other parts of the dev process. (Even if the bug is eventually rejected, as
>> seems likely given that it sat around for three years with no compelling
>> use case  and then Guido added a "very skeptical" comment.)
>>
>
> The use case which springs immediately to my mind is cryptography. To
> encrypt a stream symmetrically you can go:
>
>  cleartext-bytes ^ cryptographicly-random-bytes-from-cipher
>
> so with this one could write:
>
>  def crypted(byteses, crypto_source):
>    ''' Accept an iterable source of bytes objects and a preprimed source
> of    crypto bytes, yield encrypted versions of the bytes objects.
>    '''
>    for bs in byteses:
>      cbs = crypto_source.next_bytes(len(bs))
>      yield bs ^ cbs
>
> Cheers,
> Cameron Simpson <cs at zip.com.au>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/blake.a.griffith%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160110/85a9c9cb/attachment.html>

From abarnert at yahoo.com  Sat Jan  9 19:31:58 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 9 Jan 2016 16:31:58 -0800
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CAOiFMpxN_yp7YK+-jdBxFPzgOSc1H5OcAS_Pa3X6WzLfLFaN8w@mail.gmail.com>
References: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
 <20160108010820.GA6099@cskk.homeip.net>
 <CAOiFMpxN_yp7YK+-jdBxFPzgOSc1H5OcAS_Pa3X6WzLfLFaN8w@mail.gmail.com>
Message-ID: <3C8E1B3A-1282-4932-B2B0-2D6FF51298D2@yahoo.com>

On Jan 9, 2016, at 16:17, Blake Griffith <blake.a.griffith at gmail.com> wrote:
> 
> A little update, I got ^, &, and | working for bytearrays. You can view the diff here:
> https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1

If you upload the diff to the issue on the tracker, the reitveld code review app should be able to pick it up automatically, allowing people to comment on it inline, in a much nicer format than a mailing list thread. It's especially nice if you're adding things in stages--people who have been following along can just look at the changes between patch 3 and 4, while new people can look at all the changes in one go, etc.

> How does it look? 
> Joe, is this how I should allocate the arrays? Am I freeing them properly?
> Am I checking the input enough?
> 
> After some feedback, I'll probably add bitshifting and negation. Then work on bytes objects.
> 
> Does this warrant a pep?

Personally, I'd just make the case for the feature on the tracker issue. If one of the core devs thinks it needs a PEP, or further discussion on this list or -ideas, they'll say so there.

At present, it seems like there's not much support for the idea, but I think that's at least partly because people want to see realistic use cases (that aren't server better by the existing bitarray/bitstring/etc. modules on PyPI, or using a NumPy array, or just using ints, etc.).

>> On Fri, Jan 8, 2016 at 2:08 AM, Cameron Simpson <cs at zip.com.au> wrote:
>>> On 07Jan2016 16:12, Python-Dev <python-dev at python.org> wrote:
>>> On Jan 7, 2016, at 15:57, Martin Panter <vadmium+py at gmail.com> wrote:
>>>>> On 7 January 2016 at 22:26, Blake Griffith <blake.a.griffith at gmail.com> wrote:
>>>>> I'm interested in adding the functionality to do something like:
>>>>>>>> b'a' ^ b'b'
>>>>> b'\x03'
>>>>> Instead of the good ol' TypeError.
>>>>> 
>>>>> I think both bytes and bytearray should support all the bitwise operations.
>>>> 
>>>> There is a bug open about adding this kind of functionality:
>>>> <https://bugs.python.org/issue19251>.
>>> 
>>> And it's in the needs patch stage, which makes it perfect for the OP: in addition to learning how to hack on builtin types, he can also learn the other parts of the dev process. (Even if the bug is eventually rejected, as seems likely given that it sat around for three years with no compelling use case  and then Guido added a "very skeptical" comment.)
>> 
>> The use case which springs immediately to my mind is cryptography. To encrypt a stream symmetrically you can go:
>> 
>>  cleartext-bytes ^ cryptographicly-random-bytes-from-cipher
>> 
>> so with this one could write:
>> 
>>  def crypted(byteses, crypto_source):
>>    ''' Accept an iterable source of bytes objects and a preprimed source of    crypto bytes, yield encrypted versions of the bytes objects.
>>    '''
>>    for bs in byteses:
>>      cbs = crypto_source.next_bytes(len(bs))
>>      yield bs ^ cbs
>> 
>> Cheers,
>> Cameron Simpson <cs at zip.com.au>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/blake.a.griffith%40gmail.com
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160109/4fb53fca/attachment.html>

From vadmium+py at gmail.com  Sat Jan  9 22:21:37 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Sun, 10 Jan 2016 03:21:37 +0000
Subject: [Python-Dev] bitwise operations for bytes and bytearray
In-Reply-To: <CAOiFMpxN_yp7YK+-jdBxFPzgOSc1H5OcAS_Pa3X6WzLfLFaN8w@mail.gmail.com>
References: <6E0FFB72-142F-493C-BE29-D260302EBA16@yahoo.com>
 <20160108010820.GA6099@cskk.homeip.net>
 <CAOiFMpxN_yp7YK+-jdBxFPzgOSc1H5OcAS_Pa3X6WzLfLFaN8w@mail.gmail.com>
Message-ID: <CA+eR4cFYFnYKdHFLDO+5xwvTb+kNWc_HXdwRZNNMhvzZJ_rh5w@mail.gmail.com>

On 10 January 2016 at 00:17, Blake Griffith <blake.a.griffith at gmail.com> wrote:
> A little update, I got ^, &, and | working for bytearrays. You can view the
> diff here:
> https://github.com/python/cpython/compare/master...cowlicks:bitwise-bytes?expand=1
>
> How does it look?

I left some comments against your commits on Git Hub. Unfortunately
they seem to be made in the base Python repository, rather than your
fork, which I did not expect.

> Joe, is this how I should allocate the arrays? Am I freeing them properly?
> Am I checking the input enough?
>
> After some feedback, I'll probably add bitshifting and negation. Then work
> on bytes objects.
>
> Does this warrant a pep?

From brett at python.org  Sun Jan 10 12:43:48 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 10 Jan 2016 17:43:48 +0000
Subject: [Python-Dev] GitHub migration planning has started
Message-ID: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>

For those of you who have not heard, I made the decision a little over a
week ago to move Python's development from our home-grown workflow to one
hosted on GitHub (mainly for code hosting and code review; we're keeping
bugs.python.org for our issue tracker). The hope is that this will let core
developers work through patches faster so that we have a better turn-around
time while being at least as good as our current workflow for external
contributors (but I will be shocked if it isn't better). There are also
people involved with the migration who plan to put in the effort to make
sure external contributors can still submit patches without ever
interacting with GitHub.

If you want to help with the transition, then feel free to join the
core-workflow mailing list where all the discussions on the details of the
migration are occurring (including the PEP I'm starting to write to outline
the steps we will be taking):
https://mail.python.org/mailman/listinfo/core-workflow
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160110/f88d397b/attachment.html>

From rymg19 at gmail.com  Sun Jan 10 12:48:37 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Sun, 10 Jan 2016 11:48:37 -0600
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
Message-ID: <4896FEAF-63DE-4E49-8271-34CF7BB9AAFF@gmail.com>

Is it possible to contribute to this, even if you're not part of the core dev team?

On January 10, 2016 11:43:48 AM CST, Brett Cannon <brett at python.org> wrote:
>For those of you who have not heard, I made the decision a little over
>a
>week ago to move Python's development from our home-grown workflow to
>one
>hosted on GitHub (mainly for code hosting and code review; we're
>keeping
>bugs.python.org for our issue tracker). The hope is that this will let
>core
>developers work through patches faster so that we have a better
>turn-around
>time while being at least as good as our current workflow for external
>contributors (but I will be shocked if it isn't better). There are also
>people involved with the migration who plan to put in the effort to
>make
>sure external contributors can still submit patches without ever
>interacting with GitHub.
>
>If you want to help with the transition, then feel free to join the
>core-workflow mailing list where all the discussions on the details of
>the
>migration are occurring (including the PEP I'm starting to write to
>outline
>the steps we will be taking):
>https://mail.python.org/mailman/listinfo/core-workflow
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160110/3eb8dc0a/attachment.html>

From brett at python.org  Sun Jan 10 12:59:45 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 10 Jan 2016 17:59:45 +0000
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <4896FEAF-63DE-4E49-8271-34CF7BB9AAFF@gmail.com>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
 <4896FEAF-63DE-4E49-8271-34CF7BB9AAFF@gmail.com>
Message-ID: <CAP1=2W4=rHuZ3sFxUies=9SudhxMgNEq2QUb7C6yWMy_mzh5fw@mail.gmail.com>

On Sun, 10 Jan 2016 at 09:48 Ryan Gonzalez <rymg19 at gmail.com> wrote:

> Is it possible to contribute to this, even if you're not part of the core
> dev team?
>

Sure! There's going to be plenty of code to write, decisions to be made,
etc. While I will making most of the final decisions, I will be asking for
feedback from people, asking some to go off and write some code, etc.

-Brett


>
> On January 10, 2016 11:43:48 AM CST, Brett Cannon <brett at python.org>
> wrote:
>
>> For those of you who have not heard, I made the decision a little over a
>> week ago to move Python's development from our home-grown workflow to one
>> hosted on GitHub (mainly for code hosting and code review; we're keeping
>> bugs.python.org for our issue tracker). The hope is that this will let
>> core developers work through patches faster so that we have a better
>> turn-around time while being at least as good as our current workflow for
>> external contributors (but I will be shocked if it isn't better). There are
>> also people involved with the migration who plan to put in the effort to
>> make sure external contributors can still submit patches without ever
>> interacting with GitHub.
>>
>> If you want to help with the transition, then feel free to join the
>> core-workflow mailing list where all the discussions on the details of the
>> migration are occurring (including the PEP I'm starting to write to outline
>> the steps we will be taking):
>> https://mail.python.org/mailman/listinfo/core-workflow
>>
>> ------------------------------
>>
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>>
>>
> --
> Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160110/3fdc5e76/attachment.html>

From stephane at wirtel.be  Sun Jan 10 15:36:42 2016
From: stephane at wirtel.be (Stephane Wirtel)
Date: Sun, 10 Jan 2016 21:36:42 +0100
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
Message-ID: <20160110203642.GD28792@sg1>

On 01/10, Brett Cannon wrote:
> For those of you who have not heard, I made the decision a little over a
> week ago to move Python's development from our home-grown workflow to one
> hosted on GitHub (mainly for code hosting and code review; we're keeping
> bugs.python.org for our issue tracker). The hope is that this will let core
> developers work through patches faster so that we have a better turn-around
> time while being at least as good as our current workflow for external
> contributors (but I will be shocked if it isn't better). There are also
> people involved with the migration who plan to put in the effort to make
> sure external contributors can still submit patches without ever
> interacting with GitHub.
> 
> If you want to help with the transition, then feel free to join the
> core-workflow mailing list where all the discussions on the details of the
> migration are occurring (including the PEP I'm starting to write to outline
> the steps we will be taking):
> https://mail.python.org/mailman/listinfo/core-workflow

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be

\o/

Already on core-workflow.

-- 
St?phane Wirtel - http://wirtel.be - @matrixise

From tjreedy at udel.edu  Sun Jan 10 22:52:49 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 10 Jan 2016 22:52:49 -0500
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
Message-ID: <n6v8uk$tm1$1@ger.gmane.org>

On 1/10/2016 12:43 PM, Brett Cannon wrote:
> For those of you who have not heard, I made the decision a little over a
> week ago to move Python's development from our home-grown workflow to
> one hosted on GitHub (mainly for code hosting and code review; we're
> keeping bugs.python.org <http://bugs.python.org> for our issue tracker).
> The hope is that this will let core developers work through patches
> faster so that we have a better turn-around time while being at least as
> good as our current workflow for external contributors (but I will be
> shocked if it isn't better). There are also people involved with the
> migration who plan to put in the effort to make sure external
> contributors can still submit patches without ever interacting with GitHub.
>
> If you want to help with the transition, then feel free to join the
> core-workflow mailing list where all the discussions on the details of
> the migration are occurring (including the PEP I'm starting to write to
> outline the steps we will be taking):
> https://mail.python.org/mailman/listinfo/core-workflow

Is there a gmane mirror, or do you think this is too limited (and 
temporary) for that?

-- 
Terry Jan Reedy


From brett at python.org  Mon Jan 11 00:44:04 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 11 Jan 2016 05:44:04 +0000
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <n6v8uk$tm1$1@ger.gmane.org>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
 <n6v8uk$tm1$1@ger.gmane.org>
Message-ID: <CAP1=2W6qUZM07bJXgJo=kDq9j8gM+Vcfp+ZjQ68S7TLXoKJa=w@mail.gmail.com>

On Sun, 10 Jan 2016 at 19:53 Terry Reedy <tjreedy at udel.edu> wrote:

> On 1/10/2016 12:43 PM, Brett Cannon wrote:
> > For those of you who have not heard, I made the decision a little over a
> > week ago to move Python's development from our home-grown workflow to
> > one hosted on GitHub (mainly for code hosting and code review; we're
> > keeping bugs.python.org <http://bugs.python.org> for our issue tracker).
> > The hope is that this will let core developers work through patches
> > faster so that we have a better turn-around time while being at least as
> > good as our current workflow for external contributors (but I will be
> > shocked if it isn't better). There are also people involved with the
> > migration who plan to put in the effort to make sure external
> > contributors can still submit patches without ever interacting with
> GitHub.
> >
> > If you want to help with the transition, then feel free to join the
> > core-workflow mailing list where all the discussions on the details of
> > the migration are occurring (including the PEP I'm starting to write to
> > outline the steps we will be taking):
> > https://mail.python.org/mailman/listinfo/core-workflow
>
> Is there a gmane mirror, or do you think this is too limited (and
> temporary) for that?
>

I have no idea if a gmane mirror was set up.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160111/2e6c4840/attachment.html>

From vadmium+py at gmail.com  Mon Jan 11 00:58:38 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Mon, 11 Jan 2016 05:58:38 +0000
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <n6v8uk$tm1$1@ger.gmane.org>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
 <n6v8uk$tm1$1@ger.gmane.org>
Message-ID: <CA+eR4cHVUsRg4h8qSyUkjyB7HZjQBzT+S2vNcq2QR7OVHvA8nw@mail.gmail.com>

On 11 January 2016 at 03:52, Terry Reedy <tjreedy at udel.edu> wrote:
> On 1/10/2016 12:43 PM, Brett Cannon wrote:
>> If you want to help with the transition, then feel free to join the
>> core-workflow mailing list where all the discussions on the details of
>> the migration are occurring (including the PEP I'm starting to write to
>> outline the steps we will be taking):
>> https://mail.python.org/mailman/listinfo/core-workflow
>
>
> Is there a gmane mirror, or do you think this is too limited (and temporary)
> for that?

I guess this is it: <http://dir.gmane.org/gmane.comp.python.devel.core-workflow>

From tjreedy at udel.edu  Mon Jan 11 01:46:45 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 11 Jan 2016 01:46:45 -0500
Subject: [Python-Dev] GitHub migration planning has started
In-Reply-To: <CA+eR4cHVUsRg4h8qSyUkjyB7HZjQBzT+S2vNcq2QR7OVHvA8nw@mail.gmail.com>
References: <CAP1=2W5imOQf-irGScn_YehY1+4pb7wYxdvybp8Vxv5L9AKcgw@mail.gmail.com>
 <n6v8uk$tm1$1@ger.gmane.org>
 <CA+eR4cHVUsRg4h8qSyUkjyB7HZjQBzT+S2vNcq2QR7OVHvA8nw@mail.gmail.com>
Message-ID: <n6vj4o$ge0$1@ger.gmane.org>

On 1/11/2016 12:58 AM, Martin Panter wrote:
> On 11 January 2016 at 03:52, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 1/10/2016 12:43 PM, Brett Cannon wrote:
>>> If you want to help with the transition, then feel free to join the
>>> core-workflow mailing list where all the discussions on the details of
>>> the migration are occurring (including the PEP I'm starting to write to
>>> outline the steps we will be taking):
>>> https://mail.python.org/mailman/listinfo/core-workflow
>>
>>
>> Is there a gmane mirror, or do you think this is too limited (and temporary)
>> for that?
>
> I guess this is it: <http://dir.gmane.org/gmane.comp.python.devel.core-workflow>

Thanks.  When I right-click news.gmane.org (in Thunderbird) and select 
subscribe, it does not appear in the 'Current groups' tab.  I just 
discovered that there is also a 'New groups' tab that has it.

-- 
Terry Jan Reedy


From barry at python.org  Mon Jan 11 10:10:41 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 11 Jan 2016 10:10:41 -0500
Subject: [Python-Dev] [Python-ideas] PEP 9 - plaintext PEP format - is
 officially deprecated
In-Reply-To: <CAPkN8xLb_dVveD4nu6o4GwXTK6vE_wqA07fdPdqvHdV02J3t-A@mail.gmail.com>
References: <20160105184921.317ac5ec@limelight.wooz.org>
 <CAPkN8xLb_dVveD4nu6o4GwXTK6vE_wqA07fdPdqvHdV02J3t-A@mail.gmail.com>
Message-ID: <20160111101041.1ba6177d@limelight.wooz.org>

On Jan 11, 2016, at 03:25 PM, anatoly techtonik wrote:

>On Wed, Jan 6, 2016 at 2:49 AM, Barry Warsaw <barry at python.org> wrote:
>
>> reStructuredText is clearly a better format
>
>Can you expand on that? I use markdown everywhere

reST is better than plain text.  Markdown is not a PEP format option.

>> all recent PEP submissions have been in reST for a while now anyway.
>
>Is it possible to query exact numbers automatically?

Feel free to grep the PEPs hg repo.

>What is the tooling support for handling PEP 9 and PEP 12?

UTSL.  Everything is in the PEPs hg repo.

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160111/4b7be7bc/attachment.sig>

From victor.stinner at gmail.com  Mon Jan 11 11:49:19 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 11 Jan 2016 17:49:19 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
Message-ID: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>

Hi,

After a first round on python-ideas, here is the second version of my
PEP. The main changes since the first version are that the dictionary
version is no more exposed at the Python level and the field type now
also has a size of 64-bit on 32-bit platforms.

The PEP is part of a serie of 3 PEP adding an API to implement a
static Python optimizer specializing functions with guards. The second
PEP is currently discussed on python-ideas and I'm still working on
the third PEP.

Thanks to Red Hat for giving me time to experiment on this.


HTML version:
https://www.python.org/dev/peps/pep-0509/


PEP: 509
Title: Add a private version to dict
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 4-January-2016
Python-Version: 3.6


Abstract
========

Add a new private version to builtin ``dict`` type, incremented at each
change, to implement fast guards on namespaces.


Rationale
=========

In Python, the builtin ``dict`` type is used by many instructions. For
example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
global namespace, or in the builtins namespace (two dict lookups).
Python uses ``dict`` for the builtins namespace, globals namespace, type
namespaces, instance namespaces, etc. The local namespace (namespace of
a function) is usually optimized to an array, but it can be a dict too.

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implementing optimizations respecting the Python
semantics requires to detect when "something changes": we will call
these checks "guards".

The speedup of optimizations depends on the speed of guard checks. This
PEP proposes to add a version to dictionaries to implement fast guards
on namespaces.

Dictionary lookups can be skipped if the version does not change which
is the common case for most namespaces. The performance of a guard does
not depend on the number of watched dictionary entries, complexity of
O(1), if the dictionary version does not change.

Example of optimization: copy the value of a global variable to function
constants.  This optimization requires a guard on the global variable to
check if it was modified. If the variable is modified, the variable must
be loaded at runtime when the function is called, instead of using the
constant.

See the `PEP 510 -- Specialized functions with guards
<https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of
guards to specialize functions and for the rationale on Python static
optimizers.


Guard example
=============

Pseudo-code of an fast guard to check if a dictionary entry was modified
(created, updated or deleted) using an hypothetical
``dict_get_version(dict)`` function::

    UNSET = object()

    class GuardDictKey:
        def __init__(self, dict, key):
            self.dict = dict
            self.key = key
            self.value = dict.get(key, UNSET)
            self.version = dict_get_version(dict)

        def check(self):
            """Return True if the dictionary entry did not changed."""

            # read the version field of the dict structure
            version = dict_get_version(self.dict)
            if version == self.version:
                # Fast-path: dictionary lookup avoided
                return True

            # lookup in the dictionary
            value = self.dict.get(self.key, UNSET)
            if value is self.value:
                # another key was modified:
                # cache the new dictionary version
                self.version = version
                return True

            # the key was modified
            return False


Usage of the dict version
=========================

Specialized functions using guards
----------------------------------

The `PEP 510 -- Specialized functions with guards
<https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support
specialized functions with guards. It allows to implement static
optimizers for Python without breaking the Python semantics.

Example of a static Python optimizer: the astoptimizer of the `FAT
Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
implements many optimizations which require guards on namespaces.
Examples:

* Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
  ``builtins.__dict__['len']`` and ``globals()['len']`` are required
* Loop unrolling: to unroll the loop ``for i in range(...): ...``,
  guards on ``builtins.__dict__['range']`` and ``globals()['range']``
  are required


Pyjion
------

According of Brett Cannon, one of the two main developers of Pyjion,
Pyjion can also benefit from dictionary version to implement
optimizations.

Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
Core runtime).


Unladen Swallow
---------------

Even if dictionary version was not explicitly mentionned, optimization
globals and builtins lookup was part of the Unladen Swallow plan:
"Implement one of the several proposed schemes for speeding lookups of
globals and builtins." Source: `Unladen Swallow ProjectPlan
<https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_.

Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
implemented with LLVM. The project stopped in 2011: `Unladen Swallow
Retrospective
<http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_.


Changes
=======

Add a ``ma_version`` field to the ``PyDictObject`` structure with the C
type ``PY_INT64_T``, 64-bit unsigned integer. New empty dictionaries are
initilized to version ``0``. The version is incremented at each change:

* ``clear()`` if the dict was non-empty
* ``pop(key)`` if the key exists
* ``popitem()`` if the dict is non-empty
* ``setdefault(key, value)`` if the `key` does not exist
* ``__detitem__(key)`` if the key exists
* ``__setitem__(key, value)`` if the `key` doesn't exist or if the value
  is different
* ``update(...)`` if new values are different than existing values (the
  version can be incremented multiple times)

Example using an hypothetical ``dict_get_version(dict)`` function::

    >>> d = {}
    >>> dict_get_version(d)
    0
    >>> d['key'] = 'value'
    >>> dict_get_version(d)
    1
    >>> d['key'] = 'new value'
    >>> dict_get_version(d)
    2
    >>> del d['key']
    >>> dict_get_version(d)
    3

If a dictionary is created with items, the version is also incremented
at each dictionary insertion. Example::

    >>> d=dict(x=7, y=33)
    >>> dict_get_version(d)
    2

The version is not incremented if an existing key is set to the same
value. For efficiency, values are compared by their identity:
``new_value is old_value``, not by their content:
``new_value == old_value``. Example::

    >>> d={}
    >>> value = object()
    >>> d['key'] = value
    >>> dict_get_version(d)
    2
    >>> d['key'] = value
    >>> dict_get_version(d)
    2

.. note::
   CPython uses some singleton like integers in the range [-5; 257],
   empty tuple, empty strings, Unicode strings of a single character in
   the range [U+0000; U+00FF], etc. When a key is set twice to the same
   singleton, the version is not modified.


Implementation
==============

The `issue #26058: PEP 509: Add ma_version to PyDictObject
<https://bugs.python.org/issue26058>`_ contains a patch implementing
this PEP.

On pybench and timeit microbenchmarks, the patch does not seem to add
any overhead on dictionary operations.

When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for
a dictioanry lookup, whereas a guard check only takes 3.8 ns. Moreover,
a guard can watch for multiple keys. For example, for an optimization
using 10 global variables in a function, 10 dictionary lookups costs 148
ns, whereas the guard still only costs 3.8 ns when the version does not
change (39x as fast).


Integer overflow
================

The implementation uses the C unsigned integer type ``PY_INT64_T`` to
store the version, a 64 bits unsigned integer. The C code uses
``version++``. On integer overflow, the version is wrapped to ``0`` (and
then continue to be incremented) according to the C standard.

After an integer overflow, a guard can succeed whereas the watched
dictionary key was modified. The bug occurs if the dictionary is
modified at least ``2 ** 64`` times between two checks of the guard and
if the new version (theorical value with no integer overflow) is equal
to the old version modulo ``2 ** 64``.

If a dictionary is modified each nanosecond, an overflow takes longer
than 584 years. Using a 32-bit version, the overflow occurs only after 4
seconds. That's why a 64-bit unsigned type is also used on 32-bit
systems. A dictionary lookup at the C level takes 14.8 ns.

A risk of a bug every 584 years is acceptable.


Alternatives
============

Expose the version at Python level as a read-only __version__ property
----------------------------------------------------------------------

The first version of the PEP proposed to expose the dictionary version
as a read-only ``__version__`` property at Python level, and also to add
the property to ``collections.UserDict`` (since this type must mimick
the ``dict`` API).

There are multiple issues:

* To be consistent and avoid bad surprises, the version must be added to
  all mapping types. Implementing a new mapping type would require extra
  work for no benefit, since the version is only required on the
  ``dict`` type in practice.
* All Python implementations must implement this new property, it gives
  more work to other implementations, whereas they may not use the
  dictionary version at all.
* The ``__version__`` can be wrapped on integer overflow. It is error
  prone: using ``dict.__version__ <= guard_version`` is wrong,
  ``dict.__version__ == guard_version`` must be used instead to reduce
  the risk of bug on integer overflow (even if the integer overflow is
  unlikely in practice).
* Exposing the dictionary version at Python level can lead the
  false assumption on performances. Checking ``dict.__version__`` at
  the Python level is not faster than a dictionary lookup. A dictionary
  lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5
  ns, the difference is only 1.2 ns (3%)::


    $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33'
    10000000 loops, best of 3: 0.0487 usec per loop
    $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
'd.__version__ == 100'
    10000000 loops, best of 3: 0.0475 usec per loop

Bikeshedding on the property name:

* ``__cache_token__``: name proposed by Nick Coghlan, name coming from
  `abc.get_cache_token()
  <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_.
* ``__version__``
* ``__timestamp__``


Add a version to each dict entry
--------------------------------

A single version per dictionary requires to keep a strong reference to
the value which can keep the value alive longer than expected. If we add
also a version per dictionary entry, the guard can only store the entry
version to avoid the strong reference to the value (only strong
references to the dictionary and to the key are needed).

Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure,
the field has the C type ``PY_INT64_T``. When a key is created or
modified, the entry version is set to the dictionary version which is
incremented at any change (create, modify, delete).

Pseudo-code of an fast guard to check if a dictionary key was modified
using hypothetical ``dict_get_version(dict)``
``dict_get_entry_version(dict)`` functions::

    UNSET = object()

    class GuardDictKey:
        def __init__(self, dict, key):
            self.dict = dict
            self.key = key
            self.dict_version = dict_get_version(dict)
            self.entry_version = dict_get_entry_version(dict, key)

        def check(self):
            """Return True if the dictionary entry did not changed."""

            # read the version field of the dict structure
            dict_version = dict_get_version(self.dict)
            if dict_version == self.version:
                # Fast-path: dictionary lookup avoided
                return True

            # lookup in the dictionary
            entry_version = get_dict_key_version(dict, key)
            if entry_version == self.entry_version:
                # another key was modified:
                # cache the new dictionary version
                self.dict_version = dict_version
                return True

            # the key was modified
            return False

The main drawback of this option is the impact on the memory footprint.
It increases the size of each dictionary entry, so the overhead depends
on the number of buckets (dictionary entries, used or unused yet). For
example, it increases the size of each dictionary entry by 8 bytes on
64-bit system.

In Python, the memory footprint matters and the trend is to reduce it.
Examples:

* `PEP 393 -- Flexible String Representation
  <https://www.python.org/dev/peps/pep-0393/>`_
* `PEP 412 -- Key-Sharing Dictionary
  <https://www.python.org/dev/peps/pep-0412/>`_


Add a new dict subtype
----------------------

Add a new ``verdict`` type, subtype of ``dict``. When guards are needed,
use the ``verdict`` for namespaces (module namespace, type namespace,
instance namespace, etc.) instead of ``dict``.

Leave the ``dict`` type unchanged to not add any overhead (memory
footprint) when guards are not needed.

Technical issue: a lot of C code in the wild, including CPython core,
expecting the exact ``dict`` type. Issues:

* ``exec()`` requires a ``dict`` for globals and locals. A lot of code
  use ``globals={}``. It is not possible to cast the ``dict`` to a
  ``dict`` subtype because the caller expects the ``globals`` parameter
  to be modified (``dict`` is mutable).
* Functions call directly ``PyDict_xxx()`` functions, instead of calling
  ``PyObject_xxx()`` if the object is a ``dict`` subtype
* ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some
  functions require the exact ``dict`` type.
* ``Python/ceval.c`` does not completly supports dict subtypes for
  namespaces


The ``exec()`` issue is a blocker issue.

Other issues:

* The garbage collector has a special code to "untrack" ``dict``
  instances. If a ``dict`` subtype is used for namespaces, the garbage
  collector can be unable to break some reference cycles.
* Some functions have a fast-path for ``dict`` which would not be taken
  for ``dict`` subtypes, and so it would make Python a little bit
  slower.


Prior Art
=========

Method cache and type version tag
---------------------------------

In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It
was merged into Python 2.6.  The patch adds a "type attribute cache
version tag" (``tp_version_tag``) and a "valid version tag" flag to
types (the ``PyTypeObject`` structure).

The type version tag is not available at the Python level.

The version tag has the C type ``unsigned int``. The cache is a global
hash table of 4096 entries, shared by all types. The cache is global to
"make it fast, have a deterministic and low memory footprint, and be
easy to invalidate". Each cache entry has a version tag. A global
version tag is used to create the next version tag, it also has the C
type ``unsigned int``.

By default, a type has its "valid version tag" flag cleared to indicate
that the version tag is invalid. When the first method of the type is
cached, the version tag and the "valid version tag" flag are set. When a
type is modified, the "valid version tag" flag of the type and its
subclasses is cleared. Later, when a cache entry of these types is used,
the entry is removed because its version tag is outdated.

On integer overflow, the whole cache is cleared and the global version
tag is reset to ``0``.

See `Method cache (issue #1685986)
<https://bugs.python.org/issue1685986>`_ and `Armin's method cache
optimization updated for Python 2.6 (issue #1700288)
<https://bugs.python.org/issue1700288>`_.


Globals / builtins cache
------------------------

In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue
#10401) <http://bugs.python.org/issue10401>`_ which adds a private
``ma_version`` field to the ``PyDictObject`` structure (``dict`` type),
the field has the C type ``Py_ssize_t``.

The patch adds a "global and builtin cache" to functions and frames, and
changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the
cache.

The change on the ``PyDictObject`` structure is very similar to this
PEP.


Cached globals+builtins lookup
------------------------------

In 2006, Andrea Griffini proposed a patch implementing a `Cached
globals+builtins lookup optimization
<https://bugs.python.org/issue1616125>`_.  The patch adds a private
``timestamp`` field to the ``PyDictObject`` structure (``dict`` type),
the field has the C type ``size_t``.

Thread on python-dev: `About dictionary lookup caching
<https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_.


Guard against changing dict during iteration
--------------------------------------------

In 2013, Serhiy Storchaka proposed `Guard against changing dict during
iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which
adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict``
type), the field has the C type ``size_t``.  This field is incremented
when the dictionary is modified, and so is very similar to the proposed
dictionary version.

Sadly, the dictionary version proposed in this PEP doesn't help to
detect dictionary mutation. The dictionary version changes when values
are replaced, whereas modifying dictionary values while iterating on
dictionary keys is legit in Python.


PySizer
-------

`PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python,
Google Summer of Code 2005 project by Nick Smallbone.

This project has a patch for CPython 2.4 which adds ``key_time`` and
``value_time`` fields to dictionary entries. It uses a global
process-wide counter for dictionaries, incremented each time that a
dictionary is modified. The times are used to decide when child objects
first appeared in their parent objects.


Discussion
==========

Thread on the python-ideas mailing list: `RFC: PEP: Add dict.__version__
<https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_.


Copyright
=========

This document has been placed in the public domain.

From fijall at gmail.com  Mon Jan 11 14:09:18 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 11 Jan 2016 21:09:18 +0200
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
Message-ID: <CAK5idxRD3ZSHHv95Hh3Lpj8Fh24E_hqc9mffO7L3VdbB10ye4Q@mail.gmail.com>

Hi Victor.

You know that pypy does this stuff without changing and exposing
python semantics right? We have a version dict that does not leak
abstractions to the user.

In general, doing stuff like that where there is a public API that
leaks details of certain optimizations makes it harder and harder for
optimizing compilers to do their job properly, if you want to do
something slightly different.

Can we make this happen (as you noted in the prior art) WITHOUT
changing ANY of the things exposed to the user?

On Mon, Jan 11, 2016 at 6:49 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> ========
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =========
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =============
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.value = dict.get(key, UNSET)
>             self.version = dict_get_version(dict)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             version = dict_get_version(self.dict)
>             if version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             value = self.dict.get(self.key, UNSET)
>             if value is self.value:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.version = version
>                 return True
>
>             # the key was modified
>             return False
>
>
> Usage of the dict version
> =========================
>
> Specialized functions using guards
> ----------------------------------
>
> The `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
> implements many optimizations which require guards on namespaces.
> Examples:
>
> * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
>   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
>   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
>   are required
>
>
> Pyjion
> ------
>
> According of Brett Cannon, one of the two main developers of Pyjion,
> Pyjion can also benefit from dictionary version to implement
> optimizations.
>
> Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
> Core runtime).
>
>
> Unladen Swallow
> ---------------
>
> Even if dictionary version was not explicitly mentionned, optimization
> globals and builtins lookup was part of the Unladen Swallow plan:
> "Implement one of the several proposed schemes for speeding lookups of
> globals and builtins." Source: `Unladen Swallow ProjectPlan
> <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_.
>
> Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
> implemented with LLVM. The project stopped in 2011: `Unladen Swallow
> Retrospective
> <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_.
>
>
> Changes
> =======
>
> Add a ``ma_version`` field to the ``PyDictObject`` structure with the C
> type ``PY_INT64_T``, 64-bit unsigned integer. New empty dictionaries are
> initilized to version ``0``. The version is incremented at each change:
>
> * ``clear()`` if the dict was non-empty
> * ``pop(key)`` if the key exists
> * ``popitem()`` if the dict is non-empty
> * ``setdefault(key, value)`` if the `key` does not exist
> * ``__detitem__(key)`` if the key exists
> * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value
>   is different
> * ``update(...)`` if new values are different than existing values (the
>   version can be incremented multiple times)
>
> Example using an hypothetical ``dict_get_version(dict)`` function::
>
>     >>> d = {}
>     >>> dict_get_version(d)
>     0
>     >>> d['key'] = 'value'
>     >>> dict_get_version(d)
>     1
>     >>> d['key'] = 'new value'
>     >>> dict_get_version(d)
>     2
>     >>> del d['key']
>     >>> dict_get_version(d)
>     3
>
> If a dictionary is created with items, the version is also incremented
> at each dictionary insertion. Example::
>
>     >>> d=dict(x=7, y=33)
>     >>> dict_get_version(d)
>     2
>
> The version is not incremented if an existing key is set to the same
> value. For efficiency, values are compared by their identity:
> ``new_value is old_value``, not by their content:
> ``new_value == old_value``. Example::
>
>     >>> d={}
>     >>> value = object()
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>
> .. note::
>    CPython uses some singleton like integers in the range [-5; 257],
>    empty tuple, empty strings, Unicode strings of a single character in
>    the range [U+0000; U+00FF], etc. When a key is set twice to the same
>    singleton, the version is not modified.
>
>
> Implementation
> ==============
>
> The `issue #26058: PEP 509: Add ma_version to PyDictObject
> <https://bugs.python.org/issue26058>`_ contains a patch implementing
> this PEP.
>
> On pybench and timeit microbenchmarks, the patch does not seem to add
> any overhead on dictionary operations.
>
> When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for
> a dictioanry lookup, whereas a guard check only takes 3.8 ns. Moreover,
> a guard can watch for multiple keys. For example, for an optimization
> using 10 global variables in a function, 10 dictionary lookups costs 148
> ns, whereas the guard still only costs 3.8 ns when the version does not
> change (39x as fast).
>
>
> Integer overflow
> ================
>
> The implementation uses the C unsigned integer type ``PY_INT64_T`` to
> store the version, a 64 bits unsigned integer. The C code uses
> ``version++``. On integer overflow, the version is wrapped to ``0`` (and
> then continue to be incremented) according to the C standard.
>
> After an integer overflow, a guard can succeed whereas the watched
> dictionary key was modified. The bug occurs if the dictionary is
> modified at least ``2 ** 64`` times between two checks of the guard and
> if the new version (theorical value with no integer overflow) is equal
> to the old version modulo ``2 ** 64``.
>
> If a dictionary is modified each nanosecond, an overflow takes longer
> than 584 years. Using a 32-bit version, the overflow occurs only after 4
> seconds. That's why a 64-bit unsigned type is also used on 32-bit
> systems. A dictionary lookup at the C level takes 14.8 ns.
>
> A risk of a bug every 584 years is acceptable.
>
>
> Alternatives
> ============
>
> Expose the version at Python level as a read-only __version__ property
> ----------------------------------------------------------------------
>
> The first version of the PEP proposed to expose the dictionary version
> as a read-only ``__version__`` property at Python level, and also to add
> the property to ``collections.UserDict`` (since this type must mimick
> the ``dict`` API).
>
> There are multiple issues:
>
> * To be consistent and avoid bad surprises, the version must be added to
>   all mapping types. Implementing a new mapping type would require extra
>   work for no benefit, since the version is only required on the
>   ``dict`` type in practice.
> * All Python implementations must implement this new property, it gives
>   more work to other implementations, whereas they may not use the
>   dictionary version at all.
> * The ``__version__`` can be wrapped on integer overflow. It is error
>   prone: using ``dict.__version__ <= guard_version`` is wrong,
>   ``dict.__version__ == guard_version`` must be used instead to reduce
>   the risk of bug on integer overflow (even if the integer overflow is
>   unlikely in practice).
> * Exposing the dictionary version at Python level can lead the
>   false assumption on performances. Checking ``dict.__version__`` at
>   the Python level is not faster than a dictionary lookup. A dictionary
>   lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5
>   ns, the difference is only 1.2 ns (3%)::
>
>
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33'
>     10000000 loops, best of 3: 0.0487 usec per loop
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
> 'd.__version__ == 100'
>     10000000 loops, best of 3: 0.0475 usec per loop
>
> Bikeshedding on the property name:
>
> * ``__cache_token__``: name proposed by Nick Coghlan, name coming from
>   `abc.get_cache_token()
>   <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_.
> * ``__version__``
> * ``__timestamp__``
>
>
> Add a version to each dict entry
> --------------------------------
>
> A single version per dictionary requires to keep a strong reference to
> the value which can keep the value alive longer than expected. If we add
> also a version per dictionary entry, the guard can only store the entry
> version to avoid the strong reference to the value (only strong
> references to the dictionary and to the key are needed).
>
> Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure,
> the field has the C type ``PY_INT64_T``. When a key is created or
> modified, the entry version is set to the dictionary version which is
> incremented at any change (create, modify, delete).
>
> Pseudo-code of an fast guard to check if a dictionary key was modified
> using hypothetical ``dict_get_version(dict)``
> ``dict_get_entry_version(dict)`` functions::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.dict_version = dict_get_version(dict)
>             self.entry_version = dict_get_entry_version(dict, key)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             dict_version = dict_get_version(self.dict)
>             if dict_version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             entry_version = get_dict_key_version(dict, key)
>             if entry_version == self.entry_version:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.dict_version = dict_version
>                 return True
>
>             # the key was modified
>             return False
>
> The main drawback of this option is the impact on the memory footprint.
> It increases the size of each dictionary entry, so the overhead depends
> on the number of buckets (dictionary entries, used or unused yet). For
> example, it increases the size of each dictionary entry by 8 bytes on
> 64-bit system.
>
> In Python, the memory footprint matters and the trend is to reduce it.
> Examples:
>
> * `PEP 393 -- Flexible String Representation
>   <https://www.python.org/dev/peps/pep-0393/>`_
> * `PEP 412 -- Key-Sharing Dictionary
>   <https://www.python.org/dev/peps/pep-0412/>`_
>
>
> Add a new dict subtype
> ----------------------
>
> Add a new ``verdict`` type, subtype of ``dict``. When guards are needed,
> use the ``verdict`` for namespaces (module namespace, type namespace,
> instance namespace, etc.) instead of ``dict``.
>
> Leave the ``dict`` type unchanged to not add any overhead (memory
> footprint) when guards are not needed.
>
> Technical issue: a lot of C code in the wild, including CPython core,
> expecting the exact ``dict`` type. Issues:
>
> * ``exec()`` requires a ``dict`` for globals and locals. A lot of code
>   use ``globals={}``. It is not possible to cast the ``dict`` to a
>   ``dict`` subtype because the caller expects the ``globals`` parameter
>   to be modified (``dict`` is mutable).
> * Functions call directly ``PyDict_xxx()`` functions, instead of calling
>   ``PyObject_xxx()`` if the object is a ``dict`` subtype
> * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some
>   functions require the exact ``dict`` type.
> * ``Python/ceval.c`` does not completly supports dict subtypes for
>   namespaces
>
>
> The ``exec()`` issue is a blocker issue.
>
> Other issues:
>
> * The garbage collector has a special code to "untrack" ``dict``
>   instances. If a ``dict`` subtype is used for namespaces, the garbage
>   collector can be unable to break some reference cycles.
> * Some functions have a fast-path for ``dict`` which would not be taken
>   for ``dict`` subtypes, and so it would make Python a little bit
>   slower.
>
>
> Prior Art
> =========
>
> Method cache and type version tag
> ---------------------------------
>
> In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It
> was merged into Python 2.6.  The patch adds a "type attribute cache
> version tag" (``tp_version_tag``) and a "valid version tag" flag to
> types (the ``PyTypeObject`` structure).
>
> The type version tag is not available at the Python level.
>
> The version tag has the C type ``unsigned int``. The cache is a global
> hash table of 4096 entries, shared by all types. The cache is global to
> "make it fast, have a deterministic and low memory footprint, and be
> easy to invalidate". Each cache entry has a version tag. A global
> version tag is used to create the next version tag, it also has the C
> type ``unsigned int``.
>
> By default, a type has its "valid version tag" flag cleared to indicate
> that the version tag is invalid. When the first method of the type is
> cached, the version tag and the "valid version tag" flag are set. When a
> type is modified, the "valid version tag" flag of the type and its
> subclasses is cleared. Later, when a cache entry of these types is used,
> the entry is removed because its version tag is outdated.
>
> On integer overflow, the whole cache is cleared and the global version
> tag is reset to ``0``.
>
> See `Method cache (issue #1685986)
> <https://bugs.python.org/issue1685986>`_ and `Armin's method cache
> optimization updated for Python 2.6 (issue #1700288)
> <https://bugs.python.org/issue1700288>`_.
>
>
> Globals / builtins cache
> ------------------------
>
> In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue
> #10401) <http://bugs.python.org/issue10401>`_ which adds a private
> ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``Py_ssize_t``.
>
> The patch adds a "global and builtin cache" to functions and frames, and
> changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the
> cache.
>
> The change on the ``PyDictObject`` structure is very similar to this
> PEP.
>
>
> Cached globals+builtins lookup
> ------------------------------
>
> In 2006, Andrea Griffini proposed a patch implementing a `Cached
> globals+builtins lookup optimization
> <https://bugs.python.org/issue1616125>`_.  The patch adds a private
> ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``size_t``.
>
> Thread on python-dev: `About dictionary lookup caching
> <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_.
>
>
> Guard against changing dict during iteration
> --------------------------------------------
>
> In 2013, Serhiy Storchaka proposed `Guard against changing dict during
> iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which
> adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict``
> type), the field has the C type ``size_t``.  This field is incremented
> when the dictionary is modified, and so is very similar to the proposed
> dictionary version.
>
> Sadly, the dictionary version proposed in this PEP doesn't help to
> detect dictionary mutation. The dictionary version changes when values
> are replaced, whereas modifying dictionary values while iterating on
> dictionary keys is legit in Python.
>
>
> PySizer
> -------
>
> `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python,
> Google Summer of Code 2005 project by Nick Smallbone.
>
> This project has a patch for CPython 2.4 which adds ``key_time`` and
> ``value_time`` fields to dictionary entries. It uses a global
> process-wide counter for dictionaries, incremented each time that a
> dictionary is modified. The times are used to decide when child objects
> first appeared in their parent objects.
>
>
> Discussion
> ==========
>
> Thread on the python-ideas mailing list: `RFC: PEP: Add dict.__version__
> <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From victor.stinner at gmail.com  Mon Jan 11 14:56:26 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 11 Jan 2016 20:56:26 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxRD3ZSHHv95Hh3Lpj8Fh24E_hqc9mffO7L3VdbB10ye4Q@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAK5idxRD3ZSHHv95Hh3Lpj8Fh24E_hqc9mffO7L3VdbB10ye4Q@mail.gmail.com>
Message-ID: <CAMpsgwav+D2nHJX7Nw56F3b2ch2ZMjdpiWFRS9pg_asYjjpjqA@mail.gmail.com>

Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" <fijall at gmail.com> a ?crit :
> Hi Victor.
>
> You know that pypy does this stuff without changing and exposing
> python semantics right? We have a version dict that does not leak
> abstractions to the user.

The PEP adds a field to the C structure PyDictObject. Are you asking me to
hide it from the C structure?

The first version of my PEP added a public read-only property at Python
level, but I changed the PEP. See the alternatives section for more detail.

Victor

> In general, doing stuff like that where there is a public API that
> leaks details of certain optimizations makes it harder and harder for
> optimizing compilers to do their job properly, if you want to do
> something slightly different.
>
> Can we make this happen (as you noted in the prior art) WITHOUT
> changing ANY of the things exposed to the user?
>
> On Mon, Jan 11, 2016 at 6:49 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
> > Hi,
> >
> > After a first round on python-ideas, here is the second version of my
> > PEP. The main changes since the first version are that the dictionary
> > version is no more exposed at the Python level and the field type now
> > also has a size of 64-bit on 32-bit platforms.
> >
> > The PEP is part of a serie of 3 PEP adding an API to implement a
> > static Python optimizer specializing functions with guards. The second
> > PEP is currently discussed on python-ideas and I'm still working on
> > the third PEP.
> >
> > Thanks to Red Hat for giving me time to experiment on this.
> >
> >
> > HTML version:
> > https://www.python.org/dev/peps/pep-0509/
> >
> >
> > PEP: 509
> > Title: Add a private version to dict
> > Version: $Revision$
> > Last-Modified: $Date$
> > Author: Victor Stinner <victor.stinner at gmail.com>
> > Status: Draft
> > Type: Standards Track
> > Content-Type: text/x-rst
> > Created: 4-January-2016
> > Python-Version: 3.6
> >
> >
> > Abstract
> > ========
> >
> > Add a new private version to builtin ``dict`` type, incremented at each
> > change, to implement fast guards on namespaces.
> >
> >
> > Rationale
> > =========
> >
> > In Python, the builtin ``dict`` type is used by many instructions. For
> > example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> > global namespace, or in the builtins namespace (two dict lookups).
> > Python uses ``dict`` for the builtins namespace, globals namespace, type
> > namespaces, instance namespaces, etc. The local namespace (namespace of
> > a function) is usually optimized to an array, but it can be a dict too.
> >
> > Python is hard to optimize because almost everything is mutable: builtin
> > functions, function code, global variables, local variables, ... can be
> > modified at runtime. Implementing optimizations respecting the Python
> > semantics requires to detect when "something changes": we will call
> > these checks "guards".
> >
> > The speedup of optimizations depends on the speed of guard checks. This
> > PEP proposes to add a version to dictionaries to implement fast guards
> > on namespaces.
> >
> > Dictionary lookups can be skipped if the version does not change which
> > is the common case for most namespaces. The performance of a guard does
> > not depend on the number of watched dictionary entries, complexity of
> > O(1), if the dictionary version does not change.
> >
> > Example of optimization: copy the value of a global variable to function
> > constants.  This optimization requires a guard on the global variable to
> > check if it was modified. If the variable is modified, the variable must
> > be loaded at runtime when the function is called, instead of using the
> > constant.
> >
> > See the `PEP 510 -- Specialized functions with guards
> > <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of
> > guards to specialize functions and for the rationale on Python static
> > optimizers.
> >
> >
> > Guard example
> > =============
> >
> > Pseudo-code of an fast guard to check if a dictionary entry was modified
> > (created, updated or deleted) using an hypothetical
> > ``dict_get_version(dict)`` function::
> >
> >     UNSET = object()
> >
> >     class GuardDictKey:
> >         def __init__(self, dict, key):
> >             self.dict = dict
> >             self.key = key
> >             self.value = dict.get(key, UNSET)
> >             self.version = dict_get_version(dict)
> >
> >         def check(self):
> >             """Return True if the dictionary entry did not changed."""
> >
> >             # read the version field of the dict structure
> >             version = dict_get_version(self.dict)
> >             if version == self.version:
> >                 # Fast-path: dictionary lookup avoided
> >                 return True
> >
> >             # lookup in the dictionary
> >             value = self.dict.get(self.key, UNSET)
> >             if value is self.value:
> >                 # another key was modified:
> >                 # cache the new dictionary version
> >                 self.version = version
> >                 return True
> >
> >             # the key was modified
> >             return False
> >
> >
> > Usage of the dict version
> > =========================
> >
> > Specialized functions using guards
> > ----------------------------------
> >
> > The `PEP 510 -- Specialized functions with guards
> > <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support
> > specialized functions with guards. It allows to implement static
> > optimizers for Python without breaking the Python semantics.
> >
> > Example of a static Python optimizer: the astoptimizer of the `FAT
> > Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
> > implements many optimizations which require guards on namespaces.
> > Examples:
> >
> > * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
> >   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> > * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
> >   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
> >   are required
> >
> >
> > Pyjion
> > ------
> >
> > According of Brett Cannon, one of the two main developers of Pyjion,
> > Pyjion can also benefit from dictionary version to implement
> > optimizations.
> >
> > Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
> > Core runtime).
> >
> >
> > Unladen Swallow
> > ---------------
> >
> > Even if dictionary version was not explicitly mentionned, optimization
> > globals and builtins lookup was part of the Unladen Swallow plan:
> > "Implement one of the several proposed schemes for speeding lookups of
> > globals and builtins." Source: `Unladen Swallow ProjectPlan
> > <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_.
> >
> > Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
> > implemented with LLVM. The project stopped in 2011: `Unladen Swallow
> > Retrospective
> > <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html
>`_.
> >
> >
> > Changes
> > =======
> >
> > Add a ``ma_version`` field to the ``PyDictObject`` structure with the C
> > type ``PY_INT64_T``, 64-bit unsigned integer. New empty dictionaries are
> > initilized to version ``0``. The version is incremented at each change:
> >
> > * ``clear()`` if the dict was non-empty
> > * ``pop(key)`` if the key exists
> > * ``popitem()`` if the dict is non-empty
> > * ``setdefault(key, value)`` if the `key` does not exist
> > * ``__detitem__(key)`` if the key exists
> > * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value
> >   is different
> > * ``update(...)`` if new values are different than existing values (the
> >   version can be incremented multiple times)
> >
> > Example using an hypothetical ``dict_get_version(dict)`` function::
> >
> >     >>> d = {}
> >     >>> dict_get_version(d)
> >     0
> >     >>> d['key'] = 'value'
> >     >>> dict_get_version(d)
> >     1
> >     >>> d['key'] = 'new value'
> >     >>> dict_get_version(d)
> >     2
> >     >>> del d['key']
> >     >>> dict_get_version(d)
> >     3
> >
> > If a dictionary is created with items, the version is also incremented
> > at each dictionary insertion. Example::
> >
> >     >>> d=dict(x=7, y=33)
> >     >>> dict_get_version(d)
> >     2
> >
> > The version is not incremented if an existing key is set to the same
> > value. For efficiency, values are compared by their identity:
> > ``new_value is old_value``, not by their content:
> > ``new_value == old_value``. Example::
> >
> >     >>> d={}
> >     >>> value = object()
> >     >>> d['key'] = value
> >     >>> dict_get_version(d)
> >     2
> >     >>> d['key'] = value
> >     >>> dict_get_version(d)
> >     2
> >
> > .. note::
> >    CPython uses some singleton like integers in the range [-5; 257],
> >    empty tuple, empty strings, Unicode strings of a single character in
> >    the range [U+0000; U+00FF], etc. When a key is set twice to the same
> >    singleton, the version is not modified.
> >
> >
> > Implementation
> > ==============
> >
> > The `issue #26058: PEP 509: Add ma_version to PyDictObject
> > <https://bugs.python.org/issue26058>`_ contains a patch implementing
> > this PEP.
> >
> > On pybench and timeit microbenchmarks, the patch does not seem to add
> > any overhead on dictionary operations.
> >
> > When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for
> > a dictioanry lookup, whereas a guard check only takes 3.8 ns. Moreover,
> > a guard can watch for multiple keys. For example, for an optimization
> > using 10 global variables in a function, 10 dictionary lookups costs 148
> > ns, whereas the guard still only costs 3.8 ns when the version does not
> > change (39x as fast).
> >
> >
> > Integer overflow
> > ================
> >
> > The implementation uses the C unsigned integer type ``PY_INT64_T`` to
> > store the version, a 64 bits unsigned integer. The C code uses
> > ``version++``. On integer overflow, the version is wrapped to ``0`` (and
> > then continue to be incremented) according to the C standard.
> >
> > After an integer overflow, a guard can succeed whereas the watched
> > dictionary key was modified. The bug occurs if the dictionary is
> > modified at least ``2 ** 64`` times between two checks of the guard and
> > if the new version (theorical value with no integer overflow) is equal
> > to the old version modulo ``2 ** 64``.
> >
> > If a dictionary is modified each nanosecond, an overflow takes longer
> > than 584 years. Using a 32-bit version, the overflow occurs only after 4
> > seconds. That's why a 64-bit unsigned type is also used on 32-bit
> > systems. A dictionary lookup at the C level takes 14.8 ns.
> >
> > A risk of a bug every 584 years is acceptable.
> >
> >
> > Alternatives
> > ============
> >
> > Expose the version at Python level as a read-only __version__ property
> > ----------------------------------------------------------------------
> >
> > The first version of the PEP proposed to expose the dictionary version
> > as a read-only ``__version__`` property at Python level, and also to add
> > the property to ``collections.UserDict`` (since this type must mimick
> > the ``dict`` API).
> >
> > There are multiple issues:
> >
> > * To be consistent and avoid bad surprises, the version must be added to
> >   all mapping types. Implementing a new mapping type would require extra
> >   work for no benefit, since the version is only required on the
> >   ``dict`` type in practice.
> > * All Python implementations must implement this new property, it gives
> >   more work to other implementations, whereas they may not use the
> >   dictionary version at all.
> > * The ``__version__`` can be wrapped on integer overflow. It is error
> >   prone: using ``dict.__version__ <= guard_version`` is wrong,
> >   ``dict.__version__ == guard_version`` must be used instead to reduce
> >   the risk of bug on integer overflow (even if the integer overflow is
> >   unlikely in practice).
> > * Exposing the dictionary version at Python level can lead the
> >   false assumption on performances. Checking ``dict.__version__`` at
> >   the Python level is not faster than a dictionary lookup. A dictionary
> >   lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5
> >   ns, the difference is only 1.2 ns (3%)::
> >
> >
> >     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
'd["33"] == 33'
> >     10000000 loops, best of 3: 0.0487 usec per loop
> >     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
> > 'd.__version__ == 100'
> >     10000000 loops, best of 3: 0.0475 usec per loop
> >
> > Bikeshedding on the property name:
> >
> > * ``__cache_token__``: name proposed by Nick Coghlan, name coming from
> >   `abc.get_cache_token()
> >   <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_.
> > * ``__version__``
> > * ``__timestamp__``
> >
> >
> > Add a version to each dict entry
> > --------------------------------
> >
> > A single version per dictionary requires to keep a strong reference to
> > the value which can keep the value alive longer than expected. If we add
> > also a version per dictionary entry, the guard can only store the entry
> > version to avoid the strong reference to the value (only strong
> > references to the dictionary and to the key are needed).
> >
> > Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure,
> > the field has the C type ``PY_INT64_T``. When a key is created or
> > modified, the entry version is set to the dictionary version which is
> > incremented at any change (create, modify, delete).
> >
> > Pseudo-code of an fast guard to check if a dictionary key was modified
> > using hypothetical ``dict_get_version(dict)``
> > ``dict_get_entry_version(dict)`` functions::
> >
> >     UNSET = object()
> >
> >     class GuardDictKey:
> >         def __init__(self, dict, key):
> >             self.dict = dict
> >             self.key = key
> >             self.dict_version = dict_get_version(dict)
> >             self.entry_version = dict_get_entry_version(dict, key)
> >
> >         def check(self):
> >             """Return True if the dictionary entry did not changed."""
> >
> >             # read the version field of the dict structure
> >             dict_version = dict_get_version(self.dict)
> >             if dict_version == self.version:
> >                 # Fast-path: dictionary lookup avoided
> >                 return True
> >
> >             # lookup in the dictionary
> >             entry_version = get_dict_key_version(dict, key)
> >             if entry_version == self.entry_version:
> >                 # another key was modified:
> >                 # cache the new dictionary version
> >                 self.dict_version = dict_version
> >                 return True
> >
> >             # the key was modified
> >             return False
> >
> > The main drawback of this option is the impact on the memory footprint.
> > It increases the size of each dictionary entry, so the overhead depends
> > on the number of buckets (dictionary entries, used or unused yet). For
> > example, it increases the size of each dictionary entry by 8 bytes on
> > 64-bit system.
> >
> > In Python, the memory footprint matters and the trend is to reduce it.
> > Examples:
> >
> > * `PEP 393 -- Flexible String Representation
> >   <https://www.python.org/dev/peps/pep-0393/>`_
> > * `PEP 412 -- Key-Sharing Dictionary
> >   <https://www.python.org/dev/peps/pep-0412/>`_
> >
> >
> > Add a new dict subtype
> > ----------------------
> >
> > Add a new ``verdict`` type, subtype of ``dict``. When guards are needed,
> > use the ``verdict`` for namespaces (module namespace, type namespace,
> > instance namespace, etc.) instead of ``dict``.
> >
> > Leave the ``dict`` type unchanged to not add any overhead (memory
> > footprint) when guards are not needed.
> >
> > Technical issue: a lot of C code in the wild, including CPython core,
> > expecting the exact ``dict`` type. Issues:
> >
> > * ``exec()`` requires a ``dict`` for globals and locals. A lot of code
> >   use ``globals={}``. It is not possible to cast the ``dict`` to a
> >   ``dict`` subtype because the caller expects the ``globals`` parameter
> >   to be modified (``dict`` is mutable).
> > * Functions call directly ``PyDict_xxx()`` functions, instead of calling
> >   ``PyObject_xxx()`` if the object is a ``dict`` subtype
> > * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some
> >   functions require the exact ``dict`` type.
> > * ``Python/ceval.c`` does not completly supports dict subtypes for
> >   namespaces
> >
> >
> > The ``exec()`` issue is a blocker issue.
> >
> > Other issues:
> >
> > * The garbage collector has a special code to "untrack" ``dict``
> >   instances. If a ``dict`` subtype is used for namespaces, the garbage
> >   collector can be unable to break some reference cycles.
> > * Some functions have a fast-path for ``dict`` which would not be taken
> >   for ``dict`` subtypes, and so it would make Python a little bit
> >   slower.
> >
> >
> > Prior Art
> > =========
> >
> > Method cache and type version tag
> > ---------------------------------
> >
> > In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It
> > was merged into Python 2.6.  The patch adds a "type attribute cache
> > version tag" (``tp_version_tag``) and a "valid version tag" flag to
> > types (the ``PyTypeObject`` structure).
> >
> > The type version tag is not available at the Python level.
> >
> > The version tag has the C type ``unsigned int``. The cache is a global
> > hash table of 4096 entries, shared by all types. The cache is global to
> > "make it fast, have a deterministic and low memory footprint, and be
> > easy to invalidate". Each cache entry has a version tag. A global
> > version tag is used to create the next version tag, it also has the C
> > type ``unsigned int``.
> >
> > By default, a type has its "valid version tag" flag cleared to indicate
> > that the version tag is invalid. When the first method of the type is
> > cached, the version tag and the "valid version tag" flag are set. When a
> > type is modified, the "valid version tag" flag of the type and its
> > subclasses is cleared. Later, when a cache entry of these types is used,
> > the entry is removed because its version tag is outdated.
> >
> > On integer overflow, the whole cache is cleared and the global version
> > tag is reset to ``0``.
> >
> > See `Method cache (issue #1685986)
> > <https://bugs.python.org/issue1685986>`_ and `Armin's method cache
> > optimization updated for Python 2.6 (issue #1700288)
> > <https://bugs.python.org/issue1700288>`_.
> >
> >
> > Globals / builtins cache
> > ------------------------
> >
> > In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue
> > #10401) <http://bugs.python.org/issue10401>`_ which adds a private
> > ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type),
> > the field has the C type ``Py_ssize_t``.
> >
> > The patch adds a "global and builtin cache" to functions and frames, and
> > changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the
> > cache.
> >
> > The change on the ``PyDictObject`` structure is very similar to this
> > PEP.
> >
> >
> > Cached globals+builtins lookup
> > ------------------------------
> >
> > In 2006, Andrea Griffini proposed a patch implementing a `Cached
> > globals+builtins lookup optimization
> > <https://bugs.python.org/issue1616125>`_.  The patch adds a private
> > ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type),
> > the field has the C type ``size_t``.
> >
> > Thread on python-dev: `About dictionary lookup caching
> > <https://mail.python.org/pipermail/python-dev/2006-December/070348.html
>`_.
> >
> >
> > Guard against changing dict during iteration
> > --------------------------------------------
> >
> > In 2013, Serhiy Storchaka proposed `Guard against changing dict during
> > iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which
> > adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict``
> > type), the field has the C type ``size_t``.  This field is incremented
> > when the dictionary is modified, and so is very similar to the proposed
> > dictionary version.
> >
> > Sadly, the dictionary version proposed in this PEP doesn't help to
> > detect dictionary mutation. The dictionary version changes when values
> > are replaced, whereas modifying dictionary values while iterating on
> > dictionary keys is legit in Python.
> >
> >
> > PySizer
> > -------
> >
> > `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python,
> > Google Summer of Code 2005 project by Nick Smallbone.
> >
> > This project has a patch for CPython 2.4 which adds ``key_time`` and
> > ``value_time`` fields to dictionary entries. It uses a global
> > process-wide counter for dictionaries, incremented each time that a
> > dictionary is modified. The times are used to decide when child objects
> > first appeared in their parent objects.
> >
> >
> > Discussion
> > ==========
> >
> > Thread on the python-ideas mailing list: `RFC: PEP: Add dict.__version__
> > <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html
>`_.
> >
> >
> > Copyright
> > =========
> >
> > This document has been placed in the public domain.
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160111/76061889/attachment-0001.html>

From fijall at gmail.com  Mon Jan 11 15:10:30 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 11 Jan 2016 22:10:30 +0200
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwav+D2nHJX7Nw56F3b2ch2ZMjdpiWFRS9pg_asYjjpjqA@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAK5idxRD3ZSHHv95Hh3Lpj8Fh24E_hqc9mffO7L3VdbB10ye4Q@mail.gmail.com>
 <CAMpsgwav+D2nHJX7Nw56F3b2ch2ZMjdpiWFRS9pg_asYjjpjqA@mail.gmail.com>
Message-ID: <CAK5idxRXpDWP28r94yCLR+8tdmzMBYWS28EC0P3yWn-ATyKGZQ@mail.gmail.com>

On Mon, Jan 11, 2016 at 9:56 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Le 11 janv. 2016 8:09 PM, "Maciej Fijalkowski" <fijall at gmail.com> a ?crit :
>> Hi Victor.
>>
>> You know that pypy does this stuff without changing and exposing
>> python semantics right? We have a version dict that does not leak
>> abstractions to the user.
>
> The PEP adds a field to the C structure PyDictObject. Are you asking me to
> hide it from the C structure?
>
> The first version of my PEP added a public read-only property at Python
> level, but I changed the PEP. See the alternatives section for more detail.
>
> Victor

I asked you to hide it from python, read the wrong version :-)

Cool!

From greg at krypto.org  Mon Jan 11 18:07:33 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 11 Jan 2016 23:07:33 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
Message-ID: <CAGE7PNKdY1eZZbUcH1WHtxR+2VRwW9F-heyERGx=AMP460Trog@mail.gmail.com>

On Mon, Jan 11, 2016 at 8:50 AM Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> ========
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =========
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =============
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.value = dict.get(key, UNSET)
>             self.version = dict_get_version(dict)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             version = dict_get_version(self.dict)
>             if version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             value = self.dict.get(self.key, UNSET)
>             if value is self.value:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.version = version
>                 return True
>
>             # the key was modified
>             return False
>
>
> Usage of the dict version
> =========================
>
> Specialized functions using guards
> ----------------------------------
>
> The `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
> implements many optimizations which require guards on namespaces.
> Examples:
>
> * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
>   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
>   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
>   are required
>
>
> Pyjion
> ------
>
> According of Brett Cannon, one of the two main developers of Pyjion,
> Pyjion can also benefit from dictionary version to implement
> optimizations.
>
> Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
> Core runtime).
>
>
> Unladen Swallow
> ---------------
>
> Even if dictionary version was not explicitly mentionned, optimization
> globals and builtins lookup was part of the Unladen Swallow plan:
> "Implement one of the several proposed schemes for speeding lookups of
> globals and builtins." Source: `Unladen Swallow ProjectPlan
> <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_.
>
> Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
> implemented with LLVM. The project stopped in 2011: `Unladen Swallow
> Retrospective
> <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html
> >`_.
>
>
> Changes
> =======
>
> Add a ``ma_version`` field to the ``PyDictObject`` structure with the C
> type ``PY_INT64_T``, 64-bit unsigned integer. New empty dictionaries are
> initilized to version ``0``. The version is incremented at each change:
>
> * ``clear()`` if the dict was non-empty
> * ``pop(key)`` if the key exists
> * ``popitem()`` if the dict is non-empty
> * ``setdefault(key, value)`` if the `key` does not exist
> * ``__detitem__(key)`` if the key exists
> * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value
>   is different
> * ``update(...)`` if new values are different than existing values (the
>   version can be incremented multiple times)
>

Please be more explicit about what tests you are performing on the values.
 setitem's "if the value is different" really should mean "if value is not
dict['key']".  similarly for update, there should never be equality checks
performed on the values.  just an "is" test of it they are the same object
or not.


>
> Example using an hypothetical ``dict_get_version(dict)`` function::
>
>     >>> d = {}
>     >>> dict_get_version(d)
>     0
>     >>> d['key'] = 'value'
>     >>> dict_get_version(d)
>     1
>     >>> d['key'] = 'new value'
>     >>> dict_get_version(d)
>     2
>     >>> del d['key']
>     >>> dict_get_version(d)
>     3
>
> If a dictionary is created with items, the version is also incremented
> at each dictionary insertion. Example::
>
>     >>> d=dict(x=7, y=33)
>     >>> dict_get_version(d)
>     2
>
> The version is not incremented if an existing key is set to the same
> value. For efficiency, values are compared by their identity:
> ``new_value is old_value``, not by their content:
> ``new_value == old_value``. Example::
>
>     >>> d={}
>     >>> value = object()
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>
> .. note::
>    CPython uses some singleton like integers in the range [-5; 257],
>    empty tuple, empty strings, Unicode strings of a single character in
>    the range [U+0000; U+00FF], etc. When a key is set twice to the same
>    singleton, the version is not modified.
>
>
> Implementation
> ==============
>
> The `issue #26058: PEP 509: Add ma_version to PyDictObject
> <https://bugs.python.org/issue26058>`_ contains a patch implementing
> this PEP.
>
> On pybench and timeit microbenchmarks, the patch does not seem to add
> any overhead on dictionary operations.
>
> When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for
> a dictioanry lookup, whereas a guard check only takes 3.8 ns. Moreover,
> a guard can watch for multiple keys. For example, for an optimization
> using 10 global variables in a function, 10 dictionary lookups costs 148
> ns, whereas the guard still only costs 3.8 ns when the version does not
> change (39x as fast).
>
>
> Integer overflow
> ================
>
> The implementation uses the C unsigned integer type ``PY_INT64_T`` to
> store the version, a 64 bits unsigned integer. The C code uses
> ``version++``. On integer overflow, the version is wrapped to ``0`` (and
> then continue to be incremented) according to the C standard.
>
> After an integer overflow, a guard can succeed whereas the watched
> dictionary key was modified. The bug occurs if the dictionary is
> modified at least ``2 ** 64`` times between two checks of the guard and
> if the new version (theorical value with no integer overflow) is equal
> to the old version modulo ``2 ** 64``.
>
> If a dictionary is modified each nanosecond, an overflow takes longer
> than 584 years. Using a 32-bit version, the overflow occurs only after 4
> seconds. That's why a 64-bit unsigned type is also used on 32-bit
> systems. A dictionary lookup at the C level takes 14.8 ns.
>
> A risk of a bug every 584 years is acceptable.
>
>
> Alternatives
> ============


> Expose the version at Python level as a read-only __version__ property
> ----------------------------------------------------------------------
>
> The first version of the PEP proposed to expose the dictionary version
> as a read-only ``__version__`` property at Python level, and also to add
> the property to ``collections.UserDict`` (since this type must mimick
> the ``dict`` API).
>
> There are multiple issues:
>
> * To be consistent and avoid bad surprises, the version must be added to
>   all mapping types. Implementing a new mapping type would require extra
>   work for no benefit, since the version is only required on the
>   ``dict`` type in practice.
> * All Python implementations must implement this new property, it gives
>   more work to other implementations, whereas they may not use the
>   dictionary version at all.
> * The ``__version__`` can be wrapped on integer overflow. It is error
>   prone: using ``dict.__version__ <= guard_version`` is wrong,
>   ``dict.__version__ == guard_version`` must be used instead to reduce
>   the risk of bug on integer overflow (even if the integer overflow is
>   unlikely in practice).
> * Exposing the dictionary version at Python level can lead the
>   false assumption on performances. Checking ``dict.__version__`` at
>   the Python level is not faster than a dictionary lookup. A dictionary
>   lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5
>   ns, the difference is only 1.2 ns (3%)::
>
>
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"]
> == 33'
>     10000000 loops, best of 3: 0.0487 usec per loop
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
> 'd.__version__ == 100'
>     10000000 loops, best of 3: 0.0475 usec per loop
>
> Bikeshedding on the property name:
>
> * ``__cache_token__``: name proposed by Nick Coghlan, name coming from
>   `abc.get_cache_token()
>   <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_.
> * ``__version__``
> * ``__timestamp__``
>
>
> Add a version to each dict entry
> --------------------------------
>
> A single version per dictionary requires to keep a strong reference to
> the value which can keep the value alive longer than expected. If we add
> also a version per dictionary entry, the guard can only store the entry
> version to avoid the strong reference to the value (only strong
> references to the dictionary and to the key are needed).
>
> Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure,
> the field has the C type ``PY_INT64_T``. When a key is created or
> modified, the entry version is set to the dictionary version which is
> incremented at any change (create, modify, delete).
>
> Pseudo-code of an fast guard to check if a dictionary key was modified
> using hypothetical ``dict_get_version(dict)``
> ``dict_get_entry_version(dict)`` functions::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.dict_version = dict_get_version(dict)
>             self.entry_version = dict_get_entry_version(dict, key)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             dict_version = dict_get_version(self.dict)
>             if dict_version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             entry_version = get_dict_key_version(dict, key)
>             if entry_version == self.entry_version:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.dict_version = dict_version
>                 return True
>
>             # the key was modified
>             return False
>
> The main drawback of this option is the impact on the memory footprint.
> It increases the size of each dictionary entry, so the overhead depends
> on the number of buckets (dictionary entries, used or unused yet). For
> example, it increases the size of each dictionary entry by 8 bytes on
> 64-bit system.
>
> In Python, the memory footprint matters and the trend is to reduce it.
> Examples:
>
> * `PEP 393 -- Flexible String Representation
>   <https://www.python.org/dev/peps/pep-0393/>`_
> * `PEP 412 -- Key-Sharing Dictionary
>   <https://www.python.org/dev/peps/pep-0412/>`_
>
>
> Add a new dict subtype
> ----------------------
>
> Add a new ``verdict`` type, subtype of ``dict``. When guards are needed,
> use the ``verdict`` for namespaces (module namespace, type namespace,
> instance namespace, etc.) instead of ``dict``.
>
> Leave the ``dict`` type unchanged to not add any overhead (memory
> footprint) when guards are not needed.
>
> Technical issue: a lot of C code in the wild, including CPython core,
> expecting the exact ``dict`` type. Issues:
>
> * ``exec()`` requires a ``dict`` for globals and locals. A lot of code
>   use ``globals={}``. It is not possible to cast the ``dict`` to a
>   ``dict`` subtype because the caller expects the ``globals`` parameter
>   to be modified (``dict`` is mutable).
> * Functions call directly ``PyDict_xxx()`` functions, instead of calling
>   ``PyObject_xxx()`` if the object is a ``dict`` subtype
> * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some
>   functions require the exact ``dict`` type.
> * ``Python/ceval.c`` does not completly supports dict subtypes for
>   namespaces
>
>
> The ``exec()`` issue is a blocker issue.
>
> Other issues:
>
> * The garbage collector has a special code to "untrack" ``dict``
>   instances. If a ``dict`` subtype is used for namespaces, the garbage
>   collector can be unable to break some reference cycles.
> * Some functions have a fast-path for ``dict`` which would not be taken
>   for ``dict`` subtypes, and so it would make Python a little bit
>   slower.
>
>
> Prior Art
> =========
>
> Method cache and type version tag
> ---------------------------------
>
> In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It
> was merged into Python 2.6.  The patch adds a "type attribute cache
> version tag" (``tp_version_tag``) and a "valid version tag" flag to
> types (the ``PyTypeObject`` structure).
>
> The type version tag is not available at the Python level.
>
> The version tag has the C type ``unsigned int``. The cache is a global
> hash table of 4096 entries, shared by all types. The cache is global to
> "make it fast, have a deterministic and low memory footprint, and be
> easy to invalidate". Each cache entry has a version tag. A global
> version tag is used to create the next version tag, it also has the C
> type ``unsigned int``.
>
> By default, a type has its "valid version tag" flag cleared to indicate
> that the version tag is invalid. When the first method of the type is
> cached, the version tag and the "valid version tag" flag are set. When a
> type is modified, the "valid version tag" flag of the type and its
> subclasses is cleared. Later, when a cache entry of these types is used,
> the entry is removed because its version tag is outdated.
>
> On integer overflow, the whole cache is cleared and the global version
> tag is reset to ``0``.
>
> See `Method cache (issue #1685986)
> <https://bugs.python.org/issue1685986>`_ and `Armin's method cache
> optimization updated for Python 2.6 (issue #1700288)
> <https://bugs.python.org/issue1700288>`_.
>
>
> Globals / builtins cache
> ------------------------
>
> In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue
> #10401) <http://bugs.python.org/issue10401>`_ which adds a private
> ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``Py_ssize_t``.
>
> The patch adds a "global and builtin cache" to functions and frames, and
> changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the
> cache.
>
> The change on the ``PyDictObject`` structure is very similar to this
> PEP.
>
>
> Cached globals+builtins lookup
> ------------------------------
>
> In 2006, Andrea Griffini proposed a patch implementing a `Cached
> globals+builtins lookup optimization
> <https://bugs.python.org/issue1616125>`_.  The patch adds a private
> ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``size_t``.
>
> Thread on python-dev: `About dictionary lookup caching
> <https://mail.python.org/pipermail/python-dev/2006-December/070348.html
> >`_.
>
>
> Guard against changing dict during iteration
> --------------------------------------------
>
> In 2013, Serhiy Storchaka proposed `Guard against changing dict during
> iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which
> adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict``
> type), the field has the C type ``size_t``.  This field is incremented
> when the dictionary is modified, and so is very similar to the proposed
> dictionary version.
>
> Sadly, the dictionary version proposed in this PEP doesn't help to
> detect dictionary mutation. The dictionary version changes when values
> are replaced, whereas modifying dictionary values while iterating on
> dictionary keys is legit in Python.
>
>
> PySizer
> -------
>
> `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python,
> Google Summer of Code 2005 project by Nick Smallbone.
>
> This project has a patch for CPython 2.4 which adds ``key_time`` and
> ``value_time`` fields to dictionary entries. It uses a global
> process-wide counter for dictionaries, incremented each time that a
> dictionary is modified. The times are used to decide when child objects
> first appeared in their parent objects.
>
>
> Discussion
> ==========
>
> Thread on the python-ideas mailing list: `RFC: PEP: Add dict.__version__
> <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html
> >`_.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160111/7370b101/attachment.html>

From victor.stinner at gmail.com  Mon Jan 11 18:24:03 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 12 Jan 2016 00:24:03 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAGE7PNKdY1eZZbUcH1WHtxR+2VRwW9F-heyERGx=AMP460Trog@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAGE7PNKdY1eZZbUcH1WHtxR+2VRwW9F-heyERGx=AMP460Trog@mail.gmail.com>
Message-ID: <CAMpsgwY65bAh-Z5cCymTCeZvjkGWLNT-0aiYUpHYjbuxZkD_rQ@mail.gmail.com>

2016-01-12 0:07 GMT+01:00 Gregory P. Smith <greg at krypto.org>:
>> Changes
>> =======
>>
>> (...)
>
> Please be more explicit about what tests you are performing on the values.
> setitem's "if the value is different" really should mean "if value is not
> dict['key']".  similarly for update, there should never be equality checks
> performed on the values.  just an "is" test of it they are the same object
> or not.

Ok, done. By the way, it's also explained below: values are compared
by their identify, not by their content.

For best dict efficiency, we can not implement this micro-optimization
(to avoid a potential branch misprediction in the CPU) and always
increase the version. But for guards, the micro-optimization can avoid
a lot of dictionary lookups, especially when a guard watches for a
large number of keys.

Victor

From abarnert at yahoo.com  Mon Jan 11 18:35:35 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 11 Jan 2016 15:35:35 -0800
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwY65bAh-Z5cCymTCeZvjkGWLNT-0aiYUpHYjbuxZkD_rQ@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAGE7PNKdY1eZZbUcH1WHtxR+2VRwW9F-heyERGx=AMP460Trog@mail.gmail.com>
 <CAMpsgwY65bAh-Z5cCymTCeZvjkGWLNT-0aiYUpHYjbuxZkD_rQ@mail.gmail.com>
Message-ID: <3FEDF0A4-9B54-4087-B7F8-405C11169F3F@yahoo.com>

On Jan 11, 2016, at 15:24, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> 2016-01-12 0:07 GMT+01:00 Gregory P. Smith <greg at krypto.org>:
>>> Changes
>>> =======
>>> 
>>> (...)
>> 
>> Please be more explicit about what tests you are performing on the values.
>> setitem's "if the value is different" really should mean "if value is not
>> dict['key']".  similarly for update, there should never be equality checks
>> performed on the values.  just an "is" test of it they are the same object
>> or not.
> 
> Ok, done. By the way, it's also explained below: values are compared
> by their identify, not by their content.
> 
> For best dict efficiency, we can not implement this micro-optimization
> (to avoid a potential branch misprediction in the CPU) and always
> increase the version. But for guards, the micro-optimization can avoid
> a lot of dictionary lookups, especially when a guard watches for a
> large number of keys.

Are you saying that d[key] = d[key] may or may not increment the version, so any optimizer can't rely on the fact that it doesn't?

If so, that seems reasonable. (The worst case in incrementing the version unnecessarily is that you miss an optimization that would have been safe, right?).

From victor.stinner at gmail.com  Tue Jan 12 09:25:22 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 12 Jan 2016 15:25:22 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <3FEDF0A4-9B54-4087-B7F8-405C11169F3F@yahoo.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAGE7PNKdY1eZZbUcH1WHtxR+2VRwW9F-heyERGx=AMP460Trog@mail.gmail.com>
 <CAMpsgwY65bAh-Z5cCymTCeZvjkGWLNT-0aiYUpHYjbuxZkD_rQ@mail.gmail.com>
 <3FEDF0A4-9B54-4087-B7F8-405C11169F3F@yahoo.com>
Message-ID: <CAMpsgwbYHKAg=8RcndUW=mMyB9=h9szzp0XVohDAnp9gMCiw-Q@mail.gmail.com>

Well, it was just a remark.

2016-01-12 0:35 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
> Are you saying that d[key] = d[key] may or may not increment the version, so any optimizer can't rely on the fact that it doesn't?

Optimizers don't have to rely on this exactly behaviour. Not
incrementing the version on such case avoids dictionary lookups in the
guard.

My current patch does not increment if the value is the same, and I'm
unable to see any performance regression on *micro* benchmarks:
https://bugs.python.org/issue26058

So I'm in favor of making guards as efficient as possible and not
increment the version in dict ;-)

Victor

From jimjjewett at gmail.com  Tue Jan 12 13:34:07 2016
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Tue, 12 Jan 2016 13:34:07 -0500
Subject: [Python-Dev] PEP 509
Message-ID: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>

(1)  Please make it clear within the abstract what counts as a change.

(1a)  E.g., a second paragraph such as "Adding or removing a key, or
replacing a value, counts as a change.  Modifying an object in place,
or replacing it with itself may not be picked up."

(1b)  Is there a way to force a version update?

d[k]=d[k] seems like it should do that (absent the optimization to
prevent it), but I confess that I can't come up with a good use case
that doesn't start seeming internal to a specific optimizer.

(1c)  Section "Guard against changing dict during iteration" says
"Sadly, the dictionary version proposed in this PEP doesn't help to
detect dictionary mutation."  Why not?  Wouldn't that mutation involve
replacing a value, which ought to trigger a version change?


(2)  I would like to see a .get on the guard object, so that it could
be used in place of the dict lookup even from python.  If this doesn't
make sense (e.g., doesn't really save time since the guard has to be
used from python), please mention that in the Guard Example text.


(3)  It would be possible to define the field as reserved in the main
header, and require another header to use it even from C.

(3a)  This level of privacy might be overkill, but I would prefer that
the decision be explicit.

(3b)  The change should almost certainly be hidden from the ABI / Py_LIMITED_API

-jJ

From ethan at stoneleaf.us  Tue Jan 12 13:52:24 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Jan 2016 10:52:24 -0800
Subject: [Python-Dev] PEP 509
In-Reply-To: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
Message-ID: <56954B68.3090208@stoneleaf.us>

On 01/12/2016 10:34 AM, Jim J. Jewett wrote:

> (1c)  Section "Guard against changing dict during iteration" says
> "Sadly, the dictionary version proposed in this PEP doesn't help to
> detect dictionary mutation."  Why not?  Wouldn't that mutation involve
> replacing a value, which ought to trigger a version change?


Yes it would, but mutating a dictionary value during iteration is legal, 
so we cannot use the __version__ [1] change to tell us that something 
illegal happened.

[1] We're not going to call it __version__ are we?  Seems like 
__cache_token__ is a much better name.

--
~Ethan~

From victor.stinner at gmail.com  Tue Jan 12 16:34:09 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 12 Jan 2016 22:34:09 +0100
Subject: [Python-Dev] PEP 509
In-Reply-To: <56954B68.3090208@stoneleaf.us>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
Message-ID: <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>

2016-01-12 19:52 GMT+01:00 Ethan Furman <ethan at stoneleaf.us>:
> [1] We're not going to call it __version__ are we?  Seems like
> __cache_token__ is a much better name.

See the online version to the most recent version of the PEP:
https://www.python.org/dev/peps/pep-0509/

In the first version I proposed to expose the version, but I changed
my mind, it's not private (only exposed in the C API).

Victor

From victor.stinner at gmail.com  Tue Jan 12 16:42:35 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 12 Jan 2016 22:42:35 +0100
Subject: [Python-Dev] PEP 509
In-Reply-To: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
Message-ID: <CAMpsgway_kjfkKjWPayEkOx3djP1XbFHTy4qu1pDpXiMVv6UPA@mail.gmail.com>

2016-01-12 19:34 GMT+01:00 Jim J. Jewett <jimjjewett at gmail.com>:
> (1)  Please make it clear within the abstract what counts as a change.

I don't think that an abstract must give the long list of cases when
the version is modified or not. It's explained in detail at:
https://www.python.org/dev/peps/pep-0509/#changes

> (1b)  Is there a way to force a version update?

No. Why would you do that? (What is your use case.)

FYI there is a private API in _testcapi to set the version, for unit tests.

> (2)  I would like to see a .get on the guard object, so that it could
> be used in place of the dict lookup even from python.  If this doesn't
> make sense (e.g., doesn't really save time since the guard has to be
> used from python), please mention that in the Guard Example text.

Optimizations are out of the scope of this PEP.

Please see https://www.python.org/dev/peps/pep-0510/ to more examples
of specialization with guards.

See also https://faster-cpython.readthedocs.org/fat_python.html for
concrete optimizations using specialization with guards.

> (3)  It would be possible to define the field as reserved in the main
> header, and require another header to use it even from C.
>
> (3a)  This level of privacy might be overkill, but I would prefer that
> the decision be explicit.
>
> (3b)  The change should almost certainly be hidden from the ABI / Py_LIMITED_API

Oh, the PyDictObject structure is not part of the stable ABI. It seems
worth to mention it in the PEP.

Victor

From victor.stinner at gmail.com  Tue Jan 12 17:13:22 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 12 Jan 2016 23:13:22 +0100
Subject: [Python-Dev] PEP 510: Specialize functions with guards
Message-ID: <CAMpsgwZmb2KBi+5OtBN5Cg4-LDc_hbM6nrDVusXaKWn3dOeefQ@mail.gmail.com>

Hi,

I posted a first version of this PEP on python-ideas and I got
interesting feedback. The main changes in this second version is that
the whole API is now private: no more exposed in Python at all, only
in the Python C API (C level).

The PEP is part of a serie of 3 PEP (509, 510, 511) adding an API to implement a
static Python optimizer specializing functions with guards. I will
post a first version of the PEP 511 to python-ideas soon.

HTML version:
https://www.python.org/dev/peps/pep-0510/#changes

Victor


PEP: 510
Title: Specialize functions with guards
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 4-January-2016
Python-Version: 3.6


Abstract
========

Add functions to the Python C API to specialize pure Python functions:
add specialized codes with guards. It allows to implement static
optimizers respecting the Python semantics.


Rationale
=========

Python semantics
----------------

Python is hard to optimize because almost everything is mutable: builtin
functions, function code, global variables, local variables, ... can be
modified at runtime. Implement optimizations respecting the Python
semantics requires to detect when "something changes", we will call these
checks "guards".

This PEP proposes to add a public API to the Python C API to add
specialized codes with guards to a function. When the function is
called, a specialized code is used if nothing changed, otherwise use the
original bytecode.

Even if guards help to respect most parts of the Python semantics, it's
hard to optimize Python without making subtle changes on the exact
behaviour. CPython has a long history and many applications rely on
implementation details. A compromise must be found between "everything
is mutable" and performance.

Writing an optimizer is out of the scope of this PEP.


Why not a JIT compiler?
-----------------------

There are multiple JIT compilers for Python actively developed:

* `PyPy <http://pypy.org/>`_
* `Pyston <https://github.com/dropbox/pyston>`_
* `Numba <http://numba.pydata.org/>`_
* `Pyjion <https://github.com/microsoft/pyjion>`_

Numba is specific to numerical computation.  Pyston and Pyjion are still
young.  PyPy is the most complete Python interpreter, it is much faster
than CPython and has a very good compatibility with CPython (it respects
the Python semantics). There are still issues with Python JIT compilers
which avoid them to be widely used instead of CPython.

Many popular libraries like numpy, PyGTK, PyQt, PySide and wxPython are
implemented in C or C++ and use the Python C API. To have a small memory
footprint and better performances, Python JIT compilers do not use
reference counting to use a faster garbage collector, do not use C
structures of CPython objects and manage memory allocations differently.
PyPy has a ``cpyext`` module which emulates the Python C API but it has
worse performances than CPython and does not support the full Python C
API.

New features are first developped in CPython. In january 2016, the
latest CPython stable version is 3.5, whereas PyPy only supports Python
2.7 and 3.2, and Pyston only supports Python 2.7.

Even if PyPy has a very good compatibility with Python, some modules are
still not compatible with PyPy: see `PyPy Compatibility Wiki
<https://bitbucket.org/pypy/compatibility/wiki/Home>`_. The incomplete
support of the the Python C API is part of this problem. There are also
subtle differences between PyPy and CPython like reference counting:
object destructors are always called in PyPy, but can be called "later"
than in CPython. Using context managers helps to control when resources
are released.

Even if PyPy is much faster than CPython in a wide range of benchmarks,
some users still report worse performances than CPython on some specific
use cases or unstable performances.

When Python is used as a scripting program for programs running less
than 1 minute, JIT compilers can be slower because their startup time is
higher and the JIT compiler takes time to optimize the code. For
example, most Mercurial commands take a few seconds.

Numba now supports ahead of time compilation, but it requires decorator
to specify arguments types and it only supports numerical types.

CPython 3.5 has almost no optimization: the peephole optimizer only
implements basic optimizations. A static compiler is a compromise
between CPython 3.5 and PyPy.

.. note::
   There was also the Unladen Swallow project, but it was abandoned in
   2011.


Examples
========

Following examples are not written to show powerful optimizations
promising important speedup, but to be short and easy to understand,
just to explain the principle.

Hypothetical myoptimizer module
-------------------------------

Examples in this PEP uses an hypothetical ``myoptimizer`` module which
provides the following functions and types:

* ``specialize(func, code, guards)``: add the specialized code `code`
  with guards `guards` to the function `func`
* ``get_specialized(func)``: get the list of specialized codes as a list
  of ``(code, guards)`` tuples where `code` is a callable or code object
  and `guards` is a list of a guards
* ``GuardBuiltins(name)``: guard watching for
  ``builtins.__dict__[name]`` and ``globals()[name]``. The guard fails
  if ``builtins.__dict__[name]`` is replaced, or if ``globals()[name]``
  is set.


Using bytecode
--------------

Add specialized bytecode where the call to the pure builtin function
``chr(65)`` is replaced with its result ``"A"``::

    import myoptimizer

    def func():
        return chr(65)

    def fast_func():
        return "A"

    myoptimizer.specialize(func, fast_func.__code__,
                           [myoptimizer.GuardBuiltins("chr")])
    del fast_func

Example showing the behaviour of the guard::

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(): %s" % func())
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output::

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call uses the specialized bytecode which returns the string
``"A"``. The second call removes the specialized code because the
builtin ``chr()`` function was replaced, and executes the original
bytecode calling ``chr(65)``.

On a microbenchmark, calling the specialized bytecode takes 88 ns,
whereas the original function takes 145 ns (+57 ns): 1.6 times as fast.


Using builtin function
----------------------

Add the C builtin ``chr()`` function as the specialized code instead of
a bytecode calling ``chr(obj)``::

    import myoptimizer

    def func(arg):
        return chr(arg)

    myoptimizer.specialize(func, chr,
                           [myoptimizer.GuardBuiltins("chr")])

Example showing the behaviour of the guard::

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))
    print()

    import builtins
    builtins.chr = lambda obj: "mock"

    print("func(65): %s" % func(65))
    print("#specialized: %s" % len(myoptimizer.get_specialized(func)))

Output::

    func(): A
    #specialized: 1

    func(): mock
    #specialized: 0

The first call calls the C builtin ``chr()`` function (without creating
a Python frame). The second call removes the specialized code because
the builtin ``chr()`` function was replaced, and executes the original
bytecode.

On a microbenchmark, calling the C builtin takes 95 ns, whereas the
original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling
directly ``chr(65)`` takes 76 ns.


Choose the specialized code
===========================

Pseudo-code to choose the specialized code to call a pure Python
function::

    def call_func(func, args, kwargs):
        specialized = myoptimizer.get_specialized(func)
        nspecialized = len(specialized)
        index = 0
        while index < nspecialized:
            specialized_code, guards = specialized[index]

            for guard in guards:
                check = guard(args, kwargs)
                if check:
                    break

            if not check:
                # all guards succeeded:
                # use the specialized code
                return specialized_code
            elif check == 1:
                # a guard failed temporarely:
                # try the next specialized code
                index += 1
            else:
                assert check == 2
                # a guard will always fail:
                # remove the specialized code
                del specialized[index]

        # if a guard of each specialized code failed, or if the function
        # has no specialized code, use original bytecode
        code = func.__code__



Changes
=======

Changes to the Python C API:

* Add a ``PyFuncGuardObject`` object and a ``PyFuncGuard_Type`` type
* Add a ``PySpecializedCode`` structure
* Add the following fields to the ``PyFunctionObject`` structure::

    Py_ssize_t nb_specialized;
    PySpecializedCode *specialized;

* Add function methods:

  * ``PyFunction_Specialize()``
  * ``PyFunction_GetSpecializedCodes()``
  * ``PyFunction_GetSpecializedCode()``

None of these function and types are exposed at the Python level.

All these additions are explicitly excluded of the stable ABI.

When a function code is replaced (``func.__code__ = new_code``), all
specialized codes and guards are removed.

When a function is serialized ``pickle``, specialized codes and guards are
ignored (not serialized). Specialized codes and guards are not stored in
``.pyc`` files but created and registered at runtime, when a module is
loaded.


Function guard
--------------

Add a function guard object::

    typedef struct {
        PyObject ob_base;
        int (*init) (PyObject *guard, PyObject *func);
        int (*check) (PyObject *guard, PyObject **stack, int na, int nk);
    } PyFuncGuardObject;

The ``init()`` function initializes a guard:

* Return ``0`` on success
* Return ``1`` if the guard will always fail: ``PyFunction_Specialize()``
  must ignore the specialized code
* Raise an exception and return ``-1`` on error


The ``check()`` function checks a guard:

* Return ``0`` on success
* Return ``1`` if the guard failed temporarely
* Return ``2`` if the guard will always fail: the specialized code must
  be removed
* Raise an exception and return ``-1`` on error

*stack* is an array of arguments: indexed arguments followed by (*key*,
*value*) pairs of keyword arguments. *na* is the number of indexed
arguments. *nk* is the number of keyword arguments: the number of (*key*,
*value*) pairs. `stack` contains ``na + nk * 2`` objects.


Specialized code
----------------

Add a specialized code structure::

    typedef struct {
        PyObject *code;        /* callable or code object */
        Py_ssize_t nb_guard;
        PyObject **guards;     /* PyFuncGuardObject objects */
    } PySpecializedCode;


Function methods
----------------

Add a function method to specialize the function, add a specialized code
with guards::

    int PyFunction_Specialize(PyObject *func,
                              PyObject *code, PyObject *guards)

Result:

* Return ``0`` on success
* Return ``1`` if the specialization has been ignored
* Raise an exception and return ``-1`` on error

Add a function method to get the list of specialized codes::

    PyObject* PyFunction_GetSpecializedCodes(PyObject *func)

Return a list of (*code*, *guards*) tuples where *code* is a callable or
code object and *guards* is a list of ``PyFuncGuard`` objects. Raise an
exception and return ``NULL`` on error.

Add a function method to get the specialized code::

    PyObject* PyFunction_GetSpecializedCode(PyObject *func,
                                            PyObject **stack,
                                            int na, int nk)

See ``check()`` function of guards for *stack*, *na* and *nk* arguments.
Return a callable or a code object on success. Raise an exception and
return ``NULL`` on error.

Benchmark
---------

Microbenchmark on ``python3.6 -m timeit -s 'def f(): pass' 'f()'`` (best
of 3 runs):

* Original Python: 79 ns
* Patched Python: 79 ns

According to this microbenchmark, the changes has no overhead on calling
a Python function without specialization.


Other implementations of Python
===============================

This PEP only contains changes to the Python C API, the Python API is
unchanged. Other implementations of Python are free to not implement new
additions, or implement added functions as no-op:

* ``PyFunction_Specialize()``: always return ``1`` (the specialization
  has been ignored)
* ``PyFunction_GetSpecializedCodes()``: always return an empty list
* ``PyFunction_GetSpecializedCode()``: return the function code object,
  as the existing ``PyFunction_GET_CODE()`` macro


Discussion
==========

Thread on the python-ideas mailing list: `RFC: PEP: Specialized
functions with guards
<https://mail.python.org/pipermail/python-ideas/2016-January/037703.html>`_.


Copyright
=========

This document has been placed in the public domain.

From ethan at stoneleaf.us  Tue Jan 12 17:24:50 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Jan 2016 14:24:50 -0800
Subject: [Python-Dev] PEP 509
In-Reply-To: <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
 <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
Message-ID: <56957D32.6020401@stoneleaf.us>

On 01/12/2016 01:34 PM, Victor Stinner wrote:
> 2016-01-12 19:52 GMT+01:00 Ethan Furman <ethan at stoneleaf.us>:
>> [1] We're not going to call it __version__ are we?  Seems like
>> __cache_token__ is a much better name.
>
> See the online version to the most recent version of the PEP:
> https://www.python.org/dev/peps/pep-0509/
>
> In the first version I proposed to expose the version, but I changed
> my mind, it's not private (only exposed in the C API).

Even if not exposed at the Python layer, it's still exposed when working 
at the C layer.  Is __version__ any less confusing there?  (I only work 
in C when working on Python, and only occasionally, so my question is real.)

--
~Ethan~

From tjreedy at udel.edu  Tue Jan 12 17:57:15 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 12 Jan 2016 17:57:15 -0500
Subject: [Python-Dev] PEP 509
In-Reply-To: <56957D32.6020401@stoneleaf.us>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
 <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
 <56957D32.6020401@stoneleaf.us>
Message-ID: <n740ck$fnl$1@ger.gmane.org>

On 1/12/2016 5:24 PM, Ethan Furman wrote:
> On 01/12/2016 01:34 PM, Victor Stinner wrote:
>> 2016-01-12 19:52 GMT+01:00 Ethan Furman <ethan at stoneleaf.us>:
>>> [1] We're not going to call it __version__ are we?  Seems like
>>> __cache_token__ is a much better name.

While I understand the rationale against __version__, it strikes me as a 
better description of what it is, and easier on the brain than 
__cache_token__.  Maybe there is something even better, such as 
__seqnum__.  This is literally what the attribute
is, a sequence/revision number, as with the Python repository, without 
the connotations of 'version', as in 'Python version'.  Each commit to 
CPython changes the repository state without changing the 'version'. A 
dict.update may run thru 1000s of sequence numbers, but only the final 
result is a 'version' from the programmers point of view.

-- 
Terry Jan Reedy


From victor.stinner at gmail.com  Tue Jan 12 18:24:32 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 13 Jan 2016 00:24:32 +0100
Subject: [Python-Dev] PEP 509
In-Reply-To: <56957D32.6020401@stoneleaf.us>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
 <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
 <56957D32.6020401@stoneleaf.us>
Message-ID: <CAMpsgwaA7WRQC+O4MVBr+8+tMOCcHt123TC10Mf9jJYKCH+Y6w@mail.gmail.com>

2016-01-12 23:24 GMT+01:00 Ethan Furman <ethan at stoneleaf.us>:
> Even if not exposed at the Python layer, it's still exposed when working at
> the C layer.  Is __version__ any less confusing there?  (I only work in C
> when working on Python, and only occasionally, so my question is real.)

Fields of the PyDictObject must be prefixed with "ma_". If you read
the prior art of the PEP, you will see that Antoine Pitrou also
proposed the "ma_version" name. The existing version on types used the
"version_tag" name. Maybe I should pick this one.

Dunder names like __version__ is not used in the C language.

Do you expect "__version__" to be somehow "protected" or "private"?
The field is definitely public in the Python C API, and I don't think
that it's a problem. It's not really a choice, there is no way to hide
a field of a C structure. If you start to do random things on the C
API, it's your responsability :-)

Victor

From ethan at stoneleaf.us  Tue Jan 12 18:50:56 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 12 Jan 2016 15:50:56 -0800
Subject: [Python-Dev] PEP 509
In-Reply-To: <CAMpsgwaA7WRQC+O4MVBr+8+tMOCcHt123TC10Mf9jJYKCH+Y6w@mail.gmail.com>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
 <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
 <56957D32.6020401@stoneleaf.us>
 <CAMpsgwaA7WRQC+O4MVBr+8+tMOCcHt123TC10Mf9jJYKCH+Y6w@mail.gmail.com>
Message-ID: <56959160.3020203@stoneleaf.us>

On 01/12/2016 03:24 PM, Victor Stinner wrote:
 > 2016-01-12 23:24 GMT+01:00 Ethan Furman wrote:

 >> Even if not exposed at the Python layer, it's still exposed when
 >> working at the C layer.  Is __version__ any less confusing there?
 >> (I only work in C when working on Python, and only occasionally, so
 >> my question is real.)
 >
 > Fields of the PyDictObject must be prefixed with "ma_". If you read
 > the prior art of the PEP, you will see that Antoine Pitrou also
 > proposed the "ma_version" name. The existing version on types used the
 > "version_tag" name. Maybe I should pick this one.
 >
 > Dunder names like __version__ is not used in the C language.
 >
 > Do you expect "__version__" to be somehow "protected" or "private"?

Nope, I just don't want to be misdirected when I see the name. :)  I 
think ma_version (or ma_seqnum) will be fine.

--
~Ethan~



From Eddy at Quicksall.com  Tue Jan 12 21:55:50 2016
From: Eddy at Quicksall.com (Eddy Quicksall)
Date: Tue, 12 Jan 2016 21:55:50 -0500
Subject: [Python-Dev] Building with VS2015
Message-ID: <01f401d14dad$e984f1e0$bc8ed5a0$@com>

I downloaded https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz. 

After reading Python-3.4.4\PCbuild\readme.txt I opened pcbuild.sln and
selected release/winn32 then tried a build all (F7). But that give lots of
errors and missing .h files (mostly sqlite3.h).

 

Does anyone know what I may be doing wrong or have misunderstood?

 

Here is the output:

 

1>------ Build started: Project: make_versioninfo, Configuration: Release
Win32 ------

2>------ Build started: Project: kill_python, Configuration: Release Win32
------

3>------ Skipped Build: Project: bdist_wininst, Configuration: Release Win32
------

3>Project not selected to build for this solution configuration 

4>------ Build started: Project: python3dll, Configuration: Release Win32
------

5>------ Build started: Project: xxlimited, Configuration: Release Win32
------

1>  make_versioninfo.c

5>  xxlimited.c

2>  kill_python.c

4>

4>  Microsoft (R) Program Maintenance Utility Version 14.00.23026.0

4>  Copyright (C) Microsoft Corporation.  All rights reserved.

4>

4>     lib /def:python34stub.def
/out:W:\Python-3.4.4\PCbuild\python34stub.lib /MACHINE:x86

5>LINK : fatal error LNK1104: cannot open file 'python3.lib'

4>  Microsoft (R) Library Manager Version 14.00.23026.0

4>  Copyright (C) Microsoft Corporation.  All rights reserved.

4>

4>     Creating library W:\Python-3.4.4\PCbuild\python34stub.lib and object
W:\Python-3.4.4\PCbuild\python34stub.exp

4>     cl /LD /FeW:\Python-3.4.4\PCbuild\python3.dll python3dll.c
python3.def W:\Python-3.4.4\PCbuild\python34stub.lib

4>  Microsoft (R) C/C++ Optimizing Compiler Version 19.00.23026 for x86

4>  Copyright (C) Microsoft Corporation.  All rights reserved.

4>

4>  python3dll.c

1>  make_versioninfo.vcxproj -> W:\Python-3.4.4\PCbuild\make_versioninfo.exe

2>  Generating code

2>  All 5 functions were compiled because no usable IPDB/IOBJ from previous
compilation was found.

2>  Finished generating code

4>  Microsoft (R) Incremental Linker Version 14.00.23026.0

4>  Copyright (C) Microsoft Corporation.  All rights reserved.

4>

4>  /dll

4>  /implib:W:\Python-3.4.4\PCbuild\python3.lib

4>  /out:W:\Python-3.4.4\PCbuild\python3.dll

4>  /def:python3.def

4>  python3dll.obj

4>  W:\Python-3.4.4\PCbuild\python34stub.lib

4>     Creating library W:\Python-3.4.4\PCbuild\python3.lib and object
W:\Python-3.4.4\PCbuild\python3.exp

6>------ Build started: Project: pylauncher, Configuration: Release Win32
------

7>------ Build started: Project: pywlauncher, Configuration: Release Win32
------

7>  launcher.c

6>  launcher.c

2>  kill_python.vcxproj -> W:\Python-3.4.4\PCbuild\kill_python.exe

8>------ Build started: Project: pythoncore, Configuration: Release Win32
------

9>------ Build started: Project: sqlite3, Configuration: Release Win32
------

9>  sqlite3.c

9>c1 : fatal error C1083: Cannot open source file:
'..\externals\sqlite-3.8.11.0\sqlite3.c': No such file or directory

7>  Generating code

7>  All 41 functions were compiled because no usable IPDB/IOBJ from previous
compilation was found.

7>  Finished generating code

8>  _bisectmodule.c

8>  _codecsmodule.c

6>  Generating code

6>  All 41 functions were compiled because no usable IPDB/IOBJ from previous
compilation was found.

6>  Finished generating code

8>  _collectionsmodule.c

7>  pywlauncher.vcxproj -> W:\Python-3.4.4\PCbuild\pyw.exe

8>  _csv.c

6>  pylauncher.vcxproj -> W:\Python-3.4.4\PCbuild\py.exe

8>  _functoolsmodule.c

8>  _heapqmodule.c

8>  _json.c

8>  _localemodule.c

8>  _lsprof.c

8>  _math.c

8>  _pickle.c

8>  _randommodule.c

8>  _sre.c

8>  _stat.c

8>  _struct.c

8>  _weakref.c

8>  arraymodule.c

8>  atexitmodule.c

8>  audioop.c

8>  binascii.c

8>  Generating Code...

8>  Compiling...

8>  cmathmodule.c

8>  _datetimemodule.c

8>..\Modules\_datetimemodule.c(4843): error C2065: 'daylight': undeclared
identifier

8>..\Modules\_datetimemodule.c(4844): error C2065: 'tzname': undeclared
identifier

8>..\Modules\_datetimemodule.c(4844): error C2109: subscript requires array
or pointer type

8>..\Modules\_datetimemodule.c(4846): error C2065: 'tzname': undeclared
identifier

8>..\Modules\_datetimemodule.c(4846): error C2109: subscript requires array
or pointer type

8>  errnomodule.c

8>  faulthandler.c

8>  gcmodule.c

8>  hashtable.c

8>  itertoolsmodule.c

8>  main.c

8>..\Modules\main.c(521): error C2198: 'wcstok': too few arguments for call

8>..\Modules\main.c(523): error C2198: 'wcstok': too few arguments for call

8>  mathmodule.c

8>  md5module.c

8>  mmapmodule.c

8>  _opcode.c

8>  _operator.c

8>  parsermodule.c

8>  posixmodule.c

8>  rotatingtree.c

8>  sha1module.c

8>  sha256module.c

8>  sha512module.c

8>  signalmodule.c

8>  Generating Code...

8>  Compiling...

8>  symtablemodule.c

8>  _threadmodule.c

8>  _tracemalloc.c

8>  timemodule.c

8>..\Modules\timemodule.c(1325): error C2065: 'timezone': undeclared
identifier

8>..\Modules\timemodule.c(1329): error C2065: 'timezone': undeclared
identifier

8>..\Modules\timemodule.c(1331): error C2065: 'daylight': undeclared
identifier

8>..\Modules\timemodule.c(1332): error C2065: 'tzname': undeclared
identifier

8>..\Modules\timemodule.c(1332): error C2109: subscript requires array or
pointer type

8>..\Modules\timemodule.c(1332): error C2198: 'PyUnicode_DecodeLocale': too
few arguments for call

8>..\Modules\timemodule.c(1333): error C2065: 'tzname': undeclared
identifier

8>..\Modules\timemodule.c(1333): error C2109: subscript requires array or
pointer type

8>..\Modules\timemodule.c(1333): error C2198: 'PyUnicode_DecodeLocale': too
few arguments for call

8>  xxsubtype.c

8>  zipimport.c

8>  zlibmodule.c

8>  fileio.c

8>  bytesio.c

8>  stringio.c

8>  bufferedio.c

8>..\Modules\_io\bufferedio.c(321): warning C4244: 'function': conversion
from 'double' to '__int64', possible loss of data

8>  iobase.c

8>  textio.c

8>  _iomodule.c

8>  adler32.c

8>  compress.c

8>  crc32.c

8>  deflate.c

8>  infback.c

8>  inffast.c

8>  Generating Code...

8>  Compiling...

8>  inflate.c

8>  inftrees.c

8>  trees.c

8>  uncompr.c

8>  zutil.c

8>  _codecs_cn.c

8>  _codecs_hk.c

8>  _codecs_iso2022.c

8>  _codecs_jp.c

8>  _codecs_kr.c

8>  _codecs_tw.c

8>  multibytecodec.c

8>  _winapi.c

8>..\Modules\_winapi.c(875): warning C4996: 'GetVersion': was declared
deprecated

8>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\sysinfoapi.h(110):
note: see declaration of 'GetVersion'

8>  abstract.c

8>  accu.c

8>  boolobject.c

8>  bytes_methods.c

8>  bytearrayobject.c

8>  bytesobject.c

8>  capsule.c

8>  Generating Code...

8>  Compiling...

8>  cellobject.c

8>  classobject.c

8>  codeobject.c

8>  complexobject.c

8>  descrobject.c

8>  dictobject.c

8>  enumobject.c

8>  exceptions.c

8>  fileobject.c

8>  floatobject.c

8>  frameobject.c

8>  funcobject.c

8>  genobject.c

8>  iterobject.c

8>  listobject.c

8>  longobject.c

8>  memoryobject.c

8>  methodobject.c

8>  moduleobject.c

8>  namespaceobject.c

8>  Generating Code...

8>  Compiling...

8>  object.c

8>  obmalloc.c

8>  rangeobject.c

8>  setobject.c

8>  sliceobject.c

8>  structseq.c

8>  tupleobject.c

8>  typeobject.c

8>  unicodectype.c

8>  unicodeobject.c

8>..\Objects\unicodeobject.c(15011): warning C4996: 'GetVersionExA': was
declared deprecated

8>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\sysinfoapi.h(433):
note: see declaration of 'GetVersionExA'

8>  weakrefobject.c

8>  acceler.c

8>  bitset.c

8>  firstsets.c

8>  grammar.c

8>  grammar1.c

8>  listnode.c

8>  metagrammar.c

8>  myreadline.c

8>  node.c

8>  Generating Code...

8>  Compiling...

8>  parser.c

8>  parsetok.c

8>  tokenizer.c

8>  winreg.c

8>  config.c

8>  dl_nt.c

8>  getpathp.c

8>..\PC\getpathp.c(454): error C2198: 'wcstok': too few arguments for call

8>..\PC\getpathp.c(456): error C2198: 'wcstok': too few arguments for call

8>..\PC\getpathp.c(458): error C2198: 'wcstok': too few arguments for call

8>  msvcrtmodule.c

8>  pyhash.c

8>  random.c

8>  _warnings.c

8>  asdl.c

8>  ast.c

8>  bltinmodule.c

8>  ceval.c

8>  codecs.c

8>  compile.c

8>  dynamic_annotations.c

8>  dynload_win.c

8>  errors.c

8>  Generating Code...

8>  Compiling...

8>  fileutils.c

8>  formatter_unicode.c

8>  frozen.c

8>  future.c

8>  getargs.c

8>  getcompiler.c

8>  getcopyright.c

8>  getopt.c

8>  getplatform.c

8>  getversion.c

8>  graminit.c

8>  import.c

8>  importdl.c

8>  marshal.c

8>  modsupport.c

8>  mysnprintf.c

8>  mystrtoul.c

8>  peephole.c

8>  pyarena.c

8>  pyctype.c

8>  Generating Code...

8>  Compiling...

8>  pyfpe.c

8>  pymath.c

8>  pytime.c

8>  pystate.c

8>  pystrcmp.c

8>  pystrtod.c

8>  dtoa.c

8>  Python-ast.c

8>  pythonrun.c

8>  structmember.c

8>  symtable.c

8>  sysmodule.c

8>..\Python\sysmodule.c(782): warning C4996: 'GetVersionExA': was declared
deprecated

8>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\sysinfoapi.h(433):
note: see declaration of 'GetVersionExA'

8>  thread.c

8>  traceback.c

8>  Generating Code...

10>------ Build started: Project: python, Configuration: Release Win32
------

11>------ Build started: Project: _socket, Configuration: Release Win32
------

12>------ Build started: Project: _decimal, Configuration: Release Win32
------

13>------ Build started: Project: _ctypes, Configuration: Release Win32
------

11>  socketmodule.c

12>  _decimal.c

13>  _ctypes.c

12>  basearith.c

10>  python.c

12>  constants.c

12>  context.c

11>..\Modules\socketmodule.c(866): warning C4996: 'inet_addr': Use
inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1850):
note: see declaration of 'inet_addr'

11>..\Modules\socketmodule.c(3749): warning C4996: 'WSADuplicateSocketA':
Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2872):
note: see declaration of 'WSADuplicateSocketA'

11>..\Modules\socketmodule.c(3959): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

11>..\Modules\socketmodule.c(3993): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

11>..\Modules\socketmodule.c(4399): warning C4996: 'gethostbyname': Use
getaddrinfo() or GetAddrInfoW() instead or define
_WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2238):
note: see declaration of 'gethostbyname'

11>..\Modules\socketmodule.c(4497): warning C4996: 'gethostbyaddr': Use
getnameinfo() or GetNameInfoW() instead or define
_WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2216):
note: see declaration of 'gethostbyaddr'

11>..\Modules\socketmodule.c(4627): warning C4996: 'WSADuplicateSocketA':
Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2872):
note: see declaration of 'WSADuplicateSocketA'

11>..\Modules\socketmodule.c(4632): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

11>..\Modules\socketmodule.c(4921): warning C4996: 'inet_addr': Use
inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1850):
note: see declaration of 'inet_addr'

11>..\Modules\socketmodule.c(4963): warning C4996: 'inet_ntoa': Use
inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1868):
note: see declaration of 'inet_ntoa'

11>..\Modules\socketmodule.c(5037): warning C4996: 'WSAStringToAddressA':
Use WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

12>  convolute.c

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

11>..\Modules\socketmodule.c(5175): warning C4996: 'WSAAddressToStringA':
Use WSAAddressToStringW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3556):
note: see declaration of 'WSAAddressToStringA'

11>..\Modules\socketmodule.c(5783): warning C4996: 'GetVersion': was
declared deprecated

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\sysinfoapi.h(110):
note: see declaration of 'GetVersion'

11>..\Modules\socketmodule.c(6922): warning C4996: 'inet_addr': Use
inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1850):
note: see declaration of 'inet_addr'

11>..\Modules\socketmodule.c(6941): warning C4996: 'inet_ntoa': Use
inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

11>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1868):
note: see declaration of 'inet_ntoa'

12>  crt.c

13>  callbacks.c

12>  difradix2.c

12>  fnt.c

12>  fourstep.c

12>  io.c

10>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

11>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

12>  memory.c

14>------ Build started: Project: ssl, Configuration: Release Win32 ------

15>------ Build started: Project: _ctypes_test, Configuration: Release Win32
------

13>  callproc.c

12>  mpdecimal.c

14>  '"W:\Python-3.4.4\PCbuild\python.exe"' is not recognized as an internal
or external command,

14>  operable program or batch file.

14>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5):
error MSB3073: The command "cd "W:\Python-3.4.4\PCbuild\"

14>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5):
error MSB3073: "W:\Python-3.4.4\PCbuild\python.exe" build_ssl.py Release
Win32 -a

14>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5):
error MSB3073: " exited with code 9009.

16>------ Build started: Project: _elementtree, Configuration: Release Win32
------

15>  _ctypes_test.c

12>  numbertheory.c

13>  cfield.c

15>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

17>------ Build started: Project: _msi, Configuration: Release Win32 ------

12>  sixstep.c

12>  transpose.c

12>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

18>------ Build started: Project: _sqlite3, Configuration: Release Win32
------

19>------ Build started: Project: _ssl, Configuration: Release Win32 ------

18>  cache.c

19>  _ssl.c

18>  connection.c

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  cursor.c

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  microprotocols.c

19>..\Modules\_ssl.c(59): fatal error C1083: Cannot open include file:
'openssl/rsa.h': No such file or directory

20>------ Build started: Project: _testcapi, Configuration: Release Win32
------

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  module.c

13>  ffi.c

17>  _msi.c

13>  malloc_closure.c

13>  prep_cif.c

13>  stgdict.c

13>  win32.c

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  prepare_protocol.c

17>LINK : fatal error LNK1181: cannot open input file 'fci.lib'

21>------ Build started: Project: _testimportmultiple, Configuration:
Release Win32 ------

18>  row.c

13>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

20>  _testcapimodule.c

22>------ Build started: Project: _tkinter, Configuration: Release Win32
------

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  statement.c

21>  _testimportmultiple.c

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

18>  util.c

18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error C1083:
Cannot open include file: 'sqlite3.h': No such file or directory

21>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

23>------ Build started: Project: _bz2, Configuration: Release Win32 ------

24>------ Build started: Project: select, Configuration: Release Win32
------

22>  _tkinter.c

20>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

25>------ Build started: Project: _lzma, Configuration: Release Win32 ------

25>  _lzmamodule.c

23>  _bz2module.c

25>..\Modules\_lzmamodule.c(19): fatal error C1083: Cannot open include
file: 'lzma.h': No such file or directory

26>------ Build started: Project: unicodedata, Configuration: Release Win32
------

24>  selectmodule.c

23>..\Modules\_bz2module.c(12): fatal error C1083: Cannot open include file:
'bzlib.h': No such file or directory

22>..\Modules\_tkinter.c(55): fatal error C1083: Cannot open include file:
'tcl.h': No such file or directory

23>  blocksort.c

22>  tkappinit.c

22>..\Modules\tkappinit.c(16): fatal error C1083: Cannot open include file:
'tcl.h': No such file or directory

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\blocksort.c': No such file or directory

23>  bzlib.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\bzlib.c': No such file or directory

23>  compress.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\compress.c': No such file or directory

23>  crctable.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\crctable.c': No such file or directory

23>  decompress.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\decompress.c': No such file or directory

23>  huffman.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\huffman.c': No such file or directory

23>  randtable.c

23>c1 : fatal error C1083: Cannot open source file:
'..\externals\bzip2-1.0.6\randtable.c': No such file or directory

27>------ Build started: Project: pyexpat, Configuration: Release Win32
------

24>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

26>  unicodedata.c

28>------ Build started: Project: _hashlib, Configuration: Release Win32
------

29>------ Build started: Project: _multiprocessing, Configuration: Release
Win32 ------

27>  pyexpat.c

28>  _hashopenssl.c

29>  multiprocessing.c

27>  xmlparse.c

28>..\Modules\_hashopenssl.c(22): fatal error C1083: Cannot open include
file: 'openssl/evp.h': No such file or directory

30>------ Build started: Project: pythonw, Configuration: Release Win32
------

27>  xmlrole.c

27>  xmltok.c

29>  semaphore.c

27>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

31>------ Build started: Project: winsound, Configuration: Release Win32
------

30>  WinMain.c

31>  winsound.c

29>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

32>------ Build started: Project: _testbuffer, Configuration: Release Win32
------

26>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

33>------ Skipped Build: Project: _freeze_importlib, Configuration: Release
Win32 ------

33>Project not selected to build for this solution configuration 

34>------ Build started: Project: _overlapped, Configuration: Release Win32
------

32>  _testbuffer.c

34>  overlapped.c

30>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

35>------ Build started: Project: _testembed, Configuration: Release Win32
------

35>  _testembed.c

32>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

31>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

34>..\Modules\overlapped.c(977): warning C4996: 'WSAStringToAddressA': Use
WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to
disable deprecated API warnings

34>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

34>..\Modules\overlapped.c(988): warning C4996: 'WSAStringToAddressA': Use
WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to
disable deprecated API warnings

34>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

35>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

34>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.4.4\PCbuild\python34.lib'

========== Build: 5 succeeded, 28 failed, 1 up-to-date, 2 skipped ==========

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160112/3bd1eaec/attachment-0001.html>

From tritium-list at sdamon.com  Tue Jan 12 22:32:26 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 12 Jan 2016 22:32:26 -0500
Subject: [Python-Dev] Building with VS2015
In-Reply-To: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
Message-ID: <5695C54A.3070101@sdamon.com>

This is a mailing list for the development of python itself, not support 
for building it.  That said...

3.4 uses visual studio 2010, for starters.  3.5 uses 2015.

It also looks like you have a lot of missing dependencies.

On 1/12/2016 21:55, Eddy Quicksall wrote:
>
> I downloaded https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tgz.
>
> After reading Python-3.4.4\PCbuild\readme.txt I opened pcbuild.sln and 
> selected release/winn32 then tried a build all (F7). But that give 
> lots of errors and missing .h files (mostly sqlite3.h).
>
> Does anyone know what I may be doing wrong or have misunderstood?
>
> Here is the output:
>
> 1>------ Build started: Project: make_versioninfo, Configuration: 
> Release Win32 ------
>
> 2>------ Build started: Project: kill_python, Configuration: Release 
> Win32 ------
>
> 3>------ Skipped Build: Project: bdist_wininst, Configuration: Release 
> Win32 ------
>
> 3>Project not selected to build for this solution configuration
>
> 4>------ Build started: Project: python3dll, Configuration: Release 
> Win32 ------
>
> 5>------ Build started: Project: xxlimited, Configuration: Release 
> Win32 ------
>
> 1> make_versioninfo.c
>
> 5> xxlimited.c
>
> 2> kill_python.c
>
> 4>
>
> 4> Microsoft (R) Program Maintenance Utility Version 14.00.23026.0
>
> 4> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> 4>
>
> 4>    lib /def:python34stub.def 
> /out:W:\Python-3.4.4\PCbuild\python34stub.lib /MACHINE:x86
>
> 5>LINK : fatal error LNK1104: cannot open file 'python3.lib'
>
> 4> Microsoft (R) Library Manager Version 14.00.23026.0
>
> 4> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> 4>
>
> 4> Creating library W:\Python-3.4.4\PCbuild\python34stub.lib and 
> object W:\Python-3.4.4\PCbuild\python34stub.exp
>
> 4>    cl /LD /FeW:\Python-3.4.4\PCbuild\python3.dll python3dll.c 
> python3.def W:\Python-3.4.4\PCbuild\python34stub.lib
>
> 4> Microsoft (R) C/C++ Optimizing Compiler Version 19.00.23026 for x86
>
> 4> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> 4>
>
> 4> python3dll.c
>
> 1> make_versioninfo.vcxproj -> 
> W:\Python-3.4.4\PCbuild\make_versioninfo.exe
>
> 2> Generating code
>
> 2> All 5 functions were compiled because no usable IPDB/IOBJ from 
> previous compilation was found.
>
> 2> Finished generating code
>
> 4> Microsoft (R) Incremental Linker Version 14.00.23026.0
>
> 4> Copyright (C) Microsoft Corporation.  All rights reserved.
>
> 4>
>
> 4> /dll
>
> 4> /implib:W:\Python-3.4.4\PCbuild\python3.lib
>
> 4> /out:W:\Python-3.4.4\PCbuild\python3.dll
>
> 4> /def:python3.def
>
> 4> python3dll.obj
>
> 4> W:\Python-3.4.4\PCbuild\python34stub.lib
>
> 4> Creating library W:\Python-3.4.4\PCbuild\python3.lib and object 
> W:\Python-3.4.4\PCbuild\python3.exp
>
> 6>------ Build started: Project: pylauncher, Configuration: Release 
> Win32 ------
>
> 7>------ Build started: Project: pywlauncher, Configuration: Release 
> Win32 ------
>
> 7> launcher.c
>
> 6> launcher.c
>
> 2> kill_python.vcxproj -> W:\Python-3.4.4\PCbuild\kill_python.exe
>
> 8>------ Build started: Project: pythoncore, Configuration: Release 
> Win32 ------
>
> 9>------ Build started: Project: sqlite3, Configuration: Release Win32 
> ------
>
> 9> sqlite3.c
>
> 9>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\sqlite-3.8.11.0\sqlite3.c': No such file or directory
>
> 7> Generating code
>
> 7> All 41 functions were compiled because no usable IPDB/IOBJ from 
> previous compilation was found.
>
> 7> Finished generating code
>
> 8> _bisectmodule.c
>
> 8> _codecsmodule.c
>
> 6> Generating code
>
> 6> All 41 functions were compiled because no usable IPDB/IOBJ from 
> previous compilation was found.
>
> 6> Finished generating code
>
> 8> _collectionsmodule.c
>
> 7> pywlauncher.vcxproj -> W:\Python-3.4.4\PCbuild\pyw.exe
>
> 8> _csv.c
>
> 6> pylauncher.vcxproj -> W:\Python-3.4.4\PCbuild\py.exe
>
> 8> _functoolsmodule.c
>
> 8> _heapqmodule.c
>
> 8> _json.c
>
> 8> _localemodule.c
>
> 8> _lsprof.c
>
> 8> _math.c
>
> 8> _pickle.c
>
> 8> _randommodule.c
>
> 8> _sre.c
>
> 8> _stat.c
>
> 8> _struct.c
>
> 8> _weakref.c
>
> 8> arraymodule.c
>
> 8> atexitmodule.c
>
> 8> audioop.c
>
> 8> binascii.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> cmathmodule.c
>
> 8> _datetimemodule.c
>
> 8>..\Modules\_datetimemodule.c(4843): error C2065: 'daylight': 
> undeclared identifier
>
> 8>..\Modules\_datetimemodule.c(4844): error C2065: 'tzname': 
> undeclared identifier
>
> 8>..\Modules\_datetimemodule.c(4844): error C2109: subscript requires 
> array or pointer type
>
> 8>..\Modules\_datetimemodule.c(4846): error C2065: 'tzname': 
> undeclared identifier
>
> 8>..\Modules\_datetimemodule.c(4846): error C2109: subscript requires 
> array or pointer type
>
> 8> errnomodule.c
>
> 8> faulthandler.c
>
> 8> gcmodule.c
>
> 8> hashtable.c
>
> 8> itertoolsmodule.c
>
> 8> main.c
>
> 8>..\Modules\main.c(521): error C2198: 'wcstok': too few arguments for 
> call
>
> 8>..\Modules\main.c(523): error C2198: 'wcstok': too few arguments for 
> call
>
> 8> mathmodule.c
>
> 8> md5module.c
>
> 8> mmapmodule.c
>
> 8> _opcode.c
>
> 8> _operator.c
>
> 8> parsermodule.c
>
> 8> posixmodule.c
>
> 8> rotatingtree.c
>
> 8> sha1module.c
>
> 8> sha256module.c
>
> 8> sha512module.c
>
> 8> signalmodule.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> symtablemodule.c
>
> 8> _threadmodule.c
>
> 8> _tracemalloc.c
>
> 8> timemodule.c
>
> 8>..\Modules\timemodule.c(1325): error C2065: 'timezone': undeclared 
> identifier
>
> 8>..\Modules\timemodule.c(1329): error C2065: 'timezone': undeclared 
> identifier
>
> 8>..\Modules\timemodule.c(1331): error C2065: 'daylight': undeclared 
> identifier
>
> 8>..\Modules\timemodule.c(1332): error C2065: 'tzname': undeclared 
> identifier
>
> 8>..\Modules\timemodule.c(1332): error C2109: subscript requires array 
> or pointer type
>
> 8>..\Modules\timemodule.c(1332): error C2198: 
> 'PyUnicode_DecodeLocale': too few arguments for call
>
> 8>..\Modules\timemodule.c(1333): error C2065: 'tzname': undeclared 
> identifier
>
> 8>..\Modules\timemodule.c(1333): error C2109: subscript requires array 
> or pointer type
>
> 8>..\Modules\timemodule.c(1333): error C2198: 
> 'PyUnicode_DecodeLocale': too few arguments for call
>
> 8> xxsubtype.c
>
> 8> zipimport.c
>
> 8> zlibmodule.c
>
> 8> fileio.c
>
> 8> bytesio.c
>
> 8> stringio.c
>
> 8> bufferedio.c
>
> 8>..\Modules\_io\bufferedio.c(321): warning C4244: 'function': 
> conversion from 'double' to '__int64', possible loss of data
>
> 8> iobase.c
>
> 8> textio.c
>
> 8> _iomodule.c
>
> 8> adler32.c
>
> 8> compress.c
>
> 8> crc32.c
>
> 8> deflate.c
>
> 8> infback.c
>
> 8> inffast.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> inflate.c
>
> 8> inftrees.c
>
> 8> trees.c
>
> 8> uncompr.c
>
> 8> zutil.c
>
> 8> _codecs_cn.c
>
> 8> _codecs_hk.c
>
> 8> _codecs_iso2022.c
>
> 8> _codecs_jp.c
>
> 8> _codecs_kr.c
>
> 8> _codecs_tw.c
>
> 8> multibytecodec.c
>
> 8> _winapi.c
>
> 8>..\Modules\_winapi.c(875): warning C4996: 'GetVersion': was declared 
> deprecated
>
> 8> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\sysinfoapi.h(110): note: see declaration of 
> 'GetVersion'
>
> 8> abstract.c
>
> 8> accu.c
>
> 8> boolobject.c
>
> 8> bytes_methods.c
>
> 8> bytearrayobject.c
>
> 8> bytesobject.c
>
> 8> capsule.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> cellobject.c
>
> 8> classobject.c
>
> 8> codeobject.c
>
> 8> complexobject.c
>
> 8> descrobject.c
>
> 8> dictobject.c
>
> 8> enumobject.c
>
> 8> exceptions.c
>
> 8> fileobject.c
>
> 8> floatobject.c
>
> 8> frameobject.c
>
> 8> funcobject.c
>
> 8> genobject.c
>
> 8> iterobject.c
>
> 8> listobject.c
>
> 8> longobject.c
>
> 8> memoryobject.c
>
> 8> methodobject.c
>
> 8> moduleobject.c
>
> 8> namespaceobject.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> object.c
>
> 8> obmalloc.c
>
> 8> rangeobject.c
>
> 8> setobject.c
>
> 8> sliceobject.c
>
> 8> structseq.c
>
> 8> tupleobject.c
>
> 8> typeobject.c
>
> 8> unicodectype.c
>
> 8> unicodeobject.c
>
> 8>..\Objects\unicodeobject.c(15011): warning C4996: 'GetVersionExA': 
> was declared deprecated
>
> 8> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\sysinfoapi.h(433): note: see declaration of 
> 'GetVersionExA'
>
> 8> weakrefobject.c
>
> 8> acceler.c
>
> 8> bitset.c
>
> 8> firstsets.c
>
> 8> grammar.c
>
> 8> grammar1.c
>
> 8> listnode.c
>
> 8> metagrammar.c
>
> 8> myreadline.c
>
> 8> node.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8>  parser.c
>
> 8> parsetok.c
>
> 8> tokenizer.c
>
> 8> winreg.c
>
> 8> config.c
>
> 8> dl_nt.c
>
> 8> getpathp.c
>
> 8>..\PC\getpathp.c(454): error C2198: 'wcstok': too few arguments for call
>
> 8>..\PC\getpathp.c(456): error C2198: 'wcstok': too few arguments for call
>
> 8>..\PC\getpathp.c(458): error C2198: 'wcstok': too few arguments for call
>
> 8> msvcrtmodule.c
>
> 8> pyhash.c
>
> 8> random.c
>
> 8> _warnings.c
>
> 8> asdl.c
>
> 8> ast.c
>
> 8> bltinmodule.c
>
> 8> ceval.c
>
> 8> codecs.c
>
> 8> compile.c
>
> 8> dynamic_annotations.c
>
> 8> dynload_win.c
>
> 8> errors.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> fileutils.c
>
> 8> formatter_unicode.c
>
> 8> frozen.c
>
> 8> future.c
>
> 8> getargs.c
>
> 8> getcompiler.c
>
> 8> getcopyright.c
>
> 8> getopt.c
>
> 8> getplatform.c
>
> 8> getversion.c
>
> 8> graminit.c
>
> 8> import.c
>
> 8> importdl.c
>
> 8> marshal.c
>
> 8> modsupport.c
>
> 8> mysnprintf.c
>
> 8> mystrtoul.c
>
> 8> peephole.c
>
> 8> pyarena.c
>
> 8> pyctype.c
>
> 8> Generating Code...
>
> 8> Compiling...
>
> 8> pyfpe.c
>
> 8> pymath.c
>
> 8> pytime.c
>
> 8> pystate.c
>
> 8> pystrcmp.c
>
> 8> pystrtod.c
>
> 8> dtoa.c
>
> 8> Python-ast.c
>
> 8> pythonrun.c
>
> 8> structmember.c
>
> 8> symtable.c
>
> 8> sysmodule.c
>
> 8>..\Python\sysmodule.c(782): warning C4996: 'GetVersionExA': was 
> declared deprecated
>
> 8> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\sysinfoapi.h(433): note: see declaration of 
> 'GetVersionExA'
>
> 8> thread.c
>
> 8> traceback.c
>
> 8> Generating Code...
>
> 10>------ Build started: Project: python, Configuration: Release Win32 
> ------
>
> 11>------ Build started: Project: _socket, Configuration: Release 
> Win32 ------
>
> 12>------ Build started: Project: _decimal, Configuration: Release 
> Win32 ------
>
> 13>------ Build started: Project: _ctypes, Configuration: Release 
> Win32 ------
>
> 11> socketmodule.c
>
> 12> _decimal.c
>
> 13> _ctypes.c
>
> 12> basearith.c
>
> 10> python.c
>
> 12> constants.c
>
> 12> context.c
>
> 11>..\Modules\socketmodule.c(866): warning C4996: 'inet_addr': Use 
> inet_pton() or InetPton() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(1850): note: see declaration of 'inet_addr'
>
> 11>..\Modules\socketmodule.c(3749): warning C4996: 
> 'WSADuplicateSocketA': Use WSADuplicateSocketW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(2872): note: see declaration of 
> 'WSADuplicateSocketA'
>
> 11>..\Modules\socketmodule.c(3959): warning C4996: 'WSASocketA': Use 
> WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to 
> disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3457): note: see declaration of 
> 'WSASocketA'
>
> 11>..\Modules\socketmodule.c(3993): warning C4996: 'WSASocketA': Use 
> WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to 
> disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3457): note: see declaration of 
> 'WSASocketA'
>
> 11>..\Modules\socketmodule.c(4399): warning C4996: 'gethostbyname': 
> Use getaddrinfo() or GetAddrInfoW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(2238): note: see declaration of 
> 'gethostbyname'
>
> 11>..\Modules\socketmodule.c(4497): warning C4996: 'gethostbyaddr': 
> Use getnameinfo() or GetNameInfoW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(2216): note: see declaration of 
> 'gethostbyaddr'
>
> 11>..\Modules\socketmodule.c(4627): warning C4996: 
> 'WSADuplicateSocketA': Use WSADuplicateSocketW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(2872): note: see declaration of 
> 'WSADuplicateSocketA'
>
> 11>..\Modules\socketmodule.c(4632): warning C4996: 'WSASocketA': Use 
> WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to 
> disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3457): note: see declaration of 
> 'WSASocketA'
>
> 11>..\Modules\socketmodule.c(4921): warning C4996: 'inet_addr': Use 
> inet_pton() or InetPton() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(1850): note: see declaration of 'inet_addr'
>
> 11>..\Modules\socketmodule.c(4963): warning C4996: 'inet_ntoa': Use 
> inet_ntop() or InetNtop() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(1868): note: see declaration of 'inet_ntoa'
>
> 11>..\Modules\socketmodule.c(5037): warning C4996: 
> 'WSAStringToAddressA': Use WSAStringToAddressW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 12> convolute.c
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3623): note: see declaration of 
> 'WSAStringToAddressA'
>
> 11>..\Modules\socketmodule.c(5175): warning C4996: 
> 'WSAAddressToStringA': Use WSAAddressToStringW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3556): note: see declaration of 
> 'WSAAddressToStringA'
>
> 11>..\Modules\socketmodule.c(5783): warning C4996: 'GetVersion': was 
> declared deprecated
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\sysinfoapi.h(110): note: see declaration of 
> 'GetVersion'
>
> 11>..\Modules\socketmodule.c(6922): warning C4996: 'inet_addr': Use 
> inet_pton() or InetPton() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(1850): note: see declaration of 'inet_addr'
>
> 11>..\Modules\socketmodule.c(6941): warning C4996: 'inet_ntoa': Use 
> inet_ntop() or InetNtop() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 11> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(1868): note: see declaration of 'inet_ntoa'
>
> 12> crt.c
>
> 13> callbacks.c
>
> 12> difradix2.c
>
> 12> fnt.c
>
> 12> fourstep.c
>
> 12> io.c
>
> 10>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 11>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 12> memory.c
>
> 14>------ Build started: Project: ssl, Configuration: Release Win32 ------
>
> 15>------ Build started: Project: _ctypes_test, Configuration: Release 
> Win32 ------
>
> 13> callproc.c
>
> 12> mpdecimal.c
>
> 14> '"W:\Python-3.4.4\PCbuild\python.exe"' is not recognized as an 
> internal or external command,
>
> 14> operable program or batch file.
>
> 14>C:\Program Files 
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5): error 
> MSB3073: The command "cd "W:\Python-3.4.4\PCbuild\"
>
> 14>C:\Program Files 
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5): error 
> MSB3073: "W:\Python-3.4.4\PCbuild\python.exe" build_ssl.py Release 
> Win32 -a
>
> 14>C:\Program Files 
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(37,5): error 
> MSB3073: " exited with code 9009.
>
> 16>------ Build started: Project: _elementtree, Configuration: Release 
> Win32 ------
>
> 15> _ctypes_test.c
>
> 12> numbertheory.c
>
> 13> cfield.c
>
> 15>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 17>------ Build started: Project: _msi, Configuration: Release Win32 
> ------
>
> 12> sixstep.c
>
> 12> transpose.c
>
> 12>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 18>------ Build started: Project: _sqlite3, Configuration: Release 
> Win32 ------
>
> 19>------ Build started: Project: _ssl, Configuration: Release Win32 
> ------
>
> 18> cache.c
>
> 19> _ssl.c
>
> 18> connection.c
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> cursor.c
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> microprotocols.c
>
> 19>..\Modules\_ssl.c(59): fatal error C1083: Cannot open include file: 
> 'openssl/rsa.h': No such file or directory
>
> 20>------ Build started: Project: _testcapi, Configuration: Release 
> Win32 ------
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> module.c
>
> 13> ffi.c
>
> 17> _msi.c
>
> 13> malloc_closure.c
>
> 13> prep_cif.c
>
> 13> stgdict.c
>
> 13> win32.c
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> prepare_protocol.c
>
> 17>LINK : fatal error LNK1181: cannot open input file 'fci.lib'
>
> 21>------ Build started: Project: _testimportmultiple, Configuration: 
> Release Win32 ------
>
> 18> row.c
>
> 13>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 20> _testcapimodule.c
>
> 22>------ Build started: Project: _tkinter, Configuration: Release 
> Win32 ------
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> statement.c
>
> 21> _testimportmultiple.c
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 18> util.c
>
> 18>w:\python-3.4.4\modules\_sqlite\connection.h(33): fatal error 
> C1083: Cannot open include file: 'sqlite3.h': No such file or directory
>
> 21>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 23>------ Build started: Project: _bz2, Configuration: Release Win32 
> ------
>
> 24>------ Build started: Project: select, Configuration: Release Win32 
> ------
>
> 22> _tkinter.c
>
> 20>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 25>------ Build started: Project: _lzma, Configuration: Release Win32 
> ------
>
> 25> _lzmamodule.c
>
> 23> _bz2module.c
>
> 25>..\Modules\_lzmamodule.c(19): fatal error C1083: Cannot open 
> include file: 'lzma.h': No such file or directory
>
> 26>------ Build started: Project: unicodedata, Configuration: Release 
> Win32 ------
>
> 24> selectmodule.c
>
> 23>..\Modules\_bz2module.c(12): fatal error C1083: Cannot open include 
> file: 'bzlib.h': No such file or directory
>
> 22>..\Modules\_tkinter.c(55): fatal error C1083: Cannot open include 
> file: 'tcl.h': No such file or directory
>
> 23> blocksort.c
>
> 22> tkappinit.c
>
> 22>..\Modules\tkappinit.c(16): fatal error C1083: Cannot open include 
> file: 'tcl.h': No such file or directory
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\blocksort.c': No such file or directory
>
> 23> bzlib.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\bzlib.c': No such file or directory
>
> 23> compress.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\compress.c': No such file or directory
>
> 23> crctable.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\crctable.c': No such file or directory
>
> 23> decompress.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\decompress.c': No such file or directory
>
> 23> huffman.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\huffman.c': No such file or directory
>
> 23> randtable.c
>
> 23>c1 : fatal error C1083: Cannot open source file: 
> '..\externals\bzip2-1.0.6\randtable.c': No such file or directory
>
> 27>------ Build started: Project: pyexpat, Configuration: Release 
> Win32 ------
>
> 24>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 26> unicodedata.c
>
> 28>------ Build started: Project: _hashlib, Configuration: Release 
> Win32 ------
>
> 29>------ Build started: Project: _multiprocessing, Configuration: 
> Release Win32 ------
>
> 27> pyexpat.c
>
> 28> _hashopenssl.c
>
> 29> multiprocessing.c
>
> 27> xmlparse.c
>
> 28>..\Modules\_hashopenssl.c(22): fatal error C1083: Cannot open 
> include file: 'openssl/evp.h': No such file or directory
>
> 30>------ Build started: Project: pythonw, Configuration: Release 
> Win32 ------
>
> 27> xmlrole.c
>
> 27> xmltok.c
>
> 29> semaphore.c
>
> 27>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 31>------ Build started: Project: winsound, Configuration: Release 
> Win32 ------
>
> 30> WinMain.c
>
> 31> winsound.c
>
> 29>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 32>------ Build started: Project: _testbuffer, Configuration: Release 
> Win32 ------
>
> 26>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 33>------ Skipped Build: Project: _freeze_importlib, Configuration: 
> Release Win32 ------
>
> 33>Project not selected to build for this solution configuration
>
> 34>------ Build started: Project: _overlapped, Configuration: Release 
> Win32 ------
>
> 32> _testbuffer.c
>
> 34> overlapped.c
>
> 30>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 35>------ Build started: Project: _testembed, Configuration: Release 
> Win32 ------
>
> 35> _testembed.c
>
> 32>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 31>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 34>..\Modules\overlapped.c(977): warning C4996: 'WSAStringToAddressA': 
> Use WSAStringToAddressW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 34> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3623): note: see declaration of 
> 'WSAStringToAddressA'
>
> 34>..\Modules\overlapped.c(988): warning C4996: 'WSAStringToAddressA': 
> Use WSAStringToAddressW() instead or define 
> _WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings
>
> 34> C:\Program Files (x86)\Windows 
> Kits\8.1\Include\um\winsock2.h(3623): note: see declaration of 
> 'WSAStringToAddressA'
>
> 35>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> 34>LINK : fatal error LNK1181: cannot open input file 
> 'W:\Python-3.4.4\PCbuild\python34.lib'
>
> ========== Build: 5 succeeded, 28 failed, 1 up-to-date, 2 skipped 
> ==========
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160112/d17e6bde/attachment-0001.html>

From zachary.ware+pydev at gmail.com  Tue Jan 12 23:03:47 2016
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Tue, 12 Jan 2016 22:03:47 -0600
Subject: [Python-Dev] Building with VS2015
In-Reply-To: <5695C54A.3070101@sdamon.com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
Message-ID: <CAKJDb-OHsXnKX-R+KDL4RE3cZrN67R-Xj5EEs_NvZcgEurXvOw@mail.gmail.com>

On Tue, Jan 12, 2016 at 9:32 PM, Alexander Walters
<tritium-list at sdamon.com> wrote:
> This is a mailing list for the development of python itself, not support for
> building it.  That said...
>
> 3.4 uses visual studio 2010, for starters.  3.5 uses 2015.

Agreed with all of the above.

You'll be much happier using either 3.5 with VS 2015, or 3.4 with VS
2010.  You might coerce things the other way around, but you won't get
official support for it and you won't match the rest of the world (in
particular, binary packages from PyPI won't work).

> It also looks like you have a lot of missing dependencies.

Run `PCbuild\get_externals.bat` after installing Subversion (and
ensuring that svn.exe is on PATH) to resolve the missing dependencies.
The other errors will require patching the source, and it's too late
in the 3.4 release cycle for those patches to be accepted for 3.4.5.
I also find it much simpler to use `PCbuild\build.bat` instead of
building from the VS GUI.  A 32bit Release build with all the optional
pieces can be accomplished with `PCbuild\build.bat -e`; add '-p x64'
for a 64bit build.

-- 
Zach

From paulson at busiq.com  Wed Jan 13 14:32:23 2016
From: paulson at busiq.com (Matthew Paulson)
Date: Wed, 13 Jan 2016 14:32:23 -0500
Subject: [Python-Dev] Discussion related to memory leaks requested
Message-ID: <5696A647.6090501@busiq.com>

Hi:

I've spent some time performing memory leak analysis while using Python 
in an embedded configuration.

The pattern is:

    Py_Initialize();

    ... run empty python source file ...

    Py_Finalize();


I've identified several suspect areas including dictionary maitenace in 
import.c:~ 414

     /* Clear the modules dict. */
     PyDict_Clear(modules);
     /* Restore the original builtins dict, to ensure that any
        user data gets cleared. */
     dict = PyDict_Copy(interp->builtins);
     if (dict == NULL)
         PyErr_Clear();
     PyDict_Clear(interp->builtins);
     if (PyDict_Update(interp->builtins, interp->builtins_copy))
         PyErr_Clear();
     Py_XDECREF(dict);
     /* Clear module dict copies stored in the interpreter state */


Is there someone in the group that would like to discuss this topic.  
There seems to be other leaks as well.  I'm new to Python-dev, but 
willing to help or work with someone who is more familiar with these 
areas than I.

Thanks,

Matt


-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/f44fba42/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/f44fba42/attachment.jpe>

From brett at python.org  Wed Jan 13 15:06:11 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Jan 2016 20:06:11 +0000
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696A647.6090501@busiq.com>
References: <5696A647.6090501@busiq.com>
Message-ID: <CAP1=2W5h-oMztyEoEykPz5Tx8ck1V3M-8pkjCCR1EYJ1sfBsvw@mail.gmail.com>

Probably the best way to handle this, Matthew, is to open issues at
bugs.python.org for each of the leaks you have found and then they can be
discussed there.

And thanks for being willing to report these!

On Wed, 13 Jan 2016 at 11:42 Matthew Paulson <paulson at busiq.com> wrote:

> Hi:
>
> I've spent some time performing memory leak analysis while using Python in
> an embedded configuration.
>
> The pattern is:
>
> Py_Initialize();
>
> ... run empty python source file ...
>
> Py_Finalize();
>
>
> I've identified several suspect areas including dictionary maitenace in
> import.c:~ 414
>
>     /* Clear the modules dict. */
>     PyDict_Clear(modules);
>     /* Restore the original builtins dict, to ensure that any
>        user data gets cleared. */
>     dict = PyDict_Copy(interp->builtins);
>     if (dict == NULL)
>         PyErr_Clear();
>     PyDict_Clear(interp->builtins);
>     if (PyDict_Update(interp->builtins, interp->builtins_copy))
>         PyErr_Clear();
>     Py_XDECREF(dict);
>     /* Clear module dict copies stored in the interpreter state */
>
>
> Is there someone in the group that would like to discuss this topic.
> There seems to be other leaks as well.  I'm new to Python-dev, but willing
> to help or work with someone who is more familiar with these areas than I.
>
> Thanks,
>
> Matt
>
>
> --
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/172001b8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/172001b8/attachment-0002.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/172001b8/attachment-0003.jpe>

From vadmium+py at gmail.com  Wed Jan 13 16:15:29 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Wed, 13 Jan 2016 21:15:29 +0000
Subject: [Python-Dev] Modifying the self-signed.pythontest.net certificate
Message-ID: <CA+eR4cF61-SkCO7-2eMOutnfHtk_CDGHuBJQBvSYxx1ARhj=hg@mail.gmail.com>

In order to fix the SSL test suite
<https://bugs.python.org/issue25940>, I would like to modify the
certificate used by https://self-signed.pythontest.net. So far I have
a patch ready for the pythontestdotnet repository, but I want to know
if I can just push to that repository, or if other steps are required.

Judging by <https://bugs.python.org/issue22650>, Georg Brandl was
involved in setting the server up. So far I haven?t heard anything
from him on the SSL tests bug, so I am asking here in case somebody
else can help.

From Eddy at Quicksall.com  Wed Jan 13 15:05:11 2016
From: Eddy at Quicksall.com (Eddy Quicksall)
Date: Wed, 13 Jan 2016 15:05:11 -0500
Subject: [Python-Dev] Debugging using VS 2015
In-Reply-To: <5695C54A.3070101@sdamon.com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
Message-ID: <003201d14e3d$b6560f80$23022e80$@com>

I am using 3.5.1. I'm adding an extension for my special case. 

 

I know this list is for development and that's what I'm doing but I would
like to use VS 2015 to do my debugging. If there is another list I should
use to get the build to work then please let me know the list.

 

I have been able to build using "PCbuild\build.bat -e" but I'm not able to
build using PCbuild\pcbuild.sln. It gives lots of errors.

 

I'm using Debug/Win32.

 

Below is the output. Note that lots of .c files are missing .. Is there
another package I should get (but then why would it build correctly with
build.bat)? It could be that I need to add the equivalent to the "-e"
option.

 

1>------ Rebuild All started: Project: pythoncore, Configuration: Debug
Win32 ------

2>------ Rebuild All started: Project: tcl, Configuration: Debug Win32
------

3>------ Rebuild All started: Project: ssleay, Configuration: Debug Win32
------

4>------ Rebuild All started: Project: sqlite3, Configuration: Debug Win32
------

1>  Killing any running python_d.exe instances...

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The
command "hg id -b >
"W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited
with code 9009.

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The
command "hg id -i >
"W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited
with code 9009.

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The
command "hg id -t >
"W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited with
code 9009.

2>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
Skipping...

5>------ Rebuild All started: Project: python3dll, Configuration: Debug
Win32 ------

1>  _bisectmodule.c

5>  python3dll.c

4>  sqlite3.c

3>  d1_both.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or
directory

3>  d1_lib.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or
directory

1>  _codecsmodule.c

3>  d1_pkt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or
directory

3>  d1_srtp.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or
directory

3>  s2_clnt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or
directory

3>  s2_enc.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or
directory

3>  s2_lib.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or
directory

3>  s2_meth.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or
directory

3>  s2_pkt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_pkt.c': No such file or
directory

3>  s2_srvr.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_srvr.c': No such file or
directory

3>  s23_clnt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_clnt.c': No such file or
directory

3>  s23_lib.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_lib.c': No such file or
directory

3>  s23_meth.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_meth.c': No such file or
directory

3>  s23_pkt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_pkt.c': No such file or
directory

3>  s23_srvr.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_srvr.c': No such file or
directory

3>  s3_both.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_both.c': No such file or
directory

3>  s3_cbc.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_cbc.c': No such file or
directory

3>  s3_clnt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_clnt.c': No such file or
directory

3>  s3_enc.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_enc.c': No such file or
directory

3>  s3_lib.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_lib.c': No such file or
directory

3>  Generating Code...

3>  Compiling...

3>  s3_meth.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_meth.c': No such file or
directory

3>  s3_pkt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_pkt.c': No such file or
directory

3>  s3_srvr.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_srvr.c': No such file or
directory

3>  ssl_algs.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_algs.c': No such file or
directory

3>  ssl_asn1.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_asn1.c': No such file or
directory

3>  ssl_cert.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_cert.c': No such file or
directory

3>  ssl_ciph.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_ciph.c': No such file or
directory

3>  ssl_err.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err.c': No such file or
directory

3>  ssl_err2.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err2.c': No such file or
directory

3>  ssl_lib.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_lib.c': No such file or
directory

3>  ssl_rsa.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_rsa.c': No such file or
directory

3>  ssl_sess.c

1>  _collectionsmodule.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_sess.c': No such file or
directory

3>  t1_clnt.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_clnt.c': No such file or
directory

3>  t1_enc.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_enc.c': No such file or
directory

3>  t1_lib.c

1>  _csv.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_lib.c': No such file or
directory

3>  t1_meth.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_meth.c': No such file or
directory

3>  t1_reneg.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_reneg.c': No such file or
directory

3>  t1_srvr.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_srvr.c': No such file or
directory

3>  tls_srp.c

3>c1 : fatal error C1083: Cannot open source file:
'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\tls_srp.c': No such file or
directory

3>  Generating Code...

6>------ Rebuild All started: Project: tk, Configuration: Debug Win32 ------

1>  _functoolsmodule.c

1>  _heapqmodule.c

1>  _json.c

1>  _localemodule.c

4>     Creating library W:\Python-3.5.1\PCBuild\win32\sqlite3_d.lib and
object W:\Python-3.5.1\PCBuild\win32\sqlite3_d.exp

5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_dstub.lib and
object W:\Python-3.5.1\PCBuild\win32\python3_dstub.exp

1>  _lsprof.c

5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_d.lib and
object W:\Python-3.5.1\PCBuild\win32\python3_d.exp

5>  python3dll.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python3_d.dll

7>------ Rebuild All started: Project: libeay, Configuration: Debug Win32
------

6>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
Skipping...

8>------ Skipped Rebuild All: Project: bdist_wininst, Configuration: Debug
Win32 ------

8>Project not selected to build for this solution configuration 

9>------ Skipped Rebuild All: Project: xxlimited, Configuration: Release
Win32 ------

9>Project not selected to build for this solution configuration 

10>------ Rebuild All started: Project: pylauncher, Configuration: Debug
Win32 ------

1>  _math.c

4>  sqlite3.vcxproj -> W:\Python-3.5.1\PCBuild\win32\sqlite3_d.dll

11>------ Rebuild All started: Project: pywlauncher, Configuration: Debug
Win32 ------

10>  launcher.c

1>  _pickle.c

7>  nasm: fatal: unable to open input file
`W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm'

1>  _randommodule.c

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: The command
"setlocal

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: set
PATH=W:\Python-3.5.1\externals\\nasm-2.11.06\;%PATH%

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: nasm.exe -f
win32 -o
"W:\Python-3.5.1\externals\openssl-1.0.2d\tmp\\win32_Debug\libeay\aes-586.ob
j" "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm"" exited with
code 1.

12>------ Rebuild All started: Project: tix, Configuration: Debug Win32
------

11>  launcher.c

1>  _sre.c

1>  _stat.c

1>  _struct.c

12>    md Debug_VC13

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixClass.obj ..\generic\tixClass.c

12>  tixClass.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixCmds.obj ..\generic\tixCmds.c

12>  tixCmds.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixCompat.obj ..\generic\tixCompat.c

12>  tixCompat.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDiImg.obj ..\generic\tixDiImg.c

12>  tixDiImg.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDiITxt.obj ..\generic\tixDiITxt.c

12>  tixDiITxt.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDiStyle.obj ..\generic\tixDiStyle.c

12>  tixDiStyle.c

11>  pywlauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pyw_d.exe

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDItem.obj ..\generic\tixDItem.c

12>  tixDItem.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDiText.obj ..\generic\tixDiText.c

12>  tixDiText.c

1>  _weakref.c

1>  arraymodule.c

1>  atexitmodule.c

1>  audioop.c

1>  binascii.c

1>  Generating Code...

1>  Compiling...

1>  cmathmodule.c

1>  _datetimemodule.c

1>  errnomodule.c

1>  faulthandler.c

1>  gcmodule.c

10>  pylauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\py_d.exe

1>  hashtable.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixDiWin.obj ..\generic\tixDiWin.c

12>  tixDiWin.c

1>  itertoolsmodule.c

1>  main.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixError.obj ..\generic\tixError.c

12>  tixError.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixForm.obj ..\generic\tixForm.c

1>  mathmodule.c

12>  tixForm.c

1>  md5module.c

1>  mmapmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixFormMisc.obj ..\generic\tixFormMisc.c

12>  tixFormMisc.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGeometry.obj ..\generic\tixGeometry.c

12>  tixGeometry.c

1>  _opcode.c

1>  _operator.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrid.obj ..\generic\tixGrid.c

1>  parsermodule.c

12>  tixGrid.c

1>  posixmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrData.obj ..\generic\tixGrData.c

12>  tixGrData.c

1>  rotatingtree.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrRC.obj ..\generic\tixGrRC.c

1>  sha1module.c

12>  tixGrRC.c

1>  sha256module.c

1>  sha512module.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrFmt.obj ..\generic\tixGrFmt.c

12>  tixGrFmt.c

1>  signalmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrSel.obj ..\generic\tixGrSel.c

12>  tixGrSel.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixGrUtl.obj ..\generic\tixGrUtl.c

12>  tixGrUtl.c

1>  Compiling...

1>  symtablemodule.c

1>  _threadmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixHLCol.obj ..\generic\tixHLCol.c

12>  tixHLCol.c

1>  _tracemalloc.c

1>  timemodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixHLHdr.obj ..\generic\tixHLHdr.c

12>  tixHLHdr.c

1>  xxsubtype.c

1>  zipimport.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixHLInd.obj ..\generic\tixHLInd.c

12>  tixHLInd.c

1>  zlibmodule.c

1>  fileio.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixHList.obj ..\generic\tixHList.c

12>  tixHList.c

1>  bytesio.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixImgCmp.obj ..\generic\tixImgCmp.c

12>  tixImgCmp.c

1>  stringio.c

1>  bufferedio.c

12>..\generic\tixImgCmp.c(169): warning C4090: 'function': different 'const'
qualifiers

12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 2 different
from declaration

12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 5 different
from declaration

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixImgXpm.obj ..\generic\tixImgXpm.c

12>  tixImgXpm.c

1>  iobase.c

1>  textio.c

12>..\generic\tixImgXpm.c(88): warning C4090: 'function': different 'const'
qualifiers

12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 2 different
from declaration

12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 5 different
from declaration

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixInit.obj ..\generic\tixInit.c

1>  _iomodule.c

12>  tixInit.c

1>  adler32.c

1>  compress.c

1>  crc32.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixList.obj ..\generic\tixList.c

1>  deflate.c

12>  tixList.c

1>  infback.c

1>  inffast.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixMethod.obj ..\generic\tixMethod.c

1>  Compiling...

1>  inflate.c

12>  tixMethod.c

1>  inftrees.c

1>  trees.c

1>  uncompr.c

1>  zutil.c

1>  _codecs_cn.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixNBFrame.obj ..\generic\tixNBFrame.c

12>  tixNBFrame.c

1>  _codecs_hk.c

1>  _codecs_iso2022.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixOption.obj ..\generic\tixOption.c

12>  tixOption.c

1>  _codecs_jp.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixSmpLs.obj ..\generic\tixSmpLs.c

1>  _codecs_kr.c

12>  tixSmpLs.c

1>  _codecs_tw.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixScroll.obj ..\generic\tixScroll.c

12>  tixScroll.c

1>  multibytecodec.c

1>  _winapi.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixTList.obj ..\generic\tixTList.c

12>  tixTList.c

1>  abstract.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixUtils.obj ..\generic\tixUtils.c

12>  tixUtils.c

1>  accu.c

1>  boolobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixWCmpt.obj ..\win\tixWCmpt.c

1>  bytes_methods.c

12>  tixWCmpt.c

1>  bytearrayobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixWidget.obj ..\generic\tixWidget.c

12>  tixWidget.c

1>  bytesobject.c

1>  capsule.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixWinDraw.obj ..\win\tixWinDraw.c

12>  tixWinDraw.c

1>  Compiling...

1>  cellobject.c

1>  classobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixWinXpm.obj ..\win\tixWinXpm.c

12>  tixWinXpm.c

1>  codeobject.c

1>  complexobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
-MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
Studio\VC98\include"  -I..\win -I..\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tk-8.6.4.2\win
-IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
-IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
-D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
-FoDebug_VC13\tixWinWm.obj ..\win\tixWinWm.c

1>  descrobject.c

12>  tixWinWm.c

1>  dictobject.c

12>    link.exe -debug -debugtype:cv /RELEASE /NOLOGO /MACHINE:IX86 -dll
W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib
W:\Python-3.5.1\externals\tcl-core-8.6.4.2\win\Debug_VC13\tclstub86.lib
kernel32.lib  advapi32.lib user32.lib gdi32.lib comdlg32.lib
-out:Debug_VC13\tix84g.dll @C:\Users\eddyq\AppData\Local\Temp\nmF029.tmp

12>LINK : fatal error LNK1181: cannot open input file
'W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib'

12>NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual
Studio 14.0\VC\bin\link.exe"' : return code '0x49d'

12>  Stop.

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: The command "setlocal

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: if not exist
"W:\Python-3.5.1\externals\tcltk\lib\tix8.4.3\tix84g.dll" goto build

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: goto :eof

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: :build

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: set VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual
Studio 14.0\VC\

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: cd /D "W:\Python-3.5.1\externals\tix-8.4.3.6\win"

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: nmake /nologo -f makefile.vc MACHINE=IX86 DEBUG=1 NODEBUG=0
TCL_DBGX=g TK_DBGX=g TCL_MAJOR=8 TCL_MINOR=6 TCL_PATCH=4
BUILDDIRTOP="Debug_VC13"
TCL_DIR="W:\Python-3.5.1\externals\tcl-core-8.6.4.2"
TK_DIR="W:\Python-3.5.1\externals\tk-8.6.4.2"
INSTALL_DIR="W:\Python-3.5.1\externals\tcltk" all install

12>C:\Program Files
(x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
error MSB3073: " exited with code 2.

1>  enumobject.c

1>  exceptions.c

1>  fileobject.c

1>  floatobject.c

1>  frameobject.c

1>  funcobject.c

1>  genobject.c

1>  iterobject.c

1>  listobject.c

1>  longobject.c

1>  memoryobject.c

1>  methodobject.c

1>  moduleobject.c

1>  namespaceobject.c

1>  Generating Code...

1>  Compiling...

1>  object.c

1>  obmalloc.c

1>  odictobject.c

1>  rangeobject.c

1>  setobject.c

1>  sliceobject.c

1>  structseq.c

1>  tupleobject.c

1>  typeobject.c

1>  unicodectype.c

1>  unicodeobject.c

1>  weakrefobject.c

1>  acceler.c

1>  bitset.c

1>  firstsets.c

1>  grammar.c

1>  grammar1.c

1>  listnode.c

1>  metagrammar.c

1>  myreadline.c

1>  Generating Code...

1>  Compiling...

1>  node.c

1>  parser.c

1>  parsetok.c

1>  tokenizer.c

1>  invalid_parameter_handler.c

1>  winreg.c

1>  config.c

1>  getpathp.c

1>  msvcrtmodule.c

1>  pyhash.c

1>  random.c

1>  _warnings.c

1>  asdl.c

1>  ast.c

1>  bltinmodule.c

1>  ceval.c

1>  codecs.c

1>  compile.c

1>  dynamic_annotations.c

1>  dynload_win.c

1>  Generating Code...

1>  Compiling...

1>  errors.c

1>  fileutils.c

1>  formatter_unicode.c

1>  frozen.c

1>  future.c

1>  getargs.c

1>  getcompiler.c

1>  getcopyright.c

1>  getopt.c

1>  getplatform.c

1>  getversion.c

1>  graminit.c

1>  import.c

1>  importdl.c

1>  marshal.c

1>  modsupport.c

1>  mysnprintf.c

1>  mystrtoul.c

1>  peephole.c

1>  pyarena.c

1>  Generating Code...

1>  Compiling...

1>  pyctype.c

1>  pyfpe.c

1>  pylifecycle.c

1>  pymath.c

1>  pytime.c

1>  pystate.c

1>  pystrcmp.c

1>  pystrhex.c

1>  pystrtod.c

1>  dtoa.c

1>  Python-ast.c

1>  pythonrun.c

1>  structmember.c

1>  symtable.c

1>  sysmodule.c

1>  thread.c

1>  traceback.c

1>  dl_nt.c

1>  Generating Code...

1>  getbuildinfo.c

1>     Creating library W:\Python-3.5.1\PCBuild\win32\python35_d.lib and
object W:\Python-3.5.1\PCBuild\win32\python35_d.exp

1>  pythoncore.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python35_d.dll

13>------ Rebuild All started: Project: _socket, Configuration: Debug Win32
------

14>------ Rebuild All started: Project: _ctypes_test, Configuration: Debug
Win32 ------

15>------ Rebuild All started: Project: _elementtree, Configuration: Debug
Win32 ------

16>------ Rebuild All started: Project: _msi, Configuration: Debug Win32
------

15>  _elementtree.c

13>  socketmodule.c

16>  _msi.c

14>  _ctypes_test.c

15>  xmlparse.c

15>  xmlrole.c

15>  xmltok.c

15>  Generating Code...

13>..\Modules\socketmodule.c(1014): warning C4996: 'inet_addr': Use
inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1850):
note: see declaration of 'inet_addr'

13>..\Modules\socketmodule.c(4040): warning C4996: 'WSADuplicateSocketA':
Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2872):
note: see declaration of 'WSADuplicateSocketA'

13>..\Modules\socketmodule.c(4254): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

13>..\Modules\socketmodule.c(4288): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

13>..\Modules\socketmodule.c(4694): warning C4996: 'gethostbyname': Use
getaddrinfo() or GetAddrInfoW() instead or define
_WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2238):
note: see declaration of 'gethostbyname'

13>..\Modules\socketmodule.c(4792): warning C4996: 'gethostbyaddr': Use
getnameinfo() or GetNameInfoW() instead or define
_WINSOCK_DEPRECATED_NO_WARNINGS to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2216):
note: see declaration of 'gethostbyaddr'

13>..\Modules\socketmodule.c(4922): warning C4996: 'WSADuplicateSocketA':
Use WSADuplicateSocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(2872):
note: see declaration of 'WSADuplicateSocketA'

13>..\Modules\socketmodule.c(4927): warning C4996: 'WSASocketA': Use
WSASocketW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to disable
deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3457):
note: see declaration of 'WSASocketA'

13>..\Modules\socketmodule.c(5216): warning C4996: 'inet_addr': Use
inet_pton() or InetPton() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1850):
note: see declaration of 'inet_addr'

13>..\Modules\socketmodule.c(5259): warning C4996: 'inet_ntoa': Use
inet_ntop() or InetNtop() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(1868):
note: see declaration of 'inet_ntoa'

13>..\Modules\socketmodule.c(5333): warning C4996: 'WSAStringToAddressA':
Use WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

13>..\Modules\socketmodule.c(5477): warning C4996: 'WSAAddressToStringA':
Use WSAAddressToStringW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS
to disable deprecated API warnings

13>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3556):
note: see declaration of 'WSAAddressToStringA'

14>     Creating library W:\Python-3.5.1\PCBuild\win32\_ctypes_test_d.lib
and object W:\Python-3.5.1\PCBuild\win32\_ctypes_test_d.exp

16>     Creating library W:\Python-3.5.1\PCBuild\win32\_msi_d.lib and object
W:\Python-3.5.1\PCBuild\win32\_msi_d.exp

13>     Creating library W:\Python-3.5.1\PCBuild\win32\_socket_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_socket_d.exp

15>     Creating library W:\Python-3.5.1\PCBuild\win32\_elementtree_d.lib
and object W:\Python-3.5.1\PCBuild\win32\_elementtree_d.exp

14>  _ctypes_test.vcxproj ->
W:\Python-3.5.1\PCBuild\win32\_ctypes_test_d.pyd

16>  _msi.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_msi_d.pyd

13>  _socket.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_socket_d.pyd

15>  _elementtree.vcxproj ->
W:\Python-3.5.1\PCBuild\win32\_elementtree_d.pyd

17>------ Rebuild All started: Project: _sqlite3, Configuration: Debug Win32
------

18>------ Rebuild All started: Project: _ssl, Configuration: Debug Win32
------

19>------ Rebuild All started: Project: _testcapi, Configuration: Debug
Win32 ------

20>------ Rebuild All started: Project: _testimportmultiple, Configuration:
Debug Win32 ------

19>  _testcapimodule.c

20>  _testimportmultiple.c

17>  cache.c

18>  _ssl.c

17>  connection.c

17>  cursor.c

20>     Creating library
W:\Python-3.5.1\PCBuild\win32\_testimportmultiple_d.lib and object
W:\Python-3.5.1\PCBuild\win32\_testimportmultiple_d.exp

17>  microprotocols.c

20>  _testimportmultiple.vcxproj ->
W:\Python-3.5.1\PCBuild\win32\_testimportmultiple_d.pyd

18>..\Modules\_ssl.c(59): fatal error C1083: Cannot open include file:
'openssl/rsa.h': No such file or directory

21>------ Rebuild All started: Project: _tkinter, Configuration: Debug Win32
------

17>  module.c

22>------ Rebuild All started: Project: _bz2, Configuration: Debug Win32
------

17>  prepare_protocol.c

19>     Creating library W:\Python-3.5.1\PCBuild\win32\_testcapi_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_testcapi_d.exp

22>  _bz2module.c

17>  row.c

19>  _testcapi.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_testcapi_d.pyd

23>------ Rebuild All started: Project: select, Configuration: Debug Win32
------

23>  selectmodule.c

22>  blocksort.c

17>  statement.c

21>  _tkinter.c

17>  util.c

17>  Generating Code...

23>     Creating library W:\Python-3.5.1\PCBuild\win32\select_d.lib and
object W:\Python-3.5.1\PCBuild\win32\select_d.exp

21>..\Modules\_tkinter.c(49): fatal error C1083: Cannot open include file:
'tcl.h': No such file or directory

21>  tkappinit.c

21>..\Modules\tkappinit.c(16): fatal error C1083: Cannot open include file:
'tcl.h': No such file or directory

21>  Generating Code...

24>------ Rebuild All started: Project: _lzma, Configuration: Debug Win32
------

22>  bzlib.c

23>  select.vcxproj -> W:\Python-3.5.1\PCBuild\win32\select_d.pyd

25>------ Rebuild All started: Project: pyexpat, Configuration: Debug Win32
------

24>  _lzmamodule.c

17>     Creating library W:\Python-3.5.1\PCBuild\win32\_sqlite3_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_sqlite3_d.exp

17>  _sqlite3.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_sqlite3_d.pyd

24>     Creating library W:\Python-3.5.1\PCBuild\win32\_lzma_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_lzma_d.exp

24>  _lzma.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_lzma_d.pyd

26>------ Rebuild All started: Project: _hashlib, Configuration: Debug Win32
------

26>  _hashopenssl.c

26>..\Modules\_hashopenssl.c(23): fatal error C1083: Cannot open include
file: 'openssl/evp.h': No such file or directory

27>------ Rebuild All started: Project: python, Configuration: Debug Win32
------

27>  python.c

25>  pyexpat.c

25>  xmlparse.c

22>  compress.c

25>  xmlrole.c

25>  xmltok.c

25>  Generating Code...

22>  crctable.c

22>  decompress.c

22>  huffman.c

22>  randtable.c

28>------ Rebuild All started: Project: _multiprocessing, Configuration:
Debug Win32 ------

27>  python.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python_d.exe

29>------ Rebuild All started: Project: pythonw, Configuration: Debug Win32
------

28>  multiprocessing.c

22>  Generating Code...

29>  WinMain.c

25>     Creating library W:\Python-3.5.1\PCBuild\win32\pyexpat_d.lib and
object W:\Python-3.5.1\PCBuild\win32\pyexpat_d.exp

25>  pyexpat.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pyexpat_d.pyd

28>  semaphore.c

30>------ Rebuild All started: Project: _testbuffer, Configuration: Debug
Win32 ------

30>  _testbuffer.c

28>  Generating Code...

22>     Creating library W:\Python-3.5.1\PCBuild\win32\_bz2_d.lib and object
W:\Python-3.5.1\PCBuild\win32\_bz2_d.exp

28>     Creating library
W:\Python-3.5.1\PCBuild\win32\_multiprocessing_d.lib and object
W:\Python-3.5.1\PCBuild\win32\_multiprocessing_d.exp

22>  _bz2.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_bz2_d.pyd

31>------ Skipped Rebuild All: Project: _freeze_importlib, Configuration:
Debug Win32 ------

31>Project not selected to build for this solution configuration 

32>------ Rebuild All started: Project: _overlapped, Configuration: Debug
Win32 ------

30>     Creating library W:\Python-3.5.1\PCBuild\win32\_testbuffer_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_testbuffer_d.exp

30>  _testbuffer.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_testbuffer_d.pyd

33>------ Rebuild All started: Project: _testembed, Configuration: Debug
Win32 ------

32>  overlapped.c

33>  _testembed.c

28>  _multiprocessing.vcxproj ->
W:\Python-3.5.1\PCBuild\win32\_multiprocessing_d.pyd

34>------ Rebuild All started: Project: _testmultiphase, Configuration:
Debug Win32 ------

34>  _testmultiphase.c

29>  pythonw.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pythonw_d.exe

35>------ Rebuild All started: Project: winsound, Configuration: Debug Win32
------

32>..\Modules\overlapped.c(977): warning C4996: 'WSAStringToAddressA': Use
WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to
disable deprecated API warnings

32>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

32>..\Modules\overlapped.c(988): warning C4996: 'WSAStringToAddressA': Use
WSAStringToAddressW() instead or define _WINSOCK_DEPRECATED_NO_WARNINGS to
disable deprecated API warnings

32>  C:\Program Files (x86)\Windows Kits\8.1\Include\um\winsock2.h(3623):
note: see declaration of 'WSAStringToAddressA'

34>     Creating library W:\Python-3.5.1\PCBuild\win32\_testmultiphase_d.lib
and object W:\Python-3.5.1\PCBuild\win32\_testmultiphase_d.exp

35>  winsound.c

33>  _testembed.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_testembed_d.exe

36>------ Rebuild All started: Project: _decimal, Configuration: Debug Win32
------

32>     Creating library W:\Python-3.5.1\PCBuild\win32\_overlapped_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_overlapped_d.exp

34>  _testmultiphase.vcxproj ->
W:\Python-3.5.1\PCBuild\win32\_testmultiphase_d.pyd

37>------ Rebuild All started: Project: _ctypes, Configuration: Debug Win32
------

36>  _decimal.c

37>  _ctypes.c

36>  basearith.c

36>  constants.c

36>  context.c

32>  _overlapped.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_overlapped_d.pyd

38>------ Rebuild All started: Project: unicodedata, Configuration: Debug
Win32 ------

36>  convolute.c

36>  crt.c

38>  unicodedata.c

36>  difradix2.c

35>     Creating library W:\Python-3.5.1\PCBuild\win32\winsound_d.lib and
object W:\Python-3.5.1\PCBuild\win32\winsound_d.exp

37>  callbacks.c

36>  fnt.c

36>  fourstep.c

35>  winsound.vcxproj -> W:\Python-3.5.1\PCBuild\win32\winsound_d.pyd

36>  io.c

36>  memory.c

36>  mpdecimal.c

37>  callproc.c

36>  numbertheory.c

36>  sixstep.c

36>  transpose.c

36>  Generating Code...

38>     Creating library W:\Python-3.5.1\PCBuild\win32\unicodedata_d.lib and
object W:\Python-3.5.1\PCBuild\win32\unicodedata_d.exp

37>  cfield.c

38>  unicodedata.vcxproj -> W:\Python-3.5.1\PCBuild\win32\unicodedata_d.pyd

36>     Creating library W:\Python-3.5.1\PCBuild\win32\_decimal_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_decimal_d.exp

36>  _decimal.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_decimal_d.pyd

37>  ffi.c

37>  malloc_closure.c

37>  prep_cif.c

37>  stgdict.c

37>  win32.c

37>  Generating Code...

37>     Creating library W:\Python-3.5.1\PCBuild\win32\_ctypes_d.lib and
object W:\Python-3.5.1\PCBuild\win32\_ctypes_d.exp

37>  _ctypes.vcxproj -> W:\Python-3.5.1\PCBuild\win32\_ctypes_d.pyd

========== Rebuild All: 29 succeeded, 6 failed, 3 skipped ==========

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/a09f1f3e/attachment-0001.html>

From brett at python.org  Wed Jan 13 16:30:07 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 13 Jan 2016 21:30:07 +0000
Subject: [Python-Dev] Debugging using VS 2015
In-Reply-To: <003201d14e3d$b6560f80$23022e80$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
 <003201d14e3d$b6560f80$23022e80$@com>
Message-ID: <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>

Use PCbuild/get_externals.bat
<https://hg.python.org/cpython/file/default/PCbuild/get_externals.bat> to
download the external dependencies.

On Wed, 13 Jan 2016 at 13:25 Eddy Quicksall <Eddy at quicksall.com> wrote:

> I am using 3.5.1. I?m adding an extension for my special case.
>
>
>
> I know this list is for development and that?s what I?m doing but I would
> like to use VS 2015 to do my debugging. If there is another list I should
> use to get the build to work then please let me know the list.
>
>
>
> I have been able to build using ?PCbuild\build.bat ?e? but I?m not able to
> build using PCbuild\pcbuild.sln. It gives lots of errors.
>
>
>
> I?m using Debug/Win32.
>
>
>
> Below is the output. Note that lots of .c files are missing ?. Is there
> another package I should get (but then why would it build correctly with
> build.bat)? It could be that I need to add the equivalent to the ?-e?
> option.
>
>
>
> 1>------ Rebuild All started: Project: pythoncore, Configuration: Debug
> Win32 ------
>
> 2>------ Rebuild All started: Project: tcl, Configuration: Debug Win32
> ------
>
> 3>------ Rebuild All started: Project: ssleay, Configuration: Debug Win32
> ------
>
> 4>------ Rebuild All started: Project: sqlite3, Configuration: Debug Win32
> ------
>
> 1>  Killing any running python_d.exe instances...
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The
> command "hg id -b >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The
> command "hg id -i >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The
> command "hg id -t >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited
> with code 9009.
>
> 2>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 5>------ Rebuild All started: Project: python3dll, Configuration: Debug
> Win32 ------
>
> 1>  _bisectmodule.c
>
> 5>  python3dll.c
>
> 4>  sqlite3.c
>
> 3>  d1_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or
> directory
>
> 3>  d1_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or
> directory
>
> 1>  _codecsmodule.c
>
> 3>  d1_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or
> directory
>
> 3>  d1_srtp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or
> directory
>
> 3>  s2_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or
> directory
>
> 3>  s2_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or
> directory
>
> 3>  s2_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or
> directory
>
> 3>  s2_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or
> directory
>
> 3>  s2_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_pkt.c': No such file or
> directory
>
> 3>  s2_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_srvr.c': No such file or
> directory
>
> 3>  s23_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_clnt.c': No such file or
> directory
>
> 3>  s23_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_lib.c': No such file or
> directory
>
> 3>  s23_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_meth.c': No such file or
> directory
>
> 3>  s23_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_pkt.c': No such file or
> directory
>
> 3>  s23_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_srvr.c': No such file or
> directory
>
> 3>  s3_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_both.c': No such file or
> directory
>
> 3>  s3_cbc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_cbc.c': No such file or
> directory
>
> 3>  s3_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_clnt.c': No such file or
> directory
>
> 3>  s3_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_enc.c': No such file or
> directory
>
> 3>  s3_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_lib.c': No such file or
> directory
>
> 3>  Generating Code...
>
> 3>  Compiling...
>
> 3>  s3_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_meth.c': No such file or
> directory
>
> 3>  s3_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_pkt.c': No such file or
> directory
>
> 3>  s3_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_srvr.c': No such file or
> directory
>
> 3>  ssl_algs.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_algs.c': No such file or
> directory
>
> 3>  ssl_asn1.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_asn1.c': No such file or
> directory
>
> 3>  ssl_cert.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_cert.c': No such file or
> directory
>
> 3>  ssl_ciph.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_ciph.c': No such file or
> directory
>
> 3>  ssl_err.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err.c': No such file or
> directory
>
> 3>  ssl_err2.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err2.c': No such file or
> directory
>
> 3>  ssl_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_lib.c': No such file or
> directory
>
> 3>  ssl_rsa.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_rsa.c': No such file or
> directory
>
> 3>  ssl_sess.c
>
> 1>  _collectionsmodule.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_sess.c': No such file or
> directory
>
> 3>  t1_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_clnt.c': No such file or
> directory
>
> 3>  t1_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_enc.c': No such file or
> directory
>
> 3>  t1_lib.c
>
> 1>  _csv.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_lib.c': No such file or
> directory
>
> 3>  t1_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_meth.c': No such file or
> directory
>
> 3>  t1_reneg.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_reneg.c': No such file or
> directory
>
> 3>  t1_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_srvr.c': No such file or
> directory
>
> 3>  tls_srp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\tls_srp.c': No such file or
> directory
>
> 3>  Generating Code...
>
> 6>------ Rebuild All started: Project: tk, Configuration: Debug Win32
> ------
>
> 1>  _functoolsmodule.c
>
> 1>  _heapqmodule.c
>
> 1>  _json.c
>
> 1>  _localemodule.c
>
> 4>     Creating library W:\Python-3.5.1\PCBuild\win32\sqlite3_d.lib and
> object W:\Python-3.5.1\PCBuild\win32\sqlite3_d.exp
>
> 5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_dstub.lib
> and object W:\Python-3.5.1\PCBuild\win32\python3_dstub.exp
>
> 1>  _lsprof.c
>
> 5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_d.lib and
> object W:\Python-3.5.1\PCBuild\win32\python3_d.exp
>
> 5>  python3dll.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python3_d.dll
>
> 7>------ Rebuild All started: Project: libeay, Configuration: Debug Win32
> ------
>
> 6>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 8>------ Skipped Rebuild All: Project: bdist_wininst, Configuration: Debug
> Win32 ------
>
> 8>Project not selected to build for this solution configuration
>
> 9>------ Skipped Rebuild All: Project: xxlimited, Configuration: Release
> Win32 ------
>
> 9>Project not selected to build for this solution configuration
>
> 10>------ Rebuild All started: Project: pylauncher, Configuration: Debug
> Win32 ------
>
> 1>  _math.c
>
> 4>  sqlite3.vcxproj -> W:\Python-3.5.1\PCBuild\win32\sqlite3_d.dll
>
> 11>------ Rebuild All started: Project: pywlauncher, Configuration: Debug
> Win32 ------
>
> 10>  launcher.c
>
> 1>  _pickle.c
>
> 7>  nasm: fatal: unable to open input file
> `W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm'
>
> 1>  _randommodule.c
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: The command
> "setlocal
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: set
> PATH=W:\Python-3.5.1\externals\\nasm-2.11.06\;%PATH%
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: nasm.exe -f
> win32 -o
> "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp\\win32_Debug\libeay\aes-586.obj"
> "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm"" exited with
> code 1.
>
> 12>------ Rebuild All started: Project: tix, Configuration: Debug Win32
> ------
>
> 11>  launcher.c
>
> 1>  _sre.c
>
> 1>  _stat.c
>
> 1>  _struct.c
>
> 12>    md Debug_VC13
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixClass.obj ..\generic\tixClass.c
>
> 12>  tixClass.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixCmds.obj ..\generic\tixCmds.c
>
> 12>  tixCmds.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixCompat.obj ..\generic\tixCompat.c
>
> 12>  tixCompat.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiImg.obj ..\generic\tixDiImg.c
>
> 12>  tixDiImg.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiITxt.obj ..\generic\tixDiITxt.c
>
> 12>  tixDiITxt.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiStyle.obj ..\generic\tixDiStyle.c
>
> 12>  tixDiStyle.c
>
> 11>  pywlauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pyw_d.exe
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDItem.obj ..\generic\tixDItem.c
>
> 12>  tixDItem.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiText.obj ..\generic\tixDiText.c
>
> 12>  tixDiText.c
>
> 1>  _weakref.c
>
> 1>  arraymodule.c
>
> 1>  atexitmodule.c
>
> 1>  audioop.c
>
> 1>  binascii.c
>
> 1>  Generating Code...
>
> 1>  Compiling...
>
> 1>  cmathmodule.c
>
> 1>  _datetimemodule.c
>
> 1>  errnomodule.c
>
> 1>  faulthandler.c
>
> 1>  gcmodule.c
>
> 10>  pylauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\py_d.exe
>
> 1>  hashtable.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiWin.obj ..\generic\tixDiWin.c
>
> 12>  tixDiWin.c
>
> 1>  itertoolsmodule.c
>
> 1>  main.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixError.obj ..\generic\tixError.c
>
> 12>  tixError.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixForm.obj ..\generic\tixForm.c
>
> 1>  mathmodule.c
>
> 12>  tixForm.c
>
> 1>  md5module.c
>
> 1>  mmapmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixFormMisc.obj ..\generic\tixFormMisc.c
>
> 12>  tixFormMisc.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGeometry.obj ..\generic\tixGeometry.c
>
> 12>  tixGeometry.c
>
> 1>  _opcode.c
>
> 1>  _operator.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrid.obj ..\generic\tixGrid.c
>
> 1>  parsermodule.c
>
> 12>  tixGrid.c
>
> 1>  posixmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrData.obj ..\generic\tixGrData.c
>
> 12>  tixGrData.c
>
> 1>  rotatingtree.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrRC.obj ..\generic\tixGrRC.c
>
> 1>  sha1module.c
>
> 12>  tixGrRC.c
>
> 1>  sha256module.c
>
> 1>  sha512module.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrFmt.obj ..\generic\tixGrFmt.c
>
> 12>  tixGrFmt.c
>
> 1>  signalmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrSel.obj ..\generic\tixGrSel.c
>
> 12>  tixGrSel.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrUtl.obj ..\generic\tixGrUtl.c
>
> 12>  tixGrUtl.c
>
> 1>  Compiling...
>
> 1>  symtablemodule.c
>
> 1>  _threadmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLCol.obj ..\generic\tixHLCol.c
>
> 12>  tixHLCol.c
>
> 1>  _tracemalloc.c
>
> 1>  timemodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLHdr.obj ..\generic\tixHLHdr.c
>
> 12>  tixHLHdr.c
>
> 1>  xxsubtype.c
>
> 1>  zipimport.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLInd.obj ..\generic\tixHLInd.c
>
> 12>  tixHLInd.c
>
> 1>  zlibmodule.c
>
> 1>  fileio.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHList.obj ..\generic\tixHList.c
>
> 12>  tixHList.c
>
> 1>  bytesio.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixImgCmp.obj ..\generic\tixImgCmp.c
>
> 12>  tixImgCmp.c
>
> 1>  stringio.c
>
> 1>  bufferedio.c
>
> 12>..\generic\tixImgCmp.c(169): warning C4090: 'function': different
> 'const' qualifiers
>
> 12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 2
> different from declaration
>
> 12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 5
> different from declaration
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixImgXpm.obj ..\generic\tixImgXpm.c
>
> 12>  tixImgXpm.c
>
> 1>  iobase.c
>
> 1>  textio.c
>
> 12>..\generic\tixImgXpm.c(88): warning C4090: 'function': different
> 'const' qualifiers
>
> 12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 2 different
> from declaration
>
> 12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 5 different
> from declaration
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixInit.obj ..\generic\tixInit.c
>
> 1>  _iomodule.c
>
> 12>  tixInit.c
>
> 1>  adler32.c
>
> 1>  compress.c
>
> 1>  crc32.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixList.obj ..\generic\tixList.c
>
> 1>  deflate.c
>
> 12>  tixList.c
>
> 1>  infback.c
>
> 1>  inffast.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixMethod.obj ..\generic\tixMethod.c
>
> 1>  Compiling...
>
> 1>  inflate.c
>
> 12>  tixMethod.c
>
> 1>  inftrees.c
>
> 1>  trees.c
>
> 1>  uncompr.c
>
> 1>  zutil.c
>
> 1>  _codecs_cn.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixNBFrame.obj ..\generic\tixNBFrame.c
>
> 12>  tixNBFrame.c
>
> 1>  _codecs_hk.c
>
> 1>  _codecs_iso2022.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixOption.obj ..\generic\tixOption.c
>
> 12>  tixOption.c
>
> 1>  _codecs_jp.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixSmpLs.obj ..\generic\tixSmpLs.c
>
> 1>  _codecs_kr.c
>
> 12>  tixSmpLs.c
>
> 1>  _codecs_tw.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixScroll.obj ..\generic\tixScroll.c
>
> 12>  tixScroll.c
>
> 1>  multibytecodec.c
>
> 1>  _winapi.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixTList.obj ..\generic\tixTList.c
>
> 12>  tixTList.c
>
> 1>  abstract.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixUtils.obj ..\generic\tixUtils.c
>
> 12>  tixUtils.c
>
> 1>  accu.c
>
> 1>  boolobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWCmpt.obj ..\win\tixWCmpt.c
>
> 1>  bytes_methods.c
>
> 12>  tixWCmpt.c
>
> 1>  bytearrayobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWidget.obj ..\generic\tixWidget.c
>
> 12>  tixWidget.c
>
> 1>  bytesobject.c
>
> 1>  capsule.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinDraw.obj ..\win\tixWinDraw.c
>
> 12>  tixWinDraw.c
>
> 1>  Compiling...
>
> 1>  cellobject.c
>
> 1>  classobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinXpm.obj ..\win\tixWinXpm.c
>
> 12>  tixWinXpm.c
>
> 1>  codeobject.c
>
> 1>  complexobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinWm.obj ..\win\tixWinWm.c
>
> 1>  descrobject.c
>
> 12>  tixWinWm.c
>
> 1>  dictobject.c
>
> 12>    link.exe -debug -debugtype:cv /RELEASE /NOLOGO /MACHINE:IX86 -dll
> W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib
> W:\Python-3.5.1\externals\tcl-core-8.6.4.2\win\Debug_VC13\tclstub86.lib
> kernel32.lib  advapi32.lib user32.lib gdi32.lib comdlg32.lib
> -out:Debug_VC13\tix84g.dll @C:\Users\eddyq\AppData\Local\Temp\nmF029.tmp
>
> 12>LINK : fatal error LNK1181: cannot open input file
> 'W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib'
>
> 12>NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual
> Studio 14.0\VC\bin\link.exe"' : return code '0x49d'
>
> 12>  Stop.
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: The command "setlocal
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: if not exist
> "W:\Python-3.5.1\externals\tcltk\lib\tix8.4.3\tix84g.dll" goto build
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: goto :eof
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: :build
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: set VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual
> Studio 14.0\VC\
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: cd /D "W:\Python-3.5.1\externals\tix-8.4.3.6\win"
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: nmake /nologo -f makefile.vc MACHINE=IX86 DEBUG=1
> NODEBUG=0 TCL_DBGX=g TK_DBGX=g TCL_MAJOR=8 TCL_MINOR=6 TCL_PATCH=4
> BUILDDIRTOP="Debug_VC13"
> TCL_DIR="W:\Python-3.5.1\externals\tcl-core-8.6.4.2"
> TK_DIR="W:\Python-3.5.1\externals\tk-8.6.4.2"
> INSTALL_DIR="W:\Python-3.5.1\externals\tcltk" all install
>
> 12>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5):
> error MSB3073: " exited with code 2.
>
> 1>  enumobject.c
>
> 1>  exceptions.c
>
> 1>  fileobject.c
>
> 1>  floatobject.c
>
> 1>  frameobject.c
>
> 1>  funcobject.c
>
> 1>  genobject.c
>
> 1>  iterobject.c
>
> 1>  listobject.c
>
> 1>  longobject.c
>
> 1>  memoryobject.c
>
> 1>  methodobject.c
>
> 1>  moduleobject.c
>
> 1>  namespaceobject.c
>
> 1>  Generating Code...
>
> 1>  Compiling...
>
> 1>  object.c
>
> 1>  obmalloc.c
>
> 1>  odictobject.c
>
> 1>  rangeobject.c
>
> 1>  setobject.c
>
> 1>  sliceobject.c
>
> 1>  structseq.c
>
> 1>  tupleobject.c
>
> 1>  typeobject.c
>
> 1>  unicodectype.c
>
> 1>  unicodeobject.c
>
> 1>  weakrefobject.c
>
> 1>  acceler.c
>
> 1>  bitset.c
>
> 1>  firstsets.c
>
> 1>  grammar.c
>
> 1>  grammar1.c
>
> 1>  listnode.c
>
> 1>  metagrammar.c
>
> 1>  myreadline.c
>
> 1>  Generating Code...
>
> 1>  Compiling...
>
> 1>  node.c
>
> 1>  parser.c
>
> 1>  parsetok.c
>
> 1>  tokenizer.c
>
> 1>  invalid_parameter_handler.c
>
> 1>  winreg.c
>
> 1>  config.c
>
> 1>  getpathp.c
>
> 1>  msvcrtmodule.c
>
> 1>  pyhash.c
>
> 1>  random.c
>
> 1>  _warnings.c
>
> 1>  asdl.c
>
> 1>  ast.c
>
> 1>  bltinmodule.c
>
> 1>  ceval.c
>
> 1>  codecs.c
>
> 1>  compile.c
> <
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/e97601d0/attachment-0001.html>

From victor.stinner at gmail.com  Wed Jan 13 17:10:30 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 13 Jan 2016 23:10:30 +0100
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696A647.6090501@busiq.com>
References: <5696A647.6090501@busiq.com>
Message-ID: <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>

Hi,

2016-01-13 20:32 GMT+01:00 Matthew Paulson <paulson at busiq.com>:
> I've spent some time performing memory leak analysis while using Python in an embedded configuration.

Hum, did you try tracemalloc?

https://docs.python.org/dev/library/tracemalloc.html
https://pytracemalloc.readthedocs.org/

> Is there someone in the group that would like to discuss this topic.  There seems to be other leaks as well.  I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.

Are you able to reproduce the leak with a simple program?

Victor

From victor.stinner at gmail.com  Wed Jan 13 17:32:15 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 13 Jan 2016 23:32:15 +0100
Subject: [Python-Dev] PEP 510: Specialize functions with guards
In-Reply-To: <CAMpsgwZmb2KBi+5OtBN5Cg4-LDc_hbM6nrDVusXaKWn3dOeefQ@mail.gmail.com>
References: <CAMpsgwZmb2KBi+5OtBN5Cg4-LDc_hbM6nrDVusXaKWn3dOeefQ@mail.gmail.com>
Message-ID: <CAMpsgwbQoL=CQajsWRbY1x5sd-uEw-e+Gg+f-+rPLLpb86YqFA@mail.gmail.com>

I extracted a patch from my FAT Python project to implement the PEP 510:
https://bugs.python.org/issue26098

FYI I also extracted the runtime part of the FAT Python optimizer and
put it on GitHub:
https://github.com/haypo/fat

The fat module provides specialize(), get_specialized() and
replace_consts() functions, but also 6 different guards: GuardArgType,
GuardBuiltins, GuardDict, GuardFunc, GuardGlobals, GuardTypeDict.

Victor

From paulson at busiq.com  Wed Jan 13 17:49:41 2016
From: paulson at busiq.com (Matthew Paulson)
Date: Wed, 13 Jan 2016 17:49:41 -0500
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
References: <5696A647.6090501@busiq.com>
 <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
Message-ID: <5696D485.2030607@busiq.com>

Hi Victor:

No, I'm using the new heap analysis functions in DS2015.  We think we 
have found one issue. In the following sequence, dict has no side 
effects, yet it is used -- unless someone can shed light on why dict is 
used in this case:

/* Clear the modules dict. */
     PyDict_Clear(modules);
     /* Restore the original builtins dict, to ensure that any
        user data gets cleared. */
     dict = PyDict_Copy(interp->builtins);
     if (dict == NULL)
         PyErr_Clear();
     PyDict_Clear(interp->builtins);
     if (PyDict_Update(interp->builtins, interp->builtins_copy))
         PyErr_Clear();
     Py_XDECREF(dict);

And removing dict from this sequence seems to have fixed one of the 
issues, yielding 14k per iteration.

Simple program: Good idea.  We will try that -- right now it's embedded 
in a more complex environment, but we have tried to strip it down to a 
very simple sequence.

The next item on our list is memory that is not getting freed after 
running simple string.  It's in the parsertok sequence -- it seems that 
the syntax tree is not getting cleared -- but this opinion is preliminary.

Best,

Matt

On 1/13/2016 5:10 PM, Victor Stinner wrote:
> Hi,
>
> 2016-01-13 20:32 GMT+01:00 Matthew Paulson <paulson at busiq.com>:
>> I've spent some time performing memory leak analysis while using Python in an embedded configuration.
> Hum, did you try tracemalloc?
>
> https://docs.python.org/dev/library/tracemalloc.html
> https://pytracemalloc.readthedocs.org/
>
>> Is there someone in the group that would like to discuss this topic.  There seems to be other leaks as well.  I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.
> Are you able to reproduce the leak with a simple program?
>
> Victor
>
>

-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/fedda01a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/fedda01a/attachment-0001.jpe>

From abarnert at yahoo.com  Wed Jan 13 18:45:52 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 13 Jan 2016 15:45:52 -0800
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696D485.2030607@busiq.com>
References: <5696A647.6090501@busiq.com>
 <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
 <5696D485.2030607@busiq.com>
Message-ID: <6AFE80F6-AFF2-4B2E-BA98-E3F24C4CCF73@yahoo.com>

On Jan 13, 2016, at 14:49, Matthew Paulson <paulson at busiq.com> wrote:
> 
> Hi Victor:
> 
> No, I'm using the new heap analysis functions in DS2015. 

Isn't that going to report any memory that Python's higher level allocators hold in their freelists as leaked, even though it isn't leaked?

> We think we have found one issue. In the following sequence, dict has no side effects, yet it is used -- unless someone can shed light on why dict is used in this case:

Where do you see an issue here? The dict will have one ref, so the decref at the end should return it to the freelist.

Also, it looks like there _is_ a side effect here. When you add a bunch of elements to a dict, it grows. When you delete a bunch of elements, it generally doesn't shrink. But when you clear the dict, it does shrink. So, copying it to a temporary dict, clearing it, updating it from the temporary dict, and then releasing the temporary dict should force it to shrink.

So, the overall effect should be that you have a smaller hash table for the builtins dict, and a chunk of memory sitting on the freelists ready to be reused. If your analyzer is showing the freelists as leaked, this will look like a net leak rather than a net recovery, but that's just a problem in the analyzer.

Of course I could be wrong, but I think the first step is to rule out the possibility that you're measuring the wrong thing...

> /* Clear the modules dict. */
>     PyDict_Clear(modules);
>     /* Restore the original builtins dict, to ensure that any
>        user data gets cleared. */
>     dict = PyDict_Copy(interp->builtins);
>     if (dict == NULL)
>         PyErr_Clear();
>     PyDict_Clear(interp->builtins);
>     if (PyDict_Update(interp->builtins, interp->builtins_copy))
>         PyErr_Clear();
>     Py_XDECREF(dict);
> 
> And removing dict from this sequence seems to have fixed one of the issues, yielding 14k per iteration.

> Simple program: Good idea.  We will try that -- right now it's embedded in a more complex environment, but we have tried to strip it down to a very simple sequence.
> 
> The next item on our list is memory that is not getting freed after running simple string.  It's in the parsertok sequence -- it seems that the syntax tree is not getting cleared -- but this opinion is preliminary.
> 
> Best,
> 
> Matt
> 
>> On 1/13/2016 5:10 PM, Victor Stinner wrote:
>> Hi,
>> 
>> 2016-01-13 20:32 GMT+01:00 Matthew Paulson <paulson at busiq.com>:
>>> I've spent some time performing memory leak analysis while using Python in an embedded configuration.
>> Hum, did you try tracemalloc?
>> 
>> https://docs.python.org/dev/library/tracemalloc.html
>> https://pytracemalloc.readthedocs.org/
>> 
>>> Is there someone in the group that would like to discuss this topic.  There seems to be other leaks as well.  I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.
>> Are you able to reproduce the leak with a simple program?
>> 
>> Victor
>> 
>> 
> 
> -- 
> <MattSig.JPG>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/fd9662df/attachment.html>

From Eddy at Quicksall.com  Wed Jan 13 18:56:35 2016
From: Eddy at Quicksall.com (Eddy Quicksall)
Date: Wed, 13 Jan 2016 18:56:35 -0500
Subject: [Python-Dev] Debugging using VS 2015
In-Reply-To: <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
 <5695C54A.3070101@sdamon.com> <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
Message-ID: <005e01d14e5e$09660bb0$1c322310$@com>

Those files already exist:

 

W:\Python-3.5.1>PCbuild\get_externals.bat

Fetching external libraries...

bzip2-1.0.6 already exists, skipping.

nasm-2.11.06 already exists, skipping.

openssl-1.0.2d already exists, skipping.

sqlite-3.8.11.0 already exists, skipping.

tcl-core-8.6.4.2 already exists, skipping.

tk-8.6.4.2 already exists, skipping.

tix-8.4.3.6 already exists, skipping.

xz-5.0.5 already exists, skipping.

Finished.

 

W:\Python-3.5.1>

 

From: Brett Cannon [mailto:brett at python.org] 
Sent: Wednesday, January 13, 2016 4:30 PM
To: Eddy Quicksall; python-dev at python.org
Subject: Re: [Python-Dev] Debugging using VS 2015

 

Use PCbuild/get_externals.bat <https://hg.python.org/cpython/file/default/PCbuild/get_externals.bat>  to download the external dependencies.

 

On Wed, 13 Jan 2016 at 13:25 Eddy Quicksall <Eddy at quicksall.com> wrote:

I am using 3.5.1. I?m adding an extension for my special case. 

 

I know this list is for development and that?s what I?m doing but I would like to use VS 2015 to do my debugging. If there is another list I should use to get the build to work then please let me know the list.

 

I have been able to build using ?PCbuild\build.bat ?e? but I?m not able to build using PCbuild\pcbuild.sln. It gives lots of errors.

 

I?m using Debug/Win32.

 

Below is the output. Note that lots of .c files are missing ?. Is there another package I should get (but then why would it build correctly with build.bat)? It could be that I need to add the equivalent to the ?-e? option.

 

1>------ Rebuild All started: Project: pythoncore, Configuration: Debug Win32 ------

2>------ Rebuild All started: Project: tcl, Configuration: Debug Win32 ------

3>------ Rebuild All started: Project: ssleay, Configuration: Debug Win32 ------

4>------ Rebuild All started: Project: sqlite3, Configuration: Debug Win32 ------

1>  Killing any running python_d.exe instances...

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The command "hg id -b > "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited with code 9009.

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The command "hg id -i > "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited with code 9009.

1>  'hg' is not recognized as an internal or external command,

1>  operable program or batch file.

1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The command "hg id -t > "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited with code 9009.

2>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5): warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.  Skipping...

5>------ Rebuild All started: Project: python3dll, Configuration: Debug Win32 ------

1>  _bisectmodule.c

5>  python3dll.c

4>  sqlite3.c

3>  d1_both.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or directory

3>  d1_lib.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or directory

1>  _codecsmodule.c

3>  d1_pkt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or directory

3>  d1_srtp.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or directory

3>  s2_clnt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or directory

3>  s2_enc.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or directory

3>  s2_lib.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or directory

3>  s2_meth.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or directory

3>  s2_pkt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_pkt.c': No such file or directory

3>  s2_srvr.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_srvr.c': No such file or directory

3>  s23_clnt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_clnt.c': No such file or directory

3>  s23_lib.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_lib.c': No such file or directory

3>  s23_meth.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_meth.c': No such file or directory

3>  s23_pkt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_pkt.c': No such file or directory

3>  s23_srvr.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_srvr.c': No such file or directory

3>  s3_both.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_both.c': No such file or directory

3>  s3_cbc.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_cbc.c': No such file or directory

3>  s3_clnt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_clnt.c': No such file or directory

3>  s3_enc.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_enc.c': No such file or directory

3>  s3_lib.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_lib.c': No such file or directory

3>  Generating Code...

3>  Compiling...

3>  s3_meth.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_meth.c': No such file or directory

3>  s3_pkt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_pkt.c': No such file or directory

3>  s3_srvr.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_srvr.c': No such file or directory

3>  ssl_algs.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_algs.c': No such file or directory

3>  ssl_asn1.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_asn1.c': No such file or directory

3>  ssl_cert.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_cert.c': No such file or directory

3>  ssl_ciph.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_ciph.c': No such file or directory

3>  ssl_err.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err.c': No such file or directory

3>  ssl_err2.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err2.c': No such file or directory

3>  ssl_lib.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_lib.c': No such file or directory

3>  ssl_rsa.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_rsa.c': No such file or directory

3>  ssl_sess.c

1>  _collectionsmodule.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_sess.c': No such file or directory

3>  t1_clnt.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_clnt.c': No such file or directory

3>  t1_enc.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_enc.c': No such file or directory

3>  t1_lib.c

1>  _csv.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_lib.c': No such file or directory

3>  t1_meth.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_meth.c': No such file or directory

3>  t1_reneg.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_reneg.c': No such file or directory

3>  t1_srvr.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_srvr.c': No such file or directory

3>  tls_srp.c

3>c1 : fatal error C1083: Cannot open source file: 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\tls_srp.c': No such file or directory

3>  Generating Code...

6>------ Rebuild All started: Project: tk, Configuration: Debug Win32 ------

1>  _functoolsmodule.c

1>  _heapqmodule.c

1>  _json.c

1>  _localemodule.c

4>     Creating library W:\Python-3.5.1\PCBuild\win32\sqlite3_d.lib and object W:\Python-3.5.1\PCBuild\win32\sqlite3_d.exp

5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_dstub.lib and object W:\Python-3.5.1\PCBuild\win32\python3_dstub.exp

1>  _lsprof.c

5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_d.lib and object W:\Python-3.5.1\PCBuild\win32\python3_d.exp

5>  python3dll.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python3_d.dll

7>------ Rebuild All started: Project: libeay, Configuration: Debug Win32 ------

6>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5): warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.  Skipping...

8>------ Skipped Rebuild All: Project: bdist_wininst, Configuration: Debug Win32 ------

8>Project not selected to build for this solution configuration 

9>------ Skipped Rebuild All: Project: xxlimited, Configuration: Release Win32 ------

9>Project not selected to build for this solution configuration 

10>------ Rebuild All started: Project: pylauncher, Configuration: Debug Win32 ------

1>  _math.c

4>  sqlite3.vcxproj -> W:\Python-3.5.1\PCBuild\win32\sqlite3_d.dll

11>------ Rebuild All started: Project: pywlauncher, Configuration: Debug Win32 ------

10>  launcher.c

1>  _pickle.c

7>  nasm: fatal: unable to open input file `W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm'

1>  _randommodule.c

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: The command "setlocal

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: set PATH=W:\Python-3.5.1\externals\\nasm-2.11.06\;%PATH%

7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: nasm.exe -f win32 -o "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp\\win32_Debug\libeay\aes-586.obj" "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm"" exited with code 1.

12>------ Rebuild All started: Project: tix, Configuration: Debug Win32 ------

11>  launcher.c

1>  _sre.c

1>  _stat.c

1>  _struct.c

12>    md Debug_VC13

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixClass.obj ..\generic\tixClass.c

12>  tixClass.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixCmds.obj ..\generic\tixCmds.c

12>  tixCmds.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixCompat.obj ..\generic\tixCompat.c

12>  tixCompat.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDiImg.obj ..\generic\tixDiImg.c

12>  tixDiImg.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDiITxt.obj ..\generic\tixDiITxt.c

12>  tixDiITxt.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDiStyle.obj ..\generic\tixDiStyle.c

12>  tixDiStyle.c

11>  pywlauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pyw_d.exe

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDItem.obj ..\generic\tixDItem.c

12>  tixDItem.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDiText.obj ..\generic\tixDiText.c

12>  tixDiText.c

1>  _weakref.c

1>  arraymodule.c

1>  atexitmodule.c

1>  audioop.c

1>  binascii.c

1>  Generating Code...

1>  Compiling...

1>  cmathmodule.c

1>  _datetimemodule.c

1>  errnomodule.c

1>  faulthandler.c

1>  gcmodule.c

10>  pylauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\py_d.exe

1>  hashtable.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixDiWin.obj ..\generic\tixDiWin.c

12>  tixDiWin.c

1>  itertoolsmodule.c

1>  main.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixError.obj ..\generic\tixError.c

12>  tixError.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixForm.obj ..\generic\tixForm.c

1>  mathmodule.c

12>  tixForm.c

1>  md5module.c

1>  mmapmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixFormMisc.obj ..\generic\tixFormMisc.c

12>  tixFormMisc.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGeometry.obj ..\generic\tixGeometry.c

12>  tixGeometry.c

1>  _opcode.c

1>  _operator.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrid.obj ..\generic\tixGrid.c

1>  parsermodule.c

12>  tixGrid.c

1>  posixmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrData.obj ..\generic\tixGrData.c

12>  tixGrData.c

1>  rotatingtree.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrRC.obj ..\generic\tixGrRC.c

1>  sha1module.c

12>  tixGrRC.c

1>  sha256module.c

1>  sha512module.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrFmt.obj ..\generic\tixGrFmt.c

12>  tixGrFmt.c

1>  signalmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrSel.obj ..\generic\tixGrSel.c

12>  tixGrSel.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixGrUtl.obj ..\generic\tixGrUtl.c

12>  tixGrUtl.c

1>  Compiling...

1>  symtablemodule.c

1>  _threadmodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixHLCol.obj ..\generic\tixHLCol.c

12>  tixHLCol.c

1>  _tracemalloc.c

1>  timemodule.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixHLHdr.obj ..\generic\tixHLHdr.c

12>  tixHLHdr.c

1>  xxsubtype.c

1>  zipimport.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixHLInd.obj ..\generic\tixHLInd.c

12>  tixHLInd.c

1>  zlibmodule.c

1>  fileio.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixHList.obj ..\generic\tixHList.c

12>  tixHList.c

1>  bytesio.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixImgCmp.obj ..\generic\tixImgCmp.c

12>  tixImgCmp.c

1>  stringio.c

1>  bufferedio.c

12>..\generic\tixImgCmp.c(169): warning C4090: 'function': different 'const' qualifiers

12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 2 different from declaration

12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 5 different from declaration

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixImgXpm.obj ..\generic\tixImgXpm.c

12>  tixImgXpm.c

1>  iobase.c

1>  textio.c

12>..\generic\tixImgXpm.c(88): warning C4090: 'function': different 'const' qualifiers

12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 2 different from declaration

12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 5 different from declaration

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixInit.obj ..\generic\tixInit.c

1>  _iomodule.c

12>  tixInit.c

1>  adler32.c

1>  compress.c

1>  crc32.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixList.obj ..\generic\tixList.c

1>  deflate.c

12>  tixList.c

1>  infback.c

1>  inffast.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixMethod.obj ..\generic\tixMethod.c

1>  Compiling...

1>  inflate.c

12>  tixMethod.c

1>  inftrees.c

1>  trees.c

1>  uncompr.c

1>  zutil.c

1>  _codecs_cn.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixNBFrame.obj ..\generic\tixNBFrame.c

12>  tixNBFrame.c

1>  _codecs_hk.c

1>  _codecs_iso2022.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixOption.obj ..\generic\tixOption.c

12>  tixOption.c

1>  _codecs_jp.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixSmpLs.obj ..\generic\tixSmpLs.c

1>  _codecs_kr.c

12>  tixSmpLs.c

1>  _codecs_tw.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixScroll.obj ..\generic\tixScroll.c

12>  tixScroll.c

1>  multibytecodec.c

1>  _winapi.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixTList.obj ..\generic\tixTList.c

12>  tixTList.c

1>  abstract.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixUtils.obj ..\generic\tixUtils.c

12>  tixUtils.c

1>  accu.c

1>  boolobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixWCmpt.obj ..\win\tixWCmpt.c

1>  bytes_methods.c

12>  tixWCmpt.c

1>  bytearrayobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixWidget.obj ..\generic\tixWidget.c

12>  tixWidget.c

1>  bytesobject.c

1>  capsule.c

1>  Generating Code...

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixWinDraw.obj ..\win\tixWinDraw.c

12>  tixWinDraw.c

1>  Compiling...

1>  cellobject.c

1>  classobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixWinXpm.obj ..\win\tixWinXpm.c

12>  tixWinXpm.c

1>  codeobject.c

1>  complexobject.c

12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual Studio\VC98\include"  -I..\win -I..\generic  -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic -IW:\Python-3.5.1\externals\tk-8.6.4.2\win -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib  -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__  -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix -FoDebug_VC13\tixWinWm.obj ..\win\tixWinWm.c

1>  descrobject.c

12>  tixWinWm.c

1>  dictobject.c

12>    link.exe -debug -debugtype:cv /RELEASE /NOLOGO /MACHINE:IX86 -dll W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib W:\Python-3.5.1\externals\tcl-core-8.6.4.2\win\Debug_VC13\tclstub86.lib kernel32.lib  advapi32.lib user32.lib gdi32.lib comdlg32.lib  -out:Debug_VC13\tix84g.dll @C:\Users\eddyq\AppData\Local\Temp\nmF029.tmp

12>LINK : fatal error LNK1181: cannot open input file 'W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib'

12>NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\link.exe"' : return code '0x49d'

12>  Stop.

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: The command "setlocal

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: if not exist "W:\Python-3.5.1\externals\tcltk\lib\tix8.4.3\tix84g.dll" goto build

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: goto :eof

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: :build

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: set VCINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: cd /D "W:\Python-3.5.1\externals\tix-8.4.3.6\win"

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: nmake /nologo -f makefile.vc MACHINE=IX86 DEBUG=1 NODEBUG=0 TCL_DBGX=g TK_DBGX=g TCL_MAJOR=8 TCL_MINOR=6 TCL_PATCH=4 BUILDDIRTOP="Debug_VC13" TCL_DIR="W:\Python-3.5.1\externals\tcl-core-8.6.4.2" TK_DIR="W:\Python-3.5.1\externals\tk-8.6.4.2" INSTALL_DIR="W:\Python-3.5.1\externals\tcltk" all install

12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(44,5): error MSB3073: " exited with code 2.

1>  enumobject.c

1>  exceptions.c

1>  fileobject.c

1>  floatobject.c

1>  frameobject.c

1>  funcobject.c

1>  genobject.c

1>  iterobject.c

1>  listobject.c

1>  longobject.c

1>  memoryobject.c

1>  methodobject.c

1>  moduleobject.c

1>  namespaceobject.c

1>  Generating Code...

1>  Compiling...

1>  object.c

1>  obmalloc.c

1>  odictobject.c

1>  rangeobject.c

1>  setobject.c

1>  sliceobject.c

1>  structseq.c

1>  tupleobject.c

1>  typeobject.c

1>  unicodectype.c

1>  unicodeobject.c

1>  weakrefobject.c

1>  acceler.c

1>  bitset.c

1>  firstsets.c

1>  grammar.c

1>  grammar1.c

1>  listnode.c

1>  metagrammar.c

1>  myreadline.c

1>  Generating Code...

1>  Compiling...

1>  node.c

1>  parser.c

1>  parsetok.c

1>  tokenizer.c

1>  invalid_parameter_handler.c

1>  winreg.c

1>  config.c

1>  getpathp.c

1>  msvcrtmodule.c

1>  pyhash.c

1>  random.c

1>  _warnings.c

1>  asdl.c

1>  ast.c

1>  bltinmodule.c

1>  ceval.c

1>  codecs.c

1>  compile.c

< 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/f775d42e/attachment-0001.html>

From brett at python.org  Wed Jan 13 19:14:05 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 14 Jan 2016 00:14:05 +0000
Subject: [Python-Dev] Debugging using VS 2015
In-Reply-To: <005e01d14e5e$09660bb0$1c322310$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
 <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com>
Message-ID: <CAP1=2W4oPCMDjfWTJ91F+6=4a7_1iNrqvFHgVhVztdC6=aEkHQ@mail.gmail.com>

Then your paths might be off; I would try with a fresh checkout and make
sure you can build Python cleanly before trying your embedded case to make
sure your tooling is set up (e.g., you don't have Mercurial installed so
that's a cause of one of your error messages).

On Wed, 13 Jan 2016 at 15:58 Eddy Quicksall <Eddy at quicksall.com> wrote:

> Those files already exist:
>
>
>
> W:\Python-3.5.1>PCbuild\get_externals.bat
>
> Fetching external libraries...
>
> bzip2-1.0.6 already exists, skipping.
>
> nasm-2.11.06 already exists, skipping.
>
> openssl-1.0.2d already exists, skipping.
>
> sqlite-3.8.11.0 already exists, skipping.
>
> tcl-core-8.6.4.2 already exists, skipping.
>
> tk-8.6.4.2 already exists, skipping.
>
> tix-8.4.3.6 already exists, skipping.
>
> xz-5.0.5 already exists, skipping.
>
> Finished.
>
>
>
> W:\Python-3.5.1>
>
>
>
> *From:* Brett Cannon [mailto:brett at python.org]
> *Sent:* Wednesday, January 13, 2016 4:30 PM
> *To:* Eddy Quicksall; python-dev at python.org
> *Subject:* Re: [Python-Dev] Debugging using VS 2015
>
>
>
> Use PCbuild/get_externals.bat
> <https://hg.python.org/cpython/file/default/PCbuild/get_externals.bat> to
> download the external dependencies.
>
>
>
> On Wed, 13 Jan 2016 at 13:25 Eddy Quicksall <Eddy at quicksall.com> wrote:
>
> I am using 3.5.1. I?m adding an extension for my special case.
>
>
>
> I know this list is for development and that?s what I?m doing but I would
> like to use VS 2015 to do my debugging. If there is another list I should
> use to get the build to work then please let me know the list.
>
>
>
> I have been able to build using ?PCbuild\build.bat ?e? but I?m not able to
> build using PCbuild\pcbuild.sln. It gives lots of errors.
>
>
>
> I?m using Debug/Win32.
>
>
>
> Below is the output. Note that lots of .c files are missing ?. Is there
> another package I should get (but then why would it build correctly with
> build.bat)? It could be that I need to add the equivalent to the ?-e?
> option.
>
>
>
> 1>------ Rebuild All started: Project: pythoncore, Configuration: Debug
> Win32 ------
>
> 2>------ Rebuild All started: Project: tcl, Configuration: Debug Win32
> ------
>
> 3>------ Rebuild All started: Project: ssleay, Configuration: Debug Win32
> ------
>
> 4>------ Rebuild All started: Project: sqlite3, Configuration: Debug Win32
> ------
>
> 1>  Killing any running python_d.exe instances...
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(403,5): warning MSB3073: The
> command "hg id -b >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgbranch.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(404,5): warning MSB3073: The
> command "hg id -i >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgversion.txt"" exited
> with code 9009.
>
> 1>  'hg' is not recognized as an internal or external command,
>
> 1>  operable program or batch file.
>
> 1>W:\Python-3.5.1\PCbuild\pythoncore.vcxproj(405,5): warning MSB3073: The
> command "hg id -t >
> "W:\Python-3.5.1\PCbuild\obj\\win32_Debug\pythoncore\hgtag.txt"" exited
> with code 9009.
>
> 2>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 5>------ Rebuild All started: Project: python3dll, Configuration: Debug
> Win32 ------
>
> 1>  _bisectmodule.c
>
> 5>  python3dll.c
>
> 4>  sqlite3.c
>
> 3>  d1_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_both.c': No such file or
> directory
>
> 3>  d1_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_lib.c': No such file or
> directory
>
> 1>  _codecsmodule.c
>
> 3>  d1_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_pkt.c': No such file or
> directory
>
> 3>  d1_srtp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\d1_srtp.c': No such file or
> directory
>
> 3>  s2_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_clnt.c': No such file or
> directory
>
> 3>  s2_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_enc.c': No such file or
> directory
>
> 3>  s2_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_lib.c': No such file or
> directory
>
> 3>  s2_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_meth.c': No such file or
> directory
>
> 3>  s2_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_pkt.c': No such file or
> directory
>
> 3>  s2_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s2_srvr.c': No such file or
> directory
>
> 3>  s23_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_clnt.c': No such file or
> directory
>
> 3>  s23_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_lib.c': No such file or
> directory
>
> 3>  s23_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_meth.c': No such file or
> directory
>
> 3>  s23_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_pkt.c': No such file or
> directory
>
> 3>  s23_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s23_srvr.c': No such file or
> directory
>
> 3>  s3_both.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_both.c': No such file or
> directory
>
> 3>  s3_cbc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_cbc.c': No such file or
> directory
>
> 3>  s3_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_clnt.c': No such file or
> directory
>
> 3>  s3_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_enc.c': No such file or
> directory
>
> 3>  s3_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_lib.c': No such file or
> directory
>
> 3>  Generating Code...
>
> 3>  Compiling...
>
> 3>  s3_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_meth.c': No such file or
> directory
>
> 3>  s3_pkt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_pkt.c': No such file or
> directory
>
> 3>  s3_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\s3_srvr.c': No such file or
> directory
>
> 3>  ssl_algs.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_algs.c': No such file or
> directory
>
> 3>  ssl_asn1.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_asn1.c': No such file or
> directory
>
> 3>  ssl_cert.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_cert.c': No such file or
> directory
>
> 3>  ssl_ciph.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_ciph.c': No such file or
> directory
>
> 3>  ssl_err.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err.c': No such file or
> directory
>
> 3>  ssl_err2.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_err2.c': No such file or
> directory
>
> 3>  ssl_lib.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_lib.c': No such file or
> directory
>
> 3>  ssl_rsa.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_rsa.c': No such file or
> directory
>
> 3>  ssl_sess.c
>
> 1>  _collectionsmodule.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\ssl_sess.c': No such file or
> directory
>
> 3>  t1_clnt.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_clnt.c': No such file or
> directory
>
> 3>  t1_enc.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_enc.c': No such file or
> directory
>
> 3>  t1_lib.c
>
> 1>  _csv.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_lib.c': No such file or
> directory
>
> 3>  t1_meth.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_meth.c': No such file or
> directory
>
> 3>  t1_reneg.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_reneg.c': No such file or
> directory
>
> 3>  t1_srvr.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\t1_srvr.c': No such file or
> directory
>
> 3>  tls_srp.c
>
> 3>c1 : fatal error C1083: Cannot open source file:
> 'W:\Python-3.5.1\externals\openssl-1.0.2d\ssl\tls_srp.c': No such file or
> directory
>
> 3>  Generating Code...
>
> 6>------ Rebuild All started: Project: tk, Configuration: Debug Win32
> ------
>
> 1>  _functoolsmodule.c
>
> 1>  _heapqmodule.c
>
> 1>  _json.c
>
> 1>  _localemodule.c
>
> 4>     Creating library W:\Python-3.5.1\PCBuild\win32\sqlite3_d.lib and
> object W:\Python-3.5.1\PCBuild\win32\sqlite3_d.exp
>
> 5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_dstub.lib
> and object W:\Python-3.5.1\PCBuild\win32\python3_dstub.exp
>
> 1>  _lsprof.c
>
> 5>     Creating library W:\Python-3.5.1\PCBuild\win32\python3_d.lib and
> object W:\Python-3.5.1\PCBuild\win32\python3_d.exp
>
> 5>  python3dll.vcxproj -> W:\Python-3.5.1\PCBuild\win32\python3_d.dll
>
> 7>------ Rebuild All started: Project: libeay, Configuration: Debug Win32
> ------
>
> 6>C:\Program Files
> (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.MakeFile.Targets(41,5):
> warning MSB8005: The property 'NMakeReBuildCommandLine' doesn't exist.
> Skipping...
>
> 8>------ Skipped Rebuild All: Project: bdist_wininst, Configuration: Debug
> Win32 ------
>
> 8>Project not selected to build for this solution configuration
>
> 9>------ Skipped Rebuild All: Project: xxlimited, Configuration: Release
> Win32 ------
>
> 9>Project not selected to build for this solution configuration
>
> 10>------ Rebuild All started: Project: pylauncher, Configuration: Debug
> Win32 ------
>
> 1>  _math.c
>
> 4>  sqlite3.vcxproj -> W:\Python-3.5.1\PCBuild\win32\sqlite3_d.dll
>
> 11>------ Rebuild All started: Project: pywlauncher, Configuration: Debug
> Win32 ------
>
> 10>  launcher.c
>
> 1>  _pickle.c
>
> 7>  nasm: fatal: unable to open input file
> `W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm'
>
> 1>  _randommodule.c
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: The command
> "setlocal
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: set
> PATH=W:\Python-3.5.1\externals\\nasm-2.11.06\;%PATH%
>
> 7>W:\Python-3.5.1\PCbuild\openssl.props(69,5): error MSB3073: nasm.exe -f
> win32 -o
> "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp\\win32_Debug\libeay\aes-586.obj"
> "W:\Python-3.5.1\externals\openssl-1.0.2d\tmp32\aes-586.asm"" exited with
> code 1.
>
> 12>------ Rebuild All started: Project: tix, Configuration: Debug Win32
> ------
>
> 11>  launcher.c
>
> 1>  _sre.c
>
> 1>  _stat.c
>
> 1>  _struct.c
>
> 12>    md Debug_VC13
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixClass.obj ..\generic\tixClass.c
>
> 12>  tixClass.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixCmds.obj ..\generic\tixCmds.c
>
> 12>  tixCmds.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixCompat.obj ..\generic\tixCompat.c
>
> 12>  tixCompat.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiImg.obj ..\generic\tixDiImg.c
>
> 12>  tixDiImg.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiITxt.obj ..\generic\tixDiITxt.c
>
> 12>  tixDiITxt.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiStyle.obj ..\generic\tixDiStyle.c
>
> 12>  tixDiStyle.c
>
> 11>  pywlauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\pyw_d.exe
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDItem.obj ..\generic\tixDItem.c
>
> 12>  tixDItem.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiText.obj ..\generic\tixDiText.c
>
> 12>  tixDiText.c
>
> 1>  _weakref.c
>
> 1>  arraymodule.c
>
> 1>  atexitmodule.c
>
> 1>  audioop.c
>
> 1>  binascii.c
>
> 1>  Generating Code...
>
> 1>  Compiling...
>
> 1>  cmathmodule.c
>
> 1>  _datetimemodule.c
>
> 1>  errnomodule.c
>
> 1>  faulthandler.c
>
> 1>  gcmodule.c
>
> 10>  pylauncher.vcxproj -> W:\Python-3.5.1\PCBuild\win32\py_d.exe
>
> 1>  hashtable.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixDiWin.obj ..\generic\tixDiWin.c
>
> 12>  tixDiWin.c
>
> 1>  itertoolsmodule.c
>
> 1>  main.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixError.obj ..\generic\tixError.c
>
> 12>  tixError.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixForm.obj ..\generic\tixForm.c
>
> 1>  mathmodule.c
>
> 12>  tixForm.c
>
> 1>  md5module.c
>
> 1>  mmapmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixFormMisc.obj ..\generic\tixFormMisc.c
>
> 12>  tixFormMisc.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGeometry.obj ..\generic\tixGeometry.c
>
> 12>  tixGeometry.c
>
> 1>  _opcode.c
>
> 1>  _operator.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrid.obj ..\generic\tixGrid.c
>
> 1>  parsermodule.c
>
> 12>  tixGrid.c
>
> 1>  posixmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrData.obj ..\generic\tixGrData.c
>
> 12>  tixGrData.c
>
> 1>  rotatingtree.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrRC.obj ..\generic\tixGrRC.c
>
> 1>  sha1module.c
>
> 12>  tixGrRC.c
>
> 1>  sha256module.c
>
> 1>  sha512module.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrFmt.obj ..\generic\tixGrFmt.c
>
> 12>  tixGrFmt.c
>
> 1>  signalmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrSel.obj ..\generic\tixGrSel.c
>
> 12>  tixGrSel.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixGrUtl.obj ..\generic\tixGrUtl.c
>
> 12>  tixGrUtl.c
>
> 1>  Compiling...
>
> 1>  symtablemodule.c
>
> 1>  _threadmodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLCol.obj ..\generic\tixHLCol.c
>
> 12>  tixHLCol.c
>
> 1>  _tracemalloc.c
>
> 1>  timemodule.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLHdr.obj ..\generic\tixHLHdr.c
>
> 12>  tixHLHdr.c
>
> 1>  xxsubtype.c
>
> 1>  zipimport.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHLInd.obj ..\generic\tixHLInd.c
>
> 12>  tixHLInd.c
>
> 1>  zlibmodule.c
>
> 1>  fileio.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixHList.obj ..\generic\tixHList.c
>
> 12>  tixHList.c
>
> 1>  bytesio.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixImgCmp.obj ..\generic\tixImgCmp.c
>
> 12>  tixImgCmp.c
>
> 1>  stringio.c
>
> 1>  bufferedio.c
>
> 12>..\generic\tixImgCmp.c(169): warning C4090: 'function': different
> 'const' qualifiers
>
> 12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 2
> different from declaration
>
> 12>..\generic\tixImgCmp.c(169): warning C4028: formal parameter 5
> different from declaration
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixImgXpm.obj ..\generic\tixImgXpm.c
>
> 12>  tixImgXpm.c
>
> 1>  iobase.c
>
> 1>  textio.c
>
> 12>..\generic\tixImgXpm.c(88): warning C4090: 'function': different
> 'const' qualifiers
>
> 12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 2 different
> from declaration
>
> 12>..\generic\tixImgXpm.c(88): warning C4028: formal parameter 5 different
> from declaration
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixInit.obj ..\generic\tixInit.c
>
> 1>  _iomodule.c
>
> 12>  tixInit.c
>
> 1>  adler32.c
>
> 1>  compress.c
>
> 1>  crc32.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixList.obj ..\generic\tixList.c
>
> 1>  deflate.c
>
> 12>  tixList.c
>
> 1>  infback.c
>
> 1>  inffast.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixMethod.obj ..\generic\tixMethod.c
>
> 1>  Compiling...
>
> 1>  inflate.c
>
> 12>  tixMethod.c
>
> 1>  inftrees.c
>
> 1>  trees.c
>
> 1>  uncompr.c
>
> 1>  zutil.c
>
> 1>  _codecs_cn.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixNBFrame.obj ..\generic\tixNBFrame.c
>
> 12>  tixNBFrame.c
>
> 1>  _codecs_hk.c
>
> 1>  _codecs_iso2022.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixOption.obj ..\generic\tixOption.c
>
> 12>  tixOption.c
>
> 1>  _codecs_jp.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixSmpLs.obj ..\generic\tixSmpLs.c
>
> 1>  _codecs_kr.c
>
> 12>  tixSmpLs.c
>
> 1>  _codecs_tw.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixScroll.obj ..\generic\tixScroll.c
>
> 12>  tixScroll.c
>
> 1>  multibytecodec.c
>
> 1>  _winapi.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixTList.obj ..\generic\tixTList.c
>
> 12>  tixTList.c
>
> 1>  abstract.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixUtils.obj ..\generic\tixUtils.c
>
> 12>  tixUtils.c
>
> 1>  accu.c
>
> 1>  boolobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWCmpt.obj ..\win\tixWCmpt.c
>
> 1>  bytes_methods.c
>
> 12>  tixWCmpt.c
>
> 1>  bytearrayobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWidget.obj ..\generic\tixWidget.c
>
> 12>  tixWidget.c
>
> 1>  bytesobject.c
>
> 1>  capsule.c
>
> 1>  Generating Code...
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinDraw.obj ..\win\tixWinDraw.c
>
> 12>  tixWinDraw.c
>
> 1>  Compiling...
>
> 1>  cellobject.c
>
> 1>  classobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinXpm.obj ..\win\tixWinXpm.c
>
> 12>  tixWinXpm.c
>
> 1>  codeobject.c
>
> 1>  complexobject.c
>
> 12>    cl.exe -DWIN32 -D_WIN32 -D_MT -DSTDC_HEADERS -D_DLL -c -W3 -nologo
> -MD -FpDebug_VC13\  -Od -Zi -I"c:\Program Files\Microsoft Visual
> Studio\VC98\include"  -I..\win -I..\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\win
> -IW:\Python-3.5.1\externals\tk-8.6.4.2\xlib
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\generic
> -IW:\Python-3.5.1\externals\tcl-core-8.6.4.2\win -D__WIN32__
> -D_CRT_SECURE_NO_WARNINGS -DUSE_TCL_STUBS=1 -DUSE_TK_STUBS=1 -DBUILD_tix
> -FoDebug_VC13\tixWinWm.obj ..\win\tixWinWm.c
>
> 1>  descrobject.c
>
> 12>  tixWinWm.c
>
> 1>  dictobject.c
>
> 12>    link.exe -debug -debugtype:cv /RELEASE /NOLOGO /MACHINE:IX86 -dll
> W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib
> W:\Python-3.5.1\externals\tcl-core-8.6.4.2\win\Debug_VC13\tclstub86.lib
> kernel32.lib  advapi32.lib user32.lib gdi32.lib comdlg32.lib
> -out:Debug_VC13\tix84g.dll @C:\Users\eddyq\AppData\Local\Temp\nmF029.tmp
>
> 12>LINK : fatal error LNK1181: cannot open input file
> 'W:\Python-3.5.1\externals\tk-8.6.4.2\win\Debug_VC13\tkstub86.lib'
>
> 12>NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual
> Studio 14.0\VC\bin\link.exe"' : return code '0x49d'
>
> 12>  Stop.
>
> 12>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.Make
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/ae657763/attachment-0001.html>

From paulson at busiq.com  Wed Jan 13 19:18:21 2016
From: paulson at busiq.com (Matthew Paulson)
Date: Wed, 13 Jan 2016 19:18:21 -0500
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <6AFE80F6-AFF2-4B2E-BA98-E3F24C4CCF73@yahoo.com>
References: <5696A647.6090501@busiq.com>
 <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
 <5696D485.2030607@busiq.com> <6AFE80F6-AFF2-4B2E-BA98-E3F24C4CCF73@yahoo.com>
Message-ID: <5696E94D.3060508@busiq.com>

Hi Andrew:

These are all good points, and I defer to your experience -- I am new to 
python internals, but the fact remains that after multiple iterations of 
our embedded test case, we are seeing continued allocations (DS2015) and 
growth of the working set (windows task manager).  If your are pooling 
resources on the free list, wouldn't you expect these items to get 
reused and for things to stabilize after a while?  We're not seeing that.

I think Victor's suggestion of a very simple test case is probably the 
best idea.  I'll try to put that together in the next few days and if it 
also demonstrates the problem, then I'll submit it here.

Thanks for your time and help.

Best,

Matt

On 1/13/2016 6:45 PM, Andrew Barnert wrote:
> On Jan 13, 2016, at 14:49, Matthew Paulson <paulson at busiq.com 
> <mailto:paulson at busiq.com>> wrote:
>
>> Hi Victor:
>>
>> No, I'm using the new heap analysis functions in DS2015.
>
> Isn't that going to report any memory that Python's higher level 
> allocators hold in their freelists as leaked, even though it isn't leaked?
>
>> We think we have found one issue. In the following sequence, dict has 
>> no side effects, yet it is used -- unless someone can shed light on 
>> why dict is used in this case:
>
> Where do you see an issue here? The dict will have one ref, so the 
> decref at the end should return it to the freelist.
>
> Also, it looks like there _is_ a side effect here. When you add a 
> bunch of elements to a dict, it grows. When you delete a bunch of 
> elements, it generally doesn't shrink. But when you clear the dict, it 
> does shrink. So, copying it to a temporary dict, clearing it, updating 
> it from the temporary dict, and then releasing the temporary dict 
> should force it to shrink.
>
> So, the overall effect should be that you have a smaller hash table 
> for the builtins dict, and a chunk of memory sitting on the freelists 
> ready to be reused. If your analyzer is showing the freelists as 
> leaked, this will look like a net leak rather than a net recovery, but 
> that's just a problem in the analyzer.
>
> Of course I could be wrong, but I think the first step is to rule out 
> the possibility that you're measuring the wrong thing...
>
>> /* Clear the modules dict. */
>>     PyDict_Clear(modules);
>>     /* Restore the original builtins dict, to ensure that any
>>        user data gets cleared. */
>>     dict = PyDict_Copy(interp->builtins);
>>     if (dict == NULL)
>>         PyErr_Clear();
>>     PyDict_Clear(interp->builtins);
>>     if (PyDict_Update(interp->builtins, interp->builtins_copy))
>>         PyErr_Clear();
>>     Py_XDECREF(dict);
>>
>> And removing dict from this sequence seems to have fixed one of the 
>> issues, yielding 14k per iteration.
>
>> Simple program: Good idea.  We will try that -- right now it's 
>> embedded in a more complex environment, but we have tried to strip it 
>> down to a very simple sequence.
>>
>> The next item on our list is memory that is not getting freed after 
>> running simple string.  It's in the parsertok sequence -- it seems 
>> that the syntax tree is not getting cleared -- but this opinion is 
>> preliminary.
>>
>> Best,
>>
>> Matt
>>
>> On 1/13/2016 5:10 PM, Victor Stinner wrote:
>>> Hi,
>>>
>>> 2016-01-13 20:32 GMT+01:00 Matthew Paulson<paulson at busiq.com>:
>>>> I've spent some time performing memory leak analysis while using Python in an embedded configuration.
>>> Hum, did you try tracemalloc?
>>>
>>> https://docs.python.org/dev/library/tracemalloc.html
>>> https://pytracemalloc.readthedocs.org/
>>>
>>>> Is there someone in the group that would like to discuss this topic.  There seems to be other leaks as well.  I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.
>>> Are you able to reproduce the leak with a simple program?
>>>
>>> Victor
>>>
>>>
>>
>> -- 
>> <MattSig.JPG>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org <mailto:Python-Dev at python.org>
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: 
>> https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com

-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/20710a99/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160113/20710a99/attachment-0001.jpe>

From steve.dower at python.org  Wed Jan 13 19:50:46 2016
From: steve.dower at python.org (Steve Dower)
Date: Wed, 13 Jan 2016 16:50:46 -0800
Subject: [Python-Dev] Debugging using VS 2015
In-Reply-To: <005e01d14e5e$09660bb0$1c322310$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
 <5695C54A.3070101@sdamon.com> <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com>
Message-ID: <5696F0E6.8090700@python.org>

On 13Jan2016 1556, Eddy Quicksall wrote:
> Those files already exist:
>
> W:\Python-3.5.1>PCbuild\get_externals.bat
> Fetching external libraries...
> bzip2-1.0.6 already exists, skipping.
> nasm-2.11.06 already exists, skipping.
> openssl-1.0.2d already exists, skipping.
> sqlite-3.8.11.0 already exists, skipping.
> tcl-core-8.6.4.2 already exists, skipping.
> tk-8.6.4.2 already exists, skipping.
> tix-8.4.3.6 already exists, skipping.
> xz-5.0.5 already exists, skipping.
> Finished.
> W:\Python-3.5.1>

You may need to delete the externals/openssl-1.0.2d directory and run 
get_externals.bat again.

Sometimes when you try to build without having downloaded the externals 
it creates enough directories that the script thinks it exists when it 
isn't all there.

Cheers,
Steve

From ncoghlan at gmail.com  Wed Jan 13 20:58:47 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Jan 2016 11:58:47 +1000
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696E94D.3060508@busiq.com>
References: <5696A647.6090501@busiq.com>
 <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
 <5696D485.2030607@busiq.com>
 <6AFE80F6-AFF2-4B2E-BA98-E3F24C4CCF73@yahoo.com>
 <5696E94D.3060508@busiq.com>
Message-ID: <CADiSq7cqmPFe-g13Skuux6Dwu-kr84yL_nX+tvm8RqvT5DACzg@mail.gmail.com>

On 14 January 2016 at 10:18, Matthew Paulson <paulson at busiq.com> wrote:

> Hi Andrew:
>
> These are all good points, and I defer to your experience -- I am new to
> python internals, but the fact remains that after multiple iterations of
> our embedded test case, we are seeing continued allocations (DS2015) and
> growth of the working set (windows task manager).  If your are pooling
> resources on the free list, wouldn't you expect these items to get reused
> and for things to stabilize after a while?  We're not seeing that.
>
> I think Victor's suggestion of a very simple test case is probably the
> best idea.  I'll try to put that together in the next few days and if it
> also demonstrates the problem, then I'll submit it here.
>

If you want to throw your debugger at it, there's an existing
subinterpreter test case in _testembed that should exhibit any
Initialize/Finalize leaks:
https://github.com/python/cpython/blob/master/Programs/_testembed.c#L36

However, if there is one, our existing automated leak monitoring
unfortunately wouldn't pick it up, as the embedding tests run in a
subprocess rather than the main test process.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/0f19523a/attachment.html>

From trent at trent.me  Wed Jan 13 22:47:15 2016
From: trent at trent.me (Trent Nelson)
Date: Thu, 14 Jan 2016 03:47:15 +0000
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696E94D.3060508@busiq.com>
References: <5696A647.6090501@busiq.com>
 <CAMpsgwYpHS7750bVMVwrgwrn_Pn5y=we9f5fCeDuo9kCqRDgMw@mail.gmail.com>
 <5696D485.2030607@busiq.com> <6AFE80F6-AFF2-4B2E-BA98-E3F24C4CCF73@yahoo.com>
 <5696E94D.3060508@busiq.com>
Message-ID: <BLUPR02MB1665F6205E02F5255C56B048DCCC0@BLUPR02MB1665.namprd02.prod.outlook.com>

Gflags/umdh is pretty useful on Windows, I used it to track down a few quirky PyParallel memory leaks.  Steps:


1.       Enable global flags: gflags ?i python.exe +ust

2.       Launch Python.

3.       Enable the umdh tracer: umdh ?p:<python.exe pid> -f:d1.log

4.       Kill it after a short run.

5.       Re-launch Python.

6.       Enable it again: umdh ?p:<python.exe pid> -f:d2.log

7.       Let it run for longer ? long enough to make sure it?s leaking memory.

8.       Kill it, then generate a dump file: umdh ?d d1.log d2.log > dump.txt

(Those steps were pretty specific to my particular situation, but it should at least be a reasonable starting point for what to google to find out more.)

Here are two sample outputs that pin-pointed the exact leak path:


+   49116 (  49116 -      0)      6 allocs        BackTrace9763CA0

+       6 (      6 -      0)        BackTrace9763CA0        allocations



ntdll!RtlpCallInterceptRoutine+40

ntdll!RtlAllocateHeap+79846

SQLSRV32!SQLAllocateMemory+26

markSQLSRV32!SQLAllocConnect+F6

SQLSRV32!SQLAllocHandle+83

ODBC32!RetcodeDriverInit+2D9

ODBC32!SQLInternalDriverConnectW+2F

ODBC32!CDispenser::CreateResource+DB

comsvcs!CHolder::SafeDispenserDriver::CreateResource+43

comsvcs!CHolder::AllocResource+24D

ODBC32!CDispenser::TryAllocResource+6E

ODBC32!CDispenser::GetActiveConnection+72

ODBC32!SQLDriverConnectW+9D4

pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)

pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)

pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)

python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)

python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)

python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)

python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)



+   48432 (  48432 -      0)      6 allocs        BackTrace97635E0

+       6 (      6 -      0)        BackTrace97635E0        allocations



ntdll!RtlpCallInterceptRoutine+40

ntdll!RtlAllocateHeap+79846

SQLSRV32!SQLAllocateMemory+26

SQLSRV32!SQLAllocConnect+4D

SQLSRV32!SQLAllocHandle+83

ODBC32!RetcodeDriverInit+2D9

ODBC32!SQLInternalDriverConnectW+2F

ODBC32!CDispenser::CreateResource+DB

comsvcs!CHolder::SafeDispenserDriver::CreateResource+43

comsvcs!CHolder::AllocResource+24D

ODBC32!CDispenser::TryAllocResource+6E

ODBC32!CDispenser::GetActiveConnection+72

ODBC32!SQLDriverConnectW+9D4

pyodbc!Connect+14F (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 85)

pyodbc!Connection_New+CD (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\connection.cpp, 166)

pyodbc!mod_connect+579 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.cpp, 378)

python33!PyCFunction_Call+F3 (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 84)

python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)

python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)

python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)



You can see that the leak is coming from SQLAllocateMemory in this case, which we trace back to pyodbc Connection_New().  In this particular case, we were continually creating new connections, but never deallocating them.

Another example:


+ 9420800 ( 9420800 -      0)   2300 allocs        BackTrace8E96BE0

+    2300 (   2300 -      0)        BackTrace8E96BE0        allocations



ntdll!RtlpCallInterceptRoutine+40

ntdll!RtlAllocateHeap+79846

python33!_PyTLSHeap_Init+9E (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3040)

python33!_PyTLSHeap_Malloc+B0 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3124)

python33!_PyHeap_Malloc+55 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 3170)

python33!PyUnicode_New+149 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 1051)

python33!PyUnicode_DecodeUTF8Stateful+D2 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 4910)

python33!PyUnicode_InternFromString+16 (c:\users\trent\home\src\pyparallel\objects\unicodeobject.c, 14505)

python33!PyObject_GetAttrString+2E (c:\users\trent\home\src\pyparallel\objects\object.c, 881)

pyodbc!lowercase+18 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\pyodbcmodule.h, 69)

pyodbc!execute+772 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\cursor.cpp, 869)

pyodbc!Cursor_execute+D4 (c:\users\trent\home\src\pyparallel\contrib\pyodbc\src\cursor.cpp, 945)

python33!PyCFunction_Call+12D (c:\users\trent\home\src\pyparallel\objects\methodobject.c, 117)

python33!call_function+371 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4130)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!fast_function+113 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4219)

python33!call_function+529 (c:\users\trent\home\src\pyparallel\python\ceval.c, 4152)

python33!PyEval_EvalFrameEx+356C (c:\users\trent\home\src\pyparallel\python\ceval.c, 2745)

python33!PyEval_EvalCodeEx+B4D (c:\users\trent\home\src\pyparallel\python\ceval.c, 3500)

python33!function_call+1BB (c:\users\trent\home\src\pyparallel\objects\funcobject.c, 639)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!method_call+F9 (c:\users\trent\home\src\pyparallel\objects\classobject.c, 353)

python33!PyObject_Call+7C (c:\users\trent\home\src\pyparallel\objects\abstract.c, 2036)

python33!PyEval_CallObjectWithKeywords+16C (c:\users\trent\home\src\pyparallel\python\ceval.c, 4011)

python33!PxSocket_IOLoop+1249 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 9128)

python33!PxSocket_IOCallback+295 (c:\users\trent\home\src\pyparallel\python\pyparallel.c, 10612)

KERNEL32!BasepTpIoCallback+59

ntdll!TppIopExecuteCallback+182

ntdll!TppWorkerThread+8B4



We can trace this leak back to lowercase(), which was interning a string.  At this point, we were intercepting string interning and making it thread local allocation, but those allocations were never being freed.


From: Python-Dev [mailto:python-dev-bounces+trent=snakebite.org at python.org] On Behalf Of Matthew Paulson
Sent: Wednesday, January 13, 2016 7:18 PM
To: Andrew Barnert <abarnert at yahoo.com>
Cc: Python Dev <python-dev at python.org>; Gary F. Desrochers <desrochers at busiq.com>
Subject: Re: [Python-Dev] Discussion related to memory leaks requested

Hi Andrew:

These are all good points, and I defer to your experience -- I am new to python internals, but the fact remains that after multiple iterations of our embedded test case, we are seeing continued allocations (DS2015) and growth of the working set (windows task manager).  If your are pooling resources on the free list, wouldn't you expect these items to get reused and for things to stabilize after a while?  We're not seeing that.

I think Victor's suggestion of a very simple test case is probably the best idea.  I'll try to put that together in the next few days and if it also demonstrates the problem, then I'll submit it here.

Thanks for your time and help.

Best,

Matt
On 1/13/2016 6:45 PM, Andrew Barnert wrote:
On Jan 13, 2016, at 14:49, Matthew Paulson <paulson at busiq.com<mailto:paulson at busiq.com>> wrote:

Hi Victor:

No, I'm using the new heap analysis functions in DS2015.

Isn't that going to report any memory that Python's higher level allocators hold in their freelists as leaked, even though it isn't leaked?


We think we have found one issue. In the following sequence, dict has no side effects, yet it is used -- unless someone can shed light on why dict is used in this case:

Where do you see an issue here? The dict will have one ref, so the decref at the end should return it to the freelist.

Also, it looks like there _is_ a side effect here. When you add a bunch of elements to a dict, it grows. When you delete a bunch of elements, it generally doesn't shrink. But when you clear the dict, it does shrink. So, copying it to a temporary dict, clearing it, updating it from the temporary dict, and then releasing the temporary dict should force it to shrink.

So, the overall effect should be that you have a smaller hash table for the builtins dict, and a chunk of memory sitting on the freelists ready to be reused. If your analyzer is showing the freelists as leaked, this will look like a net leak rather than a net recovery, but that's just a problem in the analyzer.

Of course I could be wrong, but I think the first step is to rule out the possibility that you're measuring the wrong thing...

/* Clear the modules dict. */
    PyDict_Clear(modules);
    /* Restore the original builtins dict, to ensure that any
       user data gets cleared. */
    dict = PyDict_Copy(interp->builtins);
    if (dict == NULL)
        PyErr_Clear();
    PyDict_Clear(interp->builtins);
    if (PyDict_Update(interp->builtins, interp->builtins_copy))
        PyErr_Clear();
    Py_XDECREF(dict);

And removing dict from this sequence seems to have fixed one of the issues, yielding 14k per iteration.

Simple program: Good idea.  We will try that -- right now it's embedded in a more complex environment, but we have tried to strip it down to a very simple sequence.

The next item on our list is memory that is not getting freed after running simple string.  It's in the parsertok sequence -- it seems that the syntax tree is not getting cleared -- but this opinion is preliminary.

Best,

Matt
On 1/13/2016 5:10 PM, Victor Stinner wrote:

Hi,



2016-01-13 20:32 GMT+01:00 Matthew Paulson <paulson at busiq.com><mailto:paulson at busiq.com>:

I've spent some time performing memory leak analysis while using Python in an embedded configuration.

Hum, did you try tracemalloc?



https://docs.python.org/dev/library/tracemalloc.html

https://pytracemalloc.readthedocs.org/



Is there someone in the group that would like to discuss this topic.  There seems to be other leaks as well.  I'm new to Python-dev, but willing to help or work with someone who is more familiar with these areas than I.

Are you able to reproduce the leak with a simple program?



Victor





--
<MattSig.JPG>
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com

--
[cid:image001.jpg at 01D14E51.93AED0C0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/718f888d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 38491 bytes
Desc: image001.jpg
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/718f888d/attachment-0001.jpg>

From benjamin at python.org  Thu Jan 14 00:34:29 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 13 Jan 2016 21:34:29 -0800
Subject: [Python-Dev] Modifying the self-signed.pythontest.net
 certificate
In-Reply-To: <CA+eR4cF61-SkCO7-2eMOutnfHtk_CDGHuBJQBvSYxx1ARhj=hg@mail.gmail.com>
References: <CA+eR4cF61-SkCO7-2eMOutnfHtk_CDGHuBJQBvSYxx1ARhj=hg@mail.gmail.com>
Message-ID: <1452749669.3075604.491708178.7CD07DB8@webmail.messagingengine.com>



On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
> In order to fix the SSL test suite
> <https://bugs.python.org/issue25940>, I would like to modify the
> certificate used by https://self-signed.pythontest.net. So far I have
> a patch ready for the pythontestdotnet repository, but I want to know
> if I can just push to that repository, or if other steps are required.

It should suffice to update that repo. I suppose the server might have
to be kicked, too, as it's a cert change. Ping this thread if that
doesn't work.

From benjamin at python.org  Thu Jan 14 00:42:24 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 13 Jan 2016 21:42:24 -0800
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5696A647.6090501@busiq.com>
References: <5696A647.6090501@busiq.com>
Message-ID: <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>

This is a "well-known" issue. Parts of the interpreter (and especially,
extension modules) cheerfully stash objects in global variables with no
way to clean them up. Fixing this is a large project, which probably
involves implementing PEP 489.

On Wed, Jan 13, 2016, at 11:32, Matthew Paulson wrote:
> Hi:
> 
> I've spent some time performing memory leak analysis while using Python 
> in an embedded configuration.
> 
> The pattern is:
> 
>     Py_Initialize();
> 
>     ... run empty python source file ...
> 
>     Py_Finalize();
> 
> 
> I've identified several suspect areas including dictionary maitenace in 
> import.c:~ 414
> 
>      /* Clear the modules dict. */
>      PyDict_Clear(modules);
>      /* Restore the original builtins dict, to ensure that any
>         user data gets cleared. */
>      dict = PyDict_Copy(interp->builtins);
>      if (dict == NULL)
>          PyErr_Clear();
>      PyDict_Clear(interp->builtins);
>      if (PyDict_Update(interp->builtins, interp->builtins_copy))
>          PyErr_Clear();
>      Py_XDECREF(dict);
>      /* Clear module dict copies stored in the interpreter state */
> 
> 
> Is there someone in the group that would like to discuss this topic.  
> There seems to be other leaks as well.  I'm new to Python-dev, but 
> willing to help or work with someone who is more familiar with these 
> areas than I.
> 
> Thanks,
> 
> Matt
> 
> 
> -- 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> Email had 1 attachment:
> + MattSig.JPG
>   52k (image/jpeg)

From vadmium+py at gmail.com  Thu Jan 14 02:31:02 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 14 Jan 2016 07:31:02 +0000
Subject: [Python-Dev] Modifying the self-signed.pythontest.net
 certificate
In-Reply-To: <1452749669.3075604.491708178.7CD07DB8@webmail.messagingengine.com>
References: <CA+eR4cF61-SkCO7-2eMOutnfHtk_CDGHuBJQBvSYxx1ARhj=hg@mail.gmail.com>
 <1452749669.3075604.491708178.7CD07DB8@webmail.messagingengine.com>
Message-ID: <CA+eR4cGAaV=LjFyKqSFHgDqM0_Qju+bnv-itkAZ+hChDdoen=Q@mail.gmail.com>

On 14 January 2016 at 05:34, Benjamin Peterson <benjamin at python.org> wrote:
> On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
>> In order to fix the SSL test suite
>> <https://bugs.python.org/issue25940>, I would like to modify the
>> certificate used by https://self-signed.pythontest.net. So far I have
>> a patch ready for the pythontestdotnet repository, but I want to know
>> if I can just push to that repository, or if other steps are required.
>
> It should suffice to update that repo. I suppose the server might have
> to be kicked, too, as it's a cert change. Ping this thread if that
> doesn't work.

It seems to have worked, thanks. Not immediately, but after waiting
~20 minutes the certificate is now updated. I am working on an update
for all branches 3.2+ and 2.7; I expect test_httplib will be broken
until I fix it.

BTW <https://self-signed.pythontest.net/> is giving me a 403 Forbidden
response, but it was doing this before I pushed my change, so I think
it?s not my fault :)

From benjamin at python.org  Thu Jan 14 02:46:50 2016
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 13 Jan 2016 23:46:50 -0800
Subject: [Python-Dev] Modifying the self-signed.pythontest.net
 certificate
In-Reply-To: <CA+eR4cGAaV=LjFyKqSFHgDqM0_Qju+bnv-itkAZ+hChDdoen=Q@mail.gmail.com>
References: <CA+eR4cF61-SkCO7-2eMOutnfHtk_CDGHuBJQBvSYxx1ARhj=hg@mail.gmail.com>
 <1452749669.3075604.491708178.7CD07DB8@webmail.messagingengine.com>
 <CA+eR4cGAaV=LjFyKqSFHgDqM0_Qju+bnv-itkAZ+hChDdoen=Q@mail.gmail.com>
Message-ID: <1452757610.3100460.491767618.0B033637@webmail.messagingengine.com>



On Wed, Jan 13, 2016, at 23:31, Martin Panter wrote:
> On 14 January 2016 at 05:34, Benjamin Peterson <benjamin at python.org>
> wrote:
> > On Wed, Jan 13, 2016, at 13:15, Martin Panter wrote:
> >> In order to fix the SSL test suite
> >> <https://bugs.python.org/issue25940>, I would like to modify the
> >> certificate used by https://self-signed.pythontest.net. So far I have
> >> a patch ready for the pythontestdotnet repository, but I want to know
> >> if I can just push to that repository, or if other steps are required.
> >
> > It should suffice to update that repo. I suppose the server might have
> > to be kicked, too, as it's a cert change. Ping this thread if that
> > doesn't work.
> 
> It seems to have worked, thanks. Not immediately, but after waiting
> ~20 minutes the certificate is now updated. I am working on an update
> for all branches 3.2+ and 2.7; I expect test_httplib will be broken
> until I fix it.

Thank you for taking care of this issue.

> 
> BTW <https://self-signed.pythontest.net/> is giving me a 403 Forbidden
> response, but it was doing this before I pushed my change, so I think
> it?s not my fault :)

Fixed.

From ncoghlan at gmail.com  Thu Jan 14 04:45:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 14 Jan 2016 19:45:02 +1000
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
References: <5696A647.6090501@busiq.com>
 <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
Message-ID: <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>

On 14 January 2016 at 15:42, Benjamin Peterson <benjamin at python.org> wrote:
> This is a "well-known" issue. Parts of the interpreter (and especially,
> extension modules) cheerfully stash objects in global variables with no
> way to clean them up. Fixing this is a large project, which probably
> involves implementing PEP 489.

The actual multi-phase extension module import system from 489 was
implemented for 3.5, but indeed, the modules with stashed global state
haven't been converted yet.

I didn't think we loaded any of those by default, though...

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From encukou at gmail.com  Thu Jan 14 05:16:34 2016
From: encukou at gmail.com (Petr Viktorin)
Date: Thu, 14 Jan 2016 11:16:34 +0100
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
References: <5696A647.6090501@busiq.com>
 <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
 <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
Message-ID: <56977582.2090109@gmail.com>

On 01/14/2016 10:45 AM, Nick Coghlan wrote:
> On 14 January 2016 at 15:42, Benjamin Peterson <benjamin at python.org> wrote:
>> This is a "well-known" issue. Parts of the interpreter (and especially,
>> extension modules) cheerfully stash objects in global variables with no
>> way to clean them up. Fixing this is a large project, which probably
>> involves implementing PEP 489.
> 
> The actual multi-phase extension module import system from 489 was
> implemented for 3.5, but indeed, the modules with stashed global state
> haven't been converted yet.

The hairy details on why the global variables haven't yet gone away are
on import-sig [0]. Nick suggested a workable solution there that I
really need to go back to and implement.

[0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html




From skrah.temporarily at gmail.com  Thu Jan 14 05:37:59 2016
From: skrah.temporarily at gmail.com (Stefan Krah)
Date: Thu, 14 Jan 2016 10:37:59 +0000 (UTC)
Subject: [Python-Dev] Discussion related to memory leaks requested
References: <5696A647.6090501@busiq.com>
 <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
 <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
 <56977582.2090109@gmail.com>
Message-ID: <loom.20160114T113543-2@post.gmane.org>

Petr Viktorin <encukou <at> gmail.com> writes:
> The hairy details on why the global variables haven't yet gone away are
> on import-sig [0]. Nick suggested a workable solution there that I
> really need to go back to and implement.
> 
> [0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html

I want to add here that existing schemes for eliminating global variables
are inefficient (20% speed hit for _decimal), so a complete solution
would have to address that as well.


Stefan Krah


From brett at python.org  Thu Jan 14 13:36:38 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 14 Jan 2016 18:36:38 +0000
Subject: [Python-Dev] PEP 506 secrets module
In-Reply-To: <20151016005711.GC11980@ando.pearwood.info>
References: <20151016005711.GC11980@ando.pearwood.info>
Message-ID: <CAP1=2W6p3CK-_ksvf7Jxk9UC2ugniraqkRGvv2FGrVxnvuhv=Q@mail.gmail.com>

I noticed an article about default rand usage in Go
<http://blog.sgmansfield.com/2016/01/the-hidden-dangers-of-default-rand/>
from the Go Weekly newsletter and it reminded me about PEP 506 and the
secrets module. That's when I noticed that the PEP is still open. What is
the current blocker on the PEP?

On Thu, 15 Oct 2015 at 17:57 Steven D'Aprano <steve at pearwood.info> wrote:

> Hi,
>
> As extensively discussed on Python-Ideas, the secrets module and PEP 506
> is (I hope) ready for pronouncement.
>
> https://www.python.org/dev/peps/pep-0506/
>
> There is code and tests here:
>
> https://bitbucket.org/sdaprano/secrets
>
>
> or you can run
>
> hg clone https://sdaprano at bitbucket.org/sdaprano/secrets
>
>
> The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.
>
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/f2dc27ef/attachment.html>

From guido at python.org  Thu Jan 14 13:47:09 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 14 Jan 2016 10:47:09 -0800
Subject: [Python-Dev] PEP 506 secrets module
In-Reply-To: <CAP1=2W6p3CK-_ksvf7Jxk9UC2ugniraqkRGvv2FGrVxnvuhv=Q@mail.gmail.com>
References: <20151016005711.GC11980@ando.pearwood.info>
 <CAP1=2W6p3CK-_ksvf7Jxk9UC2ugniraqkRGvv2FGrVxnvuhv=Q@mail.gmail.com>
Message-ID: <CAP7+vJ+7cR3do5LOApqFjR+PvavKdOpA2Q0rLC0pYAAS4LGwKQ@mail.gmail.com>

I think the discussion petered out and nobody asked me to approve it yet
(or I lost track of it). I'm almost happy to approve it in the current
state. My only quibble is with some naming -- I'm not sure that a
super-generic name like 'equal' is better than the original
('compare_digest'), and I would have picked a different name for token_url
-- probably token_urlsafe. But maybe Steven can convince me that the names
currently in the PEP are better. (I also don't like the wishy-washy
position of the PEP on the actual specs of the proposed functions. But I'm
fine with the actual implementation shown as the spec.)

On Thu, Jan 14, 2016 at 10:36 AM, Brett Cannon <brett at python.org> wrote:

> I noticed an article about default rand usage in Go
> <http://blog.sgmansfield.com/2016/01/the-hidden-dangers-of-default-rand/>
> from the Go Weekly newsletter and it reminded me about PEP 506 and the
> secrets module. That's when I noticed that the PEP is still open. What is
> the current blocker on the PEP?
>
> On Thu, 15 Oct 2015 at 17:57 Steven D'Aprano <steve at pearwood.info> wrote:
>
>> Hi,
>>
>> As extensively discussed on Python-Ideas, the secrets module and PEP 506
>> is (I hope) ready for pronouncement.
>>
>> https://www.python.org/dev/peps/pep-0506/
>>
>> There is code and tests here:
>>
>> https://bitbucket.org/sdaprano/secrets
>>
>>
>> or you can run
>>
>> hg clone https://sdaprano at bitbucket.org/sdaprano/secrets
>>
>>
>> The code is written for and tested on Python 2.6, 2.7, 3.1 - 3.4.
>>
>>
>>
>> --
>> Steve
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/54202db4/attachment.html>

From paulson at busiq.com  Thu Jan 14 14:25:19 2016
From: paulson at busiq.com (Matthew Paulson)
Date: Thu, 14 Jan 2016 14:25:19 -0500
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
References: <5696A647.6090501@busiq.com>
 <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
 <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
Message-ID: <5697F61F.7050004@busiq.com>

Hi All:

I've created a simple program to make sure I wasn't lying to you all ;->

Here it is:

for (ii = 0; ii < 100; ii++)
         {
             Py_Initialize();

             if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
                 printf("PyRun_SimpleString() failed\n");
             else
             {
                 if (PyRun_SimpleString(p) == -1)
                     printf("PyRun_SimpleString() failed\n");

                 Py_CLEAR(code);
             }

             Py_Finalize();
         }

This sequence causes about 10k growth per iteration and after many 
cycles, there's no indication that any pooling logic is helping. Our 
"useful" example is slightly more complex, and therefore may explain why 
I was seeing about 16k per iteration.

Unless I've done something obviously wrong, I tend to believe Benjamin's 
claim that this issue is well known.

Suggestion: I have had great success with similar problems in the past 
by using a pools implementation sitting on top of what I call a "block 
memory allocator".   The bottom (block) allocator grabs large blocks 
from the heap and then doles them out to the pools layer, which in turn 
doles them out to the requester.  When client memory is freed -- it is 
NOT -- rather it's added to the pool which contains like-sized blocks -- 
call it an "organized free list". This is a very, very fast way to 
handle high allocation frequency patterns.  Finally, during shutdown, 
the pool simply vaporizes and the block allocator returns a the (fewer) 
large blocks back to the heap.  This avoids thrashing the heap, forcing 
it to coalesce inefficiently and also avoids heap fragmentation, which 
can cause unwanted growth as well...

Note that this would be a "hard-reset" of all allocated memory, and any 
global data in the text segment would also have to be cleared, but it 
would provide a fast, clean way to ensure that each invocation was 100% 
clean.

I don't claim to understand all the intricacies of the many way python 
can be embedded, but as I said, this strategy has worked very well for 
me in the past building servers written in C that have to stay up for 
months at a time.

Happy to discuss further, if anyone has any interest.

Best,

Matt




On 1/14/2016 4:45 AM, Nick Coghlan wrote:
> On 14 January 2016 at 15:42, Benjamin Peterson <benjamin at python.org> wrote:
>> This is a "well-known" issue. Parts of the interpreter (and especially,
>> extension modules) cheerfully stash objects in global variables with no
>> way to clean them up. Fixing this is a large project, which probably
>> involves implementing PEP 489.
> The actual multi-phase extension module import system from 489 was
> implemented for 3.5, but indeed, the modules with stashed global state
> haven't been converted yet.
>
> I didn't think we loaded any of those by default, though...
>
> Cheers,
> Nick.
>

-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/36a39761/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MattSig.JPG
Type: image/jpeg
Size: 38491 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160114/36a39761/attachment-0001.jpe>

From stephen at xemacs.org  Thu Jan 14 22:16:00 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 15 Jan 2016 12:16:00 +0900
Subject: [Python-Dev] PEP 509
In-Reply-To: <n740ck$fnl$1@ger.gmane.org>
References: <CA+OGgf6+hY_Nbzqbsxuk0JrPmsWmc5ekXzUKVkKgEbPA_RWVmA@mail.gmail.com>
 <56954B68.3090208@stoneleaf.us>
 <CAMpsgwZsVndnL-pK2ihA0OZ3Cmz-v-Pe0YPW+no4ON7fPp7D+w@mail.gmail.com>
 <56957D32.6020401@stoneleaf.us> <n740ck$fnl$1@ger.gmane.org>
Message-ID: <22168.25712.743971.624078@turnbull.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > While I understand the rationale against __version__, it strikes me as a 
 > better description of what it is, and easier on the brain than 
 > __cache_token__.  Maybe there is something even better, such as 
 > __seqnum__.

Or __generation__, as in "generational garbage collector"?


From ncoghlan at gmail.com  Thu Jan 14 23:19:25 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 15 Jan 2016 14:19:25 +1000
Subject: [Python-Dev] Discussion related to memory leaks requested
In-Reply-To: <5697F61F.7050004@busiq.com>
References: <5696A647.6090501@busiq.com>
 <1452750144.3077065.491709618.053F1E57@webmail.messagingengine.com>
 <CADiSq7fYG-Nw-F8HVwi+mZiMGp9FONXqsgU6oxz08H9M6yqtnw@mail.gmail.com>
 <5697F61F.7050004@busiq.com>
Message-ID: <CADiSq7ethKr1+3RA3gaBakmdGhpBRR+-zf=KKLwJ9P_tfetVKg@mail.gmail.com>

On 15 January 2016 at 05:25, Matthew Paulson <paulson at busiq.com> wrote:

> Hi All:
>
> I've created a simple program to make sure I wasn't lying to you all ;->
>
> Here it is:
>
> for (ii = 0; ii < 100; ii++)
>         {
>             Py_Initialize();
>
>             if ((code = Py_CompileString(p, "foo", Py_file_input)) == NULL)
>                 printf("PyRun_SimpleString() failed\n");
>             else
>             {
>                 if (PyRun_SimpleString(p) == -1)
>                     printf("PyRun_SimpleString() failed\n");
>
>                 Py_CLEAR(code);
>             }
>
>             Py_Finalize();
>         }
>
> This sequence causes about 10k growth per iteration and after many cycles,
> there's no indication that any pooling logic is helping.  Our "useful"
> example is slightly more complex, and therefore may explain why I was
> seeing about 16k per iteration.
>
> Unless I've done something obviously wrong, I tend to believe Benjamin's
> claim that this issue is well known.
>
> Suggestion: I have had great success with similar problems in the past by
> using a pools implementation sitting on top of what I call a "block memory
> allocator".   The bottom (block) allocator grabs large blocks from the heap
> and then doles them out to the pools layer, which in turn doles them out to
> the requester.  When client memory is freed -- it is NOT -- rather it's
> added to the pool which contains like-sized blocks -- call it an "organized
> free list". This is a very, very fast way to handle high allocation
> frequency patterns.  Finally, during shutdown, the pool simply vaporizes
> and the block allocator returns a the (fewer) large blocks back to the
> heap.  This avoids thrashing the heap, forcing it to coalesce inefficiently
> and also avoids heap fragmentation, which can cause unwanted growth as
> well...
>
> Note that this would be a "hard-reset" of all allocated memory, and any
> global data in the text segment would also have to be cleared, but it would
> provide a fast, clean way to ensure that each invocation was 100% clean.
>

CPython does use an arena based allocator, but PyFinalize doesn't purge it
(if it did, there'd be segfaults rather than memory growth when modules
keep pointers across Initialize/Finalize cycles).

Building with PYMALLOC_DEBUG and setting PYTHONMALLOCSTATS in the
environment will cause it to dump debugging info during Py_Finalize.

Building with Py_TRACE_REFS and setting PYTHONDUMPREFS also provides info
on live Python objects during shutdown.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/0933176a/attachment.html>

From status at bugs.python.org  Fri Jan 15 12:08:35 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 15 Jan 2016 18:08:35 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160115170835.81B815667D@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-01-08 - 2016-01-15)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5387 (+16)
  closed 32496 (+60)
  total  37883 (+76)

Open issues with patches: 2372 


Issues opened (51)
==================

#25668: Deadlock in logging caused by a possible race condition with "
http://bugs.python.org/issue25668  reopened by fviard

#26050: Add new StreamReader.readuntil() method
http://bugs.python.org/issue26050  opened by mmarkk

#26051: Non-data descriptors in pydoc
http://bugs.python.org/issue26051  opened by Antony.Lee

#26052: pydoc for __init__ with not docstring
http://bugs.python.org/issue26052  opened by Antony.Lee

#26053: regression in pdb output between 2.7 and 3.5
http://bugs.python.org/issue26053  opened by doughellmann

#26057: Avoid nonneeded use of PyUnicode_FromObject()
http://bugs.python.org/issue26057  opened by serhiy.storchaka

#26058: PEP 509: Add ma_version to PyDictObject
http://bugs.python.org/issue26058  opened by haypo

#26059: Integer Overflow in strop.replace()
http://bugs.python.org/issue26059  opened by Ramin Farajpour Cami

#26060: Class __dict__ iteration order changing due to type instance k
http://bugs.python.org/issue26060  opened by ncoghlan

#26065: python embedded 3.5 amd64 crash when using venv
http://bugs.python.org/issue26065  opened by Laurent Dufrechou

#26067: test_shutil fails when gid name is missing
http://bugs.python.org/issue26067  opened by Dinesh Wijekoon

#26068: re.compile() repr end quote truncated
http://bugs.python.org/issue26068  opened by ThiefMaster

#26070: Launcher fails to find in-place built binaries from earlier Py
http://bugs.python.org/issue26070  opened by mhammond

#26071: bdist_wininst created binaries fail to start and find 32bit Py
http://bugs.python.org/issue26071  opened by mhammond

#26072: pdb fails to access variables closed over
http://bugs.python.org/issue26072  opened by Antony.Lee

#26073: Update the list of magic numbers in launcher
http://bugs.python.org/issue26073  opened by serhiy.storchaka

#26075: typing.Union unifies types too broadly
http://bugs.python.org/issue26075  opened by alex.gronholm

#26076: redundant checks in tok_get in Parser\tokenizer.c
http://bugs.python.org/issue26076  opened by Oren Milman

#26077: Make slicing of immutable structures return a view instead of 
http://bugs.python.org/issue26077  opened by Filip Haglund

#26079: Build with Visual Studio 2015 using PlatformToolset=v120
http://bugs.python.org/issue26079  opened by bjoernthiel

#26081: Implement asyncio Future in C to improve performance
http://bugs.python.org/issue26081  opened by yselivanov

#26082: functools.lru_cache user specified cachedict support
http://bugs.python.org/issue26082  opened by wdv4758h

#26085: Tkinter spoils the input text
http://bugs.python.org/issue26085  opened by fresh_nick

#26086: Bug in standardmodule os
http://bugs.python.org/issue26086  opened by Johano

#26089: Duplicated keyword in distutils metadata
http://bugs.python.org/issue26089  opened by Augustin Laville

#26090: More correct string truncating in PyUnicode_FromFormat()
http://bugs.python.org/issue26090  opened by serhiy.storchaka

#26092: doctest should allow custom sys.displayhook
http://bugs.python.org/issue26092  opened by Sergey.Kirpichev

#26093: __qualname__ different when calling generator object w/ functi
http://bugs.python.org/issue26093  opened by dino.viehland

#26094: ConfigParser.get() doc to be updated according to the configpa
http://bugs.python.org/issue26094  opened by khyox

#26095: Update porting HOWTO to special-case Python 2 code, not Python
http://bugs.python.org/issue26095  opened by brett.cannon

#26098: PEP 510: Specialize functions with guards
http://bugs.python.org/issue26098  opened by haypo

#26099: site ignores ImportError when running sitecustomize and usercu
http://bugs.python.org/issue26099  opened by haypo

#26100: Add test.support.optim_args_from_interpreter_flags()
http://bugs.python.org/issue26100  opened by haypo

#26101: Lib/test/test_compileall.py fails when run directly
http://bugs.python.org/issue26101  opened by haypo

#26102: access violation in PyErrFetch if tcur==null in PyGILState_Rel
http://bugs.python.org/issue26102  opened by cberger

#26103: Contradiction in definition of "data descriptor" between (dott
http://bugs.python.org/issue26103  opened by Aaron Hall

#26106: Move licences to literal blocks
http://bugs.python.org/issue26106  opened by sizeof

#26107: code.co_lnotab: use signed line number delta to support moving
http://bugs.python.org/issue26107  opened by haypo

#26108: Calling PyInitialize with 2.7.11 on Windows x64 terminates pro
http://bugs.python.org/issue26108  opened by David Heffernan

#26109: _Py_DumpTraceback should be PyAPI_FUNC
http://bugs.python.org/issue26109  opened by John.Malmberg

#26110: Speedup method calls 1.2x
http://bugs.python.org/issue26110  opened by yselivanov

#26111: On Windows, os.scandir will keep a handle on the directory unt
http://bugs.python.org/issue26111  opened by remyroy

#26114: Rewrite math.erf() and math.erfc() from scratch
http://bugs.python.org/issue26114  opened by brett.cannon

#26117: Close directory descriptor in scandir iterator on error
http://bugs.python.org/issue26117  opened by serhiy.storchaka

#26119: Windows Installer can sometimes silently fail pip stage
http://bugs.python.org/issue26119  opened by Paul Hammant

#26120: pydoc: move __future__ imports out of the DATA block
http://bugs.python.org/issue26120  opened by Antony.Lee

#26121: Use C99 functions in math if available
http://bugs.python.org/issue26121  opened by serhiy.storchaka

#26122: Isolated mode doesn't ignore PYTHONHASHSEED
http://bugs.python.org/issue26122  opened by ncoghlan

#26123: http.client status code constants incompatible with Python 3.4
http://bugs.python.org/issue26123  opened by srittau

#26124: shlex.quote and pipes.quote do not quote shell keywords
http://bugs.python.org/issue26124  opened by Charles Daffern

#26125: Incorrect error message in the module asyncio.selector_events.
http://bugs.python.org/issue26125  opened by Paradisee



Most recent 15 issues with no replies (15)
==========================================

#26122: Isolated mode doesn't ignore PYTHONHASHSEED
http://bugs.python.org/issue26122

#26120: pydoc: move __future__ imports out of the DATA block
http://bugs.python.org/issue26120

#26117: Close directory descriptor in scandir iterator on error
http://bugs.python.org/issue26117

#26103: Contradiction in definition of "data descriptor" between (dott
http://bugs.python.org/issue26103

#26102: access violation in PyErrFetch if tcur==null in PyGILState_Rel
http://bugs.python.org/issue26102

#26094: ConfigParser.get() doc to be updated according to the configpa
http://bugs.python.org/issue26094

#26093: __qualname__ different when calling generator object w/ functi
http://bugs.python.org/issue26093

#26092: doctest should allow custom sys.displayhook
http://bugs.python.org/issue26092

#26085: Tkinter spoils the input text
http://bugs.python.org/issue26085

#26081: Implement asyncio Future in C to improve performance
http://bugs.python.org/issue26081

#26073: Update the list of magic numbers in launcher
http://bugs.python.org/issue26073

#26072: pdb fails to access variables closed over
http://bugs.python.org/issue26072

#26051: Non-data descriptors in pydoc
http://bugs.python.org/issue26051

#26040: Improve coverage and rigour of test.test_math
http://bugs.python.org/issue26040

#26038: zipfile cannot handle zip files where the archive size for a f
http://bugs.python.org/issue26038



Most recent 15 issues waiting for review (15)
=============================================

#26125: Incorrect error message in the module asyncio.selector_events.
http://bugs.python.org/issue26125

#26121: Use C99 functions in math if available
http://bugs.python.org/issue26121

#26117: Close directory descriptor in scandir iterator on error
http://bugs.python.org/issue26117

#26110: Speedup method calls 1.2x
http://bugs.python.org/issue26110

#26107: code.co_lnotab: use signed line number delta to support moving
http://bugs.python.org/issue26107

#26106: Move licences to literal blocks
http://bugs.python.org/issue26106

#26101: Lib/test/test_compileall.py fails when run directly
http://bugs.python.org/issue26101

#26100: Add test.support.optim_args_from_interpreter_flags()
http://bugs.python.org/issue26100

#26099: site ignores ImportError when running sitecustomize and usercu
http://bugs.python.org/issue26099

#26098: PEP 510: Specialize functions with guards
http://bugs.python.org/issue26098

#26089: Duplicated keyword in distutils metadata
http://bugs.python.org/issue26089

#26082: functools.lru_cache user specified cachedict support
http://bugs.python.org/issue26082

#26081: Implement asyncio Future in C to improve performance
http://bugs.python.org/issue26081

#26079: Build with Visual Studio 2015 using PlatformToolset=v120
http://bugs.python.org/issue26079

#26076: redundant checks in tok_get in Parser\tokenizer.c
http://bugs.python.org/issue26076



Top 10 most discussed issues (10)
=================================

#25887: awaiting on coroutine more than once should be an error
http://bugs.python.org/issue25887  16 msgs

#26059: Integer Overflow in strop.replace()
http://bugs.python.org/issue26059  15 msgs

#26058: PEP 509: Add ma_version to PyDictObject
http://bugs.python.org/issue26058  13 msgs

#26071: bdist_wininst created binaries fail to start and find 32bit Py
http://bugs.python.org/issue26071  12 msgs

#25596: Use scandir() to speed up the glob module
http://bugs.python.org/issue25596  10 msgs

#26068: re.compile() repr end quote truncated
http://bugs.python.org/issue26068  10 msgs

#25940: SSL tests failed due to expired svn.python.org SSL certificate
http://bugs.python.org/issue25940   9 msgs

#25994: File descriptor leaks in os.scandir()
http://bugs.python.org/issue25994   8 msgs

#26111: On Windows, os.scandir will keep a handle on the directory unt
http://bugs.python.org/issue26111   7 msgs

#25995: os.walk() consumes a lot of file descriptors
http://bugs.python.org/issue25995   6 msgs



Issues closed (58)
==================

#7944: Use the 'with' statement in conjunction with 'open' throughout
http://bugs.python.org/issue7944  closed by ezio.melotti

#11440: fix_callable should be dropped from lib2to3 / changed
http://bugs.python.org/issue11440  closed by SilentGhost

#13963: dev guide has no mention of mechanics of patch review
http://bugs.python.org/issue13963  closed by ezio.melotti

#15430: Improve filecmp documentation
http://bugs.python.org/issue15430  closed by orsenthil

#19006: UnitTest docs should have a single list of assertions
http://bugs.python.org/issue19006  closed by ezio.melotti

#19316: devguide: compiler - wording
http://bugs.python.org/issue19316  closed by ezio.melotti

#22138: patch.object doesn't restore function defaults
http://bugs.python.org/issue22138  closed by orsenthil

#22570: Better stdlib support for Path objects
http://bugs.python.org/issue22570  closed by gvanrossum

#22642: trace module: convert to argparse
http://bugs.python.org/issue22642  closed by orsenthil

#23675: glossary entry for 'method resolution order' links to somethin
http://bugs.python.org/issue23675  closed by orsenthil

#23942: Explain naming of the patch files in the bug tracker
http://bugs.python.org/issue23942  closed by ezio.melotti

#24649: python -mtrace --help is wrong
http://bugs.python.org/issue24649  closed by SilentGhost

#24752: SystemError when importing from a non-package directory
http://bugs.python.org/issue24752  closed by brett.cannon

#24786: Changes in the devguide repository are not published online in
http://bugs.python.org/issue24786  closed by ezio.melotti

#24789: ctypes doc string
http://bugs.python.org/issue24789  closed by ezio.melotti

#25347: assert_has_calls output is formatted inconsistently
http://bugs.python.org/issue25347  closed by orsenthil

#25348: Update pgo_build.bat to use --pgo flag for regrtest
http://bugs.python.org/issue25348  closed by python-dev

#25486: Resurrect inspect.getargspec() in 3.6
http://bugs.python.org/issue25486  closed by yselivanov

#25517: regex howto example in "Lookahead Assertions"
http://bugs.python.org/issue25517  closed by ezio.melotti

#25574: 2.7 incorrectly documents objects hash as equal to id
http://bugs.python.org/issue25574  closed by ezio.melotti

#25730: invisible sidebar content with code snippets
http://bugs.python.org/issue25730  closed by ezio.melotti

#25752: asyncio.readline - add customizable line separator
http://bugs.python.org/issue25752  closed by martin.panter

#25802: Finish deprecating load_module()
http://bugs.python.org/issue25802  closed by brett.cannon

#25822: Add docstrings to fields of urllib.parse results
http://bugs.python.org/issue25822  closed by orsenthil

#25967: Devguide: add 2to3 to the "Changing CPython's Grammar" checkli
http://bugs.python.org/issue25967  closed by ezio.melotti

#25986: Collections.deque maxlen: added in 2.6 or 2.7?
http://bugs.python.org/issue25986  closed by ezio.melotti

#25991: readline example eventually consumes all memory
http://bugs.python.org/issue25991  closed by ezio.melotti

#26001: Tutorial: write() does not expect string in binary mode
http://bugs.python.org/issue26001  closed by ezio.melotti

#26004: pip install lifetimes - throwing error and unable to install p
http://bugs.python.org/issue26004  closed by terry.reedy

#26025: Document pathlib.Path.__truediv__()
http://bugs.python.org/issue26025  closed by brett.cannon

#26029: Broken sentence in extending documentation
http://bugs.python.org/issue26029  closed by terry.reedy

#26030: Use PEP8 in documentation examples
http://bugs.python.org/issue26030  closed by ezio.melotti

#26054: Unable to run scripts: idle3 -r script.py
http://bugs.python.org/issue26054  closed by zach.ware

#26055: sys.argv[0] is the python file, not ""
http://bugs.python.org/issue26055  closed by Wei

#26056: installation failure
http://bugs.python.org/issue26056  closed by zach.ware

#26061: logging LogRecordFactory allow kwargs
http://bugs.python.org/issue26061  closed by vinay.sajip

#26062: IPython4 bash magic ! with {} does not work with Python 3.5.1
http://bugs.python.org/issue26062  closed by brett.cannon

#26063: Update copyright in the devguide
http://bugs.python.org/issue26063  closed by ezio.melotti

#26064: directory is getting separated
http://bugs.python.org/issue26064  closed by SilentGhost

#26066: Language on the "Cryptographic Services" documentation page is
http://bugs.python.org/issue26066  closed by python-dev

#26069: Remove the Deprecated API in trace module
http://bugs.python.org/issue26069  closed by orsenthil

#26074: Add a method to pip.. pip.require("package_name")
http://bugs.python.org/issue26074  closed by berker.peksag

#26078: Python launcher options enhancement
http://bugs.python.org/issue26078  closed by eryksun

#26080: "abandonned" -> "abandoned" in PEP 510's https://hg.python.org
http://bugs.python.org/issue26080  closed by haypo

#26083: ValueError: insecure string pickle in subprocess.Popen on Pyth
http://bugs.python.org/issue26083  closed by gregory.p.smith

#26084: HTMLParser mishandles last attribute in self-closing tag
http://bugs.python.org/issue26084  closed by ezio.melotti

#26087: PEP 0373 should be updated
http://bugs.python.org/issue26087  closed by python-dev

#26088: re
http://bugs.python.org/issue26088  closed by ezio.melotti

#26091: decimal.Decimal(0)**0 throws decimal.InvalidOperation
http://bugs.python.org/issue26091  closed by skrah

#26096: '*' glob string matches dot files in pathlib
http://bugs.python.org/issue26096  closed by gvanrossum

#26097: 2.7 documentation about TextTestRunner do not specify all the 
http://bugs.python.org/issue26097  closed by orsenthil

#26104: Reference leak in functools.partial constructor in failure cas
http://bugs.python.org/issue26104  closed by serhiy.storchaka

#26105: Python JSON module doesn't actually produce JSON
http://bugs.python.org/issue26105  closed by ezio.melotti

#26112: Error on example using "dialect" parameter. have to be: "diale
http://bugs.python.org/issue26112  closed by SilentGhost

#26113: pathlib p.match('') should return False rather than raising ex
http://bugs.python.org/issue26113  closed by gvanrossum

#26115: pathlib.glob('**') returns only directories
http://bugs.python.org/issue26115  closed by SilentGhost

#26116: CSV-module. The example code don't work. Have to be: reader = 
http://bugs.python.org/issue26116  closed by SilentGhost

#26118: String performance issue using single quotes
http://bugs.python.org/issue26118  closed by SilentGhost

From ethan at stoneleaf.us  Fri Jan 15 13:22:56 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 15 Jan 2016 10:22:56 -0800
Subject: [Python-Dev] Boolean value of an Enum member
Message-ID: <56993900.60105@stoneleaf.us>

When Enum was being designed one of the questions considered was where 
to start autonumbering: zero or one.

As I remember the discussion we chose not to start with zero because we 
didn't want an enum member to be False by default, and having a member 
with value 0 be True was discordant.  So the functional API starts with 
1 unless overridden.  In fact, according to the Enum docs:

    The reason for defaulting to ``1`` as the starting number and
    not ``0`` is that ``0`` is ``False`` in a boolean sense, but
    enum members all evaluate to ``True``.

However, if the Enum is combined with some other type (str, int, float, 
etc), then most behaviour is determined by that type -- including 
boolean evaluation.  So the empty string, 0 values, etc, will cause that 
Enum member to evaluate as False.

So the question now is:  for a standard Enum (meaning no other type 
besides Enum is involved) should __bool__ look to the value of the Enum 
member to determine True/False, or should we always be True by default 
and make the Enum creator add their own __bool__ if they want something 
different?

On the one hand we have backwards compatibility, which will take a 
version to change.

On the other hand we have a pretty basic difference in how zero/empty is 
handled between "pure" Enums and "mixed" Enums.

On the gripping hand we have . . .

Please respond with your thoughts on changing pure Enums to match mixed 
Enums or any experience you have had with relying on the "always True" 
behaviour or if you have implemented your own __bool__ to match the 
standard True/False meanings or if you have implemented your own 
__bool__ to match some other scheme entirely.

--
~Ethan~

From guido at python.org  Fri Jan 15 13:28:40 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 15 Jan 2016 10:28:40 -0800
Subject: [Python-Dev] [Python-ideas] Boolean value of an Enum member
In-Reply-To: <56993900.60105@stoneleaf.us>
References: <56993900.60105@stoneleaf.us>
Message-ID: <CAP7+vJKLgMYidjaiMxGeGKQg_E12d86K98b1i05XCorszBHhCg@mail.gmail.com>

Honestly I think it's too late to change. The proposal to change plain
Enums to False when their value is zero (or falsey) would be a huge
backward incompatibility. I don't think there's a reasonable path forward,
and also don't think there's a big reason to regret the current semantics.

On Fri, Jan 15, 2016 at 10:22 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> When Enum was being designed one of the questions considered was where to
> start autonumbering: zero or one.
>
> As I remember the discussion we chose not to start with zero because we
> didn't want an enum member to be False by default, and having a member with
> value 0 be True was discordant.  So the functional API starts with 1 unless
> overridden.  In fact, according to the Enum docs:
>
>    The reason for defaulting to ``1`` as the starting number and
>    not ``0`` is that ``0`` is ``False`` in a boolean sense, but
>    enum members all evaluate to ``True``.
>
> However, if the Enum is combined with some other type (str, int, float,
> etc), then most behaviour is determined by that type -- including boolean
> evaluation.  So the empty string, 0 values, etc, will cause that Enum
> member to evaluate as False.
>
> So the question now is:  for a standard Enum (meaning no other type
> besides Enum is involved) should __bool__ look to the value of the Enum
> member to determine True/False, or should we always be True by default and
> make the Enum creator add their own __bool__ if they want something
> different?
>
> On the one hand we have backwards compatibility, which will take a version
> to change.
>
> On the other hand we have a pretty basic difference in how zero/empty is
> handled between "pure" Enums and "mixed" Enums.
>
> On the gripping hand we have . . .
>
> Please respond with your thoughts on changing pure Enums to match mixed
> Enums or any experience you have had with relying on the "always True"
> behaviour or if you have implemented your own __bool__ to match the
> standard True/False meanings or if you have implemented your own __bool__
> to match some other scheme entirely.
>
> --
> ~Ethan~
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/a1ff96f7/attachment.html>

From barry at python.org  Fri Jan 15 13:32:58 2016
From: barry at python.org (Barry Warsaw)
Date: Fri, 15 Jan 2016 13:32:58 -0500
Subject: [Python-Dev] Boolean value of an Enum member
In-Reply-To: <56993900.60105@stoneleaf.us>
References: <56993900.60105@stoneleaf.us>
Message-ID: <20160115133258.3ca202dd@limelight.wooz.org>

On Jan 15, 2016, at 10:22 AM, Ethan Furman wrote:

>So the question now is: for a standard Enum (meaning no other type besides
>Enum is involved) should __bool__ look to the value of the Enum member to
>determine True/False, or should we always be True by default and make the
>Enum creator add their own __bool__ if they want something different?

The latter.  I think in general enums are primarily a symbolic value and don't
have truthiness.  It's also so easy to override when you define the enum that
it's not worth changing the current behavior.

Cheers,
-Barry

From davidr at openscg.com  Fri Jan 15 17:13:41 2016
From: davidr at openscg.com (Rader, David)
Date: Fri, 15 Jan 2016 17:13:41 -0500
Subject: [Python-Dev] 2.7.11 Windows Installer issues on Win2008R2
Message-ID: <CAABt7R7b6_u3w10q+ZEHXOLR6qi=EY-V7+QoOLkDsDT879CCQg@mail.gmail.com>

Problem 1:
The .manifest information for the VC runtime dll's has been changed in the
recent versions of the 2.7.x 64-bit installers for Windows. Python fails to
run on a clean Win2008R2 install after running the Python installer to
install "Just for me". The installation succeeds if "Install for all users"
is selected.

After install completes, trying to run python results in:
The application has failed to start because it's side-by-side configuration
is incorrect. Please see the application event log or use the command-line
sxstrace.exe tool for more detail.

The event viewer log shows:
Activation context generation failed for "C:\Python27\python.exe".Error in
manifest or policy file "C:\Python27\Microsoft.VC90.CRT.MANIFEST" on line
4. Component identity found in manifest does not match the identity of the
component requested. Reference is
Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8".
Definition is
Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.30729.1".
Please use sxstrace.exe for detailed diagnosis.

This means that that VC2008 SP1 dll and manifest are included in the
installer, but the Python.exe is compiled against VC2008 (_not_ SP1).
Replacing the installed manifest and VC90 with one pulled from an older
distribution with the correct 9.0.21022.8 version enables python to run.

Problem 2:
The compiled DLLs in the DLLs folder incorrectly have the VC manifest
included in them as well. This breaks the side-by-side look up, since the
VC90 dll is not in the DLLs folder. So if you try to import socket, you get
an error message like:
Traceback (most recent call last):
  File "hub\scripts\pgc.py", line 9, in <module>
    import socket
  File "C:\Python27\lib\socket.py", line 47, in <module>
    import _socket
ImportError: DLL load failed: The application has failed to start because
its si
de-by-side configuration is incorrect. Please see the application event log
or u
se the command-line sxstrace.exe tool for more detail.

Previous versions of Python for windows have had this problem but it was
corrected. It looks like it has crept back in.

--
David Rader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/daf1442b/attachment.html>

From tjreedy at udel.edu  Fri Jan 15 18:20:48 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 15 Jan 2016 18:20:48 -0500
Subject: [Python-Dev] 2.7.11 Windows Installer issues on Win2008R2
In-Reply-To: <CAABt7R7b6_u3w10q+ZEHXOLR6qi=EY-V7+QoOLkDsDT879CCQg@mail.gmail.com>
References: <CAABt7R7b6_u3w10q+ZEHXOLR6qi=EY-V7+QoOLkDsDT879CCQg@mail.gmail.com>
Message-ID: <n7bust$tad$1@ger.gmane.org>

On 1/15/2016 5:13 PM, Rader, David wrote:
[description of problems]

Please register at bugs.python.org and open a new issue for Versions 
2.7, Components: Installation, with 'benjamin.peterson' and 'loewis' on 
the Nosy List.  Copy what you wrote in the Comment: box.

-- 
Terry Jan Reedy


From Eddy at Quicksall.com  Fri Jan 15 18:31:23 2016
From: Eddy at Quicksall.com (Eddy Quicksall)
Date: Fri, 15 Jan 2016 18:31:23 -0500
Subject: [Python-Dev] C struct for Str( )
In-Reply-To: <5696F0E6.8090700@python.org>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
 <5695C54A.3070101@sdamon.com> <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com> <5696F0E6.8090700@python.org>
Message-ID: <00e201d14fec$d946ff80$8bd4fe80$@com>

I want to fill an Str() string from a C function. But I think I am using the
wrong structure (PyBytesObject). I have written a C function to dump the
Python class but as you can see the structure I'm using does not match the
data in the class. 

Can someone please tell me the correct structure:

--------- Python snip ---------
class _vendorRecord_2:
    vendorListNumber        = str()
    vendorNumber            = str()
    vendorName              = str('x' * 20)

vendorRecord_2 = _vendorRecord_2()

print(len(vendorRecord_2.vendorName))
print(vendorRecord_2.vendorName + '|')
XBaseDump_PythonString(py_object(vendorRecord_2.vendorName))

--------- C function --------
#define MS_NO_COREDLL
#undef _DEBUG
#include <W:\Python-3.5.1\Include\python.h>
DSI_DLL void CALL_TYPE XBaseDump_PythonString( PyBytesObject *pBytesObject )
{
    printf( "ob_size = %d, ob_shash = %X, ob_sval = %s\n",
pBytesObject->ob_base.ob_size, pBytesObject->ob_shash, pBytesObject->ob_sval
);
    printf( "offsetof(ob_size) = %d, offsetof(ob_sval) = %d\n", offsetof(
PyBytesObject, ob_base.ob_size ),
        offsetof( PyBytesObject, ob_sval ) );
    printf( "sizeof(PyBytesObject) = %d\n", sizeof(PyBytesObject) );
    DsmDumpBytes( pBytesObject, 32 );
}

-------- output ------------
20
xxxxxxxxxxxxxxxxxxxx|
ob_size = 20, ob_shash = 70768D53, ob_sval = ?sci
offsetof(ob_size) = 8, offsetof(ob_sval) = 16
sizeof(PyBytesObject) = 20
0000: 03 00 00 00 60 B0 DC 1D 14 00 00 00 53 8D 76 70 ....-...........
0016: E5 73 63 69 00 00 00 00 78 78 78 78 78 78 78 78 V...............




---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


From brett at python.org  Fri Jan 15 18:58:07 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 15 Jan 2016 23:58:07 +0000
Subject: [Python-Dev] C struct for Str( )
In-Reply-To: <00e201d14fec$d946ff80$8bd4fe80$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
 <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com> <5696F0E6.8090700@python.org>
 <00e201d14fec$d946ff80$8bd4fe80$@com>
Message-ID: <CAP1=2W65hxFpnyA5wk31P+aV41pm8Ai=HtT2SXF7TqYgdx0cwA@mail.gmail.com>

I don't quite see what this has to do with has to do with the development
of Python, Eddy. You can always reference the C API at
https://docs.python.org/3/c-api/index.html . And `PyBytesObject` is an
instance of `bytes` in Python.

On Fri, 15 Jan 2016 at 15:33 Eddy Quicksall <Eddy at quicksall.com> wrote:

> I want to fill an Str() string from a C function. But I think I am using
> the
> wrong structure (PyBytesObject). I have written a C function to dump the
> Python class but as you can see the structure I'm using does not match the
> data in the class.
>
> Can someone please tell me the correct structure:
>
> --------- Python snip ---------
> class _vendorRecord_2:
>     vendorListNumber        = str()
>     vendorNumber            = str()
>     vendorName              = str('x' * 20)
>
> vendorRecord_2 = _vendorRecord_2()
>
> print(len(vendorRecord_2.vendorName))
> print(vendorRecord_2.vendorName + '|')
> XBaseDump_PythonString(py_object(vendorRecord_2.vendorName))
>
> --------- C function --------
> #define MS_NO_COREDLL
> #undef _DEBUG
> #include <W:\Python-3.5.1\Include\python.h>
> DSI_DLL void CALL_TYPE XBaseDump_PythonString( PyBytesObject *pBytesObject
> )
> {
>     printf( "ob_size = %d, ob_shash = %X, ob_sval = %s\n",
> pBytesObject->ob_base.ob_size, pBytesObject->ob_shash,
> pBytesObject->ob_sval
> );
>     printf( "offsetof(ob_size) = %d, offsetof(ob_sval) = %d\n", offsetof(
> PyBytesObject, ob_base.ob_size ),
>         offsetof( PyBytesObject, ob_sval ) );
>     printf( "sizeof(PyBytesObject) = %d\n", sizeof(PyBytesObject) );
>     DsmDumpBytes( pBytesObject, 32 );
> }
>
> -------- output ------------
> 20
> xxxxxxxxxxxxxxxxxxxx|
> ob_size = 20, ob_shash = 70768D53, ob_sval = ?sci
> offsetof(ob_size) = 8, offsetof(ob_sval) = 16
> sizeof(PyBytesObject) = 20
> 0000: 03 00 00 00 60 B0 DC 1D 14 00 00 00 53 8D 76 70 ....-...........
> 0016: E5 73 63 69 00 00 00 00 78 78 78 78 78 78 78 78 V...............
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/c6ea9d06/attachment.html>

From Eddy at Quicksall.com  Fri Jan 15 19:23:39 2016
From: Eddy at Quicksall.com (Eddy Quicksall)
Date: Fri, 15 Jan 2016 19:23:39 -0500
Subject: [Python-Dev] C struct for Str( )
In-Reply-To: <CAP1=2W65hxFpnyA5wk31P+aV41pm8Ai=HtT2SXF7TqYgdx0cwA@mail.gmail.com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com>
 <5695C54A.3070101@sdamon.com> <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com> <5696F0E6.8090700@python.org>
 <00e201d14fec$d946ff80$8bd4fe80$@com>
 <CAP1=2W65hxFpnyA5wk31P+aV41pm8Ai=HtT2SXF7TqYgdx0cwA@mail.gmail.com>
Message-ID: <00e601d14ff4$2679dc30$736d9490$@com>

Sorry, I must be on the wrong list. Can you please give me the correct list?

 

Eddy

 

From: Brett Cannon [mailto:brett at python.org] 
Sent: Friday, January 15, 2016 6:58 PM
To: Eddy Quicksall; python-dev at python.org
Subject: Re: [Python-Dev] C struct for Str( )

 

I don't quite see what this has to do with has to do with the development of Python, Eddy. You can always reference the C API at https://docs.python.org/3/c-api/index.html . And `PyBytesObject` is an instance of `bytes` in Python.

 

On Fri, 15 Jan 2016 at 15:33 Eddy Quicksall <Eddy at quicksall.com> wrote:

I want to fill an Str() string from a C function. But I think I am using the
wrong structure (PyBytesObject). I have written a C function to dump the
Python class but as you can see the structure I'm using does not match the
data in the class.

Can someone please tell me the correct structure:

--------- Python snip ---------
class _vendorRecord_2:
    vendorListNumber        = str()
    vendorNumber            = str()
    vendorName              = str('x' * 20)

vendorRecord_2 = _vendorRecord_2()

print(len(vendorRecord_2.vendorName))
print(vendorRecord_2.vendorName + '|')
XBaseDump_PythonString(py_object(vendorRecord_2.vendorName))

--------- C function --------
#define MS_NO_COREDLL
#undef _DEBUG
#include <W:\Python-3.5.1\Include\python.h>
DSI_DLL void CALL_TYPE XBaseDump_PythonString( PyBytesObject *pBytesObject )
{
    printf( "ob_size = %d, ob_shash = %X, ob_sval = %s\n",
pBytesObject->ob_base.ob_size, pBytesObject->ob_shash, pBytesObject->ob_sval
);
    printf( "offsetof(ob_size) = %d, offsetof(ob_sval) = %d\n", offsetof(
PyBytesObject, ob_base.ob_size ),
        offsetof( PyBytesObject, ob_sval ) );
    printf( "sizeof(PyBytesObject) = %d\n", sizeof(PyBytesObject) );
    DsmDumpBytes( pBytesObject, 32 );
}

-------- output ------------
20
xxxxxxxxxxxxxxxxxxxx|
ob_size = 20, ob_shash = 70768D53, ob_sval = ?sci
offsetof(ob_size) = 8, offsetof(ob_sval) = 16
sizeof(PyBytesObject) = 20
0000: 03 00 00 00 60 B0 DC 1D 14 00 00 00 53 8D 76 70 ....-...........
0016: E5 73 63 69 00 00 00 00 78 78 78 78 78 78 78 78 V...............




---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/ae21b0cb/attachment.html>

From victor.pantoja at gmail.com  Fri Jan 15 20:08:15 2016
From: victor.pantoja at gmail.com (Victor Pantoja)
Date: Fri, 15 Jan 2016 23:08:15 -0200
Subject: [Python-Dev] C struct for Str( )
In-Reply-To: <00e601d14ff4$2679dc30$736d9490$@com>
References: <01f401d14dad$e984f1e0$bc8ed5a0$@com> <5695C54A.3070101@sdamon.com>
 <003201d14e3d$b6560f80$23022e80$@com>
 <CAP1=2W5bLKKxa4H998afUCzc36S0Zp9_GXGh7BGOwstSzY3n7g@mail.gmail.com>
 <005e01d14e5e$09660bb0$1c322310$@com> <5696F0E6.8090700@python.org>
 <00e201d14fec$d946ff80$8bd4fe80$@com>
 <CAP1=2W65hxFpnyA5wk31P+aV41pm8Ai=HtT2SXF7TqYgdx0cwA@mail.gmail.com>
 <00e601d14ff4$2679dc30$736d9490$@com>
Message-ID: <CAJNyChbWXobcy7rttWAVwzfT5jdXaK5xBJ_sQR+uYRirDXHwGA@mail.gmail.com>

Hi Eddy

You can try some of the lists in https://mail.python.org/mailman/listinfo.

Best,
victor

2016-01-15 22:23 GMT-02:00 Eddy Quicksall <Eddy at quicksall.com>:

> Sorry, I must be on the wrong list. Can you please give me the correct
> list?
>
>
>
> Eddy
>
>
>
> *From:* Brett Cannon [mailto:brett at python.org]
> *Sent:* Friday, January 15, 2016 6:58 PM
> *To:* Eddy Quicksall; python-dev at python.org
> *Subject:* Re: [Python-Dev] C struct for Str( )
>
>
>
> I don't quite see what this has to do with has to do with the development
> of Python, Eddy. You can always reference the C API at
> https://docs.python.org/3/c-api/index.html . And `PyBytesObject` is an
> instance of `bytes` in Python.
>
>
>
> On Fri, 15 Jan 2016 at 15:33 Eddy Quicksall <Eddy at quicksall.com> wrote:
>
> I want to fill an Str() string from a C function. But I think I am using
> the
> wrong structure (PyBytesObject). I have written a C function to dump the
> Python class but as you can see the structure I'm using does not match the
> data in the class.
>
> Can someone please tell me the correct structure:
>
> --------- Python snip ---------
> class _vendorRecord_2:
>     vendorListNumber        = str()
>     vendorNumber            = str()
>     vendorName              = str('x' * 20)
>
> vendorRecord_2 = _vendorRecord_2()
>
> print(len(vendorRecord_2.vendorName))
> print(vendorRecord_2.vendorName + '|')
> XBaseDump_PythonString(py_object(vendorRecord_2.vendorName))
>
> --------- C function --------
> #define MS_NO_COREDLL
> #undef _DEBUG
> #include <W:\Python-3.5.1\Include\python.h>
> DSI_DLL void CALL_TYPE XBaseDump_PythonString( PyBytesObject *pBytesObject
> )
> {
>     printf( "ob_size = %d, ob_shash = %X, ob_sval = %s\n",
> pBytesObject->ob_base.ob_size, pBytesObject->ob_shash,
> pBytesObject->ob_sval
> );
>     printf( "offsetof(ob_size) = %d, offsetof(ob_sval) = %d\n", offsetof(
> PyBytesObject, ob_base.ob_size ),
>         offsetof( PyBytesObject, ob_sval ) );
>     printf( "sizeof(PyBytesObject) = %d\n", sizeof(PyBytesObject) );
>     DsmDumpBytes( pBytesObject, 32 );
> }
>
> -------- output ------------
> 20
> xxxxxxxxxxxxxxxxxxxx|
> ob_size = 20, ob_shash = 70768D53, ob_sval = ?sci
> offsetof(ob_size) = 8, offsetof(ob_sval) = 16
> sizeof(PyBytesObject) = 20
> 0000: 03 00 00 00 60 B0 DC 1D 14 00 00 00 53 8D 76 70 ....-...........
> 0016: E5 73 63 69 00 00 00 00 78 78 78 78 78 78 78 78 V...............
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> This
> email has been sent from a virus-free computer protected by Avast.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.pantoja%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160115/8c7e9366/attachment.html>

From avivcohn123 at yahoo.com  Sat Jan 16 11:05:06 2016
From: avivcohn123 at yahoo.com (Aviv Cohn)
Date: Sat, 16 Jan 2016 16:05:06 +0000 (UTC)
Subject: [Python-Dev] Should inspect.getargspec take any callable?
References: <375069367.4888403.1452960306976.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <375069367.4888403.1452960306976.JavaMail.yahoo@mail.yahoo.com>

Hello :)This is my first mail here, I had an idea that I'd like to propose.
The `getargspec` function in the `inspect` module enforces the input parameter to be either a method or a function.
? ? def getargspec(func):? ? ? ? """Get the names and default values of a function's arguments.? ??? ? ? ? A tuple of four things is returned: (args, varargs, varkw, defaults).? ? ? ? 'args' is a list of the argument names (it may contain nested lists).? ? ? ? 'varargs' and 'varkw' are the names of the * and ** arguments or None.? ??? ? 'defaults' is an n-tuple of the default values of the last n arguments.? ??? ? """? ??
? ??? ? if ismethod(func):? ??? ? ? ? func = func.im_func? ??? ? if not isfunction(func):? ??? ? ? ? raise TypeError('{!r} is not a Python function'.format(func))? ??? ? args, varargs, varkw = getargs(func.func_code)? ??? ? return ArgSpec(args, varargs, varkw, func.func_defaults)
Passing in a callable which is not a function causes a TypeError to be raised.
I think in this case any callable should be allowed, allowing classes and callable objects as well.We can switch on whether `func` is a function, a class or a callable object, and pass into `getargs` the appropriate value.
What is your opinion?Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160116/134762c6/attachment.html>

From abarnert at yahoo.com  Sat Jan 16 12:23:10 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Sat, 16 Jan 2016 09:23:10 -0800
Subject: [Python-Dev] Should inspect.getargspec take any callable?
In-Reply-To: <375069367.4888403.1452960306976.JavaMail.yahoo@mail.yahoo.com>
References: <375069367.4888403.1452960306976.JavaMail.yahoo.ref@mail.yahoo.com>
 <375069367.4888403.1452960306976.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <D218C5F4-17BE-4144-A054-67125B59D129@yahoo.com>

On Jan 16, 2016, at 08:05, Aviv Cohn via Python-Dev <python-dev at python.org> wrote:
> 
> The `getargspec` function in the `inspect` module enforces the input parameter to be either a method or a function.

The `getargspec` already works with classes, callable objects, and some builtins.

It's also deprecated, in part because its API can't handle various features (like keyword-only arguments). There is an extended version that can handle some of those features, but as of 3.5 that one is deprecated as well.

The `signature` function is much easier to use, as well as being more powerful.

> 
>     def getargspec(func):
>         """Get the names and default values of a function's arguments.
>     
>         A tuple of four things is returned: (args, varargs, varkw, defaults).
>         'args' is a list of the argument names (it may contain nested lists).
>         'varargs' and 'varkw' are the names of the * and ** arguments or None.
>         'defaults' is an n-tuple of the default values of the last n arguments.
>         """
>     
>         if ismethod(func):
>             func = func.im_func
>         if not isfunction(func):
>             raise TypeError('{!r} is not a Python function'.format(func))
>         args, varargs, varkw = getargs(func.func_code)
>         return ArgSpec(args, varargs, varkw, func.func_defaults)
> 
> Passing in a callable which is not a function causes a TypeError to be raised.
> 
> I think in this case any callable should be allowed, allowing classes and callable objects as well.
> We can switch on whether `func` is a function, a class or a callable object, and pass into `getargs` the appropriate value.
> 
> What is your opinion?
> Thank you
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/abarnert%40yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160116/d61faad2/attachment.html>

From ethan at stoneleaf.us  Sat Jan 16 13:18:15 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 16 Jan 2016 10:18:15 -0800
Subject: [Python-Dev] [Python-ideas] Boolean value of an Enum member
In-Reply-To: <5699583A.8090102@canterbury.ac.nz>
References: <56993900.60105@stoneleaf.us> <5699583A.8090102@canterbury.ac.nz>
Message-ID: <569A8967.8010805@stoneleaf.us>

[resending to lists -- sorry, Greg]

On 01/15/2016 12:36 PM, Greg Ewing wrote:
> Ethan Furman wrote:

>> So the question now is:  for a standard Enum (meaning no other type
>> besides Enum is involved) should __bool__ look to the value of the
>> Enum member to determine True/False, or should we always be True by
>> default and make the Enum creator add their own __bool__ if they want
>> something different?
>
> Can't you just specify a starting value of 0 if you
> want the enum to have a false value? That doesn't
> seem too onerous to me.

You can start with zero, but unless the Enum is mixed with a numeric 
type it will evaluate to True.

Also, but there are other falsey values that a pure Enum member could 
have:  False, None, '', etc., to name a few.

However, as Barry said, writing your own is a whopping two lines of code:

   def __bool__(self):
     return bool(self._value_)

With Barry and Guido's feedback this issue is closed.

Thanks everyone!

--
~Ethan~

From ncoghlan at gmail.com  Sat Jan 16 22:32:55 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 17 Jan 2016 13:32:55 +1000
Subject: [Python-Dev] Should inspect.getargspec take any callable?
In-Reply-To: <D218C5F4-17BE-4144-A054-67125B59D129@yahoo.com>
References: <375069367.4888403.1452960306976.JavaMail.yahoo.ref@mail.yahoo.com>
 <375069367.4888403.1452960306976.JavaMail.yahoo@mail.yahoo.com>
 <D218C5F4-17BE-4144-A054-67125B59D129@yahoo.com>
Message-ID: <CADiSq7eOOMGNuf3V9XTtG4BSKH-_iy=wJFEj83N69qa-KkM2Qg@mail.gmail.com>

On 17 January 2016 at 03:23, Andrew Barnert via Python-Dev
<python-dev at python.org> wrote:
> On Jan 16, 2016, at 08:05, Aviv Cohn via Python-Dev <python-dev at python.org>
> wrote:
>
> The `getargspec` function in the `inspect` module enforces the input
> parameter to be either a method or a function.
>
>
> The `getargspec` already works with classes, callable objects, and some
> builtins.
>
> It's also deprecated, in part because its API can't handle various features
> (like keyword-only arguments). There is an extended version that can handle
> some of those features, but as of 3.5 that one is deprecated as well.
>
> The `signature` function is much easier to use, as well as being more
> powerful.

As Andrew states here, the limitations of getargspec() and
getfullargspec() are why they were deprecated in favour of
inspect.signature() in Python 3.3:
https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object

The funcsigs project provides a backport of much of that functionality
to earlier Python versions: https://pypi.python.org/pypi/funcsigs

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Sun Jan 17 14:10:40 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 17 Jan 2016 19:10:40 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
Message-ID: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>

While doing a review of http://bugs.python.org/review/26129/ I asked to
have curly braces put around all `if` statement bodies. Serhiy pointed out
that PEP 7 says curly braces are optional:
https://www.python.org/dev/peps/pep-0007/#id5. I would like to change that.

My argument is to require them to prevent bugs like the one Apple made with
OpenSSL about two years ago:
https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the curly
braces is purely an aesthetic thing while leaving them out can lead to
actual bugs.

Anyone object if I update PEP 7 to remove the optionality of curly braces
in PEP 7?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160117/f010ec86/attachment.html>

From ethan at stoneleaf.us  Sun Jan 17 16:58:52 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 17 Jan 2016 13:58:52 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <569C0E9C.1000307@stoneleaf.us>

On 01/17/2016 11:10 AM, Brett Cannon wrote:

> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can
> lead to actual bugs.

Not sure what that sentence actually says, but +1 on making them mandatory.

--
~Ethan~

From brett at python.org  Sun Jan 17 17:19:52 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 17 Jan 2016 22:19:52 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569C0E9C.1000307@stoneleaf.us>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569C0E9C.1000307@stoneleaf.us>
Message-ID: <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>

On Sun, 17 Jan 2016, 13:59 Ethan Furman <ethan at stoneleaf.us> wrote:

> On 01/17/2016 11:10 AM, Brett Cannon wrote:
>
> > https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> > curly braces is purely an aesthetic thing while leaving them out can
> > lead to actual bugs.
>
> Not sure what that sentence actually says, but +1 on making them mandatory.
>


Yeah, bad phrasing on my part. What I meant to say is leaving them off is
an aesthetic thing while requiring them is a bug prevention thing. When it
comes to writing C code I always vote for practicality over aesthetics.


> --
> ~Ethan~
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160117/91fbf5c5/attachment.html>

From robertc at robertcollins.net  Sun Jan 17 17:49:28 2016
From: robertc at robertcollins.net (Robert Collins)
Date: Mon, 18 Jan 2016 11:49:28 +1300
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569C0E9C.1000307@stoneleaf.us>
 <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>
Message-ID: <CAJ3HoZ0wSUxng_AFRA7N_JskXzgnHAZrFRy5yNSWM4QmoTTt_Q@mail.gmail.com>

+1 from me on requiring them.

On 18 January 2016 at 11:19, Brett Cannon <brett at python.org> wrote:
>
>
> On Sun, 17 Jan 2016, 13:59 Ethan Furman <ethan at stoneleaf.us> wrote:
>>
>> On 01/17/2016 11:10 AM, Brett Cannon wrote:
>>
>> > https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> > curly braces is purely an aesthetic thing while leaving them out can
>> > lead to actual bugs.
>>
>> Not sure what that sentence actually says, but +1 on making them
>> mandatory.
>
>
>
> Yeah, bad phrasing on my part. What I meant to say is leaving them off is an
> aesthetic thing while requiring them is a bug prevention thing. When it
> comes to writing C code I always vote for practicality over aesthetics.
>
>>
>> --
>> ~Ethan~
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net
>



-- 
Robert Collins <rbtcollins at hpe.com>
Distinguished Technologist
HP Converged Cloud

From guido at python.org  Sun Jan 17 21:50:51 2016
From: guido at python.org (Guido van Rossum)
Date: Sun, 17 Jan 2016 18:50:51 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAJ3HoZ0wSUxng_AFRA7N_JskXzgnHAZrFRy5yNSWM4QmoTTt_Q@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569C0E9C.1000307@stoneleaf.us>
 <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>
 <CAJ3HoZ0wSUxng_AFRA7N_JskXzgnHAZrFRy5yNSWM4QmoTTt_Q@mail.gmail.com>
Message-ID: <CAP7+vJ+ffnY-_QQD27qNiKsQS4HibtNvK-j-uawa2VoGV8-f9g@mail.gmail.com>

I'm +0. The editor I use is too smart to let me make this mistake, but I
don't object to recommending it. As usual, though, let's not start mindless
reformatting of existing code.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160117/97f06287/attachment.html>

From victor.stinner at gmail.com  Mon Jan 18 02:00:19 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 18 Jan 2016 08:00:19 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <CAMpsgwZmXOkPmw8ZJjAYxNHbsp=fkX9f9zX8MT5O77H6N8tj2g@mail.gmail.com>

I like if without braces when the body is only one line, especially when
there is no else block.

Victor


Le dimanche 17 janvier 2016, Brett Cannon <brett at python.org> a ?crit :

> While doing a review of http://bugs.python.org/review/26129/ I asked to
> have curly braces put around all `if` statement bodies. Serhiy pointed out
> that PEP 7 says curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> that.
>
> My argument is to require them to prevent bugs like the one Apple made
> with OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can lead
> to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/9de2e8c6/attachment.html>

From g.brandl at gmx.net  Mon Jan 18 02:02:20 2016
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 18 Jan 2016 08:02:20 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569C0E9C.1000307@stoneleaf.us>
 <CAP1=2W45Z+RirLVi7DfDsC4JZW1pa5Mg+=WJB9yy50KM4EZBWg@mail.gmail.com>
Message-ID: <n7i2ls$npg$1@ger.gmane.org>

On 01/17/2016 11:19 PM, Brett Cannon wrote:
> 
> 
> On Sun, 17 Jan 2016, 13:59 Ethan Furman <ethan at stoneleaf.us
> <mailto:ethan at stoneleaf.us>> wrote:
> 
>     On 01/17/2016 11:10 AM, Brett Cannon wrote:
> 
>     > https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>     > curly braces is purely an aesthetic thing while leaving them out can
>     > lead to actual bugs.
> 
>     Not sure what that sentence actually says, but +1 on making them mandatory.
> 
> 
> 
> Yeah, bad phrasing on my part. What I meant to say is leaving them off is an
> aesthetic thing while requiring them is a bug prevention thing. When it comes to
> writing C code I always vote for practicality over aesthetics.

+1.

Out of curiosity, I made a quick script to see if we had any candidates for
bugs related to this. I didn't expect any bugs to be found, since with the
amount of static checkers that have been run they should have been found.
The only problem I found was in the S390 port of libffi (#ifdef-conditional
code which wouldn't even compile).

I also found (in ast.c) two instances of semantically correct code with the
wrong indent level which I fixed (see rev 1ececa34b748).

cheers,
Georg


From storchaka at gmail.com  Mon Jan 18 03:05:46 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 18 Jan 2016 10:05:46 +0200
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <n7i6cq$f0v$1@ger.gmane.org>

On 17.01.16 21:10, Brett Cannon wrote:
> While doing a review of http://bugs.python.org/review/26129/
> <http://bugs.python..org/review/26129/> I asked to have curly braces put
> around all `if` statement bodies. Serhiy pointed out that PEP 7 says
> curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change that.
>
> My argument is to require them to prevent bugs like the one Apple made
> with OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can
> lead to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly
> braces in PEP 7?

I'm -0. The code without braces looks more clear. Especially if the body 
is one-line return, break, continue or goto statement. Sometimes it is 
appropriate to add an empty line after it for even larger clearness. On 
the other hand, there is no a precedence of bugs like the one Apple made 
in CPython sources. Mandatory braces *may be* will prevent hypothetical 
bug, but for sure make a lot of correct code harder to read.


From mal at egenix.com  Mon Jan 18 03:24:40 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 18 Jan 2016 09:24:40 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAMpsgwZmXOkPmw8ZJjAYxNHbsp=fkX9f9zX8MT5O77H6N8tj2g@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAMpsgwZmXOkPmw8ZJjAYxNHbsp=fkX9f9zX8MT5O77H6N8tj2g@mail.gmail.com>
Message-ID: <569CA148.7030404@egenix.com>

On 18.01.2016 08:00, Victor Stinner wrote:
> I like if without braces when the body is only one line, especially when
> there is no else block.

Same here.

Compilers warn about these things today, so I don't think
we need to go paranoid ;-)

> Victor
> 
> 
> Le dimanche 17 janvier 2016, Brett Cannon <brett at python.org> a ?crit :
> 
>> While doing a review of http://bugs.python.org/review/26129/ I asked to
>> have curly braces put around all `if` statement bodies. Serhiy pointed out
>> that PEP 7 says curly braces are optional:
>> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
>> that.
>>
>> My argument is to require them to prevent bugs like the one Apple made
>> with OpenSSL about two years ago:
>> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> curly braces is purely an aesthetic thing while leaving them out can lead
>> to actual bugs.
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?


-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 18 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From abarnert at yahoo.com  Mon Jan 18 03:39:42 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 18 Jan 2016 00:39:42 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <C6FBC1D9-6E91-4BC7-AB7B-708462F03492@yahoo.com>

On Jan 17, 2016, at 11:10, Brett Cannon <brett at python.org> wrote:
> 
> While doing a review of http://bugs.python.org/review/26129/ I asked to have curly braces put around all `if` statement bodies. Serhiy pointed out that PEP 7 says curly braces are optional: https://www.python.org/dev/peps/pep-0007/#id5. I would like to change that.
> 
> My argument is to require them to prevent bugs like the one Apple made with OpenSSL about two years ago: https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the curly braces is purely an aesthetic thing while leaving them out can lead to actual bugs.
> 
> Anyone object if I update PEP 7 to remove the optionality of curly braces in PEP 7?

There are two ways you could do that.

The first is to just change "braces may be omitted where C permits, but when present, they should be formatted as follows" to something like "braces must not be omitted, and should be formatted as follows", changing one-liner tests into this:

    if (!obj) {
        return -1;
    }

Alternatively, it could say something like "braces must not be omitted; when other C styles would use a braceless one-liner, a one-liner with braces should be used instead; otherwise, they should be formatted as follows", changing the same tests into:

    if (!obj) { return -1; }

The first one is obviously a much bigger change in the formatting of actual code, even if it's a simpler change to the PEP. Is that what was intended?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/bff98213/attachment-0001.html>

From ncoghlan at gmail.com  Mon Jan 18 03:47:01 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 18 Jan 2016 18:47:01 +1000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <CADiSq7fj8OoadQV+T+-S5=f8=Orq9E87m7Gtc6XYKOmmovWwSA@mail.gmail.com>

On 18 January 2016 at 05:10, Brett Cannon <brett at python.org> wrote:
> While doing a review of http://bugs.python.org/review/26129/ I asked to have
> curly braces put around all `if` statement bodies. Serhiy pointed out that
> PEP 7 says curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change that.
>
> My argument is to require them to prevent bugs like the one Apple made with
> OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the curly
> braces is purely an aesthetic thing while leaving them out can lead to
> actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces in
> PEP 7?

+1 from me, as I usually add them to code I'm editing anyway (I find
it too hard to read otherwise, especially when there's a long series
of them).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From larry at hastings.org  Mon Jan 18 03:59:12 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 18 Jan 2016 00:59:12 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <569CA960.4030401@hastings.org>



On 01/17/2016 11:10 AM, Brett Cannon wrote:
> Anyone object if I update PEP 7 to remove the optionality of curly 
> braces in PEP 7?

I'm -1.  I don't like being forced to add the curly braces when the code 
is perfectly clear without them.  If this was a frequent problem then 
I'd put up with it, but I can't recall ever making this particular 
mistake myself or seeing it in CPython source.  It seems to me like a 
fix for a problem we don't have.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/67be52de/attachment.html>

From soltysh at gmail.com  Mon Jan 18 06:42:44 2016
From: soltysh at gmail.com (Maciej Szulik)
Date: Mon, 18 Jan 2016 12:42:44 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569CA960.4030401@hastings.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
Message-ID: <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>

On Mon, Jan 18, 2016 at 9:59 AM, Larry Hastings <larry at hastings.org> wrote:

>
>
> On 01/17/2016 11:10 AM, Brett Cannon wrote:
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>
>
> I'm -1.  I don't like being forced to add the curly braces when the code
> is perfectly clear without them.  If this was a frequent problem then I'd
> put up with it, but I can't recall ever making this particular mistake
> myself or seeing it in CPython source.  It seems to me like a fix for a
> problem we don't have.
>
>
I'm +1. We don't have that problem yet and the idea Brett brought up is for
future changes that will happen.
We'll be soon moving to github, which should simplify the process of
submitting PRs from other developers
interested in making our beautiful language even more awesome. I'm quite
positive that with current review
process that kind of bug should not happen, but you never know. Having this
as a requirement is rather to
minimize the risk of potentially having such bugs. I've switched to this
style myself good couple years ago
and I find it very readable atm.

Maciej


*/arry*
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/soltysh%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/e25781cc/attachment.html>

From rosuav at gmail.com  Mon Jan 18 06:57:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Jan 2016 22:57:58 +1100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
Message-ID: <CAPTjJmqRQdZHLE4f+WL5RfMuPKfYjyZ7tzfafB84bVEFYTfS5g@mail.gmail.com>

On Mon, Jan 18, 2016 at 10:42 PM, Maciej Szulik <soltysh at gmail.com> wrote:
> On Mon, Jan 18, 2016 at 9:59 AM, Larry Hastings <larry at hastings.org> wrote:
>>
>>
>>
>> On 01/17/2016 11:10 AM, Brett Cannon wrote:
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?
>>
>>
>> I'm -1.  I don't like being forced to add the curly braces when the code
>> is perfectly clear without them.  If this was a frequent problem then I'd
>> put up with it, but I can't recall ever making this particular mistake
>> myself or seeing it in CPython source.  It seems to me like a fix for a
>> problem we don't have.
>>
>
> I'm +1. We don't have that problem yet and the idea Brett brought up is for
> future changes that will happen.
> We'll be soon moving to github, which should simplify the process of
> submitting PRs from other developers
> interested in making our beautiful language even more awesome. I'm quite
> positive that with current review
> process that kind of bug should not happen, but you never know. Having this
> as a requirement is rather to
> minimize the risk of potentially having such bugs. I've switched to this
> style myself good couple years ago
> and I find it very readable atm.

Rather than forcing people to use braces, wouldn't it be easier to
just add a linter to the toolchain that will detect those kinds of
problems and reject the commit? Which might be as simple as telling a
compiler to treat this warning as an error, and then always use that
compiler at some point before committing. (I don't know how many
compilers can check this.) I'm -1 on forcing people to use a
particular style when a script could do the same job more reliably.

ChrisA

From skrah.temporarily at gmail.com  Mon Jan 18 06:58:20 2016
From: skrah.temporarily at gmail.com (Stefan Krah)
Date: Mon, 18 Jan 2016 11:58:20 +0000 (UTC)
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <loom.20160118T125314-130@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:
> Anyone object if I update PEP 7 to remove the optionality of curly braces
in PEP 7?

I strongly prefer braces everywhere, but I'm -1 on enforcing it.



Stefan Krah


From larry at hastings.org  Mon Jan 18 07:02:38 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 18 Jan 2016 04:02:38 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAPTjJmqRQdZHLE4f+WL5RfMuPKfYjyZ7tzfafB84bVEFYTfS5g@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
 <CAPTjJmqRQdZHLE4f+WL5RfMuPKfYjyZ7tzfafB84bVEFYTfS5g@mail.gmail.com>
Message-ID: <569CD45E.2030703@hastings.org>



On 01/18/2016 03:57 AM, Chris Angelico wrote:
> Rather than forcing people to use braces, wouldn't it be easier to
> just add a linter to the toolchain that will detect those kinds of
> problems and reject the commit?

I don't understand your suggestion.  If we automatically reject commits 
that lack this style of braces, then surely we are "forcing people to 
use [them]"?


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/4b139a3f/attachment.html>

From storchaka at gmail.com  Mon Jan 18 07:04:26 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 18 Jan 2016 14:04:26 +0200
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
Message-ID: <n7ikcb$pro$1@ger.gmane.org>

On 18.01.16 13:42, Maciej Szulik wrote:
> We'll be soon moving to github, which should simplify the process of
> submitting PRs from other developers
> interested in making our beautiful language even more awesome. I'm quite
> positive that with current review
> process that kind of bug should not happen, but you never know.

If moving to GitHub will decrease the quality of source code, it is bad 
idea.



From rosuav at gmail.com  Mon Jan 18 07:09:23 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 18 Jan 2016 23:09:23 +1100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569CD45E.2030703@hastings.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
 <CAPTjJmqRQdZHLE4f+WL5RfMuPKfYjyZ7tzfafB84bVEFYTfS5g@mail.gmail.com>
 <569CD45E.2030703@hastings.org>
Message-ID: <CAPTjJmp6FASWiyF7NZFMk_2J+f3iFVmay8U3Ph7YjGsgP3or8A@mail.gmail.com>

On Mon, Jan 18, 2016 at 11:02 PM, Larry Hastings <larry at hastings.org> wrote:
> On 01/18/2016 03:57 AM, Chris Angelico wrote:
>
> Rather than forcing people to use braces, wouldn't it be easier to
> just add a linter to the toolchain that will detect those kinds of
> problems and reject the commit?
>
>
> I don't understand your suggestion.  If we automatically reject commits that
> lack this style of braces, then surely we are "forcing people to use
> [them]"?

Only in the exact situation that this is trying to prevent, where
indentation implies something that braces don't stipulate. From the
original Apple bug link:

if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    goto fail;
    goto fail;

Since there are two indented lines after the if and no braces, this
should be flagged as an error. If there's only one indented line,
braces are optional.

ChrisA

From barry at python.org  Mon Jan 18 09:49:56 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 18 Jan 2016 09:49:56 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <20160118094956.5263efe2@limelight.wooz.org>

On Jan 17, 2016, at 07:10 PM, Brett Cannon wrote:

>Anyone object if I update PEP 7 to remove the optionality of curly braces
>in PEP 7?

+1 for requiring them everywhere, -1 for a wholesale reformatting of existing
code.  New code and significant refactoring/rewriting can adopt braces
everywhere.

Cheers,
-Barry

From yselivanov.ml at gmail.com  Mon Jan 18 11:52:08 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 18 Jan 2016 11:52:08 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <569D1838.9000108@gmail.com>

I'm usually fine with code like this:

     if ( ... )
         return

But when there is a multi-line 'else' clause, I just want to use braces 
for both 'if' and its 'else'.  So, for consistency, I'm +1 for 
recommending to use braces everywhere.

And +1 for requiring to use braces if one of the clauses of the 'if' 
statement is multi-line.

Yury

On 2016-01-17 2:10 PM, Brett Cannon wrote:
> While doing a review of http://bugs.python.org/review/26129/ I asked 
> to have curly braces put around all `if` statement bodies. Serhiy 
> pointed out that PEP 7 says curly braces are optional: 
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change 
> that.
>
> My argument is to require them to prevent bugs like the one Apple made 
> with OpenSSL about two years ago: 
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the 
> curly braces is purely an aesthetic thing while leaving them out can 
> lead to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly 
> braces in PEP 7?
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com


From brett at python.org  Mon Jan 18 11:57:13 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 18 Jan 2016 16:57:13 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <n7ikcb$pro$1@ger.gmane.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CALC2N4Y_Xra9UQwzq6dL+gy1iKOCRkV6jrdmWXXKxnvrZBCSXQ@mail.gmail.com>
 <n7ikcb$pro$1@ger.gmane.org>
Message-ID: <CAP1=2W7xyTHX9W6ryqisA64EYQhn97y4-dMJdnU2v0mJEDvqaA@mail.gmail.com>

On Mon, 18 Jan 2016 at 04:05 Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 18.01.16 13:42, Maciej Szulik wrote:
> > We'll be soon moving to github, which should simplify the process of
> > submitting PRs from other developers
> > interested in making our beautiful language even more awesome. I'm quite
> > positive that with current review
> > process that kind of bug should not happen, but you never know.
>
> If moving to GitHub will decrease the quality of source code, it is bad
> idea.
>

It's not going to decrease code quality.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/b9d3f2d0/attachment-0001.html>

From fijall at gmail.com  Mon Jan 18 15:18:28 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 18 Jan 2016 21:18:28 +0100
Subject: [Python-Dev] _PyThreadState_Current
Message-ID: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>

Hi

change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and
pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can
really encounter _PyThreadState_Current being null, while crashing an
interpreter is a bit not ideal in this case.

Any chance, a) _PyThreadState_Current can be restored in visibility?
b) can I get a better API to get it in case it can be NULL, but also
in 3.5 (since it works in 3.5.0 and breaks in 3.5.1)

Cheers,
fijal

From victor.stinner at gmail.com  Mon Jan 18 15:25:18 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 18 Jan 2016 21:25:18 +0100
Subject: [Python-Dev] _PyThreadState_Current
In-Reply-To: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
References: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
Message-ID: <CAMpsgwb-Zx6whUvMenWv9ABrYnskUoPPgsUj39i22JK941P_zA@mail.gmail.com>

Hum, you can try to lie and define Py_BUILD_CORE?

Victor

2016-01-18 21:18 GMT+01:00 Maciej Fijalkowski <fijall at gmail.com>:
> Hi
>
> change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and
> pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can
> really encounter _PyThreadState_Current being null, while crashing an
> interpreter is a bit not ideal in this case.
>
> Any chance, a) _PyThreadState_Current can be restored in visibility?
> b) can I get a better API to get it in case it can be NULL, but also
> in 3.5 (since it works in 3.5.0 and breaks in 3.5.1)
>
> Cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From fijall at gmail.com  Mon Jan 18 15:31:05 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 18 Jan 2016 21:31:05 +0100
Subject: [Python-Dev] _PyThreadState_Current
In-Reply-To: <CAMpsgwb-Zx6whUvMenWv9ABrYnskUoPPgsUj39i22JK941P_zA@mail.gmail.com>
References: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
 <CAMpsgwb-Zx6whUvMenWv9ABrYnskUoPPgsUj39i22JK941P_zA@mail.gmail.com>
Message-ID: <CAK5idxTMEaMHKx=y-54FUDdYy8_g916uwwaWWhNq6F_H1mXMbg@mail.gmail.com>

Good point

On Mon, Jan 18, 2016 at 9:25 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hum, you can try to lie and define Py_BUILD_CORE?
>
> Victor
>
> 2016-01-18 21:18 GMT+01:00 Maciej Fijalkowski <fijall at gmail.com>:
>> Hi
>>
>> change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and
>> pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can
>> really encounter _PyThreadState_Current being null, while crashing an
>> interpreter is a bit not ideal in this case.
>>
>> Any chance, a) _PyThreadState_Current can be restored in visibility?
>> b) can I get a better API to get it in case it can be NULL, but also
>> in 3.5 (since it works in 3.5.0 and breaks in 3.5.1)
>>
>> Cheers,
>> fijal
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From fijall at gmail.com  Mon Jan 18 15:32:28 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 18 Jan 2016 21:32:28 +0100
Subject: [Python-Dev] _PyThreadState_Current
In-Reply-To: <CAK5idxTMEaMHKx=y-54FUDdYy8_g916uwwaWWhNq6F_H1mXMbg@mail.gmail.com>
References: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
 <CAMpsgwb-Zx6whUvMenWv9ABrYnskUoPPgsUj39i22JK941P_zA@mail.gmail.com>
 <CAK5idxTMEaMHKx=y-54FUDdYy8_g916uwwaWWhNq6F_H1mXMbg@mail.gmail.com>
Message-ID: <CAK5idxQOfs1e-uiqnQngkR2y8yO=jZSyjApYq4qK0eXkQ-m0Xw@mail.gmail.com>

seems to work thanks.

That said, I would love to have PyThreadState_Get equivalent that
would let me handle the NULL.

On Mon, Jan 18, 2016 at 9:31 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> Good point
>
> On Mon, Jan 18, 2016 at 9:25 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
>> Hum, you can try to lie and define Py_BUILD_CORE?
>>
>> Victor
>>
>> 2016-01-18 21:18 GMT+01:00 Maciej Fijalkowski <fijall at gmail.com>:
>>> Hi
>>>
>>> change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and
>>> pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can
>>> really encounter _PyThreadState_Current being null, while crashing an
>>> interpreter is a bit not ideal in this case.
>>>
>>> Any chance, a) _PyThreadState_Current can be restored in visibility?
>>> b) can I get a better API to get it in case it can be NULL, but also
>>> in 3.5 (since it works in 3.5.0 and breaks in 3.5.1)
>>>
>>> Cheers,
>>> fijal
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From victor.stinner at gmail.com  Mon Jan 18 17:43:54 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 18 Jan 2016 23:43:54 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
Message-ID: <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>

Is someone opposed to this PEP 509?

The main complain was the change on the public Python API, but the PEP
doesn't change the Python API anymore.

I'm not aware of any remaining issue on this PEP.

Victor

2016-01-11 17:49 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.
>
> Thanks to Red Hat for giving me time to experiment on this.
>
>
> HTML version:
> https://www.python.org/dev/peps/pep-0509/
>
>
> PEP: 509
> Title: Add a private version to dict
> Version: $Revision$
> Last-Modified: $Date$
> Author: Victor Stinner <victor.stinner at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 4-January-2016
> Python-Version: 3.6
>
>
> Abstract
> ========
>
> Add a new private version to builtin ``dict`` type, incremented at each
> change, to implement fast guards on namespaces.
>
>
> Rationale
> =========
>
> In Python, the builtin ``dict`` type is used by many instructions. For
> example, the ``LOAD_GLOBAL`` instruction searchs for a variable in the
> global namespace, or in the builtins namespace (two dict lookups).
> Python uses ``dict`` for the builtins namespace, globals namespace, type
> namespaces, instance namespaces, etc. The local namespace (namespace of
> a function) is usually optimized to an array, but it can be a dict too.
>
> Python is hard to optimize because almost everything is mutable: builtin
> functions, function code, global variables, local variables, ... can be
> modified at runtime. Implementing optimizations respecting the Python
> semantics requires to detect when "something changes": we will call
> these checks "guards".
>
> The speedup of optimizations depends on the speed of guard checks. This
> PEP proposes to add a version to dictionaries to implement fast guards
> on namespaces.
>
> Dictionary lookups can be skipped if the version does not change which
> is the common case for most namespaces. The performance of a guard does
> not depend on the number of watched dictionary entries, complexity of
> O(1), if the dictionary version does not change.
>
> Example of optimization: copy the value of a global variable to function
> constants.  This optimization requires a guard on the global variable to
> check if it was modified. If the variable is modified, the variable must
> be loaded at runtime when the function is called, instead of using the
> constant.
>
> See the `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ for the concrete usage of
> guards to specialize functions and for the rationale on Python static
> optimizers.
>
>
> Guard example
> =============
>
> Pseudo-code of an fast guard to check if a dictionary entry was modified
> (created, updated or deleted) using an hypothetical
> ``dict_get_version(dict)`` function::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.value = dict.get(key, UNSET)
>             self.version = dict_get_version(dict)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             version = dict_get_version(self.dict)
>             if version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             value = self.dict.get(self.key, UNSET)
>             if value is self.value:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.version = version
>                 return True
>
>             # the key was modified
>             return False
>
>
> Usage of the dict version
> =========================
>
> Specialized functions using guards
> ----------------------------------
>
> The `PEP 510 -- Specialized functions with guards
> <https://www.python.org/dev/peps/pep-0510/>`_ proposes an API to support
> specialized functions with guards. It allows to implement static
> optimizers for Python without breaking the Python semantics.
>
> Example of a static Python optimizer: the astoptimizer of the `FAT
> Python <http://faster-cpython.readthedocs.org/fat_python.html>`_ project
> implements many optimizations which require guards on namespaces.
> Examples:
>
> * Call pure builtins: to replace ``len("abc")`` with ``3``, guards on
>   ``builtins.__dict__['len']`` and ``globals()['len']`` are required
> * Loop unrolling: to unroll the loop ``for i in range(...): ...``,
>   guards on ``builtins.__dict__['range']`` and ``globals()['range']``
>   are required
>
>
> Pyjion
> ------
>
> According of Brett Cannon, one of the two main developers of Pyjion,
> Pyjion can also benefit from dictionary version to implement
> optimizations.
>
> Pyjion is a JIT compiler for Python based upon CoreCLR (Microsoft .NET
> Core runtime).
>
>
> Unladen Swallow
> ---------------
>
> Even if dictionary version was not explicitly mentionned, optimization
> globals and builtins lookup was part of the Unladen Swallow plan:
> "Implement one of the several proposed schemes for speeding lookups of
> globals and builtins." Source: `Unladen Swallow ProjectPlan
> <https://code.google.com/p/unladen-swallow/wiki/ProjectPlan>`_.
>
> Unladen Swallow is a fork of CPython 2.6.1 adding a JIT compiler
> implemented with LLVM. The project stopped in 2011: `Unladen Swallow
> Retrospective
> <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_.
>
>
> Changes
> =======
>
> Add a ``ma_version`` field to the ``PyDictObject`` structure with the C
> type ``PY_INT64_T``, 64-bit unsigned integer. New empty dictionaries are
> initilized to version ``0``. The version is incremented at each change:
>
> * ``clear()`` if the dict was non-empty
> * ``pop(key)`` if the key exists
> * ``popitem()`` if the dict is non-empty
> * ``setdefault(key, value)`` if the `key` does not exist
> * ``__detitem__(key)`` if the key exists
> * ``__setitem__(key, value)`` if the `key` doesn't exist or if the value
>   is different
> * ``update(...)`` if new values are different than existing values (the
>   version can be incremented multiple times)
>
> Example using an hypothetical ``dict_get_version(dict)`` function::
>
>     >>> d = {}
>     >>> dict_get_version(d)
>     0
>     >>> d['key'] = 'value'
>     >>> dict_get_version(d)
>     1
>     >>> d['key'] = 'new value'
>     >>> dict_get_version(d)
>     2
>     >>> del d['key']
>     >>> dict_get_version(d)
>     3
>
> If a dictionary is created with items, the version is also incremented
> at each dictionary insertion. Example::
>
>     >>> d=dict(x=7, y=33)
>     >>> dict_get_version(d)
>     2
>
> The version is not incremented if an existing key is set to the same
> value. For efficiency, values are compared by their identity:
> ``new_value is old_value``, not by their content:
> ``new_value == old_value``. Example::
>
>     >>> d={}
>     >>> value = object()
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>     >>> d['key'] = value
>     >>> dict_get_version(d)
>     2
>
> .. note::
>    CPython uses some singleton like integers in the range [-5; 257],
>    empty tuple, empty strings, Unicode strings of a single character in
>    the range [U+0000; U+00FF], etc. When a key is set twice to the same
>    singleton, the version is not modified.
>
>
> Implementation
> ==============
>
> The `issue #26058: PEP 509: Add ma_version to PyDictObject
> <https://bugs.python.org/issue26058>`_ contains a patch implementing
> this PEP.
>
> On pybench and timeit microbenchmarks, the patch does not seem to add
> any overhead on dictionary operations.
>
> When the version does not change, ``PyDict_GetItem()`` takes 14.8 ns for
> a dictioanry lookup, whereas a guard check only takes 3.8 ns. Moreover,
> a guard can watch for multiple keys. For example, for an optimization
> using 10 global variables in a function, 10 dictionary lookups costs 148
> ns, whereas the guard still only costs 3.8 ns when the version does not
> change (39x as fast).
>
>
> Integer overflow
> ================
>
> The implementation uses the C unsigned integer type ``PY_INT64_T`` to
> store the version, a 64 bits unsigned integer. The C code uses
> ``version++``. On integer overflow, the version is wrapped to ``0`` (and
> then continue to be incremented) according to the C standard.
>
> After an integer overflow, a guard can succeed whereas the watched
> dictionary key was modified. The bug occurs if the dictionary is
> modified at least ``2 ** 64`` times between two checks of the guard and
> if the new version (theorical value with no integer overflow) is equal
> to the old version modulo ``2 ** 64``.
>
> If a dictionary is modified each nanosecond, an overflow takes longer
> than 584 years. Using a 32-bit version, the overflow occurs only after 4
> seconds. That's why a 64-bit unsigned type is also used on 32-bit
> systems. A dictionary lookup at the C level takes 14.8 ns.
>
> A risk of a bug every 584 years is acceptable.
>
>
> Alternatives
> ============
>
> Expose the version at Python level as a read-only __version__ property
> ----------------------------------------------------------------------
>
> The first version of the PEP proposed to expose the dictionary version
> as a read-only ``__version__`` property at Python level, and also to add
> the property to ``collections.UserDict`` (since this type must mimick
> the ``dict`` API).
>
> There are multiple issues:
>
> * To be consistent and avoid bad surprises, the version must be added to
>   all mapping types. Implementing a new mapping type would require extra
>   work for no benefit, since the version is only required on the
>   ``dict`` type in practice.
> * All Python implementations must implement this new property, it gives
>   more work to other implementations, whereas they may not use the
>   dictionary version at all.
> * The ``__version__`` can be wrapped on integer overflow. It is error
>   prone: using ``dict.__version__ <= guard_version`` is wrong,
>   ``dict.__version__ == guard_version`` must be used instead to reduce
>   the risk of bug on integer overflow (even if the integer overflow is
>   unlikely in practice).
> * Exposing the dictionary version at Python level can lead the
>   false assumption on performances. Checking ``dict.__version__`` at
>   the Python level is not faster than a dictionary lookup. A dictionary
>   lookup has a cost of 48.7 ns and checking a guard has a cost of 47.5
>   ns, the difference is only 1.2 ns (3%)::
>
>
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}' 'd["33"] == 33'
>     10000000 loops, best of 3: 0.0487 usec per loop
>     $ ./python -m timeit -s 'd = {str(i):i for i in range(100)}'
> 'd.__version__ == 100'
>     10000000 loops, best of 3: 0.0475 usec per loop
>
> Bikeshedding on the property name:
>
> * ``__cache_token__``: name proposed by Nick Coghlan, name coming from
>   `abc.get_cache_token()
>   <https://docs.python.org/3/library/abc.html#abc.get_cache_token>`_.
> * ``__version__``
> * ``__timestamp__``
>
>
> Add a version to each dict entry
> --------------------------------
>
> A single version per dictionary requires to keep a strong reference to
> the value which can keep the value alive longer than expected. If we add
> also a version per dictionary entry, the guard can only store the entry
> version to avoid the strong reference to the value (only strong
> references to the dictionary and to the key are needed).
>
> Changes: add a ``me_version`` field to the ``PyDictKeyEntry`` structure,
> the field has the C type ``PY_INT64_T``. When a key is created or
> modified, the entry version is set to the dictionary version which is
> incremented at any change (create, modify, delete).
>
> Pseudo-code of an fast guard to check if a dictionary key was modified
> using hypothetical ``dict_get_version(dict)``
> ``dict_get_entry_version(dict)`` functions::
>
>     UNSET = object()
>
>     class GuardDictKey:
>         def __init__(self, dict, key):
>             self.dict = dict
>             self.key = key
>             self.dict_version = dict_get_version(dict)
>             self.entry_version = dict_get_entry_version(dict, key)
>
>         def check(self):
>             """Return True if the dictionary entry did not changed."""
>
>             # read the version field of the dict structure
>             dict_version = dict_get_version(self.dict)
>             if dict_version == self.version:
>                 # Fast-path: dictionary lookup avoided
>                 return True
>
>             # lookup in the dictionary
>             entry_version = get_dict_key_version(dict, key)
>             if entry_version == self.entry_version:
>                 # another key was modified:
>                 # cache the new dictionary version
>                 self.dict_version = dict_version
>                 return True
>
>             # the key was modified
>             return False
>
> The main drawback of this option is the impact on the memory footprint.
> It increases the size of each dictionary entry, so the overhead depends
> on the number of buckets (dictionary entries, used or unused yet). For
> example, it increases the size of each dictionary entry by 8 bytes on
> 64-bit system.
>
> In Python, the memory footprint matters and the trend is to reduce it.
> Examples:
>
> * `PEP 393 -- Flexible String Representation
>   <https://www.python.org/dev/peps/pep-0393/>`_
> * `PEP 412 -- Key-Sharing Dictionary
>   <https://www.python.org/dev/peps/pep-0412/>`_
>
>
> Add a new dict subtype
> ----------------------
>
> Add a new ``verdict`` type, subtype of ``dict``. When guards are needed,
> use the ``verdict`` for namespaces (module namespace, type namespace,
> instance namespace, etc.) instead of ``dict``.
>
> Leave the ``dict`` type unchanged to not add any overhead (memory
> footprint) when guards are not needed.
>
> Technical issue: a lot of C code in the wild, including CPython core,
> expecting the exact ``dict`` type. Issues:
>
> * ``exec()`` requires a ``dict`` for globals and locals. A lot of code
>   use ``globals={}``. It is not possible to cast the ``dict`` to a
>   ``dict`` subtype because the caller expects the ``globals`` parameter
>   to be modified (``dict`` is mutable).
> * Functions call directly ``PyDict_xxx()`` functions, instead of calling
>   ``PyObject_xxx()`` if the object is a ``dict`` subtype
> * ``PyDict_CheckExact()`` check fails on ``dict`` subtype, whereas some
>   functions require the exact ``dict`` type.
> * ``Python/ceval.c`` does not completly supports dict subtypes for
>   namespaces
>
>
> The ``exec()`` issue is a blocker issue.
>
> Other issues:
>
> * The garbage collector has a special code to "untrack" ``dict``
>   instances. If a ``dict`` subtype is used for namespaces, the garbage
>   collector can be unable to break some reference cycles.
> * Some functions have a fast-path for ``dict`` which would not be taken
>   for ``dict`` subtypes, and so it would make Python a little bit
>   slower.
>
>
> Prior Art
> =========
>
> Method cache and type version tag
> ---------------------------------
>
> In 2007, Armin Rigo wrote a patch to to implement a cache of methods. It
> was merged into Python 2.6.  The patch adds a "type attribute cache
> version tag" (``tp_version_tag``) and a "valid version tag" flag to
> types (the ``PyTypeObject`` structure).
>
> The type version tag is not available at the Python level.
>
> The version tag has the C type ``unsigned int``. The cache is a global
> hash table of 4096 entries, shared by all types. The cache is global to
> "make it fast, have a deterministic and low memory footprint, and be
> easy to invalidate". Each cache entry has a version tag. A global
> version tag is used to create the next version tag, it also has the C
> type ``unsigned int``.
>
> By default, a type has its "valid version tag" flag cleared to indicate
> that the version tag is invalid. When the first method of the type is
> cached, the version tag and the "valid version tag" flag are set. When a
> type is modified, the "valid version tag" flag of the type and its
> subclasses is cleared. Later, when a cache entry of these types is used,
> the entry is removed because its version tag is outdated.
>
> On integer overflow, the whole cache is cleared and the global version
> tag is reset to ``0``.
>
> See `Method cache (issue #1685986)
> <https://bugs.python.org/issue1685986>`_ and `Armin's method cache
> optimization updated for Python 2.6 (issue #1700288)
> <https://bugs.python.org/issue1700288>`_.
>
>
> Globals / builtins cache
> ------------------------
>
> In 2010, Antoine Pitrou proposed a `Globals / builtins cache (issue
> #10401) <http://bugs.python.org/issue10401>`_ which adds a private
> ``ma_version`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``Py_ssize_t``.
>
> The patch adds a "global and builtin cache" to functions and frames, and
> changes ``LOAD_GLOBAL`` and ``STORE_GLOBAL`` instructions to use the
> cache.
>
> The change on the ``PyDictObject`` structure is very similar to this
> PEP.
>
>
> Cached globals+builtins lookup
> ------------------------------
>
> In 2006, Andrea Griffini proposed a patch implementing a `Cached
> globals+builtins lookup optimization
> <https://bugs.python.org/issue1616125>`_.  The patch adds a private
> ``timestamp`` field to the ``PyDictObject`` structure (``dict`` type),
> the field has the C type ``size_t``.
>
> Thread on python-dev: `About dictionary lookup caching
> <https://mail.python.org/pipermail/python-dev/2006-December/070348.html>`_.
>
>
> Guard against changing dict during iteration
> --------------------------------------------
>
> In 2013, Serhiy Storchaka proposed `Guard against changing dict during
> iteration (issue #19332) <https://bugs.python.org/issue19332>`_ which
> adds a ``ma_count`` field to the ``PyDictObject`` structure (``dict``
> type), the field has the C type ``size_t``.  This field is incremented
> when the dictionary is modified, and so is very similar to the proposed
> dictionary version.
>
> Sadly, the dictionary version proposed in this PEP doesn't help to
> detect dictionary mutation. The dictionary version changes when values
> are replaced, whereas modifying dictionary values while iterating on
> dictionary keys is legit in Python.
>
>
> PySizer
> -------
>
> `PySizer <http://pysizer.8325.org/>`_: a memory profiler for Python,
> Google Summer of Code 2005 project by Nick Smallbone.
>
> This project has a patch for CPython 2.4 which adds ``key_time`` and
> ``value_time`` fields to dictionary entries. It uses a global
> process-wide counter for dictionaries, incremented each time that a
> dictionary is modified. The times are used to decide when child objects
> first appeared in their parent objects.
>
>
> Discussion
> ==========
>
> Thread on the python-ideas mailing list: `RFC: PEP: Add dict.__version__
> <https://mail.python.org/pipermail/python-ideas/2016-January/037702.html>`_.
>
>
> Copyright
> =========
>
> This document has been placed in the public domain.

From barry at python.org  Mon Jan 18 18:04:10 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 18 Jan 2016 18:04:10 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
Message-ID: <20160118180410.0da0bc8f@limelight.wooz.org>

On Jan 18, 2016, at 11:43 PM, Victor Stinner wrote:

>Is someone opposed to this PEP 509?
>
>The main complain was the change on the public Python API, but the PEP
>doesn't change the Python API anymore.
>
>I'm not aware of any remaining issue on this PEP.

Have you done any performance analysis for a wide range of Python
applications?  Did you address my suggestion on python-ideas to make the new C
API optionally compiled in?

I still think this is maintenance and potential performance overhead we don't
want to commit to long term unless it enables significant optimization.  Since
you probably can't prove that without some experimentation, this API should be
provisional.

Cheers,
-Barry

From brett at python.org  Mon Jan 18 18:20:21 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 18 Jan 2016 23:20:21 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>

On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:

> While doing a review of http://bugs.python.org/review/26129/ I asked to
> have curly braces put around all `if` statement bodies. Serhiy pointed out
> that PEP 7 says curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> that.
>
> My argument is to require them to prevent bugs like the one Apple made
> with OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can lead
> to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>

Currently this thread stands at:

+1
  Brett
  Ethan
  Robert
  Georg
  Nick
  Maciej Szulik
+0
  Guido
-0
  Serhiy
  MAL
-1
  Victor (maybe; didn't specifically vote)
  Larry
  Stefan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/b0d440a9/attachment.html>

From brett at python.org  Mon Jan 18 18:30:06 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 18 Jan 2016 23:30:06 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569CA960.4030401@hastings.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
Message-ID: <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>

On Mon, 18 Jan 2016 at 00:59 Larry Hastings <larry at hastings.org> wrote:

>
>
> On 01/17/2016 11:10 AM, Brett Cannon wrote:
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>
>
> I'm -1.  I don't like being forced to add the curly braces when the code
> is perfectly clear without them.
>

I'm going to assume you mean it is clearer without them, else I don't hear
an argument against beyond "I don't wanna".

For me, I don't see how::

  int x = do_something();
  if (x != 10)
    return NULL;
  do_some_more();

is any clearer or more readable than::

  int x = do_something();
  if (x != 10) {
    return NULL;
  }
  do_some_more();



> If this was a frequent problem then I'd put up with it, but I can't recall
> ever making this particular mistake myself or seeing it in CPython source.
> It seems to me like a fix for a problem we don't have.
>

I personally have come close to screwing up like this, but I caught it
before I put up a patch.

I honestly also prefer the consistency of just always using braces rather
than the sometimes rule we have. It isn't like C is a pretty, safe language
to begin with, so I would rather be consistent and avoid potential errors.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/5911c49c/attachment-0001.html>

From tjreedy at udel.edu  Mon Jan 18 19:18:08 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 18 Jan 2016 19:18:08 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
Message-ID: <n7jvce$tms$1@ger.gmane.org>

On 1/18/2016 6:20 PM, Brett Cannon wrote:
>
>
> On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org
> <mailto:brett at python.org>> wrote:
>
>     While doing a review of http://bugs.python.org/review/26129/ I asked
>     to have curly braces put around all `if` statement bodies. Serhiy
>     pointed out that PEP 7 says curly braces are optional:
>     https://www.python.org/dev/peps/pep-0007/#id5. I would like to
>     change that.
>
>     My argument is to require them to prevent bugs like the one Apple
>     made with OpenSSL about two years ago:
>     https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping
>     the curly braces is purely an aesthetic thing while leaving them out
>     can lead to actual bugs.
>
>     Anyone object if I update PEP 7 to remove the optionality of curly
>     braces in PEP 7?
>
>
> Currently this thread stands at:
>
> +1
>    Brett
>    Ethan
>    Robert
>    Georg
>    Nick
>    Maciej Szulik
> +0
>    Guido
> -0
>    Serhiy
>    MAL
> -1
>    Victor (maybe; didn't specifically vote)
>    Larry
>    Stefan

Though I don't write C anymore, I occasionally read our C sources.  I 
dislike mixed bracketing in a multiple clause if/else statement,  and 
would strongly recommend against that.  On the other hand, to my 
Python-trained eye, brackets for one line clauses are just noise.  +-0.

If coverity's scan does not flag the sort of misleading bug bait 
formatting that at least partly prompted this thread

if (a):
    b;
    c;

then I think we should find or write something that does and run it over 
existing code as well as patches.

-- 
Terry Jan Reedy


From python at mrabarnett.plus.com  Mon Jan 18 20:04:03 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Tue, 19 Jan 2016 01:04:03 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <n7jvce$tms$1@ger.gmane.org>
Message-ID: <em0288f1d4-093e-4dc9-a26d-461c8d4d55ec@andromeda>

On 2016-01-19 00:18:08, "Terry Reedy" <tjreedy at udel.edu> wrote:

>On 1/18/2016 6:20 PM, Brett Cannon wrote:
>>
>>On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org
>><mailto:brett at python.org>> wrote:
>>
>>     While doing a review of http://bugs.python.org/review/26129/ I 
>>asked
>>     to have curly braces put around all `if` statement bodies. Serhiy
>>     pointed out that PEP 7 says curly braces are optional:
>>     https://www.python.org/dev/peps/pep-0007/#id5. I would like to
>>     change that.
>>
>>     My argument is to require them to prevent bugs like the one Apple
>>     made with OpenSSL about two years ago:
>>     https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping
>>     the curly braces is purely an aesthetic thing while leaving them 
>>out
>>     can lead to actual bugs.
>>
>>     Anyone object if I update PEP 7 to remove the optionality of curly
>>     braces in PEP 7?
>>
>>
>>Currently this thread stands at:
>>
>>+1
>>    Brett
>>    Ethan
>>    Robert
>>    Georg
>>    Nick
>>    Maciej Szulik
>>+0
>>    Guido
>>-0
>>    Serhiy
>>    MAL
>>-1
>>    Victor (maybe; didn't specifically vote)
>>    Larry
>>    Stefan
>
>Though I don't write C anymore, I occasionally read our C sources.  I 
>dislike mixed bracketing in a multiple clause if/else statement,  and 
>would strongly recommend against that.  On the other hand, to my 
>Python-trained eye, brackets for one line clauses are just noise.  +-0.
>
>If coverity's scan does not flag the sort of misleading bug bait 
>formatting that at least partly prompted this thread
>
>if (a):
>    b;
>    c;
>
>then I think we should find or write something that does and run it 
>over existing code as well as patches.
>
I agree.

After all, how hard could it be? :-)


From greg.ewing at canterbury.ac.nz  Mon Jan 18 23:27:21 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 19 Jan 2016 17:27:21 +1300
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
Message-ID: <569DBB29.3080701@canterbury.ac.nz>

Brett Cannon wrote:
> For me, I don't see how::
> 
>   if (x != 10)
>     return NULL;
>   do_some_more();
> 
> is any clearer or more readable than::
> 
>   if (x != 10) {
>     return NULL;
>   }
>   do_some_more();

Maybe not for that piece of code on its own, but the version
with braces takes up one more line. Put a few of those together,
and you can't fit as much code on the screen. If it makes the
difference between being able to see e.g. the whole of a loop
at once vs. having to scroll up and down, it could make the
code as a whole harder to read.

-- 
Greg

From tritium-list at sdamon.com  Mon Jan 18 23:40:30 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Mon, 18 Jan 2016 23:40:30 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DBB29.3080701@canterbury.ac.nz>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz>
Message-ID: <569DBE3E.5020509@sdamon.com>



On 1/18/2016 23:27, Greg Ewing wrote:
> Brett Cannon wrote:
>> For me, I don't see how::
>>
>>   if (x != 10)
>>     return NULL;
>>   do_some_more();
>>
>> is any clearer or more readable than::
>>
>>   if (x != 10) {
>>     return NULL;
>>   }
>>   do_some_more();
>
> Maybe not for that piece of code on its own, but the version
> with braces takes up one more line. Put a few of those together,
> and you can't fit as much code on the screen. If it makes the
> difference between being able to see e.g. the whole of a loop
> at once vs. having to scroll up and down, it could make the
> code as a whole harder to read.
>
When someone trying to make this argument in #python for Python code... 
the response is newlines are free.  Almost this entire thread has me 
confused - the arguments against are kind of hypocritical; You are 
developing a language with a built in design ethic, and ignoring those 
ethics while building the implementation itself.

Newlines are free, use them

Explicit > Implicit - Explicitly scope everything.

I am not a core developer, but I just kind of feel its hypocritical to 
oppose always using brackets for the development of *python*

From ncoghlan at gmail.com  Tue Jan 19 00:09:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 19 Jan 2016 15:09:07 +1000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DBE3E.5020509@sdamon.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
Message-ID: <CADiSq7cNbzeWyMy-+n0yJPPYcLtJOUU=9A6VNF=h83Xzg8XMcQ@mail.gmail.com>

On 19 January 2016 at 14:40, Alexander Walters <tritium-list at sdamon.com> wrote:
>
>
> On 1/18/2016 23:27, Greg Ewing wrote:
>>
>> Brett Cannon wrote:
>>>
>>> For me, I don't see how::
>>>
>>>   if (x != 10)
>>>     return NULL;
>>>   do_some_more();
>>>
>>> is any clearer or more readable than::
>>>
>>>   if (x != 10) {
>>>     return NULL;
>>>   }
>>>   do_some_more();
>>
>>
>> Maybe not for that piece of code on its own, but the version
>> with braces takes up one more line. Put a few of those together,
>> and you can't fit as much code on the screen. If it makes the
>> difference between being able to see e.g. the whole of a loop
>> at once vs. having to scroll up and down, it could make the
>> code as a whole harder to read.
>>
> When someone trying to make this argument in #python for Python code... the
> response is newlines are free.  Almost this entire thread has me confused -
> the arguments against are kind of hypocritical; You are developing a
> language with a built in design ethic, and ignoring those ethics while
> building the implementation itself.

There are two conflicting code aesthetics at work here, and the
relevant one for the folks that prefer to avoid braces in C where they
can is:

>>> from __future__ import braces
  File "<stdin>", line 1
SyntaxError: not a chance

So if we bring *Python* into the comparison, we can see it splits the
difference between the two C variants by omitting the closing brace
and replacing the opening brace with ":":

  x = do_something()
  if (x != 10):
    return None
  do_some_more()

The additional "cost" of mandatory braces in C is thus more lines
containing only a single "}", while the benefit is simply not having
to think about the braceless variant as a possible alternative
spelling.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From greg.ewing at canterbury.ac.nz  Tue Jan 19 00:09:11 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 19 Jan 2016 18:09:11 +1300
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DBE3E.5020509@sdamon.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
Message-ID: <569DC4F7.7050406@canterbury.ac.nz>

Alexander Walters wrote:
> When someone trying to make this argument in #python for Python code... 
> the response is newlines are free.

Well, I disagree. I very rarely put blank lines in a function
in any language, because it makes it hard to scan the code
visually and pick out the beginnings of functions. So while
they may help readability locally, they hurt it globally.

If I find myself needing to put blank lines in a function in
order to make it readable, I take it as a sign that the
function ought to be split up into smaller functions.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Jan 19 00:16:37 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 19 Jan 2016 18:16:37 +1300
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DBE3E.5020509@sdamon.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
Message-ID: <569DC6B5.303@canterbury.ac.nz>

Alexander Walters wrote:
> I am not a core developer, but I just kind of feel its hypocritical to 
> oppose always using brackets for the development of *python*

If we were being *really* pythonic, we would write all
the C code without any braces at all, and feed it through
a filter that adds them based on indentation. End of
argument. :-)

-- 
Greg

From tritium-list at sdamon.com  Tue Jan 19 01:05:54 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 19 Jan 2016 01:05:54 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DC4F7.7050406@canterbury.ac.nz>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
 <569DC4F7.7050406@canterbury.ac.nz>
Message-ID: <569DD242.1080209@sdamon.com>



On 1/19/2016 00:09, Greg Ewing wrote:
> Alexander Walters wrote:
>> When someone trying to make this argument in #python for Python 
>> code... the response is newlines are free.
>
> Well, I disagree. I very rarely put blank lines in a function
> in any language, because it makes it hard to scan the code
> visually and pick out the beginnings of functions. So while
> they may help readability locally, they hurt it globally.
I find the opposite to be true when I work in C

> If I find myself needing to put blank lines in a function in
> order to make it readable, I take it as a sign that the
> function ought to be split up into smaller functions.
>
.... Well, maybe I should change my position based on this.  Any excuse 
to break code out into more functions... is usually the right idea.

From v+python at g.nevcal.com  Tue Jan 19 01:04:55 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Mon, 18 Jan 2016 22:04:55 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DC6B5.303@canterbury.ac.nz>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
 <569DC6B5.303@canterbury.ac.nz>
Message-ID: <569DD207.3070603@g.nevcal.com>

On 1/18/2016 9:16 PM, Greg Ewing wrote:
> Alexander Walters wrote:
>> I am not a core developer, but I just kind of feel its hypocritical 
>> to oppose always using brackets for the development of *python*
>
> If we were being *really* pythonic, we would write all
> the C code without any braces at all, and feed it through
> a filter that adds them based on indentation. End of
> argument. :-)
>

No argument, an extension:  If we were being *really* pythonic, we would 
re-implement CPython in Python, with a Python-to-C option (Cython?), and 
then we wouldn't need to worry about using braces for block delimiters.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/93eb3bc1/attachment.html>

From tritium-list at sdamon.com  Tue Jan 19 01:09:05 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 19 Jan 2016 01:09:05 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DD207.3070603@g.nevcal.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
 <569DC6B5.303@canterbury.ac.nz> <569DD207.3070603@g.nevcal.com>
Message-ID: <569DD301.8010509@sdamon.com>

On 1/19/2016 01:04, Glenn Linderman wrote:
> On 1/18/2016 9:16 PM, Greg Ewing wrote:
>> Alexander Walters wrote:
>>> I am not a core developer, but I just kind of feel its hypocritical 
>>> to oppose always using brackets for the development of *python*
>>
>> If we were being *really* pythonic, we would write all
>> the C code without any braces at all, and feed it through
>> a filter that adds them based on indentation. End of
>> argument. :-)
>>
>
> No argument, an extension:  If we were being *really* pythonic, we 
> would re-implement CPython in Python, with a Python-to-C option 
> (Cython?), and then we wouldn't need to worry about using braces for 
> block delimiters.
So what I can take away from this is pypy is the pythonic python.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/22e037f3/attachment.html>

From larry at hastings.org  Tue Jan 19 02:21:09 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 18 Jan 2016 23:21:09 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DBE3E.5020509@sdamon.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
Message-ID: <569DE3E5.4070906@hastings.org>


On 01/18/2016 08:40 PM, Alexander Walters wrote:
> I am not a core developer, but I just kind of feel its hypocritical to 
> oppose always using brackets for the development of *python*

CPython isn't written in Python, it's written in C.  So we use C idioms, 
which naturally are different from Python idioms.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/648ccd8d/attachment-0001.html>

From tritium-list at sdamon.com  Tue Jan 19 02:25:07 2016
From: tritium-list at sdamon.com (Alexander Walters)
Date: Tue, 19 Jan 2016 02:25:07 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DE3E5.4070906@hastings.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
 <569DE3E5.4070906@hastings.org>
Message-ID: <569DE4D3.7090607@sdamon.com>



On 1/19/2016 02:21, Larry Hastings wrote:
>
> On 01/18/2016 08:40 PM, Alexander Walters wrote:
>> I am not a core developer, but I just kind of feel its hypocritical 
>> to oppose always using brackets for the development of *python*
>
> CPython isn't written in Python, it's written in C.  So we use C 
> idioms, which naturally are different from Python idioms.
>

Yes, and always using brackets is an accepted C idiom (one of two, the 
other being only use brackets when you must).  Always using them is  
more in line with the spirit of python.

> //arry/
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/525cd5a3/attachment.html>

From g.brandl at gmx.net  Tue Jan 19 02:32:06 2016
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 19 Jan 2016 08:32:06 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <n7jvce$tms$1@ger.gmane.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <n7jvce$tms$1@ger.gmane.org>
Message-ID: <n7kops$90m$1@ger.gmane.org>

On 01/19/2016 01:18 AM, Terry Reedy wrote:

> Though I don't write C anymore, I occasionally read our C sources.  I 
> dislike mixed bracketing in a multiple clause if/else statement,  and 
> would strongly recommend against that.  On the other hand, to my 
> Python-trained eye, brackets for one line clauses are just noise.  +-0.
> 
> If coverity's scan does not flag the sort of misleading bug bait 
> formatting that at least partly prompted this thread
> 
> if (a):
>     b;
>     c;
> 
> then I think we should find or write something that does and run it over 
> existing code as well as patches.

I don't know if static checkers care about whitespace and indentation in C;
it might be a very obvious thing to do for Python programmers, but maybe not
for C static checker developers :)

And probably with good reason, since whitespace isn't a consideration except
for nicely readable code, which many people (not talking about CPython here)
apparently don't care about, you'd have tons of spurious checker messages.

In many cases (especially error handling ones like "goto fail"), I expect the
checker to flag the error anyway, but for semantic reasons, not because of
whitespace.

Georg


From pludemann at google.com  Tue Jan 19 02:34:00 2016
From: pludemann at google.com (Peter Ludemann)
Date: Mon, 18 Jan 2016 23:34:00 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DE4D3.7090607@sdamon.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <569CA960.4030401@hastings.org>
 <CAP1=2W4ZdJ0=DHdmog9hqoWr54tMAhATFspjK-RSC4eucpdLHA@mail.gmail.com>
 <569DBB29.3080701@canterbury.ac.nz> <569DBE3E.5020509@sdamon.com>
 <569DE3E5.4070906@hastings.org> <569DE4D3.7090607@sdamon.com>
Message-ID: <CACsRUKJStu1pH5HGBjYH5gvYQJr6CKERF9sEudbhA69uPzz1Eg@mail.gmail.com>

On 18 January 2016 at 23:25, Alexander Walters <tritium-list at sdamon.com>
wrote:

>
>
> On 1/19/2016 02:21, Larry Hastings wrote:
>
>
> On 01/18/2016 08:40 PM, Alexander Walters wrote:
>
> I am not a core developer, but I just kind of feel its hypocritical to
> oppose always using brackets for the development of *python*
>
>
> CPython isn't written in Python, it's written in C.  So we use C idioms,
> which naturally are different from Python idioms.
>
>
> Yes, and always using brackets is an accepted C idiom (one of two, the
> other being only use brackets when you must).  Always using them is  more
> in line with the spirit of python.
>

?And in the spirit of safety, which I think is Brett's concern.
Yes, there are tools that could catch the Apple bug
<https://www.imperialviolet.org/2014/02/22/applebug.html>, by catching
incorrect indentation. But if the rule is to *always* use braces, that just
feels safer. Even if it offends some people's ability to read everything at
one glance (my suggestion: use a smaller font or a larger screen; or
refactor the code so it isn't as big a chunk).

There's a fairly exhaustive list of styles here:
https://en.wikipedia.org/wiki/Indent_style
I'm sure that some of these have not been advocated in this thread.?



>
> */arry*
>
>
> _______________________________________________
> Python-Dev mailing listPython-Dev at python.orghttps://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160118/a6080a32/attachment.html>

From victor.stinner at gmail.com  Tue Jan 19 03:03:07 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 19 Jan 2016 09:03:07 +0100
Subject: [Python-Dev] PEP 510: Specialize functions with guards
In-Reply-To: <CAMpsgwbQoL=CQajsWRbY1x5sd-uEw-e+Gg+f-+rPLLpb86YqFA@mail.gmail.com>
References: <CAMpsgwZmb2KBi+5OtBN5Cg4-LDc_hbM6nrDVusXaKWn3dOeefQ@mail.gmail.com>
 <CAMpsgwbQoL=CQajsWRbY1x5sd-uEw-e+Gg+f-+rPLLpb86YqFA@mail.gmail.com>
Message-ID: <CAMpsgwY-tz+iPD-Abt_bXVSHTr+e3wSN+0LQa4xx22mnvZLrmw@mail.gmail.com>

Oh, I think that the PEP 510 lacks two functions to:

* remove a specific specialized code
* remove all specialized code

Victor

From mal at egenix.com  Tue Jan 19 03:48:35 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 19 Jan 2016 09:48:35 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
Message-ID: <569DF863.3000109@egenix.com>

On 19.01.2016 00:20, Brett Cannon wrote:
> On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:
> 
>> While doing a review of http://bugs.python.org/review/26129/ I asked to
>> have curly braces put around all `if` statement bodies. Serhiy pointed out
>> that PEP 7 says curly braces are optional:
>> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
>> that.
>>
>> My argument is to require them to prevent bugs like the one Apple made
>> with OpenSSL about two years ago:
>> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> curly braces is purely an aesthetic thing while leaving them out can lead
>> to actual bugs.
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?
>>
> 
> Currently this thread stands at:

Make that:

> +1
>   Brett
>   Ethan
>   Robert
>   Georg
>   Nick
>   Maciej Szulik
> +0
>   Guido
> -0
>   Serhiy
> -1
    MAL
>   Victor (maybe; didn't specifically vote)
>   Larry
>   Stefan

There are plenty other cases where typos can ruin the flow
of your code in C; the discussed case is not a very common one.

I find the whole discussion a bit strange: In Python we're
advocating not having to use curly braces, because whitespace
already provides the needed semantics for us, yet you are
now advocating that without adding extra curly braces
we'd be in danger of writing wrong code.

The Apple bug can happen in Python just as well:

if a:
    run_if_true()
else:
    run_if_false()

can turn into (say by hitting a wrong key in the editor):

if a:
    run_if_true()
    run_if_false()

(run_if_false is now run when a is true, and nothing is
done in case a is false)

So what's the correct way to address this ?

It's having a test for both branches (a is true, a is false),
not starting to write e.g.

if a:
    run_if_true()
if not a:
    run_if false()

to feel more "secure".

Also note that the extra braces won't necessarily help you.
If you remove "else" from:

if (a) {
    run_if_true();
}
else {
    run_if_false();
}

you get exactly the same Apply bug as before, only with more
curly braces.

This all sounds a bit like security theater to me ;-)

I'd say: people who feel better using the extra braces can use
them, while others who don't need them can go ahead as always
... and both groups should write more tests :-)

BTW: There are other things in PEP 7 which should probably be updated
instead, e.g. it still says we should use C89, but we're having more
and more C99 code (mostly new library functions) in CPython.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 19 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From soltysh at gmail.com  Tue Jan 19 04:00:03 2016
From: soltysh at gmail.com (Maciej Szulik)
Date: Tue, 19 Jan 2016 10:00:03 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DF863.3000109@egenix.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <569DF863.3000109@egenix.com>
Message-ID: <CALC2N4ag1zsox9QsonEMV_dzTpihU602=ino30jLi-XtjAc-Dg@mail.gmail.com>

To add my additional 0.02$ to the discussion. Quite a while ago my
coworkers and I agreed
to use strict rules regarding code formatting. That idea turned into a
formatter we've had
committed in the project itself. This forced everyone on the team to use it
and thus producing
exactly "the same looking" code. I found it very nice, although we've
debated over how
those rules should be applied for good couple months, that was at one of
the previous companies
I've worked for. Currently by day I'm working with Go, which go-fmt is by
default required for all our
projects and I found it extremely handy see [1], [2], [3].

Maciej


[1] https://github.com/openshift/origin/
[2] https://github.com/openshift/source-to-image/
[3] https://github.com/kubernetes/kubernetes

On Tue, Jan 19, 2016 at 9:48 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 19.01.2016 00:20, Brett Cannon wrote:
> > On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:
> >
> >> While doing a review of http://bugs.python.org/review/26129/ I asked to
> >> have curly braces put around all `if` statement bodies. Serhiy pointed
> out
> >> that PEP 7 says curly braces are optional:
> >> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> >> that.
> >>
> >> My argument is to require them to prevent bugs like the one Apple made
> >> with OpenSSL about two years ago:
> >> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> >> curly braces is purely an aesthetic thing while leaving them out can
> lead
> >> to actual bugs.
> >>
> >> Anyone object if I update PEP 7 to remove the optionality of curly
> braces
> >> in PEP 7?
> >>
> >
> > Currently this thread stands at:
>
> Make that:
>
> > +1
> >   Brett
> >   Ethan
> >   Robert
> >   Georg
> >   Nick
> >   Maciej Szulik
> > +0
> >   Guido
> > -0
> >   Serhiy
> > -1
>     MAL
> >   Victor (maybe; didn't specifically vote)
> >   Larry
> >   Stefan
>
> There are plenty other cases where typos can ruin the flow
> of your code in C; the discussed case is not a very common one.
>
> I find the whole discussion a bit strange: In Python we're
> advocating not having to use curly braces, because whitespace
> already provides the needed semantics for us, yet you are
> now advocating that without adding extra curly braces
> we'd be in danger of writing wrong code.
>
> The Apple bug can happen in Python just as well:
>
> if a:
>     run_if_true()
> else:
>     run_if_false()
>
> can turn into (say by hitting a wrong key in the editor):
>
> if a:
>     run_if_true()
>     run_if_false()
>
> (run_if_false is now run when a is true, and nothing is
> done in case a is false)
>
> So what's the correct way to address this ?
>
> It's having a test for both branches (a is true, a is false),
> not starting to write e.g.
>
> if a:
>     run_if_true()
> if not a:
>     run_if false()
>
> to feel more "secure".
>
> Also note that the extra braces won't necessarily help you.
> If you remove "else" from:
>
> if (a) {
>     run_if_true();
> }
> else {
>     run_if_false();
> }
>
> you get exactly the same Apply bug as before, only with more
> curly braces.
>
> This all sounds a bit like security theater to me ;-)
>
> I'd say: people who feel better using the extra braces can use
> them, while others who don't need them can go ahead as always
> ... and both groups should write more tests :-)
>
> BTW: There are other things in PEP 7 which should probably be updated
> instead, e.g. it still says we should use C89, but we're having more
> and more C99 code (mostly new library functions) in CPython.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, Jan 19 2016)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...           http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
>                       http://www.malemburg.com/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/soltysh%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/e24f85b9/attachment.html>

From victor.stinner at gmail.com  Tue Jan 19 04:42:11 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 19 Jan 2016 10:42:11 +0100
Subject: [Python-Dev] Reference cycle on the module dict (globals())
Message-ID: <CAMpsgwbXq17e5zNaSXyqUWq7L-Q=Q4yRA=wTAhxVt7VaMkr41g@mail.gmail.com>

Hi,

While working on my FAT Python optimizer project, I found an annoying
bug in my code. When at least one guard is created with a reference to
the global namespace (globals(), the module dictionary), objects of
the module are no more removed at exit.

Example:
---
import sys

class MessageAtExit:
    def __del__(self):
        print('__del__ called')

# display a message at exit, when message_at_exit is removed
message_at_exit = MessageAtExit()

# create a reference cycle:
# module -> module dict -> Guard -> module dict
guard = sys.Guard(globals())
---
(the code is adapted from a test of test_gc)

Apply attached patch to Python 3.6 to get the sys.Guard object. It's a
minimalist object to keep a strong reference to an object.

I expected the garbage collector to break such (simple?) reference cycle.

The Guard object implements a traverse module, but it is never called.

Did I miss something obvious, or is it a known issue of the garbage
collector on modules?

Victor
-------------- next part --------------
A non-text attachment was scrubbed...
Name: guard.patch
Type: text/x-patch
Size: 3804 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/e8ac9ca6/attachment-0001.bin>

From skrah.temporarily at gmail.com  Tue Jan 19 05:29:07 2016
From: skrah.temporarily at gmail.com (Stefan Krah)
Date: Tue, 19 Jan 2016 10:29:07 +0000 (UTC)
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <569DF863.3000109@egenix.com>
Message-ID: <loom.20160119T111059-148@post.gmane.org>

M.-A. Lemburg <mal <at> egenix.com> writes: 
> > Currently this thread stands at:
> 
> Make that:
> 
> > +1
> >   Brett
> >   Ethan
> >   Robert
> >   Georg
> >   Nick
> >   Maciej Szulik
> > +0
> >   Guido
> > -0
> >   Serhiy
> > -1
>     MAL
> >   Victor (maybe; didn't specifically vote)
> >   Larry
> >   Stefan

I want to clarify my position a bit: Personally, in _decimal/* I've
always used braces and I prefer that.

But from reading the Python sources in general, I got the impression
that the default style at least for one-liner if-statements is to
omit braces. So, in memoryview.c, I adapted to that style.


I think enforcing braces won't do anything for security. DJB (who
had a single exploit found in qmail in 20 years) even uses nested
for-loops without braces.

IMO secure code can only be achieved by auditing it quietly in
a terminal, not being distracted by peripheral things like
version control and web interfaces (green merge buttons!) and
trying to do formal proofs (if time allows for it).


So I would not want to enforce a style if it makes some people
unhappy.


Stefan Krah

























From encukou at gmail.com  Tue Jan 19 05:39:28 2016
From: encukou at gmail.com (Petr Viktorin)
Date: Tue, 19 Jan 2016 11:39:28 +0100
Subject: [Python-Dev] Reference cycle on the module dict (globals())
In-Reply-To: <CAMpsgwbXq17e5zNaSXyqUWq7L-Q=Q4yRA=wTAhxVt7VaMkr41g@mail.gmail.com>
References: <CAMpsgwbXq17e5zNaSXyqUWq7L-Q=Q4yRA=wTAhxVt7VaMkr41g@mail.gmail.com>
Message-ID: <569E1260.6070105@gmail.com>

On 01/19/2016 10:42 AM, Victor Stinner wrote:
> Hi,
> 
> While working on my FAT Python optimizer project, I found an annoying
> bug in my code. When at least one guard is created with a reference to
> the global namespace (globals(), the module dictionary), objects of
> the module are no more removed at exit.
> 
> Example:
> ---
> import sys
> 
> class MessageAtExit:
>     def __del__(self):
>         print('__del__ called')
> 
> # display a message at exit, when message_at_exit is removed
> message_at_exit = MessageAtExit()
> 
> # create a reference cycle:
> # module -> module dict -> Guard -> module dict
> guard = sys.Guard(globals())
> ---
> (the code is adapted from a test of test_gc)
> 
> Apply attached patch to Python 3.6 to get the sys.Guard object. It's a
> minimalist object to keep a strong reference to an object.
> 
> I expected the garbage collector to break such (simple?) reference cycle.
> 
> The Guard object implements a traverse module, but it is never called.
> 
> Did I miss something obvious, or is it a known issue of the garbage
> collector on modules?

The default type flags are for objects that don't store references.
Since you're creating a mutable container, you need to set
Py_TPFLAGS_HAVE_GC. See https://docs.python.org/3/c-api/gcsupport.html
for all the details.

From victor.stinner at gmail.com  Tue Jan 19 07:32:39 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 19 Jan 2016 13:32:39 +0100
Subject: [Python-Dev] _PyThreadState_Current
In-Reply-To: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
References: <CAK5idxRqrrbDhSqLyhmYqF-TLhGcvrVCziBCcj=Uy+CYvUvKMw@mail.gmail.com>
Message-ID: <CAMpsgwZFxzxnbUKO0TQCee8kneg6CY=iqeP9b8Qmh=FZz0+2QQ@mail.gmail.com>

Since it's a regression introduced in Python 3.5.1, I propose to
introduce a new private function _PyThreadState_FastGet() to
reintroduce the feature:
https://bugs.python.org/issue26154

Using afunction instead of using directly the variable hides how
atomic variables are implemented and so avoid compiler issues.

Victor

2016-01-18 21:18 GMT+01:00 Maciej Fijalkowski <fijall at gmail.com>:
> Hi
>
> change in between 3.5.0 and 3.5.1 (hiding _PyThreadState_Current and
> pyatomic.h) broke vmprof. The problem is that as a profile, vmprof can
> really encounter _PyThreadState_Current being null, while crashing an
> interpreter is a bit not ideal in this case.
>
> Any chance, a) _PyThreadState_Current can be restored in visibility?
> b) can I get a better API to get it in case it can be NULL, but also
> in 3.5 (since it works in 3.5.0 and breaks in 3.5.1)
>
> Cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From victor.stinner at gmail.com  Tue Jan 19 07:54:17 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 19 Jan 2016 13:54:17 +0100
Subject: [Python-Dev] Reference cycle on the module dict (globals())
In-Reply-To: <569E1260.6070105@gmail.com>
References: <CAMpsgwbXq17e5zNaSXyqUWq7L-Q=Q4yRA=wTAhxVt7VaMkr41g@mail.gmail.com>
 <569E1260.6070105@gmail.com>
Message-ID: <CAMpsgwagADCROfZs2__tO0UM6FbRY2Spw86QM6My_jy4gorBSw@mail.gmail.com>

Hi,

2016-01-19 11:39 GMT+01:00 Petr Viktorin <encukou at gmail.com>:
>> Did I miss something obvious, or is it a known issue of the garbage
>> collector on modules?
>
> The default type flags are for objects that don't store references.
> Since you're creating a mutable container, you need to set
> Py_TPFLAGS_HAVE_GC. See https://docs.python.org/3/c-api/gcsupport.html
> for all the details.

Ok, so I missed this important flag :-) Thanks!

I had to fight against the C API to fix all my bugs, but now it works
well: a guard keeps a strong reference to the global namespace, but
objects are still destroyed when the module is unloaded!

FYI I updated my PEP 510 patch to track guards with the garbage
collector, and my fat project to fix bugs related to GC:

- https://bugs.python.org/issue26098
- https://github.com/haypo/fat

Victor

From dmalcolm at redhat.com  Tue Jan 19 10:36:56 2016
From: dmalcolm at redhat.com (David Malcolm)
Date: Tue, 19 Jan 2016 10:36:56 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <n7jvce$tms$1@ger.gmane.org>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <n7jvce$tms$1@ger.gmane.org>
Message-ID: <1453217816.12356.22.camel@surprise>

On Mon, 2016-01-18 at 19:18 -0500, Terry Reedy wrote:
> On 1/18/2016 6:20 PM, Brett Cannon wrote:
> >
> >
> > On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org
> > <mailto:brett at python.org>> wrote:
> >
> >     While doing a review of http://bugs.python.org/review/26129/ I asked
> >     to have curly braces put around all `if` statement bodies. Serhiy
> >     pointed out that PEP 7 says curly braces are optional:
> >     https://www.python.org/dev/peps/pep-0007/#id5. I would like to
> >     change that.
> >
> >     My argument is to require them to prevent bugs like the one Apple
> >     made with OpenSSL about two years ago:
> >     https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping
> >     the curly braces is purely an aesthetic thing while leaving them out
> >     can lead to actual bugs.
> >
> >     Anyone object if I update PEP 7 to remove the optionality of curly
> >     braces in PEP 7?
> >
> >
> > Currently this thread stands at:
> >
> > +1
> >    Brett
> >    Ethan
> >    Robert
> >    Georg
> >    Nick
> >    Maciej Szulik
> > +0
> >    Guido
> > -0
> >    Serhiy
> >    MAL
> > -1
> >    Victor (maybe; didn't specifically vote)
> >    Larry
> >    Stefan
> 
> Though I don't write C anymore, I occasionally read our C sources.  I 
> dislike mixed bracketing in a multiple clause if/else statement,  and 
> would strongly recommend against that.  On the other hand, to my 
> Python-trained eye, brackets for one line clauses are just noise.  +-0.
> 
> If coverity's scan does not flag the sort of misleading bug bait 
> formatting that at least partly prompted this thread
> 
> if (a):
>     b;
>     c;
> 
> then I think we should find or write something that does and run it over 
> existing code as well as patches.

FWIW, for the forthcoming gcc 6, I've implemented a new
-Wmisleading-indentation warning that catches this.  It's currently
enabled by -Wall:

sslKeyExchange.c: In function 'SSLVerifySignedServerKeyExchange':
sslKeyExchange.c:631:8: warning: statement is indented as if it were guarded by... [-Wmisleading-indentation]
        goto fail;
        ^~~~
sslKeyExchange.c:629:4: note: ...this 'if' clause, but it is not
    if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
    ^~


(not that I've had time for core Python development lately, but FWIW in
gcc-python-plugin I mandate braces for single-statement clauses).

Dave


From yselivanov.ml at gmail.com  Tue Jan 19 11:55:43 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 19 Jan 2016 11:55:43 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
Message-ID: <569E6A8F.60903@gmail.com>



On 2016-01-18 6:20 PM, Brett Cannon wrote:
>
>
> On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org 
> <mailto:brett at python.org>> wrote:
>
>     While doing a review of http://bugs.python.org/review/26129/ I
>     asked to have curly braces put around all `if` statement bodies.
>     Serhiy pointed out that PEP 7 says curly braces are optional:
>     https://www.python.org/dev/peps/pep-0007/#id5. I would like to
>     change that.
>
>     My argument is to require them to prevent bugs like the one Apple
>     made with OpenSSL about two years ago:
>     https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping
>     the curly braces is purely an aesthetic thing while leaving them
>     out can lead to actual bugs.
>
>     Anyone object if I update PEP 7 to remove the optionality of curly
>     braces in PEP 7?
>
>
> Currently this thread stands at:
>
> +1
>   Brett
>   Ethan
>   Robert
>   Georg
>   Nick
>   Maciej Szulik


I guess you forgot to count Barry and me as +1s.

Yury


From jimjjewett at gmail.com  Tue Jan 19 11:56:41 2016
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Tue, 19 Jan 2016 08:56:41 -0800 (PST)
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <C6FBC1D9-6E91-4BC7-AB7B-708462F03492@yahoo.com>
Message-ID: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>



> On Jan 17, 2016, at 11:10, Brett Cannon <brett at python.org> wrote:

>> While doing a review of http://bugs.python.org/review/26129/ 
>> ... update PEP 7 to remove the optionality of curly braces

On Mon Jan 18 03:39:42 EST 2016, Andrew Barnert pointed out:

> There are two ways you could do that.

[The one most people are talking about, which often makes an if-clause
visually too heavy ... though Alexander Walters pointed out that
"Any excuse to break code out into more functions... is usually the
right idea."]

    if (!obj) {
        return -1;
    }


> Alternatively, it could say something like "braces must not be omitted;
> when other C styles would use a braceless one-liner, a one-liner with
> braces should be used instead; otherwise, they should be formatted as follows"

That "otherwise" gets a bit awkward, but I like the idea.  Perhaps
"braces must not be omitted, and should normally be formatted as
follows. ... Where other C styles would permit a braceless one-liner,
the expression and braces may be moved to a single line, as follows: "

    if (x > 5) { y++ }

I think that is clearly better, but it may be *too* lightweight for
flow control.

    if (!obj)
        { return -1; }

does work for me, and I think the \n{} may actually be useful for
warning that flow control takes a jump.


One reason I posted was to point to a specific example already in PEP 7
itself:

    if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
        type->tp_dictoffset == b_size &&
        (size_t)t_size == b_size + sizeof(PyObject *))
        return 0; /* "Forgive" adding a __dict__ only */

For me, that return is already visually lost, simply because it shares
an indentation with the much larger test expression.  Would that be
better as either:

    /* "Forgive" adding a __dict__ only */
    if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
        type->tp_dictoffset == b_size &&
        (size_t)t_size == b_size + sizeof(PyObject *)) { return 0; }

or:

    /* "Forgive" adding a __dict__ only */
    if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 &&
        type->tp_dictoffset == b_size &&
        (size_t)t_size == b_size + sizeof(PyObject *)) {
            return 0;
        }
        
-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ

From brett at python.org  Tue Jan 19 12:07:36 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 19 Jan 2016 17:07:36 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569DF863.3000109@egenix.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <569DF863.3000109@egenix.com>
Message-ID: <CAP1=2W7XxNy-CbFn9LgRCN8Pjc-3g4TmXX=-wgZvcObdZM8mXg@mail.gmail.com>

On Tue, 19 Jan 2016 at 00:48 M.-A. Lemburg <mal at egenix.com> wrote:

> On 19.01.2016 00:20, Brett Cannon wrote:
> > On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:
> >
> >> While doing a review of http://bugs.python.org/review/26129/ I asked to
> >> have curly braces put around all `if` statement bodies. Serhiy pointed
> out
> >> that PEP 7 says curly braces are optional:
> >> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> >> that.
> >>
> >> My argument is to require them to prevent bugs like the one Apple made
> >> with OpenSSL about two years ago:
> >> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> >> curly braces is purely an aesthetic thing while leaving them out can
> lead
> >> to actual bugs.
> >>
> >> Anyone object if I update PEP 7 to remove the optionality of curly
> braces
> >> in PEP 7?
> >>
> >
> > Currently this thread stands at:
>
> Make that:
>
> > +1
> >   Brett
> >   Ethan
> >   Robert
> >   Georg
> >   Nick
> >   Maciej Szulik
> > +0
> >   Guido
> > -0
> >   Serhiy
> > -1
>     MAL
> >   Victor (maybe; didn't specifically vote)
> >   Larry
> >   Stefan
>
> There are plenty other cases where typos can ruin the flow
> of your code in C; the discussed case is not a very common one.
>
> I find the whole discussion a bit strange: In Python we're
> advocating not having to use curly braces, because whitespace
> already provides the needed semantics for us, yet you are
> now advocating that without adding extra curly braces
> we'd be in danger of writing wrong code.
>

Yes, because in one language whitespace represents semantics while the
other is just formatting; I don't have to question the meaning of when
something is indented in Python, but in C I have to question whether the
indentation is an accident or the missing braces is the accident.


>
> The Apple bug can happen in Python just as well:
>
> if a:
>     run_if_true()
> else:
>     run_if_false()
>
> can turn into (say by hitting a wrong key in the editor):
>
> if a:
>     run_if_true()
>     run_if_false()
>
> (run_if_false is now run when a is true, and nothing is
> done in case a is false)
>
> So what's the correct way to address this ?
>
> It's having a test for both branches (a is true, a is false),
> not starting to write e.g.
>
> if a:
>     run_if_true()
> if not a:
>     run_if false()
>
> to feel more "secure".
>

OK, but what if the block was instead:

  if (a)
    run_if_true();
    Py_INCREF(a);

? Unit tests are not going to easily turn up a refcount leak, and the
number of times I have needed to email python-dev when Antoine's daily
refcount email has found a leak for several days shows people do not watch
closely for this. It's one thing when the expressions are obviously
mutually exclusive, but that's an ideal case. This isn't always about
losing an `else` statement as it can come from inserting a new statement
and not noticing that the braces are missing.


>
> Also note that the extra braces won't necessarily help you.
> If you remove "else" from:
>
> if (a) {
>     run_if_true();
> }
> else {
>     run_if_false();
> }
>
> you get exactly the same Apply bug as before, only with more
> curly braces.
>
> This all sounds a bit like security theater to me ;-)
>

That's fine. I also want format consistency by always using braces.


>
> I'd say: people who feel better using the extra braces can use
> them, while others who don't need them can go ahead as always
> ... and both groups should write more tests :-)
>

Well, I'm dropping out of this discussion because I have enough on my plate
with the GitHub migration than to keep fighting this.


>
> BTW: There are other things in PEP 7 which should probably be updated
> instead, e.g. it still says we should use C89, but we're having more
> and more C99 code (mostly new library functions) in CPython.
>

That's a whole other discussion (which I support, but I'm not going to lead
since I'm burned out at the moment on C-related discussions).

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/307ba4e8/attachment.html>

From abarnert at yahoo.com  Tue Jan 19 12:27:49 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Tue, 19 Jan 2016 09:27:49 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
Message-ID: <40C5A9D5-2EA9-4DC6-BFFF-FE95CCA5AD3D@yahoo.com>

> On Jan 19, 2016, at 08:56, Jim J. Jewett <jimjjewett at gmail.com> wrote:
> 
> On Mon Jan 18 03:39:42 EST 2016, Andrew Barnert pointed out:
>> 
>> Alternatively, it could say something like "braces must not be omitted;
>> when other C styles would use a braceless one-liner, a one-liner with
>> braces should be used instead; otherwise, they should be formatted as follows"
> 
> That "otherwise" gets a bit awkward, but I like the idea.  Perhaps
> "braces must not be omitted, and should normally be formatted as
> follows. ... Where other C styles would permit a braceless one-liner,
> the expression and braces may be moved to a single line, as follows: "
> 
>    if (x > 5) { y++ }
> 
> I think that is clearly better, but it may be *too* lightweight for
> flow control.
> 
>    if (!obj)
>        { return -1; }
> 
> does work for me, and I think the \n{} may actually be useful for
> warning that flow control takes a jump.

Your wording is much better than mine. And so is your suggestion. Giving people the option of 1 or 3 lines, but not 2, seems a little silly. And, while I rarely use or see your 2-line version in C, I use it quite a bit in C++ (and related languages like D), so it doesn't look at all weird to me. But I'll leave it up to people who only do C (and Python) and/or who are more familiar with the CPython code base to judge.

From ethan at stoneleaf.us  Tue Jan 19 13:29:07 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 19 Jan 2016 10:29:07 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
Message-ID: <569E8073.3010106@stoneleaf.us>

On 01/19/2016 08:56 AM, Jim J. Jewett wrote:

> That "otherwise" gets a bit awkward, but I like the idea.  Perhaps
> "braces must not be omitted, and should normally be formatted as
> follows. ... Where other C styles would permit a braceless one-liner,
> the expression and braces may be moved to a single line, as follows: "
>
>      if (x > 5) { y++ }
>
> I think that is clearly better, but it may be *too* lightweight for
> flow control.
>
>      if (!obj)
>          { return -1; }
>
> does work for me, and I think the \n{} may actually be useful for
> warning that flow control takes a jump.

Either of those two, with preference for the second on multiline ifs, 
seems quite readable to me.

--
~Ethan~


From guido at python.org  Tue Jan 19 14:36:21 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Jan 2016 11:36:21 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569E8073.3010106@stoneleaf.us>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
Message-ID: <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>

Let's not switch to either of those options. Visually I much prefer either
of these:

if (test) {
    blah;
}

or

if (test)
    blah;

over the versions with '{ blah; }' (regardless of whether it's on the same
line as 'if' or on the next line). It looks like the shorter versions are
mostly used inside macros, where aesthetics usually go out the door anyways
in favor of robustness.

Since this discussion is never going to end until someone says "enough",
let me just attempt that (though technically it's Brett's call) -- let's go
with the strong recommendation to prefer

if (test) {
    blah;
}

and stop there.

On Tue, Jan 19, 2016 at 10:29 AM, Ethan Furman <ethan at stoneleaf.us> wrote:

> On 01/19/2016 08:56 AM, Jim J. Jewett wrote:
>
> That "otherwise" gets a bit awkward, but I like the idea.  Perhaps
>> "braces must not be omitted, and should normally be formatted as
>> follows. ... Where other C styles would permit a braceless one-liner,
>> the expression and braces may be moved to a single line, as follows: "
>>
>>      if (x > 5) { y++ }
>>
>> I think that is clearly better, but it may be *too* lightweight for
>> flow control.
>>
>>      if (!obj)
>>          { return -1; }
>>
>> does work for me, and I think the \n{} may actually be useful for
>> warning that flow control takes a jump.
>>
>
> Either of those two, with preference for the second on multiline ifs,
> seems quite readable to me.
>
> --
> ~Ethan~
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/857c738f/attachment.html>

From brett at python.org  Tue Jan 19 15:12:25 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 19 Jan 2016 20:12:25 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
Message-ID: <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>

Here is a proposed update:

diff -r 633f51d10a67 pep-0007.txt
--- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
+++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
@@ -75,9 +75,9 @@
       }

 * Code structure: one space between keywords like ``if``, ``for`` and
-  the following left paren; no spaces inside the paren; braces may be
-  omitted where C permits but when present, they should be formatted
-  as shown::
+  the following left paren; no spaces inside the paren; braces are
+  strongly preferred but may be omitted where C permits, and they
+  should be formatted as shown::

       if (mro != NULL) {
           ...


On Tue, 19 Jan 2016 at 11:37 Guido van Rossum <guido at python.org> wrote:

> Let's not switch to either of those options. Visually I much prefer either
> of these:
>
> if (test) {
>     blah;
> }
>
> or
>
> if (test)
>     blah;
>
> over the versions with '{ blah; }' (regardless of whether it's on the same
> line as 'if' or on the next line). It looks like the shorter versions are
> mostly used inside macros, where aesthetics usually go out the door anyways
> in favor of robustness.
>
> Since this discussion is never going to end until someone says "enough",
> let me just attempt that (though technically it's Brett's call) -- let's go
> with the strong recommendation to prefer
>
> if (test) {
>     blah;
> }
>
> and stop there.
>
> On Tue, Jan 19, 2016 at 10:29 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>
>> On 01/19/2016 08:56 AM, Jim J. Jewett wrote:
>>
>> That "otherwise" gets a bit awkward, but I like the idea.  Perhaps
>>> "braces must not be omitted, and should normally be formatted as
>>> follows. ... Where other C styles would permit a braceless one-liner,
>>> the expression and braces may be moved to a single line, as follows: "
>>>
>>>      if (x > 5) { y++ }
>>>
>>> I think that is clearly better, but it may be *too* lightweight for
>>> flow control.
>>>
>>>      if (!obj)
>>>          { return -1; }
>>>
>>> does work for me, and I think the \n{} may actually be useful for
>>> warning that flow control takes a jump.
>>>
>>
>> Either of those two, with preference for the second on multiline ifs,
>> seems quite readable to me.
>>
>> --
>> ~Ethan~
>
>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/90ca7fd8/attachment.html>

From francismb at email.de  Tue Jan 19 15:22:27 2016
From: francismb at email.de (francismb)
Date: Tue, 19 Jan 2016 21:22:27 +0100
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
Message-ID: <569E9B03.80200@email.de>

Hi Brett,

On 01/19/2016 12:20 AM, Brett Cannon wrote:
> On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:
> 
>> While doing a review of http://bugs.python.org/review/26129/ I asked to
>> have curly braces put around all `if` statement bodies. Serhiy pointed out
>> that PEP 7 says curly braces are optional:
>> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
>> that.
>>
>> My argument is to require them to prevent bugs like the one Apple made
>> with OpenSSL about two years ago:
>> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
>> curly braces is purely an aesthetic thing while leaving them out can lead
>> to actual bugs.
>>
>> Anyone object if I update PEP 7 to remove the optionality of curly braces
>> in PEP 7?
>>
> 

What about about a code formatter bot ? (new workflow). If one just
could agree, then those reviews should just disappear (?).

Regards,
francis

From guido at python.org  Tue Jan 19 15:32:46 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 19 Jan 2016 12:32:46 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <569E9B03.80200@email.de>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
 <CAP1=2W6+h_=WQn=Cqs3FMwyfbh_pC3+1BUtjhzgzO7ZwdZgZtA@mail.gmail.com>
 <569E9B03.80200@email.de>
Message-ID: <CAP7+vJKtPpZ7hvn1uxq8Lyg9eXEoEyEYANGdcokEghrc4n+c7g@mail.gmail.com>

A formatter bot would be quite complicated to introduce without
disrupitions of everybody's workflow (remember that we have about half a
million lines of C code in the Python repo). If you want to discuss that
please start a new thread on python-dev.

On Tue, Jan 19, 2016 at 12:22 PM, francismb <francismb at email.de> wrote:

> Hi Brett,
>
> On 01/19/2016 12:20 AM, Brett Cannon wrote:
> > On Sun, 17 Jan 2016 at 11:10 Brett Cannon <brett at python.org> wrote:
> >
> >> While doing a review of http://bugs.python.org/review/26129/ I asked to
> >> have curly braces put around all `if` statement bodies. Serhiy pointed
> out
> >> that PEP 7 says curly braces are optional:
> >> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> >> that.
> >>
> >> My argument is to require them to prevent bugs like the one Apple made
> >> with OpenSSL about two years ago:
> >> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> >> curly braces is purely an aesthetic thing while leaving them out can
> lead
> >> to actual bugs.
> >>
> >> Anyone object if I update PEP 7 to remove the optionality of curly
> braces
> >> in PEP 7?
> >>
> >
>
> What about about a code formatter bot ? (new workflow). If one just
> could agree, then those reviews should just disappear (?).
>
> Regards,
> francis
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/7f02fd26/attachment-0001.html>

From greg at krypto.org  Tue Jan 19 15:59:13 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Tue, 19 Jan 2016 20:59:13 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
References: <CAP1=2W4go_mYeTf91EEWQUG9-aGSyu3uz9Mhrni0vB6jXpVMUA@mail.gmail.com>
Message-ID: <CAGE7PNKLJQVFgQbTyt=uP1ZGQ1VRwKCuMaByOeiYymvsHa76WA@mail.gmail.com>

On Sun, Jan 17, 2016 at 11:12 AM Brett Cannon <brett at python.org> wrote:

> While doing a review of http://bugs.python.org/review/26129/ I asked to
> have curly braces put around all `if` statement bodies. Serhiy pointed out
> that PEP 7 says curly braces are optional:
> https://www.python.org/dev/peps/pep-0007/#id5. I would like to change
> that.
>
> My argument is to require them to prevent bugs like the one Apple made
> with OpenSSL about two years ago:
> https://www.imperialviolet.org/2014/02/22/applebug.html. Skipping the
> curly braces is purely an aesthetic thing while leaving them out can lead
> to actual bugs.
>
> Anyone object if I update PEP 7 to remove the optionality of curly braces
> in PEP 7?
>

+1, always using {}s is just good C style.

(and, duh, of course we do *not* go modifying code for this retroactively,
pep8 vs our existing python code is evidence of that)

If I had _my_ way we'd require clang format for C/C++ files and yapf for
all Python files before accepting a commit.  Like any good modern open
source project should.  People who don't like defensive bug reducing coding
practices should be glad I don't get my way. :P

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/bf95fd6f/attachment.html>

From francismb at email.de  Tue Jan 19 15:59:18 2016
From: francismb at email.de (francismb)
Date: Tue, 19 Jan 2016 21:59:18 +0100
Subject: [Python-Dev] Code formatter bot
Message-ID: <569EA3A6.4010802@email.de>

Dear Core-Devs,
what's your opinion about a code-formatter bot for cpython.
Pros, Cons, where could be applicable (new commits, new workflow, it
doesn't make sense), ...


- At least it should follow PEP 7 ;-)
- ...


Thanks in advance,
francis


From srkunze at mail.de  Tue Jan 19 16:44:30 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Tue, 19 Jan 2016 22:44:30 +0100
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <569EA3A6.4010802@email.de>
References: <569EA3A6.4010802@email.de>
Message-ID: <569EAE3E.20509@mail.de>

Not a core dev, but I would definitely recommend using them.

Best,
Sven

On 19.01.2016 21:59, francismb wrote:
> Dear Core-Devs,
> what's your opinion about a code-formatter bot for cpython.
> Pros, Cons, where could be applicable (new commits, new workflow, it
> doesn't make sense), ...
>
>
> - At least it should follow PEP 7 ;-)
> - ...
>
>
> Thanks in advance,
> francis
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


From senthil at uthcode.com  Tue Jan 19 19:29:18 2016
From: senthil at uthcode.com (Senthil Kumaran)
Date: Tue, 19 Jan 2016 16:29:18 -0800
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <569EA3A6.4010802@email.de>
References: <569EA3A6.4010802@email.de>
Message-ID: <CAPOVWOTzQPfDvLQT2XJx9MEVX=rS31X620ZSoqdTa-nM=GKYOg@mail.gmail.com>

On Tue, Jan 19, 2016 at 12:59 PM, francismb <francismb at email.de> wrote:

> Pros, Cons, where could be applicable (new commits, new workflow, it
> doesn't make sense), ...
>

-1. formatting should be done by humans (with the help of tools) before
committing.
It should not be left to a robot to make automatic changes.

We already some pre-commit hooks which do basic checks. If anything more
automated is desirable then enhancement to the pre-commit hooks could be
the place to look for.
As far as I know, none of the core devs have expressed any complaints the
pre-commit hooks.


-- 
Senthil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160119/8f9e2395/attachment.html>

From vadmium+py at gmail.com  Tue Jan 19 22:33:37 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Wed, 20 Jan 2016 03:33:37 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
Message-ID: <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>

On 19 January 2016 at 20:12, Brett Cannon <brett at python.org> wrote:
> Here is a proposed update:
>
> diff -r 633f51d10a67 pep-0007.txt
> --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
> +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
> @@ -75,9 +75,9 @@
>        }
>
>  * Code structure: one space between keywords like ``if``, ``for`` and
> -  the following left paren; no spaces inside the paren; braces may be
> -  omitted where C permits but when present, they should be formatted
> -  as shown::
> +  the following left paren; no spaces inside the paren; braces are
> +  strongly preferred but may be omitted where C permits, and they
> +  should be formatted as shown::
>
>        if (mro != NULL) {
>            ...

This change seems to be accidentally smuggled in, in the guise of a
PEP 512 update :)

My view is I prefer always using curly brackets in my own code. It is
easier to add printf() debugging without making mistakes. I support
?strongly preferring? them in the style guide, which is as much as a
style guide can do anyway.

From greg at krypto.org  Wed Jan 20 02:29:43 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 20 Jan 2016 07:29:43 +0000
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <CAPOVWOTzQPfDvLQT2XJx9MEVX=rS31X620ZSoqdTa-nM=GKYOg@mail.gmail.com>
References: <569EA3A6.4010802@email.de>
 <CAPOVWOTzQPfDvLQT2XJx9MEVX=rS31X620ZSoqdTa-nM=GKYOg@mail.gmail.com>
Message-ID: <CAGE7PNJ9WZGm9251NxhBrdFrVtxWUUy-zJe6j3+zZYq54198Pg@mail.gmail.com>

Indeed, automated code formatting is a good thing. But a bot is the wrong
approach. You want a code formatting checker as a potential pre-submit hook
(like we have had for white space issues in the past), but until you have
super high confidence in it you need to make sure it is not a blocker for a
commit or push, just a default check that can be explicitly skipped for the
infrequent cases where it is wrong. It's a workflow thing. Devs should have
their editor setup to auto-run the correct formatter or easily run it if
not automatic.

-gps

On Tue, Jan 19, 2016 at 4:29 PM Senthil Kumaran <senthil at uthcode.com> wrote:

>
> On Tue, Jan 19, 2016 at 12:59 PM, francismb <francismb at email.de> wrote:
>
>> Pros, Cons, where could be applicable (new commits, new workflow, it
>> doesn't make sense), ...
>>
>
> -1. formatting should be done by humans (with the help of tools) before
> committing.
> It should not be left to a robot to make automatic changes.
>
> We already some pre-commit hooks which do basic checks. If anything more
> automated is desirable then enhancement to the pre-commit hooks could be
> the place to look for.
> As far as I know, none of the core devs have expressed any complaints the
> pre-commit hooks.
>
>
> --
> Senthil
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/438c1528/attachment.html>

From ben+python at benfinney.id.au  Wed Jan 20 03:35:45 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Wed, 20 Jan 2016 19:35:45 +1100
Subject: [Python-Dev] Code formatter bot
References: <569EA3A6.4010802@email.de>
Message-ID: <85wpr4myvy.fsf@benfinney.id.au>

francismb <francismb at email.de> writes:

> what's your opinion about a code-formatter bot for cpython.

What is the proposal? The opinions will surely depend on:

* What formatting is to be applied automatically?

  * If you propose to enforce rigid interpretations of the style-guide
    PEPs as automatic rules, that will incur the wrath of those who have
    made clear those PEPs are not to be used that way. There is a clear
    opinion (from at least the BDFL) that the style guide PEPs are
    guidelines to be applied with human judgement.

  * If on the other hand you propose to enforce only those rules which
    are strict enough to be applied automatically (e.g. ?don't mix
    spaces and TABs?, ?encode source using UTF-8 only?) then that's best
    done by editor plug-ins like EditorConfig[0].

* At which point in the workflow will the formatting be applied?

  * If you propose to change the code *after* the programmer sees it in
    their text editor, that is sure to be unpopular. The code committed
    to VCS should exactly match what the programmer sees when they
    choose to commit.

  * If you propose to reject the code at time of committing, there are
    VCS hooks that can do that; I don't know what different system you
    propose.

  * If you propose to present formatting violations as errors in the
    programmer's text editor so they can be corrected before using the
    VCS, then there are tools like EditorConfig[0] to do that.

[0]: EditorConfig <URL:http://editorconfig.org/>

-- 
 \         ?Pinky, are you pondering what I'm pondering?? ?I think so, |
  `\         Brain, but isn't a cucumber that small called a gherkin?? |
_o__)                                           ?_Pinky and The Brain_ |
Ben Finney


From victor.stinner at gmail.com  Wed Jan 20 12:40:14 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 20 Jan 2016 18:40:14 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
Message-ID: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>

Hi,

I proposed a patch for the devguide to give the current status of all
Python branches: active, bugfix, security only, end-of-line, with
their end-of-life when applicable (past date or scheduled date)
http://bugs.python.org/issue26165

What do you think? Does it look correct?

We will have to update this table each time that the status of a
branch change. Hopefully, it's not a common event, so it will not
require a lot of work for release managers :-)

Victor

From brett at python.org  Wed Jan 20 12:45:11 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 17:45:11 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
Message-ID: <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>

On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com> wrote:

> On 19 January 2016 at 20:12, Brett Cannon <brett at python.org> wrote:
> > Here is a proposed update:
> >
> > diff -r 633f51d10a67 pep-0007.txt
> > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
> > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
> > @@ -75,9 +75,9 @@
> >        }
> >
> >  * Code structure: one space between keywords like ``if``, ``for`` and
> > -  the following left paren; no spaces inside the paren; braces may be
> > -  omitted where C permits but when present, they should be formatted
> > -  as shown::
> > +  the following left paren; no spaces inside the paren; braces are
> > +  strongly preferred but may be omitted where C permits, and they
> > +  should be formatted as shown::
> >
> >        if (mro != NULL) {
> >            ...
>
> This change seems to be accidentally smuggled in, in the guise of a
> PEP 512 update :)
>

Darn, sorry about that; forgot I had that change sitting in my peps
checkout. I'll revert it when I get home (unless the change is actually
acceptable to Guido).

-Brett


>
> My view is I prefer always using curly brackets in my own code. It is
> easier to add printf() debugging without making mistakes. I support
> ?strongly preferring? them in the style guide, which is as much as a
> style guide can do anyway.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/1139362c/attachment.html>

From brett at python.org  Wed Jan 20 12:53:33 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 17:53:33 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
Message-ID: <CAP1=2W6YXfLNzrm+NvfkvWkH29X5Wf0MkiUFpd_6BxuLoRKrYA@mail.gmail.com>

On Wed, 20 Jan 2016 at 09:41 Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> I proposed a patch for the devguide to give the current status of all
> Python branches: active, bugfix, security only, end-of-line, with
> their end-of-life when applicable (past date or scheduled date)
> http://bugs.python.org/issue26165
>
> What do you think? Does it look correct?
>

I would update it have a "first release" date column and also the projected
EOL for Python 3.5. Otherwise LGTM.

-Brett


>
> We will have to update this table each time that the status of a
> branch change. Hopefully, it's not a common event, so it will not
> require a lot of work for release managers :-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/3c56d0ea/attachment.html>

From abarnert at yahoo.com  Wed Jan 20 12:58:12 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Wed, 20 Jan 2016 09:58:12 -0800
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <85wpr4myvy.fsf@benfinney.id.au>
References: <569EA3A6.4010802@email.de> <85wpr4myvy.fsf@benfinney.id.au>
Message-ID: <2859591B-BAF8-4D29-9509-AF2E6C48D076@yahoo.com>

On Jan 20, 2016, at 00:35, Ben Finney <ben+python at benfinney.id.au> wrote:
> 
> francismb <francismb at email.de> writes:
> 
>> what's your opinion about a code-formatter bot for cpython.
> 
> What is the proposal? The opinions will surely depend on:

... plus:

* How does the formatter bot deal with "legacy code"? Large parts of CPython predate PEPs 7 and 8, and the decision was made long ago not to reformat existing code unless that code is being substantially modified for some other reason. Similarly, when the PEPs are updated, the usual decision is to not reformat old code.

* When code _is_ auto-reformatted, what tools do you have to help git's merge logic, Reitveld, human readers looking at diffs or blame/annotate locally or on the web, etc. look past the hundreds of trivial changes to highlight the ones that matter?

* What's the argument for specifically automating code formatting instead of any of the other things a commit-triggered linter can catch just as easily?

But one comment on Ben's comment:

>  * If on the other hand you propose to enforce only those rules which
>    are strict enough to be applied automatically (e.g. ?don't mix
>    spaces and TABs?, ?encode source using UTF-8 only?) then that's best
>    done by editor plug-ins like EditorConfig[0].

In my experience (although mostly with projects with a lot fewer contributors than CPython...), it can be helpful to have both suggested editor plugins that do the auto formatting on the dev's computer, and VCS-triggered checkers that ensure the formatting was correct. That catches those occasional cases where you do a quick "trivial" edit in nano instead of your usual editor and then forget you did so and try to check in), without the nasty side-effects you mention later (like committing code you haven't seen).

(Of course writing plugins that understand "legacy code" in the exact same way as the commit filter can be tricky, but in that case, it's better to know that one or the other isn't working as intended--both so a human can decide, and so people can see the bug in the plugin or filter--than to automatically make changes that weren't wanted.)


From ben+python at benfinney.id.au  Wed Jan 20 13:09:31 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Thu, 21 Jan 2016 05:09:31 +1100
Subject: [Python-Dev] Code formatter bot
References: <569EA3A6.4010802@email.de> <85wpr4myvy.fsf@benfinney.id.au>
 <2859591B-BAF8-4D29-9509-AF2E6C48D076@yahoo.com>
Message-ID: <85oacgm8bo.fsf@benfinney.id.au>

Andrew Barnert via Python-Dev <python-dev at python.org> writes:

> [?] it can be helpful to have both suggested editor plugins that do
> the auto formatting on the dev's computer, and VCS-triggered checkers
> that ensure the formatting was correct.

Right, I was not intending the different stages to be exclusive. I was
seeking clarification from the OP on the intended stages proposed to be
automated.

It's also worth noting that neither of the above approaches you mention
would qualify IMO to be termed a ?bot?, since they still leave it to the
human operator to deal with formatting violations before the edits
reach the VCS.

But that's another clarification I'm seeking from the OP: what ?bot? is
being proposed?

-- 
 \      ?Nothing is more sacred than the facts.? ?Sam Harris, _The End |
  `\                                                   of Faith_, 2004 |
_o__)                                                                  |
Ben Finney


From yselivanov.ml at gmail.com  Wed Jan 20 13:02:56 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 13:02:56 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
Message-ID: <569FCBD0.4040307@gmail.com>

On 2016-01-18 5:43 PM, Victor Stinner wrote:
> Is someone opposed to this PEP 509?
>
> The main complain was the change on the public Python API, but the PEP
> doesn't change the Python API anymore.
>
> I'm not aware of any remaining issue on this PEP.

Victor,

I've been experimenting with the PEP to implement a per-opcode
cache in ceval loop (I'll share my progress on that in a few
days).  This allows to significantly speedup LOAD_GLOBAL and
LOAD_METHOD opcodes, to the point, where they don't require
any dict lookups at all.  Some macro-benchmarks (such as
chameleon_v2) demonstrate impressive ~10% performance boost.

I rely on your dict->ma_version to implement cache invalidation.

However, besides guarding against version change, I also want
to guard against the dict being swapped for another dict, to
avoid situations like this:


     def foo():
         print(bar)

     exec(foo.__code__, {'bar': 1}, {})
     exec(foo.__code__, {'bar': 2}, {})


What I propose is to add a pointer "ma_extra" (same 64bits),
which will be set to NULL for most dict instances (instead of
ma_version).  "ma_extra" can then point to a struct that has a
globally unique dict ID (uint64), and a version tag (unit64).
A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
efficiently fetch the version/unique ID of the dict for guards.

"ma_extra" would also make it easier for us to extend dicts
in the future.

Yury

From fijall at gmail.com  Wed Jan 20 13:15:21 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Jan 2016 19:15:21 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FCBD0.4040307@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
Message-ID: <CAK5idxRfvMeorOHaVjuYttf5+CiQxo6aiyVUdELGLWG3Fzhg7g@mail.gmail.com>

The easiest version is to have global numbering (as opposed to local).

Anyway, I would strongly suggest getting some benchmarks done and
showing performance benefits first, because you don't want PEPs to be
final when you don't exactly know the details.

On Wed, Jan 20, 2016 at 7:02 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On 2016-01-18 5:43 PM, Victor Stinner wrote:
>>
>> Is someone opposed to this PEP 509?
>>
>> The main complain was the change on the public Python API, but the PEP
>> doesn't change the Python API anymore.
>>
>> I'm not aware of any remaining issue on this PEP.
>
>
> Victor,
>
> I've been experimenting with the PEP to implement a per-opcode
> cache in ceval loop (I'll share my progress on that in a few
> days).  This allows to significantly speedup LOAD_GLOBAL and
> LOAD_METHOD opcodes, to the point, where they don't require
> any dict lookups at all.  Some macro-benchmarks (such as
> chameleon_v2) demonstrate impressive ~10% performance boost.
>
> I rely on your dict->ma_version to implement cache invalidation.
>
> However, besides guarding against version change, I also want
> to guard against the dict being swapped for another dict, to
> avoid situations like this:
>
>
>     def foo():
>         print(bar)
>
>     exec(foo.__code__, {'bar': 1}, {})
>     exec(foo.__code__, {'bar': 2}, {})
>
>
> What I propose is to add a pointer "ma_extra" (same 64bits),
> which will be set to NULL for most dict instances (instead of
> ma_version).  "ma_extra" can then point to a struct that has a
> globally unique dict ID (uint64), and a version tag (unit64).
> A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
> efficiently fetch the version/unique ID of the dict for guards.
>
> "ma_extra" would also make it easier for us to extend dicts
> in the future.
>
> Yury
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From brett at python.org  Wed Jan 20 13:22:50 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 18:22:50 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FCBD0.4040307@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
Message-ID: <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>

On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

> On 2016-01-18 5:43 PM, Victor Stinner wrote:
> > Is someone opposed to this PEP 509?
> >
> > The main complain was the change on the public Python API, but the PEP
> > doesn't change the Python API anymore.
> >
> > I'm not aware of any remaining issue on this PEP.
>
> Victor,
>
> I've been experimenting with the PEP to implement a per-opcode
> cache in ceval loop (I'll share my progress on that in a few
> days).  This allows to significantly speedup LOAD_GLOBAL and
> LOAD_METHOD opcodes, to the point, where they don't require
> any dict lookups at all.  Some macro-benchmarks (such as
> chameleon_v2) demonstrate impressive ~10% performance boost.
>

Ooh, now my brain is trying to figure out the design of the cache. :)


>
> I rely on your dict->ma_version to implement cache invalidation.
>
> However, besides guarding against version change, I also want
> to guard against the dict being swapped for another dict, to
> avoid situations like this:
>
>
>      def foo():
>          print(bar)
>
>      exec(foo.__code__, {'bar': 1}, {})
>      exec(foo.__code__, {'bar': 2}, {})
>
>
> What I propose is to add a pointer "ma_extra" (same 64bits),
> which will be set to NULL for most dict instances (instead of
> ma_version).  "ma_extra" can then point to a struct that has a
> globally unique dict ID (uint64), and a version tag (unit64).
> A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
> efficiently fetch the version/unique ID of the dict for guards.
>
> "ma_extra" would also make it easier for us to extend dicts
> in the future.
>

Why can't you simply use the id of the dict object as the globally unique
dict ID? It's already globally unique amongst all Python objects which
makes it inherently unique amongst dicts.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/592786ef/attachment.html>

From fijall at gmail.com  Wed Jan 20 13:36:53 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Jan 2016 19:36:53 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
Message-ID: <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>

On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon <brett at python.org> wrote:
>
>
> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>>
>> On 2016-01-18 5:43 PM, Victor Stinner wrote:
>> > Is someone opposed to this PEP 509?
>> >
>> > The main complain was the change on the public Python API, but the PEP
>> > doesn't change the Python API anymore.
>> >
>> > I'm not aware of any remaining issue on this PEP.
>>
>> Victor,
>>
>> I've been experimenting with the PEP to implement a per-opcode
>> cache in ceval loop (I'll share my progress on that in a few
>> days).  This allows to significantly speedup LOAD_GLOBAL and
>> LOAD_METHOD opcodes, to the point, where they don't require
>> any dict lookups at all.  Some macro-benchmarks (such as
>> chameleon_v2) demonstrate impressive ~10% performance boost.
>
>
> Ooh, now my brain is trying to figure out the design of the cache. :)
>
>>
>>
>> I rely on your dict->ma_version to implement cache invalidation.
>>
>> However, besides guarding against version change, I also want
>> to guard against the dict being swapped for another dict, to
>> avoid situations like this:
>>
>>
>>      def foo():
>>          print(bar)
>>
>>      exec(foo.__code__, {'bar': 1}, {})
>>      exec(foo.__code__, {'bar': 2}, {})
>>
>>
>> What I propose is to add a pointer "ma_extra" (same 64bits),
>> which will be set to NULL for most dict instances (instead of
>> ma_version).  "ma_extra" can then point to a struct that has a
>> globally unique dict ID (uint64), and a version tag (unit64).
>> A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
>> efficiently fetch the version/unique ID of the dict for guards.
>>
>> "ma_extra" would also make it easier for us to extend dicts
>> in the future.
>
>
> Why can't you simply use the id of the dict object as the globally unique
> dict ID? It's already globally unique amongst all Python objects which makes
> it inherently unique amongst dicts.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

Brett, you need two things - the ID of the dict and the version tag.
What we do in pypy is we have a small object (called, surprisingly,
VersionTag()) and we use the ID of that. That way you can change the
version id of an existing dict and have only one field.

From yselivanov.ml at gmail.com  Wed Jan 20 13:38:49 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 13:38:49 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxRfvMeorOHaVjuYttf5+CiQxo6aiyVUdELGLWG3Fzhg7g@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAK5idxRfvMeorOHaVjuYttf5+CiQxo6aiyVUdELGLWG3Fzhg7g@mail.gmail.com>
Message-ID: <569FD439.9040704@gmail.com>

On 2016-01-20 1:15 PM, Maciej Fijalkowski wrote:
> [..]
>
> Anyway, I would strongly suggest getting some benchmarks done and
> showing performance benefits first, because you don't want PEPs to be
> final when you don't exactly know the details.

I agree 100%.

Yury


From tjreedy at udel.edu  Wed Jan 20 13:39:58 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Jan 2016 13:39:58 -0500
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
Message-ID: <n7okad$9ol$1@ger.gmane.org>

On 1/20/2016 12:40 PM, Victor Stinner wrote:
> Hi,
>
> I proposed a patch for the devguide to give the current status of all
> Python branches: active, bugfix, security only, end-of-line, with
> their end-of-life when applicable (past date or scheduled date)
> http://bugs.python.org/issue26165
>
> What do you think? Does it look correct?

I thought end-of-life was 5 years after initial release, not 5 years 
after last bugfix.  That would put eol for 3.4 in Feb 2019, I believe.

> We will have to update this table each time that the status of a
> branch change. Hopefully, it's not a common event, so it will not
> require a lot of work for release managers :-)

I believe there is some text describing current releases somewhere that 
also needs to be changed.  The release pep or scripts should have a 
reminder in the sections about the transitions.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Wed Jan 20 13:42:24 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 20 Jan 2016 13:42:24 -0500
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
Message-ID: <n7okeu$9ol$2@ger.gmane.org>

On 1/20/2016 12:45 PM, Brett Cannon wrote:
>
>
> On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com
> <mailto:vadmium%2Bpy at gmail.com>> wrote:
>
>     On 19 January 2016 at 20:12, Brett Cannon <brett at python.org
>     <mailto:brett at python.org>> wrote:
>      > Here is a proposed update:
>      >
>      > diff -r 633f51d10a67 pep-0007.txt
>      > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
>      > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
>      > @@ -75,9 +75,9 @@
>      >        }
>      >
>      >  * Code structure: one space between keywords like ``if``,
>     ``for`` and
>      > -  the following left paren; no spaces inside the paren; braces
>     may be
>      > -  omitted where C permits but when present, they should be formatted
>      > -  as shown::
>      > +  the following left paren; no spaces inside the paren; braces are
>      > +  strongly preferred but may be omitted where C permits, and they
>      > +  should be formatted as shown::
>      >
>      >        if (mro != NULL) {
>      >            ...
>
>     This change seems to be accidentally smuggled in, in the guise of a
>     PEP 512 update :)
>
>
> Darn, sorry about that; forgot I had that change sitting in my peps
> checkout. I'll revert it when I get home (unless the change is actually
> acceptable to Guido).

I thought that the above was your intentional compromise change given 
the range of opinions ;-).

-- 
Terry Jan Reedy


From yselivanov.ml at gmail.com  Wed Jan 20 13:45:51 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 13:45:51 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
Message-ID: <569FD5DF.2000700@gmail.com>

Brett,

On 2016-01-20 1:22 PM, Brett Cannon wrote:
>
>
> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com 
> <mailto:yselivanov.ml at gmail.com>> wrote:
>
>     On 2016-01-18 5:43 PM, Victor Stinner wrote:
>     > Is someone opposed to this PEP 509?
>     >
>     > The main complain was the change on the public Python API, but
>     the PEP
>     > doesn't change the Python API anymore.
>     >
>     > I'm not aware of any remaining issue on this PEP.
>
>     Victor,
>
>     I've been experimenting with the PEP to implement a per-opcode
>     cache in ceval loop (I'll share my progress on that in a few
>     days).  This allows to significantly speedup LOAD_GLOBAL and
>     LOAD_METHOD opcodes, to the point, where they don't require
>     any dict lookups at all.  Some macro-benchmarks (such as
>     chameleon_v2) demonstrate impressive ~10% performance boost.
>
>
> Ooh, now my brain is trying to figure out the design of the cache. :)

Yeah, it's tricky.  I'll need some time to draft a comprehensible
overview.  And I want to implement a couple more optimizations and
benchmark it better.

BTW, I've some updates (html5lib benchmark for py3, new benchmarks
for calling C methods, and I want to port some PyPy benchmakrs)
to the benchmarks suite.  Should I just commit them, or should I
use bugs.python.org?

>
>     I rely on your dict->ma_version to implement cache invalidation.
>
>     However, besides guarding against version change, I also want
>     to guard against the dict being swapped for another dict, to
>     avoid situations like this:
>
>
>          def foo():
>              print(bar)
>
>          exec(foo.__code__, {'bar': 1}, {})
>          exec(foo.__code__, {'bar': 2}, {})
>
>
>     What I propose is to add a pointer "ma_extra" (same 64bits),
>     which will be set to NULL for most dict instances (instead of
>     ma_version).  "ma_extra" can then point to a struct that has a
>     globally unique dict ID (uint64), and a version tag (unit64).
>     A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
>     efficiently fetch the version/unique ID of the dict for guards.
>
>     "ma_extra" would also make it easier for us to extend dicts
>     in the future.
>
>
> Why can't you simply use the id of the dict object as the globally 
> unique dict ID? It's already globally unique amongst all Python 
> objects which makes it inherently unique amongst dicts.

We have a freelist for dicts -- so if the dict dies, there
could be a new dict in its place, with the same ma_version.

While the probability of such hiccups is low, we still have
to account for it.

Yury


From yselivanov.ml at gmail.com  Wed Jan 20 14:00:31 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 14:00:31 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
Message-ID: <569FD94F.2000808@gmail.com>



On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote:
> On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon <brett at python.org> wrote:
>>
>> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com> wrote:
[..]
>>> "ma_extra" would also make it easier for us to extend dicts
>>> in the future.
>>
>> Why can't you simply use the id of the dict object as the globally unique
>> dict ID? It's already globally unique amongst all Python objects which makes
>> it inherently unique amongst dicts.
>>
>>
> Brett, you need two things - the ID of the dict and the version tag.
> What we do in pypy is we have a small object (called, surprisingly,
> VersionTag()) and we use the ID of that. That way you can change the
> version id of an existing dict and have only one field.


Yeah, that's essentially what I propose with ma_extra.

Yury

From fijall at gmail.com  Wed Jan 20 14:02:53 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Jan 2016 20:02:53 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FD94F.2000808@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FD94F.2000808@gmail.com>
Message-ID: <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>

On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>
>
> On 2016-01-20 1:36 PM, Maciej Fijalkowski wrote:
>>
>> On Wed, Jan 20, 2016 at 7:22 PM, Brett Cannon <brett at python.org> wrote:
>>>
>>>
>>> On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com>
>>> wrote:
>
> [..]
>>>>
>>>> "ma_extra" would also make it easier for us to extend dicts
>>>> in the future.
>>>
>>>
>>> Why can't you simply use the id of the dict object as the globally unique
>>> dict ID? It's already globally unique amongst all Python objects which
>>> makes
>>> it inherently unique amongst dicts.
>>>
>>>
>> Brett, you need two things - the ID of the dict and the version tag.
>> What we do in pypy is we have a small object (called, surprisingly,
>> VersionTag()) and we use the ID of that. That way you can change the
>> version id of an existing dict and have only one field.
>
>
>
> Yeah, that's essentially what I propose with ma_extra.
>
> Yury

The trick is we use only one field :-)

you're proposing to have both fields - version tag and dict id. Why
not just use the id of the object (without any fields)?

From yselivanov.ml at gmail.com  Wed Jan 20 14:08:48 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 14:08:48 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FD94F.2000808@gmail.com>
 <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>
Message-ID: <569FDB40.9060906@gmail.com>


On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote:
> On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>
[..]
>>> Brett, you need two things - the ID of the dict and the version tag.
>>> What we do in pypy is we have a small object (called, surprisingly,
>>> VersionTag()) and we use the ID of that. That way you can change the
>>> version id of an existing dict and have only one field.
>> Yeah, that's essentially what I propose with ma_extra.
>>
>> Yury
> The trick is we use only one field :-)
>
> you're proposing to have both fields - version tag and dict id. Why
> not just use the id of the object (without any fields)?

What if your old dict is GCed, its "VersionTag()" (1) object is
freed, and you have a new dict, for which a new "VersionTag()" (2)
object happens to be allocated at the same address as (1)?

Yury


From fijall at gmail.com  Wed Jan 20 14:09:54 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Jan 2016 20:09:54 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FDB40.9060906@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FD94F.2000808@gmail.com>
 <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>
 <569FDB40.9060906@gmail.com>
Message-ID: <CAK5idxTXD2_-6Osr4f=Yq+1rNvkQQ4te6M8Bo6wO3xMrdJHM6A@mail.gmail.com>

On Wed, Jan 20, 2016 at 8:08 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>
> On 2016-01-20 2:02 PM, Maciej Fijalkowski wrote:
>>
>> On Wed, Jan 20, 2016 at 8:00 PM, Yury Selivanov <yselivanov.ml at gmail.com>
>> wrote:
>>
> [..]
>>>>
>>>> Brett, you need two things - the ID of the dict and the version tag.
>>>> What we do in pypy is we have a small object (called, surprisingly,
>>>> VersionTag()) and we use the ID of that. That way you can change the
>>>> version id of an existing dict and have only one field.
>>>
>>> Yeah, that's essentially what I propose with ma_extra.
>>>
>>> Yury
>>
>> The trick is we use only one field :-)
>>
>> you're proposing to have both fields - version tag and dict id. Why
>> not just use the id of the object (without any fields)?
>
>
> What if your old dict is GCed, its "VersionTag()" (1) object is
> freed, and you have a new dict, for which a new "VersionTag()" (2)
> object happens to be allocated at the same address as (1)?
>
> Yury
>

You don't free a version tag that's stored in the guard. You store the
object and not id

From yselivanov.ml at gmail.com  Wed Jan 20 14:23:43 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 14:23:43 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxTXD2_-6Osr4f=Yq+1rNvkQQ4te6M8Bo6wO3xMrdJHM6A@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FD94F.2000808@gmail.com>
 <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>
 <569FDB40.9060906@gmail.com>
 <CAK5idxTXD2_-6Osr4f=Yq+1rNvkQQ4te6M8Bo6wO3xMrdJHM6A@mail.gmail.com>
Message-ID: <569FDEBF.2090709@gmail.com>



On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote:
>> >
> You don't free a version tag that's stored in the guard. You store the
> object and not id

Ah, got it.  Although for my current cache design it would be
more memory efficient to use the dict itself to store its own
unique id and tag, hence my "ma_extra" proposal.  In any case,
the current "ma_version" proposal is flawed :(

Yury

From fijall at gmail.com  Wed Jan 20 14:31:42 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 20 Jan 2016 20:31:42 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FDEBF.2090709@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FD94F.2000808@gmail.com>
 <CAK5idxTbDgjjBOndMrzmoU8tyMm5ZX9+r9fHpiviX5j-buiZDg@mail.gmail.com>
 <569FDB40.9060906@gmail.com>
 <CAK5idxTXD2_-6Osr4f=Yq+1rNvkQQ4te6M8Bo6wO3xMrdJHM6A@mail.gmail.com>
 <569FDEBF.2090709@gmail.com>
Message-ID: <CAK5idxT-hhUQZ1s02Yw+n73aHgCUfhdbGDnZ2k-fdOXdWNeSOA@mail.gmail.com>

there is also the problem that you don't want it on all dicts. So
having two extra words is more to pay than having extra objects (also,
comparison is cheaper for guards)

On Wed, Jan 20, 2016 at 8:23 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
>
>
> On 2016-01-20 2:09 PM, Maciej Fijalkowski wrote:
>>>
>>> >
>>
>> You don't free a version tag that's stored in the guard. You store the
>> object and not id
>
>
> Ah, got it.  Although for my current cache design it would be
> more memory efficient to use the dict itself to store its own
> unique id and tag, hence my "ma_extra" proposal.  In any case,
> the current "ma_version" proposal is flawed :(
>
> Yury

From v+python at g.nevcal.com  Wed Jan 20 14:45:42 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 20 Jan 2016 11:45:42 -0800
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
Message-ID: <569FE3E6.303@g.nevcal.com>

On 1/20/2016 10:36 AM, Maciej Fijalkowski wrote:
>> Why can't you simply use the id of the dict object as the globally unique
>> >dict ID? It's already globally unique amongst all Python objects which makes
>> >it inherently unique amongst dicts.
>> >
>> >_______________________________________________
>> >Python-Dev mailing list
>> >Python-Dev at python.org
>> >https://mail.python.org/mailman/listinfo/python-dev
>> >Unsubscribe:
>> >https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>> >
> Brett, you need two things - the ID of the dict and the version tag.
> What we do in pypy is we have a small object (called, surprisingly,
> VersionTag()) and we use the ID of that. That way you can change the
> version id of an existing dict and have only one field.
For the reuse case, can't you simply keep the ma_version "live" in dict 
items on the free list, rather than starting over at (presumably) 0 ?  
Then if the dict is reused, it bumps the ma_version, and the fallback 
code goes on with (presumably) relocating the original dict (oops, it's 
gone), and dealing with the fallout.

Then you can use the regular dict id as the globally unique dict id?  
And only need the one uint64, rather than a separately allocated extra 
pair of uint64?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/5c1b4ebb/attachment-0001.html>

From brett at python.org  Wed Jan 20 15:18:53 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 20:18:53 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <n7okeu$9ol$2@ger.gmane.org>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
Message-ID: <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>

On Wed, 20 Jan 2016 at 10:45 Terry Reedy <tjreedy at udel.edu> wrote:

> On 1/20/2016 12:45 PM, Brett Cannon wrote:
> >
> >
> > On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com
> > <mailto:vadmium%2Bpy at gmail.com>> wrote:
> >
> >     On 19 January 2016 at 20:12, Brett Cannon <brett at python.org
> >     <mailto:brett at python.org>> wrote:
> >      > Here is a proposed update:
> >      >
> >      > diff -r 633f51d10a67 pep-0007.txt
> >      > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
> >      > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
> >      > @@ -75,9 +75,9 @@
> >      >        }
> >      >
> >      >  * Code structure: one space between keywords like ``if``,
> >     ``for`` and
> >      > -  the following left paren; no spaces inside the paren; braces
> >     may be
> >      > -  omitted where C permits but when present, they should be
> formatted
> >      > -  as shown::
> >      > +  the following left paren; no spaces inside the paren; braces
> are
> >      > +  strongly preferred but may be omitted where C permits, and they
> >      > +  should be formatted as shown::
> >      >
> >      >        if (mro != NULL) {
> >      >            ...
> >
> >     This change seems to be accidentally smuggled in, in the guise of a
> >     PEP 512 update :)
> >
> >
> > Darn, sorry about that; forgot I had that change sitting in my peps
> > checkout. I'll revert it when I get home (unless the change is actually
> > acceptable to Guido).
>
> I thought that the above was your intentional compromise change given
> the range of opinions ;-).
>

It is, but Guido is the author of PEP 7 and so I didn't want to check in
the change without his approval first.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/98bc6ea2/attachment.html>

From brett at python.org  Wed Jan 20 15:20:02 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 20:20:02 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <n7okad$9ol$1@ger.gmane.org>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
Message-ID: <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>

On Wed, 20 Jan 2016 at 10:40 Terry Reedy <tjreedy at udel.edu> wrote:

> On 1/20/2016 12:40 PM, Victor Stinner wrote:
> > Hi,
> >
> > I proposed a patch for the devguide to give the current status of all
> > Python branches: active, bugfix, security only, end-of-line, with
> > their end-of-life when applicable (past date or scheduled date)
> > http://bugs.python.org/issue26165
> >
> > What do you think? Does it look correct?
>
> I thought end-of-life was 5 years after initial release, not 5 years
> after last bugfix.


It is, which is why I requested the first release date be a column.


>   That would put eol for 3.4 in Feb 2019, I believe.
>
> > We will have to update this table each time that the status of a
> > branch change. Hopefully, it's not a common event, so it will not
> > require a lot of work for release managers :-)
>
> I believe there is some text describing current releases somewhere that
> also needs to be changed.  The release pep or scripts should have a
> reminder in the sections about the transitions.
>

PEP 101 would need a tweak to remind the RM to update the devguide.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/5e5792de/attachment.html>

From brett at python.org  Wed Jan 20 15:23:59 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 20:23:59 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FD5DF.2000700@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <569FD5DF.2000700@gmail.com>
Message-ID: <CAP1=2W5TfSTWsetKochoi683RrwZsCC=N9SQCOWHcbMsGHsxcQ@mail.gmail.com>

On Wed, 20 Jan 2016 at 10:46 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

> Brett,
>
> On 2016-01-20 1:22 PM, Brett Cannon wrote:
> >
> >
> > On Wed, 20 Jan 2016 at 10:11 Yury Selivanov <yselivanov.ml at gmail.com
> > <mailto:yselivanov.ml at gmail.com>> wrote:
> >
> >     On 2016-01-18 5:43 PM, Victor Stinner wrote:
> >     > Is someone opposed to this PEP 509?
> >     >
> >     > The main complain was the change on the public Python API, but
> >     the PEP
> >     > doesn't change the Python API anymore.
> >     >
> >     > I'm not aware of any remaining issue on this PEP.
> >
> >     Victor,
> >
> >     I've been experimenting with the PEP to implement a per-opcode
> >     cache in ceval loop (I'll share my progress on that in a few
> >     days).  This allows to significantly speedup LOAD_GLOBAL and
> >     LOAD_METHOD opcodes, to the point, where they don't require
> >     any dict lookups at all.  Some macro-benchmarks (such as
> >     chameleon_v2) demonstrate impressive ~10% performance boost.
> >
> >
> > Ooh, now my brain is trying to figure out the design of the cache. :)
>
> Yeah, it's tricky.  I'll need some time to draft a comprehensible
> overview.  And I want to implement a couple more optimizations and
> benchmark it better.
>
> BTW, I've some updates (html5lib benchmark for py3, new benchmarks
> for calling C methods, and I want to port some PyPy benchmakrs)
> to the benchmarks suite.  Should I just commit them, or should I
> use bugs.python.org?
>

I actually emailed speed@ to see if people were interested in finally
sitting down with all the various VM implementations at PyCon and trying to
come up with a reasonable base set of benchmarks that better reflect modern
Python usage, but I never heard back.

Anyway, issues on bugs.python.org are probably best to talk about new
benchmarks before adding them (fixes and updates to pre-existing benchmarks
can just go in).


>
> >
> >     I rely on your dict->ma_version to implement cache invalidation.
> >
> >     However, besides guarding against version change, I also want
> >     to guard against the dict being swapped for another dict, to
> >     avoid situations like this:
> >
> >
> >          def foo():
> >              print(bar)
> >
> >          exec(foo.__code__, {'bar': 1}, {})
> >          exec(foo.__code__, {'bar': 2}, {})
> >
> >
> >     What I propose is to add a pointer "ma_extra" (same 64bits),
> >     which will be set to NULL for most dict instances (instead of
> >     ma_version).  "ma_extra" can then point to a struct that has a
> >     globally unique dict ID (uint64), and a version tag (unit64).
> >     A macro like PyDict_GET_ID and PyDict_GET_VERSION could then
> >     efficiently fetch the version/unique ID of the dict for guards.
> >
> >     "ma_extra" would also make it easier for us to extend dicts
> >     in the future.
> >
> >
> > Why can't you simply use the id of the dict object as the globally
> > unique dict ID? It's already globally unique amongst all Python
> > objects which makes it inherently unique amongst dicts.
>
> We have a freelist for dicts -- so if the dict dies, there
> could be a new dict in its place, with the same ma_version.
>

Ah, I figured it would be too simple to use something we already had.


>
> While the probability of such hiccups is low, we still have
> to account for it.
>

Yep.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/3802ec6b/attachment.html>

From yselivanov.ml at gmail.com  Wed Jan 20 15:27:12 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 15:27:12 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FE3E6.303@g.nevcal.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com>
Message-ID: <569FEDA0.1040506@gmail.com>



On 2016-01-20 2:45 PM, Glenn Linderman wrote:
> For the reuse case, can't you simply keep the ma_version "live" in 
> dict items on the free list, rather than starting over at (presumably) 
> 0 ?  Then if the dict is reused, it bumps the ma_version, and the 
> fallback code goes on with (presumably) relocating the original dict 
> (oops, it's gone), and dealing with the fallout.

Not all dicts are created from a freelist, and not all dicts go to the 
freelist when they are GCed.

You still can have this situation:

- dict "A" is used as f_locals for a frame, its ma_version is set to X
- dict "A" is GCed, but the freelist is full, so it's just freed
- after a while, you call the code object, a new dict "B" is allocated 
with malloc (since now the freelist happens to be empty) for f_locals
- it happens that "B" is allocated where "A" was, and its ma_version 
happens to be X by an accident

>
> Then you can use the regular dict id as the globally unique dict id?  
> And only need the one uint64, rather than a separately allocated extra 
> pair of uint64?

In my design only namespace dicts will have a non-NULL ma_extra, which 
means that only a fraction of dicts will actually have a separated pair 
of uint64s.

I think that we should either use one global ma_version (as Maciej 
suggested) or ma_extra, as it gives more flexibility for us to extend 
dicts in the future.

A global (shared between all dicts) unit64 ma_version is actually quite 
reliable -- if a program does 1,000,000 dict modifications per second, 
it would take it 600,000 years till wrap-around.

Yury

From brett at python.org  Wed Jan 20 15:50:41 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 20:50:41 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FEDA0.1040506@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
Message-ID: <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>

On Wed, 20 Jan 2016 at 12:27 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

>
>
> On 2016-01-20 2:45 PM, Glenn Linderman wrote:
> > For the reuse case, can't you simply keep the ma_version "live" in
> > dict items on the free list, rather than starting over at (presumably)
> > 0 ?  Then if the dict is reused, it bumps the ma_version, and the
> > fallback code goes on with (presumably) relocating the original dict
> > (oops, it's gone), and dealing with the fallout.
>
> Not all dicts are created from a freelist, and not all dicts go to the
> freelist when they are GCed.
>
> You still can have this situation:
>
> - dict "A" is used as f_locals for a frame, its ma_version is set to X
> - dict "A" is GCed, but the freelist is full, so it's just freed
> - after a while, you call the code object, a new dict "B" is allocated
> with malloc (since now the freelist happens to be empty) for f_locals
> - it happens that "B" is allocated where "A" was, and its ma_version
> happens to be X by an accident
>
> >
> > Then you can use the regular dict id as the globally unique dict id?
> > And only need the one uint64, rather than a separately allocated extra
> > pair of uint64?
>
> In my design only namespace dicts will have a non-NULL ma_extra, which
> means that only a fraction of dicts will actually have a separated pair
> of uint64s.
>
> I think that we should either use one global ma_version (as Maciej
> suggested) or ma_extra, as it gives more flexibility for us to extend
> dicts in the future.
>
> A global (shared between all dicts) unit64 ma_version is actually quite
> reliable -- if a program does 1,000,000 dict modifications per second,
> it would take it 600,000 years till wrap-around.
>

Since you're going to need to hold the GIL for the modifications there
won't be any locking or contention problems, so it sounds like the global
value is the best since that's simple, uses the least amount of memory, and
will be easiest to use as a modification check since that will be a simple
uint64 comparison instead of comparing a GUID + version.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/339a907d/attachment.html>

From mark at hotpy.org  Wed Jan 20 16:06:01 2016
From: mark at hotpy.org (Mark Shannon)
Date: Wed, 20 Jan 2016 21:06:01 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
Message-ID: <569FF6B9.6000907@hotpy.org>



On 11/01/16 16:49, Victor Stinner wrote:
> Hi,
>
> After a first round on python-ideas, here is the second version of my
> PEP. The main changes since the first version are that the dictionary
> version is no more exposed at the Python level and the field type now
> also has a size of 64-bit on 32-bit platforms.
>
> The PEP is part of a serie of 3 PEP adding an API to implement a
> static Python optimizer specializing functions with guards. The second
> PEP is currently discussed on python-ideas and I'm still working on
> the third PEP.

If anyone wants to experiment (at the C, not Python, level) with dict 
versioning to optimise load-global/builtins, then you can do so without 
adding a version number.

A "version" can created by splitting the dict with "make_keys_shared" 
and then making the keys-object immutable by setting "dk_usable" to zero.
This means that any change to the keys will force a keys-object change, 
but changes to the values will not.
For many optimisations, this is want you want.

Using this trick:
To read a global, check that the keys is the expected keys and read the 
value straight out of the values array at the known index.

To read a builtins, check that the module keys is the expected keys and 
thus cannot shadow the builtins, then read the builtins as above.

I don't know how much help this will be for a static optimiser, but it 
could work well for a dynamic optimiser. I used this optimisation in 
HotPy for optimising object attribute lookups.


Cheers,
Mark.



From victor.stinner at gmail.com  Wed Jan 20 16:22:01 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 20 Jan 2016 22:22:01 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
Message-ID: <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>

I pushed my table, it will be online in a few hours (I don't know when
the devguide is recompiled?):
http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment

By the way, it would be super cool to rebuild the PEPs with a
post-commit hook server-side, rather than having to wait the crontab
which requires to wait ~30 minutes (1h? I don't know exactly).

2016-01-20 21:20 GMT+01:00 Brett Cannon <brett at python.org>:
> It is, which is why I requested the first release date be a column.

I added a Scheduled column with a link to the Release Schedule PEP of
each version. I also added a column with the date of the first date. I
added 5 years to estimate the end-of-line. I used the same month and
same date, with a comment above explaining that the release manager is
free to adujst the end-of-line date.

Thanks for the feedback.

> PEP 101 would need a tweak to remind the RM to update the devguide.

Can someone please mention this table in the PEP

Victor

From v+python at g.nevcal.com  Wed Jan 20 16:18:25 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 20 Jan 2016 13:18:25 -0800
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
 <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
Message-ID: <569FF9A1.8010108@g.nevcal.com>

On 1/20/2016 12:50 PM, Brett Cannon wrote:
>
>     A global (shared between all dicts) unit64 ma_version is actually
>     quite
>     reliable -- if a program does 1,000,000 dict modifications per second,
>     it would take it 600,000 years till wrap-around.
>

But would invalidate everything, instead of just a fraction of things, 
on every update to anything that is monitored...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/2860261e/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Wed Jan 20 17:01:03 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Jan 2016 11:01:03 +1300
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FCBD0.4040307@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
Message-ID: <56A0039F.6070409@canterbury.ac.nz>

Yury Selivanov wrote:
> What I propose is to add a pointer "ma_extra" (same 64bits),
> which will be set to NULL for most dict instances (instead of
> ma_version).  "ma_extra" can then point to a struct that has a
> globally unique dict ID (uint64), and a version tag (unit64).

Why not just use a single global counter for allocating
dict version tags, instead of one per dict?

-- 
Greg

From brett at python.org  Wed Jan 20 17:01:28 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 22:01:28 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
Message-ID: <CAP1=2W53izMTaOxodoXrwi39m7f8=NJUrVH=18F-CkgKafytEg@mail.gmail.com>

On Wed, 20 Jan 2016 at 13:22 Victor Stinner <victor.stinner at gmail.com>
wrote:

> I pushed my table, it will be online in a few hours (I don't know when
> the devguide is recompiled?):
>
> http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment
>
> By the way, it would be super cool to rebuild the PEPs with a
> post-commit hook server-side, rather than having to wait the crontab
> which requires to wait ~30 minutes (1h? I don't know exactly).
>

This is a proposed optional, future feature leading from moving to GitHub:
https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/a2cf42f4/attachment.html>

From francismb at email.de  Wed Jan 20 17:18:12 2016
From: francismb at email.de (francismb)
Date: Wed, 20 Jan 2016 23:18:12 +0100
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <569EA3A6.4010802@email.de>
References: <569EA3A6.4010802@email.de>
Message-ID: <56A007A4.80705@email.de>

Thanks again to all persons that commented so far.

> what's your opinion about a code-formatter bot for cpython.
> Pros, Cons, where could be applicable (new commits, new workflow, it
> doesn't make sense), ...
> 
> 
> - At least it should follow PEP 7 ;-)
> - ...

There seems to be too much implicit information on the lines above.
Sorry for that. I'll try to clarify.

>From my point of view (not a core-dev) the reviews seem to be one
of the bottlenecks to the commit throughput, and I noticed on the
other PEP 7 thread that time is taken to review such, IMHO, (and as
Andrew also noted) trivialities. Thus the basic idea was to get that
noise away from the reviews, somehow (better upfront, but why not
after, accidental PEP 7 noise commits can happen). Well, people
should first agree on that PEP and then some automatic could come.

Please notice that the interaction is not just: core-dev committing
to a repo or a bot committing to a repo but external-contributor that
tries to minimize review iterations. I have no problem with some
process (call it now a bot or script) that just changes a patch/file to
reduce that cycle (but it could, run on all the workflow points you
mentioned, plus on the PR-site as a kind of an advisor/mentor).


Regards,
francis

From victor.stinner at gmail.com  Wed Jan 20 17:28:00 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 20 Jan 2016 23:28:00 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W53izMTaOxodoXrwi39m7f8=NJUrVH=18F-CkgKafytEg@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAP1=2W53izMTaOxodoXrwi39m7f8=NJUrVH=18F-CkgKafytEg@mail.gmail.com>
Message-ID: <CAMpsgwbLBnnVDGXNrQ5CQKZRmWv7Z2uUPLOZ0HEoKxv0DOOv=A@mail.gmail.com>

2016-01-20 23:01 GMT+01:00 Brett Cannon <brett at python.org>:
> This is a proposed optional, future feature leading from moving to GitHub:
> https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content

I'm using the free service ReadTheDocs.org and it's really impressive
how fast it is to update the HTML page after a push. It's usually less
than 10 seconds.

Victor

From guido at python.org  Wed Jan 20 17:31:01 2016
From: guido at python.org (Guido van Rossum)
Date: Wed, 20 Jan 2016 14:31:01 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
Message-ID: <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>

The wording is totally fine! You might still want to revert it and
re-commit it so it doesn't look like a mistake when reviewing the log.

BTW When can we start using git for the peps repo?

On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Wed, 20 Jan 2016 at 10:45 Terry Reedy <tjreedy at udel.edu> wrote:
>
>> On 1/20/2016 12:45 PM, Brett Cannon wrote:
>> >
>> >
>> > On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com
>> > <mailto:vadmium%2Bpy at gmail.com>> wrote:
>> >
>> >     On 19 January 2016 at 20:12, Brett Cannon <brett at python.org
>> >     <mailto:brett at python.org>> wrote:
>> >      > Here is a proposed update:
>> >      >
>> >      > diff -r 633f51d10a67 pep-0007.txt
>> >      > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
>> >      > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
>> >      > @@ -75,9 +75,9 @@
>> >      >        }
>> >      >
>> >      >  * Code structure: one space between keywords like ``if``,
>> >     ``for`` and
>> >      > -  the following left paren; no spaces inside the paren; braces
>> >     may be
>> >      > -  omitted where C permits but when present, they should be
>> formatted
>> >      > -  as shown::
>> >      > +  the following left paren; no spaces inside the paren; braces
>> are
>> >      > +  strongly preferred but may be omitted where C permits, and
>> they
>> >      > +  should be formatted as shown::
>> >      >
>> >      >        if (mro != NULL) {
>> >      >            ...
>> >
>> >     This change seems to be accidentally smuggled in, in the guise of a
>> >     PEP 512 update :)
>> >
>> >
>> > Darn, sorry about that; forgot I had that change sitting in my peps
>> > checkout. I'll revert it when I get home (unless the change is actually
>> > acceptable to Guido).
>>
>> I thought that the above was your intentional compromise change given
>> the range of opinions ;-).
>>
>
> It is, but Guido is the author of PEP 7 and so I didn't want to check in
> the change without his approval first.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/874b6a9f/attachment.html>

From brett at python.org  Wed Jan 20 17:41:16 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 20 Jan 2016 22:41:16 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
Message-ID: <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>

On Wed, 20 Jan 2016 at 14:31 Guido van Rossum <guido at python.org> wrote:

> The wording is totally fine! You might still want to revert it and
> re-commit it so it doesn't look like a mistake when reviewing the log.
>

Sure thing!


>
> BTW When can we start using git for the peps repo?
>

Depends on how fast the parts covered in
https://www.python.org/dev/peps/pep-0512/#requirements-for-code-only-repositories
 and
https://www.python.org/dev/peps/pep-0512/#requirements-for-web-related-repositories
takes.
My hope is before PyCon US (although if we choose to not enforce the CLA on
PEP contributions then it could happen even faster).

-Brett


>
> On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Wed, 20 Jan 2016 at 10:45 Terry Reedy <tjreedy at udel.edu> wrote:
>>
>>> On 1/20/2016 12:45 PM, Brett Cannon wrote:
>>> >
>>> >
>>> > On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com
>>> > <mailto:vadmium%2Bpy at gmail.com>> wrote:
>>> >
>>> >     On 19 January 2016 at 20:12, Brett Cannon <brett at python.org
>>> >     <mailto:brett at python.org>> wrote:
>>> >      > Here is a proposed update:
>>> >      >
>>> >      > diff -r 633f51d10a67 pep-0007.txt
>>> >      > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
>>> >      > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
>>> >      > @@ -75,9 +75,9 @@
>>> >      >        }
>>> >      >
>>> >      >  * Code structure: one space between keywords like ``if``,
>>> >     ``for`` and
>>> >      > -  the following left paren; no spaces inside the paren; braces
>>> >     may be
>>> >      > -  omitted where C permits but when present, they should be
>>> formatted
>>> >      > -  as shown::
>>> >      > +  the following left paren; no spaces inside the paren; braces
>>> are
>>> >      > +  strongly preferred but may be omitted where C permits, and
>>> they
>>> >      > +  should be formatted as shown::
>>> >      >
>>> >      >        if (mro != NULL) {
>>> >      >            ...
>>> >
>>> >     This change seems to be accidentally smuggled in, in the guise of a
>>> >     PEP 512 update :)
>>> >
>>> >
>>> > Darn, sorry about that; forgot I had that change sitting in my peps
>>> > checkout. I'll revert it when I get home (unless the change is actually
>>> > acceptable to Guido).
>>>
>>> I thought that the above was your intentional compromise change given
>>> the range of opinions ;-).
>>>
>>
>> It is, but Guido is the author of PEP 7 and so I didn't want to check in
>> the change without his approval first.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/b13fd417/attachment.html>

From victor.stinner at gmail.com  Wed Jan 20 18:45:50 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 00:45:50 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <569FF9A1.8010108@g.nevcal.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
 <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
 <569FF9A1.8010108@g.nevcal.com>
Message-ID: <CAMpsgwZZtF+fovOM7acBwYRVWzN0jcBRKri7UJYX65X0nX2=Gg@mail.gmail.com>

Hi,

2016-01-20 22:18 GMT+01:00 Glenn Linderman <v+python at g.nevcal.com>:
> On 1/20/2016 12:50 PM, Brett Cannon wrote:
>>
>> A global (shared between all dicts) unit64 ma_version is actually quite
>> reliable -- if a program does 1,000,000 dict modifications per second,
>> it would take it 600,000 years till wrap-around.

I think that Yury found a bug in FAT Python. I didn't test the case
when the builtins dictionary is replaced after the definition of the
function. To be more concrete: when a function is executed in a
different namespace using exec(code, namespace). That's why I like the
PEP process, it helps to find all issues before going too far :-)

I like the idea of global counter for dictionary versions. It means
that the dictionary constructor increases this counter instead of
always starting to 0.

FYI a fat.GuardDict keeps a strong reference to the dictionary. For
some guards, I hesitated to store the object identifier and/or using a
weak reference. An object identifier is not reliable because the
object can be destroyed and a new object, completly different, or of
the same type, can get the same identifier.

> But would invalidate everything, instead of just a fraction of things, on
> every update to anything that is monitored...

I don't understand this point.

In short, the guard only has to compare two 64 bit integers in the
fast-path, when nothing changed. For a namespace, it means that no
value was replaced in this namespace.

If a different namespace is modified, the version of the watched
namespace does not change, so we are still in the fast-path.

If a value is replaced in the watched namespace, but not the watched
variable, we have to take a slow-path, hopefully only once.

The worst case is when a value different than the watched value is
modified between each guard check. In this case, we always need a dict
lookup. An heuristic can be chosen to decide to give up after N tries.
Currently, fat.GuardDict always retries.

Victor

From brett at python.org  Wed Jan 20 19:08:12 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 00:08:12 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwZZtF+fovOM7acBwYRVWzN0jcBRKri7UJYX65X0nX2=Gg@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
 <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
 <569FF9A1.8010108@g.nevcal.com>
 <CAMpsgwZZtF+fovOM7acBwYRVWzN0jcBRKri7UJYX65X0nX2=Gg@mail.gmail.com>
Message-ID: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>

On Wed, 20 Jan 2016 at 15:46 Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> 2016-01-20 22:18 GMT+01:00 Glenn Linderman <v+python at g.nevcal.com>:
> > On 1/20/2016 12:50 PM, Brett Cannon wrote:
> >>
> >> A global (shared between all dicts) unit64 ma_version is actually quite
> >> reliable -- if a program does 1,000,000 dict modifications per second,
> >> it would take it 600,000 years till wrap-around.
>
> I think that Yury found a bug in FAT Python. I didn't test the case
> when the builtins dictionary is replaced after the definition of the
> function. To be more concrete: when a function is executed in a
> different namespace using exec(code, namespace). That's why I like the
> PEP process, it helps to find all issues before going too far :-)
>
> I like the idea of global counter for dictionary versions. It means
> that the dictionary constructor increases this counter instead of
> always starting to 0.
>
> FYI a fat.GuardDict keeps a strong reference to the dictionary. For
> some guards, I hesitated to store the object identifier and/or using a
> weak reference. An object identifier is not reliable because the
> object can be destroyed and a new object, completly different, or of
> the same type, can get the same identifier.
>
> > But would invalidate everything, instead of just a fraction of things, on
> > every update to anything that is monitored...
>
> I don't understand this point.
>

I think Glenn was assuming we had a single, global version # that all dicts
shared *without* having a per-dict version ID. The key thing here is that
we have a global counter that tracks the number of mutations for *all*
dictionaries
but whose value we store as a *per-dictionary* value. That ends up making
the version ID inherently both a token representing the state of any dict
but also the uniqueness of the dict since no two dictionaries will ever
have the same version ID.


>
> In short, the guard only has to compare two 64 bit integers in the
> fast-path, when nothing changed. For a namespace, it means that no
> value was replaced in this namespace.
>
> If a different namespace is modified, the version of the watched
> namespace does not change, so we are still in the fast-path.
>
> If a value is replaced in the watched namespace, but not the watched
> variable, we have to take a slow-path, hopefully only once.
>
> The worst case is when a value different than the watched value is
> modified between each guard check. In this case, we always need a dict
> lookup. An heuristic can be chosen to decide to give up after N tries.
> Currently, fat.GuardDict always retries.
>

Does "retries" mean "check if the value really changed, and if it hasn't
then just update the version ID the guard checks"?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/59b1cbb9/attachment-0001.html>

From brett at python.org  Wed Jan 20 19:09:43 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 00:09:43 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwbLBnnVDGXNrQ5CQKZRmWv7Z2uUPLOZ0HEoKxv0DOOv=A@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAP1=2W53izMTaOxodoXrwi39m7f8=NJUrVH=18F-CkgKafytEg@mail.gmail.com>
 <CAMpsgwbLBnnVDGXNrQ5CQKZRmWv7Z2uUPLOZ0HEoKxv0DOOv=A@mail.gmail.com>
Message-ID: <CAP1=2W6GH5i9Pi2bNWg+bFSLET8UtZ0sZ-5KhXzUQq7N-zvrFw@mail.gmail.com>

On Wed, 20 Jan 2016 at 14:28 Victor Stinner <victor.stinner at gmail.com>
wrote:

> 2016-01-20 23:01 GMT+01:00 Brett Cannon <brett at python.org>:
> > This is a proposed optional, future feature leading from moving to
> GitHub:
> >
> https://www.python.org/dev/peps/pep-0512/#web-hooks-for-re-generating-web-content
>
> I'm using the free service ReadTheDocs.org and it's really impressive
> how fast it is to update the HTML page after a push. It's usually less
> than 10 seconds.
>

I have no idea if the way our docs are built would work on readthedocs.org,
but if it could then I would definitely vote to move our docs there and
have the PSF make a regular donation for the service.

But this is a discussion to have on core-workflow@ and not here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/d0cc76e9/attachment.html>

From v+python at g.nevcal.com  Wed Jan 20 19:10:56 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 20 Jan 2016 16:10:56 -0800
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
 <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
 <569FF9A1.8010108@g.nevcal.com>
 <CAMpsgwZZtF+fovOM7acBwYRVWzN0jcBRKri7UJYX65X0nX2=Gg@mail.gmail.com>
 <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
Message-ID: <56A02210.7030002@g.nevcal.com>

On 1/20/2016 4:08 PM, Brett Cannon wrote:
>
>
> On Wed, 20 Jan 2016 at 15:46 Victor Stinner <victor.stinner at gmail.com 
> <mailto:victor.stinner at gmail.com>> wrote:
>
>     Hi,
>
>     2016-01-20 22:18 GMT+01:00 Glenn Linderman <v+python at g.nevcal.com
>     <mailto:v%2Bpython at g.nevcal.com>>:
>     > On 1/20/2016 12:50 PM, Brett Cannon wrote:
>     >>
>     >> A global (shared between all dicts) unit64 ma_version is
>     actually quite
>     >> reliable -- if a program does 1,000,000 dict modifications per
>     second,
>     >> it would take it 600,000 years till wrap-around.
>
>     I think that Yury found a bug in FAT Python. I didn't test the case
>     when the builtins dictionary is replaced after the definition of the
>     function. To be more concrete: when a function is executed in a
>     different namespace using exec(code, namespace). That's why I like the
>     PEP process, it helps to find all issues before going too far :-)
>
>     I like the idea of global counter for dictionary versions. It means
>     that the dictionary constructor increases this counter instead of
>     always starting to 0.
>
>     FYI a fat.GuardDict keeps a strong reference to the dictionary. For
>     some guards, I hesitated to store the object identifier and/or using a
>     weak reference. An object identifier is not reliable because the
>     object can be destroyed and a new object, completly different, or of
>     the same type, can get the same identifier.
>
>     > But would invalidate everything, instead of just a fraction of
>     things, on
>     > every update to anything that is monitored...
>
>     I don't understand this point.
>
>
> I think Glenn was assuming we had a single, global version # that all 
> dicts shared *without* having a per-dict version ID. The key thing 
> here is that we have a global counter that tracks the number of 
> mutations for *all* dictionaries but whose value we store as a 
> *per-dictionary* value. That ends up making the version ID inherently 
> both a token representing the state of any dict but also the 
> uniqueness of the dict since no two dictionaries will ever have the 
> same version ID.

This would work. You were correct about my assumptions.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160120/2416edb9/attachment.html>

From brett at python.org  Wed Jan 20 19:16:31 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 00:16:31 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
Message-ID: <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>

I just checked with Van and we should have CLAs for even PEP contributors,
so it will have to go through the same steps as the other ancillary
repositories.

On Wed, 20 Jan 2016 at 14:41 Brett Cannon <brett at python.org> wrote:

> On Wed, 20 Jan 2016 at 14:31 Guido van Rossum <guido at python.org> wrote:
>
>> The wording is totally fine! You might still want to revert it and
>> re-commit it so it doesn't look like a mistake when reviewing the log.
>>
>
> Sure thing!
>
>
>>
>> BTW When can we start using git for the peps repo?
>>
>
> Depends on how fast the parts covered in
> https://www.python.org/dev/peps/pep-0512/#requirements-for-code-only-repositories
>  and
> https://www.python.org/dev/peps/pep-0512/#requirements-for-web-related-repositories takes.
> My hope is before PyCon US (although if we choose to not enforce the CLA on
> PEP contributions then it could happen even faster).
>
> -Brett
>
>
>>
>> On Wed, Jan 20, 2016 at 12:18 PM, Brett Cannon <brett at python.org> wrote:
>>
>>>
>>>
>>> On Wed, 20 Jan 2016 at 10:45 Terry Reedy <tjreedy at udel.edu> wrote:
>>>
>>>> On 1/20/2016 12:45 PM, Brett Cannon wrote:
>>>> >
>>>> >
>>>> > On Tue, 19 Jan 2016 at 19:33 Martin Panter <vadmium+py at gmail.com
>>>> > <mailto:vadmium%2Bpy at gmail.com>> wrote:
>>>> >
>>>> >     On 19 January 2016 at 20:12, Brett Cannon <brett at python.org
>>>> >     <mailto:brett at python.org>> wrote:
>>>> >      > Here is a proposed update:
>>>> >      >
>>>> >      > diff -r 633f51d10a67 pep-0007.txt
>>>> >      > --- a/pep-0007.txt Mon Jan 18 10:52:57 2016 -0800
>>>> >      > +++ b/pep-0007.txt Tue Jan 19 12:11:44 2016 -0800
>>>> >      > @@ -75,9 +75,9 @@
>>>> >      >        }
>>>> >      >
>>>> >      >  * Code structure: one space between keywords like ``if``,
>>>> >     ``for`` and
>>>> >      > -  the following left paren; no spaces inside the paren; braces
>>>> >     may be
>>>> >      > -  omitted where C permits but when present, they should be
>>>> formatted
>>>> >      > -  as shown::
>>>> >      > +  the following left paren; no spaces inside the paren;
>>>> braces are
>>>> >      > +  strongly preferred but may be omitted where C permits, and
>>>> they
>>>> >      > +  should be formatted as shown::
>>>> >      >
>>>> >      >        if (mro != NULL) {
>>>> >      >            ...
>>>> >
>>>> >     This change seems to be accidentally smuggled in, in the guise of
>>>> a
>>>> >     PEP 512 update :)
>>>> >
>>>> >
>>>> > Darn, sorry about that; forgot I had that change sitting in my peps
>>>> > checkout. I'll revert it when I get home (unless the change is
>>>> actually
>>>> > acceptable to Guido).
>>>>
>>>> I thought that the above was your intentional compromise change given
>>>> the range of opinions ;-).
>>>>
>>>
>>> It is, but Guido is the author of PEP 7 and so I didn't want to check in
>>> the change without his approval first.
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>>
>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/70a19d29/attachment.html>

From victor.stinner at gmail.com  Wed Jan 20 19:23:44 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 01:23:44 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com>
 <CAP1=2W7-PvDeeDyc5HVKd6zkic2WoTAX496cTH47vLnX9J+VKg@mail.gmail.com>
 <CAK5idxTGiE=Y0R_T0j=c4H=1n_DJ1a35mwyhk=unKZOZo2k4xg@mail.gmail.com>
 <569FE3E6.303@g.nevcal.com> <569FEDA0.1040506@gmail.com>
 <CAP1=2W7+4uqZ0uGaJhSOfi7QQtG4G7Q55apuVWZxk_5NbjpDRg@mail.gmail.com>
 <569FF9A1.8010108@g.nevcal.com>
 <CAMpsgwZZtF+fovOM7acBwYRVWzN0jcBRKri7UJYX65X0nX2=Gg@mail.gmail.com>
 <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
Message-ID: <CAMpsgwYK8_HWc10udZ+JD9SkxWu8_eisDX1iWtvgsyiG2h+JQA@mail.gmail.com>

2016-01-21 1:08 GMT+01:00 Brett Cannon <brett at python.org>:
> On Wed, 20 Jan 2016 at 15:46 Victor Stinner <victor.stinner at gmail.com>
>> The worst case is when a value different than the watched value is
>> modified between each guard check. In this case, we always need a dict
>> lookup. An heuristic can be chosen to decide to give up after N tries.
>> Currently, fat.GuardDict always retries.
>
> Does "retries" mean "check if the value really changed, and if it hasn't
> then just update the version ID the guard checks"?

If the dict version changes (because a value different than the
watched value is modified) each time that the guard is checked, the
guard always require a dict lookup to check if the watched value
changed.

Victor

From victor.stinner at gmail.com  Wed Jan 20 19:33:03 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 01:33:03 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W6GH5i9Pi2bNWg+bFSLET8UtZ0sZ-5KhXzUQq7N-zvrFw@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAP1=2W53izMTaOxodoXrwi39m7f8=NJUrVH=18F-CkgKafytEg@mail.gmail.com>
 <CAMpsgwbLBnnVDGXNrQ5CQKZRmWv7Z2uUPLOZ0HEoKxv0DOOv=A@mail.gmail.com>
 <CAP1=2W6GH5i9Pi2bNWg+bFSLET8UtZ0sZ-5KhXzUQq7N-zvrFw@mail.gmail.com>
Message-ID: <CAMpsgwaocjheJ5+n7+9CyXnCZZgjWh5CWyoFDP+WY8086WKOkA@mail.gmail.com>

2016-01-21 1:09 GMT+01:00 Brett Cannon <brett at python.org>:
> On Wed, 20 Jan 2016 at 14:28 Victor Stinner <victor.stinner at gmail.com>
>> I'm using the free service ReadTheDocs.org and it's really impressive
>> how fast it is to update the HTML page after a push. It's usually less
>> than 10 seconds.
>
> I have no idea if the way our docs are built would work on readthedocs.org,
> but if it could then I would definitely vote to move our docs there and have
> the PSF make a regular donation for the service.

Oh, I was talking about small documentations of personal projects. I
didn't propose to move Python docs to readthedocs.org. I don't know if
it makes sense.

It's just to say that we can do better than 30 minutes of the current system :-)

Victor

From abarnert at yahoo.com  Wed Jan 20 20:54:41 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Thu, 21 Jan 2016 01:54:41 +0000 (UTC)
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
References: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
Message-ID: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>

On Wednesday, January 20, 2016 4:10 PM, Brett Cannon <brett at python.org> wrote:


>I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries but whose value we store as a per-dictionary value. That ends up making the version ID inherently both a token representing the state of any dict but also the uniqueness of the dict since no two dictionaries will ever have the same version ID.

This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on _any_ dict. So, imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, so you end up with 8 cores each running at 1/64th the speed of a single core but burning 100% CPU. Obviously a real-life implementation wouldn't be _that_ stupid; you'd special-case the version-bumping (maybe unconditionally bump it N times before starting the transaction, and then as long as you don't bump more than N times during the transaction, you can commit without touching it), but there's still going to be a lot of contention.

And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext. At first glance that sounds like a stupid idea--why would you want to run an optimizer through a slow emulator? But the optimizer only runs once and transforms the function code, which runs a zillion times, so who cares how slow the optimizer is? Of course it may still be true that many of the AoT optimizations that FAT makes don't apply very well to PyPy, in which case it doesn't matter. But I don't think we can assume that a priori.

Is there a way to define this loosely enough so that the implementation _can_ be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary?

From brett at python.org  Wed Jan 20 21:04:58 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 02:04:58 +0000
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
References: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
 <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CAP1=2W7Og1Pmdo7ZH2i9D9uPTkpMC0Ea0ZnwckpGOUKZKM-hNA@mail.gmail.com>

On Wed, 20 Jan 2016, 17:54 Andrew Barnert <abarnert at yahoo.com> wrote:

> On Wednesday, January 20, 2016 4:10 PM, Brett Cannon <brett at python.org>
> wrote:
>
>
> >I think Glenn was assuming we had a single, global version # that all
> dicts shared without having a per-dict version ID. The key thing here is
> that we have a global counter that tracks the number of mutations for all
> dictionaries but whose value we store as a per-dictionary value. That ends
> up making the version ID inherently both a token representing the state of
> any dict but also the uniqueness of the dict since no two dictionaries will
> ever have the same version ID.
>
> This idea worries me. I'm not sure why, but I think because of threading.
> After all, it's pretty rare for two threads to both want to work on the
> same dict, but very, very common for two threads to both want to work on
> _any_ dict. So, imagine someone manages to remove the GIL from CPython by
> using STM: now most transactions are bumping that global counter, meaning
> most transactions fail and have to be retried, so you end up with 8 cores
> each running at 1/64th the speed of a single core but burning 100% CPU.
> Obviously a real-life implementation wouldn't be _that_ stupid; you'd
> special-case the version-bumping (maybe unconditionally bump it N times
> before starting the transaction, and then as long as you don't bump more
> than N times during the transaction, you can commit without touching it),
> but there's still going to be a lot of contention.
>

This is all being regarded as an implementation detail of CPython, so in
this hypothetical STM world we can drop all of this (or lock it).


> And that also affects something like PyPy being able to use
> FAT-Python-style AoT optimizations via cpyext. At first glance that sounds
> like a stupid idea--why would you want to run an optimizer through a slow
> emulator? But the optimizer only runs once and transforms the function
> code, which runs a zillion times, so who cares how slow the optimizer is?
> Of course it may still be true that many of the AoT optimizations that FAT
> makes don't apply very well to PyPy, in which case it doesn't matter. But I
> don't think we can assume that a priori.
>
> Is there a way to define this loosely enough so that the implementation
> _can_ be a single global counter, if that turns out to be most efficient,
> but can also be a counter per dictionary and a globally-unique ID per
> dictionary?
>

There's no need to if this is all under the hood and in no way affects
anyone but the eval loop and those who choose to use it. We can make sure
to preface all of this with underscores so it's obvious they are private
and so use at your own peril.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/fa7b1fd7/attachment-0001.html>

From yselivanov.ml at gmail.com  Wed Jan 20 21:13:38 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 20 Jan 2016 21:13:38 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
References: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
 <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <56A03ED2.3010204@gmail.com>



On 2016-01-20 8:54 PM, Andrew Barnert via Python-Dev wrote:
>> >I think Glenn was assuming we had a single, global version # that all dicts shared without having a per-dict version ID. The key thing here is that we have a global counter that tracks the number of mutations for all dictionaries but whose value we store as a per-dictionary value. That ends up making the version ID inherently both a token representing the state of any dict but also the uniqueness of the dict since no two dictionaries will ever have the same version ID.
> This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on_any_  dict. So, imagine someone manages to remove the GIL from CPython by using STM: now most transactions are bumping that global counter, meaning most transactions fail and have to be retried, so you end up with 8 cores each running at 1/64th the speed of a single core but burning 100% CPU. Obviously a real-life implementation wouldn't be_that_  stupid; you'd special-case the version-bumping (maybe unconditionally bump it N times before starting the transaction, and then as long as you don't bump more than N times during the transaction, you can commit without touching it), but there's still going to be a lot of contention.

Well, PEP 509 proposes to add ma_version only for CPython.  It's an 
implementation detail of CPython.  Victor's FAT optimizer is also 
tailored specifically for CPython, and making it work on PyPy would 
require a completely different set of hacks.

To remove the GIL or implement an efficient STM one will have to rewrite 
(and potentially break) so much code in CPython, that ma_version won't 
be a concern.

For now, though, ma_version will be protected by GIL, so threading 
shouldn't be a problem.

>
> And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext. At first glance that sounds like a stupid idea--why would you want to run an optimizer through a slow emulator? But the optimizer only runs once and transforms the function code, which runs a zillion times, so who cares how slow the optimizer is? Of course it may still be true that many of the AoT optimizations that FAT makes don't apply very well to PyPy, in which case it doesn't matter. But I don't think we can assume that a priori.

The idea of FAT is that it will also generate optimized code objects 
with guards.  I doubt it would make any sense to use it under PyPy or 
any jitted Python implementation.  JITs have a far better understanding 
of the code than any static optimizer.

>
> Is there a way to define this loosely enough so that the implementation_can_  be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary?

Defining it "loosely" means that you can't trust it.  I'd just 
explicitly say that:

- ma_version is an implementation detail of CPython and may not be 
implemented on other platforms;
- ma_version can be removed from future CPython releases;
- ma_version can be used by code optimizers tailored specifically for 
CPython and CPython itself.

Yury

From yselivanov.ml at gmail.com  Thu Jan 21 00:08:57 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Thu, 21 Jan 2016 00:08:57 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <56A0039F.6070409@canterbury.ac.nz>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz>
Message-ID: <56A067E9.8090409@gmail.com>



On 2016-01-20 5:01 PM, Greg Ewing wrote:
> Yury Selivanov wrote:
>> What I propose is to add a pointer "ma_extra" (same 64bits),
>> which will be set to NULL for most dict instances (instead of
>> ma_version).  "ma_extra" can then point to a struct that has a
>> globally unique dict ID (uint64), and a version tag (unit64).
>
> Why not just use a single global counter for allocating
> dict version tags, instead of one per dict?
>


Yeah, I think that's what we agreed on: 
https://mail.python.org/pipermail/python-dev/2016-January/142837.html

The only advantage of ma_extra pointer is that it allows to add more 
stuff to dicts in the future.

Yury

From ncoghlan at gmail.com  Thu Jan 21 02:07:43 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 21 Jan 2016 17:07:43 +1000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
 <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
Message-ID: <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>

On 21 January 2016 at 10:16, Brett Cannon <brett at python.org> wrote:
> I just checked with Van and we should have CLAs for even PEP contributors,
> so it will have to go through the same steps as the other ancillary
> repositories.

We should probably mention that in PEP 1 - I wasn't aware of that
requirement, so I've never explicitly checked CLA status for folks
contributing packaging related PEPs. (And looking at the
just-checked-in PEP 513, I apparently have a CLA to chase up...)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Thu Jan 21 02:46:21 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 08:46:21 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
Message-ID: <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>

2016-01-20 22:22 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> I pushed my table, it will be online in a few hours (I don't know when
> the devguide is recompiled?):
> http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment

Hum ok, it takes more than a few hours in fact. It's still not online
10 hours after my push :-/
https://docs.python.org/devguide/

Victor

From greg.ewing at canterbury.ac.nz  Thu Jan 21 00:20:42 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 21 Jan 2016 18:20:42 +1300
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
References: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
 <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <56A06AAA.7020507@canterbury.ac.nz>

Andrew Barnert via Python-Dev wrote:
> imagine someone manages to remove the GIL from CPython by using
> STM: now most transactions are bumping that global counter, meaning most
> transactions fail and have to be retried,

If this becomes a problem, the tag could be split into two
parts of m and n bits, with m + n = 64. Use a global counter
for allocating the high half, and increment the low half
locally. When the low half overflows, allocate a new high
half.

A value of n = 16 or so ought to reduce contention for the
global counter to something fairly negligible, I would
think, without much risk of the high half ever wrapping
around.

-- 
Greg

From victor.stinner at gmail.com  Thu Jan 21 03:28:25 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 09:28:25 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
References: <CAP1=2W6AZTsnbYJsr-gkPK_VXMLNiNGKYDsicCDhoz6r+d7P+g@mail.gmail.com>
 <126377093.6733723.1453341281211.JavaMail.yahoo@mail.yahoo.com>
Message-ID: <CAMpsgwazshBD+NnCiFPWOiQDj4L0xzx16KBb1T4NFgj3YJ3ZnQ@mail.gmail.com>

2016-01-21 2:54 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
> This idea worries me. I'm not sure why, but I think because of threading. After all, it's pretty rare for two threads to both want to work on the same dict, but very, very common for two threads to both want to work on _any_ dict. So, imagine someone manages to remove the GIL from CPython by using STM: (...)

That's a huge project :-) PyPy works one this, but PyPy is not CPython.

> (...) now most transactions are bumping that global counter, meaning most transactions fail and have to be retried,

Python has atomic types which work well with multiple concurrent threads.

> And that also affects something like PyPy being able to use FAT-Python-style AoT optimizations via cpyext.

I don't think that using a static optimizer with PyPy  makes sense.
Reminder that a dynamic optimize (JIT compiler) beats a static
compiler on performance ;-)

PyPy has a very different design, it's a very bad idea to implement
optimizations on cpyext which is emulated and known to be slow.

> Is there a way to define this loosely enough so that the implementation _can_ be a single global counter, if that turns out to be most efficient, but can also be a counter per dictionary and a globally-unique ID per dictionary?

I don't see how a single global counter for all dictionary can be used
to implement fast guards on namespaces. See the rationale of the PEP
509: I wrote it to implement fast guards on namespaces.

Victor

From victor.stinner at gmail.com  Thu Jan 21 03:34:07 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 21 Jan 2016 09:34:07 +0100
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <56A067E9.8090409@gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz>
 <56A067E9.8090409@gmail.com>
Message-ID: <CAMpsgwbVkjXXpJyXEMccWukrjY76pOztU0Xhgbnpj72U0ZNRUA@mail.gmail.com>

2016-01-21 6:08 GMT+01:00 Yury Selivanov <yselivanov.ml at gmail.com>:
> Yeah, I think that's what we agreed on:
> https://mail.python.org/pipermail/python-dev/2016-January/142837.html
>
> The only advantage of ma_extra pointer is that it allows to add more stuff
> to dicts in the future.

I don't agree on ma_extra since I don't understand it :-) What is the
advantage compared to a simple integer? If it's a pointer, it requires
to dereference the pointer? You say that it can be NULL, does it mean
that we also have to first test if the pointer is NULL. Does it mean
that we have to allocate a second memory block to store a version tag
object? When do you create a version tag or not?

Note: The PEP 509 proposes to use 64-bit integer for the version on
32-bit systems to avoid integer overflow after a few seconds. I first
proposed to use the size_t type (which has the size of a pointer) but
it doesn't work.

I tried to fix FAT Python to handle your use case: function defined in
a namespace and run in a different namespace (especially for the
builtin dictionary). It looks like I don't need the discussion change
to use a global version, FAT Python guards have a different design
than your cache.

But if a global counter doesn't make the slow more complex and opens
new kinds of optimization, I agree to change my PEP 509.

Victor

From yselivanov.ml at gmail.com  Thu Jan 21 11:12:16 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Thu, 21 Jan 2016 11:12:16 -0500
Subject: [Python-Dev] PEP 509: Add a private version to dict
In-Reply-To: <CAMpsgwbVkjXXpJyXEMccWukrjY76pOztU0Xhgbnpj72U0ZNRUA@mail.gmail.com>
References: <CAMpsgwYe=cwR5m8N+aYnW33+ETXEVOu8TxOjTb5Z5PvJCvyanw@mail.gmail.com>
 <CAMpsgwZgeaMQJgp06cS+X1dTCeMA2=S4uwkOMZQeaOnejaWxQg@mail.gmail.com>
 <569FCBD0.4040307@gmail.com> <56A0039F.6070409@canterbury.ac.nz>
 <56A067E9.8090409@gmail.com>
 <CAMpsgwbVkjXXpJyXEMccWukrjY76pOztU0Xhgbnpj72U0ZNRUA@mail.gmail.com>
Message-ID: <56A10360.1060604@gmail.com>



On 2016-01-21 3:34 AM, Victor Stinner wrote:
[..]
> But if a global counter doesn't make the slow more complex and opens
> new kinds of optimization, I agree to change my PEP 509.

Please.  That would allow us to use ma_version to
implement caches in CPython itself.


Yury

From brett at python.org  Thu Jan 21 12:12:10 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 17:12:10 +0000
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
 <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
 <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>
Message-ID: <CAP1=2W4F6RoO1GVBsRW_7vnbCHx8sKhpNgQ9KZa8+SQc7ucbPA@mail.gmail.com>

On Wed, 20 Jan 2016 at 23:07 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 January 2016 at 10:16, Brett Cannon <brett at python.org> wrote:
> > I just checked with Van and we should have CLAs for even PEP
> contributors,
> > so it will have to go through the same steps as the other ancillary
> > repositories.
>
> We should probably mention that in PEP 1


Yep.


> - I wasn't aware of that
> requirement, so I've never explicitly checked CLA status for folks
> contributing packaging related PEPs. (And looking at the
> just-checked-in PEP 513, I apparently have a CLA to chase up...)
>

Yeah, I didn't know either until I asked Van, so I think it's new to
everyone. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/7d4ce71e/attachment.html>

From brett at python.org  Thu Jan 21 12:18:13 2016
From: brett at python.org (Brett Cannon)
Date: Thu, 21 Jan 2016 17:18:13 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
Message-ID: <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>

It's live: https://docs.python.org/devguide/#status-of-python-branches

On Wed, 20 Jan 2016 at 23:47 Victor Stinner <victor.stinner at gmail.com>
wrote:

> 2016-01-20 22:22 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
> > I pushed my table, it will be online in a few hours (I don't know when
> > the devguide is recompiled?):
> >
> http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment
>
> Hum ok, it takes more than a few hours in fact. It's still not online
> 10 hours after my push :-/
> https://docs.python.org/devguide/
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/b8af1524/attachment.html>

From zachary.ware+pydev at gmail.com  Thu Jan 21 12:18:13 2016
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 21 Jan 2016 11:18:13 -0600
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAP1=2W4F6RoO1GVBsRW_7vnbCHx8sKhpNgQ9KZa8+SQc7ucbPA@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
 <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
 <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>
 <CAP1=2W4F6RoO1GVBsRW_7vnbCHx8sKhpNgQ9KZa8+SQc7ucbPA@mail.gmail.com>
Message-ID: <CAKJDb-PqF2HmDLW0AcwxhuG3d5ycPnjTX0fYC0FKDO71AuUQWA@mail.gmail.com>

On Thu, Jan 21, 2016 at 11:12 AM, Brett Cannon <brett at python.org> wrote:
> On Wed, 20 Jan 2016 at 23:07 Nick Coghlan <ncoghlan at gmail.com> wrote:
>> - I wasn't aware of that
>> requirement, so I've never explicitly checked CLA status for folks
>> contributing packaging related PEPs. (And looking at the
>> just-checked-in PEP 513, I apparently have a CLA to chase up...)
>
> Yeah, I didn't know either until I asked Van, so I think it's new to
> everyone. :)

It's quite surprising to me, since all (as far as I know) PEPs are
explicitly public domain.  I am very much not a lawyer, though :)

-- 
Zach

From guido at python.org  Thu Jan 21 12:40:34 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 21 Jan 2016 09:40:34 -0800
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
Message-ID: <CAP7+vJ+cn++cQdBf1MEBUCR2hL+jwefTJ_hZyikV=7CD=q5SvA@mail.gmail.com>

Thanks Victor for doing this. I'm starting a campaign to tell people about
it on Twitter. :-)

On Thu, Jan 21, 2016 at 9:18 AM, Brett Cannon <brett at python.org> wrote:

> It's live: https://docs.python.org/devguide/#status-of-python-branches
>
> On Wed, 20 Jan 2016 at 23:47 Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> 2016-01-20 22:22 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:
>> > I pushed my table, it will be online in a few hours (I don't know when
>> > the devguide is recompiled?):
>> >
>> http://docs.python.org/devguide/triaging.html#generating-special-links-in-a-comment
>>
>> Hum ok, it takes more than a few hours in fact. It's still not online
>> 10 hours after my push :-/
>> https://docs.python.org/devguide/
>>
>> Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/92ca4804/attachment.html>

From p.f.moore at gmail.com  Thu Jan 21 13:42:40 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 21 Jan 2016 18:42:40 +0000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
Message-ID: <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>

On 21 January 2016 at 17:18, Brett Cannon <brett at python.org> wrote:
> It's live: https://docs.python.org/devguide/#status-of-python-branches

Nice :-)

Minor nit, the status column says "end of life", but the text below
the table uses the term "end of line" (as does the comment "Versions
older than 2.6 reached their end-of-line". From my experience, "end of
life" is the more common term.

Paul

From steve.dower at python.org  Thu Jan 21 14:50:37 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 21 Jan 2016 11:50:37 -0800
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAKJDb-PqF2HmDLW0AcwxhuG3d5ycPnjTX0fYC0FKDO71AuUQWA@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
 <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
 <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>
 <CAP1=2W4F6RoO1GVBsRW_7vnbCHx8sKhpNgQ9KZa8+SQc7ucbPA@mail.gmail.com>
 <CAKJDb-PqF2HmDLW0AcwxhuG3d5ycPnjTX0fYC0FKDO71AuUQWA@mail.gmail.com>
Message-ID: <E1aMLG9-00076O-RF@se2-syd.hostedmail.net.au>

I'm still yet to meet a lawyer who trusts "public domain" statements...

The CLA will ensure we have enough rights to republish the PEP on p.o or future sites, and it doesn't prevent authors from also releasing the text elsewhere under other terms.

Top-posted from my Windows Phone

-----Original Message-----
From: "Zachary Ware" <zachary.ware+pydev at gmail.com>
Sent: ?1/?21/?2016 9:23
To: "Python-Dev" <python-dev at python.org>
Subject: Re: [Python-Dev] Update PEP 7 to require curly braces in C

On Thu, Jan 21, 2016 at 11:12 AM, Brett Cannon <brett at python.org> wrote:
> On Wed, 20 Jan 2016 at 23:07 Nick Coghlan <ncoghlan at gmail.com> wrote:
>> - I wasn't aware of that
>> requirement, so I've never explicitly checked CLA status for folks
>> contributing packaging related PEPs. (And looking at the
>> just-checked-in PEP 513, I apparently have a CLA to chase up...)
>
> Yeah, I didn't know either until I asked Van, so I think it's new to
> everyone. :)

It's quite surprising to me, since all (as far as I know) PEPs are
explicitly public domain.  I am very much not a lawyer, though :)

-- 
Zach
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160121/1ac42051/attachment.html>

From emile at fenx.com  Thu Jan 21 16:27:44 2016
From: emile at fenx.com (Emile van Sebille)
Date: Thu, 21 Jan 2016 13:27:44 -0800
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
 <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
Message-ID: <n7rih8$v99$1@ger.gmane.org>

On 1/21/2016 10:42 AM, Paul Moore wrote:
> On 21 January 2016 at 17:18, Brett Cannon <brett at python.org> wrote:
>> It's live: https://docs.python.org/devguide/#status-of-python-branches
>
> Nice :-)
>
> Minor nit, the status column says "end of life", but the text below
> the table uses the term "end of line" (as does the comment "Versions
> older than 2.6 reached their end-of-line". From my experience, "end of
> life" is the more common term.

I'd prefer end-of-support -- bet you can't count how many pre 2.5 
installations are still live.

Emile



From stephen at xemacs.org  Fri Jan 22 00:25:38 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Jan 2016 14:25:38 +0900
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <n7rih8$v99$1@ger.gmane.org>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
 <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
 <n7rih8$v99$1@ger.gmane.org>
Message-ID: <22177.48466.360573.863828@turnbull.sk.tsukuba.ac.jp>

Emile van Sebille writes:

 > I'd prefer end-of-support -- bet you can't count how many pre 2.5 
 > installations are still live.

I see your point, but (having just been thinking about CLAs and
Schneier's blog) have to suggest that software that has an explicit
"security support" period and is in use after that ends isn't "live".

It is "undead".

Hope-the-admin-is-named-Alice-ly y'rs,



From stephen at xemacs.org  Fri Jan 22 00:32:40 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 22 Jan 2016 14:32:40 +0900
Subject: [Python-Dev] Update PEP 7 to require curly braces in C
In-Reply-To: <CAKJDb-PqF2HmDLW0AcwxhuG3d5ycPnjTX0fYC0FKDO71AuUQWA@mail.gmail.com>
References: <569e6ac9.0611370a.41841.ffffd93b@mx.google.com>
 <569E8073.3010106@stoneleaf.us>
 <CAP7+vJJyGsSYOmbGNQyvQC=zF8bsb24fe7_-k13=zBZoYvLD=w@mail.gmail.com>
 <CAP1=2W6Nn9CtPe=c9HF=oYS-yjNA2AiqjxC7SbZ78mA-6NigXQ@mail.gmail.com>
 <CA+eR4cHYVk4Jf2HCZr+2NTaxz1-aoVCjriO_swQSedj7sOmg9w@mail.gmail.com>
 <CAP1=2W6Nq0CNSrkH4WGvT2qpdS0vzVjaQ-R9-Y5N5V7Qfv=pXg@mail.gmail.com>
 <n7okeu$9ol$2@ger.gmane.org>
 <CAP1=2W4tu5POFMxE-CAZKKMxZWc=rwH5R=Q0ZYUaK-cUkyE1Qw@mail.gmail.com>
 <CAP7+vJKW5uNEy-WUBMdfxZSA6W=+iEM4w_-6pMpuhaqMz7ZPAQ@mail.gmail.com>
 <CAP1=2W7+UafFxVBVFaKLMTGrieLdCpS8S7z8hJrJn1SF9HTyKg@mail.gmail.com>
 <CAP1=2W744tYr7u96PLE1=bcB3hjHie1iqDy3ENpdsfrEhx_OJg@mail.gmail.com>
 <CADiSq7dnuhieSYvOgFSHAO1AVdnyDe+xgtNUgkj9_hCipXMmtw@mail.gmail.com>
 <CAP1=2W4F6RoO1GVBsRW_7vnbCHx8sKhpNgQ9KZa8+SQc7ucbPA@mail.gmail.com>
 <CAKJDb-PqF2HmDLW0AcwxhuG3d5ycPnjTX0fYC0FKDO71AuUQWA@mail.gmail.com>
Message-ID: <22177.48888.899524.150195@turnbull.sk.tsukuba.ac.jp>

Zachary Ware writes:

 > It's quite surprising to me, since all (as far as I know) PEPs are
 > explicitly public domain.  I am very much not a lawyer, though :)

The reason for any explicit CLA is CYA: you can't be sure that the
contributor DTRTs, so the CLA shows that the contribution was received
in good faith.  (Some CLAs also include indemnification clauses, such
as the contributor warrants that they own the copyright, etc, but all
CLAs are useful to show good faith, which may matter big time if the
contributor turns out to have a all-IP-yours-is-ours employment
contract or the like.)

You don't really need to be a lawyer to understand this stuff, you
just need to think like a security person.
https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html

From ncoghlan at gmail.com  Fri Jan 22 02:54:05 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 22 Jan 2016 17:54:05 +1000
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <n7rih8$v99$1@ger.gmane.org>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
 <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
 <n7rih8$v99$1@ger.gmane.org>
Message-ID: <CADiSq7eHEP9hP=kO_o+ve0tzZSREF4PKAifv78rPNNA24XkoJw@mail.gmail.com>

On 22 January 2016 at 07:27, Emile van Sebille <emile at fenx.com> wrote:
> On 1/21/2016 10:42 AM, Paul Moore wrote:
>>
>> On 21 January 2016 at 17:18, Brett Cannon <brett at python.org> wrote:
>>>
>>> It's live: https://docs.python.org/devguide/#status-of-python-branches
>>
>> Nice :-)
>>
>> Minor nit, the status column says "end of life", but the text below
>> the table uses the term "end of line" (as does the comment "Versions
>> older than 2.6 reached their end-of-line". From my experience, "end of
>> life" is the more common term.
>
> I'd prefer end-of-support -- bet you can't count how many pre 2.5
> installations are still live.

I can count the number of folks contributing changes to the upstream
Python 2.5 branch: zero. Even if somebody offered a patch for it, we
wouldn't accept it - that maintenance branch is dead, which is what
the "End of Life" refers to.

Folks are still free to run it (all past Python releases remain
online, all the way back to 1.1), and downstreams may still offer
support for it, but that's their call.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Fri Jan 22 03:26:59 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 22 Jan 2016 09:26:59 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
 <CACac1F-EsG8y5y1zooMgk-_2FRUxuv4EgXW6-ioHEe5n_f6vig@mail.gmail.com>
Message-ID: <CAMpsgwYhzZSmS2ja8OzH7fXF16O=9XUHbLiTQp2=MZVp1CYhkQ@mail.gmail.com>

2016-01-21 19:42 GMT+01:00 Paul Moore <p.f.moore at gmail.com>:
> Minor nit, the status column says "end of life", but the text below
> the table uses the term "end of line"

Ooops, it's a funny typo. Fixed. Thanks for the report!

Victor

From victor.stinner at gmail.com  Fri Jan 22 11:03:39 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 22 Jan 2016 17:03:39 +0100
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
Message-ID: <CAMpsgwZQ0cvctVO7EQFQrAU52-bcROB+MzsE_L5aAa+mLUzxeQ@mail.gmail.com>

2016-01-21 18:18 GMT+01:00 Brett Cannon <brett at python.org>:
> It's live: https://docs.python.org/devguide/#status-of-python-branches

There is a very strange bug in this website.

This URL shows the table:
https://docs.python.org/devguide/

This URL doesn't show the table:
https://docs.python.org/devguide/index.html

Outdated version of the guide?

This bug can be seen without a browser, using wget:

$ wget  -O- https://docs.python.org/devguide/ 2>&1|grep 'Python branches'
<span id="branchstatus"></span><h2>Status of Python branches<(...)
<li><a class="reference internal"
href="#status-of-python-branches">Status of Python branches</a></li>

$ wget  -O- https://docs.python.org/devguide/index.html 2>&1|grep
'Python branches'
<not found, empty output>

Victor

From berker.peksag at gmail.com  Fri Jan 22 11:21:48 2016
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Fri, 22 Jan 2016 18:21:48 +0200
Subject: [Python-Dev] Devguide: Add a table summarizing status of Python
 branches
In-Reply-To: <CAMpsgwZQ0cvctVO7EQFQrAU52-bcROB+MzsE_L5aAa+mLUzxeQ@mail.gmail.com>
References: <CAMpsgwbbc=U-VchGns7py9O3SOEpsZ2mV+TfLdXRynDx9-FO0Q@mail.gmail.com>
 <n7okad$9ol$1@ger.gmane.org>
 <CAP1=2W51i6BwcNKA4=U9Q0Vxx=U1xp7vws9zvjTwKrOMWx2EMA@mail.gmail.com>
 <CAMpsgwaM2j5ZSNq0g7rExvufPJGBOUDyy_TOb0kUnjsfBxMqFA@mail.gmail.com>
 <CAMpsgwak2-CxS=pjG612Aduppku_A1vEa6wfq-bpFmYDtC-tDw@mail.gmail.com>
 <CAP1=2W4dBrsoWF_F20zVbh6GZZGS_JcpQb=XrEmHbUKLV0J1ow@mail.gmail.com>
 <CAMpsgwZQ0cvctVO7EQFQrAU52-bcROB+MzsE_L5aAa+mLUzxeQ@mail.gmail.com>
Message-ID: <CAF4280KwoheAsg5+1x-Qw0uo=g5-oR6w2NzxX2PGu_yt9npBEw@mail.gmail.com>

On Fri, Jan 22, 2016 at 6:03 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2016-01-21 18:18 GMT+01:00 Brett Cannon <brett at python.org>:
>> It's live: https://docs.python.org/devguide/#status-of-python-branches
>
> There is a very strange bug in this website.
>
> This URL shows the table:
> https://docs.python.org/devguide/
>
> This URL doesn't show the table:
> https://docs.python.org/devguide/index.html
>
> Outdated version of the guide?

It looks like a cache issue. I purged the cache for /devguide/index.html:

$ wget -O- https://docs.python.org/devguide/index.html 2>&1 | grep
"Python branches"
<span id="branchstatus"></span><h2>Status of Python branches<a
class="headerlink" href="#status-of-python-branches" title="Permalink
to this headline">?</a></h2>
<li><a class="reference internal"
href="#status-of-python-branches">Status of Python branches</a></li>

--Berker

From status at bugs.python.org  Fri Jan 22 12:08:33 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 22 Jan 2016 18:08:33 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160122170833.EF922560CE@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-01-15 - 2016-01-22)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5354 (-33)
  closed 32585 (+89)
  total  37939 (+56)

Open issues with patches: 2348 


Issues opened (31)
==================

#25791: Raise an ImportWarning when __spec__.parent/__package__ isn't 
http://bugs.python.org/issue25791  reopened by brett.cannon

#26128: Let the subprocess.STARTUPINFO constructor take arguments
http://bugs.python.org/issue26128  opened by cool-RR

#26130: redundant local copy of a char pointer in classify in Parser\p
http://bugs.python.org/issue26130  opened by Oren Milman

#26131: Raise ImportWarning when loader.load_module() is used
http://bugs.python.org/issue26131  opened by brett.cannon

#26132: 2.7.11 Windows Installer issues on Win2008R2
http://bugs.python.org/issue26132  opened by David Rader

#26133: asyncio: ugly error related to signal handlers at exit if the 
http://bugs.python.org/issue26133  opened by Alex Brandt

#26134: HTTPPasswordMgrWithPriorAuth does not work with DigestAuthenti
http://bugs.python.org/issue26134  opened by guesommer

#26136: DeprecationWarning for PEP 479 (generator_stop)
http://bugs.python.org/issue26136  opened by martin.panter

#26137: [idea] use the Microsoft Antimalware Scan Interface
http://bugs.python.org/issue26137  opened by Alexander Riccio

#26140: inspect.iscoroutinefunction raises TypeError when checks Mock 
http://bugs.python.org/issue26140  opened by miyakogi

#26141: typing module documentation incomplete
http://bugs.python.org/issue26141  opened by Ben.Darnell

#26143: Ensure that IDLE's stdlib imports are from the stdlib
http://bugs.python.org/issue26143  opened by terry.reedy

#26144: test_pkg test_4 and/or test_7 sometimes fail
http://bugs.python.org/issue26144  opened by martin.panter

#26145: PEP 511: Add sys.set_code_transformers()
http://bugs.python.org/issue26145  opened by haypo

#26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const
http://bugs.python.org/issue26146  opened by haypo

#26148: String literals are not interned if in a tuple
http://bugs.python.org/issue26148  opened by serhiy.storchaka

#26149: Suggest PyCharm Community as an editor for Unix platforms
http://bugs.python.org/issue26149  opened by John Hagen

#26152: A non-breaking space in a source
http://bugs.python.org/issue26152  opened by Drekin

#26153: PyImport_GetModuleDict: no module dictionary! when `__del__` t
http://bugs.python.org/issue26153  opened by minrk

#26155: 3.5.1 installer issue on Win 7 32 bit
http://bugs.python.org/issue26155  opened by TarotRedhand

#26158: File truncate() not defaulting to current position as document
http://bugs.python.org/issue26158  opened by fornax

#26159: Unsafe to BaseEventLoop.set_debug(False) when PYTHONASYNCIODEB
http://bugs.python.org/issue26159  opened by Bradley McLean

#26160: Tutorial incorrectly claims that (explicit) relative imports d
http://bugs.python.org/issue26160  opened by Kevin.Norris

#26167: Improve copy.copy speed for built-in types (list/set/dict)
http://bugs.python.org/issue26167  opened by josh.r

#26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure
http://bugs.python.org/issue26168  opened by squidevil

#26173: test_ssl.bad_cert_test() exception handling
http://bugs.python.org/issue26173  opened by martin.panter

#26175: Fully implement IOBase abstract on SpooledTemporaryFile
http://bugs.python.org/issue26175  opened by Gary Fernie

#26176: EmailMessage example doesn't work
http://bugs.python.org/issue26176  opened by Srujan Chaitanya

#26177: tkinter: Canvas().keys returns empty strings.
http://bugs.python.org/issue26177  opened by terry.reedy

#26180: multiprocessing.util._afterfork_registry leak in threaded envi
http://bugs.python.org/issue26180  opened by mzamazal

#26181: argparse can't handle positional argument after list (help mes
http://bugs.python.org/issue26181  opened by atpage



Most recent 15 issues with no replies (15)
==========================================

#26181: argparse can't handle positional argument after list (help mes
http://bugs.python.org/issue26181

#26180: multiprocessing.util._afterfork_registry leak in threaded envi
http://bugs.python.org/issue26180

#26176: EmailMessage example doesn't work
http://bugs.python.org/issue26176

#26173: test_ssl.bad_cert_test() exception handling
http://bugs.python.org/issue26173

#26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure
http://bugs.python.org/issue26168

#26159: Unsafe to BaseEventLoop.set_debug(False) when PYTHONASYNCIODEB
http://bugs.python.org/issue26159

#26153: PyImport_GetModuleDict: no module dictionary! when `__del__` t
http://bugs.python.org/issue26153

#26141: typing module documentation incomplete
http://bugs.python.org/issue26141

#26136: DeprecationWarning for PEP 479 (generator_stop)
http://bugs.python.org/issue26136

#26131: Raise ImportWarning when loader.load_module() is used
http://bugs.python.org/issue26131

#26128: Let the subprocess.STARTUPINFO constructor take arguments
http://bugs.python.org/issue26128

#26122: Isolated mode doesn't ignore PYTHONHASHSEED
http://bugs.python.org/issue26122

#26120: pydoc: move __future__ imports out of the DATA block
http://bugs.python.org/issue26120

#26117: Close directory descriptor in scandir iterator on error
http://bugs.python.org/issue26117

#26103: Contradiction in definition of "data descriptor" between (dott
http://bugs.python.org/issue26103



Most recent 15 issues waiting for review (15)
=============================================

#26177: tkinter: Canvas().keys returns empty strings.
http://bugs.python.org/issue26177

#26175: Fully implement IOBase abstract on SpooledTemporaryFile
http://bugs.python.org/issue26175

#26167: Improve copy.copy speed for built-in types (list/set/dict)
http://bugs.python.org/issue26167

#26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const
http://bugs.python.org/issue26146

#26145: PEP 511: Add sys.set_code_transformers()
http://bugs.python.org/issue26145

#26144: test_pkg test_4 and/or test_7 sometimes fail
http://bugs.python.org/issue26144

#26140: inspect.iscoroutinefunction raises TypeError when checks Mock 
http://bugs.python.org/issue26140

#26130: redundant local copy of a char pointer in classify in Parser\p
http://bugs.python.org/issue26130

#26125: Incorrect error message in the module asyncio.selector_events.
http://bugs.python.org/issue26125

#26121: Use C99 functions in math if available
http://bugs.python.org/issue26121

#26117: Close directory descriptor in scandir iterator on error
http://bugs.python.org/issue26117

#26110: Speedup method calls 1.2x
http://bugs.python.org/issue26110

#26098: PEP 510: Specialize functions with guards
http://bugs.python.org/issue26098

#26089: Duplicated keyword in distutils metadata
http://bugs.python.org/issue26089

#26082: functools.lru_cache user specified cachedict support
http://bugs.python.org/issue26082



Top 10 most discussed issues (10)
=================================

#26158: File truncate() not defaulting to current position as document
http://bugs.python.org/issue26158  14 msgs

#25702: Link Time Optimizations support for GCC and CLANG
http://bugs.python.org/issue25702  12 msgs

#19475: Add timespec optional flag to datetime isoformat() to choose t
http://bugs.python.org/issue19475   9 msgs

#25878: CPython on Windows builds with /W3, not /W4
http://bugs.python.org/issue25878   8 msgs

#25907: Documentation i18n: Added trans tags in sphinx templates
http://bugs.python.org/issue25907   8 msgs

#26145: PEP 511: Add sys.set_code_transformers()
http://bugs.python.org/issue26145   8 msgs

#26152: A non-breaking space in a source
http://bugs.python.org/issue26152   7 msgs

#23883: __all__ lists are incomplete
http://bugs.python.org/issue23883   6 msgs

#25934: ICC compiler: ICC treats denormal floating point numbers as 0.
http://bugs.python.org/issue25934   6 msgs

#26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const
http://bugs.python.org/issue26146   6 msgs



Issues closed (84)
==================

#5626: misleading comment in socket.gethostname() documentation
http://bugs.python.org/issue5626  closed by berker.peksag

#8604: Adding an atomic FS write API
http://bugs.python.org/issue8604  closed by haypo

#9006: xml-rpc Server object does not propagate the encoding to Unmar
http://bugs.python.org/issue9006  closed by serhiy.storchaka

#12869: PyOS_StdioReadline is printing the prompt on stderr
http://bugs.python.org/issue12869  closed by martin.panter

#14046: argparse: assertion failure if optional argument has square/ro
http://bugs.python.org/issue14046  closed by martin.panter

#15809: 2.7 IDLE  console uses incorrect encoding.
http://bugs.python.org/issue15809  closed by serhiy.storchaka

#16620: Avoid using private function glob.glob1() in msi module and to
http://bugs.python.org/issue16620  closed by serhiy.storchaka

#16907: Distutils fails to build extension in path with spaces
http://bugs.python.org/issue16907  closed by zach.ware

#16956: Allow signed line number deltas in the code object's  line num
http://bugs.python.org/issue16956  closed by haypo

#17633: zipimport's handling of namespace packages is incorrect
http://bugs.python.org/issue17633  closed by brett.cannon

#18620: multiprocessing page leaves out important part of Pool example
http://bugs.python.org/issue18620  closed by berker.peksag

#21385: in debug mode, compile(ast) fails with an assertion error if a
http://bugs.python.org/issue21385  closed by haypo

#21847: Fix xmlrpc in unicodeless build
http://bugs.python.org/issue21847  closed by serhiy.storchaka

#21949: Document the Py_SIZE() macro.
http://bugs.python.org/issue21949  closed by berker.peksag

#23795: argparse -- incorrect usage for mutually exclusive
http://bugs.python.org/issue23795  closed by martin.panter

#23962: Incorrect TimeoutError referenced in concurrent.futures docume
http://bugs.python.org/issue23962  closed by orsenthil

#23965: test_ssl failure on Fedora 22
http://bugs.python.org/issue23965  closed by ncoghlan

#24520: Stop using deprecated floating-point environment functions on 
http://bugs.python.org/issue24520  closed by haypo

#24761: ERROR: test_dh_params (test.test_ssl.ThreadedTests)
http://bugs.python.org/issue24761  closed by martin.panter

#24832: Issue building viewable docs with newer sphinx (default theme 
http://bugs.python.org/issue24832  closed by benjamin.peterson

#24985: Python install test fails - OpenSSL - "dh key too small"
http://bugs.python.org/issue24985  closed by martin.panter

#25058: Right square bracket argparse metavar
http://bugs.python.org/issue25058  closed by martin.panter

#25089: Can't run Python Launcher on Windows
http://bugs.python.org/issue25089  closed by steve.dower

#25366: test_venv fails with --without-threads
http://bugs.python.org/issue25366  closed by berker.peksag

#25613: fix ssl tests with sslv3 disabled
http://bugs.python.org/issue25613  closed by martin.panter

#25644: Unable to open IDLE on Windows 10 with Python 2.7.10
http://bugs.python.org/issue25644  closed by steve.dower

#25694: test.libregrtest not installed
http://bugs.python.org/issue25694  closed by steve.dower

#25704: Update the devguide to 3.5
http://bugs.python.org/issue25704  closed by berker.peksag

#25731: Assigning and deleting __new__ attr on the class does not allo
http://bugs.python.org/issue25731  closed by python-dev

#25759: Python 2.7.11rc1 not building with Visual Studio 2015
http://bugs.python.org/issue25759  closed by steve.dower

#25765: Installation error
http://bugs.python.org/issue25765  closed by steve.dower

#25799: 2.7.11rc1 not added to Win10 app list (start menu)
http://bugs.python.org/issue25799  closed by terry.reedy

#25824: 32-bit 2.7.11 installer creates registry keys that are incompa
http://bugs.python.org/issue25824  closed by steve.dower

#25843: code_richcompare() don't use constant type when comparing code
http://bugs.python.org/issue25843  closed by haypo

#25850: Building extensions with MSVC 2015 Express fails
http://bugs.python.org/issue25850  closed by steve.dower

#25859: EOFError in test_nntplib.NetworkedNNTPTests.test_starttls()
http://bugs.python.org/issue25859  closed by martin.panter

#25876: test_gdb: use subprocess._args_from_interpreter_flags() to tes
http://bugs.python.org/issue25876  closed by haypo

#25909: Incorrect documentation for PyMapping_Items and like
http://bugs.python.org/issue25909  closed by orsenthil

#25925: Coverage support for CPython 2
http://bugs.python.org/issue25925  closed by zach.ware

#25935: OrderedDict prevents garbage collection if a circulary referen
http://bugs.python.org/issue25935  closed by serhiy.storchaka

#25982: multiprocessing docs for Namespace lacks class definition
http://bugs.python.org/issue25982  closed by orsenthil

#25993: Crashed when call time.time() after using _mm_xor_si64
http://bugs.python.org/issue25993  closed by steve.dower

#26013: Pickle protocol 2.0 not loading in python 3.5
http://bugs.python.org/issue26013  closed by serhiy.storchaka

#26017: Update https://docs.python.org/3/installing/index.html to alwa
http://bugs.python.org/issue26017  closed by orsenthil

#26035: traceback.print_tb() takes `tb`, not `traceback` as a keyword 
http://bugs.python.org/issue26035  closed by orsenthil

#26059: Integer Overflow in strop.replace()
http://bugs.python.org/issue26059  closed by serhiy.storchaka

#26065: python embedded 3.5 amd64 crash when using venv
http://bugs.python.org/issue26065  closed by steve.dower

#26070: Launcher fails to find in-place built binaries from earlier Py
http://bugs.python.org/issue26070  closed by steve.dower

#26071: bdist_wininst created binaries fail to start and find 32bit Py
http://bugs.python.org/issue26071  closed by steve.dower

#26073: Update the list of magic numbers in launcher
http://bugs.python.org/issue26073  closed by steve.dower

#26077: Make slicing of immutable structures return a view instead of 
http://bugs.python.org/issue26077  closed by serhiy.storchaka

#26099: site ignores ImportError when running sitecustomize and usercu
http://bugs.python.org/issue26099  closed by haypo

#26100: PEP 511: Add test.support.optim_args_from_interpreter_flags()
http://bugs.python.org/issue26100  closed by haypo

#26101: Lib/test/test_compileall.py fails when run directly
http://bugs.python.org/issue26101  closed by haypo

#26106: Move licences to literal blocks
http://bugs.python.org/issue26106  closed by haypo

#26107: PEP 511: code.co_lnotab: use signed line number delta to suppo
http://bugs.python.org/issue26107  closed by haypo

#26108: Calling PyInitialize with 2.7.11 on Windows x64 terminates pro
http://bugs.python.org/issue26108  closed by steve.dower

#26114: Rewrite math.erf() and math.erfc() from scratch
http://bugs.python.org/issue26114  closed by brett.cannon

#26126: Possible subtle bug when normalizing and str.translate()ing
http://bugs.python.org/issue26126  closed by SilentGhost

#26127: Broken link in docs for tokenize
http://bugs.python.org/issue26127  closed by martin.panter

#26129: Difference in behaviour with grp.getgrgid and pwd.getpwuid
http://bugs.python.org/issue26129  closed by larry

#26135: Documentation Recommends Deprecated `imp` Module
http://bugs.python.org/issue26135  closed by orsenthil

#26138: Disable /W4 warning (non-standard dllimport behavior)
http://bugs.python.org/issue26138  closed by skrah

#26139: libmpdec: disable /W4 warning (non-standard dllimport behavior
http://bugs.python.org/issue26139  closed by skrah

#26142: Formatting bug on https://docs.python.org/2.7/c-api/intro.html
http://bugs.python.org/issue26142  closed by orsenthil

#26147: Encoding errors in xmlrpc
http://bugs.python.org/issue26147  closed by serhiy.storchaka

#26150: SequenceMatcher's algorithm is not correct
http://bugs.python.org/issue26150  closed by tim.peters

#26151: str(bytes) does __repr__() instead of __str__()
http://bugs.python.org/issue26151  closed by haypo

#26154: Add private _PyThreadState_UncheckedGet() to get the current t
http://bugs.python.org/issue26154  closed by haypo

#26156: Bad name into power operator syntax
http://bugs.python.org/issue26156  closed by yselivanov

#26157: Typo in asyncio documentation
http://bugs.python.org/issue26157  closed by berker.peksag

#26161: Use Py_uintptr_t instead of void* for atomic pointers
http://bugs.python.org/issue26161  closed by haypo

#26162: thread error
http://bugs.python.org/issue26162  closed by eryksun

#26163: FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet)
http://bugs.python.org/issue26163  closed by haypo

#26164: test_with_pip() of test_venv fails on Windows buildbots
http://bugs.python.org/issue26164  closed by haypo

#26165: devguide: table summarizing status of Python branches
http://bugs.python.org/issue26165  closed by haypo

#26166: zlib compressor/decompressor objects should support copy proto
http://bugs.python.org/issue26166  closed by serhiy.storchaka

#26169: Pasting 900000 chars into a tk Entry widget fails
http://bugs.python.org/issue26169  closed by serhiy.storchaka

#26170: pip Crash on Unpacking in get_platform() line 119
http://bugs.python.org/issue26170  closed by dstufft

#26171: heap overflow in zipimporter module
http://bugs.python.org/issue26171  closed by python-dev

#26172: iBook can't open ePub
http://bugs.python.org/issue26172  closed by python-dev

#26174: Exception alias cause destruction of existing variable
http://bugs.python.org/issue26174  closed by brett.cannon

#26178: Python C-API: __all__ Creator
http://bugs.python.org/issue26178  closed by Devyn Johnson

#26179: Python C-API "unused-parameter" warnings
http://bugs.python.org/issue26179  closed by haypo

From robertpancoast77 at gmail.com  Fri Jan 22 14:56:37 2016
From: robertpancoast77 at gmail.com (=?UTF-8?Q?=C6=A6OB_COASTN?=)
Date: Fri, 22 Jan 2016 14:56:37 -0500
Subject: [Python-Dev] python3 k1om dissociation permanence
Message-ID: <CAHoh2tSmPrLO9KAsaEvpvrA7kn+xdmDJ8vxpRNDrSrY+JuH3Xg@mail.gmail.com>

Hello,

Enabling the build system for Intel MIC k1om is non-trivial using Python-3.4.4
Using Python2 for the k1om is very popular, but I require Python3
support on k1om.

The first requirement to complete this build involved the download and
extraction of pre-built MPSS RPM's.
Then built required host python bins using GCC.
Lastly, build MIC bins using ICC.
The exacts are appended to the end of this message.

I would like to discuss a few change requirements that trial and error
has revealed.

1.) libffi requires the University OF Cantabria patch because the k1om
is not binary compatible with x86_64. [attached]

These libffi changes could be implemented using the __MIC__ or __KNC__ macros.
*see https://software.intel.com/en-us/node/514528

2.) ./configure script halts during failure to locate readelf for the host.

I simply commented out these lines in the ./configure file:
 #if test "$cross_compiling" = yes; then
    #case "$READELF" in
    #readelf|:)
    #as_fn_error $? "readelf for the host is required for cross
builds" "$LINENO" 5
    #;;
    #esac
#fi

Ideally, ./configure would support ICC and k1om.
Am I missing something in the configure/make commands below?
Is it possible to bypass the readelf requirement when cross-compiling for k1om?

Additionally, are any of the command line parameters below unnecessary?
PKG_CONFIG_LIBDIR
PKG_CONFIG_PATH
PYTHON_FOR_BUILD
_PYTHON_HOST_PLATFORM
HOSTPGEN
HOSTARCH
BUILDARCH


Thanks,
Rob




#copy/unpack the k1om bins tarball
cd /home/
wget mpss-3.4.6-k1om.tar
tar xvf mpss-3.4.6-k1om.tar
cd /home/mpss-3.4.6/k1om/
for rpm in *.rpm; do rpm2cpio $rpm | cpio -idm; done

#vars
PythonVersion=Python-3.4.4
k1om_rpm=/home/mpss-3.4.6/k1om/
INSTALLPREFIX=/home/Python/release/$PythonVersion-mic
SRC=/home/Python/$PythonVersion

echo "Compiling host Python"
cd $SRC && make distclean
cd $SRC && ./configure
cd $SRC && make python Parser/pgen
rm -f $SRC/hostpython
mv $SRC/python $SRC/hostpython
rm -f $SRC/Parser/hostpgen
mv $SRC/Parser/pgen $SRC/Parser/hostpgen
cd $SRC && make distclean

echo "Configuring Python for MIC..."
cd $SRC && CONFIG_SITE=config.site \
./configure \
CC="icc -mmic" \
CFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \
CXX="icpc -mmic" \
CPPFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \
PKG_CONFIG_LIBDIR="$k1om_rpm/usr/lib64/pkgconfig" \
PKG_CONFIG_PATH="$k1om_rpm/usr/lib64/pkgconfig" \
--host=x86_64-k1om-linux \
--build=x86_64-linux-gnu \
--with-cxx-main="icpc -mmic" \
--disable-ipv6
echo "done"

echo "Compiling Python for MIC..."
cd $SRC && make \
PYTHON_FOR_BUILD=./hostpython \
_PYTHON_HOST_PLATFORM=x86_64-k1om-linux \
HOSTPGEN=./Parser/hostpgen \
HOSTARCH=x86_64-k1om-linux \
BUILDARCH=x86_64-linux-gnu \
EXTRA_CFLAGS="-fp-model precise -shared -fPIC" \
LDFLAGS="-L$k1om_rpm/lib64 -L$k1om_rpm/usr/lib64"
echo "done"

echo "Installing Python for MIC"
mkdir -p $INSTALLPREFIX
cd $SRC && make install \
PYTHON_FOR_BUILD=./hostpython \
_PYTHON_HOST_PLATFORM=x86_64-k1om-linux \
prefix=$INSTALLPREFIX
echo "done"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-k1om-libffi.patch
Type: application/octet-stream
Size: 18705 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160122/693e9b7d/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-READELF.patch
Type: application/octet-stream
Size: 790 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160122/693e9b7d/attachment-0001.obj>

From brett at python.org  Fri Jan 22 16:50:04 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 22 Jan 2016 21:50:04 +0000
Subject: [Python-Dev] python3 k1om dissociation permanence
In-Reply-To: <CAHoh2tSmPrLO9KAsaEvpvrA7kn+xdmDJ8vxpRNDrSrY+JuH3Xg@mail.gmail.com>
References: <CAHoh2tSmPrLO9KAsaEvpvrA7kn+xdmDJ8vxpRNDrSrY+JuH3Xg@mail.gmail.com>
Message-ID: <CAP1=2W5VRxffynTwGAUNn_zOq2FVjuoCHrxsb5a+YbB_6CTgTg@mail.gmail.com>

If you could, ?OB, can you open issues on bugs.python.org for each of these
problems/changes? Otherwise we will lose track of this.

On Fri, 22 Jan 2016 at 11:57 ?OB COASTN <robertpancoast77 at gmail.com> wrote:

> Hello,
>
> Enabling the build system for Intel MIC k1om is non-trivial using
> Python-3.4.4
> Using Python2 for the k1om is very popular, but I require Python3
> support on k1om.
>
> The first requirement to complete this build involved the download and
> extraction of pre-built MPSS RPM's.
> Then built required host python bins using GCC.
> Lastly, build MIC bins using ICC.
> The exacts are appended to the end of this message.
>
> I would like to discuss a few change requirements that trial and error
> has revealed.
>
> 1.) libffi requires the University OF Cantabria patch because the k1om
> is not binary compatible with x86_64. [attached]
>
> These libffi changes could be implemented using the __MIC__ or __KNC__
> macros.
> *see https://software.intel.com/en-us/node/514528
>
> 2.) ./configure script halts during failure to locate readelf for the host.
>
> I simply commented out these lines in the ./configure file:
>  #if test "$cross_compiling" = yes; then
>     #case "$READELF" in
>     #readelf|:)
>     #as_fn_error $? "readelf for the host is required for cross
> builds" "$LINENO" 5
>     #;;
>     #esac
> #fi
>
> Ideally, ./configure would support ICC and k1om.
> Am I missing something in the configure/make commands below?
> Is it possible to bypass the readelf requirement when cross-compiling for
> k1om?
>
> Additionally, are any of the command line parameters below unnecessary?
> PKG_CONFIG_LIBDIR
> PKG_CONFIG_PATH
> PYTHON_FOR_BUILD
> _PYTHON_HOST_PLATFORM
> HOSTPGEN
> HOSTARCH
> BUILDARCH
>
>
> Thanks,
> Rob
>
>
>
>
> #copy/unpack the k1om bins tarball
> cd /home/
> wget mpss-3.4.6-k1om.tar
> tar xvf mpss-3.4.6-k1om.tar
> cd /home/mpss-3.4.6/k1om/
> for rpm in *.rpm; do rpm2cpio $rpm | cpio -idm; done
>
> #vars
> PythonVersion=Python-3.4.4
> k1om_rpm=/home/mpss-3.4.6/k1om/
> INSTALLPREFIX=/home/Python/release/$PythonVersion-mic
> SRC=/home/Python/$PythonVersion
>
> echo "Compiling host Python"
> cd $SRC && make distclean
> cd $SRC && ./configure
> cd $SRC && make python Parser/pgen
> rm -f $SRC/hostpython
> mv $SRC/python $SRC/hostpython
> rm -f $SRC/Parser/hostpgen
> mv $SRC/Parser/pgen $SRC/Parser/hostpgen
> cd $SRC && make distclean
>
> echo "Configuring Python for MIC..."
> cd $SRC && CONFIG_SITE=config.site \
> ./configure \
> CC="icc -mmic" \
> CFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \
> CXX="icpc -mmic" \
> CPPFLAGS="-I$k1om_rpm/include -I$k1om_rpm/usr/include -wd10006" \
> PKG_CONFIG_LIBDIR="$k1om_rpm/usr/lib64/pkgconfig" \
> PKG_CONFIG_PATH="$k1om_rpm/usr/lib64/pkgconfig" \
> --host=x86_64-k1om-linux \
> --build=x86_64-linux-gnu \
> --with-cxx-main="icpc -mmic" \
> --disable-ipv6
> echo "done"
>
> echo "Compiling Python for MIC..."
> cd $SRC && make \
> PYTHON_FOR_BUILD=./hostpython \
> _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \
> HOSTPGEN=./Parser/hostpgen \
> HOSTARCH=x86_64-k1om-linux \
> BUILDARCH=x86_64-linux-gnu \
> EXTRA_CFLAGS="-fp-model precise -shared -fPIC" \
> LDFLAGS="-L$k1om_rpm/lib64 -L$k1om_rpm/usr/lib64"
> echo "done"
>
> echo "Installing Python for MIC"
> mkdir -p $INSTALLPREFIX
> cd $SRC && make install \
> PYTHON_FOR_BUILD=./hostpython \
> _PYTHON_HOST_PLATFORM=x86_64-k1om-linux \
> prefix=$INSTALLPREFIX
> echo "done"
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160122/966ab5aa/attachment-0001.html>

From brett at python.org  Fri Jan 22 18:44:57 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 22 Jan 2016 23:44:57 +0000
Subject: [Python-Dev] do people use sys._mercurial?
Message-ID: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>

Since we are going to be switching over to Git, sys._mercurial is going to
be made to return a dummy value of `('CPython', '', '')` once we switch to
Git. But my question is do we bother to replace it with sys._git? I wanted
to make sure that the effort is worth it to keep changing these
VCS-specific attributes every time we change our VCS (and no, we are not
going to adopt a generic one; already had that debate). So do please speak
up if you actually have found value from sys._mercurial.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160122/6b92b162/attachment.html>

From ncoghlan at gmail.com  Fri Jan 22 21:47:14 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 23 Jan 2016 12:47:14 +1000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
Message-ID: <CADiSq7cZ4beZQkmbM3ikrvthktiHDcpnvghmBC8T+oSU67CpPw@mail.gmail.com>

On 23 January 2016 at 09:44, Brett Cannon <brett at python.org> wrote:
> Since we are going to be switching over to Git, sys._mercurial is going to
> be made to return a dummy value of `('CPython', '', '')` once we switch to
> Git. But my question is do we bother to replace it with sys._git? I wanted
> to make sure that the effort is worth it to keep changing these VCS-specific
> attributes every time we change our VCS (and no, we are not going to adopt a
> generic one; already had that debate). So do please speak up if you actually
> have found value from sys._mercurial.

It's incorporated into the output of
"platform.python_(branch|revision|build)()", so I assume most people
would use those if they needed to report exact build information,
rather than accessing the attribute directly.

We also use it ourselves in printing the appropriate banner when
running an interactive prompt (note the first line of the banner):

Python 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00)
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._mercurial
('CPython', 'default', '32a4e7b337c9')

And the regression test suite:

$ ./python -m test
== CPython 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00) [GCC
5.3.1 20151207 (Red Hat 5.3.1-2)]
==   Linux-4.3.3-301.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three
little-endian
==   hash algorithm: siphash24 64bit
==   /home/ncoghlan/devel/cpython/build/test_python_13167
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0,
optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
ignore_environment=0, verbose=0, bytes_warning=0, quiet=0,
hash_randomization=1, isolated=0)
[  1/401] test_grammar
...

I guess that means "Update platform.py, and test the interactive
prompt and regression test banners still work as expected when built
from git" needs to be added to the CPython migration part of the PEP.
(In looking into this, I found several of the docstrings in
platform.py still referred to Subversion, even though the
platform._sys_version() helping had been updated to handle Mercurial)

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From nad at python.org  Fri Jan 22 22:33:39 2016
From: nad at python.org (Ned Deily)
Date: Fri, 22 Jan 2016 22:33:39 -0500
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
Message-ID: <E6D1FE9D-D4E7-46F6-9046-CEA1D72D81E4@python.org>

On Jan 22, 2016, at 18:44, Brett Cannon <brett at python.org> wrote:
> Since we are going to be switching over to Git, sys._mercurial is going to be made to return a dummy value of `('CPython', '', '')` once we switch to Git. But my question is do we bother to replace it with sys._git? I wanted to make sure that the effort is worth it to keep changing these VCS-specific attributes every time we change our VCS (and no, we are not going to adopt a generic one; already had that debate). So do please speak up if you actually have found value from sys._mercurial.

As long as the git revision tag (if any) and hash show up in sys.version and the interpreter interactive (REPL) mode header as they do today with hg and previously with svn (that is, when the interpreter is built from a vcs checkout), I'm happy:

$ python3
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

--
  Ned Deily
  nad at python.org -- []


From guido at python.org  Sat Jan 23 00:50:03 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 22 Jan 2016 21:50:03 -0800
Subject: [Python-Dev] Typehinting repo moved to python/typing
Message-ID: <CAP7+vJK_f4RspBua4ynbv2qjxanXDNiZJGd6njLSwJOXX=r2GA@mail.gmail.com>

This is just a note that with Benjamin's help we've moved the
ambv/typehinting repo on GitHub into the python org, so its URL is now
https://github.com/python/typing .

This repo was used most intensely for discussions during PEP 484's drafting
period. It also contains the code for typing.py, repackaged for earlier
releases on PyPI. The issue tracker is still open for proposals to change
PEP 484, which is not unheard of given its provisional status. If you find
a pointer to the original location of this repo in a file you can update,
please go ahead (though GitHub is pretty good at forwarding URLs from
renamed repos).

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160122/480167b4/attachment.html>

From rosuav at gmail.com  Sat Jan 23 01:03:12 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 23 Jan 2016 17:03:12 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure - test_socket
 issue with UDP6?
Message-ID: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>

I just had a major crash on the system that hosts the
angelico-debian-amd64 buildbot, and as usual, checked it carefully
after bringing everything up. It seems now to be timing out after an
hour of operation:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3132/steps/test/logs/stdio

This is happening across all the 3.* branches on my buildbot, but NOT
on 2.7, and not on any other buildbots. That makes it look like some
kind of config problem at my end. In seeking to diagnose it, I
duplicated the 3.x build directory and manually ran the commands to
run a test, and it stalled out (I didn't let it go to the whole hour
but it sat there for some minutes) at the same place: in test_socket.

Running just that test file:

$ ./python Lib/test/test_socket.py
... chomp lots of lines ...
testRecvmsgPeek (__main__.RecvmsgUDP6Test) ...

seems to indicate that the stall is due to IPv6 and UDP. The VM should
have full IPv6 support, although my ISPs don't carry IPv6 traffic, so
it won't be able to reach the internet proper; but it should be able
to do all manner of local tests. The test runs just fine on my main
system, so it's only failing in the buildbot VM. Any ideas as to
what's going on or how to diagnose?

By the way, this looks odd:

make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600
 in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs)

The parameter says 3600 (which corresponds to the error message at the
end), but it echoes back that the timeout is 3900 seconds.

ChrisA

From zachary.ware+pydev at gmail.com  Sat Jan 23 01:39:07 2016
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Sat, 23 Jan 2016 00:39:07 -0600
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
Message-ID: <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>

On Sat, Jan 23, 2016 at 12:03 AM, Chris Angelico <rosuav at gmail.com> wrote:
> By the way, this looks odd:
>
> make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600
>  in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs)
>
> The parameter says 3600 (which corresponds to the error message at the
> end), but it echoes back that the timeout is 3900 seconds.

I'm no help on the main issue, but to explain the timeout difference:
TESTTIMEOUT is a makefile variable that sets the faulthandler timeout
that tries to make Python bail out with a stack trace instead of
letting buildbot kill Python silently.  The 3900 second timeout is
buildbot's "there's been no output in this long, assume it's hung and
kill it" timeout.  The difference between the two is to give
faulthandler a chance to do its thing.

-- 
Zach

From rosuav at gmail.com  Sat Jan 23 02:47:14 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 23 Jan 2016 18:47:14 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>
Message-ID: <CAPTjJmpDRgauGMnKuwgfoozM-+j2JSTMi9wHBAb4Wq=DFm3ZOA@mail.gmail.com>

On Sat, Jan 23, 2016 at 5:39 PM, Zachary Ware
<zachary.ware+pydev at gmail.com> wrote:
> On Sat, Jan 23, 2016 at 12:03 AM, Chris Angelico <rosuav at gmail.com> wrote:
>> By the way, this looks odd:
>>
>> make buildbottest TESTOPTS= TESTPYTHONOPTS= TESTTIMEOUT=3600
>>  in dir /root/buildarea/3.x.angelico-debian-amd64/build (timeout 3900 secs)
>>
>> The parameter says 3600 (which corresponds to the error message at the
>> end), but it echoes back that the timeout is 3900 seconds.
>
> I'm no help on the main issue, but to explain the timeout difference:
> TESTTIMEOUT is a makefile variable that sets the faulthandler timeout
> that tries to make Python bail out with a stack trace instead of
> letting buildbot kill Python silently.  The 3900 second timeout is
> buildbot's "there's been no output in this long, assume it's hung and
> kill it" timeout.  The difference between the two is to give
> faulthandler a chance to do its thing.

Ah, cool. That's one mystery cleared up, at least.

ChrisA

From victor.stinner at gmail.com  Sat Jan 23 09:37:36 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 23 Jan 2016 15:37:36 +0100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>
Message-ID: <CAMpsgwZy=QX+bNAaaQbt-2ySrcHNsPVhZW6Xqd8_oAtV+qTvpw@mail.gmail.com>

3600 seconds is the maximum duration of a single test file. We may reduce
it since a single test file should not take longer than 30 min. Maybe we
can do better and put the timeout on a single test function.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160123/1fd2cc0d/attachment.html>

From rosuav at gmail.com  Sat Jan 23 09:55:58 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sun, 24 Jan 2016 01:55:58 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAMpsgwZy=QX+bNAaaQbt-2ySrcHNsPVhZW6Xqd8_oAtV+qTvpw@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAKJDb-MuYcMSQb_6OqD=5p1suwAaazw+ANVkFDiPM_NZjRp_mg@mail.gmail.com>
 <CAMpsgwZy=QX+bNAaaQbt-2ySrcHNsPVhZW6Xqd8_oAtV+qTvpw@mail.gmail.com>
Message-ID: <CAPTjJmpAHMP-3SOOu2tGHF+6eCc6WPtPyXcgBxf3uUgVXrMe4w@mail.gmail.com>

On Sun, Jan 24, 2016 at 1:37 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 3600 seconds is the maximum duration of a single test file. We may reduce it
> since a single test file should not take longer than 30 min. Maybe we can do
> better and put the timeout on a single test function.

I'd be inclined to put some strong timeouts on test_socket.py (at some
level or other of granularity). Most of those tests should be finished
in a fraction of a second; a few might take a few seconds, maybe. None
should take a whole minute. But they might easily sit around that
long.

But I'd rather know what I messed up in my recreation of the VM's configs.

ChrisA

From brett at python.org  Sat Jan 23 14:30:32 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 23 Jan 2016 19:30:32 +0000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <E6D1FE9D-D4E7-46F6-9046-CEA1D72D81E4@python.org>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <E6D1FE9D-D4E7-46F6-9046-CEA1D72D81E4@python.org>
Message-ID: <CAP1=2W4eMNKekdsvAQt4r1JE6OjkSY9gw2CcNKE-v+UyGMSRrQ@mail.gmail.com>

It seems that people do use the values so there will be a sys._git
attribute.

On Fri, 22 Jan 2016 at 19:57 Ned Deily <nad at python.org> wrote:

> On Jan 22, 2016, at 18:44, Brett Cannon <brett at python.org> wrote:
> > Since we are going to be switching over to Git, sys._mercurial is going
> to be made to return a dummy value of `('CPython', '', '')` once we switch
> to Git. But my question is do we bother to replace it with sys._git? I
> wanted to make sure that the effort is worth it to keep changing these
> VCS-specific attributes every time we change our VCS (and no, we are not
> going to adopt a generic one; already had that debate). So do please speak
> up if you actually have found value from sys._mercurial.
>
> As long as the git revision tag (if any) and hash show up in sys.version
> and the interpreter interactive (REPL) mode header as they do today with hg
> and previously with svn (that is, when the interpreter is built from a vcs
> checkout), I'm happy:
>
> $ python3
> Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44)
> [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>>
>
> --
>   Ned Deily
>   nad at python.org -- []
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160123/a95dc2c8/attachment.html>

From francismb at email.de  Sat Jan 23 14:45:19 2016
From: francismb at email.de (francismb)
Date: Sat, 23 Jan 2016 20:45:19 +0100
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
Message-ID: <56A3D84F.70803@email.de>

Hi,

On 01/23/2016 12:44 AM, Brett Cannon wrote:
> Since we are going to be switching over to Git, sys._mercurial is going to
> be made to return a dummy value of `('CPython', '', '')` once we switch to
> Git.

for me sys._mercurial it's already returning that (?) : what should
return now? (it's a bug?)


$ ipython
Python 2.7.11 (default, Dec  9 2015, 00:29:25)
Type "copyright", "credits" or "license" for more information.

IPython 2.4.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import sys

In [2]: sys._mercurial
Out[2]: ('CPython', '', '')

In [3]:


Regards,
francis

From brett at python.org  Sat Jan 23 14:48:56 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 23 Jan 2016 19:48:56 +0000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <56A3D84F.70803@email.de>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <56A3D84F.70803@email.de>
Message-ID: <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>

On Sat, 23 Jan 2016 at 11:45 francismb <francismb at email.de> wrote:

> Hi,
>
> On 01/23/2016 12:44 AM, Brett Cannon wrote:
> > Since we are going to be switching over to Git, sys._mercurial is going
> to
> > be made to return a dummy value of `('CPython', '', '')` once we switch
> to
> > Git.
>
> for me sys._mercurial it's already returning that (?) : what should
> return now? (it's a bug?)
>

Depends on your OS and how CPython was built whether it returns that value
or something more useful. IOW it's not a bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160123/14ae5a9b/attachment.html>

From random832 at fastmail.com  Sat Jan 23 19:09:41 2016
From: random832 at fastmail.com (Random832)
Date: Sat, 23 Jan 2016 19:09:41 -0500
Subject: [Python-Dev] do people use sys._mercurial?
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
Message-ID: <m2k2mzltx6.fsf@fastmail.com>

Brett Cannon <brett at python.org> writes:
> (and no, we are not going to adopt a generic one; already had that
> debate).

Do you have a link to the relevant discussions?


From brett at python.org  Sat Jan 23 19:29:12 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 24 Jan 2016 00:29:12 +0000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <m2k2mzltx6.fsf@fastmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <m2k2mzltx6.fsf@fastmail.com>
Message-ID: <CAP1=2W6ZPeiL+efr1YBd4eBTUDVx7UddjAp5tjgx+-V3L+qaVA@mail.gmail.com>

Some discussion happened on core-workflow@, otherwise you can look through
the python-dev archives for when we added sys._mercurial.

On Sat, 23 Jan 2016, 16:25 Random832 <random832 at fastmail.com> wrote:

> Brett Cannon <brett at python.org> writes:
> > (and no, we are not going to adopt a generic one; already had that
> > debate).
>
> Do you have a link to the relevant discussions?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160124/2f5dd01d/attachment.html>

From ncoghlan at gmail.com  Sun Jan 24 07:17:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Jan 2016 22:17:32 +1000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <56A3D84F.70803@email.de>
 <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>
Message-ID: <CADiSq7ceO+LL6k4zmpHzZjHYKERNHXRGKCV7sVTEuG6YqW7qcA@mail.gmail.com>

On 24 January 2016 at 05:48, Brett Cannon <brett at python.org> wrote:
> On Sat, 23 Jan 2016 at 11:45 francismb <francismb at email.de> wrote:
>> for me sys._mercurial it's already returning that (?) : what should
>> return now? (it's a bug?)
>
> Depends on your OS and how CPython was built whether it returns that value
> or something more useful. IOW it's not a bug.

Linux distros tend to build Python from a tarball rather than a source
checkout, for example, which means the build directory doesn't include
any VCS details:

$ python3
Python 3.4.3 (default, Jun 29 2015, 12:16:01)
[GCC 5.1.1 20150618 (Red Hat 5.1.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._mercurial
('CPython', '', '')

While my local checkout does have those details:

$ ./python
Python 3.6.0a0 (default:32a4e7b337c9, Jan 23 2016, 12:30:00)
[GCC 5.3.1 20151207 (Red Hat 5.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._mercurial
('CPython', 'default', '32a4e7b337c9')

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun Jan 24 07:13:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 24 Jan 2016 22:13:32 +1000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CAP1=2W6ZPeiL+efr1YBd4eBTUDVx7UddjAp5tjgx+-V3L+qaVA@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <m2k2mzltx6.fsf@fastmail.com>
 <CAP1=2W6ZPeiL+efr1YBd4eBTUDVx7UddjAp5tjgx+-V3L+qaVA@mail.gmail.com>
Message-ID: <CADiSq7etHQ8JOyQLMraBCoAZDhkqojqREH_gk01xaxdJHNJiMg@mail.gmail.com>

On 24 January 2016 at 10:29, Brett Cannon <brett at python.org> wrote:
> Some discussion happened on core-workflow@, otherwise you can look through
> the python-dev archives for when we added sys._mercurial.

We actually forgot one relevant point in those discussions: there's
already a generic API for accessing this information in the platform
module.

At the sys module level, though, the key point is that the precise
semantic interpretation is VCS dependent, so changing the variable
name conveys immediately to users both that the semantics have
changed, and which VCS is now being used to define them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From francismb at email.de  Sun Jan 24 10:48:07 2016
From: francismb at email.de (francismb)
Date: Sun, 24 Jan 2016 16:48:07 +0100
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <569EA3A6.4010802@email.de>
References: <569EA3A6.4010802@email.de>
Message-ID: <56A4F237.6060709@email.de>

Hi,
from your valuable feedback, here is what I thing could be a previous
requirements list (lets call it for e.g. autopep7 script by now):

- It should follow PEP 7 :-)
- It should check PEP 7 compliance on a per file basis (for simplicity)
- It should be embeddable on the test suite, returning PASS or FAILURE
- It should be usable as a pre-commit hook
- It could return a patch to apply to the non compliant file making it
compliant
- It could return the lines (an the reason why) that aren't compliant


Some other details on using some autopep7 on the infrastructure (as part
of a workflow, by wrapping or enhancing it):

- It could be used it on a regular basis against the cpython repo to be
able to see how much files doesn't follow PEP 7. The first time shows
the legacy ones (plus the current failures).

- It should be used in pair with a "skip" list to avoid checking the
legacy code mentioned (or a list of pairs file:reason to skip).

- It could be used on the CI-side to flag PEP 7 compliance (core-devs
doesn't have to need to point to that in reviews). Or could be used as a
part of series of checks that generate a "ready-to-review" flag.

- It could be used to continually check and proposes patches (on
bug.python.org).

- It could be used to continually check, log the issue on the tracker
and commit the patch to the cpython repository.


Regards,
francis

From francismb at email.de  Sun Jan 24 12:40:14 2016
From: francismb at email.de (francismb)
Date: Sun, 24 Jan 2016 18:40:14 +0100
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <CADiSq7ceO+LL6k4zmpHzZjHYKERNHXRGKCV7sVTEuG6YqW7qcA@mail.gmail.com>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <56A3D84F.70803@email.de>
 <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>
 <CADiSq7ceO+LL6k4zmpHzZjHYKERNHXRGKCV7sVTEuG6YqW7qcA@mail.gmail.com>
Message-ID: <56A50C7E.3090209@email.de>

Hi,

On 01/24/2016 01:17 PM, Nick Coghlan wrote:
> 
> Linux distros tend to build Python from a tarball rather than a source
> checkout, for example, which means the build directory doesn't include
> any VCS details:
>

Does that helps traceability (reproducibility)? If distros use (?) the
tarball from the release why it doesn't have, at least, the information
from where that tarball was generated from (the check out point) ?

Regards,
francis


From raymond.hettinger at gmail.com  Sun Jan 24 19:28:36 2016
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 24 Jan 2016 16:28:36 -0800
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <569EA3A6.4010802@email.de>
References: <569EA3A6.4010802@email.de>
Message-ID: <A2779BD4-09FF-46C7-A9C3-83C45A2D86EE@gmail.com>


> On Jan 19, 2016, at 12:59 PM, francismb <francismb at email.de> wrote:
> 
> Dear Core-Devs,
> what's your opinion about a code-formatter bot for cpython.
> Pros, Cons, where could be applicable (new commits, new workflow, it
> doesn't make sense), ...
> 
> 
> - At least it should follow PEP 7 ;-)

Please don't do this.  It misses the spirit of how the style-guides are intended to be used.

"I personally hate with a vengeance that there are tools named after style guide PEPs that claim to enforce the guidelines from those PEPs. The tools' rigidity and simplicity reflects badly on the PEPs, which try hard not to be rigid or simplistic." -- GvR
https://mail.python.org/pipermail/python-dev/2016-January/142643.html

"PEP 8 unto thyself, not onto others" -- Me
https://www.youtube.com/watch?v=wf-BqAjZb8M
(the most popular talk from last year's Pycon)

Almost nothing that is wrong with CPython is stylistic, the real issues are more substantive.  That is where you should devote your talents.


Raymond Hettinger

From stephen at xemacs.org  Sun Jan 24 21:07:17 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 25 Jan 2016 11:07:17 +0900
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <56A50C7E.3090209@email.de>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <56A3D84F.70803@email.de>
 <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>
 <CADiSq7ceO+LL6k4zmpHzZjHYKERNHXRGKCV7sVTEuG6YqW7qcA@mail.gmail.com>
 <56A50C7E.3090209@email.de>
Message-ID: <22181.33621.40740.909756@turnbull.sk.tsukuba.ac.jp>

francismb writes:

 > Does that helps traceability (reproducibility)? If distros use (?) 
 > the tarball from the release why it doesn't have, at least, the
 > information from where that tarball was generated from (the check
 > out point) ?

The pointer goes in the other direction: there will be a tag in the
repo named to indicate the version.

From ncoghlan at gmail.com  Sun Jan 24 21:24:34 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 25 Jan 2016 12:24:34 +1000
Subject: [Python-Dev] do people use sys._mercurial?
In-Reply-To: <56A50C7E.3090209@email.de>
References: <CAP1=2W4+H-V-YDN4Suwnmian11J+hqUJap1NcFbe+LKbaPtBuw@mail.gmail.com>
 <56A3D84F.70803@email.de>
 <CAP1=2W5GtUjN2uvS1v4H3zFnab=3Kovsn81KyoF_+KQdNKOupQ@mail.gmail.com>
 <CADiSq7ceO+LL6k4zmpHzZjHYKERNHXRGKCV7sVTEuG6YqW7qcA@mail.gmail.com>
 <56A50C7E.3090209@email.de>
Message-ID: <CADiSq7cGRAg-n4gRfFcMrA523g_mLrc0PCD5xCsrPmpCX2AL0g@mail.gmail.com>

On 25 January 2016 at 03:40, francismb <francismb at email.de> wrote:
> On 01/24/2016 01:17 PM, Nick Coghlan wrote:
>> Linux distros tend to build Python from a tarball rather than a source
>> checkout, for example, which means the build directory doesn't include
>> any VCS details:
> Does that helps traceability (reproducibility)? If distros use (?) the
> tarball from the release why it doesn't have, at least, the information
> from where that tarball was generated from (the check out point) ?

The main reason is that distro packaging processes long predate
Subversion's popularisation of atomic commits in open source version
control tools, and are designed to cope with release processes that
involve uploading a source tarball to a web server, so they don't
assume VCS tags or revision IDs will be available.

However, distro processes also capture the source code itself, and
often apply additional distro-specific patches, at which point
claiming to correspond directly to any given upstream commit would be
inaccurate.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Mon Jan 25 13:16:29 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 25 Jan 2016 19:16:29 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
Message-ID: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>

Hi,

Summary: FAT Python is not faster, but it will be ;-)

--

When I started the FAT Python as a fork of CPython 3.6, I put
everything in the same repository. Last weeks, I focused on splitting
my giant patch (10k lines) into small reviewable patches. I wrote 3
PEP (509 dict version, 510 function specialziation, 511 code
tranformers) and I enhanced the API to make it usable for more use
cases than just FAT Python. I also created fatoptimizer (the AST
optimizer) and fat (runtime dependency of the optimizer) projects on
GitHub to separate clearly what should be outside Python core. For all
links, see:

   http://faster-cpython.readthedocs.org/fat_python.html

For the fatoptimizer project, my constraint is to be able to run the
full Python test suite unmodified. In practice, I have to disable some
optimizations by putting a "__fatoptimizer__= {...}" configuration to
some test files. For example, I have to disable constant folding on
test_bool because it tests that False+2 gives 2 at runtime, whereas
the optimizer replaces directly False+2 with 2 during the compilation.
Well, test_bool.py is not the best example because all tests pass with
the constant folding optimization (if I comment my
"__fatoptimizer__={...}" change).

This constraint ensures that the optimizer "works" and doesn't break
(too much ;-)) the Python semantics, but it's more difficult to
implement powerful optimizations.

I also found and fixed various kinds of bugs. In my code obviously,
but also in the Python core, in various places. Some bugs only concern
AST transformers which is a new feature, but I had to fix them. For
example, Python didn't support negative line number delta in
co_lntotab of code objects, and so line number were all wrong on
optimized code. I merged my enhancement in the default branch of
CPython (issue #26107).

In short, I focused on having something working (respecting the Python
semantics), rather than spending time on writing optimizations.

--

When I asked explicitly "Is someone opposed to this PEP 509 [dict
verion] ?", Barry Warsaw answered that a performance analysis is
required. Extract of his mail:

   "I still think this is maintenance and potential performance
overhead we don't want to commit to long term unless it enables
significant optimization.  Since you probably can't prove that without
some experimentation, this API should be provisional."

Last week, I ran some benchmarks and I have to admin that I was
disappointed. Not only fatoptimizer doesn't make Python faster, but it
makes it much slower on some tests!

   http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html

Quickly, I identified a major performance issue when nested functions
are specialized, especially in Lib/json/encoder.py (tested by
bm_json_v2.py benchmark). I fixed my optimizer to not specialize
nested functions anymore. This simple change fixed the main
performance issue. Reminder: in performance critical code, don't use
nested functions! I will maybe propose patches for Lib/json/encoder.py
to stop using nested functions.

I only ran benchmarks with the optimizer enabled. I now have to
measure the overhead of my patches (PEP 509, 510 and 511) adding the
API fat AST optimizers. The overhead must be negligible. For me, it's
a requirement of the whole project. Changes must not make Python
slower when the optimizer is not used.

fatoptimizer is faster on microbenchmarks, but I had to write manually
some optimizations:

   http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html

IMHO fatoptimizer is not faster on macro benchmarks because it is not
smart enough (yet) to generate the most interesting optimizations,
like function inlining and specialization for argument types. You can
estimate the speedup if you specialize manually your functions.

--

Barry also wrote: "Did you address my suggestion on python-ideas to
make the new C API optionally compiled in?"

Well, it is an option, but I would prefer to have the API for AST
optimizer directly built in Python.

The first beta version of Python 3.6 is scheduled in September 2016
(deadline for new features in Python 3.6), so I still have a few
months to implement more powerful optimizations and prove that it can
be faster ;-)

Victor

From francismb at email.de  Mon Jan 25 14:55:38 2016
From: francismb at email.de (francismb)
Date: Mon, 25 Jan 2016 20:55:38 +0100
Subject: [Python-Dev] Code formatter bot
In-Reply-To: <A2779BD4-09FF-46C7-A9C3-83C45A2D86EE@gmail.com>
References: <569EA3A6.4010802@email.de>
 <A2779BD4-09FF-46C7-A9C3-83C45A2D86EE@gmail.com>
Message-ID: <56A67DBA.8010005@email.de>

Hi,

On 01/25/2016 01:28 AM, Raymond Hettinger wrote:
>
>> - At least it should follow PEP 7 ;-)
> 
> Please don't do this.  It misses the spirit of how the style-guides are intended to be used.
>
> "I personally hate with a vengeance that there are tools named after style guide PEPs that claim to enforce the guidelines from those PEPs. The tools' rigidity and simplicity reflects badly on the PEPs, which try hard not to be rigid or simplistic." -- GvR
> https://mail.python.org/pipermail/python-dev/2016-January/142643.html
> 
Good point, tools need to get better (more context sensitive), maybe
should work only as advisors and have just funny names (and not claim
the spirit).


> "PEP 8 unto thyself, not onto others" -- Me
> https://www.youtube.com/watch?v=wf-BqAjZb8M
> (the most popular talk from last year's Pycon)
> 
> Almost nothing that is wrong with CPython is stylistic, the real issues are more substantive.  That is where you should devote your talents.
> 
Good advices and refactoring on your talk (and I love the gorilla part
but I think that the "color_distraction" trick doesn't work for "non
native" speakers as me ;-)) and indeed, that wanted to be my point:

Let's review what the code does and not 007-Things. It's was really
about the workflow not style, but I'll follow your advice.


Regards,
francis


PS: Interesting is also how/why do people introduce bug's by
reformatting or trying to beautify things (and how those can be avoided,
by allowing change).

From gmludo at gmail.com  Mon Jan 25 16:20:24 2016
From: gmludo at gmail.com (Ludovic Gasc)
Date: Mon, 25 Jan 2016 22:20:24 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
Message-ID: <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>

Hi,

Just thanks for this big contribution.
And maybe this project could give new ideas to optimize Python, who knows ?

At least, you've win a beer for the FOSDEM event this week-end ;-)

Have a nice week.


--
Ludovic Gasc (GMLudo)
http://www.gmludo.eu/

2016-01-25 19:16 GMT+01:00 Victor Stinner <victor.stinner at gmail.com>:

> Hi,
>
> Summary: FAT Python is not faster, but it will be ;-)
>
> --
>
> When I started the FAT Python as a fork of CPython 3.6, I put
> everything in the same repository. Last weeks, I focused on splitting
> my giant patch (10k lines) into small reviewable patches. I wrote 3
> PEP (509 dict version, 510 function specialziation, 511 code
> tranformers) and I enhanced the API to make it usable for more use
> cases than just FAT Python. I also created fatoptimizer (the AST
> optimizer) and fat (runtime dependency of the optimizer) projects on
> GitHub to separate clearly what should be outside Python core. For all
> links, see:
>
>    http://faster-cpython.readthedocs.org/fat_python.html
>
> For the fatoptimizer project, my constraint is to be able to run the
> full Python test suite unmodified. In practice, I have to disable some
> optimizations by putting a "__fatoptimizer__= {...}" configuration to
> some test files. For example, I have to disable constant folding on
> test_bool because it tests that False+2 gives 2 at runtime, whereas
> the optimizer replaces directly False+2 with 2 during the compilation.
> Well, test_bool.py is not the best example because all tests pass with
> the constant folding optimization (if I comment my
> "__fatoptimizer__={...}" change).
>
> This constraint ensures that the optimizer "works" and doesn't break
> (too much ;-)) the Python semantics, but it's more difficult to
> implement powerful optimizations.
>
> I also found and fixed various kinds of bugs. In my code obviously,
> but also in the Python core, in various places. Some bugs only concern
> AST transformers which is a new feature, but I had to fix them. For
> example, Python didn't support negative line number delta in
> co_lntotab of code objects, and so line number were all wrong on
> optimized code. I merged my enhancement in the default branch of
> CPython (issue #26107).
>
> In short, I focused on having something working (respecting the Python
> semantics), rather than spending time on writing optimizations.
>
> --
>
> When I asked explicitly "Is someone opposed to this PEP 509 [dict
> verion] ?", Barry Warsaw answered that a performance analysis is
> required. Extract of his mail:
>
>    "I still think this is maintenance and potential performance
> overhead we don't want to commit to long term unless it enables
> significant optimization.  Since you probably can't prove that without
> some experimentation, this API should be provisional."
>
> Last week, I ran some benchmarks and I have to admin that I was
> disappointed. Not only fatoptimizer doesn't make Python faster, but it
> makes it much slower on some tests!
>
>    http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html
>
> Quickly, I identified a major performance issue when nested functions
> are specialized, especially in Lib/json/encoder.py (tested by
> bm_json_v2.py benchmark). I fixed my optimizer to not specialize
> nested functions anymore. This simple change fixed the main
> performance issue. Reminder: in performance critical code, don't use
> nested functions! I will maybe propose patches for Lib/json/encoder.py
> to stop using nested functions.
>
> I only ran benchmarks with the optimizer enabled. I now have to
> measure the overhead of my patches (PEP 509, 510 and 511) adding the
> API fat AST optimizers. The overhead must be negligible. For me, it's
> a requirement of the whole project. Changes must not make Python
> slower when the optimizer is not used.
>
> fatoptimizer is faster on microbenchmarks, but I had to write manually
> some optimizations:
>
>    http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html
>
> IMHO fatoptimizer is not faster on macro benchmarks because it is not
> smart enough (yet) to generate the most interesting optimizations,
> like function inlining and specialization for argument types. You can
> estimate the speedup if you specialize manually your functions.
>
> --
>
> Barry also wrote: "Did you address my suggestion on python-ideas to
> make the new C API optionally compiled in?"
>
> Well, it is an option, but I would prefer to have the API for AST
> optimizer directly built in Python.
>
> The first beta version of Python 3.6 is scheduled in September 2016
> (deadline for new features in Python 3.6), so I still have a few
> months to implement more powerful optimizations and prove that it can
> be faster ;-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/gmludo%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160125/bd36a6a8/attachment.html>

From victor.stinner at gmail.com  Mon Jan 25 16:43:16 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 25 Jan 2016 22:43:16 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
Message-ID: <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>

Hi,

2016-01-25 22:20 GMT+01:00 Ludovic Gasc <gmludo at gmail.com>:
> Just thanks for this big contribution.
> And maybe this project could give new ideas to optimize Python, who knows ?

Sorry for my long email. I should try to summarize next time :-) In
short: FAT Python is not fast today, but it will be faster if you give
me a few more months to implement optimizations which will unlock the
real power of AST optimizers ;-)

I have a looooong list of ideas of optimizations:
https://fatoptimizer.readthedocs.org/en/latest/todo.html

According to microbenchmarks, the most promising optimizations are
functions inlining (Python function calls are slow :-/) and specialize
the code for the type of arguments.

And I agree to wait until fatoptimizer is proven to be faster than the
regular CPython before taking a decision of my 3 PEPs (509, 510, 511).

Victor

From srkunze at mail.de  Mon Jan 25 16:51:11 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Mon, 25 Jan 2016 22:51:11 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
Message-ID: <56A698CF.3080402@mail.de>

Hi Victor,

I encourage you to proceed here. I would love to see your PEPs (509-511) 
incorporated into CPython. It's not that I consider Python slow 
(although some folks claim so), but performance improvements are always 
welcome; especially when I glance over diagrams like those: 
http://blog.carlesmateo.com/wp-content/uploads/2014/10/blog-carlesmateo-com-performance-several-languages-php7-phantomjs-nodejs-java-bash-go-perl-luajit-hhvm3_9-scale_mod5.png

So, I join Barry when he says, we want more benchmarking and definite 
results, however, I might be less strict than he is and say:

- even if FAT might not optimize significantly (whatever definition we 
apply), PEP 509 to 511 are a great win for CPython
- they provide a great infrastructure for optimizing CPython AND 
extending/experimenting Python as an ecosystem
- FAT provides interesting insights into the field of optimizing a 
dynamic language

So, keep up the good work. I am eager to see where this goes.

If there's anything I can do, let me know. :)

Best,
Sven

On 25.01.2016 19:16, Victor Stinner wrote:
> Hi,
>
> Summary: FAT Python is not faster, but it will be ;-)
>
> --
>
> When I started the FAT Python as a fork of CPython 3.6, I put
> everything in the same repository. Last weeks, I focused on splitting
> my giant patch (10k lines) into small reviewable patches. I wrote 3
> PEP (509 dict version, 510 function specialziation, 511 code
> tranformers) and I enhanced the API to make it usable for more use
> cases than just FAT Python. I also created fatoptimizer (the AST
> optimizer) and fat (runtime dependency of the optimizer) projects on
> GitHub to separate clearly what should be outside Python core. For all
> links, see:
>
>     http://faster-cpython.readthedocs.org/fat_python.html
>
> For the fatoptimizer project, my constraint is to be able to run the
> full Python test suite unmodified. In practice, I have to disable some
> optimizations by putting a "__fatoptimizer__= {...}" configuration to
> some test files. For example, I have to disable constant folding on
> test_bool because it tests that False+2 gives 2 at runtime, whereas
> the optimizer replaces directly False+2 with 2 during the compilation.
> Well, test_bool.py is not the best example because all tests pass with
> the constant folding optimization (if I comment my
> "__fatoptimizer__={...}" change).
>
> This constraint ensures that the optimizer "works" and doesn't break
> (too much ;-)) the Python semantics, but it's more difficult to
> implement powerful optimizations.
>
> I also found and fixed various kinds of bugs. In my code obviously,
> but also in the Python core, in various places. Some bugs only concern
> AST transformers which is a new feature, but I had to fix them. For
> example, Python didn't support negative line number delta in
> co_lntotab of code objects, and so line number were all wrong on
> optimized code. I merged my enhancement in the default branch of
> CPython (issue #26107).
>
> In short, I focused on having something working (respecting the Python
> semantics), rather than spending time on writing optimizations.
>
> --
>
> When I asked explicitly "Is someone opposed to this PEP 509 [dict
> verion] ?", Barry Warsaw answered that a performance analysis is
> required. Extract of his mail:
>
>     "I still think this is maintenance and potential performance
> overhead we don't want to commit to long term unless it enables
> significant optimization.  Since you probably can't prove that without
> some experimentation, this API should be provisional."
>
> Last week, I ran some benchmarks and I have to admin that I was
> disappointed. Not only fatoptimizer doesn't make Python faster, but it
> makes it much slower on some tests!
>
>     http://fatoptimizer.readthedocs.org/en/latest/benchmarks.html
>
> Quickly, I identified a major performance issue when nested functions
> are specialized, especially in Lib/json/encoder.py (tested by
> bm_json_v2.py benchmark). I fixed my optimizer to not specialize
> nested functions anymore. This simple change fixed the main
> performance issue. Reminder: in performance critical code, don't use
> nested functions! I will maybe propose patches for Lib/json/encoder.py
> to stop using nested functions.
>
> I only ran benchmarks with the optimizer enabled. I now have to
> measure the overhead of my patches (PEP 509, 510 and 511) adding the
> API fat AST optimizers. The overhead must be negligible. For me, it's
> a requirement of the whole project. Changes must not make Python
> slower when the optimizer is not used.
>
> fatoptimizer is faster on microbenchmarks, but I had to write manually
> some optimizations:
>
>     http://fatoptimizer.readthedocs.org/en/latest/microbenchmarks.html
>
> IMHO fatoptimizer is not faster on macro benchmarks because it is not
> smart enough (yet) to generate the most interesting optimizations,
> like function inlining and specialization for argument types. You can
> estimate the speedup if you specialize manually your functions.
>
> --
>
> Barry also wrote: "Did you address my suggestion on python-ideas to
> make the new C API optionally compiled in?"
>
> Well, it is an option, but I would prefer to have the API for AST
> optimizer directly built in Python.
>
> The first beta version of Python 3.6 is scheduled in September 2016
> (deadline for new features in Python 3.6), so I still have a few
> months to implement more powerful optimizations and prove that it can
> be faster ;-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


From barry at python.org  Mon Jan 25 17:14:51 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 25 Jan 2016 17:14:51 -0500
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
Message-ID: <20160125171451.49a43f0a@subdivisions.wooz.org>

On Jan 25, 2016, at 07:16 PM, Victor Stinner wrote:

>Barry also wrote: "Did you address my suggestion on python-ideas to
>make the new C API optionally compiled in?"
>
>Well, it is an option, but I would prefer to have the API for AST
>optimizer directly built in Python.

In my plan, it would be, but it would have to be enabled with a configure
switch, and provisionally protected by an #ifdef.

>And I agree to wait until fatoptimizer is proven to be faster than the
>regular CPython before taking a decision of my 3 PEPs (509, 510, 511).

+1 - and just to be clear, I hope you succeed beyond your wildest
imagination. :)

Cheers,
-Barry

From abarnert at yahoo.com  Mon Jan 25 17:28:57 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Jan 2016 14:28:57 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
 <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>
Message-ID: <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>

On Jan 25, 2016, at 13:43, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> According to microbenchmarks, the most promising optimizations are
> functions inlining (Python function calls are slow :-/) and specialize
> the code for the type of arguments.

Can you specialize a function with a C API function, or only with bytecode? I'm not sure how much benefit you'd get out of specializing list vs. generic iterable or int vs. whatever from an AST transform, but substituting raw C code, on the other hand...

From brett at python.org  Mon Jan 25 17:34:20 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 25 Jan 2016 22:34:20 +0000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
 <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>
 <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>
Message-ID: <CAP1=2W64hwOEFoLmgRgbnreY5U=zvT0An7pfQkRscwvVdgCUmg@mail.gmail.com>

On Mon, 25 Jan 2016 at 14:30 Andrew Barnert via Python-Dev <
python-dev at python.org> wrote:

> On Jan 25, 2016, at 13:43, Victor Stinner <victor.stinner at gmail.com>
> wrote:
> >
> > According to microbenchmarks, the most promising optimizations are
> > functions inlining (Python function calls are slow :-/) and specialize
> > the code for the type of arguments.
>
> Can you specialize a function with a C API function, or only with
> bytecode? I'm not sure how much benefit you'd get out of specializing list
> vs. generic iterable or int vs. whatever from an AST transform, but
> substituting raw C code, on the other hand...
>

Victor's work is only manipulation of ASTs and bytecode. If you want
something that low-level you need to either reach for Cython or hope a
project like Pyjion pays off.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160125/fa881a4b/attachment-0001.html>

From victor.stinner at gmail.com  Mon Jan 25 17:46:45 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 25 Jan 2016 23:46:45 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
 <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>
 <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>
Message-ID: <CAMpsgwaYMfzu-5-tdwkX-Brew=cdJCjjigYgQESg6TZ_Xjspow@mail.gmail.com>

Hi,

2016-01-25 23:28 GMT+01:00 Andrew Barnert <abarnert at yahoo.com>:
> On Jan 25, 2016, at 13:43, Victor Stinner <victor.stinner at gmail.com> wrote:
>>
>> According to microbenchmarks, the most promising optimizations are
>> functions inlining (Python function calls are slow :-/) and specialize
>> the code for the type of arguments.
>
> Can you specialize a function with a C API function, or only with bytecode? I'm not sure how much benefit you'd get out of specializing list vs. generic iterable or int vs. whatever from an AST transform, but substituting raw C code, on the other hand...

As I wrote in the first part of my email, I redesigned to API to make
it more generic. One of my change was to change
PyFunction_Specialize() to not only accept code objects, but any
callable object.

The PEP 510 even contains an example using a builtin function as the
specialized code:
https://www.python.org/dev/peps/pep-0510/#using-builtin-function

"On a microbenchmark, calling the C builtin takes 95 ns, whereas the
original bytecode takes 155 ns (+60 ns): 1.6 times as fast. Calling
directly chr(65) takes 76 ns."

You can design an AST optimizer to compile some functions to C and
then register them as specialized code at runtime. I have a side
project to use Cython and/or pythran to specialize some functions
using type annotation on parameters.

Victor

From abarnert at yahoo.com  Mon Jan 25 17:50:31 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Jan 2016 14:50:31 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwaYMfzu-5-tdwkX-Brew=cdJCjjigYgQESg6TZ_Xjspow@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <CAON-fpEq+4Hp5Tb-akiCX8zPLAHuT4yxr9PdJCvzcP68yYHShA@mail.gmail.com>
 <CAMpsgwZ5k9_LeBWeB0pRQJ6SzxsW2SXBxfXEMe5NE0xGg1KyFg@mail.gmail.com>
 <E33584E8-3733-4FE8-951E-B253FB337431@yahoo.com>
 <CAMpsgwaYMfzu-5-tdwkX-Brew=cdJCjjigYgQESg6TZ_Xjspow@mail.gmail.com>
Message-ID: <505C9ADE-804D-4EA9-97D2-A67800A6F5FD@yahoo.com>

On Jan 25, 2016, at 14:46, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> You can design an AST optimizer to compile some functions to C and
> then register them as specialized code at runtime. I have a side
> project to use Cython and/or pythran to specialize some functions
> using type annotation on parameters.

That last part is exactly what I was thinking of. One way in which cythonizing your code isn't 100% compatible is that if you, say, shadow or replace int or range, the cython code is now wrong. Which is exactly the kind of thing FAT can guard against. Which is very cool. Glad to see you already thought of that before me. :)

From victor.stinner at gmail.com  Mon Jan 25 17:58:12 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 25 Jan 2016 23:58:12 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <56A698CF.3080402@mail.de>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
Message-ID: <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>

2016-01-25 22:51 GMT+01:00 Sven R. Kunze <srkunze at mail.de>:
> - they provide a great infrastructure for optimizing CPython AND
> extending/experimenting Python as an ecosystem

I hope that these API will create more optimizer projects than just
fatoptimizer.

For example, I expect more specialized optimizers like numba or
pythran which are very efficient but more specific (ex: numeric
computations) than fatoptimizer. Maybe not new optimizers, but just
glue to existing static compilers (numba, pythran, cython, etc.).


> If there's anything I can do, let me know. :)

Oh, they are a lot of things to do! My patches for PEP 509, 510 and
511 still need some love (reviews):

http://bugs.python.org/issue26058
http://bugs.python.org/issue26098
http://bugs.python.org/issue26145

I'm finishing my patch adding ast.Constant. This one is less
controversal, it has no impact on performance nor the Python
semantics:

http://bugs.python.org/issue26146


But these patches are boring C code. You may prefer to work on the
funny fatoptimizer project which is written in pure Python:

https://fatoptimizer.readthedocs.org/en/latest/

Victor

From songofacandy at gmail.com  Mon Jan 25 21:21:45 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 26 Jan 2016 11:21:45 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
Message-ID: <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>

I'm very interested in it.

Ruby 2.2 and PHP 7 are faster than Python 2.
Python 3 is slower than Python 2.
Performance is a attractive feature.  Python 3 lacks it.

How can I help your work?

On Tue, Jan 26, 2016 at 7:58 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> 2016-01-25 22:51 GMT+01:00 Sven R. Kunze <srkunze at mail.de>:
> > - they provide a great infrastructure for optimizing CPython AND
> > extending/experimenting Python as an ecosystem
>
> I hope that these API will create more optimizer projects than just
> fatoptimizer.
>
> For example, I expect more specialized optimizers like numba or
> pythran which are very efficient but more specific (ex: numeric
> computations) than fatoptimizer. Maybe not new optimizers, but just
> glue to existing static compilers (numba, pythran, cython, etc.).
>
>
> > If there's anything I can do, let me know. :)
>
> Oh, they are a lot of things to do! My patches for PEP 509, 510 and
> 511 still need some love (reviews):
>
> http://bugs.python.org/issue26058
> http://bugs.python.org/issue26098
> http://bugs.python.org/issue26145
>
> I'm finishing my patch adding ast.Constant. This one is less
> controversal, it has no impact on performance nor the Python
> semantics:
>
> http://bugs.python.org/issue26146
>
>
> But these patches are boring C code. You may prefer to work on the
> funny fatoptimizer project which is written in pure Python:
>
> https://fatoptimizer.readthedocs.org/en/latest/
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>



-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/30a490d9/attachment.html>

From brett at python.org  Mon Jan 25 21:34:43 2016
From: brett at python.org (Brett Cannon)
Date: Tue, 26 Jan 2016 02:34:43 +0000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
Message-ID: <CAP1=2W6-RtZ-gwCPAAhZisBF7joHmBDg8XqYjZdrbJ2goocAdw@mail.gmail.com>

On Mon, 25 Jan 2016 at 18:22 INADA Naoki <songofacandy at gmail.com> wrote:

> I'm very interested in it.
>
> Ruby 2.2 and PHP 7 are faster than Python 2.
> Python 3 is slower than Python 2.
> Performance is a attractive feature.  Python 3 lacks it.
>

That is not a fair statement to make about Python 3. It entirely depends on
your workload whether it is faster or slower.

https://gist.github.com/brettcannon/9d19cc184ea45b3e7ca0

-Brett


>
> How can I help your work?
>
> On Tue, Jan 26, 2016 at 7:58 AM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> 2016-01-25 22:51 GMT+01:00 Sven R. Kunze <srkunze at mail.de>:
>> > - they provide a great infrastructure for optimizing CPython AND
>> > extending/experimenting Python as an ecosystem
>>
>> I hope that these API will create more optimizer projects than just
>> fatoptimizer.
>>
>> For example, I expect more specialized optimizers like numba or
>> pythran which are very efficient but more specific (ex: numeric
>> computations) than fatoptimizer. Maybe not new optimizers, but just
>> glue to existing static compilers (numba, pythran, cython, etc.).
>>
>>
>> > If there's anything I can do, let me know. :)
>>
>> Oh, they are a lot of things to do! My patches for PEP 509, 510 and
>> 511 still need some love (reviews):
>>
>> http://bugs.python.org/issue26058
>> http://bugs.python.org/issue26098
>> http://bugs.python.org/issue26145
>>
>> I'm finishing my patch adding ast.Constant. This one is less
>> controversal, it has no impact on performance nor the Python
>> semantics:
>>
>> http://bugs.python.org/issue26146
>>
>>
>> But these patches are boring C code. You may prefer to work on the
>> funny fatoptimizer project which is written in pure Python:
>>
>> https://fatoptimizer.readthedocs.org/en/latest/
>>
>> Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>>
> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
>>
>
>
>
> --
> INADA Naoki  <songofacandy at gmail.com>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/b6fdb245/attachment.html>

From abarnert at yahoo.com  Mon Jan 25 22:02:52 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Jan 2016 19:02:52 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
Message-ID: <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>

On Jan 25, 2016, at 18:21, INADA Naoki <songofacandy at gmail.com> wrote:
> 
> I'm very interested in it.
> 
> Ruby 2.2 and PHP 7 are faster than Python 2.
> Python 3 is slower than Python 2.

Says who?

That was certainly true in the 3.2 days, but nowadays, most things that differ seem to be faster in 3.x. Maybe it's just the kinds of programs I write, but speedup in decoding UTF-8 that's usually ASCII (and then processing the decoded unicode when it's usually 1/4th the size), faster listcomps, and faster datetime seem to matter more than slower logging or slower imports. And that's just when running the same code; when you actually use new features, yield from is much faster than looping over yield; scandir blows away listdir; asyncio blows away asyncore or threading even harder; etc.

Maybe if you do different things, you have a different experience. But if you have a specific problem, you'd do a lot better to file specific bugs for that problem than to just hope that everything magically gets so much faster that your bottleneck no longer matters.

> Performance is a attractive feature.  Python 3 lacks it.

When performance matters, people don't use Python 2, Ruby, or PHP, any more than they use Python 3. Or, rather, they use _any_ of those languages for the 95% of their code that doesn't matter, and C (often through existing libraries like NumPy--and try to find a good equivalent of that for Ruby or PHP) for the 5% that does.

From songofacandy at gmail.com  Mon Jan 25 22:32:07 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 26 Jan 2016 12:32:07 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
Message-ID: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>

On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Jan 25, 2016, at 18:21, INADA Naoki <songofacandy at gmail.com> wrote:
> >
> > I'm very interested in it.
> >
> > Ruby 2.2 and PHP 7 are faster than Python 2.
> > Python 3 is slower than Python 2.
>
> Says who?
>

For example, http://benchmarksgame.alioth.debian.org/u64q/php.html
In Japanese, many people compares language performance by microbench like
fibbonacci.


>
> That was certainly true in the 3.2 days, but nowadays, most things that
> differ seem to be faster in 3.x.


Python is little faster than ever in these years.
But PHP and Ruby are much more faster than these years.

Matz announced Ruby 3x3. Ruby hackers will make more effort to optimize
ruby.
http://engineering.appfolio.com/appfolio-engineering/2015/11/18/ruby-3x3



> Maybe it's just the kinds of programs I write, but speedup in decoding
> UTF-8 that's usually ASCII (and then processing the decoded unicode when
> it's usually 1/4th the size), faster listcomps, and faster datetime seem to
> matter more than slower logging or slower imports. And that's just when
> running the same code; when you actually use new features, yield from is
> much faster than looping over yield; scandir blows away listdir; asyncio
> blows away asyncore or threading even harder; etc.
>

I know.
But people compares language speed by simple microbench like fibbonacci.
They doesn't use listcomp or libraries to compare *language* speed.


> Maybe if you do different things, you have a different experience. But if
> you have a specific problem, you'd do a lot better to file specific bugs
> for that problem than to just hope that everything magically gets so much
> faster that your bottleneck no longer matters.
>

I did it sometimes.
But I'd like to base language performance like function call more faster.


>
> > Performance is a attractive feature.  Python 3 lacks it.
>
> When performance matters, people don't use Python 2, Ruby, or PHP, any
> more than they use Python 3. Or, rather, they use _any_ of those languages
> for the 95% of their code that doesn't matter, and C (often through
> existing libraries like NumPy--and try to find a good equivalent of that
> for Ruby or PHP) for the 5% that does.


In the case of Web devs, many people choose main language from PHP, Ruby
and Python.
When peformance matters, they choose sub language from node.js, Go and
Scala.

While performance is not a matter when choosing first language, slowest of
three makes bad impression
and people feel less attractive about Python.

-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/409a5159/attachment.html>

From rymg19 at gmail.com  Mon Jan 25 22:34:35 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 25 Jan 2016 21:34:35 -0600
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
Message-ID: <8E5CCA97-D191-408A-8925-6A598397C622@gmail.com>



On January 25, 2016 9:32:07 PM CST, INADA Naoki <songofacandy at gmail.com> wrote:
>On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert <abarnert at yahoo.com>
>wrote:
>
>> On Jan 25, 2016, at 18:21, INADA Naoki <songofacandy at gmail.com>
>wrote:
>> >
>> > I'm very interested in it.
>> >
>> > Ruby 2.2 and PHP 7 are faster than Python 2.
>> > Python 3 is slower than Python 2.
>>
>> Says who?
>>
>
>For example, http://benchmarksgame.alioth.debian.org/u64q/php.html
>In Japanese, many people compares language performance by microbench
>like
>fibbonacci.
>

...does writing Fibonacci in a foreign language make a performance difference? Or did you mean "In Japan?"

>
>>
>> That was certainly true in the 3.2 days, but nowadays, most things
>that
>> differ seem to be faster in 3.x.
>
>
>Python is little faster than ever in these years.
>But PHP and Ruby are much more faster than these years.
>
>Matz announced Ruby 3x3. Ruby hackers will make more effort to optimize
>ruby.
>http://engineering.appfolio.com/appfolio-engineering/2015/11/18/ruby-3x3
>
>
>
>> Maybe it's just the kinds of programs I write, but speedup in
>decoding
>> UTF-8 that's usually ASCII (and then processing the decoded unicode
>when
>> it's usually 1/4th the size), faster listcomps, and faster datetime
>seem to
>> matter more than slower logging or slower imports. And that's just
>when
>> running the same code; when you actually use new features, yield from
>is
>> much faster than looping over yield; scandir blows away listdir;
>asyncio
>> blows away asyncore or threading even harder; etc.
>>
>
>I know.
>But people compares language speed by simple microbench like
>fibbonacci.
>They doesn't use listcomp or libraries to compare *language* speed.
>
>
>> Maybe if you do different things, you have a different experience.
>But if
>> you have a specific problem, you'd do a lot better to file specific
>bugs
>> for that problem than to just hope that everything magically gets so
>much
>> faster that your bottleneck no longer matters.
>>
>
>I did it sometimes.
>But I'd like to base language performance like function call more
>faster.
>
>
>>
>> > Performance is a attractive feature.  Python 3 lacks it.
>>
>> When performance matters, people don't use Python 2, Ruby, or PHP,
>any
>> more than they use Python 3. Or, rather, they use _any_ of those
>languages
>> for the 95% of their code that doesn't matter, and C (often through
>> existing libraries like NumPy--and try to find a good equivalent of
>that
>> for Ruby or PHP) for the 5% that does.
>
>
>In the case of Web devs, many people choose main language from PHP,
>Ruby
>and Python.
>When peformance matters, they choose sub language from node.js, Go and
>Scala.
>
>While performance is not a matter when choosing first language, slowest
>of
>three makes bad impression
>and people feel less attractive about Python.

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

From rosuav at gmail.com  Mon Jan 25 22:59:36 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 26 Jan 2016 14:59:36 +1100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
Message-ID: <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>

On Tue, Jan 26, 2016 at 2:32 PM, INADA Naoki <songofacandy at gmail.com> wrote:
>
> I know.
> But people compares language speed by simple microbench like fibbonacci.
> They doesn't use listcomp or libraries to compare *language* speed.
>

Well, that's a stupid way to decide on a language. Here, look: Python
is faster than C. Proof!

rosuav at sikorsky:~$ time python3 fib.py
2880067194370816120

real 0m0.033s
user 0m0.032s
sys 0m0.000s
rosuav at sikorsky:~$ cat fib.py
import functools

@functools.lru_cache()
def fib(n):
    if n < 2: return n
    return fib(n-2) + fib(n-1)

print(fib(90))

rosuav at sikorsky:~$ gcc fib.c && time ./a.out
1134903170

real 0m9.104s
user 0m9.064s
sys 0m0.000s
rosuav at sikorsky:~$ cat fib.c
#include <stdio.h>

unsigned long fib(unsigned long n)
{
    if (n < 2) return n;
    return fib(n-2) + fib(n-1);
}

int main()
{
    printf("%lu\n",fib(45));
}


Algorithmic differences - even subtle ones - can easily outdo choice
of language for run-time performance. And if you try to write a true C
equivalent of that Python code, good luck - I'll have the Python one
written and running while you're still trying to figure out how to
write a cache, much less how to keep the syntax clean as you add a
cache to an existing function.

Of course, rewriting the whole thing to work iteratively instead of
double-recursively will make a dramatic difference to both programs.
That's an unsubtle algorithmic difference, and if you're coding like
that, you probably can't see the difference in performance between any
two languages at anything up to a machine word (about the 90th or so
Fibonacci number, on a 64-bit system) - all you'll see is the startup
performance. As soon as you go beyond a machine word, Python massively
trumps C, because its default integer type is a bignum. Going beyond a
machine word in C is a hassle. Going beyond a machine word in Python 2
is almost insignificant - hey look, now your repr has an 'L' on the
end, and performance is immeasurably worse. In Python 3, there's no
machine word whatsoever.

So, yeah... Python beats C for Fibonacci calculation, too. You just
have to not be stupid with your benchmarking.

ChrisA

From songofacandy at gmail.com  Tue Jan 26 00:02:31 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 26 Jan 2016 14:02:31 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
Message-ID: <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>

Do you say I and many people are so fool?
People use same algorithm on every language when compares base language
performance [1].

[1] There are no solid definition about "Base language performance".
   But it includes function call, method lookup, GC.  It may include basic
string and arithmetic operations.


See here for example:
http://d.hatena.ne.jp/satosystems/20121228/1356655565

This article is written in 2012.
In this article, php 5.3 takes 85sec, Python 2.7 takes 53sec and CRuby 1.8
takes 213sec. (!!)

For now:

$ python2 -V
Python 2.7.11
$ time python2 -S fib.py
39088169

real 0m17.133s
user 0m16.970s
sys 0m0.055s

$ python3 -V
Python 3.5.1
$ time python3 -S fib.py
39088169

real 0m21.380s
user 0m21.337s
sys 0m0.028s

$ php -v
PHP 7.0.2 (cli) (built: Jan  7 2016 10:40:21) ( NTS )
Copyright (c) 1997-2015 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2015 Zend Technologies
$ time php fib.php
39088169

real 0m7.706s
user 0m7.654s
sys 0m0.027s

$ ruby -v
ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin14]
$ time ruby fib.rb
39088169

real 0m6.195s
user 0m6.124s
sys 0m0.032s


Fibonacci microbench measures performance of function call.
When I said "Base language performance", I meant performance of
function call, attribute lookup, GC, etc...

PHP and Ruby made grate effort to improve base language performance.
While I'm fan of Python,  I respect people made PHP and Ruby faster.

Of course, I respect people making Python faster too.
But I wonder if CPython is more faster, especially about global lookup and
function call.

-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/30442262/attachment.html>

From abarnert at yahoo.com  Tue Jan 26 00:44:07 2016
From: abarnert at yahoo.com (Andrew Barnert)
Date: Mon, 25 Jan 2016 21:44:07 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
Message-ID: <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com>

On Jan 25, 2016, at 19:32, INADA Naoki <songofacandy at gmail.com> wrote:
> 
>> On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert <abarnert at yahoo.com> wrote:
>> On Jan 25, 2016, at 18:21, INADA Naoki <songofacandy at gmail.com> wrote:
>> >
>> > I'm very interested in it.
>> >
>> > Ruby 2.2 and PHP 7 are faster than Python 2.
>> > Python 3 is slower than Python 2.
>> 
>> Says who?
> 
> For example, http://benchmarksgame.alioth.debian.org/u64q/php.html
> In Japanese, many people compares language performance by microbench like fibbonacci.

"In Japan, the hand is sharper than a knife [man splits board with karate chop], but the same doesn't work with a tomato [man splatters tomato all over himself with karate chop]."

A cheap knife really is better than a karate master at chopping tomatoes. And Python 2 really is better than Python 3 at doing integer arithmetic on the edge of what can fit into a machine word. But so what? Without seeing any of your Japanese web code, much less running a profiler, I'm willing to bet that your code is rarely CPU-bound, and, when it is, it spends a lot more time doing things like processing Unicode strings that are almost always UCS-2 (about 110% slower on Python 2) than doing this kind of arithmetic (9% faster on Python 2), or cutting tomatoes (TypeError on both versions).

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160125/a3a387ad/attachment.html>

From songofacandy at gmail.com  Tue Jan 26 01:13:19 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 26 Jan 2016 15:13:19 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <9A44AFB5-CE53-4C5B-9A49-6E85F9C148A2@yahoo.com>
Message-ID: <CAEfz+TzyumsK0hsvL0sO3_rWGFkxu13TD1O08eXMxo8-d1L=sA@mail.gmail.com>

On Tue, Jan 26, 2016 at 2:44 PM, Andrew Barnert <abarnert at yahoo.com> wrote:

> On Jan 25, 2016, at 19:32, INADA Naoki <songofacandy at gmail.com> wrote:
>
> On Tue, Jan 26, 2016 at 12:02 PM, Andrew Barnert <abarnert at yahoo.com>
> wrote:
>
>> On Jan 25, 2016, at 18:21, INADA Naoki <songofacandy at gmail.com> wrote:
>> >
>> > I'm very interested in it.
>> >
>> > Ruby 2.2 and PHP 7 are faster than Python 2.
>> > Python 3 is slower than Python 2.
>>
>> Says who?
>>
>
> For example, http://benchmarksgame.alioth.debian.org/u64q/php.html
> In Japanese, many people compares language performance by microbench like
> fibbonacci.
>
>
> "In Japan, the hand is sharper than a knife [man splits board with karate
> chop], but the same doesn't work with a tomato [man splatters tomato all
> over himself with karate chop]."
>
> A cheap knife really is better than a karate master at chopping tomatoes.
> And Python 2 really is better than Python 3 at doing integer arithmetic on
> the edge of what can fit into a machine word. But so what? Without seeing
> any of your Japanese web code, much less running a profiler, I'm willing to
> bet that your code is rarely CPU-bound, and, when it is, it spends a lot
> more time doing things like processing Unicode strings that are almost
> always UCS-2 (about 110% slower on Python 2) than doing this kind of
> arithmetic (9% faster on Python 2), or cutting tomatoes (TypeError on both
> versions).
>
>
Calm down, please.
I didn't say "microbench is more important than macrobench".

While editor is not a main problem of software development, people likes
comparing vim and emacs.
Like that, Japanese dev people likes comparing speed.

While it's not a real problem of typical application, new people should
choose first (and probably main)
editor and language.
Slowest on such a basic microbench gives bad impression for them.

Additionally, some application (e.g. traversing DOM) makes much function
calls.
Faster function call may makes some *real* application faster.

-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/1a2ed9c3/attachment.html>

From stephen at xemacs.org  Tue Jan 26 02:16:15 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 26 Jan 2016 16:16:15 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
Message-ID: <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp>

INADA Naoki writes:

 > For example, http://benchmarksgame.alioth.debian.org/u64q/php.html
 > In Japanese, many people compares language performance by
 > microbench like fibbonacci.

True enough.  But as a teacher in a Japanese engineering school, I am
ashamed to see that posted to a public list.  We Japanese ;-) need to
get over that tendency (and not just in programming, my own biggest
battle is against those who worship goodness-of-fit statistics).  Our
universities are doing an awful job at getting "big picture thinking"
across to our students.

I see that you respond that you're talking about "base language
performance" which is a big improvement over blind faith in micro-
benchmarks, but still it's a matter of pulling your nose back from the
bark so you can see the tree.  At least from the point of view of the
computations the people around me are doing with these languages, you
still miss the forest.

 > While performance is not a matter when choosing first language,
 > slowest of three makes bad impression and people feel less
 > attractive about Python.

My school is the one that Matz graduated from (different department),
and as far as I can see (which is pretty local, I admit), Python is a
clear first and growing as a teaching language and as a glue language
for research computations.  Java is second for both[1], Ruby is rare,
and PHP just isn't on the map.  I see some lab home pages using
Ruby-on-Rails, and a couple of Mediawiki installations and the like
based on PHP.

For me personally, PHP and Ruby are both unusable for *performance*
reasons: I need something like Pandas, which is based on NumPy, and at
least the last time I checked a couple years ago Ruby had no
equivalent to NumPy, let alone Pandas.  (For PHP I asked the nearest
PHP fanboy web programmer, so I could be way wrong about no efficient
computation for PHP.)

At least as used at this university, I just don't see a performance
issue that Python needs to be specifically worried about.

That said, optimization is important, and IMO anything that allows 1%
of people currently writing (and debugging ...) modules in C for
performance reasons to write them in pure Python instead is a win.
But to me, relieving programmer pain, not microbenchmarks, should be
the performance measure we care about (even if others do care about
those benchmarks).  Measure them, yes, but focus on them only when
we're pretty sure that poor benchmark numbers are associated with
programmer pain.


Footnotes: 
[1]  Java was clearly first as a teaching language until 2010 or so, and
since most of our grad students and assistant profs were students here
in that era, I suspect its persistence is just the inertia that goes
with academic nepotism.


From ncoghlan at gmail.com  Tue Jan 26 05:06:03 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 26 Jan 2016 20:06:03 +1000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7fSV1aVS2mTbPZVZ_uaxcz5mZq5ue_J8yDszQpcRBbzmw@mail.gmail.com>

On 26 January 2016 at 17:16, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Our universities are doing an awful job at getting "big picture thinking"
> across to our students.

That problem isn't specific to Japan - I'm not aware of *anywhere*
that does a particularly good job of teaching developers not to get
tribal about their programming language choice, and instead evaluate
their options based on their specific problem, the team that will be
working on it, and the pre-existing problem solving tools available in
that ecosystem.

As a result, folks making programming language choices based on
criteria that aren't actually relevant to the problem they're trying
to solve is going to be a fact of life. While improving those kinds of
metrics isn't a *reason* to do anything, it does count as an added
bonus when it comes as a beneficial side effect of working on
something else (such as the function specialisation infrastructure
underpinning Victor's optimiser project).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Tue Jan 26 08:38:13 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 26 Jan 2016 08:38:13 -0500
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
Message-ID: <n87ssb$m4j$1@ger.gmane.org>

On 1/26/2016 12:02 AM, INADA Naoki wrote:

> People use same algorithm on every language when compares base language
> performance [1].

The python code is NOT using the same algorithm.  The proof is that the 
Python function will return the correct value for, say fib(50) while 
most if not all the other versions will not.  The domain of an algorithm 
is part of what characterizes an algorithm.

-- 
Terry Jan Reedy


From rymg19 at gmail.com  Tue Jan 26 10:28:42 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 26 Jan 2016 09:28:42 -0600
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
Message-ID: <C95FE5C9-69AD-427A-84DE-D0485C45CA78@gmail.com>



On January 25, 2016 9:59:36 PM CST, Chris Angelico <rosuav at gmail.com> wrote:
>On Tue, Jan 26, 2016 at 2:32 PM, INADA Naoki <songofacandy at gmail.com>
>wrote:
>>
>> I know.
>> But people compares language speed by simple microbench like
>fibbonacci.
>> They doesn't use listcomp or libraries to compare *language* speed.
>>
>
>Well, that's a stupid way to decide on a language. Here, look: Python
>is faster than C. Proof!
>
>rosuav at sikorsky:~$ time python3 fib.py
>2880067194370816120
>
>real 0m0.033s
>user 0m0.032s
>sys 0m0.000s
>rosuav at sikorsky:~$ cat fib.py
>import functools
>
>@functools.lru_cache()
>def fib(n):
>    if n < 2: return n
>    return fib(n-2) + fib(n-1)
>
>print(fib(90))
>
>rosuav at sikorsky:~$ gcc fib.c && time ./a.out
>1134903170
>
>real 0m9.104s
>user 0m9.064s
>sys 0m0.000s
>rosuav at sikorsky:~$ cat fib.c
>#include <stdio.h>
>
>unsigned long fib(unsigned long n)
>{
>    if (n < 2) return n;
>    return fib(n-2) + fib(n-1);
>}
>
>int main()
>{
>    printf("%lu\n",fib(45));
>}
>

*cough* -O3 *cough*

>
>Algorithmic differences - even subtle ones - can easily outdo choice
>of language for run-time performance. And if you try to write a true C
>equivalent of that Python code, good luck - I'll have the Python one
>written and running while you're still trying to figure out how to
>write a cache, much less how to keep the syntax clean as you add a
>cache to an existing function.
>
>Of course, rewriting the whole thing to work iteratively instead of
>double-recursively will make a dramatic difference to both programs.
>That's an unsubtle algorithmic difference, and if you're coding like
>that, you probably can't see the difference in performance between any
>two languages at anything up to a machine word (about the 90th or so
>Fibonacci number, on a 64-bit system) - all you'll see is the startup
>performance. As soon as you go beyond a machine word, Python massively
>trumps C, because its default integer type is a bignum. Going beyond a
>machine word in C is a hassle. Going beyond a machine word in Python 2
>is almost insignificant - hey look, now your repr has an 'L' on the
>end, and performance is immeasurably worse. In Python 3, there's no
>machine word whatsoever.
>
>So, yeah... Python beats C for Fibonacci calculation, too. You just
>have to not be stupid with your benchmarking.
>
>ChrisA
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.

From victor.stinner at gmail.com  Tue Jan 26 10:32:45 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 26 Jan 2016 16:32:45 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
Message-ID: <CAMpsgwZ0vy4A0N6RtCQc77B2hT-ijEwvvbK2bAiAJcfZ_MFk6Q@mail.gmail.com>

Hi,

2016-01-26 3:21 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
> How can I help your work?

I don't know exactly yet, but I started to write a documentation to
explain how to contribute:
http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute

You may contact me directly ;-)

Victor

From rosuav at gmail.com  Tue Jan 26 10:40:08 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 27 Jan 2016 02:40:08 +1100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <C95FE5C9-69AD-427A-84DE-D0485C45CA78@gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <C95FE5C9-69AD-427A-84DE-D0485C45CA78@gmail.com>
Message-ID: <CAPTjJmpgJtttbSEux6ro+=cRJr5EDdUZCnMEsWaTum0Z51=Ntw@mail.gmail.com>

On Wed, Jan 27, 2016 at 2:28 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>rosuav at sikorsky:~$ gcc fib.c && time ./a.out
>>1134903170
>>
>>real 0m9.104s
>>user 0m9.064s
>>sys 0m0.000s
>>rosuav at sikorsky:~$ cat fib.c
>>#include <stdio.h>
>>
>>unsigned long fib(unsigned long n)
>>{
>>    if (n < 2) return n;
>>    return fib(n-2) + fib(n-1);
>>}
>>
>>int main()
>>{
>>    printf("%lu\n",fib(45));
>>}
>>
>
> *cough* -O3 *cough*
>

Oh, I'm sorry. Let's try it again.

rosuav at sikorsky:~$ gcc -O3 fib.c && time ./a.out
1134903170

real 0m3.153s
user 0m3.088s
sys 0m0.052s

Cool! Seems to be linear. From which we can deduce that Python on my
system was compiled at about -O275.

ChrisA

From srkunze at mail.de  Tue Jan 26 12:10:59 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Tue, 26 Jan 2016 18:10:59 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwZ0vy4A0N6RtCQc77B2hT-ijEwvvbK2bAiAJcfZ_MFk6Q@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <CAMpsgwZ0vy4A0N6RtCQc77B2hT-ijEwvvbK2bAiAJcfZ_MFk6Q@mail.gmail.com>
Message-ID: <56A7A8A3.5040705@mail.de>

Hi,

will look into it soon. :)

Best,
Sven

On 26.01.2016 16:32, Victor Stinner wrote:
> Hi,
>
> 2016-01-26 3:21 GMT+01:00 INADA Naoki <songofacandy at gmail.com>:
>> How can I help your work?
> I don't know exactly yet, but I started to write a documentation to
> explain how to contribute:
> http://faster-cpython.readthedocs.org/fat_python.html#how-can-you-contribute
>
> You may contact me directly ;-)
>
> Victor


From srkunze at mail.de  Tue Jan 26 12:35:56 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Tue, 26 Jan 2016 18:35:56 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
Message-ID: <56A7AE7C.3080806@mail.de>

I completely agree with INADA.

It's like saying, because a specific crossroad features a higher 
accident rate, *people need to change their driving behavior*.
*No!* People won't change and it's not necessary either. The crossroad 
needs to be changed to be safer.


Same goes for Python. If it's slow using the very same piece of code 
(even superficially), you better make the language faster.
Developers won't change and they won't change their code either. Just 
not necessary.


Btw. it would be a great feature for Python 3 to be faster than Python 
2. I've heard a lot of complaints of the scientific community that 
Python is slow. Would Python 3 be significantly faster than Python 2, 
that'll be a huge reason to upgrade (and would create pressure to 
upgrade libs as well).
They are satisfied with Python so far, but would there be a language 
equally readable/maintainable and 10x faster (of course proven by some 
weird micro benchmarks - incomprehensible to most nervous subscribers to 
this list), they would readily switch over. I for one hope that *Python 
itself will be that language* in the foreseeable future. This is some 
sort of marketing but also requires hard facts indeed.

Best,
Sven

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/1604d951/attachment.html>

From stephen at xemacs.org  Tue Jan 26 12:50:13 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Jan 2016 02:50:13 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CADiSq7fSV1aVS2mTbPZVZ_uaxcz5mZq5ue_J8yDszQpcRBbzmw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <22183.7487.315172.752001@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7fSV1aVS2mTbPZVZ_uaxcz5mZq5ue_J8yDszQpcRBbzmw@mail.gmail.com>
Message-ID: <22183.45525.814657.897955@turnbull.sk.tsukuba.ac.jp>

Nick Coghlan writes:
 > On 26 January 2016 at 17:16, Stephen J. Turnbull <stephen at xemacs.org> wrote:

 > > Our universities are doing an awful job at getting "big picture
 > > thinking" across to our students.
 > 
 > That problem isn't specific to Japan - I'm not aware of *anywhere*
 > that does a particularly good job of teaching developers not to get
 > tribal about their programming language choice,

But that's a different issue.  The approach that Naoki describes isn't
"tribal", in fact it's the exact opposite: it's an attempt to base
such decisions on strictly objective measures.

 > and instead evaluate their options based on their specific problem,
 > the team that will be working on it, and the pre-existing problem
 > solving tools available in that ecosystem.

One of which is the language itself, and the team's experience with
it.  "We're a C++/Java/Python/Ruby/Brainf!ck/assembler/COBOL shop"
isn't a bad heuristic in most cases -- outside of research and/or "we
also walk dogs" generalist consultancies -- especially when you're
under pressure from the bean-counting department to reduce costs.
That heuristic is hard to distinguish from "tribalism", though.

AFAICT, in fact, the generalists (including massive entities like IBM
and Google, as well as companies like Red Hat and cooperatives like
Debian) are quite good at the kind of evaluation you describe.

And Python has been quite good at targeting the kinds of improvements
that make it appealing to people who can and do do that kind of
evaluation, in more and more areas.

 > While improving those kinds of metrics isn't a *reason* to do
 > anything, it does count as an added bonus when it comes as a
 > beneficial side effect of working on something else (such as the
 > function specialisation infrastructure underpinning Victor's
 > optimiser project).

Sure, but that's a "have your cake and eat it too" situation.
Nobody's going to complain about that!

If Naoki -- or anybody else -- wants to work on optimizations enabled
by FAT Python, more power to them, I say.  I just would like to see
them reviewed with the goal of making Python a better language for
solving a wide variety of problems, rather than strictly focusing on
benchmarks.  If the benchmarks can be improved without closing off
potential syntax improvements or restricting the domain of algorithms
(cf Terry's comment), wonderful!

I thought Chris's point about efficient algorithms that would be hard
to implement correctly in other languages but are easy to do in Python
*because* of Python's carefully crafted, but not always maximally
speedy, semantics was especially apropos here.  Of course his claim
that Python is faster than C is tongue in cheek, and a caching version
of fibonacci wouldn't be that hard to write in C, and an iterative
version even easier.  But others have pointed out many other syntaxes
(comprehensions, generators, yield from, and so on) that put together
often make efficient computation TOOWTDI.  That, to me, is The Python
Advantage.




From stephen at xemacs.org  Tue Jan 26 12:51:54 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 27 Jan 2016 02:51:54 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <n87ssb$m4j$1@ger.gmane.org>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
 <n87ssb$m4j$1@ger.gmane.org>
Message-ID: <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp>

Terry Reedy writes:
 > On 1/26/2016 12:02 AM, INADA Naoki wrote:
 > 
 > > People use same algorithm on every language when compares base language
 > > performance [1].
 > 
 > The python code is NOT using the same algorithm.  The proof is that the 
 > Python function will return the correct value for, say fib(50) while 
 > most if not all the other versions will not.

True, but that's not a reasonable criterion for "same algorithm" in
this context.  Naoki's application ("base language performance"
benchmarking) requires fib(n) only for n < 40, and run it in a loop
100 times if you want 2 more decimal places of precision ("40" is
appropriate for an implementation with 32-bit ints).  On that
restricted domain the algorithm *is* the same.

If you want to argue that the bigger domain is a better one to use for
evaluating programming languages, be our guest.  But then you're
comparing apples (speed) to oranges (domain), and Naoki (or the
Japanese benchmarkers) can argue that a smaller, more risky, domain is
covered by "consenting adults" -- if you know there's a risk, you need
to write code to deal with it, but if you know there isn't, you
shouldn't have to accept lower performance.

Obviously, I don't think that's an appropriate tradeoff myself, but
that's based on "IMHO" not "comparison is invalid because algorithms
differ".


From truong_nguyen8 at yahoo.com  Tue Jan 26 18:18:08 2016
From: truong_nguyen8 at yahoo.com (Truong Nguyen)
Date: Tue, 26 Jan 2016 23:18:08 +0000 (UTC)
Subject: [Python-Dev] Hoping to Find a Mentor
References: <970197771.523029.1453850288839.JavaMail.yahoo.ref@mail.yahoo.com>
Message-ID: <970197771.523029.1453850288839.JavaMail.yahoo@mail.yahoo.com>

Dear Everyone,
My name is Truong Nguyen and I like web development. I'm writing hoping to find a mentor (who likes to teach), and opportunity to contribute to the python.org website to gain skills in full stack development. Python Software Foundation interest me because the language is used in a wide variety of fields.
My background is not in computer science, but I've earned a B.S. in Applied Mathematics and have an equivalent minor in computer science. I did front end project with HTML, CSS3 & Media Query, and JavaScript myself, but hope to work with someone to learn from.
I'm a hard worker and responsible person and do my best to contribute to the organization.
Have a nice day,
Truong Nguyen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/75da2883/attachment.html>

From brett at snarky.ca  Tue Jan 26 12:31:16 2016
From: brett at snarky.ca (Brett Cannon)
Date: Tue, 26 Jan 2016 17:31:16 +0000
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (cbd4a6a2657e): sum=134
In-Reply-To: <20160126084734.122727.74008@psf.io>
References: <20160126084734.122727.74008@psf.io>
Message-ID: <CAP1=2W6THMRWB_n=krdSCMCcnTd-rmFS71S+PCzUrQM4zNEuwg@mail.gmail.com>

Looks like Victor's ast.Constant change introduced a refleak.

On Tue, 26 Jan 2016 at 00:47 <solipsis at pitrou.net> wrote:

> results for cbd4a6a2657e on branch "default"
> --------------------------------------------
>
> test_ast leaked [39, 39, 39] references, sum=117
> test_ast leaked [5, 5, 5] memory blocks, sum=15
> test_collections leaked [-2, 0, 0] references, sum=-2
> test_functools leaked [0, 2, 2] memory blocks, sum=4
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/psf-users/antoine/refleaks/reflogqIZGVY', '--timeout', '7200']
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160126/632156a4/attachment.html>

From sjoerdjob at sjec.nl  Wed Jan 27 00:23:41 2016
From: sjoerdjob at sjec.nl (Sjoerd Job Postmus)
Date: Wed, 27 Jan 2016 06:23:41 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
Message-ID: <20160127052341.GA14190@sjoerdjob.com>

On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote:
> ...
> Oh, they are a lot of things to do! ...

Just wondering, do you also need a set of (abusive) test-cases which
check 100% conformity to the CPython semantics? I'm sure many of us
would be able to whip up some ideas of things that are possible with
Python and are of the kind "but you should not do that! That's bad
programming!" which may or may not break the optimizations (especially
specialized functions).

I'm thinking of things like

    def override_length_function_test():
        global len
        value = len("abc")
	len = lambda obj: ord(obj[0]))
	value += len("abc")
	assert value == 3 + 97, "Outdated `len` used."

And also cases where `len` was changed not in the function itself, but
in another function that was called indirectly (maybe 4 functions down,
one of which was monkey-patched in after the compilation step):

    module_a.py
    def test_func(callback):
        value = len("abc")
	callback()
	value += len("abc")
	assert value == 3 + 97, "Outdated `len` used."

    module_b.py
    import module_a
    def override_length_function_test():
        def callback():
	    module_a.len = lambda obj: ord(obj[0])
	assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used."

(I'm sure some of the other readers of this list can be even more
creative in trying to show that making optimizations like this can break
semantics.)

Other fun tricks I'd like to try is overriding the `len` method from a
signal handler, what happens when you monkey-patch a dependent method,
having `__getitem__` and `__getattr__` on some value override `len`.

Basically: trying things that I normally should not try during my
working hours, on account of wanting to still have a job the next day.

Kind regards,
Sjoerd Job

From victor.stinner at gmail.com  Wed Jan 27 03:09:36 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 27 Jan 2016 09:09:36 +0100
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
 (cbd4a6a2657e): sum=134
In-Reply-To: <CAP1=2W6THMRWB_n=krdSCMCcnTd-rmFS71S+PCzUrQM4zNEuwg@mail.gmail.com>
References: <20160126084734.122727.74008@psf.io>
 <CAP1=2W6THMRWB_n=krdSCMCcnTd-rmFS71S+PCzUrQM4zNEuwg@mail.gmail.com>
Message-ID: <CAMpsgwZWOxyXrqsXWAy8t5-DbE3+DWMiNWpzsRnEaKyWULGSAA@mail.gmail.com>

Hi,

I pushed a fix before you sent your message. At least test_ast should be
fixed.

https://hg.python.org/cpython/rev/c5df914e73ad

FYI I'm unable to reproduce the test_collections leak.

Victor

Le mardi 26 janvier 2016, Brett Cannon <brett at snarky.ca> a ?crit :

> Looks like Victor's ast.Constant change introduced a refleak.
>
> On Tue, 26 Jan 2016 at 00:47 <solipsis at pitrou.net
> <javascript:_e(%7B%7D,'cvml','solipsis at pitrou.net');>> wrote:
>
>> results for cbd4a6a2657e on branch "default"
>> --------------------------------------------
>>
>> test_ast leaked [39, 39, 39] references, sum=117
>> test_ast leaked [5, 5, 5] memory blocks, sum=15
>> test_collections leaked [-2, 0, 0] references, sum=-2
>> test_functools leaked [0, 2, 2] memory blocks, sum=4
>>
>>
>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>> '3:3:/home/psf-users/antoine/refleaks/reflogqIZGVY', '--timeout', '7200']
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> <javascript:_e(%7B%7D,'cvml','Python-checkins at python.org');>
>> https://mail.python.org/mailman/listinfo/python-checkins
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/826e2ddd/attachment.html>

From p.f.moore at gmail.com  Wed Jan 27 04:28:15 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 27 Jan 2016 09:28:15 +0000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <20160127052341.GA14190@sjoerdjob.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <20160127052341.GA14190@sjoerdjob.com>
Message-ID: <CACac1F_9deRtfX4KtfndC9vYk9ugq1mRmKkZBwiY1FUOWWV3pw@mail.gmail.com>

On 27 January 2016 at 05:23, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
> On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote:
>> ...
>> Oh, they are a lot of things to do! ...
>
> Just wondering, do you also need a set of (abusive) test-cases which
> check 100% conformity to the CPython semantics? I'm sure many of us
> would be able to whip up some ideas of things that are possible with
> Python and are of the kind "but you should not do that! That's bad
> programming!" which may or may not break the optimizations (especially
> specialized functions).
>
> I'm thinking of things like
>
>     def override_length_function_test():
>         global len
>         value = len("abc")
>         len = lambda obj: ord(obj[0]))
>         value += len("abc")
>         assert value == 3 + 97, "Outdated `len` used."
>
> And also cases where `len` was changed not in the function itself, but
> in another function that was called indirectly (maybe 4 functions down,
> one of which was monkey-patched in after the compilation step):
>
>     module_a.py
>     def test_func(callback):
>         value = len("abc")
>         callback()
>         value += len("abc")
>         assert value == 3 + 97, "Outdated `len` used."
>
>     module_b.py
>     import module_a
>     def override_length_function_test():
>         def callback():
>             module_a.len = lambda obj: ord(obj[0])
>         assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used."
>
> (I'm sure some of the other readers of this list can be even more
> creative in trying to show that making optimizations like this can break
> semantics.)
>
> Other fun tricks I'd like to try is overriding the `len` method from a
> signal handler, what happens when you monkey-patch a dependent method,
> having `__getitem__` and `__getattr__` on some value override `len`.
>
> Basically: trying things that I normally should not try during my
> working hours, on account of wanting to still have a job the next day.
>
> Kind regards,
> Sjoerd Job

Maybe I'm just nasty, but IMO those kinds of "torture tests" are just
as valuable in general, so I'd encourage people to submit patches to
the main Python test suite to add them.

Paul

From victor.stinner at gmail.com  Wed Jan 27 04:31:54 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 27 Jan 2016 10:31:54 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CACac1F_9deRtfX4KtfndC9vYk9ugq1mRmKkZBwiY1FUOWWV3pw@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <20160127052341.GA14190@sjoerdjob.com>
 <CACac1F_9deRtfX4KtfndC9vYk9ugq1mRmKkZBwiY1FUOWWV3pw@mail.gmail.com>
Message-ID: <CAMpsgwYJZzfhgjMqz9X1g5aQA0-HVUbbcrU5DF5mCpmCYGXuSg@mail.gmail.com>

Python has test_dynamic which tests such corner cases.

For example, test_modify_builtins_while_generator_active(): "Modify
the builtins out from under a live generator."
https://hg.python.org/cpython/file/58266f5101cc/Lib/test/test_dynamic.py#l49

Victor

2016-01-27 10:28 GMT+01:00 Paul Moore <p.f.moore at gmail.com>:
> On 27 January 2016 at 05:23, Sjoerd Job Postmus <sjoerdjob at sjec.nl> wrote:
>> On Mon, Jan 25, 2016 at 11:58:12PM +0100, Victor Stinner wrote:
>>> ...
>>> Oh, they are a lot of things to do! ...
>>
>> Just wondering, do you also need a set of (abusive) test-cases which
>> check 100% conformity to the CPython semantics? I'm sure many of us
>> would be able to whip up some ideas of things that are possible with
>> Python and are of the kind "but you should not do that! That's bad
>> programming!" which may or may not break the optimizations (especially
>> specialized functions).
>>
>> I'm thinking of things like
>>
>>     def override_length_function_test():
>>         global len
>>         value = len("abc")
>>         len = lambda obj: ord(obj[0]))
>>         value += len("abc")
>>         assert value == 3 + 97, "Outdated `len` used."
>>
>> And also cases where `len` was changed not in the function itself, but
>> in another function that was called indirectly (maybe 4 functions down,
>> one of which was monkey-patched in after the compilation step):
>>
>>     module_a.py
>>     def test_func(callback):
>>         value = len("abc")
>>         callback()
>>         value += len("abc")
>>         assert value == 3 + 97, "Outdated `len` used."
>>
>>     module_b.py
>>     import module_a
>>     def override_length_function_test():
>>         def callback():
>>             module_a.len = lambda obj: ord(obj[0])
>>         assert module_a.test_func(callback) == 3 + 97, "Outdated `len` used."
>>
>> (I'm sure some of the other readers of this list can be even more
>> creative in trying to show that making optimizations like this can break
>> semantics.)
>>
>> Other fun tricks I'd like to try is overriding the `len` method from a
>> signal handler, what happens when you monkey-patch a dependent method,
>> having `__getitem__` and `__getattr__` on some value override `len`.
>>
>> Basically: trying things that I normally should not try during my
>> working hours, on account of wanting to still have a job the next day.
>>
>> Kind regards,
>> Sjoerd Job
>
> Maybe I'm just nasty, but IMO those kinds of "torture tests" are just
> as valuable in general, so I'd encourage people to submit patches to
> the main Python test suite to add them.
>
> Paul

From victor.stinner at gmail.com  Wed Jan 27 04:39:43 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 27 Jan 2016 10:39:43 +0100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
Message-ID: <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>

2016-01-23 7:03 GMT+01:00 Chris Angelico <rosuav at gmail.com>:
> I just had a major crash on the system that hosts the
> angelico-debian-amd64 buildbot, and as usual, checked it carefully
> after bringing everything up. It seems now to be timing out after an
> hour of operation:
>
> http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/3132/steps/test/logs/stdio

I opened http://bugs.python.org/issue26206 to track this issue.

> Running just that test file:
>
> $ ./python Lib/test/test_socket.py
> ... chomp lots of lines ...
> testRecvmsgPeek (__main__.RecvmsgUDP6Test) ...
>
> seems to indicate that the stall is due to IPv6 and UDP. The VM should
> have full IPv6 support, although my ISPs don't carry IPv6 traffic, so
> it won't be able to reach the internet proper; but it should be able
> to do all manner of local tests.

Try to apply attached patch and run:

$ ./python -m test -v -m testRecvmsgPeek test_socket
(...)
testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK
<socket.socket fd=5, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 44347, 0, 0)>
SERV SOCK <socket.socket fd=4, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 40488, 0, 0)>
CLI SOCK ('::1', 44347, 0, 0)
SERV SOCK ('::1', 40488, 0, 0)
ok
testRecvmsgPeek (test.test_socket.RecvmsgIntoUDP6Test) ... CLI SOCK
<socket.socket fd=5, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 52721, 0, 0)>
SERV SOCK <socket.socket fd=4, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 43967, 0, 0)>
CLI SOCK ('::1', 52721, 0, 0)
SERV SOCK ('::1', 43967, 0, 0)
ok
(...)

As you can see: the test uses the local loopback interface.
Inet6TestBase.host is "::1".

You can try to run a UDP server using netcat: "nc -l -u ::1 12345".
Keep the command running in a terminal, and then run in a different
terminal: "echo abc | nc -u ::1 12345". You should receive abc in the
server.

Victor
-------------- next part --------------
A non-text attachment was scrubbed...
Name: debug.patch
Type: text/x-patch
Size: 928 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/ff147f2b/attachment.bin>

From tjreedy at udel.edu  Wed Jan 27 05:59:37 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 27 Jan 2016 05:59:37 -0500
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <56A7AE7C.3080806@mail.de>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
Message-ID: <n8a7v0$6g1$1@ger.gmane.org>

On 1/26/2016 12:35 PM, Sven R. Kunze wrote:
> I completely agree with INADA.

I an not sure you do.

> It's like saying, because a specific crossroad features a higher
> accident rate, *people need to change their driving behavior*.
> *No!* People won't change and it's not necessary either. The crossroad
> needs to be changed to be safer.

Safer crossroads tend to be slower unless one switched to alternate 
designs that eliminate crossing streams of traffic.  Python is safer, in 
everyday use as well as in artificial benchmarks, and is slower as a result.

Languages that don't have integers but use residue classes (with 
wraparound) or finite integer classes (with overflow) as a (faster) 
substitute have, in practice, lots of accidents (bugs) when used by 
non-experts.  Guido noticed this, gave up on changing coder behavior, 
and put the expert behavior of checking for wraparound/overflow and 
switching to real integers (longs) into the language.  (I forget when 
this was added.)

The purpose of the artificially low input to fib() is to hide and avoid 
the bugginess of most languages.  The analogous trick with testing 
crossroads would be to artificially restrict the density of cars to mask 
the accident-proneness of a 'fast, consenting-adults' crossroads with no 
stop signs and no stop lights.

> Same goes for Python. If it's slow using the very same piece of code
> (even superficially), you better make the language faster.
> Developers won't change and they won't change their code either. Just
> not necessary.

Instead of making people rewrite fib to dramatically increase speed, we 
added the lru-cache decorator to get most of the benefit without a 
rewrite.  But Inada rejected this Python speedup.  An ast optimizer 
could potentially do the same speedup without the explicit decorator. 
(No side-effects?  Multiple recursive calls?  Add a cache!)

> Btw. it would be a great feature for Python 3 to be faster than Python
> 2.

We all agree on that.  One way for this to happen is to add optimizers 
that would make Python 'cheat' on micrebenchmarks

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Wed Jan 27 06:16:19 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 27 Jan 2016 21:16:19 +1000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <56A7AE7C.3080806@mail.de>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
Message-ID: <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>

On 27 January 2016 at 03:35, Sven R. Kunze <srkunze at mail.de> wrote:
> I completely agree with  INADA.
>
> It's like saying, because a specific crossroad features a higher accident
> rate, people need to change their driving behavior.
> No! People won't change and it's not necessary either. The crossroad needs
> to be changed to be safer.

Umm, no, that's not how this works - developers contribute to
community driven projects for their *own* reasons. Nobody gets to tell
them what to do unless they're paying them.

Micro-optimising a poor algorithm won't deliver macro level
improvements because macro level code uses things like space-speed
trade-offs to improve the algorithmic efficiency (as in the example of
applying functools.lru_cache to a naive recursive fibonacci
implementation).

Victor's work on FAT optimiser is interesting because it offers
opportunities to speed up even code that is already algorithmically
efficient, as well as making CPython a better platform for
experimenting with those kinds of changes.

More generally though, much larger opportunities for improvement lie
in persuading people to *stop writing code*, and instead spending more
of their time on finding and assembling code other people have
*already written* into solutions to interesting problems. *That's* the
kind of improvement that turns enormously complex problems like facial
recognition into 25 line Python scripts:
https://realpython.com/blog/python/face-recognition-with-python/

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Wed Jan 27 07:16:45 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 27 Jan 2016 07:16:45 -0500
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
 <n87ssb$m4j$1@ger.gmane.org>
 <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp>
Message-ID: <n8acfl$ggg$1@ger.gmane.org>

On 1/26/2016 12:51 PM, Stephen J. Turnbull wrote:
> Terry Reedy writes:
>   > On 1/26/2016 12:02 AM, INADA Naoki wrote:
>   >
>   > > People use same algorithm on every language when compares base language
>   > > performance [1].
>   >
>   > The python code is NOT using the same algorithm.  The proof is that the
>   > Python function will return the correct value for, say fib(50) while
>   > most if not all the other versions will not.

Let me try to be clearer.

1.  Like everyone else, I would like Python function calls to be faster 
either in general or in special cases detected during compilation.  This 
will require micro-benchmark for function calls that do just that. 
First time an empty loop, then time a loop with a call to an empty 
function.  Do the same for various signatures, and maybe other special 
cases.

2. Cross-language micro-benchmarks aimed at timing specific operations 
are tough. To run on multiple languages, they must be restricted to a 
lowest-common-denominator  of features.  It is impossible to make every 
implementation perform exactly the same set of operations.  Some 
languages may bundle features together.  Some languages and 
implementations have optimizations that avoid unneeded operations.  Not 
all optimizatons can be turned off.

3. While there are trends as to the speed of implementations of a 
language, benchmarks time particular implementations.  Shedskin would 
compile fib to a C function that runs much faster than fib with the 
restricted subset of CPython allowed for the benchmark.

> True, but that's not a reasonable criterion for "same algorithm" in
> this context.  Naoki's application ("base language performance"
> benchmarking) requires fib(n) only for n < 40, and run it in a loop
> 100 times if you want 2 more decimal places of precision ("40" is
> appropriate for an implementation with 32-bit ints).

So you agree that the limit of 39 is not intrinsic to the fib function 
or its uses, but is an after-the-fact limit imposed to mask the bug 
proneness of using substitutes for integers.

To my mind, a fairer and more useful benchmark of 'base language 
performance' based on fib would use a wider domain.  The report would 
say that CPython (with lru_cache disallowed) is slow but works over a 
wide range of inputs, while some other implementations of other 
languages run faster for small inputs but fail catastrophically for 
larger inputs.  Users could then make a more informed pick.

Also see my answer to Sven Kunze.

-- 
Terry Jan Reedy


From rosuav at gmail.com  Wed Jan 27 07:47:33 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 27 Jan 2016 23:47:33 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>
Message-ID: <CAPTjJmoWDFoSeAWVBm5s1CutZ2fp7+L2Q_=JTe2a+5WKcAFi=Q@mail.gmail.com>

On Wed, Jan 27, 2016 at 8:39 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2016-01-23 7:03 GMT+01:00 Chris Angelico <rosuav at gmail.com>:
>> Running just that test file:
>>
>> $ ./python Lib/test/test_socket.py
>> ... chomp lots of lines ...
>> testRecvmsgPeek (__main__.RecvmsgUDP6Test) ...
>>
>> seems to indicate that the stall is due to IPv6 and UDP. The VM should
>> have full IPv6 support, although my ISPs don't carry IPv6 traffic, so
>> it won't be able to reach the internet proper; but it should be able
>> to do all manner of local tests.
>
> Try to apply attached patch and run:
>
> $ ./python -m test -v -m testRecvmsgPeek test_socket
> (...)
> testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK
> <socket.socket fd=5, family=AddressFamily.AF_INET6,
> type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 44347, 0, 0)>
> SERV SOCK <socket.socket fd=4, family=AddressFamily.AF_INET6,
> type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 40488, 0, 0)>
> CLI SOCK ('::1', 44347, 0, 0)
> SERV SOCK ('::1', 40488, 0, 0)
> ok
> testRecvmsgPeek (test.test_socket.RecvmsgIntoUDP6Test) ... CLI SOCK
> <socket.socket fd=5, family=AddressFamily.AF_INET6,
> type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 52721, 0, 0)>
> SERV SOCK <socket.socket fd=4, family=AddressFamily.AF_INET6,
> type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 43967, 0, 0)>
> CLI SOCK ('::1', 52721, 0, 0)
> SERV SOCK ('::1', 43967, 0, 0)
> ok
> (...)
>
> As you can see: the test uses the local loopback interface.
> Inet6TestBase.host is "::1".

Confirmed. It does two tests of IPv4 which run just fine, and then:

testRecvmsgPeek (test.test_socket.RecvmsgUDP6Test) ... CLI SOCK
<socket.socket fd=5, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 47518, 0, 0)>
CLI SOCK ('::1', 47518, 0, 0)
SERV SOCK <socket.socket fd=4, family=AddressFamily.AF_INET6,
type=SocketKind.SOCK_DGRAM, proto=0, laddr=('::1', 42421, 0, 0)>
SERV SOCK ('::1', 42421, 0, 0)

and hangs until I interrupt it.

> You can try to run a UDP server using netcat: "nc -l -u ::1 12345".
> Keep the command running in a terminal, and then run in a different
> terminal: "echo abc | nc -u ::1 12345". You should receive abc in the
> server.

(Oddly, the default netcat-traditional doesn't seem to support this,
but installing netcat-openbsd adds IPv6 support.)

Yep, and that works flawlessly. It's nothing weird about that
particular port, either - nc can use 42421 without a problem.

After digging through test_socket.py for over an hour (the MRO for
RecvmsgUDP6Test is enormous!!), I've boiled the issue down to this:

import socket
MSG = b'asdf qwer zxcv'
serv = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
serv.bind(("::1", 0))
cli = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
cli.bind(("::1", 0))
cli.sendto(MSG, serv.getsockname())
print(serv.recvmsg(len(MSG) - 3, 0, socket.MSG_PEEK))
print(serv.recvmsg(len(MSG), 0, socket.MSG_PEEK))
print(serv.recvmsg(len(MSG)))

On my main system, this produces three lines of output: the first has
truncated text, the second has full text, and the third also has full
text. This proves that MSG_PEEK is working correctly. On the buildbot,
though, the first one stalls out. Commenting that line out produces
correct results - peek the full data, then read it, and all is well.

Any idea why partial read on a datagram socket would sometimes stall?

ChrisA

From stephen at xemacs.org  Wed Jan 27 12:12:16 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 28 Jan 2016 02:12:16 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <n8acfl$ggg$1@ger.gmane.org>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
 <n87ssb$m4j$1@ger.gmane.org>
 <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp>
 <n8acfl$ggg$1@ger.gmane.org>
Message-ID: <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > So you agree that the limit of 39 is not intrinsic to the fib function 
 > or its uses, but is an after-the-fact limit imposed to mask the bug 
 > proneness of using substitutes for integers.

I don't know what the limit used in the benchmark is, but it must be
quite a bit lower than 50 for 32-bit integers and could be greater
than 90 for 64-bit integers.  And it's not "masking bugs", it's
"respecting the domain of valid input", s'il vous plait.

 > To my mind, a fairer and more useful benchmark of 'base language 
 > performance' based on fib would use a wider domain.

"Fair", maybe.  But why play this game at all?  These benchmarks are
simply not useful to users choosing languages, unless they already
know the difficulties of interpreting benchmarks and are willing to
expend the effort to account for them.

Without that knowledge and effort, choosing a programming language
based on microbenchmarks is like choosing a car based on the
leg-length of the model sitting on the hood in the TV commercial.

 > The report would say that CPython (with lru_cache disallowed) is
 > slow but works over a wide range of inputs,

No, the report would say "Use of this benchmark for cross-language
comparison of function call speed is more or less inaccurate due to
differences in representation of integers and in handling the
possibility of exceptions in 'integer' arithmetic."  You are picking
one tiny difference, but there are potentially many, some quite a bit
larger on the tested domain (for example, some languages may be able
to optimize fib() to unboxed integers, in which case they'll blow away
all those that don't).

 > Users could then make a more informed pick.

My point in my reply to Nick is that users aren't making informed
picks.  If they were, we wouldn't even be thinking about having this
conversation.  I'm not sure what they are doing (maybe, as Nick
suggests, justifying their "tribal" prejudices?), but it's not
that. ;-)  Sure, other things being equal, better benchmarks will
improve runtime performance, but other things are so far from being
equal even an economist can't say "ceteris paribus" here.

To expand that point: I don't really see a point in users (ie,
developers in Python and other such languages) looking at these
benchmarks except for the fun of feeling like implementers, to be
honest.  Even the implementers shouldn't much care about cross-
language benchmarks, except that when a "similar"[1] language does
significantly better on a particular benchmark, it's often useful to
wonder "how dey do dat?!"  Typically the answer is "they 'cheat'" ==
"fail one of the properties we consider required", but sometimes it's
"ooh, that's cute, and I bet we could make Python work the same way"
or "urkh, we can't do *that* (yuck!) but we could FATten up Python
with similar effect".  (Let me take this opportunity to say "Thank
you, Victor!")

Of course in the case of a controlled experiment like "configure in
Victor's changes and run the benchmarks to make sure they're not
detectably slower", they're invaluable regression tests, and more or
less valuable (ie, YMMV) as measures of improvement to compare to
costs they may impose in other features or (even more fuzzy) in
developer time.


Footnotes: 
[1]  Whatever that means....


From srkunze at mail.de  Wed Jan 27 13:11:27 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 27 Jan 2016 19:11:27 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <n8a7v0$6g1$1@ger.gmane.org>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de> <n8a7v0$6g1$1@ger.gmane.org>
Message-ID: <56A9084F.4090200@mail.de>

On 27.01.2016 11:59, Terry Reedy wrote:
> On 1/26/2016 12:35 PM, Sven R. Kunze wrote:
>> I completely agree with INADA.
>
> I an not sure you do.
>

I am sure I am. He wants to solve a problem the way that is natural to 
him as a unique human being.

>> It's like saying, because a specific crossroad features a higher
>> accident rate, *people need to change their driving behavior*.
>> *No!* People won't change and it's not necessary either. The crossroad
>> needs to be changed to be safer.
>
> Safer crossroads tend to be slower unless one switched to alternate 
> designs that eliminate crossing streams of traffic.

So Python can be safer AND faster ( = different design) if we try hard 
enough.

> Languages that don't have integers but use residue classes (with 
> wraparound) or finite integer classes (with overflow) as a (faster) 
> substitute have, in practice, lots of accidents (bugs) when used by 
> non-experts.  Guido noticed this, gave up on changing coder behavior, 
> and put the expert behavior of checking for wraparound/overflow and 
> switching to real integers (longs) into the language.  (I forget when 
> this was added.)

I am glad he did because it helps humans solve their problems in a 
natural way without artificial boundaries. :)

> The purpose of the artificially low input to fib() is to hide and 
> avoid the bugginess of most languages.  The analogous trick with 
> testing crossroads would be to artificially restrict the density of 
> cars to mask the accident-proneness of a 'fast, consenting-adults' 
> crossroads with no stop signs and no stop lights.
>

I am completely with you here, however I disagree about suspected 
hiding/avoiding mentality. You say:

Python -> *no problem with big integers but slow at small integers*
Other Language -> *faster but breaks at big integers*

Yes. That's it.

We haven't solved the human side, however. A human AGAIN would need to 
compromise on either speed or safety.


My point is: it would be insanely great if Python could be more like 
"*fast AND no problem with big integers*". No compromise here (at least 
no noticeable).

So, people could entirely *concentrate on their problem domain* without 
every worrying about such tiny little, nitty-gritty computer science 
details. I love computer science but people of other domains don't have 
the time nor the knowledge to decide properly. That's the reason why 
they might decide by using some weird micro-benchmarks. Just humans.

>> Same goes for Python. If it's slow using the very same piece of code
>> (even superficially), you better make the language faster.
>> Developers won't change and they won't change their code either. Just
>> not necessary.
>
> Instead of making people rewrite fib to dramatically increase speed, 
> we added the lru-cache decorator to get most of the benefit without a 
> rewrite.  But Inada rejected this Python speedup.  An ast optimizer 
> could potentially do the same speedup without the explicit decorator. 
> (No side-effects?  Multiple recursive calls? Add a cache!)
>

Bingo! That's the spirit.

Why that decorator in the first place? Hey, I mean, if I ever want to 
write some cryptic-looking source code with 3-letters abbreviations 
(LRU), I use Assembler again. But I discovered and love Python and I 
never want to go back when my problem domain does not require me to. So, 
when a machine can detect such an optimization, hell, do it, please. 
It's more likely that I apply it at the wrong function AND only in 10% 
of the correct cases: missing 90% and introducing some wild errors.

Again human stuff.

>> Btw. it would be a great feature for Python 3 to be faster than Python
>> 2.
>
> We all agree on that.  One way for this to happen is to add optimizers 
> that would make Python 'cheat' on micrebenchmarks

Then, we are all set. :)

Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/7b44e792/attachment.html>

From yselivanov.ml at gmail.com  Wed Jan 27 13:25:27 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 13:25:27 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
Message-ID: <56A90B97.7090001@gmail.com>

Hi,


tl;dr The summary is that I have a patch that improves CPython 
performance up to 5-10% on macro benchmarks.  Benchmarks results on 
Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available 
at [1].  There are no slowdowns that I could reproduce consistently.

There are twodifferent optimizations that yield this speedup: 
LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop.


LOAD_METHOD & CALL_METHOD
-------------------------

We had a lot of conversations with Victor about his PEP 509, and he sent 
me a link to his amazing compilation of notes about CPython performance 
[2].  One optimization that he pointed out to me was LOAD/CALL_METHOD 
opcodes, an idea first originated in PyPy.

There is a patch that implements this optimization, it's tracked here: 
[3].  There are some low level details that I explained in the issue, 
but I'll go over the high level design in this email as well.

Every time you access a method attribute on an object, a BoundMethod 
object is created. It is a fairly expensive operation, despite a 
freelist of BoundMethods (so that memory allocation is generally 
avoided).  The idea is to detect what looks like a method call in the 
compiler, and emit a pair of specialized bytecodes for that.

So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have 
LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD.

LOAD_METHOD looks at the object on top of the stack, and checks if the 
name resolves to a method or to a regular attribute.  If it's a method, 
then we push the unbound method object and the object to the stack.  If 
it's an attribute, we push the resolved attribute and NULL.

When CALL_METHOD looks at the stack it knows how to call the unbound 
method properly (pushing the object as a first arg), or how to call a 
regular callable.

This idea does make CPython faster around 2-4%.  And it surely doesn't 
make it slower.  I think it's a safe bet to at least implement this 
optimization in CPython 3.6.

So far, the patch only optimizes positional-only method calls. It's 
possible to optimize all kind of calls, but this will necessitate 3 more 
opcodes (explained in the issue).  We'll need to do some careful 
benchmarking to see if it's really needed.


Per-opcode cache in ceval
-------------------------

While reading PEP 509, I was thinking about how we can use 
dict->ma_version in ceval to speed up globals lookups.  One of the key 
assumptions (and this is what makes JITs possible) is that real-life 
programs don't modify globals and rebind builtins (often), and that most 
code paths operate on objects of the same type.

In CPython, all pure Python functions have code objects.  When you call 
a function, ceval executes its code object in a frame. Frames contain 
contextual information, including pointers to the globals and builtins 
dict.  The key observation here is that almost all code objects always 
have same pointers to the globals (the module they were defined in) and 
to the builtins.  And it's not a good programming practice to mutate 
globals or rebind builtins.

Let's look at this function:

def spam():
     print(ham)

Here are its opcodes:

   2           0 LOAD_GLOBAL              0 (print)
               3 LOAD_GLOBAL              1 (ham)
               6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
               9 POP_TOP
              10 LOAD_CONST               0 (None)
              13 RETURN_VALUE

The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3.  Let's look at 
the first one, that loads the 'print' function from builtins.  The 
opcode knows the following bits of information:

- its offset (0),
- its argument (0 -> 'print'),
- its type (LOAD_GLOBAL).

And these bits of information will *never* change.  So if this opcode 
could resolve the 'print' name (from globals or builtins, likely the 
latter) and save the pointer to it somewhere, along with 
globals->ma_version and builtins->ma_version, it could, on its second 
call, just load this cached info back, check that the globals and 
builtins dict haven't changed and push the cached ref to the stack.  
That would save it from doing two dict lookups.

We can also optimize LOAD_METHOD.  There are high chances, that 'obj' in 
'obj.method()' will be of the same type every time we execute the code 
object.  So if we'd have an opcodes cache, LOAD_METHOD could then cache 
a pointer to the resolved unbound method, a pointer to obj.__class__, 
and tp_version_tag of obj.__class__.  Then it would only need to check 
if the cached object type is the same (and that it wasn't modified) and 
that obj.__dict__ doesn't override 'method'.  Long story short, this 
caching really speeds up method calls on types implemented in C.  
list.append becomes very fast, because list doesn't have a __dict__, so 
the check is very cheap (with cache).

A straightforward way to implement such a cache is simple, but consumes 
a lot of memory, that would be just wasted, since we only need such a 
cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative 
about the cache design.  Here's what I came up with:

1. We add a few fields to the code object.

2. ceval will count how many times each code object is executed.

3. When the code object is executed over ~900 times, we mark it as 
"hot".  We also create an 'unsigned char' array "MAPPING", with length 
set to match the length of the code object.  So we have a 1-to-1 mapping 
between opcodes and MAPPING array.

4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and 
LOAD_METHOD do "MAPPING[opcode_offset()]++".

5. After 1024 calls to the code object, ceval loop will iterate through 
the MAPPING, counting all opcodes that were executed more than 50 times.

6. We then create an array of cache structs "CACHE" (here's a link to 
the updated code.h file: [6]).  We update MAPPING to be a mapping 
between opcode position and position in the CACHE. The code object is 
now "optimized".

7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use 
the CACHE array for fast path.

8. When there is a cache miss, i.e. the builtins/global/obj.__dict__ 
were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and 
it will never try to use the cache again.

Here's a link to the issue tracker with the first version of the patch: 
[5].  I'm working on the patch in a github repo here: [4].


Summary
-------

There are many things about this algorithm that we can improve/tweak.  
Perhaps we should profile code objects longer, or account for time they 
were executed.  Maybe we shouldn't deoptimize opcodes on their first 
cache miss.  Maybe we can come up with better data structures.  We also 
need to profile the memory and see how much more this cache will require.

One thing I'm certain about, is that we can get a 5-10% speedup of 
CPython with relatively low memory impact.  And I think it's worth 
exploring that!

If you're interested in these kind of optimizations, please help with 
code reviews, ideas, profiling and benchmarks.  The latter is especially 
important, I'd never imagine how hard it is to come up with a good macro 
benchmark.

I also want to thank my company MagicStack (magic.io) for sponsoring 
this work.

Thanks,
Yury


[1] https://gist.github.com/1st1/aed69d63a2ff4de4c7be
[2] http://faster-cpython.readthedocs.org/index.html
[3] http://bugs.python.org/issue26110
[4] https://github.com/1st1/cpython/tree/opcache2
[5] http://bugs.python.org/issue26219
[6] 
https://github.com/python/cpython/compare/master...1st1:opcache2?expand=1#diff-b253e61c56dfa646a6b1b9e7aaad418aR18

From brett at python.org  Wed Jan 27 13:33:38 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Jan 2016 18:33:38 +0000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <56A9084F.4090200@mail.de>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de> <n8a7v0$6g1$1@ger.gmane.org>
 <56A9084F.4090200@mail.de>
Message-ID: <CAP1=2W75=mR1R6pCY8qO00kwuViBCAmhU_ibctAB5MLUrXzy7g@mail.gmail.com>

On Wed, 27 Jan 2016 at 10:12 Sven R. Kunze <srkunze at mail.de> wrote:

> On 27.01.2016 11:59, Terry Reedy wrote:
>
> On 1/26/2016 12:35 PM, Sven R. Kunze wrote:
>
> I completely agree with INADA.
>
>
> I an not sure you do.
>
>
> I am sure I am. He wants to solve a problem the way that is natural to him
> as a unique human being.
>
>
> It's like saying, because a specific crossroad features a higher
> accident rate, *people need to change their driving behavior*.
> *No!* People won't change and it's not necessary either. The crossroad
> needs to be changed to be safer.
>
>
> Safer crossroads tend to be slower unless one switched to alternate
> designs that eliminate crossing streams of traffic.
>
>
> So Python can be safer AND faster ( = different design) if we try hard
> enough.
>
>
> Languages that don't have integers but use residue classes (with
> wraparound) or finite integer classes (with overflow) as a (faster)
> substitute have, in practice, lots of accidents (bugs) when used by
> non-experts.  Guido noticed this, gave up on changing coder behavior, and
> put the expert behavior of checking for wraparound/overflow and switching
> to real integers (longs) into the language.  (I forget when this was
> added.)
>
>
> I am glad he did because it helps humans solve their problems in a natural
> way without artificial boundaries. :)
>
>
> The purpose of the artificially low input to fib() is to hide and avoid
> the bugginess of most languages.  The analogous trick with testing
> crossroads would be to artificially restrict the density of cars to mask
> the accident-proneness of a 'fast, consenting-adults' crossroads with no
> stop signs and no stop lights.
>
>
> I am completely with you here, however I disagree about suspected
> hiding/avoiding mentality. You say:
>
> Python -> *no problem with big integers but slow at small integers*
> Other Language -> *faster but breaks at big integers*
>
> Yes. That's it.
>
> We haven't solved the human side, however. A human AGAIN would need to
> compromise on either speed or safety.
>
>
> My point is: it would be insanely great if Python could be more like "*fast
> AND no problem with big integers*". No compromise here (at least no
> noticeable).
>

And this is why this entire email thread has devolved into a conversation
that isn't really going anywhere. This whole thread has completely lost
track of the point of Victor's earlier email saying "I'm still working on
my FAT work and don't take any notice of the performance numbers until more
stuff gets finished". And this discussion of what benchmarks to care about
is rather pointless since the core team has an implicit understanding that
any performance improvement is taken into consideration in terms of
balancing complexity in CPython with how much improvement it gets us. So if
someone wants to speed up Fibonacci then they are welcome to try, but the
solution must be maintainable in proportion to the speed increase it buys
Python as a whole.

-Brett




>
> So, people could entirely *concentrate on their problem domain* without
> every worrying about such tiny little, nitty-gritty computer science
> details. I love computer science but people of other domains don't have the
> time nor the knowledge to decide properly. That's the reason why they might
> decide by using some weird micro-benchmarks. Just humans.
>
>
> Same goes for Python. If it's slow using the very same piece of code
> (even superficially), you better make the language faster.
> Developers won't change and they won't change their code either. Just
> not necessary.
>
>
> Instead of making people rewrite fib to dramatically increase speed, we
> added the lru-cache decorator to get most of the benefit without a
> rewrite.  But Inada rejected this Python speedup.  An ast optimizer could
> potentially do the same speedup without the explicit decorator. (No
> side-effects?  Multiple recursive calls?  Add a cache!)
>
>
> Bingo! That's the spirit.
>
> Why that decorator in the first place? Hey, I mean, if I ever want to
> write some cryptic-looking source code with 3-letters abbreviations (LRU),
> I use Assembler again. But I discovered and love Python and I never want to
> go back when my problem domain does not require me to. So, when a machine
> can detect such an optimization, hell, do it, please. It's more likely that
> I apply it at the wrong function AND only in 10% of the correct cases:
> missing 90% and introducing some wild errors.
>
> Again human stuff.
>
>
> Btw. it would be a great feature for Python 3 to be faster than Python
> 2.
>
>
> We all agree on that.  One way for this to happen is to add optimizers
> that would make Python 'cheat' on micrebenchmarks
>
>
> Then, we are all set. :)
>
> Best,
> Sven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/c6f09bb8/attachment.html>

From srkunze at mail.de  Wed Jan 27 13:40:44 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 27 Jan 2016 19:40:44 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
 <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
Message-ID: <56A90F2C.9000401@mail.de>

On 27.01.2016 12:16, Nick Coghlan wrote:
> On 27 January 2016 at 03:35, Sven R. Kunze <srkunze at mail.de> wrote:
>> I completely agree with  INADA.
>>
>> It's like saying, because a specific crossroad features a higher accident
>> rate, people need to change their driving behavior.
>> No! People won't change and it's not necessary either. The crossroad needs
>> to be changed to be safer.
> Umm, no, that's not how this works

That's exactly how it works, Nick.

INADA uses Python as I use crossroads each day. Daily human business.

If you read his post carefully, you can discover that he just presented 
to you his perspective of the world. Moreover, I can assure you that 
he's not alone. As usual with humans it's not about facts or 
mathematically proven theorems but *perception*. It's more about 
marketing, little important details (or unimportant ones depending on 
whom you ask) and so on. Stating that he has a wrong perspective will 
not change anything. Believing that Python is treated unfair will not 
change that either.

Most people believe what they see. When they see a "FUNCTION CALL", it's 
the same in every language. Why? Because it looks like a function call ( 
name + parentheses ), it's called "function call" even if it's 
implemented completely differently. It even doesn't matter if we use 
commas, 'def', return types, etc. Because people understand the bigger 
concept, so that is what people want to compare.

Average Joe doesn't care and does not understand. He looks at the 
benchmarks. That is something he can understand. "While performance is 
not a matter when choosing first language, slowest of three makes bad 
impression and people feel less attractive about Python." << just like that

Not saying that INADA is an Average Joe, but I think you get the idea.
>   - developers contribute to
> community driven projects for their *own* reasons. Nobody gets to tell
> them what to do unless they're paying them.

Bit off-topic.

> Micro-optimising a poor algorithm won't deliver macro level
> improvements because macro level code uses things like space-speed
> trade-offs to improve the algorithmic efficiency (as in the example of
> applying functools.lru_cache to a naive recursive fibonacci
> implementation).

I completely agree, Nick. :) But that isn't the issue here.

> Victor's work on FAT optimiser is interesting because it offers
> opportunities to speed up even code that is already algorithmically
> efficient, as well as making CPython a better platform for
> experimenting with those kinds of changes.

Exactly. :)

> More generally though, much larger opportunities for improvement lie
> in persuading people to *stop writing code*, and instead spending more
> of their time on finding and assembling code other people have
> *already written* into solutions to interesting problems. *That's* the
> kind of improvement that turns enormously complex problems like facial
> recognition into 25 line Python scripts:
> https://realpython.com/blog/python/face-recognition-with-python/

Interesting post. :) Thanks.

Btw. I completely agree with you on the "improve programming education", 
but not everybody can do it; and not everybody wants to learn and to 
practice it properly.

Best,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/3d97ab08/attachment-0001.html>

From srkunze at mail.de  Wed Jan 27 13:43:31 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Wed, 27 Jan 2016 19:43:31 +0100
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAP1=2W75=mR1R6pCY8qO00kwuViBCAmhU_ibctAB5MLUrXzy7g@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de> <n8a7v0$6g1$1@ger.gmane.org>
 <56A9084F.4090200@mail.de>
 <CAP1=2W75=mR1R6pCY8qO00kwuViBCAmhU_ibctAB5MLUrXzy7g@mail.gmail.com>
Message-ID: <56A90FD3.80505@mail.de>

On 27.01.2016 19:33, Brett Cannon wrote:
> And this is why this entire email thread has devolved into a 
> conversation that isn't really going anywhere. This whole thread has 
> completely lost track of the point of Victor's earlier email saying 
> "I'm still working on my FAT work and don't take any notice of the 
> performance numbers until more stuff gets finished". And this 
> discussion of what benchmarks to care about is rather pointless since 
> the core team has an implicit understanding that any performance 
> improvement is taken into consideration in terms of balancing 
> complexity in CPython with how much improvement it gets us. So if 
> someone wants to speed up Fibonacci then they are welcome to try, but 
> the solution must be maintainable in proportion to the speed increase 
> it buys Python as a whole.

+1

From brett at python.org  Wed Jan 27 15:01:15 2016
From: brett at python.org (Brett Cannon)
Date: Wed, 27 Jan 2016 20:01:15 +0000
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A90B97.7090001@gmail.com>
References: <56A90B97.7090001@gmail.com>
Message-ID: <CAP1=2W5+WCuSEhJGhn6y0=F93=C0=ghwBhzfopHNgSCdyEB3bw@mail.gmail.com>

On Wed, 27 Jan 2016 at 10:26 Yury Selivanov <yselivanov.ml at gmail.com> wrote:

> Hi,
>
>
> tl;dr The summary is that I have a patch that improves CPython
> performance up to 5-10% on macro benchmarks.  Benchmarks results on
> Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available
> at [1].  There are no slowdowns that I could reproduce consistently.
>
> There are twodifferent optimizations that yield this speedup:
> LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop.
>
>
> LOAD_METHOD & CALL_METHOD
> -------------------------
>
> We had a lot of conversations with Victor about his PEP 509, and he sent
> me a link to his amazing compilation of notes about CPython performance
> [2].  One optimization that he pointed out to me was LOAD/CALL_METHOD
> opcodes, an idea first originated in PyPy.
>
> There is a patch that implements this optimization, it's tracked here:
> [3].  There are some low level details that I explained in the issue,
> but I'll go over the high level design in this email as well.
>
> Every time you access a method attribute on an object, a BoundMethod
> object is created. It is a fairly expensive operation, despite a
> freelist of BoundMethods (so that memory allocation is generally
> avoided).  The idea is to detect what looks like a method call in the
> compiler, and emit a pair of specialized bytecodes for that.
>
> So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have
> LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD.
>
> LOAD_METHOD looks at the object on top of the stack, and checks if the
> name resolves to a method or to a regular attribute.  If it's a method,
> then we push the unbound method object and the object to the stack.  If
> it's an attribute, we push the resolved attribute and NULL.
>
> When CALL_METHOD looks at the stack it knows how to call the unbound
> method properly (pushing the object as a first arg), or how to call a
> regular callable.
>
> This idea does make CPython faster around 2-4%.  And it surely doesn't
> make it slower.  I think it's a safe bet to at least implement this
> optimization in CPython 3.6.
>
> So far, the patch only optimizes positional-only method calls. It's
> possible to optimize all kind of calls, but this will necessitate 3 more
> opcodes (explained in the issue).  We'll need to do some careful
> benchmarking to see if it's really needed.
>
>
> Per-opcode cache in ceval
> -------------------------
>
> While reading PEP 509, I was thinking about how we can use
> dict->ma_version in ceval to speed up globals lookups.  One of the key
> assumptions (and this is what makes JITs possible) is that real-life
> programs don't modify globals and rebind builtins (often), and that most
> code paths operate on objects of the same type.
>
> In CPython, all pure Python functions have code objects.  When you call
> a function, ceval executes its code object in a frame. Frames contain
> contextual information, including pointers to the globals and builtins
> dict.  The key observation here is that almost all code objects always
> have same pointers to the globals (the module they were defined in) and
> to the builtins.  And it's not a good programming practice to mutate
> globals or rebind builtins.
>
> Let's look at this function:
>
> def spam():
>      print(ham)
>
> Here are its opcodes:
>
>    2           0 LOAD_GLOBAL              0 (print)
>                3 LOAD_GLOBAL              1 (ham)
>                6 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
>                9 POP_TOP
>               10 LOAD_CONST               0 (None)
>               13 RETURN_VALUE
>
> The opcodes we want to optimize are LAOD_GLOBAL, 0 and 3.  Let's look at
> the first one, that loads the 'print' function from builtins.  The
> opcode knows the following bits of information:
>
> - its offset (0),
> - its argument (0 -> 'print'),
> - its type (LOAD_GLOBAL).
>
> And these bits of information will *never* change.  So if this opcode
> could resolve the 'print' name (from globals or builtins, likely the
> latter) and save the pointer to it somewhere, along with
> globals->ma_version and builtins->ma_version, it could, on its second
> call, just load this cached info back, check that the globals and
> builtins dict haven't changed and push the cached ref to the stack.
> That would save it from doing two dict lookups.
>
> We can also optimize LOAD_METHOD.  There are high chances, that 'obj' in
> 'obj.method()' will be of the same type every time we execute the code
> object.  So if we'd have an opcodes cache, LOAD_METHOD could then cache
> a pointer to the resolved unbound method, a pointer to obj.__class__,
> and tp_version_tag of obj.__class__.  Then it would only need to check
> if the cached object type is the same (and that it wasn't modified) and
> that obj.__dict__ doesn't override 'method'.  Long story short, this
> caching really speeds up method calls on types implemented in C.
> list.append becomes very fast, because list doesn't have a __dict__, so
> the check is very cheap (with cache).
>

What would it take to make this work with Python-defined classes? I guess
that would require knowing the version of the instance's __dict__, the
instance's __class__ version, the MRO, and where the method object was
found in the MRO and any intermediary classes to know if it was suddenly
shadowed? I think that's everything. :)

Obviously that's a lot, but I wonder how many classes have a deep
inheritance model vs. inheriting only from `object`? In that case you only
have to check self.__dict__.ma_version, self.__class__,
self.__class__.__dict__.ma_version, and self.__class__.__class__ == `type`.
I guess another way to look at this is to get an idea of how complex do the
checks have to get before caching something like this is not worth it
(probably also depends on how often you mutate self.__dict__ thanks to
mutating attributes, but you could in that instance just decide to always
look at self.__dict__ for the method's key and then do the ma_version cache
check for everything coming from the class).

Otherwise we can consider looking at the the caching strategies that Self
helped pioneer (http://bibliography.selflanguage.org/) that all of the
various JS engines lifted and consider caching all method lookups.


>
> A straightforward way to implement such a cache is simple, but consumes
> a lot of memory, that would be just wasted, since we only need such a
> cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be creative
> about the cache design.  Here's what I came up with:
>
> 1. We add a few fields to the code object.
>
> 2. ceval will count how many times each code object is executed.
>
> 3. When the code object is executed over ~900 times, we mark it as
> "hot".


What happens if you simply consider all code as hot? Is the overhead of
building the mapping such that you really need this, or is this simply to
avoid some memory/startup cost?


>   We also create an 'unsigned char' array "MAPPING", with length
> set to match the length of the code object.  So we have a 1-to-1 mapping
> between opcodes and MAPPING array.
>
> 4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and
> LOAD_METHOD do "MAPPING[opcode_offset()]++".
>
> 5. After 1024 calls to the code object, ceval loop will iterate through
> the MAPPING, counting all opcodes that were executed more than 50 times.
>

Where did the "50 times" boundary come from? Was this measured somehow or
did you just guess at a number?


>
> 6. We then create an array of cache structs "CACHE" (here's a link to
> the updated code.h file: [6]).  We update MAPPING to be a mapping
> between opcode position and position in the CACHE. The code object is
> now "optimized".
>
> 7. When the code object is "optimized", LOAD_METHOD and LOAD_GLOBAL use
> the CACHE array for fast path.
>
> 8. When there is a cache miss, i.e. the builtins/global/obj.__dict__
> were mutated, the opcode marks its entry in 'CACHE' as deoptimized, and
> it will never try to use the cache again.
>
> Here's a link to the issue tracker with the first version of the patch:
> [5].  I'm working on the patch in a github repo here: [4].
>
>
> Summary
> -------
>
> There are many things about this algorithm that we can improve/tweak.
> Perhaps we should profile code objects longer, or account for time they
> were executed.  Maybe we shouldn't deoptimize opcodes on their first
> cache miss.  Maybe we can come up with better data structures.  We also
> need to profile the memory and see how much more this cache will require.
>
> One thing I'm certain about, is that we can get a 5-10% speedup of
> CPython with relatively low memory impact.  And I think it's worth
> exploring that!
>

Great!


>
> If you're interested in these kind of optimizations, please help with
> code reviews, ideas, profiling and benchmarks.  The latter is especially
> important, I'd never imagine how hard it is to come up with a good macro
> benchmark.
>

Have you tried hg.python.org/benchmarks? Or are you looking for new
benchmarks? If the latter then we should probably strike up a discussion on
speed@ and start considering a new, unified benchmark suite that CPython,
PyPy, Pyston, Jython, and IronPython can all agree on.


>
> I also want to thank my company MagicStack (magic.io) for sponsoring
> this work.
>

Yep, thanks to all the companies sponsoring people doing work lately to try
and speed things up!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/2d5ac0e7/attachment.html>

From damien.p.george at gmail.com  Wed Jan 27 15:10:03 2016
From: damien.p.george at gmail.com (Damien George)
Date: Wed, 27 Jan 2016 20:10:03 +0000
Subject: [Python-Dev] Speeding up CPython 5-10%
Message-ID: <CANw+Mznb1s26Ktx8HrGiZ79fiJDm4mOFbpGaefUrxA2mfyWZ-g@mail.gmail.com>

Hi Yuri,

I think these are great ideas to speed up CPython.  They are probably
the simplest yet most effective ways to get performance improvements
in the VM.

MicroPython has had LOAD_METHOD/CALL_METHOD from the start (inspired
by PyPy, and the main reason to have it is because you don't need to
allocate on the heap when doing a simple method call).  The specific
opcodes are:

LOAD_METHOD # same behaviour as you propose
CALL_METHOD # for calls with positional and/or keyword args
CALL_METHOD_VAR_KW # for calls with one or both of */**

We also have LOAD_ATTR, CALL_FUNCTION and CALL_FUNCTION_VAR_KW for
non-method calls.

MicroPython also has dictionary lookup caching, but it's a bit
different to your proposal.  We do something much simpler: each opcode
that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR,
etc) includes a single byte in the opcode which is an offset-guess
into the dictionary to find the desired element.  Eg for LOAD_GLOBAL
we have (pseudo code):

CASE(LOAD_GLOBAL):
key = DECODE_KEY;
offset_guess = DECODE_BYTE;
if (global_dict[offset_guess].key == key) {
    // found the element straight away
} else {
    // not found, do a full lookup and save the offset
    offset_guess = dict_lookup(global_dict, key);
    UPDATE_BYTECODE(offset_guess);
}
PUSH(global_dict[offset_guess].elem);

We have found that such caching gives a massive performance increase,
on the order of 20%.  The issue (for us) is that it increases bytecode
size by a considerable amount, requires writeable bytecode, and can be
non-deterministic in terms of lookup time.  Those things are important
in the embedded world, but not so much on the desktop.

Good luck with it!

Regards,
Damien.

From yselivanov.ml at gmail.com  Wed Jan 27 15:26:35 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 15:26:35 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CAP1=2W5+WCuSEhJGhn6y0=F93=C0=ghwBhzfopHNgSCdyEB3bw@mail.gmail.com>
References: <56A90B97.7090001@gmail.com>
 <CAP1=2W5+WCuSEhJGhn6y0=F93=C0=ghwBhzfopHNgSCdyEB3bw@mail.gmail.com>
Message-ID: <56A927FB.8080307@gmail.com>



On 2016-01-27 3:01 PM, Brett Cannon wrote:
>
>
> [..]
>
>
>     We can also optimize LOAD_METHOD.  There are high chances, that
>     'obj' in
>     'obj.method()' will be of the same type every time we execute the code
>     object.  So if we'd have an opcodes cache, LOAD_METHOD could then
>     cache
>     a pointer to the resolved unbound method, a pointer to obj.__class__,
>     and tp_version_tag of obj.__class__.  Then it would only need to check
>     if the cached object type is the same (and that it wasn't
>     modified) and
>     that obj.__dict__ doesn't override 'method'.  Long story short, this
>     caching really speeds up method calls on types implemented in C.
>     list.append becomes very fast, because list doesn't have a
>     __dict__, so
>     the check is very cheap (with cache).
>
>
> What would it take to make this work with Python-defined classes?

It already works for Python-defined classes.  But it's a bit more 
expensive because you still have to check object's __dict__.  Still, 
there is a very noticeable performance increase (see the results of 
benchmark runs).

> I guess that would require knowing the version of the instance's 
> __dict__, the instance's __class__ version, the MRO, and where the 
> method object was found in the MRO and any intermediary classes to 
> know if it was suddenly shadowed? I think that's everything. :)

No, unfortunately we can't use the version of the instance's __dict__ as 
it is very volatile.  The current implementation of opcode cache works 
because types are much more stable.  Remember, the cache is per *code 
object*, so it should work for all times when code object is executed.

class F:
   def spam(self):
     self.ham()   # <- version of self.__dict__ is unstable
                  #    so we'll endup invalidating the cache
                  #    too often

__class__ version, MRO changes etc are covered by tp_version_tag, which 
I use as one of guards.

>
> Obviously that's a lot, but I wonder how many classes have a deep 
> inheritance model vs. inheriting only from `object`? In that case you 
> only have to check self.__dict__.ma_version, self.__class__, 
> self.__class__.__dict__.ma_version, and self.__class__.__class__ == 
> `type`. I guess another way to look at this is to get an idea of how 
> complex do the checks have to get before caching something like this 
> is not worth it (probably also depends on how often you mutate 
> self.__dict__ thanks to mutating attributes, but you could in that 
> instance just decide to always look at self.__dict__ for the method's 
> key and then do the ma_version cache check for everything coming from 
> the class).
>
> Otherwise we can consider looking at the the caching strategies that 
> Self helped pioneer (http://bibliography.selflanguage.org/) that all 
> of the various JS engines lifted and consider caching all method lookups.

Yeah, hidden classes are great.  But the infrastructure to support them 
properly is huge.  I think that to make them work you'll need a JIT -- 
to trace, deoptimize, optimize, and do it all with a reasonable memory 
footprint.  My patch is much smaller and simpler, something we can 
realistically tune and ship in 3.6.

>
>     A straightforward way to implement such a cache is simple, but
>     consumes
>     a lot of memory, that would be just wasted, since we only need such a
>     cache for LOAD_GLOBAL and LOAD_METHOD opcodes. So we have to be
>     creative
>     about the cache design.  Here's what I came up with:
>
>     1. We add a few fields to the code object.
>
>     2. ceval will count how many times each code object is executed.
>
>     3. When the code object is executed over ~900 times, we mark it as
>     "hot".
>
>
> What happens if you simply consider all code as hot? Is the overhead 
> of building the mapping such that you really need this, or is this 
> simply to avoid some memory/startup cost?

That's the first step for this patch.  I think we need to profile 
several big applications (I'll do it later for some of my code bases) 
and see how big is the memory impact if we optimize everything.

In any case, I expect it to be noticeable (which may be acceptable), so 
we'll probably try to optimize it.

>       We also create an 'unsigned char' array "MAPPING", with length
>     set to match the length of the code object.  So we have a 1-to-1
>     mapping
>     between opcodes and MAPPING array.
>
>     4. Next ~100 calls, while the code object is "hot", LOAD_GLOBAL and
>     LOAD_METHOD do "MAPPING[opcode_offset()]++".
>
>     5. After 1024 calls to the code object, ceval loop will iterate
>     through
>     the MAPPING, counting all opcodes that were executed more than 50
>     times.
>
>
> Where did the "50 times" boundary come from? Was this measured somehow 
> or did you just guess at a number?

If the number is too low, then you'll optimize code in branches that are 
rarely executed.  So I picked 50, because I only trace opcodes for 100 
calls.

All of those numbers can be (should be?) changed, and I think we should 
experiment with different heuristics.

> [..]
>
>
>     If you're interested in these kind of optimizations, please help with
>     code reviews, ideas, profiling and benchmarks.  The latter is
>     especially
>     important, I'd never imagine how hard it is to come up with a good
>     macro
>     benchmark.
>
>
> Have you tried hg.python.org/benchmarks <http://hg.python.org/benchmarks>?

Yes: https://gist.github.com/1st1/aed69d63a2ff4de4c7be

> Or are you looking for new benchmarks? If the latter then we should 
> probably strike up a discussion on speed@ and start considering a new, 
> unified benchmark suite that CPython, PyPy, Pyston, Jython, and 
> IronPython can all agree on.

Yes, IMHO we need better benchmarks.  Some of the existing ones are very 
unstable -- I can run them three times and get three completely 
different results.  Benchmarking is hard :)

I'll create a few issues on bugs.python.org with new/updated benchmarks, 
and will join the speed@ mailing list.

Yury

From v+python at g.nevcal.com  Wed Jan 27 15:32:41 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 27 Jan 2016 12:32:41 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <CAPTjJmp3NTTsmZfhr3EH3zpHvO8uana_KXTpCj+euuY00Moj0A@mail.gmail.com>
 <CAEfz+TxQfsngoGUCZW550-=8dCrUTfanaPg0DTpKHXYUXCHHXg@mail.gmail.com>
 <n87ssb$m4j$1@ger.gmane.org>
 <22183.45626.891297.272536@turnbull.sk.tsukuba.ac.jp>
 <n8acfl$ggg$1@ger.gmane.org>
 <22184.64112.8831.955609@turnbull.sk.tsukuba.ac.jp>
Message-ID: <56A92969.2030600@g.nevcal.com>

On 1/27/2016 9:12 AM, Stephen J. Turnbull wrote:
> Without that knowledge and effort, choosing a programming language
> based on microbenchmarks is like choosing a car based on the
> leg-length of the model sitting on the hood in the TV commercial.

+1  QOTD
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/c0670cb2/attachment.html>

From yselivanov.ml at gmail.com  Wed Jan 27 15:37:49 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 15:37:49 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CANw+Mznb1s26Ktx8HrGiZ79fiJDm4mOFbpGaefUrxA2mfyWZ-g@mail.gmail.com>
References: <CANw+Mznb1s26Ktx8HrGiZ79fiJDm4mOFbpGaefUrxA2mfyWZ-g@mail.gmail.com>
Message-ID: <56A92A9D.70508@gmail.com>



On 2016-01-27 3:10 PM, Damien George wrote:
> Hi Yuri,
>
> I think these are great ideas to speed up CPython.  They are probably
> the simplest yet most effective ways to get performance improvements
> in the VM.

Thanks!

>
> MicroPython has had LOAD_METHOD/CALL_METHOD from the start (inspired
> by PyPy, and the main reason to have it is because you don't need to
> allocate on the heap when doing a simple method call).  The specific
> opcodes are:
>
> LOAD_METHOD # same behaviour as you propose
> CALL_METHOD # for calls with positional and/or keyword args
> CALL_METHOD_VAR_KW # for calls with one or both of */**
>
> We also have LOAD_ATTR, CALL_FUNCTION and CALL_FUNCTION_VAR_KW for
> non-method calls.

Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all 
kind of method calls.  However, I'm not sure how big the impact will be, 
need to do more benchmarking.

BTW, how do you benchmark MicroPython?

>
> MicroPython also has dictionary lookup caching, but it's a bit
> different to your proposal.  We do something much simpler: each opcode
> that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR,
> etc) includes a single byte in the opcode which is an offset-guess
> into the dictionary to find the desired element.  Eg for LOAD_GLOBAL
> we have (pseudo code):
>
> CASE(LOAD_GLOBAL):
> key = DECODE_KEY;
> offset_guess = DECODE_BYTE;
> if (global_dict[offset_guess].key == key) {
>      // found the element straight away
> } else {
>      // not found, do a full lookup and save the offset
>      offset_guess = dict_lookup(global_dict, key);
>      UPDATE_BYTECODE(offset_guess);
> }
> PUSH(global_dict[offset_guess].elem);
>
> We have found that such caching gives a massive performance increase,
> on the order of 20%.  The issue (for us) is that it increases bytecode
> size by a considerable amount, requires writeable bytecode, and can be
> non-deterministic in terms of lookup time.  Those things are important
> in the embedded world, but not so much on the desktop.

That's a neat idea!  You're right, it does require bytecode to become 
writeable.  I considered implementing a similar strategy, but this would 
be a big change for CPython.  So I decided to minimize the impact of the 
patch and leave the opcodes untouched.


Thanks!
Yury

From damien.p.george at gmail.com  Wed Jan 27 16:20:27 2016
From: damien.p.george at gmail.com (Damien George)
Date: Wed, 27 Jan 2016 21:20:27 +0000
Subject: [Python-Dev] Speeding up CPython 5-10%
Message-ID: <CANw+MzmMrb+AgFLHvH1fsq1641MhOFcckg9Ro+6r25JjhCDJCA@mail.gmail.com>

Hi Yury,

(Sorry for misspelling your name previously!)

> Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all
> kind of method calls.  However, I'm not sure how big the impact will be,
> need to do more benchmarking.

I never did such fine grained analysis with MicroPython.  I don't
think there are many uses of * and ** that it'd be worth it, but
definitely there are lots of uses of plain keywords.  Also, you'd want
to consider how simple/complex it is to treat all these different
opcodes in the compiler.  For us, it's simpler to treat everything the
same.  Otherwise your LOAD_METHOD part of the compiler will need to
peek deep into the AST to see what kind of call it is.

> BTW, how do you benchmark MicroPython?

Haha, good question!  Well, we use Pystone 1.2 (unmodified) to do
basic benchmarking, and find it to be quite good.  We track our code
live at:

http://micropython.org/resources/code-dashboard/

You can see there the red line, which is the Pystone result.  There
was a big jump around Jan 2015 which is when we introduced opcode
dictionary caching.  And since then it's been very gradually
increasing due to small optimisations here and there.

Pystone is actually a great benchmark for embedded systems because it
gives very reliable results there (almost zero variation across runs)
and if we can squeeze 5 more Pystones out with some change then we
know that it's a good optimisation (for efficiency at least).

For us, low RAM usage and small code size are the most important
factors, and we track those meticulously.  But in fact, smaller code
size quite often correlates with more efficient code because there's
less to execute and it fits in the CPU cache (at least on the
desktop).

We do have some other benchmarks, but they are highly specialised for
us.  For example, how fast can you bit bang a GPIO pin using pure
Python code.  Currently we get around 200kHz on a 168MHz MCU, which
shows that pure (Micro)Python code is about 100 times slower than C.

> That's a neat idea!  You're right, it does require bytecode to become
> writeable.  I considered implementing a similar strategy, but this would
> be a big change for CPython.  So I decided to minimize the impact of the
> patch and leave the opcodes untouched.

I think you need to consider "big" changes, especially ones like this
that can have a great (and good) impact.  But really, this is a
behind-the-scenes change that *should not* affect end users, and so
you should not have any second thoughts about doing it.  One problem I
see with CPython is that it exposes way too much to the user (both
Python programmer and C extension writer) and this hurts both language
evolution (you constantly need to provide backwards compatibility) and
ability to optimise.

Cheers,
Damien.

From yselivanov.ml at gmail.com  Wed Jan 27 16:53:55 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 16:53:55 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CANw+MzmMrb+AgFLHvH1fsq1641MhOFcckg9Ro+6r25JjhCDJCA@mail.gmail.com>
References: <CANw+MzmMrb+AgFLHvH1fsq1641MhOFcckg9Ro+6r25JjhCDJCA@mail.gmail.com>
Message-ID: <56A93C73.6030606@gmail.com>

Damien,

On 2016-01-27 4:20 PM, Damien George wrote:
> Hi Yury,
>
> (Sorry for misspelling your name previously!)

NP.  As long as the first letter is "y" I don't care ;)

>
>> Yes, we'll need to add CALL_METHOD{_VAR|_KW|etc} opcodes to optimize all
>> kind of method calls.  However, I'm not sure how big the impact will be,
>> need to do more benchmarking.
> I never did such fine grained analysis with MicroPython.  I don't
> think there are many uses of * and ** that it'd be worth it, but
> definitely there are lots of uses of plain keywords.  Also, you'd want
> to consider how simple/complex it is to treat all these different
> opcodes in the compiler.  For us, it's simpler to treat everything the
> same.  Otherwise your LOAD_METHOD part of the compiler will need to
> peek deep into the AST to see what kind of call it is.
>
>> BTW, how do you benchmark MicroPython?
> Haha, good question!  Well, we use Pystone 1.2 (unmodified) to do
> basic benchmarking, and find it to be quite good.  We track our code
> live at:
>
> http://micropython.org/resources/code-dashboard/

The dashboard is cool!

An off-topic: have you ever tried hg.python.org/benchmarks
or compare MicroPython vs CPython?  I'm curious if MicroPython
is faster -- in that case we'll try to copy some optimization
ideas.

> You can see there the red line, which is the Pystone result.  There
> was a big jump around Jan 2015 which is when we introduced opcode
> dictionary caching.  And since then it's been very gradually
> increasing due to small optimisations here and there.

Do you use opcode dictionary caching only for LOAD_GLOBAL-like
opcodes?  Do you have an equivalent of LOAD_FAST, or you use
dicts to store local variables?

>> That's a neat idea!  You're right, it does require bytecode to become
>> writeable.  I considered implementing a similar strategy, but this would
>> be a big change for CPython.  So I decided to minimize the impact of the
>> patch and leave the opcodes untouched.
> I think you need to consider "big" changes, especially ones like this
> that can have a great (and good) impact.  But really, this is a
> behind-the-scenes change that *should not* affect end users, and so
> you should not have any second thoughts about doing it.

If we change the opcode size, it will probably affect libraries
that compose or modify code objects.  Modules like "dis" will
also need to be updated.  And that's probably just a tip of the
iceberg.

We can still implement your approach if we add a separate
private 'unsigned char' array to each code object, so that
LOAD_GLOBAL can store the key offsets.  It should be a bit
faster than my current patch, since it has one less level
of indirection.  But this way we loose the ability to
optimize LOAD_METHOD, simply because it requires more memory
for its cache.  In any case, I'll experiment!

> One problem I
> see with CPython is that it exposes way too much to the user (both
> Python programmer and C extension writer) and this hurts both language
> evolution (you constantly need to provide backwards compatibility) and
> ability to optimise.

Right.  Even though CPython explicitly states that opcodes
and code objects might change in the future, we still have to
be careful about changing them.

Yury

From yselivanov.ml at gmail.com  Wed Jan 27 17:01:17 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 17:01:17 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A90B97.7090001@gmail.com>
References: <56A90B97.7090001@gmail.com>
Message-ID: <56A93E2D.5060707@gmail.com>

As Brett suggested, I've just run the benchmarks suite with memory 
tracking on.  The results are here: 
https://gist.github.com/1st1/1851afb2773526fd7c58

Looks like the memory increase is around 1%.

One synthetic micro-benchmark, unpack_sequence, contains hundreds of 
lines that load a global variable and does nothing else, consumes 5%.

Yury

From v+python at g.nevcal.com  Wed Jan 27 15:46:15 2016
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Wed, 27 Jan 2016 12:46:15 -0800
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A92A9D.70508@gmail.com>
References: <CANw+Mznb1s26Ktx8HrGiZ79fiJDm4mOFbpGaefUrxA2mfyWZ-g@mail.gmail.com>
 <56A92A9D.70508@gmail.com>
Message-ID: <56A92C97.30806@g.nevcal.com>

On 1/27/2016 12:37 PM, Yury Selivanov wrote:
>
>>
>> MicroPython also has dictionary lookup caching, but it's a bit
>> different to your proposal.  We do something much simpler: each opcode
>> that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR,
>> etc) includes a single byte in the opcode which is an offset-guess
>> into the dictionary to find the desired element.  Eg for LOAD_GLOBAL
>> we have (pseudo code):
>>
>> CASE(LOAD_GLOBAL):
>> key = DECODE_KEY;
>> offset_guess = DECODE_BYTE;
>> if (global_dict[offset_guess].key == key) {
>>      // found the element straight away
>> } else {
>>      // not found, do a full lookup and save the offset
>>      offset_guess = dict_lookup(global_dict, key);
>>      UPDATE_BYTECODE(offset_guess);
>> }
>> PUSH(global_dict[offset_guess].elem);
>>
>> We have found that such caching gives a massive performance increase,
>> on the order of 20%.  The issue (for us) is that it increases bytecode
>> size by a considerable amount, requires writeable bytecode, and can be
>> non-deterministic in terms of lookup time.  Those things are important
>> in the embedded world, but not so much on the desktop.
>
> That's a neat idea!  You're right, it does require bytecode to become 
> writeable.

Would it?

Remember "fixup lists"?  Maybe they still exist for loading function 
addresses from one DLL into the code of another at load time?

So the equivalent for bytecode requires a static table of offset_guess, 
and the offsets into that table are allocated by the byte-code loader at 
byte-code load time, and the byte-code is "fixed up" at load time to use 
the correct offsets into the offset_guess table.  It takes one more 
indirection to find the guess, but if the result is a 20% improvement, 
maybe you'd still get 19%...


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160127/40458bb2/attachment.html>

From yselivanov.ml at gmail.com  Wed Jan 27 17:40:26 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 17:40:26 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A90B97.7090001@gmail.com>
References: <56A90B97.7090001@gmail.com>
Message-ID: <56A9475A.3040300@gmail.com>

BTW, this optimization also makes some old optimization tricks obsolete.

1. No need to write 'def func(len=len)'.  Globals lookups will be fast.

2. No need to save bound methods:

obj = []
obj_append = obj.append
for _ in range(10**6):
    obj_append(something)

This hand-optimized code would only be marginally faster, because of 
LOAD_METHOD and how it's cached.


Yury

From yselivanov.ml at gmail.com  Wed Jan 27 17:58:11 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Wed, 27 Jan 2016 17:58:11 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A92C97.30806@g.nevcal.com>
References: <CANw+Mznb1s26Ktx8HrGiZ79fiJDm4mOFbpGaefUrxA2mfyWZ-g@mail.gmail.com>
 <56A92A9D.70508@gmail.com> <56A92C97.30806@g.nevcal.com>
Message-ID: <56A94B83.5020705@gmail.com>



On 2016-01-27 3:46 PM, Glenn Linderman wrote:
> On 1/27/2016 12:37 PM, Yury Selivanov wrote:
>>
>>>
>>> MicroPython also has dictionary lookup caching, but it's a bit
>>> different to your proposal.  We do something much simpler: each opcode
>>> that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR,
>>> etc) includes a single byte in the opcode which is an offset-guess
>>> into the dictionary to find the desired element.  Eg for LOAD_GLOBAL
>>> we have (pseudo code):
>>>
>>> CASE(LOAD_GLOBAL):
>>> key = DECODE_KEY;
>>> offset_guess = DECODE_BYTE;
>>> if (global_dict[offset_guess].key == key) {
>>>      // found the element straight away
>>> } else {
>>>      // not found, do a full lookup and save the offset
>>>      offset_guess = dict_lookup(global_dict, key);
>>>      UPDATE_BYTECODE(offset_guess);
>>> }
>>> PUSH(global_dict[offset_guess].elem);
>>>
>>> We have found that such caching gives a massive performance increase,
>>> on the order of 20%.  The issue (for us) is that it increases bytecode
>>> size by a considerable amount, requires writeable bytecode, and can be
>>> non-deterministic in terms of lookup time.  Those things are important
>>> in the embedded world, but not so much on the desktop.
>>
>> That's a neat idea!  You're right, it does require bytecode to become 
>> writeable.
>
> Would it?
>
> Remember "fixup lists"?  Maybe they still exist for loading function 
> addresses from one DLL into the code of another at load time?
>
> So the equivalent for bytecode requires a static table of 
> offset_guess, and the offsets into that table are allocated by the 
> byte-code loader at byte-code load time, and the byte-code is "fixed 
> up" at load time to use the correct offsets into the offset_guess 
> table.  It takes one more indirection to find the guess, but if the 
> result is a 20% improvement, maybe you'd still get 19%...

Right, in my current patch I have an offset table per code object. 
Essentially, this offset table adds 8bits per opcode.  It also means 
that only first 255 LOAD_GLOBAL/LOAD_METHOD opcodes *per-code-object* 
are optimized (because the offset table only can store 8bit offsets), 
which is usually enough (I think you need to have more than a 500 lines 
of code function to reach that limit).

Yury

From stephen at xemacs.org  Wed Jan 27 19:53:51 2016
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Thu, 28 Jan 2016 09:53:51 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAP1=2W75=mR1R6pCY8qO00kwuViBCAmhU_ibctAB5MLUrXzy7g@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de> <n8a7v0$6g1$1@ger.gmane.org>
 <56A9084F.4090200@mail.de>
 <CAP1=2W75=mR1R6pCY8qO00kwuViBCAmhU_ibctAB5MLUrXzy7g@mail.gmail.com>
Message-ID: <22185.26271.894312.852637@turnbull.sk.tsukuba.ac.jp>

Brett Cannon writes:

 > the core team has an implicit understanding that any performance
 > improvement is taken into consideration in terms of balancing
 > complexity in CPython with how much improvement it gets us.

EIBTI.<wink/>  I can shut up now.  Thank you!


From ncoghlan at gmail.com  Thu Jan 28 02:58:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Jan 2016 17:58:07 +1000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <56A90F2C.9000401@mail.de>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
 <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
 <56A90F2C.9000401@mail.de>
Message-ID: <CADiSq7d4q4iBUw6XvSjwyjbOHgZdaU0f=XwpGFgGt8yHePPWnQ@mail.gmail.com>

On 28 January 2016 at 04:40, Sven R. Kunze <srkunze at mail.de> wrote:
> On 27.01.2016 12:16, Nick Coghlan wrote:
>> Umm, no, that's not how this works
> That's exactly how it works, Nick.
>
> INADA uses Python as I use crossroads each day. Daily human business.
>
> If you read his post carefully, you can discover that he just presented to
> you his perspective of the world. Moreover, I can assure you that he's not
> alone. As usual with humans it's not about facts or mathematically proven
> theorems but perception. It's more about marketing, little important details
> (or unimportant ones depending on whom you ask) and so on. Stating that he
> has a wrong perspective will not change anything.

The only part I disagree with is requesting that *other people* care
about marketing numbers if that's not something they're already
inclined to care about. I'm not in any way disputing that folks make
decisions based on inappropriate metrics, nor that it bothers some
folks that there are dozens of perfectly viable programming languages
people may choose to use instead of Python.

The fact remains that contributors to open source projects work on
what they want to work on or on what their employers pay them to work
on (for a lucky few, those are the same thing), so telling other
contributors that they're working on the "wrong thing" because their
priorities differ from our priorities is almost always going to be
irritating rather than helpful.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Thu Jan 28 03:30:43 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 28 Jan 2016 17:30:43 +0900
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CADiSq7d4q4iBUw6XvSjwyjbOHgZdaU0f=XwpGFgGt8yHePPWnQ@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
 <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
 <56A90F2C.9000401@mail.de>
 <CADiSq7d4q4iBUw6XvSjwyjbOHgZdaU0f=XwpGFgGt8yHePPWnQ@mail.gmail.com>
Message-ID: <CAEfz+TwMZyzYcY80v3PyLf7jVSQ6131S66hyRk4tY-wzYr1YpA@mail.gmail.com>

Please stop.

I'm sorry about messing up this thread.
I just wanted to represent why I'm very interested in Victor's efforts.

Regards.

On Thu, Jan 28, 2016 at 4:58 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 28 January 2016 at 04:40, Sven R. Kunze <srkunze at mail.de> wrote:
> > On 27.01.2016 12:16, Nick Coghlan wrote:
> >> Umm, no, that's not how this works
> > That's exactly how it works, Nick.
> >
> > INADA uses Python as I use crossroads each day. Daily human business.
> >
> > If you read his post carefully, you can discover that he just presented
> to
> > you his perspective of the world. Moreover, I can assure you that he's
> not
> > alone. As usual with humans it's not about facts or mathematically proven
> > theorems but perception. It's more about marketing, little important
> details
> > (or unimportant ones depending on whom you ask) and so on. Stating that
> he
> > has a wrong perspective will not change anything.
>
> The only part I disagree with is requesting that *other people* care
> about marketing numbers if that's not something they're already
> inclined to care about. I'm not in any way disputing that folks make
> decisions based on inappropriate metrics, nor that it bothers some
> folks that there are dozens of perfectly viable programming languages
> people may choose to use instead of Python.
>
> The fact remains that contributors to open source projects work on
> what they want to work on or on what their employers pay them to work
> on (for a lucky few, those are the same thing), so telling other
> contributors that they're working on the "wrong thing" because their
> priorities differ from our priorities is almost always going to be
> irritating rather than helpful.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>



-- 
INADA Naoki  <songofacandy at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160128/69feed97/attachment-0001.html>

From vadmium+py at gmail.com  Thu Jan 28 03:35:59 2016
From: vadmium+py at gmail.com (Martin Panter)
Date: Thu, 28 Jan 2016 08:35:59 +0000
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAPTjJmoWDFoSeAWVBm5s1CutZ2fp7+L2Q_=JTe2a+5WKcAFi=Q@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>
 <CAPTjJmoWDFoSeAWVBm5s1CutZ2fp7+L2Q_=JTe2a+5WKcAFi=Q@mail.gmail.com>
Message-ID: <CA+eR4cHDDeetbRSx4tg_0kzh_H0x-hBxy0EUsqMXQiZ54MPfMg@mail.gmail.com>

> After digging through test_socket.py for over an hour (the MRO for
> RecvmsgUDP6Test is enormous!!), I've boiled the issue down to this:
>
> import socket
> MSG = b'asdf qwer zxcv'
> serv = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
> serv.bind(("::1", 0))
> cli = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
> cli.bind(("::1", 0))
> cli.sendto(MSG, serv.getsockname())
> print(serv.recvmsg(len(MSG) - 3, 0, socket.MSG_PEEK))
> print(serv.recvmsg(len(MSG), 0, socket.MSG_PEEK))
> print(serv.recvmsg(len(MSG)))
>
> On my main system, this produces three lines of output: the first has
> truncated text, the second has full text, and the third also has full
> text. This proves that MSG_PEEK is working correctly. On the buildbot,
> though, the first one stalls out. Commenting that line out produces
> correct results - peek the full data, then read it, and all is well.
>
> Any idea why partial read on a datagram socket would sometimes stall?

I think it would stall if there is no data to receive. Maybe check the
return value of sendto(), to ensure it is sending the whole message.

Attached is a C program which should do the equivalent of your
boiled-down Python script, in case that helps:

$ gcc -Wall peek-udp6.c -o peek-udp6
$ ./peek-udp6
Bytes sent: 14
Received [asdf qwer z]
Received [asdf qwer zxcv]
Received [asdf qwer zxcv]

Other things that come to mind are to see if there is anything odd
about the buildbot?s Linux kernel and glibc versions. Maybe run the
Python script under ?strace? to see if anything strange is going on.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: peek-udp6.c
Type: text/x-csrc
Size: 2404 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160128/f9c68103/attachment.c>

From ncoghlan at gmail.com  Thu Jan 28 04:39:52 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 28 Jan 2016 19:39:52 +1000
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CAEfz+TwMZyzYcY80v3PyLf7jVSQ6131S66hyRk4tY-wzYr1YpA@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
 <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
 <56A90F2C.9000401@mail.de>
 <CADiSq7d4q4iBUw6XvSjwyjbOHgZdaU0f=XwpGFgGt8yHePPWnQ@mail.gmail.com>
 <CAEfz+TwMZyzYcY80v3PyLf7jVSQ6131S66hyRk4tY-wzYr1YpA@mail.gmail.com>
Message-ID: <CADiSq7fnYnE6ksN1hQv-hCMeeDo0P4hKzjAaU+za048eF+VMjQ@mail.gmail.com>

On 28 January 2016 at 18:30, INADA Naoki <songofacandy at gmail.com> wrote:
> Please stop.
>
> I'm sorry about messing up this thread.

Not your fault at all! This is just a particular bugbear of mine,
since software architecture design (including appropriate programming
language selection) is an even more poorly understood discipline than
software development in general :)

> I just wanted to represent why I'm very interested in Victor's efforts.

And thanks for posting that, as it is indeed cool that the
optimisation efforts currently being discussed may result in
performance improvements on some of the simplified micro-benchmarks
popular in programming language shootouts.

There's no way you could have anticipated the subsequent tangential
discussion on motives for contributing to open source projects, and
the impact that has on what we can reasonably expect from fellow
contributors.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Thu Jan 28 05:41:08 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 28 Jan 2016 21:41:08 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CA+eR4cHDDeetbRSx4tg_0kzh_H0x-hBxy0EUsqMXQiZ54MPfMg@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>
 <CAPTjJmoWDFoSeAWVBm5s1CutZ2fp7+L2Q_=JTe2a+5WKcAFi=Q@mail.gmail.com>
 <CA+eR4cHDDeetbRSx4tg_0kzh_H0x-hBxy0EUsqMXQiZ54MPfMg@mail.gmail.com>
Message-ID: <CAPTjJmpv4L9znRyp5MFLraEedthZzkTaNJVWWrroz7qHwe_9ug@mail.gmail.com>

On Thu, Jan 28, 2016 at 7:35 PM, Martin Panter <vadmium+py at gmail.com> wrote:
> Other things that come to mind are to see if there is anything odd
> about the buildbot?s Linux kernel and glibc versions. Maybe run the
> Python script under ?strace? to see if anything strange is going on.
>

I did that, and a few other things. Most notably, commenting out the
partial-read resulted in a flawless run, and strace showed it stalling
in the partial read too.

However, as I was doing so (and I just discarded a draft message where
I'd been typing up notes), my entire system went kerblooie, and I've
spent the last day rebuilding stuff from scratch. When I get around to
it, I'll rebuild the buildbot VM - and it'll be Debian Jessie (current
stable), because there's no particular reason to use Wheezy
(oldstable). So the problem will almost certainly disappear.

ChrisA

From stefan at bytereef.org  Thu Jan 28 08:49:50 2016
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 28 Jan 2016 14:49:50 +0100
Subject: [Python-Dev] [Python-checkins] BAD Benchmark Results for Python
 Default 2016-01-26
In-Reply-To: <A0E8B717-2D51-4154-BCE2-7211ECB04CC5@intel.com>
References: <2acd60eb-a5d1-4b31-853d-7c7f3f0841fa@irsmsx101.ger.corp.intel.com>
 <A0E8B717-2D51-4154-BCE2-7211ECB04CC5@intel.com>
Message-ID: <20160128134950.GA3196@bytereef.org>


IMO the timings of the benchmark suite are a bit unstable -- this is not the
fault of Intel's setup, I noticed it also when running the suite myself.


On Tue, Jan 26, 2016 at 06:48:54PM +0000, Stewart, David C wrote:
> Wow, what happened to Python default to cause such a regression?
> 
> 
> 
> 
> On 1/26/16, 7:31 AM, "lp_benchmark_robot" <lp_benchmark_robot at intel.com> wrote:
> 
> >Results for project Python default, build date 2016-01-26 03:07:40 +0000
> >commit:		cbd4a6a2657e
> >previous commit:	f700bc0412bc
> >revision date:	2016-01-26 02:54:37 +0000
> >environment:	Haswell-EP
> >	cpu:		Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, stepping 2, LLC 45 MB
> >	mem:		128 GB
> >	os:		CentOS 7.1
> >	kernel:	Linux 3.10.0-229.4.2.el7.x86_64
> >
> >Baseline results were generated using release v3.4.3, with hash b4cbecbc0781
> >from 2015-02-25 12:15:33+00:00
> >
> >----------------------------------------------------------------------------------
> >              benchmark   relative   change since   change since   current rev run
> >                          std_dev*       last run       baseline          with PGO
> >----------------------------------------------------------------------------------
> >:-)           django_v2      0.21%         -2.93%          8.95%            16.19%
> >:-|             pybench      0.10%          0.05%         -1.87%             5.40%
> >:-(            regex_v8      2.72%         -0.02%         -4.67%             4.57%
> >:-|               nbody      0.13%         -0.92%         -1.33%             7.40%
> >:-|        json_dump_v2      0.20%          0.87%         -1.59%            11.48%
> >:-|      normal_startup      0.90%         -0.57%          0.10%             5.35%
> >----------------------------------------------------------------------------------
> >* Relative Standard Deviation (Standard Deviation/Average)
> >
> >If this is not displayed properly please visit our results page here: http://languagesperformance.intel.com/bad-benchmark-results-for-python-default-2016-01-26/
> >
> >Note: Benchmark results are measured in seconds.
> >
> >Subject Label Legend:
> >Attributes are determined based on the performance evolution of the workloads
> >compared to the previous measurement iteration.
> >NEUTRAL: performance did not change by more than 1% for any workload
> >GOOD: performance improved by more than 1% for at least one workload and there
> >is no regression greater than 1%
> >BAD: performance dropped by more than 1% for at least one workload and there is
> >no improvement greater than 1%
> >UGLY: performance improved by more than 1% for at least one workload and also
> >dropped by more than 1% for at least one workload
> >
> >
> >Our lab does a nightly source pull and build of the Python project and measures
> >performance changes against the previous stable version and the previous nightly
> >measurement. This is provided as a service to the community so that quality
> >issues with current hardware can be identified quickly.
> >
> >Intel technologies' features and benefits depend on system configuration and may
> >require enabled hardware, software or service activation. Performance varies
> >depending on system configuration.
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins


From larry at hastings.org  Thu Jan 28 09:57:20 2016
From: larry at hastings.org (Larry Hastings)
Date: Thu, 28 Jan 2016 06:57:20 -0800
Subject: [Python-Dev] Fun with ancient unsupported platforms
Message-ID: <56AA2C50.7050700@hastings.org>



Check out and cd into Python trunk.

% grep -Ri win16 * | wc
      10      66     625

% grep -Ri nextstep | wc
      23     119    1328

% grep -Ri rhapsody * | wc
      47     269    3390

% grep -Ri msdos * | wc
      56     381    3895
  % grep -Ri ms-dos * | wc
      20     180    1425


win16!  *laughs* *wipes tear from eye*

It's currently 2016.  Perhaps it's time to remove all vestiges of these 
unsupported operating systems nobody's cared about since a year that 
started with a '1'?


//arry/

p.s. I suspect some of the uses of "rhapsody" are actually live code 
used for OS X.  So this isn't necessarily dead code, some of it is 
merely long-out-of-date comments.

p.p.s. At least there were no references to "taligent", "copland", or 
"gershwin"...!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160128/a5580cf6/attachment.html>

From christian at python.org  Thu Jan 28 10:40:48 2016
From: christian at python.org (Christian Heimes)
Date: Thu, 28 Jan 2016 16:40:48 +0100
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <56AA2C50.7050700@hastings.org>
References: <56AA2C50.7050700@hastings.org>
Message-ID: <56AA3680.8060602@python.org>

On 2016-01-28 15:57, Larry Hastings wrote:
> 
> 
> Check out and cd into Python trunk.
> 
> % grep -Ri win16 * | wc
>      10      66     625
> 
> % grep -Ri nextstep | wc
>      23     119    1328
> 
> % grep -Ri rhapsody * | wc
>      47     269    3390
> 
> % grep -Ri msdos * | wc
>      56     381    3895
>  % grep -Ri ms-dos * | wc
>      20     180    1425
> 
> 
> win16!  *laughs* *wipes tear from eye*
> 
> It's currently 2016.  Perhaps it's time to remove all vestiges of these
> unsupported operating systems nobody's cared about since a year that
> started with a '1'?

The platform module has more hilarious comments:

Still needed:
* more support for WinCE
* support for MS-DOS (PythonDX ?)
* support for Amiga and other still unsupported platforms running Python

Christian


From rymg19 at gmail.com  Thu Jan 28 10:44:32 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 28 Jan 2016 09:44:32 -0600
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <56AA2C50.7050700@hastings.org>
References: <56AA2C50.7050700@hastings.org>
Message-ID: <EA75B4F2-1990-4FE2-AB2F-C3F395949FF9@gmail.com>

win16 doesn't seem to have important stuff:

https://github.com/python/cpython/search?utf8=?&q="win16"

On January 28, 2016 8:57:20 AM CST, Larry Hastings <larry at hastings.org> wrote:
>
>
>Check out and cd into Python trunk.
>
>% grep -Ri win16 * | wc
>      10      66     625
>
>% grep -Ri nextstep | wc
>      23     119    1328
>
>% grep -Ri rhapsody * | wc
>      47     269    3390
>
>% grep -Ri msdos * | wc
>      56     381    3895
>  % grep -Ri ms-dos * | wc
>      20     180    1425
>
>
>win16!  *laughs* *wipes tear from eye*
>
>It's currently 2016.  Perhaps it's time to remove all vestiges of these
>
>unsupported operating systems nobody's cared about since a year that 
>started with a '1'?
>
>
>//arry/
>
>p.s. I suspect some of the uses of "rhapsody" are actually live code 
>used for OS X.  So this isn't necessarily dead code, some of it is 
>merely long-out-of-date comments.
>
>p.p.s. At least there were no references to "taligent", "copland", or 
>"gershwin"...!
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Python-Dev mailing list
>Python-Dev at python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com

-- 
Sent from my Nexus 5 with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160128/fe2c36e4/attachment.html>

From barry at python.org  Thu Jan 28 11:00:31 2016
From: barry at python.org (Barry Warsaw)
Date: Thu, 28 Jan 2016 11:00:31 -0500
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <EA75B4F2-1990-4FE2-AB2F-C3F395949FF9@gmail.com>
References: <56AA2C50.7050700@hastings.org>
 <EA75B4F2-1990-4FE2-AB2F-C3F395949FF9@gmail.com>
Message-ID: <20160128110031.7e7e2295@subdivisions.wooz.org>

Just as long as you can still build and run Python on Guido's ancient SGI
machine <wink>.

-Barry

From victor.stinner at gmail.com  Thu Jan 28 11:29:41 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 28 Jan 2016 17:29:41 +0100
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <56AA2C50.7050700@hastings.org>
References: <56AA2C50.7050700@hastings.org>
Message-ID: <CAMpsgwbxYW7+A-kx6K6xpuQSkXvdK4EtcojBNT0RfDsB+zGadw@mail.gmail.com>

We slowly remove old platforms, but only if the code specific to these
old platforms is annoying to maintain. For example, I wrote the
change:

https://hg.python.org/cpython/rev/a1605d2508af
"""
Issue #22591: Drop support of MS-DOS

Drop support of MS-DOS, especially of the DJGPP compiler (MS-DOS port of GCC).

Today is a sad day. Good bye MS-DOS, good bye my friend :'-(
"""
=> http://bugs.python.org/issue22591

Victor

2016-01-28 15:57 GMT+01:00 Larry Hastings <larry at hastings.org>:
>
>
> Check out and cd into Python trunk.
>
> % grep -Ri win16 * | wc
>      10      66     625
>
> % grep -Ri nextstep | wc
>      23     119    1328
>
> % grep -Ri rhapsody * | wc
>      47     269    3390
>
> % grep -Ri msdos * | wc
>      56     381    3895
>  % grep -Ri ms-dos * | wc
>      20     180    1425
>
>
> win16!  *laughs* *wipes tear from eye*
>
> It's currently 2016.  Perhaps it's time to remove all vestiges of these
> unsupported operating systems nobody's cared about since a year that started
> with a '1'?
>
>
> /arry
>
> p.s. I suspect some of the uses of "rhapsody" are actually live code used
> for OS X.  So this isn't necessarily dead code, some of it is merely
> long-out-of-date comments.
>
> p.p.s. At least there were no references to "taligent", "copland", or
> "gershwin"...!
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From pmiscml at gmail.com  Thu Jan 28 13:38:36 2016
From: pmiscml at gmail.com (Paul Sokolovsky)
Date: Thu, 28 Jan 2016 20:38:36 +0200
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <CAMpsgwbxYW7+A-kx6K6xpuQSkXvdK4EtcojBNT0RfDsB+zGadw@mail.gmail.com>
References: <56AA2C50.7050700@hastings.org>
 <CAMpsgwbxYW7+A-kx6K6xpuQSkXvdK4EtcojBNT0RfDsB+zGadw@mail.gmail.com>
Message-ID: <20160128203836.7e65ed2d@x230>

Hello,

On Thu, 28 Jan 2016 17:29:41 +0100
Victor Stinner <victor.stinner at gmail.com> wrote:

> We slowly remove old platforms, but only if the code specific to these
> old platforms is annoying to maintain. For example, I wrote the
> change:
> 
> https://hg.python.org/cpython/rev/a1605d2508af
> """
> Issue #22591: Drop support of MS-DOS
> 
> Drop support of MS-DOS, especially of the DJGPP compiler (MS-DOS port
> of GCC).
> 
> Today is a sad day. Good bye MS-DOS, good bye my friend :'-(
> """
> => http://bugs.python.org/issue22591

Well, MicroPython is ready to take the baton - FreeDOS (that's how they
call MS-DOS now) support was added recently:
https://github.com/micropython/micropython/commit/64a909ef5113925adef19f275f62473de8ee68c5

> 
> Victor
> 

[]

-- 
Best regards,
 Paul                          mailto:pmiscml at gmail.com

From rosuav at gmail.com  Thu Jan 28 22:47:03 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 29 Jan 2016 14:47:03 +1100
Subject: [Python-Dev] Buildbot timing out - test suite failure -
 test_socket issue with UDP6?
In-Reply-To: <CAPTjJmpv4L9znRyp5MFLraEedthZzkTaNJVWWrroz7qHwe_9ug@mail.gmail.com>
References: <CAPTjJmrYBMGD9c5q3a0ap-+PBHg+eHwCTsK+d6CHFdQTPoK9Yg@mail.gmail.com>
 <CAMpsgwb5cDsYiLONYg53DMCgoN-J+rktNfnOepdaP5NdhDYTtg@mail.gmail.com>
 <CAPTjJmoWDFoSeAWVBm5s1CutZ2fp7+L2Q_=JTe2a+5WKcAFi=Q@mail.gmail.com>
 <CA+eR4cHDDeetbRSx4tg_0kzh_H0x-hBxy0EUsqMXQiZ54MPfMg@mail.gmail.com>
 <CAPTjJmpv4L9znRyp5MFLraEedthZzkTaNJVWWrroz7qHwe_9ug@mail.gmail.com>
Message-ID: <CAPTjJmr0ZKHApLScGzd-BCqNZvu6uSqVw0Fzys1gujdJoix_rw@mail.gmail.com>

On Thu, Jan 28, 2016 at 9:41 PM, Chris Angelico <rosuav at gmail.com> wrote:
> However, as I was doing so (and I just discarded a draft message where
> I'd been typing up notes), my entire system went kerblooie, and I've
> spent the last day rebuilding stuff from scratch. When I get around to
> it, I'll rebuild the buildbot VM - and it'll be Debian Jessie (current
> stable), because there's no particular reason to use Wheezy
> (oldstable). So the problem will almost certainly disappear.

Sure enough, the problem is no longer reproducible. Sorry folks.
Hopefully it wasn't actually a bug anywhere, but was some bizarre
piece of VM setup weirdness.

The "AMD64 Debian root" buildbot (angelico-debian-amd64) is now
running Debian Jessie, rather than the Wheezy it was before.

ChrisA

From pludemann at google.com  Fri Jan 29 03:05:09 2016
From: pludemann at google.com (Peter Ludemann)
Date: Fri, 29 Jan 2016 00:05:09 -0800
Subject: [Python-Dev] FAT Python (lack of) performance
In-Reply-To: <CADiSq7fnYnE6ksN1hQv-hCMeeDo0P4hKzjAaU+za048eF+VMjQ@mail.gmail.com>
References: <CAMpsgwbixc8NM3Hh3nr9nw8U4E-bdn5__yVAiOwuph8yJe_LAw@mail.gmail.com>
 <56A698CF.3080402@mail.de>
 <CAMpsgwZEqXLSk=PK4zbvT2AKS6DmwShxPvBjPbB+jXOh3pNASw@mail.gmail.com>
 <CAEfz+Tw24dRXhaBBh_GnLd_snxOjkWo7QmXf6UEs7kjK3OU+ug@mail.gmail.com>
 <D713E9D8-78F4-4A3C-8557-C4A57DE04B85@yahoo.com>
 <CAEfz+TyxSC_=uJkdXd8tgbRVyusXwkoLFowj5xuynn+QxUQ1_w@mail.gmail.com>
 <56A7AE7C.3080806@mail.de>
 <CADiSq7foUzpqms-6qcqghPJcedf+cJD3-571YdRGSrWQOOfb6A@mail.gmail.com>
 <56A90F2C.9000401@mail.de>
 <CADiSq7d4q4iBUw6XvSjwyjbOHgZdaU0f=XwpGFgGt8yHePPWnQ@mail.gmail.com>
 <CAEfz+TwMZyzYcY80v3PyLf7jVSQ6131S66hyRk4tY-wzYr1YpA@mail.gmail.com>
 <CADiSq7fnYnE6ksN1hQv-hCMeeDo0P4hKzjAaU+za048eF+VMjQ@mail.gmail.com>
Message-ID: <CACsRUKKFfqrAcsoxKFO1a6W4u5BaqxbUHt4pOdiUr5DuSCtjGA@mail.gmail.com>

About benchmarks ... I've been there and it's not benchmarks that decide
whether something succeeds or fails.
(I found this old discussion
<https://groups.google.com/forum/#!topic/comp.lang.lisp/pF_wjA_A9fc> which
mentions FIB (also TAK <https://en.wikipedia.org/wiki/Tak_(function)>,
which is rather more brutal)  ... do you recognize the language that got an
implementation that was competitive with C in performance, was vastly more
expressive, yet failed to catch on?)

OTOH, good performance is never a bad thing and sometimes is a necessity;
so I applau this work.


On 28 January 2016 at 01:39, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 28 January 2016 at 18:30, INADA Naoki <songofacandy at gmail.com> wrote:
> > Please stop.
> >
> > I'm sorry about messing up this thread.
>
> Not your fault at all! This is just a particular bugbear of mine,
> since software architecture design (including appropriate programming
> language selection) is an even more poorly understood discipline than
> software development in general :)
>
> > I just wanted to represent why I'm very interested in Victor's efforts.
>
> And thanks for posting that, as it is indeed cool that the
> optimisation efforts currently being discussed may result in
> performance improvements on some of the simplified micro-benchmarks
> popular in programming language shootouts.
>
> There's no way you could have anticipated the subsequent tangential
> discussion on motives for contributing to open source projects, and
> the impact that has on what we can reasonably expect from fellow
> contributors.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/pludemann%40google.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160129/e4f8cbb9/attachment.html>

From stefan_ml at behnel.de  Fri Jan 29 05:00:02 2016
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 29 Jan 2016 11:00:02 +0100
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A90B97.7090001@gmail.com>
References: <56A90B97.7090001@gmail.com>
Message-ID: <n8fd74$j33$1@ger.gmane.org>

Yury Selivanov schrieb am 27.01.2016 um 19:25:
> tl;dr The summary is that I have a patch that improves CPython performance
> up to 5-10% on macro benchmarks.  Benchmarks results on Macbook Pro/Mac OS
> X, desktop CPU/Linux, server CPU/Linux are available at [1].  There are no
> slowdowns that I could reproduce consistently.
> 
> There are two different optimizations that yield this speedup:
> LOAD_METHOD/CALL_METHOD opcodes and per-opcode cache in ceval loop.
> 
> LOAD_METHOD & CALL_METHOD
> -------------------------
> 
> We had a lot of conversations with Victor about his PEP 509, and he sent me
> a link to his amazing compilation of notes about CPython performance [2]. 
> One optimization that he pointed out to me was LOAD/CALL_METHOD opcodes, an
> idea first originated in PyPy.
> 
> There is a patch that implements this optimization, it's tracked here:
> [3].  There are some low level details that I explained in the issue, but
> I'll go over the high level design in this email as well.
> 
> Every time you access a method attribute on an object, a BoundMethod object
> is created. It is a fairly expensive operation, despite a freelist of
> BoundMethods (so that memory allocation is generally avoided).  The idea is
> to detect what looks like a method call in the compiler, and emit a pair of
> specialized bytecodes for that.
> 
> So instead of LOAD_GLOBAL/LOAD_ATTR/CALL_FUNCTION we will have
> LOAD_GLOBAL/LOAD_METHOD/CALL_METHOD.
> 
> LOAD_METHOD looks at the object on top of the stack, and checks if the name
> resolves to a method or to a regular attribute.  If it's a method, then we
> push the unbound method object and the object to the stack.  If it's an
> attribute, we push the resolved attribute and NULL.
> 
> When CALL_METHOD looks at the stack it knows how to call the unbound method
> properly (pushing the object as a first arg), or how to call a regular
> callable.
> 
> This idea does make CPython faster around 2-4%.  And it surely doesn't make
> it slower.  I think it's a safe bet to at least implement this optimization
> in CPython 3.6.
> 
> So far, the patch only optimizes positional-only method calls. It's
> possible to optimize all kind of calls, but this will necessitate 3 more
> opcodes (explained in the issue).  We'll need to do some careful
> benchmarking to see if it's really needed.

I implemented a similar but simpler optimisation in Cython a while back:

http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html

Instead of avoiding the creation of method objects, as you proposed, it
just normally calls getattr and if that returns a bound method object, it
uses inlined calling code that avoids re-packing the argument tuple.
Interestingly, I got speedups of 5-15% for some of the Python benchmarks,
but I don't quite remember which ones (at least raytrace and richards, I
think), nor do I recall the overall gain, which (I assume) is what you are
referring to with your 2-4% above. Might have been in the same order.

Stefan



From damien.p.george at gmail.com  Fri Jan 29 07:38:53 2016
From: damien.p.george at gmail.com (Damien George)
Date: Fri, 29 Jan 2016 12:38:53 +0000
Subject: [Python-Dev] Speeding up CPython 5-10%
Message-ID: <CANw+Mzm2LYh7qMd3xx-VcFJO6qPq2E3KpaiCGQySUB3=DSQKwg@mail.gmail.com>

Hi Yury,

> An off-topic: have you ever tried hg.python.org/benchmarks
> or compare MicroPython vs CPython?  I'm curious if MicroPython
> is faster -- in that case we'll try to copy some optimization
> ideas.

I've tried a small number of those benchmarks, but not in any rigorous
way, and not enough to compare properly with CPython.  Maybe one day I
(or someone) will get to it and report results :)

One thing that makes MP fast is the use of pointer tagging and
stuffing of small integers within object pointers.  Thus integer
arithmetic below 2**30 (on 32-bit arch) requires no heap.

> Do you use opcode dictionary caching only for LOAD_GLOBAL-like
> opcodes?  Do you have an equivalent of LOAD_FAST, or you use
> dicts to store local variables?

The opcodes that have dict caching are:

LOAD_NAME
LOAD_GLOBAL
LOAD_ATTR
STORE_ATTR
LOAD_METHOD (not implemented yet in mainline repo)

For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST).
Actually, there are 16 dedicated opcodes for loading from positions
0-15, and 16 for storing to these positions.  Eg:

LOAD_FAST_0
LOAD_FAST_1
...

Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte.

> If we change the opcode size, it will probably affect libraries
> that compose or modify code objects.  Modules like "dis" will
> also need to be updated.  And that's probably just a tip of the
> iceberg.
>
> We can still implement your approach if we add a separate
> private 'unsigned char' array to each code object, so that
> LOAD_GLOBAL can store the key offsets.  It should be a bit
> faster than my current patch, since it has one less level
> of indirection.  But this way we loose the ability to
> optimize LOAD_METHOD, simply because it requires more memory
> for its cache.  In any case, I'll experiment!

Problem with that approach (having a separate array for offset_guess)
is that how do you know where to look into that array for a given
LOAD_GLOBAL opcode?  The second LOAD_GLOBAL in your bytecode should
look into the second entry in the array, but how does it know?

I'd love to experiment implementing my original caching idea with
CPython, but no time!

Cheers,
Damien.

From yselivanov.ml at gmail.com  Fri Jan 29 09:49:11 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Fri, 29 Jan 2016 09:49:11 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <n8fd74$j33$1@ger.gmane.org>
References: <56A90B97.7090001@gmail.com> <n8fd74$j33$1@ger.gmane.org>
Message-ID: <56AB7BE7.9060902@gmail.com>



On 2016-01-29 5:00 AM, Stefan Behnel wrote:
> Yury Selivanov schrieb am 27.01.2016 um 19:25:
>> [..]
>>
>> LOAD_METHOD looks at the object on top of the stack, and checks if the name
>> resolves to a method or to a regular attribute.  If it's a method, then we
>> push the unbound method object and the object to the stack.  If it's an
>> attribute, we push the resolved attribute and NULL.
>>
>> When CALL_METHOD looks at the stack it knows how to call the unbound method
>> properly (pushing the object as a first arg), or how to call a regular
>> callable.
>>
>> This idea does make CPython faster around 2-4%.  And it surely doesn't make
>> it slower.  I think it's a safe bet to at least implement this optimization
>> in CPython 3.6.
>>
>> So far, the patch only optimizes positional-only method calls. It's
>> possible to optimize all kind of calls, but this will necessitate 3 more
>> opcodes (explained in the issue).  We'll need to do some careful
>> benchmarking to see if it's really needed.
> I implemented a similar but simpler optimisation in Cython a while back:
>
> http://blog.behnel.de/posts/faster-python-calls-in-cython-021.html
>
> Instead of avoiding the creation of method objects, as you proposed, it
> just normally calls getattr and if that returns a bound method object, it
> uses inlined calling code that avoids re-packing the argument tuple.
> Interestingly, I got speedups of 5-15% for some of the Python benchmarks,
> but I don't quite remember which ones (at least raytrace and richards, I
> think), nor do I recall the overall gain, which (I assume) is what you are
> referring to with your 2-4% above. Might have been in the same order.
>
That's great!

I'm still working on the patch, but so far it looks like adding
just LOAD_METHOD/CALL_METHOD (that avoid instantiating BoundMethods)
gives us 10-15% faster method calls.

Combining them with my opcode cache makes them 30-35% faster.

Yury

From yselivanov.ml at gmail.com  Fri Jan 29 10:06:38 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Fri, 29 Jan 2016 10:06:38 -0500
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <CANw+Mzm2LYh7qMd3xx-VcFJO6qPq2E3KpaiCGQySUB3=DSQKwg@mail.gmail.com>
References: <CANw+Mzm2LYh7qMd3xx-VcFJO6qPq2E3KpaiCGQySUB3=DSQKwg@mail.gmail.com>
Message-ID: <56AB7FFE.7010004@gmail.com>

Hi Damien,

BTW I just saw (and backed!) your new Kickstarter campaign
to port MicroPython to ESP8266, good stuff!

On 2016-01-29 7:38 AM, Damien George wrote:
> Hi Yury,
>
> [..]
>> Do you use opcode dictionary caching only for LOAD_GLOBAL-like
>> opcodes?  Do you have an equivalent of LOAD_FAST, or you use
>> dicts to store local variables?
> The opcodes that have dict caching are:
>
> LOAD_NAME
> LOAD_GLOBAL
> LOAD_ATTR
> STORE_ATTR
> LOAD_METHOD (not implemented yet in mainline repo)
>
> For local variables we use LOAD_FAST and STORE_FAST (and DELETE_FAST).
> Actually, there are 16 dedicated opcodes for loading from positions
> 0-15, and 16 for storing to these positions.  Eg:
>
> LOAD_FAST_0
> LOAD_FAST_1
> ...
>
> Mostly this is done to save RAM, since LOAD_FAST_0 is 1 byte.

Interesting.  This might actually make CPython slightly faster
too.  Worth trying.

>
>> If we change the opcode size, it will probably affect libraries
>> that compose or modify code objects.  Modules like "dis" will
>> also need to be updated.  And that's probably just a tip of the
>> iceberg.
>>
>> We can still implement your approach if we add a separate
>> private 'unsigned char' array to each code object, so that
>> LOAD_GLOBAL can store the key offsets.  It should be a bit
>> faster than my current patch, since it has one less level
>> of indirection.  But this way we loose the ability to
>> optimize LOAD_METHOD, simply because it requires more memory
>> for its cache.  In any case, I'll experiment!
> Problem with that approach (having a separate array for offset_guess)
> is that how do you know where to look into that array for a given
> LOAD_GLOBAL opcode?  The second LOAD_GLOBAL in your bytecode should
> look into the second entry in the array, but how does it know?
>
>

I've changed my approach a little bit.  Now I have a simple
function [1] to initialize the cache for code objects that
are called frequently enough.

It walks through the code object's opcodes and creates the
appropriate  offset/cache tables.

Then, in ceval loop I have a couple of convenient macros
to work with the cache [2].  They use INSTR_OFFSET() macro
to locate the cache entry via the offset table initialized
by [1].

Thanks,
Yury

[1] https://github.com/1st1/cpython/blob/opcache4/Objects/codeobject.c#L167
[2] https://github.com/1st1/cpython/blob/opcache4/Python/ceval.c#L1164

From steve.dower at python.org  Fri Jan 29 12:05:18 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Jan 2016 09:05:18 -0800
Subject: [Python-Dev] More optimisation ideas
Message-ID: <56AB9BCE.2080000@python.org>

Since we're all talking about making Python faster, I thought I'd drop 
some previous ideas I've had here in case (1) someone wants to actually 
do them, and (2) they really are new ideas that haven't failed in the 
past. Mostly I was thinking about startup time.

Here are the list of modules imported on clean startup on my Windows, 
US-English machine (from -v and cleaned up a bit):

import _frozen_importlib
import _imp
import sys
import '_warnings'
import '_thread'
import '_weakref'
import '_frozen_importlib_external'
import '_io'
import 'marshal'
import 'nt'
import '_thread'
import '_weakref'
import 'winreg'
import 'zipimport'
import '_codecs'
import 'codecs'
import 'encodings.aliases'
import 'encodings'
import 'encodings.mbcs'
import '_signal'
import 'encodings.utf_8'
import 'encodings.latin_1'
import '_weakrefset'
import 'abc'
import 'io'
import 'encodings.cp437'
import 'errno'
import '_stat'
import 'stat'
import 'genericpath'
import 'ntpath'
import '_collections_abc'
import 'os'
import '_sitebuiltins'
import 'sysconfig'
import '_locale'
import '_bootlocale'
import 'encodings.cp1252'
import 'site'

Obviously the easiest first thing is to remove or delay unnecessary 
imports. But a while ago I used a native profiler to trace through this 
and the most impactful modules were the encodings:

import 'encodings.mbcs'
import 'encodings.utf_8'
import 'encodings.latin_1'
import 'encodings.cp437'
import 'encodings.cp1252'

While I don't doubt that we need all of these for *some* reason, 
aliases, cp437 and cp1252 are relatively expensive modules to import. 
Mostly due to having large static dictionaries or data structures 
generated on startup.

Given this is static and mostly read-only information[1], I see no 
reason why we couldn't either generate completely static versions of 
them, or better yet compile the resulting data structures into the core 
binary.

([1]: If being able to write to some of the encoding data is used by 
some people, I vote for breaking that for 3.6 and making it read-only.)

This is probably the code snippet that bothered me the most:

     ### Encoding table
     encoding_table=codecs.charmap_build(decoding_table)

It shows up in many of the encodings modules, and while it is not a bad 
function in itself, we are obviously generating a known data structure 
on every startup. Storing these in static data is a tradeoff between 
disk space and startup performance, and one I think it likely to be 
worthwhile.

Anyway, just an idea if someone wants to try it and see what 
improvements we can get. I'd love to do it myself, but when it actually 
comes to finding time I keep coming up short.

Cheers,
Steve


P.S. If you just want to discuss optimisation techniques or benchmarking 
in general, without specific application to CPython 3.6, there's a whole 
internet out there. Please don't make me the cause of a pointless 
centithread. :)

From status at bugs.python.org  Fri Jan 29 12:08:30 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 29 Jan 2016 18:08:30 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20160129170830.60F3156645@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-01-22 - 2016-01-29)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5381 (+27)
  closed 32615 (+30)
  total  37996 (+57)

Open issues with patches: 2359 


Issues opened (39)
==================

#26182: Deprecation warnings for the future async and await keywords
http://bugs.python.org/issue26182  opened by marco.buttu

#26184: raise an error when create_module() is not defined by exec_mod
http://bugs.python.org/issue26184  opened by brett.cannon

#26185: zipfile.ZipInfo slots can raise unexpected AttributeError
http://bugs.python.org/issue26185  opened by Matthew Zipay

#26186: LazyLoader rejecting use of SourceFileLoader
http://bugs.python.org/issue26186  opened by brett.cannon

#26187: sqlite3 trace callback prints duplicate line
http://bugs.python.org/issue26187  opened by palaviv

#26188: Provide more helpful error message when `await` is called insi
http://bugs.python.org/issue26188  opened by Nicholas Chammas

#26192: python3 k1om dissociation permanence: libffi
http://bugs.python.org/issue26192  opened by mancoast

#26193: python3 k1om dissociation permanence: readelf
http://bugs.python.org/issue26193  opened by mancoast

#26194: Undefined behavior for deque.insert() when len(d) == maxlen
http://bugs.python.org/issue26194  opened by rhettinger

#26195: Windows frozen .exe multiprocessing.Queue access is denied exc
http://bugs.python.org/issue26195  opened by alex_python_org

#26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow 
http://bugs.python.org/issue26198  opened by hniksic

#26200: SETREF adds unnecessary work in some cases
http://bugs.python.org/issue26200  opened by rhettinger

#26204: compiler: ignore constants used as statements? (don't emit LOA
http://bugs.python.org/issue26204  opened by haypo

#26205: Inconsistency concerning nested scopes
http://bugs.python.org/issue26205  opened by Roscoe R. Higgins

#26207: distutils msvccompiler fails due to mspdb140.dll error on debu
http://bugs.python.org/issue26207  opened by haypo

#26208: decimal C module's exceptions don't match the Python version
http://bugs.python.org/issue26208  opened by encukou

#26209: TypeError in smtpd module with string arguments
http://bugs.python.org/issue26209  opened by lorenzo.ancora

#26210: `HTMLParser.handle_data` may be invoked although `HTMLParser.r
http://bugs.python.org/issue26210  opened by Hibou57

#26212: Python with ncurses6.0 will not load _curses module on Solaris
http://bugs.python.org/issue26212  opened by jonesrw

#26213: Document BUILD_*_UNPACK opcodes
http://bugs.python.org/issue26213  opened by brett.cannon

#26214: textwrap should minimize number of breaks in extra long words
http://bugs.python.org/issue26214  opened by Tuomas Salo

#26215: Make GC_Head a compile-time option
http://bugs.python.org/issue26215  opened by yuriy_levchenko

#26216: run runtktests.py error when test tkinter
http://bugs.python.org/issue26216  opened by allensll

#26218: Set PrependPath default to true
http://bugs.python.org/issue26218  opened by Wallison Resende Santos

#26219: implement per-opcode cache in ceval
http://bugs.python.org/issue26219  opened by yselivanov

#26221: asynco run_in_executor swallows StopIteration
http://bugs.python.org/issue26221  opened by ikelly

#26222: Missing code in linux_distribution python 2.7.11
http://bugs.python.org/issue26222  opened by Rasmus Rynning Rasmussen

#26223: decimal.to_eng_string() does not implement engineering notatio
http://bugs.python.org/issue26223  opened by serge.stroobandt

#26224: Add "version added" for documentation of asyncio.timeout for d
http://bugs.python.org/issue26224  opened by Udi Oron

#26225: New misleading wording in execution model documenation
http://bugs.python.org/issue26225  opened by abarnert

#26226: Various test suite failures on Windows
http://bugs.python.org/issue26226  opened by ebarry

#26228: pty.spawn hangs on FreeBSD 9.3, 10.x
http://bugs.python.org/issue26228  opened by chris.torek

#26229: Make number serialization ES6/V8 compatible
http://bugs.python.org/issue26229  opened by anders.rundgren.net at gmail.com

#26231: HTTPResponse.close() should consume all remaining data in body
http://bugs.python.org/issue26231  opened by Jacky

#26233: select.epoll.poll() should avoid calling malloc() each time
http://bugs.python.org/issue26233  opened by haypo

#26234: The typing module includes 're' and 'io' in __all__
http://bugs.python.org/issue26234  opened by gvanrossum

#26235: argparse docs: Positional * argument in mutually exclusive gro
http://bugs.python.org/issue26235  opened by paul.j3

#26236: urllib2 initiate irregular call to gethostbyaddr
http://bugs.python.org/issue26236  opened by juliadolgova

#26238: httplib use wrong hostname in https request with SNI support
http://bugs.python.org/issue26238  opened by lvhancy



Most recent 15 issues with no replies (15)
==========================================

#26238: httplib use wrong hostname in https request with SNI support
http://bugs.python.org/issue26238

#26236: urllib2 initiate irregular call to gethostbyaddr
http://bugs.python.org/issue26236

#26235: argparse docs: Positional * argument in mutually exclusive gro
http://bugs.python.org/issue26235

#26224: Add "version added" for documentation of asyncio.timeout for d
http://bugs.python.org/issue26224

#26216: run runtktests.py error when test tkinter
http://bugs.python.org/issue26216

#26214: textwrap should minimize number of breaks in extra long words
http://bugs.python.org/issue26214

#26209: TypeError in smtpd module with string arguments
http://bugs.python.org/issue26209

#26200: SETREF adds unnecessary work in some cases
http://bugs.python.org/issue26200

#26195: Windows frozen .exe multiprocessing.Queue access is denied exc
http://bugs.python.org/issue26195

#26193: python3 k1om dissociation permanence: readelf
http://bugs.python.org/issue26193

#26192: python3 k1om dissociation permanence: libffi
http://bugs.python.org/issue26192

#26187: sqlite3 trace callback prints duplicate line
http://bugs.python.org/issue26187

#26185: zipfile.ZipInfo slots can raise unexpected AttributeError
http://bugs.python.org/issue26185

#26184: raise an error when create_module() is not defined by exec_mod
http://bugs.python.org/issue26184

#26176: EmailMessage example doesn't work
http://bugs.python.org/issue26176



Most recent 15 issues waiting for review (15)
=============================================

#26233: select.epoll.poll() should avoid calling malloc() each time
http://bugs.python.org/issue26233

#26228: pty.spawn hangs on FreeBSD 9.3, 10.x
http://bugs.python.org/issue26228

#26224: Add "version added" for documentation of asyncio.timeout for d
http://bugs.python.org/issue26224

#26219: implement per-opcode cache in ceval
http://bugs.python.org/issue26219

#26204: compiler: ignore constants used as statements? (don't emit LOA
http://bugs.python.org/issue26204

#26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow 
http://bugs.python.org/issue26198

#26194: Undefined behavior for deque.insert() when len(d) == maxlen
http://bugs.python.org/issue26194

#26192: python3 k1om dissociation permanence: libffi
http://bugs.python.org/issue26192

#26185: zipfile.ZipInfo slots can raise unexpected AttributeError
http://bugs.python.org/issue26185

#26177: tkinter: Canvas().keys returns empty strings.
http://bugs.python.org/issue26177

#26175: Fully implement IOBase abstract on SpooledTemporaryFile
http://bugs.python.org/issue26175

#26173: test_ssl.bad_cert_test() exception handling
http://bugs.python.org/issue26173

#26168: Py_BuildValue may leak 'N' arguments on PyTuple_New failure
http://bugs.python.org/issue26168

#26167: Improve copy.copy speed for built-in types (list/set/dict)
http://bugs.python.org/issue26167

#26145: PEP 511: Add sys.set_code_transformers()
http://bugs.python.org/issue26145



Top 10 most discussed issues (10)
=================================

#26194: Undefined behavior for deque.insert() when len(d) == maxlen
http://bugs.python.org/issue26194  17 msgs

#26207: distutils msvccompiler fails due to mspdb140.dll error on debu
http://bugs.python.org/issue26207  12 msgs

#26198: PyArg_ParseTuple with format "et#" and "es#" detects overflow 
http://bugs.python.org/issue26198  11 msgs

#26039: More flexibility in zipfile interface
http://bugs.python.org/issue26039   7 msgs

#26219: implement per-opcode cache in ceval
http://bugs.python.org/issue26219   7 msgs

#26223: decimal.to_eng_string() does not implement engineering notatio
http://bugs.python.org/issue26223   7 msgs

#19883: Integer overflow in zipimport.c
http://bugs.python.org/issue19883   6 msgs

#20598: argparse docs: '7'.split() is confusing magic
http://bugs.python.org/issue20598   6 msgs

#25314: Documentation: argparse's actions store_{true,false} default t
http://bugs.python.org/issue25314   6 msgs

#26204: compiler: ignore constants used as statements? (don't emit LOA
http://bugs.python.org/issue26204   6 msgs



Issues closed (27)
==================

#18018: SystemError: Parent module '' not loaded, cannot perform relat
http://bugs.python.org/issue18018  closed by brett.cannon

#18898: Apply the setobject optimizations to dictionaries
http://bugs.python.org/issue18898  closed by rhettinger

#19023: ctypes docs: Unimplemented and undocumented features
http://bugs.python.org/issue19023  closed by martin.panter

#22363: argparse AssertionError with add_mutually_exclusive_group and 
http://bugs.python.org/issue22363  closed by martin.panter

#24705: sysconfig._parse_makefile doesn't expand ${} vars appearing be
http://bugs.python.org/issue24705  closed by berker.peksag

#25296: Simple End-of-life guide covering all unsupported versions
http://bugs.python.org/issue25296  closed by berker.peksag

#26034: venv documentation out of date
http://bugs.python.org/issue26034  closed by berker.peksag

#26146: PEP 511: Add ast.Constant to allow AST optimizer to emit const
http://bugs.python.org/issue26146  closed by haypo

#26181: argparse can't handle positional argument after list (help mes
http://bugs.python.org/issue26181  closed by martin.panter

#26183: 2.7.11 won't clean install on Windows 10 x64
http://bugs.python.org/issue26183  closed by Roger Cook

#26189: Interpreter returns control to cmd.exe early
http://bugs.python.org/issue26189  closed by Ivan.Pozdeev

#26190: GC memory leak using weak and cyclic references
http://bugs.python.org/issue26190  closed by koehlma

#26191: pip on Windows doesn't honor Case
http://bugs.python.org/issue26191  closed by SilentGhost

#26196: Argparse breaks when a switch is given an argument beginning w
http://bugs.python.org/issue26196  closed by eric.smith

#26197: arange from numpy function has some limits....I propose a pyth
http://bugs.python.org/issue26197  closed by ebarry

#26199: fix broken link to hamcrest.library.integration.match_equality
http://bugs.python.org/issue26199  closed by berker.peksag

#26201: Faster type checking in listobject.c
http://bugs.python.org/issue26201  closed by rhettinger

#26202: The range() object is deepcopied as atomic
http://bugs.python.org/issue26202  closed by serhiy.storchaka

#26203: nesting venv in virtualenv segfaults
http://bugs.python.org/issue26203  closed by brett.cannon

#26206: test_socket.testRecvmsgPeek() timeout on "AMD64 Debian root 3.
http://bugs.python.org/issue26206  closed by martin.panter

#26211: HTMLParser: ???AssertionError: we should not get here!???
http://bugs.python.org/issue26211  closed by Hibou57

#26217: Fatal error when importing ``test.test_os`` in debug mode on W
http://bugs.python.org/issue26217  closed by ebarry

#26220: Unicode HOWTO references a question mark that isn't in snippet
http://bugs.python.org/issue26220  closed by martin.panter

#26227: Windows: socket.gethostbyaddr(name) fails for non-ASCII hostna
http://bugs.python.org/issue26227  closed by haypo

#26230: Cookies do not correct if cookiename includes [ or ]
http://bugs.python.org/issue26230  closed by tom_lt

#26232: Mock(spec=spec) has no effect
http://bugs.python.org/issue26232  closed by michael.foord

#26237: UnboundLocalError error while handling exception
http://bugs.python.org/issue26237  closed by haypo

From storchaka at gmail.com  Fri Jan 29 13:14:00 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 29 Jan 2016 20:14:00 +0200
Subject: [Python-Dev] How to resolve distinguishing between documentation
 and implementation
Message-ID: <n8ga5a$osd$1@ger.gmane.org>

How to resolve distinguishing between documentation and implementation 
if current implementation is incorrect, but third-party code can 
implicitly depends on it?

For example see issue26198. Currently buffer overflow of predefined 
buffer for "es#" and "et#" format units causes TypeError (with 
misleading message, but this is other story). The correct and 
*documented* exception is ValueError. User code can depend on current 
behavior, because TypeError is what is raised now for this type of 
errors, and this is what is raised for other types of errors. Unlikely 
authors of such code read the documentation, otherwise this issue would 
be reported earlier. On other hand, looks these format units are rarely 
used with predefined buffer (never in the stdlib since 3.5).

I think it is obvious that the code in the development branch should be 
changed to produce documented and more logical exception. But what about 
bugfix releases? Changing the documentation would be misleading, 
changing the code can break existing code (unlikely, but).


From brett at python.org  Fri Jan 29 13:26:32 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 29 Jan 2016 18:26:32 +0000
Subject: [Python-Dev] How to resolve distinguishing between
 documentation and implementation
In-Reply-To: <n8ga5a$osd$1@ger.gmane.org>
References: <n8ga5a$osd$1@ger.gmane.org>
Message-ID: <CAP1=2W5d85GnufTdhzBwBjHBkDoo_2W9GDWccCbOtK80J95yRw@mail.gmail.com>

On Fri, 29 Jan 2016 at 10:14 Serhiy Storchaka <storchaka at gmail.com> wrote:

> How to resolve distinguishing between documentation and implementation
> if current implementation is incorrect, but third-party code can
> implicitly depends on it?
>
> For example see issue26198. Currently buffer overflow of predefined
> buffer for "es#" and "et#" format units causes TypeError (with
> misleading message, but this is other story). The correct and
> *documented* exception is ValueError. User code can depend on current
> behavior, because TypeError is what is raised now for this type of
> errors, and this is what is raised for other types of errors. Unlikely
> authors of such code read the documentation, otherwise this issue would
> be reported earlier. On other hand, looks these format units are rarely
> used with predefined buffer (never in the stdlib since 3.5).
>
> I think it is obvious that the code in the development branch should be
> changed to produce documented and more logical exception. But what about
> bugfix releases? Changing the documentation would be misleading,
> changing the code can break existing code (unlikely, but).
>

When the potential breakage is low, I would move to the more reasonable
exception and add the appropriate note to the docs and What's New about how
to port pre-existing code by either changing the exception caught or
catching both exceptions until support < 3.6 can be dropped by the user.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160129/ffc5949d/attachment.html>

From guido at python.org  Fri Jan 29 13:24:48 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 29 Jan 2016 10:24:48 -0800
Subject: [Python-Dev] How to resolve distinguishing between
 documentation and implementation
In-Reply-To: <n8ga5a$osd$1@ger.gmane.org>
References: <n8ga5a$osd$1@ger.gmane.org>
Message-ID: <CAP7+vJJZFDsgd7s2EffEyYenxpYXEAOsJU-6+GrLj21q6oS1SA@mail.gmail.com>

Typically we fix this in the next feature release but not in bugfix
releases, and that's what I recommend in this case. But deciding
remains an art, not an exact science.

People who catch specific lists of exceptions based on experience are
bound to run into trouble occasionally, so I have little pity on them
when they upgrade to the next feature release (especially when the new
behavior is more logical and follows an established standard). For
bugfix releases I like to be much more conservative, since people
often don't have control over which bugfix release is installed.

On Fri, Jan 29, 2016 at 10:14 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> How to resolve distinguishing between documentation and implementation if
> current implementation is incorrect, but third-party code can implicitly
> depends on it?
>
> For example see issue26198. Currently buffer overflow of predefined buffer
> for "es#" and "et#" format units causes TypeError (with misleading message,
> but this is other story). The correct and *documented* exception is
> ValueError. User code can depend on current behavior, because TypeError is
> what is raised now for this type of errors, and this is what is raised for
> other types of errors. Unlikely authors of such code read the documentation,
> otherwise this issue would be reported earlier. On other hand, looks these
> format units are rarely used with predefined buffer (never in the stdlib
> since 3.5).
>
> I think it is obvious that the code in the development branch should be
> changed to produce documented and more logical exception. But what about
> bugfix releases? Changing the documentation would be misleading, changing
> the code can break existing code (unlikely, but).
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From francismb at email.de  Fri Jan 29 16:54:35 2016
From: francismb at email.de (francismb)
Date: Fri, 29 Jan 2016 22:54:35 +0100
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56AB9BCE.2080000@python.org>
References: <56AB9BCE.2080000@python.org>
Message-ID: <56ABDF9B.9040502@email.de>

Hi,

> 
> Storing these in static data is a tradeoff between
> disk space and startup performance, and one I think it likely to be
> worthwhile.

it's really an important trade off? As far a I understand from your
email those modules are always being loaded and the final data created.
won't the space be there (on mem or disk)?

Thanks in advance!
francis



From steve.dower at python.org  Fri Jan 29 22:48:41 2016
From: steve.dower at python.org (Steve Dower)
Date: Fri, 29 Jan 2016 19:48:41 -0800
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56ABDF9B.9040502@email.de>
References: <56AB9BCE.2080000@python.org> <56ABDF9B.9040502@email.de>
Message-ID: <E1aPMWv-0005XF-5R@se2-syd.hostedmail.net.au>

It doesn't currently end up on disk. Some tables are partially or completely stored on disk as Python source code (some are partially generated from simple rules), but others are generated by inverting those. That process takes time that could be avoided by storing the generated tables, and storing all of it in a format that doesn't require parsing, compiling and executing (such as a native array).

Potentially it could be a win all around if we stopped including the (larger) source files, but that doesn't seem like a good idea for maintaining portability to other implementations. The main thought is making the compiler binary bigger to avoid generating encoding tables at startup.

Top-posted from my Windows Phone

-----Original Message-----
From: "francismb" <francismb at email.de>
Sent: ?1/?29/?2016 13:56
To: "python-dev at python.org" <python-dev at python.org>
Subject: Re: [Python-Dev] More optimisation ideas

Hi,

> 
> Storing these in static data is a tradeoff between
> disk space and startup performance, and one I think it likely to be
> worthwhile.

it's really an important trade off? As far a I understand from your
email those modules are always being loaded and the final data created.
won't the space be there (on mem or disk)?

Thanks in advance!
francis


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160129/fbfa4ee4/attachment-0001.html>

From steve at pearwood.info  Fri Jan 29 23:28:35 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 30 Jan 2016 15:28:35 +1100
Subject: [Python-Dev] Speeding up CPython 5-10%
In-Reply-To: <56A90B97.7090001@gmail.com>
References: <56A90B97.7090001@gmail.com>
Message-ID: <20160130042835.GJ4619@ando.pearwood.info>

On Wed, Jan 27, 2016 at 01:25:27PM -0500, Yury Selivanov wrote:
> Hi,
> 
> 
> tl;dr The summary is that I have a patch that improves CPython 
> performance up to 5-10% on macro benchmarks.  Benchmarks results on 
> Macbook Pro/Mac OS X, desktop CPU/Linux, server CPU/Linux are available 
> at [1].  There are no slowdowns that I could reproduce consistently.

Have you looked at Cesare Di Mauro's wpython? As far as I know, it's now 
unmaintained, and the project repo on Google Code appears to be dead (I 
get a 404), but I understand that it was significantly faster than 
CPython back in the 2.6 days.

https://wpython.googlecode.com/files/Beyond%20Bytecode%20-%20A%20Wordcode-based%20Python.pdf



-- 
Steve

From oscar.j.benjamin at gmail.com  Sat Jan 30 08:54:54 2016
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sat, 30 Jan 2016 13:54:54 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <E1aPMWv-0005XF-5R@se2-syd.hostedmail.net.au>
References: <56AB9BCE.2080000@python.org> <56ABDF9B.9040502@email.de>
 <E1aPMWv-0005XF-5R@se2-syd.hostedmail.net.au>
Message-ID: <CAHVvXxR_edmw-thvhpBzgGc1wvca6UQKP3oR70qXD51nDVZUWQ@mail.gmail.com>

On 30 January 2016 at 03:48, Steve Dower <steve.dower at python.org> wrote:
>
> It doesn't currently end up on disk. Some tables are partially or completely
> stored on disk as Python source code (some are partially generated from
> simple rules), but others are generated by inverting those. That process
> takes time that could be avoided by storing the generated tables, and
> storing all of it in a format that doesn't require parsing, compiling and
> executing (such as a native array).
>
> Potentially it could be a win all around if we stopped including the
> (larger) source files, but that doesn't seem like a good idea for
> maintaining portability to other implementations. The main thought is making
> the compiler binary bigger to avoid generating encoding tables at startup.

When I last tried to profile startup on Windows (I haven't used
Windows for some time now) it seemed that the time was totally
dominated by file system access. Essentially the limiting factor was
the inordinate number of stat calls and small file accesses. Although
this was probably Python 2.x which may not import those particular
modules and maybe it depends on virus scanner software etc.

Things may have changed now but I concluded that substantive gains
could only come from improving FS access. Perhaps something like
zipping up the standard library would see a big improvement.

--
Oscar

From storchaka at gmail.com  Sat Jan 30 09:45:44 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 30 Jan 2016 16:45:44 +0200
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56AB9BCE.2080000@python.org>
References: <56AB9BCE.2080000@python.org>
Message-ID: <n8iiap$t6v$1@ger.gmane.org>

On 29.01.16 19:05, Steve Dower wrote:
> This is probably the code snippet that bothered me the most:
>
>      ### Encoding table
>      encoding_table=codecs.charmap_build(decoding_table)
>
> It shows up in many of the encodings modules, and while it is not a bad
> function in itself, we are obviously generating a known data structure
> on every startup. Storing these in static data is a tradeoff between
> disk space and startup performance, and one I think it likely to be
> worthwhile.

$ ./python -m timeit -s "import codecs; from encodings.cp437 import 
decoding_table" -- "codecs.charmap_build(decoding_table)"
100000 loops, best of 3: 4.36 usec per loop

Getting rid from charmap_build() would save you at most 4.4 microseconds 
per encoding. 0.0005 seconds if you have imported *all* standard encodings!

And how you expected to store encoding_table in more efficient way?


From steve.dower at python.org  Sat Jan 30 11:31:32 2016
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Jan 2016 08:31:32 -0800
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <n8iiap$t6v$1@ger.gmane.org>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
Message-ID: <56ACE564.7080107@python.org>

On 30Jan2016 0645, Serhiy Storchaka wrote:
> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
> decoding_table" -- "codecs.charmap_build(decoding_table)"
> 100000 loops, best of 3: 4.36 usec per loop
>
> Getting rid from charmap_build() would save you at most 4.4 microseconds
> per encoding. 0.0005 seconds if you have imported *all* standard encodings!

Just as happy to be proven wrong. Perhaps I misinterpreted my original 
profiling and then, embarrassingly, ran with the result for a long time 
without retesting.

> And how you expected to store encoding_table in more efficient way?

There's nothing inefficient about its storage, but as it does not change 
it would be trivial to store it statically. Then "building" the map is 
simply obtaining a pointer into an already loaded memory page. Much 
faster than building it on load, but both are clearly insignificant 
compared to other factors.

Cheers,
Steve


From storchaka at gmail.com  Sat Jan 30 13:20:08 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 30 Jan 2016 20:20:08 +0200
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56ACE564.7080107@python.org>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org>
Message-ID: <n8iusq$rif$1@ger.gmane.org>

On 30.01.16 18:31, Steve Dower wrote:
> On 30Jan2016 0645, Serhiy Storchaka wrote:
>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
>> decoding_table" -- "codecs.charmap_build(decoding_table)"
>> 100000 loops, best of 3: 4.36 usec per loop
>>
>> Getting rid from charmap_build() would save you at most 4.4 microseconds
>> per encoding. 0.0005 seconds if you have imported *all* standard
>> encodings!
>
> Just as happy to be proven wrong. Perhaps I misinterpreted my original
> profiling and then, embarrassingly, ran with the result for a long time
> without retesting.

AFAIK the most time is spent in system calls like stat or open. 
Archiving the stdlib into the ZIP file and using zipimport can decrease 
Python startup time (perhaps there is an open issue about this).



From brett at python.org  Sat Jan 30 14:09:43 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 30 Jan 2016 19:09:43 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <n8iusq$rif$1@ger.gmane.org>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
Message-ID: <CAP1=2W5v+7P_QjOQMaznowq8sXZwoxgUz3xhc2PpD33NYxXdeA@mail.gmail.com>

On Sat, 30 Jan 2016 at 10:21 Serhiy Storchaka <storchaka at gmail.com> wrote:

> On 30.01.16 18:31, Steve Dower wrote:
> > On 30Jan2016 0645, Serhiy Storchaka wrote:
> >> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
> >> decoding_table" -- "codecs.charmap_build(decoding_table)"
> >> 100000 loops, best of 3: 4.36 usec per loop
> >>
> >> Getting rid from charmap_build() would save you at most 4.4 microseconds
> >> per encoding. 0.0005 seconds if you have imported *all* standard
> >> encodings!
> >
> > Just as happy to be proven wrong. Perhaps I misinterpreted my original
> > profiling and then, embarrassingly, ran with the result for a long time
> > without retesting.
>
> AFAIK the most time is spent in system calls like stat or open.
> Archiving the stdlib into the ZIP file and using zipimport can decrease
> Python startup time (perhaps there is an open issue about this).
>

Check the archives, but  I did trying freezing the entire stdlib and it
didn't really make a difference in startup, so I don't know if this still
holds true anymore.

At this point I think all of our knowledge of what takes the most amount of
time during startup is outdated and someone should try to really profile
the whole thing to see where the hotspots are (e.g., is it stat calls from
imports, is it actually some specific function, is it just so many little
things adding up to a big thing, etc.).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160130/ffde28f9/attachment.html>

From steve.dower at python.org  Sat Jan 30 14:15:26 2016
From: steve.dower at python.org (Steve Dower)
Date: Sat, 30 Jan 2016 11:15:26 -0800
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <n8iusq$rif$1@ger.gmane.org>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
Message-ID: <E1aPazp-0002sm-WA@se2-syd.hostedmail.net.au>

Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup.

Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer.

Top-posted from my Windows Phone

-----Original Message-----
From: "Serhiy Storchaka" <storchaka at gmail.com>
Sent: ?1/?30/?2016 10:22
To: "python-dev at python.org" <python-dev at python.org>
Subject: Re: [Python-Dev] More optimisation ideas

On 30.01.16 18:31, Steve Dower wrote:
> On 30Jan2016 0645, Serhiy Storchaka wrote:
>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
>> decoding_table" -- "codecs.charmap_build(decoding_table)"
>> 100000 loops, best of 3: 4.36 usec per loop
>>
>> Getting rid from charmap_build() would save you at most 4.4 microseconds
>> per encoding. 0.0005 seconds if you have imported *all* standard
>> encodings!
>
> Just as happy to be proven wrong. Perhaps I misinterpreted my original
> profiling and then, embarrassingly, ran with the result for a long time
> without retesting.

AFAIK the most time is spent in system calls like stat or open. 
Archiving the stdlib into the ZIP file and using zipimport can decrease 
Python startup time (perhaps there is an open issue about this).


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160130/f9357a58/attachment.html>

From srkunze at mail.de  Sat Jan 30 15:29:20 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Sat, 30 Jan 2016 21:29:20 +0100
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <n8iusq$rif$1@ger.gmane.org>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
Message-ID: <56AD1D20.5090205@mail.de>

On 30.01.2016 19:20, Serhiy Storchaka wrote:
> AFAIK the most time is spent in system calls like stat or open. 
> Archiving the stdlib into the ZIP file and using zipimport can 
> decrease Python startup time (perhaps there is an open issue about this).

Oh, please don't. One thing I love about Python is the ease of access.

I personally think that startup time is not really a big issue; even 
when it comes to microbenchmarks.

Best,
Sven

From brett at python.org  Sat Jan 30 15:32:55 2016
From: brett at python.org (Brett Cannon)
Date: Sat, 30 Jan 2016 20:32:55 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56AD1D20.5090205@mail.de>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
 <56AD1D20.5090205@mail.de>
Message-ID: <CAP1=2W60b9fQnUbFE3drCVhK+-ybOSBKP60pdGof=Q9vxFBykg@mail.gmail.com>

On Sat, Jan 30, 2016, 12:30 Sven R. Kunze <srkunze at mail.de> wrote:

> On 30.01.2016 19:20, Serhiy Storchaka wrote:
> > AFAIK the most time is spent in system calls like stat or open.
> > Archiving the stdlib into the ZIP file and using zipimport can
> > decrease Python startup time (perhaps there is an open issue about this).
>
> Oh, please don't. One thing I love about Python is the ease of access.
>

It wouldn't be a requirement, just a nootion


> I personally think that startup time is not really a big issue; even
> when it comes to microbenchmarks.
>

You might not, but just about every command-line app does.

-brett


> Best,
> Sven
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160130/ea057046/attachment.html>

From srkunze at mail.de  Sat Jan 30 16:50:13 2016
From: srkunze at mail.de (Sven R. Kunze)
Date: Sat, 30 Jan 2016 22:50:13 +0100
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <CAP1=2W60b9fQnUbFE3drCVhK+-ybOSBKP60pdGof=Q9vxFBykg@mail.gmail.com>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
 <56AD1D20.5090205@mail.de>
 <CAP1=2W60b9fQnUbFE3drCVhK+-ybOSBKP60pdGof=Q9vxFBykg@mail.gmail.com>
Message-ID: <56AD3015.6010003@mail.de>

On 30.01.2016 21:32, Brett Cannon wrote:
> On Sat, Jan 30, 2016, 12:30 Sven R. Kunze <srkunze at mail.de 
> <mailto:srkunze at mail.de>> wrote:
>
>     On 30.01.2016 19:20, Serhiy Storchaka wrote:
>     > AFAIK the most time is spent in system calls like stat or open.
>     > Archiving the stdlib into the ZIP file and using zipimport can
>     > decrease Python startup time (perhaps there is an open issue
>     about this).
>
>     Oh, please don't. One thing I love about Python is the ease of access.
>
>
> It wouldn't be a requirement, just a nootion
>

That's good. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160130/c3e96f32/attachment.html>

From larry at hastings.org  Sun Jan 31 10:11:06 2016
From: larry at hastings.org (Larry Hastings)
Date: Sun, 31 Jan 2016 07:11:06 -0800
Subject: [Python-Dev] Fun with ancient unsupported platforms
In-Reply-To: <56AA2C50.7050700@hastings.org>
References: <56AA2C50.7050700@hastings.org>
Message-ID: <56AE240A.1040807@hastings.org>


On 01/28/2016 06:57 AM, Larry Hastings wrote:
> It's currently 2016.  Perhaps it's time to remove all vestiges of 
> these unsupported operating systems nobody's cared about since a year 
> that started with a '1'?

We dropped support for Irix in 2.3.  We dropped support for Irix threads 
in 3.2.  All our supported platforms have Thread Local Storage (TLS) 
support.  Maybe we can drop our 250-line portable TLS library from 
Python/thread.c?


//arry/

p.s. Derpy code in Python/thread_nt.h.  It literally looks like this:

    /* use native Windows TLS functions */
    #define Py_HAVE_NATIVE_TLS

    #ifdef Py_HAVE_NATIVE_TLS

It seems this developer had the short-term memory of a goldfish.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/76a87161/attachment.html>

From antoine at python.org  Sun Jan 31 11:56:21 2016
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 31 Jan 2016 16:56:21 +0000 (UTC)
Subject: [Python-Dev] More optimisation ideas
References: <56AB9BCE.2080000@python.org>
Message-ID: <loom.20160131T175237-527@post.gmane.org>


Hi,

If you want to make startup time faster for a broad range of applications,
please consider adding a lazy import facility in the stdlib.
I recently tried to write a lazy import mechanism using import hooks
(to make it portable from 2.6 to 3.5), it seems nearly impossible to do
so (or, at least, for an average Python programmer like me).

This would be much more useful (for actual users, not for architecture
astronauts) than refactoring the importlib APIs in each feature version...

Thanks in advance

Antoine.



From brett at python.org  Sun Jan 31 12:02:30 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 31 Jan 2016 17:02:30 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <loom.20160131T175237-527@post.gmane.org>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
Message-ID: <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>

A lazy importer was added in Python 3.5 and it was not possible without the
module spec refactoring.

On Sun, 31 Jan 2016, 08:57 Antoine Pitrou <antoine at python.org> wrote:

>
> Hi,
>
> If you want to make startup time faster for a broad range of applications,
> please consider adding a lazy import facility in the stdlib.
> I recently tried to write a lazy import mechanism using import hooks
> (to make it portable from 2.6 to 3.5), it seems nearly impossible to do
> so (or, at least, for an average Python programmer like me).
>
> This would be much more useful (for actual users, not for architecture
> astronauts) than refactoring the importlib APIs in each feature version...
>
> Thanks in advance
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/e29d70a6/attachment.html>

From antoine at python.org  Sun Jan 31 12:09:49 2016
From: antoine at python.org (Antoine Pitrou)
Date: Sun, 31 Jan 2016 17:09:49 +0000 (UTC)
Subject: [Python-Dev] More optimisation ideas
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
Message-ID: <loom.20160131T180641-918@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:

> 
> 
> A lazy importer was added in Python 3.5 and it was not possible
> without the module spec refactoring.

Wow... Thank you, I didn't know about that.

Now for the next question: how am I supposed to use it?

The following documentation leaves me absolutely clueless:

"""This class only works with loaders that define exec_module() as control
over what module type is used for the module is required. For those same
reasons, the loader?s create_module() method will be ignored (i.e., the
loader?s method should only return None). Finally, modules which substitute
the object placed into sys.modules will not work as there is no way to
properly replace the module references throughout the interpreter safely;
ValueError is raised if such a substitution is detected."""

(reference:
https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader)

I want to import lazily the modules from package "foobar.*", but not
other modules as other libraries may depend on import side effects.
How do I do that? The quoted snippet doesn't really help.

Regards

Antoine.

From donald at stufft.io  Sun Jan 31 12:11:23 2016
From: donald at stufft.io (Donald Stufft)
Date: Sun, 31 Jan 2016 12:11:23 -0500
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
Message-ID: <B4703C79-B82D-442B-BFB8-50A0EEA6837D@stufft.io>


> On Jan 31, 2016, at 12:02 PM, Brett Cannon <brett at python.org> wrote:
> 
> A lazy importer was added in Python 3.5


Is there any docs on how to actually use the LazyLoader in 3.5? I can?t seem to find any but I don?t really know the import system that well.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/58d22a83/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/58d22a83/attachment.sig>

From brett at python.org  Sun Jan 31 12:26:19 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 31 Jan 2016 17:26:19 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <B4703C79-B82D-442B-BFB8-50A0EEA6837D@stufft.io>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
 <B4703C79-B82D-442B-BFB8-50A0EEA6837D@stufft.io>
Message-ID: <CAP1=2W4-2KcEw4XFkkAszad5Cavcpfpp9GmH9SAqnsKjdvREPg@mail.gmail.com>

There are no example docs for it yet, but enough people have asked this
week about how to set up a custom importer that I will write up a generic
example case which will make sense for a lazy loader (need to file the
issue before I forget).

On Sun, 31 Jan 2016, 09:11 Donald Stufft <donald at stufft.io> wrote:

>
> On Jan 31, 2016, at 12:02 PM, Brett Cannon <brett at python.org> wrote:
>
> A lazy importer was added in Python 3.5
>
>
> Is there any docs on how to actually use the LazyLoader in 3.5? I can?t
> seem to find any but I don?t really know the import system that well.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/dbd99b8f/attachment.html>

From brett at python.org  Sun Jan 31 12:57:12 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 31 Jan 2016 17:57:12 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <CAP1=2W4-2KcEw4XFkkAszad5Cavcpfpp9GmH9SAqnsKjdvREPg@mail.gmail.com>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
 <B4703C79-B82D-442B-BFB8-50A0EEA6837D@stufft.io>
 <CAP1=2W4-2KcEw4XFkkAszad5Cavcpfpp9GmH9SAqnsKjdvREPg@mail.gmail.com>
Message-ID: <CAP1=2W4d-FHCTz=n6vMpX4B77g+4pZF36iHgkLrpAG3FQCc2Qg@mail.gmail.com>

I have opened http://bugs.python.org/issue26252 to track writing the
example (and before ppl go playing with the lazy loader, be aware of
http://bugs.python.org/issue26186).

On Sun, 31 Jan 2016 at 09:26 Brett Cannon <brett at python.org> wrote:

> There are no example docs for it yet, but enough people have asked this
> week about how to set up a custom importer that I will write up a generic
> example case which will make sense for a lazy loader (need to file the
> issue before I forget).
>
> On Sun, 31 Jan 2016, 09:11 Donald Stufft <donald at stufft.io> wrote:
>
>>
>> On Jan 31, 2016, at 12:02 PM, Brett Cannon <brett at python.org> wrote:
>>
>> A lazy importer was added in Python 3.5
>>
>>
>> Is there any docs on how to actually use the LazyLoader in 3.5? I can?t
>> seem to find any but I don?t really know the import system that well.
>>
>> -----------------
>> Donald Stufft
>> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
>> DCFA
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/ecd5e439/attachment.html>

From mal at egenix.com  Sun Jan 31 13:43:20 2016
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 31 Jan 2016 19:43:20 +0100
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <E1aPazp-0002sm-WA@se2-syd.hostedmail.net.au>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
 <E1aPazp-0002sm-WA@se2-syd.hostedmail.net.au>
Message-ID: <56AE55C8.8000807@egenix.com>

On 30.01.2016 20:15, Steve Dower wrote:
> Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup.
> 
> Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer.

Freezing the entire stdlib does improve the startup time,
simply because it removes stat calls, which dominate the startup
time at least on Unix.

It also allows sharing the stdlib byte code in memory, since it gets
stored in static C structs which the OS will happily mmap into
multiple processes for you without any additional effort.

Our eGenix PyRun does exactly that. Even though the original
motivation is a different one, the gained improvement in
startup time is a nice side effect:

    http://www.egenix.com/products/python/PyRun/

Aside: The encodings don't really make much difference here. The
dictionaries aren't all that big, so generating them on the fly doesn't
really create much overhead. The trade off in terms of maintainability/speed
definitely leans toward maintainability. For the larger encoding
tables we already have C implementations with appropriate data
structures to make lookup speed vs. storage needs efficient.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 31 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


> Top-posted from my Windows Phone
> 
> -----Original Message-----
> From: "Serhiy Storchaka" <storchaka at gmail.com>
> Sent: ?1/?30/?2016 10:22
> To: "python-dev at python.org" <python-dev at python.org>
> Subject: Re: [Python-Dev] More optimisation ideas
> 
> On 30.01.16 18:31, Steve Dower wrote:
>> On 30Jan2016 0645, Serhiy Storchaka wrote:
>>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
>>> decoding_table" -- "codecs.charmap_build(decoding_table)"
>>> 100000 loops, best of 3: 4.36 usec per loop
>>>
>>> Getting rid from charmap_build() would save you at most 4.4 microseconds
>>> per encoding. 0.0005 seconds if you have imported *all* standard
>>> encodings!
>>
>> Just as happy to be proven wrong. Perhaps I misinterpreted my original
>> profiling and then, embarrassingly, ran with the result for a long time
>> without retesting.
> 
> AFAIK the most time is spent in system calls like stat or open. 
> Archiving the stdlib into the ZIP file and using zipimport can decrease 
> Python startup time (perhaps there is an open issue about this).
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> 


From brett at python.org  Sun Jan 31 15:23:00 2016
From: brett at python.org (Brett Cannon)
Date: Sun, 31 Jan 2016 20:23:00 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <56AE55C8.8000807@egenix.com>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
 <E1aPazp-0002sm-WA@se2-syd.hostedmail.net.au> <56AE55C8.8000807@egenix.com>
Message-ID: <CAP1=2W5=N2Q_gkw_FuLR2u7+MDKsNKu-ae0tOScEGY4HmoFJPQ@mail.gmail.com>

So freezing the stdlib helps on UNIX and not on OS X (if my old testing is
still accurate). I guess the next question is what it does on Windows and
if we would want to ever consider freezing the stdlib as part of the build
process (and if we would want to change the order of importers on
sys.meta_path so frozen modules came after file-based ones).

On Sun, 31 Jan 2016, 10:43 M.-A. Lemburg <mal at egenix.com> wrote:

> On 30.01.2016 20:15, Steve Dower wrote:
> > Brett tried freezing the entire stdlib at one point (as we do for parts
> of importlib) and reported no significant improvement. Since that rules out
> code compilation as well as the OS calls, it'd seem the priority is to
> execute less code on startup.
> >
> > Details of that work were posted to python-dev about twelve months ago,
> IIRC. Maybe a little longer.
>
> Freezing the entire stdlib does improve the startup time,
> simply because it removes stat calls, which dominate the startup
> time at least on Unix.
>
> It also allows sharing the stdlib byte code in memory, since it gets
> stored in static C structs which the OS will happily mmap into
> multiple processes for you without any additional effort.
>
> Our eGenix PyRun does exactly that. Even though the original
> motivation is a different one, the gained improvement in
> startup time is a nice side effect:
>
>     http://www.egenix.com/products/python/PyRun/
>
> Aside: The encodings don't really make much difference here. The
> dictionaries aren't all that big, so generating them on the fly doesn't
> really create much overhead. The trade off in terms of
> maintainability/speed
> definitely leans toward maintainability. For the larger encoding
> tables we already have C implementations with appropriate data
> structures to make lookup speed vs. storage needs efficient.
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, Jan 31 2016)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...           http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
>                       http://www.malemburg.com/
>
>
> > Top-posted from my Windows Phone
> >
> > -----Original Message-----
> > From: "Serhiy Storchaka" <storchaka at gmail.com>
> > Sent: ?1/?30/?2016 10:22
> > To: "python-dev at python.org" <python-dev at python.org>
> > Subject: Re: [Python-Dev] More optimisation ideas
> >
> > On 30.01.16 18:31, Steve Dower wrote:
> >> On 30Jan2016 0645, Serhiy Storchaka wrote:
> >>> $ ./python -m timeit -s "import codecs; from encodings.cp437 import
> >>> decoding_table" -- "codecs.charmap_build(decoding_table)"
> >>> 100000 loops, best of 3: 4.36 usec per loop
> >>>
> >>> Getting rid from charmap_build() would save you at most 4.4
> microseconds
> >>> per encoding. 0.0005 seconds if you have imported *all* standard
> >>> encodings!
> >>
> >> Just as happy to be proven wrong. Perhaps I misinterpreted my original
> >> profiling and then, embarrassingly, ran with the result for a long time
> >> without retesting.
> >
> > AFAIK the most time is spent in system calls like stat or open.
> > Archiving the stdlib into the ZIP file and using zipimport can decrease
> > Python startup time (perhaps there is an open issue about this).
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
> >
> >
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mal%40egenix.com
> >
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/79d2a1be/attachment-0001.html>

From aarmour at cipmail.org  Sun Jan 31 15:16:27 2016
From: aarmour at cipmail.org (ty armour)
Date: Sun, 31 Jan 2016 15:16:27 -0500
Subject: [Python-Dev] just a friendly developer
Message-ID: <CAC=V4WfiGyT1vQ3T46F8Q5HM_kCsT8svk7r7KDU3RWbmWGEiqA@mail.gmail.com>

I am looking for tutorials on basically how to write the python language.
As well I want to know how to write wrappers for things like alsa and
directx and coreaudio and jackd and pulseaudio.

I would be looking to write wrappers for coreaudio in mac and coreaudio for
windows as well as alsa. i think it would benefit you to have this
information.

I am looking to aid in development of computers and languages and tutorials
on how to write the actual python language would be super useful.

but yeah im looking to write applications like guitarix rakarrack blender
ardour  lmms.

I am also going to build my own computers so these tutorials would help a
lot.

anyhow thanks for your time
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160131/0d113c44/attachment.html>

From ethan at stoneleaf.us  Sun Jan 31 17:43:22 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 31 Jan 2016 14:43:22 -0800
Subject: [Python-Dev] just a friendly developer
In-Reply-To: <CAC=V4WfiGyT1vQ3T46F8Q5HM_kCsT8svk7r7KDU3RWbmWGEiqA@mail.gmail.com>
References: <CAC=V4WfiGyT1vQ3T46F8Q5HM_kCsT8svk7r7KDU3RWbmWGEiqA@mail.gmail.com>
Message-ID: <56AE8E0A.2010906@stoneleaf.us>

On 01/31/2016 12:16 PM, ty armour wrote:

 > I am looking for tutorials on basically how to write the python
 > language.

Try asking on the python-list [1] mailing list, as that is for general 
discussion of Python.  This list is for developing Python itself.

--
~Ethan~


[1] https://mail.python.org/mailman/listinfo/python-list

From tjreedy at udel.edu  Sun Jan 31 18:35:44 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 31 Jan 2016 18:35:44 -0500
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <loom.20160131T180641-918@post.gmane.org>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
 <loom.20160131T180641-918@post.gmane.org>
Message-ID: <n8m5oj$k22$1@ger.gmane.org>

On 1/31/2016 12:09 PM, Antoine Pitrou wrote:

> The following documentation leaves me absolutely clueless:
>
> """This class only works with loaders that define exec_module() as control
> over what module type is used for the module is required.

No wonder.  I cannot parse it as an English sentence. It needs rewriting.

> For those same
> reasons, the loader?s create_module() method will be ignored (i.e., the
> loader?s method should only return None). Finally, modules which substitute
> the object placed into sys.modules will not work as there is no way to
> properly replace the module references throughout the interpreter safely;
> ValueError is raised if such a substitution is detected."""
>
> (reference:
> https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader)

-- 
Terry Jan Reedy



From brett at python.org  Sun Jan 31 20:09:27 2016
From: brett at python.org (Brett Cannon)
Date: Mon, 01 Feb 2016 01:09:27 +0000
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <n8m5oj$k22$1@ger.gmane.org>
References: <56AB9BCE.2080000@python.org>
 <loom.20160131T175237-527@post.gmane.org>
 <CAP1=2W4gzQ2i0y61h3VqNVs2ZGfcACycJ_UE4WuxDr9iCSPLFQ@mail.gmail.com>
 <loom.20160131T180641-918@post.gmane.org> <n8m5oj$k22$1@ger.gmane.org>
Message-ID: <CAP1=2W4q_-PZN_DOHjZdKremnfGkzu5gYKozrM5A176ryC+qSw@mail.gmail.com>

On Sun, 31 Jan 2016, 15:36 Terry Reedy <tjreedy at udel.edu> wrote:

> On 1/31/2016 12:09 PM, Antoine Pitrou wrote:
>
> > The following documentation leaves me absolutely clueless:
> >
> > """This class only works with loaders that define exec_module() as
> control
> > over what module type is used for the module is required.
>
> No wonder.  I cannot parse it as an English sentence. It needs rewriting.
>

Feel free to open an issue to clarify the wording.

-Brett


> > For those same
> > reasons, the loader?s create_module() method will be ignored (i.e., the
> > loader?s method should only return None). Finally, modules which
> substitute
> > the object placed into sys.modules will not work as there is no way to
> > properly replace the module references throughout the interpreter safely;
> > ValueError is raised if such a substitution is detected."""
> >
> > (reference:
> >
> https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader
> )
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20160201/1e0c5ce6/attachment.html>

From ben+python at benfinney.id.au  Sun Jan 31 21:27:35 2016
From: ben+python at benfinney.id.au (Ben Finney)
Date: Mon, 01 Feb 2016 13:27:35 +1100
Subject: [Python-Dev] just a friendly developer
References: <CAC=V4WfiGyT1vQ3T46F8Q5HM_kCsT8svk7r7KDU3RWbmWGEiqA@mail.gmail.com>
Message-ID: <857fipdv1k.fsf@benfinney.id.au>

ty armour <aarmour at cipmail.org> writes:

> I am looking for tutorials on basically how to write the python
> language.

Newcomers to Python are especially invited to the Python ?tutor? forum
<URL:https://mail.python.org/mailman/listinfo/tutor>, for collaborative
tutoring in a friendly environment.

-- 
 \       ?The internet's completely over.? Anyway, all these computers |
  `\    and digital gadgets are no good. They just fill your head with |
_o__)     numbers and that can't be good for you.? ?Prince, 2010-07-05 |
Ben Finney


From steve at pearwood.info  Sun Jan 31 22:12:27 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 1 Feb 2016 14:12:27 +1100
Subject: [Python-Dev] More optimisation ideas
In-Reply-To: <CAP1=2W5=N2Q_gkw_FuLR2u7+MDKsNKu-ae0tOScEGY4HmoFJPQ@mail.gmail.com>
References: <56AB9BCE.2080000@python.org> <n8iiap$t6v$1@ger.gmane.org>
 <56ACE564.7080107@python.org> <n8iusq$rif$1@ger.gmane.org>
 <E1aPazp-0002sm-WA@se2-syd.hostedmail.net.au> <56AE55C8.8000807@egenix.com>
 <CAP1=2W5=N2Q_gkw_FuLR2u7+MDKsNKu-ae0tOScEGY4HmoFJPQ@mail.gmail.com>
Message-ID: <20160201031226.GF31806@ando.pearwood.info>

On Sun, Jan 31, 2016 at 08:23:00PM +0000, Brett Cannon wrote:
> So freezing the stdlib helps on UNIX and not on OS X (if my old testing is
> still accurate). I guess the next question is what it does on Windows and
> if we would want to ever consider freezing the stdlib as part of the build
> process (and if we would want to change the order of importers on
> sys.meta_path so frozen modules came after file-based ones).

I find that being able to easily open stdlib .py files in a text editor 
to read the source is extremely valuable. I've learned much more from 
reading the source than from (e.g.) StackOverflow. Likewise, it's often 
handy to do a grep over the stdlib. When you talk about freezing the 
stdlib, what exactly does that mean?

- will the source files still be there?

- how will this affect people writing patches for bugs?



-- 
Steve