From status at bugs.python.org  Fri May  2 18:07:53 2014
From: status at bugs.python.org (Python tracker)
Date: Fri,  2 May 2014 18:07:53 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20140502160753.687B85691B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2014-04-25 - 2014-05-02)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4605 ( +5)
  closed 28582 (+61)
  total  33187 (+66)

Open issues with patches: 2109 


Issues opened (49)
==================

#21350: bug in file.writelines accepting buffers
http://bugs.python.org/issue21350  opened by bdkearns

#21352: improve documentation indexing
http://bugs.python.org/issue21352  opened by bgailer

#21354: PyCFunction_New no longer exposed by python DLL breaking bdist
http://bugs.python.org/issue21354  opened by mhammond

#21356: Support LibreSSL (instead of OpenSSL): make RAND_egd optional
http://bugs.python.org/issue21356  opened by Edd.Barrett

#21357: Increase filecmp test coverage from 63% to 76%
http://bugs.python.org/issue21357  opened by diana

#21358: Augmented assignment doc: clarify 'only evaluated once'
http://bugs.python.org/issue21358  opened by terry.reedy

#21359: IDLE Redo command accelerator acts as Undo with current OS X C
http://bugs.python.org/issue21359  opened by ned.deily

#21360: mailbox.Maildir should ignore files named with a leading dot
http://bugs.python.org/issue21360  opened by liw

#21362: concurrent.futures does not validate that max_workers is prope
http://bugs.python.org/issue21362  opened by Claudiu.Popa

#21363: io.TextIOWrapper always closes wrapped files
http://bugs.python.org/issue21363  opened by aronacher

#21364: Documentation Recommends Broken Pattern
http://bugs.python.org/issue21364  opened by aronacher

#21365: asyncio.Task reference misses the most important fact about it
http://bugs.python.org/issue21365  opened by pfalcon

#21366: Document that return in finally overwrites prev value
http://bugs.python.org/issue21366  opened by brandjon

#21367: multiprocessing.JoinableQueue requires new kwarg
http://bugs.python.org/issue21367  opened by lee.clemens

#21368: Check for systemd locale on startup if current locale is set t
http://bugs.python.org/issue21368  opened by ncoghlan

#21370: segfault from simple traceback.format_exc call
http://bugs.python.org/issue21370  opened by John.Rusnak

#21372: multiprocessing.util.register_after_fork inconsistency
http://bugs.python.org/issue21372  opened by schlamar

#21373: robotparser: Automatically call modified function in read()
http://bugs.python.org/issue21373  opened by mlorant

#21375: Fix and test overflow behavior in the C version of heapq
http://bugs.python.org/issue21375  opened by rhettinger

#21376: asyncio docs refer to wrong TimeoutError
http://bugs.python.org/issue21376  opened by qmega

#21381: Python 3.4+ interpreter built on/with OS X 10.7 deployment tar
http://bugs.python.org/issue21381  opened by Christian.Tismer

#21382: Signal module doesnt raises ValueError Exception
http://bugs.python.org/issue21382  opened by rsevcan

#21383: "make touch" fails when the build directory is not the source 
http://bugs.python.org/issue21383  opened by ned.deily

#21384: Windows: Make handle non inheritable by default
http://bugs.python.org/issue21384  opened by haypo

#21385: Compiling modified AST crashes on debug build unless linenumbe
http://bugs.python.org/issue21385  opened by Ztane

#21386: ipaddress.IPv4Address.is_global not implemented
http://bugs.python.org/issue21386  opened by rluethi

#21387: Memory leaks when embedded interpreter is reinitialized
http://bugs.python.org/issue21387  opened by Evgeniy.Stepanov

#21389: The repr of BoundMethod objects sometimes incorrectly identifi
http://bugs.python.org/issue21389  opened by Steven.Barker

#21390: readline: setlocale() returns NULL on Android
http://bugs.python.org/issue21390  opened by lizhenhua.dev

#21391: PATCH: using the abspath shortcut in Lib/shutil
http://bugs.python.org/issue21391  opened by Yinon

#21392: Python on Cygwin: subprocess.call BlockingIOError: [Errno 11] 
http://bugs.python.org/issue21392  opened by dellair.jie

#21393: Python/random.c: close hCryptProv at exit
http://bugs.python.org/issue21393  opened by haypo

#21396: Python io implementation doesn't flush with write_through=True
http://bugs.python.org/issue21396  opened by akira

#21397: tempfile.py: Fix grammar in docstring, comment typos
http://bugs.python.org/issue21397  opened by modocache

#21398: LC_CTYPE=C:  pydoc leaves terminal in an unusable state
http://bugs.python.org/issue21398  opened by skrah

#21400: Code coverage documentation is out-of-date.
http://bugs.python.org/issue21400  opened by matrixise

#21401: python2 -3 does not warn about str/unicode to bytes conversion
http://bugs.python.org/issue21401  opened by Joshua.J.Cogliati

#21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists
http://bugs.python.org/issue21402  opened by Zero

#21403: cElementTree's Element creation handles attrib argument differ
http://bugs.python.org/issue21403  opened by santa4nt

#21404: Compression level for tarfile/zipfile
http://bugs.python.org/issue21404  opened by Sworddragon

#21405: Allow using symbols from Unicode block "Superscripts and Subsc
http://bugs.python.org/issue21405  opened by rominf

#21406: Some socket constants are not enums
http://bugs.python.org/issue21406  opened by pitrou

#21408: delegation of `!=` to the right-hand side argument is not alwa
http://bugs.python.org/issue21408  opened by exarkun

#21409: setup.py check - should fail and retrun a non 0 exit code
http://bugs.python.org/issue21409  opened by dundeemt

#21410: setup.py check --restructuredtext  -- appears to pass if docut
http://bugs.python.org/issue21410  opened by dundeemt

#21411: Enable Treat Warning as Error on 32-bit Windows
http://bugs.python.org/issue21411  opened by zach.ware

#21412: core dump in PyThreadState_Get when built --with-pymalloc
http://bugs.python.org/issue21412  opened by jbeck

#21413: urllib.request.urlopen dies on non-basic/digest auth schemes
http://bugs.python.org/issue21413  opened by samwyse

#21414: Add an intersperse function to itertools
http://bugs.python.org/issue21414  opened by javawizard



Most recent 15 issues with no replies (15)
==========================================

#21414: Add an intersperse function to itertools
http://bugs.python.org/issue21414

#21413: urllib.request.urlopen dies on non-basic/digest auth schemes
http://bugs.python.org/issue21413

#21411: Enable Treat Warning as Error on 32-bit Windows
http://bugs.python.org/issue21411

#21410: setup.py check --restructuredtext  -- appears to pass if docut
http://bugs.python.org/issue21410

#21397: tempfile.py: Fix grammar in docstring, comment typos
http://bugs.python.org/issue21397

#21389: The repr of BoundMethod objects sometimes incorrectly identifi
http://bugs.python.org/issue21389

#21386: ipaddress.IPv4Address.is_global not implemented
http://bugs.python.org/issue21386

#21385: Compiling modified AST crashes on debug build unless linenumbe
http://bugs.python.org/issue21385

#21383: "make touch" fails when the build directory is not the source 
http://bugs.python.org/issue21383

#21373: robotparser: Automatically call modified function in read()
http://bugs.python.org/issue21373

#21372: multiprocessing.util.register_after_fork inconsistency
http://bugs.python.org/issue21372

#21366: Document that return in finally overwrites prev value
http://bugs.python.org/issue21366

#21360: mailbox.Maildir should ignore files named with a leading dot
http://bugs.python.org/issue21360

#21352: improve documentation indexing
http://bugs.python.org/issue21352

#21350: bug in file.writelines accepting buffers
http://bugs.python.org/issue21350



Most recent 15 issues waiting for review (15)
=============================================

#21397: tempfile.py: Fix grammar in docstring, comment typos
http://bugs.python.org/issue21397

#21396: Python io implementation doesn't flush with write_through=True
http://bugs.python.org/issue21396

#21393: Python/random.c: close hCryptProv at exit
http://bugs.python.org/issue21393

#21391: PATCH: using the abspath shortcut in Lib/shutil
http://bugs.python.org/issue21391

#21390: readline: setlocale() returns NULL on Android
http://bugs.python.org/issue21390

#21386: ipaddress.IPv4Address.is_global not implemented
http://bugs.python.org/issue21386

#21383: "make touch" fails when the build directory is not the source 
http://bugs.python.org/issue21383

#21375: Fix and test overflow behavior in the C version of heapq
http://bugs.python.org/issue21375

#21373: robotparser: Automatically call modified function in read()
http://bugs.python.org/issue21373

#21362: concurrent.futures does not validate that max_workers is prope
http://bugs.python.org/issue21362

#21357: Increase filecmp test coverage from 63% to 76%
http://bugs.python.org/issue21357

#21354: PyCFunction_New no longer exposed by python DLL breaking bdist
http://bugs.python.org/issue21354

#21350: bug in file.writelines accepting buffers
http://bugs.python.org/issue21350

#21347: Don't use a list argument together with shell=True in subproce
http://bugs.python.org/issue21347

#21344: save scores or ratios in difflib get_close_matches
http://bugs.python.org/issue21344



Top 10 most discussed issues (10)
=================================

#21233: Add *Calloc functions to CPython memory allocation API
http://bugs.python.org/issue21233  53 msgs

#6839: zipfile can't extract file
http://bugs.python.org/issue6839  36 msgs

#21305: PEP 466: update os.urandom
http://bugs.python.org/issue21305  16 msgs

#18314: Have os.unlink remove junction points
http://bugs.python.org/issue18314  13 msgs

#2159: dbmmodule inquiry function is performance prohibitive
http://bugs.python.org/issue2159   9 msgs

#20962: Rather modest chunk size in gzip.GzipFile
http://bugs.python.org/issue20962   9 msgs

#21332: subprocess bufsize=1 docs are misleading
http://bugs.python.org/issue21332   9 msgs

#21398: LC_CTYPE=C:  pydoc leaves terminal in an unusable state
http://bugs.python.org/issue21398   9 msgs

#21037: add an AddressSanitizer build option
http://bugs.python.org/issue21037   8 msgs

#21403: cElementTree's Element creation handles attrib argument differ
http://bugs.python.org/issue21403   8 msgs



Issues closed (58)
==================

#9291: mimetypes initialization fails on Windows because of non-Latin
http://bugs.python.org/issue9291  closed by tim.golden

#9307: Py_TPFLAGS_LONG_SUBCLASS is not documented
http://bugs.python.org/issue9307  closed by pitrou

#9815: assertRaises as a context manager keeps tracebacks and frames 
http://bugs.python.org/issue9815  closed by pitrou

#10872: Add mode to TextIOWrapper repr
http://bugs.python.org/issue10872  closed by pitrou

#13204: sys.flags.__new__ crashes
http://bugs.python.org/issue13204  closed by pitrou

#17145: memoryview(array.array)
http://bugs.python.org/issue17145  closed by skrah

#17386: Bring Doc/make.bat as close to Doc/Makefile as possible
http://bugs.python.org/issue17386  closed by python-dev

#18185: Error in test_set.TestVariousIteratorArgs.test_inline_methods
http://bugs.python.org/issue18185  closed by berker.peksag

#18243: mktime_tz documentation out-of-date
http://bugs.python.org/issue18243  closed by r.david.murray

#18604: Consolidate gui available checks in test.support
http://bugs.python.org/issue18604  closed by python-dev

#18727: test for writing dictionary rows to CSV
http://bugs.python.org/issue18727  closed by pitrou

#19001: test_gdb fails on Fedora buildbot
http://bugs.python.org/issue19001  closed by skrah

#19385: dbm.dumb should be consistent when the database is closed
http://bugs.python.org/issue19385  closed by python-dev

#19420: Leak in _hashopenssl.c
http://bugs.python.org/issue19420  closed by pitrou

#19630: marshal.dump cannot write to temporary file
http://bugs.python.org/issue19630  closed by tim.golden

#19940: ssl.cert_time_to_seconds() returns wrong results if local time
http://bugs.python.org/issue19940  closed by pitrou

#19962: Create a 'python.bat' script to invoke interpreter from source
http://bugs.python.org/issue19962  closed by zach.ware

#19977: Use "surrogateescape" error handler for sys.stdin and sys.stdo
http://bugs.python.org/issue19977  closed by haypo

#20094: intermitent failures with test_dbm
http://bugs.python.org/issue20094  closed by ethan.furman

#20230: structseq types should expose _fields
http://bugs.python.org/issue20230  closed by skrah

#20355: -W command line options should have higher priority than PYTHO
http://bugs.python.org/issue20355  closed by pitrou

#20951: SSLSocket.send() returns 0 for non-blocking socket
http://bugs.python.org/issue20951  closed by pitrou

#21026: Document sitecustomize.py problems with pythonw
http://bugs.python.org/issue21026  closed by python-dev

#21040: socketserver: use selectors module
http://bugs.python.org/issue21040  closed by neologix

#21048: Index 'as' in import and with statements
http://bugs.python.org/issue21048  closed by terry.reedy

#21055: Index (augmented) assignment symbols
http://bugs.python.org/issue21055  closed by python-dev

#21057: TextIOWrapper does not support reading bytearrays or memoryvie
http://bugs.python.org/issue21057  closed by pitrou

#21138: mimetypes.MimeType UnicodeDecodeError
http://bugs.python.org/issue21138  closed by tim.golden

#21207: urandom persistent fd - not re-openned after fd close
http://bugs.python.org/issue21207  closed by pitrou

#21225: io.py: Improve docstrings for classes
http://bugs.python.org/issue21225  closed by pitrou

#21273: don't defined socket constants which are not implemented for G
http://bugs.python.org/issue21273  closed by pitrou

#21282: setup.py: More informative error msg for modules which built b
http://bugs.python.org/issue21282  closed by python-dev

#21312: Update thread_foobar.h to include timed locking and TLS suppor
http://bugs.python.org/issue21312  closed by pitrou

#21316: mark test_devpoll to be meaningful only for Solaris
http://bugs.python.org/issue21316  closed by jcea

#21320: dict() allows keyword expansion with integer keys, e.g. dict(*
http://bugs.python.org/issue21320  closed by eric.smith

#21321: itertools.islice() doesn't release reference to the source ite
http://bugs.python.org/issue21321  closed by pitrou

#21325: Missing Generic EXIF library for images in the standard librar
http://bugs.python.org/issue21325  closed by terry.reedy

#21340: Possible concurrency bug in asyncio, AttributeError in tasks.p
http://bugs.python.org/issue21340  closed by python-dev

#21341: Configuring 'font' with ttk.Style for 'TEntry' does not change
http://bugs.python.org/issue21341  closed by terry.reedy

#21349: crash in winreg SetValueEx with memoryview
http://bugs.python.org/issue21349  closed by tim.golden

#21351: refcounts not respected at process exit
http://bugs.python.org/issue21351  closed by minrk

#21353: document Popen.args attribute
http://bugs.python.org/issue21353  closed by gregory.p.smith

#21355: Shallow defaults to True, not 1 [filecmp.cmp]
http://bugs.python.org/issue21355  closed by python-dev

#21361: Add how to run a single test case to the devguide
http://bugs.python.org/issue21361  closed by python-dev

#21369: Extended modes for tarfile.TarFile()
http://bugs.python.org/issue21369  closed by lars.gustaebel

#21371: struct lconv does not have decimal_point on android platform
http://bugs.python.org/issue21371  closed by skrah

#21374: DecimalTuple.__module__ is set to _frozen_importlib
http://bugs.python.org/issue21374  closed by skrah

#21377: PyBytes_Concat could try to concat in-place
http://bugs.python.org/issue21377  closed by pitrou

#21378: ImportError: No module named 'concurrent.futures'
http://bugs.python.org/issue21378  closed by Bill.Bergmann

#21379: StopIteration is silenced when raised by generator within cont
http://bugs.python.org/issue21379  closed by ncoghlan

#21380: timezone support in strftime methods broken
http://bugs.python.org/issue21380  closed by ned.deily

#21388: decimal.InvalidContext is unused
http://bugs.python.org/issue21388  closed by skrah

#21394: Lib/random.py: use more efficient code to convert bytes to int
http://bugs.python.org/issue21394  closed by haypo

#21395: runtests.rst: link "build Python" to setup.rst#compiling
http://bugs.python.org/issue21395  closed by python-dev

#21399: inspect and class methods
http://bugs.python.org/issue21399  closed by yselivanov

#21407: Add function signatures to _decimal
http://bugs.python.org/issue21407  closed by skrah

#21415: Python __new__ method doc typo (it's a class and not a static 
http://bugs.python.org/issue21415  closed by steven.daprano

#1533105: NetBSD build with --with-pydebug causes SIGSEGV
http://bugs.python.org/issue1533105  closed by skrah

From zachary.ware+pydev at gmail.com  Fri May  2 22:06:57 2014
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Fri, 2 May 2014 15:06:57 -0500
Subject: [Python-Dev] [Python-checkins] devguide: Fix broken link to
 Skip's optimizer paper, update bug link
In-Reply-To: <3gL49p33dHz7Ll6@mail.python.org>
References: <3gL49p33dHz7Ll6@mail.python.org>
Message-ID: <CAKJDb-MG8F94rS_OkCp7QNicqm8J=+3kyiRFvGiPrygdniMXaw@mail.gmail.com>

On Fri, May 2, 2014 at 3:02 PM, zach.ware <python-checkins at python.org> wrote:
> http://hg.python.org/devguide/rev/375b0b0b186b
> changeset:   696:375b0b0b186b
> user:        Zachary Ware <zachary.ware at gmail.com>
> date:        Fri May 02 14:44:20 2014 -0500
> summary:
>   Fix broken link to Skip's optimizer paper, update bug link
>
> files:
>   compiler.rst |  4 ++--
>   1 files changed, 2 insertions(+), 2 deletions(-)
>
>
> diff --git a/compiler.rst b/compiler.rst
> --- a/compiler.rst
> +++ b/compiler.rst
> @@ -553,10 +553,10 @@
>  .. _SPARK: http://pages.cpsc.ucalgary.ca/~aycock/spark/
>
>  .. [#skip-peephole] Skip Montanaro's Peephole Optimizer Paper
> -   (http://www.python.org/workshops/1998-11/proceedings/papers/montanaro/montanaro.html)
> +   (http://www.smontanaro.net/python/spam7/optimizer.html)

Is this a good link or is there a 'better' one available?  This was
the first result of a Google search for "skip montanaro peephole
optimizer" and seems to be straight from the source, so I went with it
:)

Regards,
-- 
Zach

From skip at pobox.com  Sat May  3 00:32:19 2014
From: skip at pobox.com (Skip Montanaro)
Date: Fri, 2 May 2014 17:32:19 -0500
Subject: [Python-Dev] [Python-checkins] devguide: Fix broken link to
 Skip's optimizer paper, update bug link
In-Reply-To: <CAKJDb-MG8F94rS_OkCp7QNicqm8J=+3kyiRFvGiPrygdniMXaw@mail.gmail.com>
References: <3gL49p33dHz7Ll6@mail.python.org>
 <CAKJDb-MG8F94rS_OkCp7QNicqm8J=+3kyiRFvGiPrygdniMXaw@mail.gmail.com>
Message-ID: <CANc-5UxUMZyWYG-uDLRwtMLBN1dm0DOjZUMPtgOQByv2i5c-7g@mail.gmail.com>

On Fri, May 2, 2014 at 3:06 PM, Zachary Ware
<zachary.ware+pydev at gmail.com> wrote:
>> -   (http://www.python.org/workshops/1998-11/proceedings/papers/montanaro/montanaro.html)
>> +   (http://www.smontanaro.net/python/spam7/optimizer.html)
>
> Is this a good link or is there a 'better' one available?

Zach,

That's as a good a link as I know of. (Lot of water under the bridge
since then!)

Skip

From jessica.mckellar at gmail.com  Mon May  5 05:02:22 2014
From: jessica.mckellar at gmail.com (Jessica McKellar)
Date: Sun, 4 May 2014 20:02:22 -0700
Subject: [Python-Dev] Python under the sea and in space
Message-ID: <CAKDZRci2bdXSrRtn+UaqZ2i6vNt1s+ch_g__Ww13qRxWp6NkLg@mail.gmail.com>

Hi folks,

I'm trying to determine the greatest depth (in the ocean or underground)
and highest altitude at which Python code has been executed.

Please note that I'm interested in where the code was executed, and not,
say, where data that Python analyzed was acquired. I know for example that
NASA has analyzed plenty of data from space in Python, but the analysis was
done in labs here on Earth. :)

Do you have some good candidates? Please let me know!

Regards,
-Jessica
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140504/4d2ac68d/attachment.html>

From larry at hastings.org  Mon May  5 13:08:41 2014
From: larry at hastings.org (Larry Hastings)
Date: Mon, 05 May 2014 04:08:41 -0700
Subject: [Python-Dev] [RELEASED] Python 3.4.1rc1
Message-ID: <53677139.6010008@hastings.org>



On behalf of the Python development community and the Python 3.4 release 
team, I'm pleased to announce the availability of Python 3.4.1rc1.  
Python 3.4.1rc1 has over three hundred bugfixes and other improvements 
over 3.4.0. One notable change: the version of OpenSSL bundled with the 
Windows installer no longer has the "HeartBleed" vulnerability.

You can download it here:

    https://www.python.org/download/releases/3.4.1


One note: the "topics" data file for the pydoc is broken in this 
release.  This means the pydoc command and the built-in "help" will fail 
on some topics.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/4458597e/attachment.html>

From anthony.tuininga at gmail.com  Mon May  5 23:32:46 2014
From: anthony.tuininga at gmail.com (Anthony Tuininga)
Date: Mon, 5 May 2014 15:32:46 -0600
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
Message-ID: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>

Hi,

I am the author of cx_Freeze which creates "frozen" executables from Python
scripts. To this point I have been using frozen modules (compiled C) but
this has the side effect of bundling parts of Python with a cx_Freeze
release -- and this has bitten me on a number of occasions when newer
versions of Python use slightly incompatible modules. :-)

By scanning the code I discovered that Python automatically searches on
startup

<PYTHONHOME>/lib/pythonNN.zip

where NN is replaced by the major and minor version that is being executed.
Using this would allow me to ensure that bootstrap modules are not bundled
with cx_Freeze but acquired from the distribution that is actually using it
-- thereby eliminating the hassle noted above.

I have tested this approach and found it works flawlessly. There is,
however, no documentation that I can find for the contents of sys.path --
the documentation simply says that it is an installation default and
doesn't specify what that default is.

So my question is: can I safely make use of this "feature"? It has remained
in place since at least Python 2.6 but I don't want to assume anything.
Please advise! Thanks.

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/d4363e54/attachment.html>

From p.f.moore at gmail.com  Mon May  5 23:50:29 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 5 May 2014 22:50:29 +0100
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
Message-ID: <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>

On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com> wrote:
> So my question is: can I safely make use of this "feature"? It has remained
> in place since at least Python 2.6 but I don't want to assume anything.
> Please advise! Thanks.

I believe this is a 100% supported feature of Python (since 2.6, as
you mention). Some of the zip import/run features have unfortunately
been underdocumented, and I suspect that this is simply one of those.

Paul

From anthony.tuininga at gmail.com  Mon May  5 23:52:07 2014
From: anthony.tuininga at gmail.com (Anthony Tuininga)
Date: Mon, 5 May 2014 15:52:07 -0600
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
Message-ID: <CAE1XR-42=N_BOx_4354VNCEOtTfNZ1a2xrhoKLi9BHCCvWZ=5g@mail.gmail.com>

On Mon, May 5, 2014 at 3:50 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com> wrote:
> > So my question is: can I safely make use of this "feature"? It has
> remained
> > in place since at least Python 2.6 but I don't want to assume anything.
> > Please advise! Thanks.
>
> I believe this is a 100% supported feature of Python (since 2.6, as
> you mention). Some of the zip import/run features have unfortunately
> been underdocumented, and I suspect that this is simply one of those.
>
> Paul
>

That's great news. Thanks!

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/f4f554c7/attachment.html>

From greg at krypto.org  Tue May  6 00:10:22 2014
From: greg at krypto.org (Gregory P. Smith)
Date: Mon, 5 May 2014 15:10:22 -0700
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CAE1XR-42=N_BOx_4354VNCEOtTfNZ1a2xrhoKLi9BHCCvWZ=5g@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
 <CAE1XR-42=N_BOx_4354VNCEOtTfNZ1a2xrhoKLi9BHCCvWZ=5g@mail.gmail.com>
Message-ID: <CAGE7PNKYL4Y_AA_phYE=B8nku4AewLemeMCr0s4bBbbrDMOigQ@mail.gmail.com>

On Mon, May 5, 2014 at 2:52 PM, Anthony Tuininga <anthony.tuininga at gmail.com
> wrote:

> On Mon, May 5, 2014 at 3:50 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
>> On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com> wrote:
>> > So my question is: can I safely make use of this "feature"? It has
>> remained
>> > in place since at least Python 2.6 but I don't want to assume anything.
>> > Please advise! Thanks.
>>
>> I believe this is a 100% supported feature of Python (since 2.6, as
>> you mention). Some of the zip import/run features have unfortunately
>> been underdocumented, and I suspect that this is simply one of those.
>>
>> Paul
>>
>
> That's great news. Thanks!
>
> Anthony
>

There is at least one caveat to using a .zip file for the standard library:
http://bugs.python.org/issue19081

Never change the .zip file in use by a running process.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/b9c70a99/attachment.html>

From ncoghlan at gmail.com  Tue May  6 00:16:05 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 6 May 2014 08:16:05 +1000
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
Message-ID: <CADiSq7dC+n+oN4_s-uUpBxJ3fbkGxe2FRgsMwUxaUkh+WJtz1g@mail.gmail.com>

On 6 May 2014 07:51, "Paul Moore" <p.f.moore at gmail.com> wrote:
>
> On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com> wrote:
> > So my question is: can I safely make use of this "feature"? It has
remained
> > in place since at least Python 2.6 but I don't want to assume anything.
> > Please advise! Thanks.
>
> I believe this is a 100% supported feature of Python (since 2.6, as
> you mention). Some of the zip import/run features have unfortunately
> been underdocumented, and I suspect that this is simply one of those.

sys.path initialisation in general is woefully underspecified and
thoroughly confusing :(

Fixing that is actually one of the motivating forces behind the proposed
startup improvements in PEP 432.

Cheers,
Nick.

>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140506/06b746bd/attachment.html>

From anthony.tuininga at gmail.com  Tue May  6 00:14:51 2014
From: anthony.tuininga at gmail.com (Anthony Tuininga)
Date: Mon, 5 May 2014 16:14:51 -0600
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CAGE7PNKYL4Y_AA_phYE=B8nku4AewLemeMCr0s4bBbbrDMOigQ@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
 <CAE1XR-42=N_BOx_4354VNCEOtTfNZ1a2xrhoKLi9BHCCvWZ=5g@mail.gmail.com>
 <CAGE7PNKYL4Y_AA_phYE=B8nku4AewLemeMCr0s4bBbbrDMOigQ@mail.gmail.com>
Message-ID: <CAE1XR-5d0z5giqwRVaXJ2a8X7orvfRX+cvaWLNnKPLo5QEQNZg@mail.gmail.com>

Thanks. I think I can live with that restriction. :-) I do not read/write
to the same zip file in the same process.

Anthony


On Mon, May 5, 2014 at 4:10 PM, Gregory P. Smith <greg at krypto.org> wrote:

>
> On Mon, May 5, 2014 at 2:52 PM, Anthony Tuininga <
> anthony.tuininga at gmail.com> wrote:
>
>> On Mon, May 5, 2014 at 3:50 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>>> On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com>
>>> wrote:
>>> > So my question is: can I safely make use of this "feature"? It has
>>> remained
>>> > in place since at least Python 2.6 but I don't want to assume anything.
>>> > Please advise! Thanks.
>>>
>>> I believe this is a 100% supported feature of Python (since 2.6, as
>>> you mention). Some of the zip import/run features have unfortunately
>>> been underdocumented, and I suspect that this is simply one of those.
>>>
>>> Paul
>>>
>>
>> That's great news. Thanks!
>>
>> Anthony
>>
>
> There is at least one caveat to using a .zip file for the standard
> library: http://bugs.python.org/issue19081
>
> Never change the .zip file in use by a running process.
>
> -gps
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/c20f2aa2/attachment.html>

From anthony.tuininga at gmail.com  Tue May  6 00:37:31 2014
From: anthony.tuininga at gmail.com (Anthony Tuininga)
Date: Mon, 5 May 2014 16:37:31 -0600
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CADiSq7dC+n+oN4_s-uUpBxJ3fbkGxe2FRgsMwUxaUkh+WJtz1g@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <CACac1F_-mFEi=nB=-ifv-KeHLgVYiPCLDZ3hb2yZzbojbc43Dg@mail.gmail.com>
 <CADiSq7dC+n+oN4_s-uUpBxJ3fbkGxe2FRgsMwUxaUkh+WJtz1g@mail.gmail.com>
Message-ID: <CAE1XR-5wSKgE6tEkX4FMe=f80YddU04M6yvYfdsEOCW8yHSEeg@mail.gmail.com>

On Mon, May 5, 2014 at 4:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 6 May 2014 07:51, "Paul Moore" <p.f.moore at gmail.com> wrote:
> >
> > On 5 May 2014 22:32, Anthony Tuininga <anthony.tuininga at gmail.com>
> wrote:
> > > So my question is: can I safely make use of this "feature"? It has
> remained
> > > in place since at least Python 2.6 but I don't want to assume anything.
> > > Please advise! Thanks.
> >
> > I believe this is a 100% supported feature of Python (since 2.6, as
> > you mention). Some of the zip import/run features have unfortunately
> > been underdocumented, and I suspect that this is simply one of those.
>
> sys.path initialisation in general is woefully underspecified and
> thoroughly confusing :(
>
> Fixing that is actually one of the motivating forces behind the proposed
> startup improvements in PEP 432.
>
> Cheers,
> Nick.
>

Thanks for the link. I would tend to agree with your assessment. :-) I've
had to delve into the startup code a number of times to make cx_Freeze
behave itself so any help here would be much appreciated!

Anthony
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140505/4584acd2/attachment.html>

From james at 3dengineer.com  Tue May  6 16:33:41 2014
From: james at 3dengineer.com (James Swift)
Date: Tue, 6 May 2014 16:33:41 +0200
Subject: [Python-Dev] Behaviour change of object().format()
Message-ID: <CAMQS46r2yfDU7mDeVYOoL0YE0=fZ_Aeuk0-gKA2KJXa18=7M1Q@mail.gmail.com>

Hi,

In 3.3 I could do the following

>>> "{x:s}".format(**{'x': [1, 2, 3]})
'[1, 2, 3]'

But in 3.4

>>> "{x:s}".format(**{'x': [1, 2, 3]})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__


Is this intentional?

regards,
James Swift
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140506/d68a8110/attachment.html>

From james at 3dengineer.com  Tue May  6 16:45:52 2014
From: james at 3dengineer.com (James Swift)
Date: Tue, 6 May 2014 16:45:52 +0200
Subject: [Python-Dev] Behaviour change of object().format() in 3.4
Message-ID: <CAMQS46oUNmmBTpz+M5Zx8BC1mfEBHkimMeFhxiJVNJKkw4NdoQ@mail.gmail.com>

Hi,

In 3.3 I could do the following

>>> "{x:s}".format(**{'x': [1, 2, 3]})
'[1, 2, 3]'

But in 3.4

>>> "{x:s}".format(**{'x': [1, 2, 3]})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__


Is this intentional?

regards,
James Swift
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140506/c152883a/attachment.html>

From rdmurray at bitdance.com  Tue May  6 17:48:16 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 06 May 2014 11:48:16 -0400
Subject: [Python-Dev] Behaviour change of object().format() in 3.4
In-Reply-To: <CAMQS46oUNmmBTpz+M5Zx8BC1mfEBHkimMeFhxiJVNJKkw4NdoQ@mail.gmail.com>
References: <CAMQS46oUNmmBTpz+M5Zx8BC1mfEBHkimMeFhxiJVNJKkw4NdoQ@mail.gmail.com>
Message-ID: <20140506154816.C239C250D5C@webabinitio.net>

On Tue, 06 May 2014 16:45:52 +0200, James Swift <james at 3dengineer.com> wrote:
> Hi,
> 
> In 3.3 I could do the following
> 
> >>> "{x:s}".format(**{'x': [1, 2, 3]})
> '[1, 2, 3]'
> 
> But in 3.4
> 
> >>> "{x:s}".format(**{'x': [1, 2, 3]})
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: non-empty format string passed to object.__format__
> 
> 
> Is this intentional?

Yes.  There was a deprecation warning for this in 3.3, and it
is now an error in 3.4.

For more information, see http://bugs.python.org/issue7994.

--David

From eric at trueblade.com  Tue May  6 20:40:40 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Tue, 06 May 2014 14:40:40 -0400
Subject: [Python-Dev] Behaviour change of object().format()
In-Reply-To: <CAMQS46r2yfDU7mDeVYOoL0YE0=fZ_Aeuk0-gKA2KJXa18=7M1Q@mail.gmail.com>
References: <CAMQS46r2yfDU7mDeVYOoL0YE0=fZ_Aeuk0-gKA2KJXa18=7M1Q@mail.gmail.com>
Message-ID: <53692CA8.6060300@trueblade.com>

On 05/06/2014 10:33 AM, James Swift wrote:
> Hi,
> 
> In 3.3 I could do the following
> 
>>>> "{x:s}".format(**{'x': [1, 2, 3]})
> '[1, 2, 3]'
> 
> But in 3.4
> 
>>>> "{x:s}".format(**{'x': [1, 2, 3]})
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> TypeError: non-empty format string passed to object.__format__
> 
> 
> Is this intentional?

Yes. See http://bugs.python.org/issue7994 for the rationale.

You can use:
>>> "{x!s:15s}".format(**{'x': [1, 2, 3]})
'[1, 2, 3]      '

That is, convert it first to a string (with !s) then use whatever string
formatting specifier you want (here, '15s'). This should work under all
3.x versions (I think, haven't checked).

BTW, for this specific case, you might want to use format_map:
>>> "{x!s:15s}".format_map({'x': [1, 2, 3]})
'[1, 2, 3]      '

Eric.


From tjreedy at udel.edu  Tue May  6 19:14:01 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 06 May 2014 13:14:01 -0400
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
Message-ID: <lkb58t$ud5$1@ger.gmane.org>

On 5/5/2014 5:32 PM, Anthony Tuininga wrote:
> Hi,
>
> I am the author of cx_Freeze which creates "frozen" executables from
> Python scripts. To this point I have been using frozen modules (compiled
> C) but this has the side effect of bundling parts of Python with a
> cx_Freeze release -- and this has bitten me on a number of occasions
> when newer versions of Python use slightly incompatible modules. :-)
>
> By scanning the code I discovered that Python automatically searches on
> startup
>
> <PYTHONHOME>/lib/pythonNN.zip
>
> where NN is replaced by the major and minor version that is being
> executed. Using this would allow me to ensure that bootstrap modules are
> not bundled with cx_Freeze but acquired from the distribution that is
> actually using it -- thereby eliminating the hassle noted above.
>
> I have tested this approach and found it works flawlessly. There is,
> however, no documentation that I can find for the contents of sys.path
> -- the documentation simply says that it is an installation default and
> doesn't specify what that default is.
>
> So my question is: can I safely make use of this "feature"? It has
> remained in place since at least Python 2.6 but I don't want to assume
> anything. Please advise! Thanks.

I would say yes. AFAIK /lib has always by used for the stdlib on Windows 
and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of 
the site module doc. We do not lightly change directory structure. PEP 
273 then says "The zip archive must be treated exactly as a subdirectory 
tree,". I take this to mean that lib/pythonxy.zip should work the same 
as lib/pythonxy/ does.

You can depend on the feature working as long as it works. Your bigger 
worry is accidental regression. I suggest testing release candidates 
(there is one for 3.4.1 now, I believe) and also an early alpha and beta 
for new release. The candidate may have a zip patch affecting central 
directory reading. If not, there is an issue on the tracker. You could 
monitor patches by filtering the new-issue announcements for 'zip'.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue May  6 21:21:49 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 06 May 2014 15:21:49 -0400
Subject: [Python-Dev] Python under the sea and in space
In-Reply-To: <CAKDZRci2bdXSrRtn+UaqZ2i6vNt1s+ch_g__Ww13qRxWp6NkLg@mail.gmail.com>
References: <CAKDZRci2bdXSrRtn+UaqZ2i6vNt1s+ch_g__Ww13qRxWp6NkLg@mail.gmail.com>
Message-ID: <lkbcoh$q48$1@ger.gmane.org>

On 5/4/2014 11:02 PM, Jessica McKellar wrote:
> Hi folks,
>
> I'm trying to determine the greatest depth (in the ocean or underground)
> and highest altitude at which Python code has been executed.
>
> Please note that I'm interested in where the code was executed, and not,
> say, where data that Python analyzed was acquired. I know for example
> that NASA has analyzed plenty of data from space in Python, but the
> analysis was done in labs here on Earth. :)
>
> Do you have some good candidates? Please let me know!

I suggest posting this on python-list; its readers seem to work with a 
wide range of applications.

-- 
Terry Jan Reedy


From stefan at bytereef.org  Tue May  6 23:35:33 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Tue, 6 May 2014 23:35:33 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
Message-ID: <20140506213533.GA32765@sleipnir.bytereef.org>

Just a warning, in case any of the new packaging team forgot to contact
http://cve.mitre.org/ .


Stefan Krah




From victor.stinner at gmail.com  Wed May  7 00:18:24 2014
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 7 May 2014 00:18:24 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <20140506213533.GA32765@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
Message-ID: <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>

Hi,

I don't understand your email. Can you please elaborate?

Victor

2014-05-06 23:35 GMT+02:00 Stefan Krah <stefan at bytereef.org>:
> Just a warning, in case any of the new packaging team forgot to contact
> http://cve.mitre.org/ .
>
>
> Stefan Krah
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From eric at trueblade.com  Wed May  7 13:45:11 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 07 May 2014 07:45:11 -0400
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <lkb58t$ud5$1@ger.gmane.org>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <lkb58t$ud5$1@ger.gmane.org>
Message-ID: <536A1CC7.10908@trueblade.com>

On 5/6/2014 1:14 PM, Terry Reedy wrote:
> On 5/5/2014 5:32 PM, Anthony Tuininga wrote:
>> Hi,
>>
>> I am the author of cx_Freeze which creates "frozen" executables from
>> Python scripts. To this point I have been using frozen modules (compiled
>> C) but this has the side effect of bundling parts of Python with a
>> cx_Freeze release -- and this has bitten me on a number of occasions
>> when newer versions of Python use slightly incompatible modules. :-)
>>
>> By scanning the code I discovered that Python automatically searches on
>> startup
>>
>> <PYTHONHOME>/lib/pythonNN.zip
>>
>> where NN is replaced by the major and minor version that is being
>> executed. Using this would allow me to ensure that bootstrap modules are
>> not bundled with cx_Freeze but acquired from the distribution that is
>> actually using it -- thereby eliminating the hassle noted above.
>>
>> I have tested this approach and found it works flawlessly. There is,
>> however, no documentation that I can find for the contents of sys.path
>> -- the documentation simply says that it is an installation default and
>> doesn't specify what that default is.
>>
>> So my question is: can I safely make use of this "feature"? It has
>> remained in place since at least Python 2.6 but I don't want to assume
>> anything. Please advise! Thanks.
> 
> I would say yes. AFAIK /lib has always by used for the stdlib on Windows
> and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of
> the site module doc. We do not lightly change directory structure. PEP
> 273 then says "The zip archive must be treated exactly as a subdirectory
> tree,". I take this to mean that lib/pythonxy.zip should work the same
> as lib/pythonxy/ does.
> 
> You can depend on the feature working as long as it works. Your bigger
> worry is accidental regression. I suggest testing release candidates
> (there is one for 3.4.1 now, I believe) and also an early alpha and beta
> for new release. The candidate may have a zip patch affecting central
> directory reading. If not, there is an issue on the tracker. You could
> monitor patches by filtering the new-issue announcements for 'zip'.

The best course of action would be to integrate any such tests into the
stdlib test suite, where a regression would immediately be detected.

Eric.


From tjreedy at udel.edu  Wed May  7 19:13:43 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 07 May 2014 13:13:43 -0400
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <536A1CC7.10908@trueblade.com>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <lkb58t$ud5$1@ger.gmane.org> <536A1CC7.10908@trueblade.com>
Message-ID: <lkdpkc$ohi$1@ger.gmane.org>

On 5/7/2014 7:45 AM, Eric V. Smith wrote:
> On 5/6/2014 1:14 PM, Terry Reedy wrote:
>> On 5/5/2014 5:32 PM, Anthony Tuininga wrote:
>>> Hi,
>>>
>>> I am the author of cx_Freeze which creates "frozen" executables from
>>> Python scripts. To this point I have been using frozen modules (compiled
>>> C) but this has the side effect of bundling parts of Python with a
>>> cx_Freeze release -- and this has bitten me on a number of occasions
>>> when newer versions of Python use slightly incompatible modules. :-)
>>>
>>> By scanning the code I discovered that Python automatically searches on
>>> startup
>>>
>>> <PYTHONHOME>/lib/pythonNN.zip
>>>
>>> where NN is replaced by the major and minor version that is being
>>> executed. Using this would allow me to ensure that bootstrap modules are
>>> not bundled with cx_Freeze but acquired from the distribution that is
>>> actually using it -- thereby eliminating the hassle noted above.
>>>
>>> I have tested this approach and found it works flawlessly. There is,
>>> however, no documentation that I can find for the contents of sys.path
>>> -- the documentation simply says that it is an installation default and
>>> doesn't specify what that default is.
>>>
>>> So my question is: can I safely make use of this "feature"? It has
>>> remained in place since at least Python 2.6 but I don't want to assume
>>> anything. Please advise! Thanks.
>>
>> I would say yes. AFAIK /lib has always by used for the stdlib on Windows
>> and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of
>> the site module doc. We do not lightly change directory structure. PEP
>> 273 then says "The zip archive must be treated exactly as a subdirectory
>> tree,". I take this to mean that lib/pythonxy.zip should work the same
>> as lib/pythonxy/ does.
>>
>> You can depend on the feature working as long as it works. Your bigger
>> worry is accidental regression. I suggest testing release candidates
>> (there is one for 3.4.1 now, I believe) and also an early alpha and beta
>> for new release. The candidate may have a zip patch affecting central
>> directory reading. If not, there is an issue on the tracker. You could
>> monitor patches by filtering the new-issue announcements for 'zip'.
>
> The best course of action would be to integrate any such tests into the
> stdlib test suite, where a regression would immediately be detected.

We do not and should not put tests of 3rd party products that we do not 
distribute ourselves into the stdlib test suite. Anthony should test 
*his* pythonxy.dll himself. To the extent feasible, we should test the 
promised behavior of python.exe regardless of what any 3rd party 
developer does.

If pythonxy.zip supplements rather than replacing pythonxy/*.py, this 
should be easy enough: copy .zip into place, run python in a subprocess, 
check the result, delete .zip.


-- 
Terry Jan Reedy


From eric at trueblade.com  Wed May  7 20:02:19 2014
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 07 May 2014 14:02:19 -0400
Subject: [Python-Dev] Existence of pythonNN.zip in sys.path
In-Reply-To: <lkdpkc$ohi$1@ger.gmane.org>
References: <CAE1XR-7ZXgJam50Ne9v_ASzGEVGKQ8E=gQ4=LoghPr9o-dccTg@mail.gmail.com>
 <lkb58t$ud5$1@ger.gmane.org> <536A1CC7.10908@trueblade.com>
 <lkdpkc$ohi$1@ger.gmane.org>
Message-ID: <536A752B.8030506@trueblade.com>

On 05/07/2014 01:13 PM, Terry Reedy wrote:
> On 5/7/2014 7:45 AM, Eric V. Smith wrote:
>> On 5/6/2014 1:14 PM, Terry Reedy wrote:
>>> On 5/5/2014 5:32 PM, Anthony Tuininga wrote:
>>>> Hi,
>>>>
>>>> I am the author of cx_Freeze which creates "frozen" executables from
>>>> Python scripts. To this point I have been using frozen modules
>>>> (compiled
>>>> C) but this has the side effect of bundling parts of Python with a
>>>> cx_Freeze release -- and this has bitten me on a number of occasions
>>>> when newer versions of Python use slightly incompatible modules. :-)
>>>>
>>>> By scanning the code I discovered that Python automatically searches on
>>>> startup
>>>>
>>>> <PYTHONHOME>/lib/pythonNN.zip
>>>>
>>>> where NN is replaced by the major and minor version that is being
>>>> executed. Using this would allow me to ensure that bootstrap modules
>>>> are
>>>> not bundled with cx_Freeze but acquired from the distribution that is
>>>> actually using it -- thereby eliminating the hassle noted above.
>>>>
>>>> I have tested this approach and found it works flawlessly. There is,
>>>> however, no documentation that I can find for the contents of sys.path
>>>> -- the documentation simply says that it is an installation default and
>>>> doesn't specify what that default is.
>>>>
>>>> So my question is: can I safely make use of this "feature"? It has
>>>> remained in place since at least Python 2.6 but I don't want to assume
>>>> anything. Please advise! Thanks.
>>>
>>> I would say yes. AFAIK /lib has always by used for the stdlib on Windows
>>> and /lib/pythonxy on *nix. Both are mentioned in the 4th paragraph of
>>> the site module doc. We do not lightly change directory structure. PEP
>>> 273 then says "The zip archive must be treated exactly as a subdirectory
>>> tree,". I take this to mean that lib/pythonxy.zip should work the same
>>> as lib/pythonxy/ does.
>>>
>>> You can depend on the feature working as long as it works. Your bigger
>>> worry is accidental regression. I suggest testing release candidates
>>> (there is one for 3.4.1 now, I believe) and also an early alpha and beta
>>> for new release. The candidate may have a zip patch affecting central
>>> directory reading. If not, there is an issue on the tracker. You could
>>> monitor patches by filtering the new-issue announcements for 'zip'.
>>
>> The best course of action would be to integrate any such tests into the
>> stdlib test suite, where a regression would immediately be detected.
> 
> We do not and should not put tests of 3rd party products that we do not
> distribute ourselves into the stdlib test suite. Anthony should test
> *his* pythonxy.dll himself. To the extent feasible, we should test the
> promised behavior of python.exe regardless of what any 3rd party
> developer does.
> 
> If pythonxy.zip supplements rather than replacing pythonxy/*.py, this
> should be easy enough: copy .zip into place, run python in a subprocess,
> check the result, delete .zip.

I'm sorry, I misread this. I thought the question was about putting the
stdlib itself in pythonxy.zip. I agree we shouldn't test someone else
using that filename.

Eric.



From stefan at bytereef.org  Thu May  8 14:12:02 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 8 May 2014 14:12:02 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
Message-ID: <20140508121201.GA28465@sleipnir.bytereef.org>

Victor Stinner <victor.stinner at gmail.com> wrote:
> I don't understand your email. Can you please elaborate?

There is nothing wrong with the package.  The remark is a joke provoked by
a long history of a campaign [1] against external packages on distutils-sig.

Many tools (like crate.io, when it was still up) have made derogatory
remarks about external packages.  Now the latest version of the officially
sanctioned download tool (pip) spits out copious warnings, one of which
is the subject of this thread.


External packages are being singled out unfairly:

  1) Anyone can upload any package to PyPI (i.e. the index is not curated
     at all).

  2) Last time I looked, access credentials (via "lost login") were sent
     out in plaintext.

  3) AFAIK people can upload a different (malicious) version of a
     package with *the exact same name*.

  4) pip generally downloads the latest version, so a malicious person
     can provide a good package for several years until people trust
     him, then change to a trojaned version.

  5) Looking at the list of certificates that is in my default cert
     store, I don't find SSL trustworthy at all.

  6) D.J. Bernstein, who is somewhat security minded, has been shipping
     his software *for years* with just plain HTTP and published checksums.


To sum it up:

  1) Don't use pip to install packages directly from PyPI if security
     really matters.

  2) The best security we currently get is either

      a) with package signatures (*if* you can get the author's key via
         a trustworthy channel, which is rarely the case).

      b) with decent checksums that are recorded on public mailing
         lists at the time the package is announced (it would be
         hard for an attacker to modify all mailing list archives.)

     Whether a package is internal or external is orthogonal to both points.


With all these points, I find it questionable for an "official" install
tool to make security related remarks about just one category of weaknesses.

After all, people might be led to believe that pip is some sort of apt-get
and all uploaded packages are safe.


Stefan Krah



[1] Note that the joke is quite innocent in comparison to what I've read on
    distutils-sig about the subject.




From donald at stufft.io  Thu May  8 14:52:36 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 08:52:36 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508121201.GA28465@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
Message-ID: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>


On May 8, 2014, at 8:12 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Victor Stinner <victor.stinner at gmail.com> wrote:
>> I don't understand your email. Can you please elaborate?
> 
> There is nothing wrong with the package.  The remark is a joke provoked by
> a long history of a campaign [1] against external packages on distutils-sig.
> 
> Many tools (like crate.io, when it was still up) have made derogatory
> remarks about external packages.  Now the latest version of the officially
> sanctioned download tool (pip) spits out copious warnings, one of which
> is the subject of this thread.

pip doesn't install externally hosted packaged by default because people should
be aware of what servers they depend on. Installs that depend on externally
hosted packages are brittle and more prone to failure. Every single external
server adds *another* SPOF into any particular install set. Even if every
external server has a 99.9% uptime, when you combine multiple of them the total
uptime of any particular set of requirements drops quickly. This has been a
major complaint that people have had over time.  However they are not a
security issue so they are easy to turn back on globally if a person wishes to
make their own deployment more brittle (--allow-all-external). 

However in addition to externally hosted packages, there are also unverifiable
packages. These are packages where there is no path to verify the download.
Generally an unverifiable package is one that either is linked to directly but
has no hash information encoded in the URL in a way that pip understands or,
the more common case, it is a package that is linked from a page that is linked
too on PyPI. Since we cannot build a path of trust to verify the package it is
trivial for an attacker to MITM it and execute arbitrary Python code on the
end users machine. This case *is* a security issue and pip goes out of it's way
to *greatly* discourage depending on packages that require this kind of
install.

> 
> 
> External packages are being singled out unfairly:
> 
>  1) Anyone can upload any package to PyPI (i.e. the index is not curated
>     at all).

Sure, the threat model of PyPI isn't that you can install anything willy nilly
and be safe. It's that when you ask for package X version Y, you should get
what the *author* published and not what attacker who happens to be in a
position to MITM you sends you.

> 
>  2) Last time I looked, access credentials (via "lost login") were sent
>     out in plaintext.

The existence of other security issues is not an excuse to not fix a security
issue. There are other problems and we're slowly working on trying to clear
them out.

> 
>  3) AFAIK people can upload a different (malicious) version of a
>     package with *the exact same name*.

Yes, a malicious author is needfully outside of the threat model for PyPI/pip.

> 
>  4) pip generally downloads the latest version, so a malicious person
>     can provide a good package for several years until people trust
>     him, then change to a trojaned version.

Yes, a malicious author is needfully outside of the threat model for PyPI/pip.

> 
>  5) Looking at the list of certificates that is in my default cert
>     store, I don't find SSL trustworthy at all.

TLS may not be the best method of authenticating package downloads, however it
is an easy one and it is significantly better than having no authentication at
all. You move your attack surface from "anyone who has a privileged network
position" to "anyone who has a privileged network position and also has the
ability to generate erroneously signed CA certificates". That is a
significantly smaller attack surface.

> 
>  6) D.J. Bernstein, who is somewhat security minded, has been shipping
>     his software *for years* with just plain HTTP and published checksums.

Argument from authority doesn't really hold up very well when DJB doesn't
distribute is software in a way that is intended to be consumed mechanically.
Also while he may be a crypto expert he is far from an expert on successfully
distributing software, unless you also think that the signature checking in
most OS provided package managers is pointless.

> 
> 
> To sum it up:
> 
>  1) Don't use pip to install packages directly from PyPI if security
>     really matters.

?security? isn?t a useful term on it?s own. You have to define a threat
model. A better answer here is don?t use pip to install packages directly
from PyPI if a malicious author is within your threat model (modulo other
issue we?re still working on, but as I said above, the existence of a second
security issue isn?t a reason not to fix the first issue).

> 
>  2) The best security we currently get is either
> 
>      a) with package signatures (*if* you can get the author's key via
>         a trustworthy channel, which is rarely the case).
> 
>      b) with decent checksums that are recorded on public mailing
>         lists at the time the package is announced (it would be
>         hard for an attacker to modify all mailing list archives.)
> 
>     Whether a package is internal or external is orthogonal to both points.

Again as I said above, the internal vs external isn?t a security related decision
it is a ?uptime? related decision. Verifiable vs unverifiable *is* a security
related decision.

> 
> 
> With all these points, I find it questionable for an "official" install
> tool to make security related remarks about just one category of weaknesses.
> 
> After all, people might be led to believe that pip is some sort of apt-get
> and all uploaded packages are safe.
> 
> 
> Stefan Krah
> 
> 
> 
> [1] Note that the joke is quite innocent in comparison to what I've read on
>    distutils-sig about the subject.
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/fa88483c/attachment.sig>

From mal at egenix.com  Thu May  8 15:39:53 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 08 May 2014 15:39:53 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
Message-ID: <536B8929.6040407@egenix.com>

Well, to be fair and leaving aside uptime concerns and the general
desire to always install packages from some server instead of
a safe and trusted local directory (probably too obvious ;-),
it would certainly be possible to add support for
trusted externally hosted packages.

However, for some reason there's a strong resistance against
doing this, which I frankly don't understand.

I agree with Stefan that the warning message wording is less
than ideal. You'd normally call such blanket statements FUD,
esp. since there are plenty external hosting services which
are reliable and safe to use.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From rosuav at gmail.com  Thu May  8 15:55:05 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 8 May 2014 23:55:05 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <536B8929.6040407@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
Message-ID: <CAPTjJmqat7S8Fm0gHkSDfPM4fr6WUQXHLZYJD+ntEE1c+MQ7TQ@mail.gmail.com>

On Thu, May 8, 2014 at 11:39 PM, M.-A. Lemburg <mal at egenix.com> wrote:
> I agree with Stefan that the warning message wording is less
> than ideal. You'd normally call such blanket statements FUD,
> esp. since there are plenty external hosting services which
> are reliable and safe to use.

No, it's not FUD. Every external dependency adds another thing that
can fail. I was just arguing this with one of my younger brothers this
evening; he had some stuff stored in Google Docs, and he couldn't
access it because of some outage. With a local git/hg repository, he
would have had all his data right there, and even losing access to a
central hub server just means you can't pull/push. Using someone
else's server makes you depend on that server and everything in
between.

Obviously that's often a cost worth paying (hey, I'm posting this from
Gmail, so I completely lose access to all these emails if it's
inaccessible), but it's a legitimate concern, not FUD.

(That doesn't stop the wording from being "less than ideal", though.)

ChrisA

From ncoghlan at gmail.com  Thu May  8 15:57:26 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 8 May 2014 23:57:26 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <536B8929.6040407@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
Message-ID: <CADiSq7fKw5L9u_gs56cMSk_yLUAoayF9HihsfWUSgPJ_Gf9+Ag@mail.gmail.com>

On 8 May 2014 23:39, M.-A. Lemburg <mal at egenix.com> wrote:
> However, for some reason there's a strong resistance against
> doing this, which I frankly don't understand.

Because we're taking responsibility for the end-to-end user experience
of PyPI, and are expressly trying to eliminate the elements of that
user experience that are beyond the control of the PyPI admin team
(even the question of "does this software actually work?" is in our
sights if you consider a long enough time span). That's hard enough
with just a couple of service providers (Fastly and Rackspace) in the
mix - it quickly becomes impossible if every new dependency from an
installation request adds a new point of failure.

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Thu May  8 15:58:08 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 09:58:08 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <536B8929.6040407@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
Message-ID: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>


On May 8, 2014, at 9:39 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> Well, to be fair and leaving aside uptime concerns and the general
> desire to always install packages from some server instead of
> a safe and trusted local directory (probably too obvious ;-),
> it would certainly be possible to add support for
> trusted externally hosted packages.

There is support for trusted externally hosted packages, you put the URL in
PyPI and include a hash in the fragment like so:

http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56

The hash can be md5 or any of the sha-2 family.

Now this does not mean that ``pip install cdecimal`` will automatically install
this, because whether or not you're willing to install from servers other than
PyPI[1] is a policy decision for the end user of pip. The only real contention
point there is whether installing from other servers should be on or off by
default. PEP438 selected off by default, and I agree with that decision.
Installing externally hosted files, which are able to be safely downloaded[2],
was a surprising behavior to *everyone* I've talked to who hadn't already
discovered that pip/easy_install did that. For the people it wasn't surprising
too, they said it was surprising when they had originally discovered it[3].

[1] To be specific, other than the configured index(es), which happens to
    default to PyPI.
[2] For the definition of safe that PyPI/pip operate under, which is that the
    author of a package is assumed to be trusted by the person electing to
    download their package.
[3] I suspect people who were around when PyPI *couldn't* host files and were
    only an index would be the exception to this.

> 
> However, for some reason there's a strong resistance against
> doing this, which I frankly don't understand.
> 
> I agree with Stefan that the warning message wording is less
> than ideal. You'd normally call such blanket statements FUD,
> esp. since there are plenty external hosting services which
> are reliable and safe to use.
> 

I don't think the warning is FUD, and it doesn't mention anything security
related at all. The exact text of the warning is in the subject of the email
here:

    cdecimal an externally hosted file and may be unreliable

Which is true as far as I can tell, it is externally hosted, and it may be
unreliable[1]. If there is a better wording for that I?m happy to have it and
will gladly commit it myself to pip.

[1] In my experience dealing with complaints of pip's users, one of their big
    ones was that some dependency they use was, typically unknown to them,
    hosted externally and they found out it was hosted externally because the
    server it was hosted on went down.



-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/199c08d1/attachment.sig>

From donald at stufft.io  Thu May  8 16:10:35 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 10:10:35 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
Message-ID: <F1D5DA9A-B144-409C-98F0-E3A2E82B8540@stufft.io>


On May 8, 2014, at 9:58 AM, Donald Stufft <donald at stufft.io> wrote:

> Now this does not mean that ``pip install cdecimal`` will automatically install
> this, because whether or not you're willing to install from servers other than
> PyPI[1] is a policy decision for the end user of pip.

I forgot to add, for externally hosted files that are able to be safely
downloaded pip's users can either elect to use any/all of them via:

   $ pip install --allow-all-external cdecimal
   $ PIP_ALLOW_ALL_EXTERNAL=1 pip install cdecimal
   $ echo "--allow-all-external\ncdecimal" > requirements.txt
   $ pip install -r requirements.txt
   $ echo "[global]\nallow-allow-external=true" > ~/.pip/pip.conf
   $ pip install cdecimal

They can also elect to allow externally hosted files for *only* cdecimal using:

    $ pip install --allow-external cdecimal decimal
    $ PIP_ALLOW_EXTERNAL=cdecimal pip install cdecimal
    $ echo "--allow-external decimal\ncdecimal" > requirements.txt
    $ pip install -r requirements.txt
    $ echo "[global]\nallow-external=decimal" > ~/.pip/pip.conf
    $ pip install cdecimal

I may have the syntax on the non command flag options slightly wrong, those
are auto generated for us in our option system and I don't personally use those
options.

Also the reason for the extra verbosity in ``--allow-external`` is so you
can allow it on something you don't directly depend on, like:

    $ pip install --allow-external cdecimal foobar-which-depends-on-cdecimal

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/4b472931/attachment.sig>

From rdmurray at bitdance.com  Thu May  8 16:11:39 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 08 May 2014 10:11:39 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
Message-ID: <20140508141140.4AEB1250DBB@webabinitio.net>

On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft <donald at stufft.io> wrote:
> I don't think the warning is FUD, and it doesn't mention anything security
> related at all. The exact text of the warning is in the subject of the email
> here:
> 
>     cdecimal an externally hosted file and may be unreliable
> 
> Which is true as far as I can tell, it is externally hosted, and it may be
> unreliable[1]. If there is a better wording for that I???m happy to have it and
> will gladly commit it myself to pip.
> 
> [1] In my experience dealing with complaints of pip's users, one of their big
>     ones was that some dependency they use was, typically unknown to them,
>     hosted externally and they found out it was hosted externally because the
>     server it was hosted on went down.

"unreliable" reads as "not safe", ie: insecure.

You probably want something like "and access to it may be unreliable".

--David

From donald at stufft.io  Thu May  8 16:21:28 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 10:21:28 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508141140.4AEB1250DBB@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
Message-ID: <54BDABBF-D8D9-4498-86E6-56F7F86C0A57@stufft.io>


On May 8, 2014, at 10:11 AM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft <donald at stufft.io> wrote:
>> I don't think the warning is FUD, and it doesn't mention anything security
>> related at all. The exact text of the warning is in the subject of the email
>> here:
>> 
>>    cdecimal an externally hosted file and may be unreliable
>> 
>> Which is true as far as I can tell, it is externally hosted, and it may be
>> unreliable[1]. If there is a better wording for that I?m happy to have it and
>> will gladly commit it myself to pip.
>> 
>> [1] In my experience dealing with complaints of pip's users, one of their big
>>    ones was that some dependency they use was, typically unknown to them,
>>    hosted externally and they found out it was hosted externally because the
>>    server it was hosted on went down.
> 
> "unreliable" reads as "not safe", ie: insecure.
> 
> You probably want something like "and access to it may be unreliable".
> 
> --David

Done: https://github.com/pypa/pip/commit/69bf7067

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/25ff9c4f/attachment-0001.sig>

From rdmurray at bitdance.com  Thu May  8 16:21:34 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 08 May 2014 10:21:34 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508141140.4AEB1250DBB@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
Message-ID: <20140508142135.09EE0250DBC@webabinitio.net>

On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" <rdmurray at bitdance.com> wrote:
> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft <donald at stufft.io> wrote:
> > I don't think the warning is FUD, and it doesn't mention anything security
> > related at all. The exact text of the warning is in the subject of the email
> > here:
> > 
> >     cdecimal an externally hosted file and may be unreliable
> > 
> > Which is true as far as I can tell, it is externally hosted, and it may be
> > unreliable[1]. If there is a better wording for that I??????m happy to have it and
> > will gladly commit it myself to pip.
> > 
> > [1] In my experience dealing with complaints of pip's users, one of their big
> >     ones was that some dependency they use was, typically unknown to them,
> >     hosted externally and they found out it was hosted externally because the
> >     server it was hosted on went down.
> 
> "unreliable" reads as "not safe", ie: insecure.
> 
> You probably want something like "and access to it may be unreliable".

Actually, thinking about this some more, *most* end-users aren't going
to care that there's another point of failure here, they only care if it
works or not when they try to install it.  So something like
"cdecimal is not hosted on pypi; download may fail if external server
is unavailable" might be clearer.

And once you're at that point, as a user I'm going to grumble, "Well, why
the heck didn't you just try?", as I figure out how to re-execute the
command so that it does try.

--David

From stephane at wirtel.be  Thu May  8 16:17:46 2014
From: stephane at wirtel.be (=?utf-8?q?St=C3=A9phane?= Wirtel)
Date: Thu, 08 May 2014 16:17:46 +0200
Subject: [Python-Dev] EuroPython CPython Sprint?
Message-ID: <EF8B6398-8670-4205-996C-5C08DC8DE5C7@wirtel.be>

Hi all,

What do you think about a CPython sprint at EuroPython 2014?

Regards,

Stephane
--
St?phane Wirtel - http://wirtel.be - @matrixise

From solipsis at pitrou.net  Thu May  8 16:31:52 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 8 May 2014 16:31:52 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
Message-ID: <20140508163152.4b554316@fsol>

On Thu, 08 May 2014 10:21:34 -0400
"R. David Murray" <rdmurray at bitdance.com> wrote:
> > 
> > "unreliable" reads as "not safe", ie: insecure.
> > 
> > You probably want something like "and access to it may be unreliable".
> 
> Actually, thinking about this some more, *most* end-users aren't going
> to care that there's another point of failure here, they only care if it
> works or not when they try to install it.  So something like
> "cdecimal is not hosted on pypi; download may fail if external server
> is unavailable" might be clearer.
> 
> And once you're at that point, as a user I'm going to grumble, "Well, why
> the heck didn't you just try?", as I figure out how to re-execute the
> command so that it does try.

Agreed. That warning looks rather pointless and only aimed at trying to
enforce the pip developers' ideological preferences.

Regards

Antoine.



From bcannon at gmail.com  Thu May  8 16:33:57 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Thu, 08 May 2014 14:33:57 +0000
Subject: [Python-Dev]  EuroPython CPython Sprint?
References: <EF8B6398-8670-4205-996C-5C08DC8DE5C7@wirtel.be>
Message-ID: <CAP1=2W6qfgssmKe+Za+hXhqfKjt3T4K3mv=pJVyC4d9q+DLuEA@mail.gmail.com>

On Thu May 08 2014 at 10:25:44 AM, St?phane Wirtel <stephane at wirtel.be>
wrote:

> Hi all,
>
> What do you think about a CPython sprint at EuroPython 2014?
>

Great, although I think that answer would be considered obvious since there
is no real negative to holding sprints. =) Are you indirectly asking if
anyone plans to lead such a sprint?

-Brett


>
> Regards,
>
> Stephane
> --
> St?phane Wirtel - http://wirtel.be - @matrixise
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/925666d9/attachment.html>

From stefan at bytereef.org  Thu May  8 16:36:50 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 8 May 2014 16:36:50 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
Message-ID: <20140508143650.GA29239@sleipnir.bytereef.org>

Donald Stufft <donald at stufft.io> wrote:
> There is support for trusted externally hosted packages, you put the URL in
> PyPI and include a hash in the fragment like so:
> 
> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56

That is exactly the mode I was using until today.  This mode produced the
subject's warning message.

Today I've switched to manual install mode with manual sha256sum verification
which is *far* safer than anything you get via pip right now.


> [2] For the definition of safe that PyPI/pip operate under, which is that the
>     author of a package is assumed to be trusted by the person electing to
>     download their package.

No, there are other holes, which you have conceded in your previous mail.


> I don't think the warning is FUD, and it doesn't mention anything security
> related at all. The exact text of the warning is in the subject of the email
> here:
> 
>     cdecimal an externally hosted file and may be unreliable
> 
> Which is true as far as I can tell, it is externally hosted, and it may be
> unreliable[1]. If there is a better wording for that I?m happy to have it and
> will gladly commit it myself to pip.

Do you honestly not see a difference between the cited warning and the
*intended* warning "the server's availability may be unreliable"?

Even the latter is FUD or a truism (it applies to any server).

The real question is:  Why is there a warning if the person running pip
has explicitly allowed external packages?


Stefan Krah



From donald at stufft.io  Thu May  8 16:37:15 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 10:37:15 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508142135.09EE0250DBC@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
Message-ID: <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>


On May 8, 2014, at 10:21 AM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Thu, 08 May 2014 10:11:39 -0400, "R. David Murray" <rdmurray at bitdance.com> wrote:
>> On Thu, 08 May 2014 09:58:08 -0400, Donald Stufft <donald at stufft.io> wrote:
>>> I don't think the warning is FUD, and it doesn't mention anything security
>>> related at all. The exact text of the warning is in the subject of the email
>>> here:
>>> 
>>>    cdecimal an externally hosted file and may be unreliable
>>> 
>>> Which is true as far as I can tell, it is externally hosted, and it may be
>>> unreliable[1]. If there is a better wording for that I???m happy to have it and
>>> will gladly commit it myself to pip.
>>> 
>>> [1] In my experience dealing with complaints of pip's users, one of their big
>>>    ones was that some dependency they use was, typically unknown to them,
>>>    hosted externally and they found out it was hosted externally because the
>>>    server it was hosted on went down.
>> 
>> "unreliable" reads as "not safe", ie: insecure.
>> 
>> You probably want something like "and access to it may be unreliable".
> 
> Actually, thinking about this some more, *most* end-users aren't going
> to care that there's another point of failure here, they only care if it
> works or not when they try to install it.  So something like
> "cdecimal is not hosted on pypi; download may fail if external server
> is unavailable" might be clearer.
> 
> And once you're at that point, as a user I'm going to grumble, "Well, why
> the heck didn't you just try?", as I figure out how to re-execute the
> command so that it does try.
> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Most users are not going to care up until the point where the external server
is unavailable, and then they care a whole lot. On the tin it sounds reasonable
to just download the external file if the server is up however we've done
that for a long time and the end result has been end user pain.

Now requiring someone to add a flag in order to download an externally hosted
file is also end user pain. The difference between the two pains is when they
happen. The requiring a flag pain happens at the point of decision, when you
decide to make your deployment depend on something hosted externally. The 
default to allow pain happens sometime in the future, when you may or may not
have any idea why suddenly your installs aren't working (and when you look,
PyPI is up so you're really very confused why this particular file doesn't
work). Even worse is the case when a project has some old files, but the newer
versions aren't hosted and suddenly people are getting very old releases which
is even more confusing to the end users.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/c846cb77/attachment.sig>

From donald at stufft.io  Thu May  8 16:38:38 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 10:38:38 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508163152.4b554316@fsol>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net> <20140508163152.4b554316@fsol>
Message-ID: <0ACCF3E1-E5CE-438C-8279-2B1C1FABE1DF@stufft.io>


On May 8, 2014, at 10:31 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Thu, 08 May 2014 10:21:34 -0400
> "R. David Murray" <rdmurray at bitdance.com> wrote:
>>> 
>>> "unreliable" reads as "not safe", ie: insecure.
>>> 
>>> You probably want something like "and access to it may be unreliable".
>> 
>> Actually, thinking about this some more, *most* end-users aren't going
>> to care that there's another point of failure here, they only care if it
>> works or not when they try to install it.  So something like
>> "cdecimal is not hosted on pypi; download may fail if external server
>> is unavailable" might be clearer.
>> 
>> And once you're at that point, as a user I'm going to grumble, "Well, why
>> the heck didn't you just try?", as I figure out how to re-execute the
>> command so that it does try.
> 
> Agreed. That warning looks rather pointless and only aimed at trying to
> enforce the pip developers' ideological preferences.
> 
> Regards
> 
> Antoine.
> 

The pip developers didn?t make this decision. It was discussed on distutils-sig
hammered out in a PEP, and then accepted. We took part in that discussion,
but ultimately we implemented PEP438.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/af48eddb/attachment.sig>

From mal at egenix.com  Thu May  8 16:42:42 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 08 May 2014 16:42:42 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
Message-ID: <536B97E2.8070805@egenix.com>

On 08.05.2014 15:58, Donald Stufft wrote:
> 
> On May 8, 2014, at 9:39 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> Well, to be fair and leaving aside uptime concerns and the general
>> desire to always install packages from some server instead of
>> a safe and trusted local directory (probably too obvious ;-),
>> it would certainly be possible to add support for
>> trusted externally hosted packages.
> 
> There is support for trusted externally hosted packages, you put the URL in
> PyPI and include a hash in the fragment like so:
> 
> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56
> 
> The hash can be md5 or any of the sha-2 family.
> 
> Now this does not mean that ``pip install cdecimal`` will automatically install
> this, because whether or not you're willing to install from servers other than
> PyPI[1] is a policy decision for the end user of pip. 

Hmm, if you call that feature "trusted externally hosted packages",
pip should really do trust them, right ? ;-)

I can understand that pip defaults to not trusting URLs which don't
meet the above feature requirements, but not that it still warns
about unreliable externally hosted packages even if the above
feature is used.

At the moment, pip will refuse to use an externally hosted files even
if the package author uses the above hashed URLs; even with HTTPS
and proper SSL certificate chain.

> The only real contention
> point there is whether installing from other servers should be on or off by
> default. PEP438 selected off by default, and I agree with that decision.
> Installing externally hosted files, which are able to be safely downloaded[2],
> was a surprising behavior to *everyone* I've talked to who hadn't already
> discovered that pip/easy_install did that. For the people it wasn't surprising
> too, they said it was surprising when they had originally discovered it[3].
> 
> [1] To be specific, other than the configured index(es), which happens to
>     default to PyPI.
> [2] For the definition of safe that PyPI/pip operate under, which is that the
>     author of a package is assumed to be trusted by the person electing to
>     download their package.
> [3] I suspect people who were around when PyPI *couldn't* host files and were
>     only an index would be the exception to this.
> 
>>
>> However, for some reason there's a strong resistance against
>> doing this, which I frankly don't understand.
>>
>> I agree with Stefan that the warning message wording is less
>> than ideal. You'd normally call such blanket statements FUD,
>> esp. since there are plenty external hosting services which
>> are reliable and safe to use.
>>
> 
> I don't think the warning is FUD, and it doesn't mention anything security
> related at all. The exact text of the warning is in the subject of the email
> here:
> 
>     cdecimal an externally hosted file and may be unreliable
> 
> Which is true as far as I can tell, it is externally hosted, and it may be
> unreliable[1]. If there is a better wording for that I?m happy to have it and
> will gladly commit it myself to pip.

The current version of pip writes:

Downloading/unpacking pkg
  Could not find any downloads that satisfy the requirement pkg
  Some externally hosted files were ignored (use --allow-external pkg to allow).
Cleaning up...
No distributions at all found for pkg

This wording if fine, IMO. The wording Stefan quoted gets generated
for dependencies. This should probably be changed to the same wording
(including the reference to the right command line option to use).

> [1] In my experience dealing with complaints of pip's users, one of their big
>     ones was that some dependency they use was, typically unknown to them,
>     hosted externally and they found out it was hosted externally because the
>     server it was hosted on went down.

I think that's a general problem, not one of some server being down:
users put too much trust into the dependencies of packages they use.

Regardless of how safe/reliable we make things w/r to file hosting,
this problem does not go away. It's just too easy for people to
get tricked into trusting packages they don't even know about.

Nothing we'll ever change, though. People are lazy and easily
drop all such concerns for ease of use :-(

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From mal at egenix.com  Thu May  8 16:52:46 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 08 May 2014 16:52:46 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <CADiSq7fKw5L9u_gs56cMSk_yLUAoayF9HihsfWUSgPJ_Gf9+Ag@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>
 <CADiSq7fKw5L9u_gs56cMSk_yLUAoayF9HihsfWUSgPJ_Gf9+Ag@mail.gmail.com>
Message-ID: <536B9A3E.9030905@egenix.com>

On 08.05.2014 15:57, Nick Coghlan wrote:
> On 8 May 2014 23:39, M.-A. Lemburg <mal at egenix.com> wrote:
>> However, for some reason there's a strong resistance against
>> doing this, which I frankly don't understand.
> 
> Because we're taking responsibility for the end-to-end user experience
> of PyPI, and are expressly trying to eliminate the elements of that
> user experience that are beyond the control of the PyPI admin team

Oh, I guess you'd have to rewrite most of those 40k packages then :-)

Seriously, the word "eliminate" in there does not sit well with
our goals for openness. External services like github, sourceforge,
bitbucket, dropbox, cdns, etc. are not per-se evil and unreliable.

pip should acknowledge this and not try to "eliminate" all hosting
services in the world per default [sound of Empire Strikes
Back theme] ;-)

> (even the question of "does this software actually work?" is in our
> sights if you consider a long enough time span). That's hard enough
> with just a couple of service providers (Fastly and Rackspace) in the
> mix - it quickly becomes impossible if every new dependency from an
> installation request adds a new point of failure.

Like I said: the best option is to use a local directory which
only contains packages files that you have inspected and
actually trust :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From donald at stufft.io  Thu May  8 16:57:18 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 10:57:18 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508143650.GA29239@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
Message-ID: <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>


On May 8, 2014, at 10:36 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald Stufft <donald at stufft.io> wrote:
>> There is support for trusted externally hosted packages, you put the URL in
>> PyPI and include a hash in the fragment like so:
>> 
>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56
> 
> That is exactly the mode I was using until today.  This mode produced the
> subject's warning message.
> 
> Today I've switched to manual install mode with manual sha256sum verification
> which is *far* safer than anything you get via pip right now.

It is not safer in any meaingful way.

If someone is in a position to compromise the integrity of PyPI's TLS, they
can replace the hash on that page with something else. Now you've attempted to
work around this by telling people to go look up the release announcement
hash. However if someone can compromise the integrity of PyPI's TLS, they can
also compromise the integrity of https://mail.python.org/, or GMane, or any
other TLS based website[1].

All of that assumes that the end user is going to bother to verify the hash
*at all* which almost none of them will and they'll just check the http url
into their requirements.txt file and be downloading things over HTTP and
be vulnerable to arbitrary code execution via MITM.

> 
> 
>> [2] For the definition of safe that PyPI/pip operate under, which is that the
>>    author of a package is assumed to be trusted by the person electing to
>>    download their package.
> 
> No, there are other holes, which you have conceded in your previous mail.

The presence of other holes is not a useful argument to avoid closing a hole.
We're working to close all of them, and that ends up meaning we close one at
a time.

> 
> 
>> I don't think the warning is FUD, and it doesn't mention anything security
>> related at all. The exact text of the warning is in the subject of the email
>> here:
>> 
>>    cdecimal an externally hosted file and may be unreliable
>> 
>> Which is true as far as I can tell, it is externally hosted, and it may be
>> unreliable[1]. If there is a better wording for that I?m happy to have it and
>> will gladly commit it myself to pip.
> 
> Do you honestly not see a difference between the cited warning and the
> *intended* warning "the server's availability may be unreliable??

Do I? No I don?t. However I?ve since adjusted the message based on
R David Murray?s feedback to make sure it specifically says that access
may be unreliable instead of just that the package itself may be unreliable.

> 
> Even the latter is FUD or a truism (it applies to any server).

No, because the use of an external host *adds* additional unreliability. If
PyPI is down, then all packages are down, including ones that host externally.
If the cdecimal server is down, then that one specific package is unavailable.

It is impossible to do anything but reduce the overall availability by adding
additional SPOFs.

> 
> The real question is:  Why is there a warning if the person running pip
> has explicitly allowed external packages?
> 

Why is there a warning? Originally that warning was there because the first
part of the transition to this "mode" of defaults was to give an option to
disable externally hosted files, but leave it on by default. In this phase
we gave this warning to tell the people who just leave things to their default
about this file.

Should the warning itself still exist? I don't know, but a better avenue for
asking for a change in pip is via our issue tracker instead of whining on
python-dev.

> Stefan Krah
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/2af61db6/attachment-0001.sig>

From stephane at wirtel.be  Thu May  8 16:59:44 2014
From: stephane at wirtel.be (=?utf-8?q?St=C3=A9phane?= Wirtel)
Date: Thu, 08 May 2014 16:59:44 +0200
Subject: [Python-Dev] EuroPython CPython Sprint?
In-Reply-To: <CAP1=2W6qfgssmKe+Za+hXhqfKjt3T4K3mv=pJVyC4d9q+DLuEA@mail.gmail.com>
References: <EF8B6398-8670-4205-996C-5C08DC8DE5C7@wirtel.be>
 <CAP1=2W6qfgssmKe+Za+hXhqfKjt3T4K3mv=pJVyC4d9q+DLuEA@mail.gmail.com>
Message-ID: <0B3881FC-46E7-40BB-875F-79B1AD7225F4@wirtel.be>

On 8 May 2014, at 16:33, Brett Cannon wrote:

> On Thu May 08 2014 at 10:25:44 AM, St?phane Wirtel <stephane at wirtel.be>
> wrote:
>
>> Hi all,
>>
>> What do you think about a CPython sprint at EuroPython 2014?
>>
>
> Great, although I think that answer would be considered obvious since there
> is no real negative to holding sprints. =) Are you indirectly asking if
> anyone plans to lead such a sprint?
Yes, I do,

I can make the request to the EuroPython team for the sprint but:
* I am not a Python Core Dev
* I can't find the right issues for the sprinters
* I can't help the sprinters with the issues.
* I am a newbie in the CPython code 

Thus, who would be interested in a CPython sprint?
>
> -Brett

St?phane,
--
St?phane Wirtel - http://wirtel.be - @matrixise

From ncoghlan at gmail.com  Thu May  8 17:05:19 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 9 May 2014 01:05:19 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <536B9A3E.9030905@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <CADiSq7fKw5L9u_gs56cMSk_yLUAoayF9HihsfWUSgPJ_Gf9+Ag@mail.gmail.com>
 <536B9A3E.9030905@egenix.com>
Message-ID: <CADiSq7ew+SqxR8xJjyQH7eP6ybsm3G9Qvs0fTgtWoguvk9LkCQ@mail.gmail.com>

On 9 May 2014 00:52, "M.-A. Lemburg" <mal at egenix.com> wrote:
>
> On 08.05.2014 15:57, Nick Coghlan wrote:
>
> > (even the question of "does this software actually work?" is in our
> > sights if you consider a long enough time span). That's hard enough
> > with just a couple of service providers (Fastly and Rackspace) in the
> > mix - it quickly becomes impossible if every new dependency from an
> > installation request adds a new point of failure.
>
> Like I said: the best option is to use a local directory which
> only contains packages files that you have inspected and
> actually trust :-)

Correct, but that raises the barrier to entry too high. The pip defaults
are aimed at providing an experience with the fewest points of failure that
is currently achievable, with a minimal learning curve.

We still have a long way to go, but if people want to influence those
design decisions, the relevant lists are pypa-dev (for pip specific
discussions) and distutils-sig (for higher level cross-tool design
decisions)

Cheers,
Nick.

>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Source  (#1, May 08 2014)
> >>> Python Projects, Consulting and Support ...   http://www.egenix.com/
> >>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
> >>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
>
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/4ae16268/attachment.html>

From stefan at bytereef.org  Thu May  8 17:19:07 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 8 May 2014 17:19:07 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
Message-ID: <20140508151906.GA29543@sleipnir.bytereef.org>

Donald Stufft <donald at stufft.io> wrote:
> hosted packages are brittle and more prone to failure. Every single external
> server adds *another* SPOF into any particular install set. Even if every
> external server has a 99.9% uptime, when you combine multiple of them the total
> uptime of any particular set of requirements drops quickly. This has been a
> major complaint that people have had over time.

We have been through that many times; to me this is an indication that
people are using pip under circumstances when they should not.  pip is
not apt-get.

[I am aware that *you* know that, just stating it again for the broader
 audience.]


> >  2) Last time I looked, access credentials (via "lost login") were sent
> >     out in plaintext.
> 
> The existence of other security issues is not an excuse to not fix a security
> issue. There are other problems and we're slowly working on trying to clear
> them out.

It is, however, a reason to avoid error messages that could *imply* (rightly
or wrongly) to users that the combination of pip and internal packages is
safe.


> >  3) AFAIK people can upload a different (malicious) version of a
> >     package with *the exact same name*.
> 
> Yes, a malicious author is needfully outside of the threat model for PyPI/pip.

How so?  I'm avoiding this attack by publishing sha256sums at release time.
The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging
up a checksum mismatch from the mailing list archives.


> >  6) D.J. Bernstein, who is somewhat security minded, has been shipping
> >     his software *for years* with just plain HTTP and published checksums.
> 
> Argument from authority doesn't really hold up very well when DJB doesn't
> distribute is software in a way that is intended to be consumed mechanically.
> Also while he may be a crypto expert he is far from an expert on successfully
> distributing software, unless you also think that the signature checking in
> most OS provided package managers is pointless.

That is sort of a strawman.  The whole point *is* that certain distribution
models don't lend themselves to mechanical consumption.  I cannot speak
for DJB, perhaps he is just thinking that GPG signing is pointless if
users can't validate the signature and SSL is pointless because one cannot
trust governments.

OS package signing is useful since the packages are curated.  If anyone
could upload a package to Debian, whereupon it would be signed with the
official key, apt-get would lose its usefulness.



Stefan Krah



From rdmurray at bitdance.com  Thu May  8 17:21:27 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 08 May 2014 11:21:27 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
 <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>
Message-ID: <20140508152127.E4219250DBB@webabinitio.net>

On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft <donald at stufft.io> wrote:
> Most users are not going to care up until the point where the external server
> is unavailable, and then they care a whole lot. On the tin it sounds reasonable
> to just download the external file if the server is up however we've done
> that for a long time and the end result has been end user pain.
> 
> Now requiring someone to add a flag in order to download an externally hosted
> file is also end user pain. The difference between the two pains is when they
> happen. The requiring a flag pain happens at the point of decision, when you
> decide to make your deployment depend on something hosted externally. The 
> default to allow pain happens sometime in the future, when you may or may not
> have any idea why suddenly your installs aren't working (and when you look,
> PyPI is up so you're really very confused why this particular file doesn't
> work). Even worse is the case when a project has some old files, but the newer
> versions aren't hosted and suddenly people are getting very old releases which
> is even more confusing to the end users.

Ah, I understand now.

Your perspective is as someone who is using pip for *deployment*.

I'm speaking from a python+plus+pip end-user perspective, which is going
to be even more common now that it is part of the Python distribution.

I'm not sure how you reconcile these two worlds.  I would venture to
suggest that if you are using it for deployment you really ought to
be using a local package repository[*], not the global one; but, as
someone observed, the sensible thing to do and what people actually
do are often very different; and, since I haven't done this particular
kind of deployment scenario in Python myself, there may be reasons
I'm missing.

However, your last mention of "end users" confuses me.  Why are "end
users" getting old packages in a deployment situation?  Isn't it the
developer/operations people (and the latter would, I assume, control
the 'external packages' flag) who would be seeing that?  Maybe you mean
something by deployment different from how I use the word?

--David

[*] I found it *such* a pain to do this for perl/cpan.  I have a
project for a customer where I have to do this, and the hoops I had
to jump through to get a reliable deployment (where packages wouldn't
be unexpectedly upgraded under my feet) were nasty.  (This was several
years ago and I haven't revisited it, so maybe things have gotten better,
or I just missed something.)

I haven't had to do it for python yet, oddly enough, so I don't know
how hard it is for Python.

From donald at stufft.io  Thu May  8 17:24:47 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 11:24:47 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508151906.GA29543@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <20140508151906.GA29543@sleipnir.bytereef.org>
Message-ID: <5151B70D-7224-47B1-BA95-257652E60469@stufft.io>


On May 8, 2014, at 11:19 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald Stufft <donald at stufft.io> wrote:
>> hosted packages are brittle and more prone to failure. Every single external
>> server adds *another* SPOF into any particular install set. Even if every
>> external server has a 99.9% uptime, when you combine multiple of them the total
>> uptime of any particular set of requirements drops quickly. This has been a
>> major complaint that people have had over time.
> 
> We have been through that many times; to me this is an indication that
> people are using pip under circumstances when they should not.  pip is
> not apt-get.
> 
> [I am aware that *you* know that, just stating it again for the broader
> audience.]
> 
> 
>>> 2) Last time I looked, access credentials (via "lost login") were sent
>>>    out in plaintext.
>> 
>> The existence of other security issues is not an excuse to not fix a security
>> issue. There are other problems and we're slowly working on trying to clear
>> them out.
> 
> It is, however, a reason to avoid error messages that could *imply* (rightly
> or wrongly) to users that the combination of pip and internal packages is
> safe.
> 
> 
>>> 3) AFAIK people can upload a different (malicious) version of a
>>>    package with *the exact same name*.
>> 
>> Yes, a malicious author is needfully outside of the threat model for PyPI/pip.
> 
> How so?  I'm avoiding this attack by publishing sha256sums at release time.
> The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging
> up a checksum mismatch from the mailing list archives.

Practically speaking exactly zero people will ever bother to dig up the checksum
from a mailing list archive, or even verify the hash that you have right on PyPI
itself. Worse security that people actually use is infinitely better than better
security that nobody uses.

> 
> 
>>> 6) D.J. Bernstein, who is somewhat security minded, has been shipping
>>>    his software *for years* with just plain HTTP and published checksums.
>> 
>> Argument from authority doesn't really hold up very well when DJB doesn't
>> distribute is software in a way that is intended to be consumed mechanically.
>> Also while he may be a crypto expert he is far from an expert on successfully
>> distributing software, unless you also think that the signature checking in
>> most OS provided package managers is pointless.
> 
> That is sort of a strawman.  The whole point *is* that certain distribution
> models don't lend themselves to mechanical consumption.  I cannot speak
> for DJB, perhaps he is just thinking that GPG signing is pointless if
> users can't validate the signature and SSL is pointless because one cannot
> trust governments.
> 
> OS package signing is useful since the packages are curated.  If anyone
> could upload a package to Debian, whereupon it would be signed with the
> official key, apt-get would lose its usefulness.

People are going to mechanically consume them. You can tell them it?s wrong
all you want but they are going to do it.

> 
> 
> 
> Stefan Krah
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/b6f9e385/attachment.sig>

From donald at stufft.io  Thu May  8 17:32:28 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 11:32:28 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508152127.E4219250DBB@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
 <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>
 <20140508152127.E4219250DBB@webabinitio.net>
Message-ID: <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io>


On May 8, 2014, at 11:21 AM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Thu, 08 May 2014 10:37:15 -0400, Donald Stufft <donald at stufft.io> wrote:
>> Most users are not going to care up until the point where the external server
>> is unavailable, and then they care a whole lot. On the tin it sounds reasonable
>> to just download the external file if the server is up however we've done
>> that for a long time and the end result has been end user pain.
>> 
>> Now requiring someone to add a flag in order to download an externally hosted
>> file is also end user pain. The difference between the two pains is when they
>> happen. The requiring a flag pain happens at the point of decision, when you
>> decide to make your deployment depend on something hosted externally. The 
>> default to allow pain happens sometime in the future, when you may or may not
>> have any idea why suddenly your installs aren't working (and when you look,
>> PyPI is up so you're really very confused why this particular file doesn't
>> work). Even worse is the case when a project has some old files, but the newer
>> versions aren't hosted and suddenly people are getting very old releases which
>> is even more confusing to the end users.
> 
> Ah, I understand now.
> 
> Your perspective is as someone who is using pip for *deployment*.

Deployment, or any kind of situation where you want to have a reproducible
build. Generally via deployment yes.

> 
> I'm speaking from a python+plus+pip end-user perspective, which is going
> to be even more common now that it is part of the Python distribution.
> 
> I'm not sure how you reconcile these two worlds.  I would venture to
> suggest that if you are using it for deployment you really ought to
> be using a local package repository[*], not the global one; but, as
> someone observed, the sensible thing to do and what people actually
> do are often very different; and, since I haven't done this particular
> kind of deployment scenario in Python myself, there may be reasons
> I'm missing.

People simply don?t do this[1]. Especially in a world with things like Heroku
existing which makes it stupid simple to use pip to install from PyPI but
installing from your own server requires standing up some infrastructure.

> 
> However, your last mention of "end users" confuses me.  Why are "end
> users" getting old packages in a deployment situation?  Isn't it the
> developer/operations people (and the latter would, I assume, control
> the 'external packages' flag) who would be seeing that?  Maybe you mean
> something by deployment different from how I use the word?

Someone using pip, this may be a developer who is just trying to recreate
their production environment locally, this may be someone using chef/puppet
to automate installing via pip, this may be someone pushing to Heroku.

The old versions thing is more that it?s really confusing when you type
``pip install foo`` on a monday and get 2.0, and ``pip install foo`` on weds
and get 0.4.

> 
> --David
> 
> [*] I found it *such* a pain to do this for perl/cpan.  I have a
> project for a customer where I have to do this, and the hoops I had
> to jump through to get a reliable deployment (where packages wouldn't
> be unexpectedly upgraded under my feet) were nasty.  (This was several
> years ago and I haven't revisited it, so maybe things have gotten better,
> or I just missed something.)
> 
> I haven't had to do it for python yet, oddly enough, so I don't know
> how hard it is for Python.

For Python with pip you can use a requirements.txt file to create a set of
dependencies that are pinned to exact versions like:

foo==2.0
bar==2.3

And pip will (theoretically, our dep solving is real bad ATM) install exactly
those versions from your index server. Generally this means PyPI which
means the author can delete the version and upload a new file with the
same version number. However it?s also trivial to stand up your own
server. It can be as easy as pointing nginx/Apache at a static directory with
autoindex = True. (See: https://wheels.caremad.io/).

On top of that there is peep which adds a secure message digest on it to
make sure that the author/index didn?t swap things out on you, and there
is some discussion about how best to add that to pip itself.

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/3812d75d/attachment.sig>

From stefan at bytereef.org  Thu May  8 17:34:54 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 8 May 2014 17:34:54 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
 <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
Message-ID: <20140508153454.GB29543@sleipnir.bytereef.org>

Donald Stufft <donald at stufft.io> wrote:
> > Today I've switched to manual install mode with manual sha256sum verification
> > which is *far* safer than anything you get via pip right now.
> 
> It is not safer in any meaingful way.
> 
> If someone is in a position to compromise the integrity of PyPI's TLS, they
> can replace the hash on that page with something else. Now you've attempted to
> work around this by telling people to go look up the release announcement
> hash. However if someone can compromise the integrity of PyPI's TLS, they can
> also compromise the integrity of https://mail.python.org/, or GMane, or any
> other TLS based website[1].

Of course it is safer.  Suppose a file is stored on PyPI:

  1) Attacker guesses my username (or is it even visible, I'm not sure).

  2) Clicks on "lost login".

  3) Intercepts mail (difficult, but far from the TLS attack category).
     Maybe on a home or university network.  Or a rogue person at a
     mail provider.

  4) Changes the uploaded file together with the hash.


pip would be perfectly happy, checking the hash via Google would turn
up a mismatch.


Stefan Krah



From mal at egenix.com  Thu May  8 17:37:57 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 08 May 2014 17:37:57 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <536B97E2.8070805@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com>
Message-ID: <536BA4D5.7080208@egenix.com>

On 08.05.2014 16:42, M.-A. Lemburg wrote:
> On 08.05.2014 15:58, Donald Stufft wrote:
>>
>> On May 8, 2014, at 9:39 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>>> Well, to be fair and leaving aside uptime concerns and the general
>>> desire to always install packages from some server instead of
>>> a safe and trusted local directory (probably too obvious ;-),
>>> it would certainly be possible to add support for
>>> trusted externally hosted packages.
>>
>> There is support for trusted externally hosted packages, you put the URL in
>> PyPI and include a hash in the fragment like so:
>>
>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56
>>
>> The hash can be md5 or any of the sha-2 family.
>>
>> Now this does not mean that ``pip install cdecimal`` will automatically install
>> this, because whether or not you're willing to install from servers other than
>> PyPI[1] is a policy decision for the end user of pip. 
> 
> Hmm, if you call that feature "trusted externally hosted packages",
> pip should really do trust them, right ? ;-)
> 
> I can understand that pip defaults to not trusting URLs which don't
> meet the above feature requirements, but not that it still warns
> about unreliable externally hosted packages even if the above
> feature is used.
> 
> At the moment, pip will refuse to use an externally hosted files even
> if the package author uses the above hashed URLs; even with HTTPS
> and proper SSL certificate chain.

Could this perhaps be changed/reconsidered for pip ?

Note that easy_install/setuptools does not have such problems.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 08 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From donald at stufft.io  Thu May  8 17:43:37 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 11:43:37 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508153454.GB29543@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
 <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
 <20140508153454.GB29543@sleipnir.bytereef.org>
Message-ID: <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io>


On May 8, 2014, at 11:34 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald Stufft <donald at stufft.io> wrote:
>>> Today I've switched to manual install mode with manual sha256sum verification
>>> which is *far* safer than anything you get via pip right now.
>> 
>> It is not safer in any meaingful way.
>> 
>> If someone is in a position to compromise the integrity of PyPI's TLS, they
>> can replace the hash on that page with something else. Now you've attempted to
>> work around this by telling people to go look up the release announcement
>> hash. However if someone can compromise the integrity of PyPI's TLS, they can
>> also compromise the integrity of https://mail.python.org/, or GMane, or any
>> other TLS based website[1].
> 
> Of course it is safer.  Suppose a file is stored on PyPI:
> 
>  1) Attacker guesses my username (or is it even visible, I'm not sure).
> 
>  2) Clicks on "lost login".
> 
>  3) Intercepts mail (difficult, but far from the TLS attack category).
>     Maybe on a home or university network.  Or a rogue person at a
>     mail provider.
> 
>  4) Changes the uploaded file together with the hash.
> 
> 
> pip would be perfectly happy, checking the hash via Google would turn
> up a mismatch.

I said ?meaningful?. Almost nobody is going to ever bother googling it and
the likelihood that someone is able to MITM *you* specifically is far lesser
than the likelihood that someone is going to MITM one of the cdecimal users.

Additionally your messages aren?t signed and email isn?t an authenticated
profile so if someone was able to get your password they could simply spoof
and email from you to the mailing list with new hashes, or edit out the description
telling people to go google some stuff.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/0f171a61/attachment.sig>

From donald at stufft.io  Thu May  8 17:46:14 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 11:46:14 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <536BA4D5.7080208@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
Message-ID: <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>


On May 8, 2014, at 11:37 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 08.05.2014 16:42, M.-A. Lemburg wrote:
>> On 08.05.2014 15:58, Donald Stufft wrote:
>>> 
>>> On May 8, 2014, at 9:39 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> 
>>>> Well, to be fair and leaving aside uptime concerns and the general
>>>> desire to always install packages from some server instead of
>>>> a safe and trusted local directory (probably too obvious ;-),
>>>> it would certainly be possible to add support for
>>>> trusted externally hosted packages.
>>> 
>>> There is support for trusted externally hosted packages, you put the URL in
>>> PyPI and include a hash in the fragment like so:
>>> 
>>> http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56
>>> 
>>> The hash can be md5 or any of the sha-2 family.
>>> 
>>> Now this does not mean that ``pip install cdecimal`` will automatically install
>>> this, because whether or not you're willing to install from servers other than
>>> PyPI[1] is a policy decision for the end user of pip. 
>> 
>> Hmm, if you call that feature "trusted externally hosted packages",
>> pip should really do trust them, right ? ;-)
>> 
>> I can understand that pip defaults to not trusting URLs which don't
>> meet the above feature requirements, but not that it still warns
>> about unreliable externally hosted packages even if the above
>> feature is used.
>> 
>> At the moment, pip will refuse to use an externally hosted files even
>> if the package author uses the above hashed URLs; even with HTTPS
>> and proper SSL certificate chain.
> 
> Could this perhaps be changed/reconsidered for pip ?
> 
> Note that easy_install/setuptools does not have such problems.

Anything can be changes or reconsidered of course. I feel pretty strongly that
an installer should not install things from places other than the index without
a specific opt in. That discussion would be best done on distutils-sig as it
would require reversing the decision in PEP438.

I really don't feel strongly one way or the other about the *warning* that
happens when you allow an external file. It exists primarily because at the
time it was implemented external files were default to allowed.


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/35c8293a/attachment.sig>

From stefan at bytereef.org  Thu May  8 18:03:25 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Thu, 8 May 2014 18:03:25 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
 <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
 <20140508153454.GB29543@sleipnir.bytereef.org>
 <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io>
Message-ID: <20140508160325.GA29870@sleipnir.bytereef.org>

Donald Stufft <donald at stufft.io> wrote:
> I said ?meaningful?. Almost nobody is going to ever bother googling it and
> the likelihood that someone is able to MITM *you* specifically is far lesser
> than the likelihood that someone is going to MITM one of the cdecimal users.

I'm doing this for important installs. -- That is how I installed qmail
and djbdns.


> Additionally your messages aren?t signed and email isn?t an authenticated
> profile so if someone was able to get your password they could simply spoof
> and email from you to the mailing list with new hashes, or edit out the description
> telling people to go google some stuff.

Signing messages is pointless if the key isn't well connected.  Also, I'm
reading the lists and would notice a "release".  Most importantly, the
checksum mismatch would still be found, since the old messages with the
correct sum would still exist under the scenario we're talking about
(i.e. not GHCQ hacking into Belgacom routers).


Stefan Krah



From donald at stufft.io  Thu May  8 18:22:16 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 12:22:16 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508160325.GA29870@sleipnir.bytereef.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
 <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
 <20140508153454.GB29543@sleipnir.bytereef.org>
 <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io>
 <20140508160325.GA29870@sleipnir.bytereef.org>
Message-ID: <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io>


On May 8, 2014, at 12:03 PM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald Stufft <donald at stufft.io> wrote:
>> I said ?meaningful?. Almost nobody is going to ever bother googling it and
>> the likelihood that someone is able to MITM *you* specifically is far lesser
>> than the likelihood that someone is going to MITM one of the cdecimal users.
> 
> I'm doing this for important installs. -- That is how I installed qmail
> and djbdns.
> 
> 
>> Additionally your messages aren?t signed and email isn?t an authenticated
>> profile so if someone was able to get your password they could simply spoof
>> and email from you to the mailing list with new hashes, or edit out the description
>> telling people to go google some stuff.
> 
> Signing messages is pointless if the key isn't well connected.  Also, I'm
> reading the lists and would notice a "release".  Most importantly, the
> checksum mismatch would still be found, since the old messages with the
> correct sum would still exist under the scenario we're talking about
> (i.e. not GHCQ hacking into Belgacom routers).
> 
> 

I?m unsure if you?re being willfully dense or if you?re just not understanding
what I mean when I say ?almost?. Of course there are going to be a few outliers
where people do bother to do that, but it?s not going to be common place at
all.

But whatever, I?ve removed the warning that occurs when you install an
externally hosted file [1] and it will be included in pip 1.6. I have not
changed the defaults for --allow-all-external nor have I removed the warning
that occurs when someone elects to install an unverifiable download.

[1] https://github.com/pypa/pip/commit/9f56b79e8d

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/f14cba3b/attachment.sig>

From rdmurray at bitdance.com  Thu May  8 18:42:16 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 08 May 2014 12:42:16 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
 <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>
 <20140508152127.E4219250DBB@webabinitio.net>
 <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io>
Message-ID: <20140508164217.28F92B14088@webabinitio.net>

On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft <donald at stufft.io> wrote:
> On May 8, 2014, at 11:21 AM, R. David Murray <rdmurray at bitdance.com> wrote:
> > Ah, I understand now.
> > 
> > Your perspective is as someone who is using pip for *deployment*.
> 
> Deployment, or any kind of situation where you want to have a reproducible
> build. Generally via deployment yes.
[...]
> For Python with pip you can use a requirements.txt file to create a set of
> dependencies that are pinned to exact versions like:
> 
> foo==2.0
> bar==2.3
> 
> And pip will (theoretically, our dep solving is real bad ATM) install exactly
> those versions from your index server. Generally this means PyPI which

OK, this makes sense, then.  (I wish perl/cpan had something
similar...maybe it does, but I couldn't find it at the time.)

This still leaves the fact that there is a disconnect between the
"needs" of two different audiences for PIP: people who deploy things,
and everyone else who just uses pip to install stuff.

The second group is going to overwhelm the first group, if it doesn't
already.

And I think that's all the comments I have on this issue :)

--David

From donald at stufft.io  Thu May  8 18:57:00 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 12:57:00 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140508164217.28F92B14088@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508141140.4AEB1250DBB@webabinitio.net>
 <20140508142135.09EE0250DBC@webabinitio.net>
 <42CA0782-9910-4563-AD92-21780DA2E1E7@stufft.io>
 <20140508152127.E4219250DBB@webabinitio.net>
 <3AAC8E13-FAB8-4DB5-8333-298D6F897704@stufft.io>
 <20140508164217.28F92B14088@webabinitio.net>
Message-ID: <B3072FFC-29B3-4E13-A97D-5016B0417D18@stufft.io>


On May 8, 2014, at 12:42 PM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Thu, 08 May 2014 11:32:28 -0400, Donald Stufft <donald at stufft.io> wrote:
>> On May 8, 2014, at 11:21 AM, R. David Murray <rdmurray at bitdance.com> wrote:
>>> Ah, I understand now.
>>> 
>>> Your perspective is as someone who is using pip for *deployment*.
>> 
>> Deployment, or any kind of situation where you want to have a reproducible
>> build. Generally via deployment yes.
> [...]
>> For Python with pip you can use a requirements.txt file to create a set of
>> dependencies that are pinned to exact versions like:
>> 
>> foo==2.0
>> bar==2.3
>> 
>> And pip will (theoretically, our dep solving is real bad ATM) install exactly
>> those versions from your index server. Generally this means PyPI which
> 
> OK, this makes sense, then.  (I wish perl/cpan had something
> similar...maybe it does, but I couldn't find it at the time.)
> 
> This still leaves the fact that there is a disconnect between the
> "needs" of two different audiences for PIP: people who deploy things,
> and everyone else who just uses pip to install stuff.

Yup balancing between the two is something we have to do in every
decision we make. When PEP438 was being discussed I did a pretty
extensive amount of investigation into what affect this change would
have [1]. What I found was that:

- The sizable majority was projects would host things on PyPI
- There was a significant chunk of projects where a single release or two
  would only be available externally and it was an accident that they weren?t
  uploaded.
- Of the links that were available externally, very few of them were available
  in a way that was verifiable and were thus insecure to install.

Because of this it was determined that simply allowing externally hosted
files without also allowing externally hosted and unverified files would not
actually have a significant impact for the vast bulk of the projects that
were not hosted on PyPI.

> 
> The second group is going to overwhelm the first group, if it doesn't
> already.

Generally yes, because not every who uses pip to deploy uses pip to
install locally, but most people who use pip to deploy also use pip
locally.

> 
> And I think that's all the comments I have on this issue :)

[1]: https://github.com/dstufft/pypi.linkcheck

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/46899527/attachment.sig>

From brian at python.org  Thu May  8 18:59:19 2014
From: brian at python.org (Brian Curtin)
Date: Thu, 8 May 2014 11:59:19 -0500
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
Message-ID: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>

This is mostly a question for Martin, but perhaps someone else would also know.

I'm trying to build the 2.7 installers so I can backport the path
option from 3.3, but I can't seem to figure out which version of Tix
is necessary to have a complete build. So far any of them on
http://svn.python.org/projects/external don't appear to be configured
to work with the combination of tcl and tk that are used on 2.7, per
the buildbot external scripts. It's another issue that Tix isn't even
downloaded by the scripts, but I'll do it manually right now just to
get this going.

Any tips?

From martin at v.loewis.de  Thu May  8 21:36:22 2014
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 08 May 2014 21:36:22 +0200
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
In-Reply-To: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
References: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
Message-ID: <536BDCB6.6030300@v.loewis.de>

Am 08.05.14 18:59, schrieb Brian Curtin:
> This is mostly a question for Martin, but perhaps someone else would also know.
> 
> I'm trying to build the 2.7 installers so I can backport the path
> option from 3.3, but I can't seem to figure out which version of Tix
> is necessary to have a complete build. So far any of them on
> http://svn.python.org/projects/external don't appear to be configured
> to work with the combination of tcl and tk that are used on 2.7, per
> the buildbot external scripts. It's another issue that Tix isn't even
> downloaded by the scripts, but I'll do it manually right now just to
> get this going.
> 
> Any tips?

Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one
where python.mak was pointing to Tcl 8.5.2. It may not have been tagged.

In 2.7, it wasn't checked out out automatically because Tix requires the
original Tcl/Tk path names, so building it automatically would have failed.

Regards,
Martin



From p.f.moore at gmail.com  Thu May  8 23:02:16 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 8 May 2014 22:02:16 +0100
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
Message-ID: <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>

On 8 May 2014 16:46, Donald Stufft <donald at stufft.io> wrote:
> Anything can be changes or reconsidered of course. I feel pretty strongly that
> an installer should not install things from places other than the index without
> a specific opt in. That discussion would be best done on distutils-sig as it
> would require reversing the decision in PEP438.

I think it's worth reconsidering. Since this behaviour was
implemented, there have been many instances of confusion and
unhappiness with the situation, both from package developers and pip
users. I don't think that's good for pip. I would like to see PEP 438
reviewed with the intention of working out how to fix the user
experience (ideally while retaining the reliability enhancements, but
accepting that compromises may be needed).

Some points:

1. Often a user won't know that a package is externally hosted until
an install fails. Having to add a flag and retry is a lousy
experience. Why not at a minimum have an "externally hosted - are you
sure?" prompt?
2. The way the flags are implemented (notably the need to repeat the
package name) is clearly confusing and irritating for users, even if
the reasons are logical. We should look at fixing that.
3. The user complaints against pip are significant and ongoing. I
don't think they should be ignored. If PEP 438 cannot be reconsidered
in the light of user feedback, then I think the pip developers may
need to have a discussion about whether we conform to the PEP (clearly
Donald thinks we should, I'm not sure I do, and I don't know about the
others). But I don't think it's healthy for pip to be looking at
deliberately ignoring an accepted PEP, so I'd prefer it if the debate
was around addressing the issues in the PEP itself.
4. Maybe distutils-sig *isn't* the right place to discuss revising PEP
438. The issues getting raised are end-user problems, and the users
most affected are unlikely to be on that list.

Socially, this change does not seem to be having the effect of
persuading more package developers to host on PyPI. The stick doesn't
appear to have worked, maybe we should be trying to find a carrot? Or
maybe we have to accept that some developers have sound reasons for
not hosting on PyPI and work with them to find an acceptable
compromise? Has anyone checked what Stefan's reasons are for not
hosting cdecimal on PyPI? Do they represent a use case that the PEP
hasn't considered?

> I really don't feel strongly one way or the other about the *warning* that
> happens when you allow an external file. It exists primarily because at the
> time it was implemented external files were default to allowed.

I think it's reasonable to remove the warning. If the user chooses to
allow an external file, it makes sense to assume they understand the
implications and not nag them about their decision. Particularly given
the level of controversy the warning is generating.

On a personal note, I'm uncomfortable with the way this change is
perceived as a case of *pip* enforcing a behaviour that the pip
developers feel should be required. I actually don't like this change
particularly. So having pip implement the behaviour required by that
PEP is to me simply a case of compliance with the agreed standard. But
now, as a pip developer, being held responsible for the resulting user
pain, and being expected to defend it, does not make me happy.

Paul

From donald at stufft.io  Thu May  8 23:22:05 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 17:22:05 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
Message-ID: <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>


On May 8, 2014, at 5:02 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 8 May 2014 16:46, Donald Stufft <donald at stufft.io> wrote:
>> Anything can be changes or reconsidered of course. I feel pretty strongly that
>> an installer should not install things from places other than the index without
>> a specific opt in. That discussion would be best done on distutils-sig as it
>> would require reversing the decision in PEP438.
> 
> I think it's worth reconsidering. Since this behaviour was
> implemented, there have been many instances of confusion and
> unhappiness with the situation, both from package developers and pip
> users. I don't think that's good for pip. I would like to see PEP 438
> reviewed with the intention of working out how to fix the user
> experience (ideally while retaining the reliability enhancements, but
> accepting that compromises may be needed).

I think most of the confusion has been over the fact that ?allow-external
takes a package name, not that it exists at all.

> 
> Some points:
> 
> 1. Often a user won't know that a package is externally hosted until
> an install fails. Having to add a flag and retry is a lousy
> experience. Why not at a minimum have an "externally hosted - are you
> sure?" prompt?

A prompt is OK with me.

> 2. The way the flags are implemented (notably the need to repeat the
> package name) is clearly confusing and irritating for users, even if
> the reasons are logical. We should look at fixing that.

Yea, there?s a ticket for this.

> 3. The user complaints against pip are significant and ongoing. I
> don't think they should be ignored. If PEP 438 cannot be reconsidered
> in the light of user feedback, then I think the pip developers may
> need to have a discussion about whether we conform to the PEP (clearly
> Donald thinks we should, I'm not sure I do, and I don't know about the
> others). But I don't think it's healthy for pip to be looking at
> deliberately ignoring an accepted PEP, so I'd prefer it if the debate
> was around addressing the issues in the PEP itself.

I don?t think the problem with with the PEP.

> 4. Maybe distutils-sig *isn't* the right place to discuss revising PEP
> 438. The issues getting raised are end-user problems, and the users
> most affected are unlikely to be on that list.
> 
> Socially, this change does not seem to be having the effect of
> persuading more package developers to host on PyPI. The stick doesn't
> appear to have worked, maybe we should be trying to find a carrot?

Do you have any data to point to that says it hasn?t worked? Just to see
what impact it has had, I?m running my scripts again that I ran a year
ago to see what has changed, already I can see they are processing
MUCH faster than last year.

> Or
> maybe we have to accept that some developers have sound reasons for
> not hosting on PyPI and work with them to find an acceptable
> compromise? Has anyone checked what Stefan's reasons are for not
> hosting cdecimal on PyPI? Do they represent a use case that the PEP
> hasn't considered?

If I recall correctly his reasoning is that he finds the legal requirements
associated with uploading to PyPI to be unsatisfactory.

> 
>> I really don't feel strongly one way or the other about the *warning* that
>> happens when you allow an external file. It exists primarily because at the
>> time it was implemented external files were default to allowed.
> 
> I think it's reasonable to remove the warning. If the user chooses to
> allow an external file, it makes sense to assume they understand the
> implications and not nag them about their decision. Particularly given
> the level of controversy the warning is generating.

The warning is gone as of a few hours ago.

> 
> On a personal note, I'm uncomfortable with the way this change is
> perceived as a case of *pip* enforcing a behaviour that the pip
> developers feel should be required. I actually don't like this change
> particularly. So having pip implement the behaviour required by that
> PEP is to me simply a case of compliance with the agreed standard. But
> now, as a pip developer, being held responsible for the resulting user
> pain, and being expected to defend it, does not make me happy.
> 
> Paul

I think the pain is being overrepresented and the positives are being ignored.
The problem is the benefits of this PEP are much like the benefits of TLS
too. For the vast majority of people they don?t notice anything different except
installing things is faster and more reliable. They don?t attribute that to the
PEP or this decision, they just internalize it as the new norm. However the
people who this does affect will seek out why it broke and raise an issue
citing that thing specifically. This creates a perception of lots of pain for no
gain when the reality is not that.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/f77af49c/attachment-0001.sig>

From zachary.ware+pydev at gmail.com  Thu May  8 23:20:53 2014
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 8 May 2014 16:20:53 -0500
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
In-Reply-To: <536BDCB6.6030300@v.loewis.de>
References: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
 <536BDCB6.6030300@v.loewis.de>
Message-ID: <CAKJDb-MWH5QHRFcNU7u0ivY9gcubhtk544YPPKd1Wifyr9V_dQ@mail.gmail.com>

On Thu, May 8, 2014 at 2:36 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Am 08.05.14 18:59, schrieb Brian Curtin:
>> This is mostly a question for Martin, but perhaps someone else would also know.
>>
>> I'm trying to build the 2.7 installers so I can backport the path
>> option from 3.3, but I can't seem to figure out which version of Tix
>> is necessary to have a complete build. So far any of them on
>> http://svn.python.org/projects/external don't appear to be configured
>> to work with the combination of tcl and tk that are used on 2.7, per
>> the buildbot external scripts. It's another issue that Tix isn't even
>> downloaded by the scripts, but I'll do it manually right now just to
>> get this going.
>>
>> Any tips?
>
> Ah, 2.7. I was using a checkout of tix-8.4.3.x from Nov 13, 2010, one
> where python.mak was pointing to Tcl 8.5.2. It may not have been tagged.
>
> In 2.7, it wasn't checked out out automatically because Tix requires the
> original Tcl/Tk path names, so building it automatically would have failed.

I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple
of weeks ago (see http://bugs.python.org/issue21303), but hadn't
gotten anything done with Tix yet.  It should just need python.mak
updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as
soon as I can.

-- 
Zach

From ncoghlan at gmail.com  Fri May  9 00:20:10 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 9 May 2014 08:20:10 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
Message-ID: <CADiSq7dqZnwQ1hz33vtMk26C=3svEgwuZMu7k2xmtMVQNYkERA@mail.gmail.com>

On 9 May 2014 07:23, "Donald Stufft" <donald at stufft.io> wrote:
> On May 8, 2014, at 5:02 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>
> > Or
> > maybe we have to accept that some developers have sound reasons for
> > not hosting on PyPI and work with them to find an acceptable
> > compromise? Has anyone checked what Stefan's reasons are for not
> > hosting cdecimal on PyPI? Do they represent a use case that the PEP
> > hasn't considered?
>
> If I recall correctly his reasoning is that he finds the legal
requirements
> associated with uploading to PyPI to be unsatisfactory.

I actually need to follow up on that, because the terms *were* legally
questionable last time I looked (also too hard to review, since as far as I
am aware, they're only presented during new user sign-up).

I'll deal with that at work today.

Cheers,
Nick.

P.S. Still the wrong list, folks. Most of the people you need to convince
to get changes made to pip, PyPI and the packaging ecosystem in general
aren't here.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/445b1604/attachment.html>

From donald at stufft.io  Fri May  9 00:22:30 2014
From: donald at stufft.io (Donald Stufft)
Date: Thu, 8 May 2014 18:22:30 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <CADiSq7dqZnwQ1hz33vtMk26C=3svEgwuZMu7k2xmtMVQNYkERA@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <CADiSq7dqZnwQ1hz33vtMk26C=3svEgwuZMu7k2xmtMVQNYkERA@mail.gmail.com>
Message-ID: <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io>


On May 8, 2014, at 6:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 9 May 2014 07:23, "Donald Stufft" <donald at stufft.io> wrote:
> > On May 8, 2014, at 5:02 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> >
> > > Or
> > > maybe we have to accept that some developers have sound reasons for
> > > not hosting on PyPI and work with them to find an acceptable
> > > compromise? Has anyone checked what Stefan's reasons are for not
> > > hosting cdecimal on PyPI? Do they represent a use case that the PEP
> > > hasn't considered?
> >
> > If I recall correctly his reasoning is that he finds the legal requirements
> > associated with uploading to PyPI to be unsatisfactory.
> 
> I actually need to follow up on that, because the terms *were* legally questionable last time I looked (also too hard to review, since as far as I am aware, they're only presented during new user sign-up).
> 
> I'll deal with that at work today.
> 
> 
> 
I?m pretty sure VanL wrote the terms and has explicitly said they won?t change and are exactly as broad as they need to be without being any broader[1]. They are linked to from the footer of every UI centric PyPI page.

[1] https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/a47922bb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/a47922bb/attachment.sig>

From ncoghlan at gmail.com  Fri May  9 01:07:18 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 9 May 2014 09:07:18 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <CADiSq7dqZnwQ1hz33vtMk26C=3svEgwuZMu7k2xmtMVQNYkERA@mail.gmail.com>
 <67BA12D2-4483-4D32-BB22-BD842CDC7BCA@stufft.io>
Message-ID: <CADiSq7cM8dRpF_jE0xdy9B5OwserK2RK2pA1LaJUrjj0p-913w@mail.gmail.com>

On 9 May 2014 08:22, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On May 8, 2014, at 6:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>>
>> I actually need to follow up on that, because the terms *were* legally
questionable last time I looked (also too hard to review, since as far as I
am aware, they're only presented during new user sign-up).
>>
>> I'll deal with that at work today.
>>
>>
> I?m pretty sure VanL wrote the terms and has explicitly said they won?t
change and are exactly as broad as they need to be without being any
broader[1]. They are linked to from the footer of every UI centric PyPI
page.

Thanks for the additional references. It should be possible to clarify the
terms to address Red Hat's (and other users') concerns while still
addressing the points Van is worried about (for example, adding a footnote
explaining the use cases that need to be covered, and being explicit that
users must respect *both* licenses).

However, this subthread is now even further off-topic for this list, so
I'll take it to python-legal-sig later today with some specific proposed
wording changes.

Cheers,
Nick.

>
> [1]
https://mail.python.org/pipermail/python-legal-sig/2013-March/000003.html
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
DCFA
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/943a3715/attachment.html>

From donald at stufft.io  Fri May  9 06:34:19 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 00:34:19 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
Message-ID: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>


On May 8, 2014, at 5:22 PM, Donald Stufft <donald at stufft.io> wrote:

>> Socially, this change does not seem to be having the effect of
>> persuading more package developers to host on PyPI. The stick doesn't
>> appear to have worked, maybe we should be trying to find a carrot?
> 
> Do you have any data to point to that says it hasn?t worked? Just to see
> what impact it has had, I?m running my scripts again that I ran a year
> ago to see what has changed, already I can see they are processing
> MUCH faster than last year.

The data has finished processing, it represents a time diff of approximately
one year. The pip release that caused all of this was released about 4-5 months
ago.

Overall PyPI has seen a 50% growth in installable projects in that time. If the
change would have had no effect we'd expect to see a ~50% increase across the
board. However what we've seen is a a 60% (+10% of expected) increase in
projects that can only be installed from PyPI and a 12% decrease in projects
that have any unsafe files (-62% of expected).

Further more we can see that if pip were to change the default of
--allow-all-external it would take 23 projects from unable to be installed by
default to able to be installed by default. This represents 0.2% of installable
projects on PyPI. It would take an additional 40 projects and make one or more
additional files able to be downloaded by default.

Some other data points:

* We've gone from 86% of projects being installable from PyPI to 92%.
* We've gone from 5% of projects being only unsafely installable to 3%
* We've gone from 14% of projects having any files unsafe to install to 8%
* We've gone from 0.004% of projects being safely hosted externally to 0.2%

Looking at these numbers I think it's safe to say that in this time period that
the "hosting hygiene" of a PyPI project is more likely to be a better state
than it was a year ago. We cannot state for a fact if this is because of this
change or not, however given that the fallout is ~23 (or ~63) projects out of
38,835 I think it is incredibly reasonable to leave the defaults alone since
there is a reasonably high chance that they played at least some part in that
change.

I'd love to get these numbers to the point where the number of projects
installable strictly from PyPI is 100% (or at least 100% installable safely),
however 92% (or 92.2%) is getting pretty close to that and hopefully that
number will just continue to grow until it hits 100%.

For reference, here's the raw numbers as well as some summary of the data here:

    https://gist.github.com/dstufft/b14008d11c0a5760dbed

And the repository where the raw data as well as the scripts used to collect
and process it is here:

    https://github.com/dstufft/pypi.linkcheck

linkcollector.py collections while linkwriter.py writes out the json file, and
stats2.py processes and gives the numbers from the gist above. links.json is
the data from a year ago, and 2014-05-08.links.json is the data from today.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/475123f3/attachment.sig>

From donald at stufft.io  Fri May  9 06:45:42 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 00:45:42 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>
Message-ID: <3097645C-A16F-4386-B8BF-46CCBA9021D5@stufft.io>


On May 9, 2014, at 12:34 AM, Donald Stufft <donald at stufft.io> wrote:

> The data has finished processing, it represents a time diff of approximately
> one year. The pip release that caused all of this was released about 4-5 months
> ago.

Oh I forgot to mention:

In order to make the comparison as accurate as possible I've used the same
collection script as I did a year ago. This is prior to the real advent of
Wheels so these numbers do not take into accounts Wheels at all (which pip can
also install) but it *does* include Eggs which pip cannot install.

Further more it also includes #egg=dev urls which point to a VCS (and it would
consider those an unverifiable/unsafe download).

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/79820f97/attachment.sig>

From mal at egenix.com  Fri May  9 10:12:03 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 09 May 2014 10:12:03 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>	<536B97E2.8070805@egenix.com>
 <536BA4D5.7080208@egenix.com>	<70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>	<CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
Message-ID: <536C8DD3.6090601@egenix.com>

On 08.05.2014 23:22, Donald Stufft wrote:
> 
>> On a personal note, I'm uncomfortable with the way this change is
>> perceived as a case of *pip* enforcing a behaviour that the pip
>> developers feel should be required. I actually don't like this change
>> particularly. So having pip implement the behaviour required by that
>> PEP is to me simply a case of compliance with the agreed standard. But
>> now, as a pip developer, being held responsible for the resulting user
>> pain, and being expected to defend it, does not make me happy.
> 
> I think the pain is being overrepresented and the positives are being ignored.
> The problem is the benefits of this PEP are much like the benefits of TLS
> too. For the vast majority of people they don?t notice anything different except
> installing things is faster and more reliable. They don?t attribute that to the
> PEP or this decision, they just internalize it as the new norm. However the
> people who this does affect will seek out why it broke and raise an issue
> citing that thing specifically. This creates a perception of lots of pain for no
> gain when the reality is not that.

Donald: I don't think anyone is arguing that hosting packages on
PyPI is a bad thing and PyPI as a service has gotten a lot better
than it was a few years ago.

However, I find it troubling that we as Python developers are *forcing*
the whole Python world to put their code into PyPI.

There are plenty good reasons not to do this, and sometimes it's
even impossible if you want to stay legal (see the PEP for details).

Accordingly, we should respect those reasons make it possible for
Python packages to live elsewhere, without having our tools put
those packages into a bad light or making it harder for Python
users to install such packages than needed.

With the checksum uploaded to PyPI, the only argument against
fetching packages from other places on the Internet is reliability,
not security.

PyPI is not the only reliable file hosting system on the Internet,
so this argument is rather weak.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 09 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From p.f.moore at gmail.com  Fri May  9 11:01:35 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 9 May 2014 10:01:35 +0100
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>
Message-ID: <CACac1F-gZ3ZmJET+Us65CaRzf4GfXyjTZogBQRM+-CrhEaZZ1A@mail.gmail.com>

On 9 May 2014 05:34, Donald Stufft <donald at stufft.io> wrote:
> On May 8, 2014, at 5:22 PM, Donald Stufft <donald at stufft.io> wrote:
>
>>> Socially, this change does not seem to be having the effect of
>>> persuading more package developers to host on PyPI. The stick doesn't
>>> appear to have worked, maybe we should be trying to find a carrot?
>>
>> Do you have any data to point to that says it hasn?t worked? Just to see
>> what impact it has had, I?m running my scripts again that I ran a year
>> ago to see what has changed, already I can see they are processing
>> MUCH faster than last year.
>
> The data has finished processing, it represents a time diff of approximately
> one year. The pip release that caused all of this was released about 4-5 months
> ago.
>
> Overall PyPI has seen a 50% growth in installable projects in that time. If the
> change would have had no effect we'd expect to see a ~50% increase across the
> board. However what we've seen is a a 60% (+10% of expected) increase in
> projects that can only be installed from PyPI and a 12% decrease in projects
> that have any unsafe files (-62% of expected).

Donald,
Thanks for taking the time to get those figures. It does appear that
there are less cases that would be affected than the number of
complaints would imply.

The only concern I have about this type of analysis is that it doesn't
"weight" projects. It may be (and again, I have no data to back this
up) that the projects that are affected detrimentally by this change
are unusually popular or otherwise significant. There's obviously no
way to assess this sensibly other than by making a judgement on the
level of complaints.

But arguing numbers was never my intention here, so let's just say
that I concede that the change has had a positive effect, which is
great.
Paul

From stefan at bytereef.org  Fri May  9 11:43:59 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 9 May 2014 11:43:59 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io>
References: <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <20140508143650.GA29239@sleipnir.bytereef.org>
 <8C923957-DF9F-4C8F-A90C-AB13BDCEB21B@stufft.io>
 <20140508153454.GB29543@sleipnir.bytereef.org>
 <91E5F9B4-DDD2-4820-980A-F55DDA8432B7@stufft.io>
 <20140508160325.GA29870@sleipnir.bytereef.org>
 <2663A769-6D86-46E2-99BD-877F4A101048@stufft.io>
Message-ID: <20140509094359.GA6254@sleipnir.bytereef.org>

Donald Stufft <donald at stufft.io> wrote:
> I?m unsure if you?re being willfully dense or if you?re just not understanding
> what I mean when I say ?almost?. Of course there are going to be a few outliers
> where people do bother to do that, but it?s not going to be common place at
> all.

I suggest that you confine that style of discussion to distutils-sig.


Stefan Krah



From donald at stufft.io  Fri May  9 13:44:25 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 07:44:25 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <536C8DD3.6090601@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>	<536B97E2.8070805@egenix.com>
 <536BA4D5.7080208@egenix.com>	<70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>	<CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
Message-ID: <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>


On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 08.05.2014 23:22, Donald Stufft wrote:
>> 
>>> On a personal note, I'm uncomfortable with the way this change is
>>> perceived as a case of *pip* enforcing a behaviour that the pip
>>> developers feel should be required. I actually don't like this change
>>> particularly. So having pip implement the behaviour required by that
>>> PEP is to me simply a case of compliance with the agreed standard. But
>>> now, as a pip developer, being held responsible for the resulting user
>>> pain, and being expected to defend it, does not make me happy.
>> 
>> I think the pain is being overrepresented and the positives are being ignored.
>> The problem is the benefits of this PEP are much like the benefits of TLS
>> too. For the vast majority of people they don?t notice anything different except
>> installing things is faster and more reliable. They don?t attribute that to the
>> PEP or this decision, they just internalize it as the new norm. However the
>> people who this does affect will seek out why it broke and raise an issue
>> citing that thing specifically. This creates a perception of lots of pain for no
>> gain when the reality is not that.
> 
> Donald: I don't think anyone is arguing that hosting packages on
> PyPI is a bad thing and PyPI as a service has gotten a lot better
> than it was a few years ago.

Didn?t mean to imply that I thought it was otherwise.

> 
> However, I find it troubling that we as Python developers are *forcing*
> the whole Python world to put their code into PyPI.

Forcing is a strong word. As of right now we?re "forcing" you to put it onto
PyPI if you want to install it without *any* flags to pip. You're more then
capable of hosting it externally if you wish, and pip will even tell the people
who try to install it what they need to enable in order to install it. It even
allows people to say "I don't care about any of this crap, just make all the
external stuff work".

Even if pip removed the external link handling all together and required you
to do something like:

    $ pip install --extra-index-url https://example.com/our-packages/ foo
    $ pip install --find-links https://example.com/our-packages/ foo

We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
discouraging people from not hosting on PyPI and providing incentives to doing
that.

> 
> There are plenty good reasons not to do this, and sometimes it's
> even impossible if you want to stay legal (see the PEP for details).

I re-read the reasons, I'm not sure I really agree with most (or all?) of them
however if people really want to do it, then there is nothing stopping them.
It's their choice and people on the *other* side of that who are installing
these packages also get to make a choice if they want to allow it or not.

> 
> Accordingly, we should respect those reasons make it possible for
> Python packages to live elsewhere, without having our tools put
> those packages into a bad light or making it harder for Python
> users to install such packages than needed.

I'm not sure what "put it into a bad light" is supposed to mean here. Is it
about the warning? I've already removed that completely.

As far as making it harder than it "needs" to be, that's a hard line to draw. I
personally know people for whom things that are hosted externally have caused
them a lot of pain to the point where they have to automatically spider PyPI
for their own dependencies every hour or so in order to isolate themselves from
the failure of random dependencies suddenly no longer being installable.

This breakage is going to happen for basically any project that installs their
dependencies with any frequency. One of the big projects that I'm aware of that
has had this problem is Openstack who, in their CI, installs things several
hundred times a day. If a service they depend on goes down that causes
significant disruption for them. Now they've solved it by the above solution
of hosting their own mirror of PyPI that includes spidering for externally
installable things. They have one or two packages left that don't host directly
on PyPI at which point they'll no longer need that afaik.

I don't think telling every downstream of us of any size that in order to get
reliable installs they are going to have to work around PyPI, and in some cases
even develop custom software in order to do so effectively, is a very user
friendly position.

> 
> With the checksum uploaded to PyPI, the only argument against
> fetching packages from other places on the Internet is reliability,
> not security.

I've never said differently. There are some minor privacy things in there
but primarily it's about reliability and that people should generally be aware
where their packages are coming from.

> 
> PyPI is not the only reliable file hosting system on the Internet,
> so this argument is rather weak.

It actually doesn't matter how reliable the other systems are. Reliability wise
the best possible outcome is the external host has 100% (extremely unlikely)
uptime and it has no affect on reliability. The realistic outcome is that
the external hosts will not have 100% uptime and will infact decrease the
total availability of the install sets on PyPI.

This isn't an opinion it is fact.

But in reality reliability also includes more things than just the server
temporarily going down. It also includes things like the maintainer has quit
maintaining that package, left Python all together, died even. There are a
decent number of packages on PyPI that people use which have no active
maintainer at all. If people rely on a package that isn't hosted on PyPI they
take a larger risk that their dependencies are going to become uninstallable
due to the above. When scanning the unverifiable links it is amazing how many
of those links are now 404s or 500s or even "nodename not founds". All of these
cause issues for people and are not hypothetical, I've dealt with people
complaining about one or another of them.

When things are hosted on PyPI then we (we being the infra team) are able to
ensure that the likelihood of ``pip install <something>`` actually works and
continues to work. When things are hosted externally the best we can do is
say sorry. At that point the only major thing outside of our control that can
cause someones installations to suddenly start failing is if the package
author willfully removes their things from PyPI. That is unfortunate but it
is also their right and there's nothing to be done for that. Thankfully that
case is far less common than the case of "an external site just isn't there".

I think it's important to point out that one of the driving factors that caused
me to finally push for changes and what lead to PEP438 being created was that
Mercurial's external hosted was being extremely flaky. I can't remember the
exact details but I want to say that over the span of a week or two I was
getting massive numbers of users complaining that ``pip install Mercurial``
was suddenly failing. This isn't to knock on the Mercurial folks or anything
but to simply point out that these problems aren't things that just happen to
(under|un)maintained software nor are they hypothetical. This PEP was born of
the frustration that was being relayed to me by end users of PyPI/pip.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/a7b69885/attachment.sig>

From p.f.moore at gmail.com  Fri May  9 13:55:57 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 9 May 2014 12:55:57 +0100
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
Message-ID: <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>

On 9 May 2014 12:44, Donald Stufft <donald at stufft.io> wrote:
> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
> discouraging people from not hosting on PyPI and providing incentives to doing
> that.

But you're doing so by inflicting pain on people using pip to install
those packages. Those users complain about *pip*, not about the
packages. Better to directly impact the package maintainers, rather
than their users (who are innocent victims). Better still of course to
encourage people to improve things, not to punish them for not doing
so.

> I think it's important to point out that one of the driving factors that caused
> me to finally push for changes and what lead to PEP438 being created was that
> Mercurial's external hosted was being extremely flaky. I can't remember the
> exact details but I want to say that over the span of a week or two I was
> getting massive numbers of users complaining that ``pip install Mercurial``
> was suddenly failing. This isn't to knock on the Mercurial folks or anything
> but to simply point out that these problems aren't things that just happen to
> (under|un)maintained software nor are they hypothetical. This PEP was born of
> the frustration that was being relayed to me by end users of PyPI/pip.

So now "pip install Mercurial" always fails? And adding a flag allows
it to work as well as before, but no better? How did that fix the
issue? Seriously - I'm missing something here.

Paul

From donald at stufft.io  Fri May  9 14:02:12 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 08:02:12 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <CACac1F-gZ3ZmJET+Us65CaRzf4GfXyjTZogBQRM+-CrhEaZZ1A@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <6CE44603-0213-4591-B63C-3B9B2782FB62@stufft.io>
 <CACac1F-gZ3ZmJET+Us65CaRzf4GfXyjTZogBQRM+-CrhEaZZ1A@mail.gmail.com>
Message-ID: <35017217-6F7A-455C-BF14-AEB58FB24955@stufft.io>


On May 9, 2014, at 5:01 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 05:34, Donald Stufft <donald at stufft.io> wrote:
>> On May 8, 2014, at 5:22 PM, Donald Stufft <donald at stufft.io> wrote:
>> 
>>>> Socially, this change does not seem to be having the effect of
>>>> persuading more package developers to host on PyPI. The stick doesn't
>>>> appear to have worked, maybe we should be trying to find a carrot?
>>> 
>>> Do you have any data to point to that says it hasn?t worked? Just to see
>>> what impact it has had, I?m running my scripts again that I ran a year
>>> ago to see what has changed, already I can see they are processing
>>> MUCH faster than last year.
>> 
>> The data has finished processing, it represents a time diff of approximately
>> one year. The pip release that caused all of this was released about 4-5 months
>> ago.
>> 
>> Overall PyPI has seen a 50% growth in installable projects in that time. If the
>> change would have had no effect we'd expect to see a ~50% increase across the
>> board. However what we've seen is a a 60% (+10% of expected) increase in
>> projects that can only be installed from PyPI and a 12% decrease in projects
>> that have any unsafe files (-62% of expected).
> 
> Donald,
> Thanks for taking the time to get those figures. It does appear that
> there are less cases that would be affected than the number of
> complaints would imply.

Of course, I don?t like making claims without backing them up if I can :)

> 
> The only concern I have about this type of analysis is that it doesn't
> "weight" projects. It may be (and again, I have no data to back this
> up) that the projects that are affected detrimentally by this change
> are unusually popular or otherwise significant. There's obviously no
> way to assess this sensibly other than by making a judgement on the
> level of complaints.

Yea, I don?t have a good way to weight those projects in any way. Normally
I could get some sort of estimate by looking at the download numbers from
PyPI but well ;)

For the record, here?s the list of projects that are hosted *only* safely externally
or that have *any* safely externally hosted files:

https://gist.github.com/dstufft/1b16c305f97fff6cef2f

Most of these don?t stand out to me at all. The only ones that do are:

* pyOpenSSL which has one older release that is hosted that way
* argparse which has the latest release hosted this way but has
  older releases hosted on PyPI
* new relic which only hosts older releases externally
* beautifulsoup4 which hosts things safely externally *and* on PyPI
* Paste which has one ?external? thing which is actually only external
  because it used a cheeseshop.python.org link instead of a pypi.python.org
  link.
* ipython which has one older release hosted safely externally but the
  latest is on PyPI
* netifaces which has one older release hosted safely externally but the
  rest are on PyPI

> 
> But arguing numbers was never my intention here, so let's just say
> that I concede that the change has had a positive effect, which is
> great.
> Paul

I didn?t mean to try to imply that it was :) I just wanted to make sure that
*my* claims were true, or if they weren?t I wanted to be able to say that
I was wrong. Since I had the numbers computed already it didn?t make
any sense not to share them here.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/ba0535d3/attachment.sig>

From ncoghlan at gmail.com  Fri May  9 14:02:53 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 9 May 2014 22:02:53 +1000
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
Message-ID: <CADiSq7c=yWRqULfQBkrb5VNwt9NMFASRG+=VeJp=nfEJGR7teQ@mail.gmail.com>

On 9 May 2014 21:55, Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 May 2014 12:44, Donald Stufft <donald at stufft.io> wrote:
>> I think it's important to point out that one of the driving factors that caused
>> me to finally push for changes and what lead to PEP438 being created was that
>> Mercurial's external hosted was being extremely flaky. I can't remember the
>> exact details but I want to say that over the span of a week or two I was
>> getting massive numbers of users complaining that ``pip install Mercurial``
>> was suddenly failing. This isn't to knock on the Mercurial folks or anything
>> but to simply point out that these problems aren't things that just happen to
>> (under|un)maintained software nor are they hypothetical. This PEP was born of
>> the frustration that was being relayed to me by end users of PyPI/pip.
>
> So now "pip install Mercurial" always fails? And adding a flag allows
> it to work as well as before, but no better? How did that fix the
> issue? Seriously - I'm missing something here.

Mercurial downloads are now available directly from PyPI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From donald at stufft.io  Fri May  9 14:06:19 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 08:06:19 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
Message-ID: <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>


On May 9, 2014, at 7:55 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 12:44, Donald Stufft <donald at stufft.io> wrote:
>> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
>> discouraging people from not hosting on PyPI and providing incentives to doing
>> that.
> 
> But you're doing so by inflicting pain on people using pip to install
> those packages. Those users complain about *pip*, not about the
> packages. Better to directly impact the package maintainers, rather
> than their users (who are innocent victims). Better still of course to
> encourage people to improve things, not to punish them for not doing
> so.

We can?t directly impact the package maintainers and the vast bulk of people
who have had a problem who have complained about it to pip also need to
add the ?allow-unverifiable flag and would not simply be able to be fixed
by allowing safely externally hosted files.

Looking at the numbers and what packages are hosted externally, allowing
safely externally hosted files would have practically no benefit to pip?s end users.
The only case that I can see with a quick scan would be it would allow the latest
version of argparse.

TBH I think the biggest source of confusion reduction would be to remove the
?safely externally hosted? category all together and just make it hosted on
PyPI -> Install by default, hosted off PyPI -> requires a per package flag. However
I?m sure the vocal minority would have a problem with that.

> 
>> I think it's important to point out that one of the driving factors that caused
>> me to finally push for changes and what lead to PEP438 being created was that
>> Mercurial's external hosted was being extremely flaky. I can't remember the
>> exact details but I want to say that over the span of a week or two I was
>> getting massive numbers of users complaining that ``pip install Mercurial``
>> was suddenly failing. This isn't to knock on the Mercurial folks or anything
>> but to simply point out that these problems aren't things that just happen to
>> (under|un)maintained software nor are they hypothetical. This PEP was born of
>> the frustration that was being relayed to me by end users of PyPI/pip.
> 
> So now "pip install Mercurial" always fails? And adding a flag allows
> it to work as well as before, but no better? How did that fix the
> issue? Seriously - I'm missing something here.

No, This caused Mercurial to upload their packages to PyPI.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/52edd213/attachment.sig>

From p.f.moore at gmail.com  Fri May  9 14:21:14 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 9 May 2014 13:21:14 +0100
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
 <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
Message-ID: <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>

On 9 May 2014 13:06, Donald Stufft <donald at stufft.io> wrote:
>>> I think it's important to point out that one of the driving factors that caused
>>> me to finally push for changes and what lead to PEP438 being created was that
>>> Mercurial's external hosted was being extremely flaky. I can't remember the
>>> exact details but I want to say that over the span of a week or two I was
>>> getting massive numbers of users complaining that ``pip install Mercurial``
>>> was suddenly failing. This isn't to knock on the Mercurial folks or anything
>>> but to simply point out that these problems aren't things that just happen to
>>> (under|un)maintained software nor are they hypothetical. This PEP was born of
>>> the frustration that was being relayed to me by end users of PyPI/pip.
>>
>> So now "pip install Mercurial" always fails? And adding a flag allows
>> it to work as well as before, but no better? How did that fix the
>> issue? Seriously - I'm missing something here.
>
> No, This caused Mercurial to upload their packages to PyPI.

You're claiming that Mercurial moved to hosting on PyPI solely because
users suddenly needed to add a flag to install from pip? As opposed to
because PyPI gave them a more reliable hosting platform, for example?
OK. I certainly can't give any evidence to dispute that claim,
although I'm surprised.

Paul

From donald at stufft.io  Fri May  9 14:25:34 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 08:25:34 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
 <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
 <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
Message-ID: <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io>


On May 9, 2014, at 8:21 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 13:06, Donald Stufft <donald at stufft.io> wrote:
>>>> I think it's important to point out that one of the driving factors that caused
>>>> me to finally push for changes and what lead to PEP438 being created was that
>>>> Mercurial's external hosted was being extremely flaky. I can't remember the
>>>> exact details but I want to say that over the span of a week or two I was
>>>> getting massive numbers of users complaining that ``pip install Mercurial``
>>>> was suddenly failing. This isn't to knock on the Mercurial folks or anything
>>>> but to simply point out that these problems aren't things that just happen to
>>>> (under|un)maintained software nor are they hypothetical. This PEP was born of
>>>> the frustration that was being relayed to me by end users of PyPI/pip.
>>> 
>>> So now "pip install Mercurial" always fails? And adding a flag allows
>>> it to work as well as before, but no better? How did that fix the
>>> issue? Seriously - I'm missing something here.
>> 
>> No, This caused Mercurial to upload their packages to PyPI.
> 
> You're claiming that Mercurial moved to hosting on PyPI solely because
> users suddenly needed to add a flag to install from pip? As opposed to
> because PyPI gave them a more reliable hosting platform, for example?
> OK. I certainly can't give any evidence to dispute that claim,
> although I'm surprised.
> 
> Paul

I don?t know that for a fact but If my memory is correct that?s a reasonable
assumption based on the timeline.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/cf5450af/attachment-0001.sig>

From stefan at bytereef.org  Fri May  9 14:39:03 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 9 May 2014 14:39:03 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
References: <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
 <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
 <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
Message-ID: <20140509123903.GA7377@sleipnir.bytereef.org>

Paul Moore <p.f.moore at gmail.com> wrote:
> You're claiming that Mercurial moved to hosting on PyPI solely because
> users suddenly needed to add a flag to install from pip? As opposed to
> because PyPI gave them a more reliable hosting platform, for example?
> OK. I certainly can't give any evidence to dispute that claim,
> although I'm surprised.

That is my understanding of the posted statistics, too.  PyPI was overloaded,
the improved infrastructure fixed that, so more projects now host on PyPI.



Stefan Krah



From p.f.moore at gmail.com  Fri May  9 14:47:49 2014
From: p.f.moore at gmail.com (Paul Moore)
Date: Fri, 9 May 2014 13:47:49 +0100
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
 <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
 <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
 <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io>
Message-ID: <CACac1F90un45Sv16GjK8gFJyB8BuC6YUbpb4GR_05NJShWCdyg@mail.gmail.com>

On 9 May 2014 13:25, Donald Stufft <donald at stufft.io> wrote:
>> You're claiming that Mercurial moved to hosting on PyPI solely because
>> users suddenly needed to add a flag to install from pip? As opposed to
>> because PyPI gave them a more reliable hosting platform, for example?
>> OK. I certainly can't give any evidence to dispute that claim,
>> although I'm surprised.
>
> I don?t know that for a fact but If my memory is correct that?s a reasonable
> assumption based on the timeline.

As Stefan pointed out, improved PyPI infrastructure such as the CDN
seems like a much more likely reason for Mercurial to have switched.
Paul

From solipsis at pitrou.net  Fri May  9 14:59:21 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 9 May 2014 14:59:21 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <CACac1F_iOqH5PYi1v8NuYux+QcrFq50nggKY-b2sye7eD+Qf=g@mail.gmail.com>
 <7F6C1802-12DB-4B7F-9A5D-C81B18BD7CC1@stufft.io>
 <CACac1F9YwrDmJQng1-7zqrg3VkaByCgVb1tiioCfem6o1e2RqA@mail.gmail.com>
 <3B66A600-8073-4EB9-B25E-56665C1DDA64@stufft.io>
 <CACac1F90un45Sv16GjK8gFJyB8BuC6YUbpb4GR_05NJShWCdyg@mail.gmail.com>
Message-ID: <20140509145921.5fa6cee5@fsol>

On Fri, 9 May 2014 13:47:49 +0100
Paul Moore <p.f.moore at gmail.com> wrote:
> On 9 May 2014 13:25, Donald Stufft <donald at stufft.io> wrote:
> >> You're claiming that Mercurial moved to hosting on PyPI solely because
> >> users suddenly needed to add a flag to install from pip? As opposed to
> >> because PyPI gave them a more reliable hosting platform, for example?
> >> OK. I certainly can't give any evidence to dispute that claim,
> >> although I'm surprised.
> >
> > I don?t know that for a fact but If my memory is correct that?s a reasonable
> > assumption based on the timeline.
> 
> As Stefan pointed out, improved PyPI infrastructure such as the CDN
> seems like a much more likely reason for Mercurial to have switched.

Here is the only reference I could find:
http://www.selenic.com/pipermail/mercurial-devel/2013-July/052129.html

Regards

Antoine.



From ethan at stoneleaf.us  Fri May  9 15:09:28 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 09 May 2014 06:09:28 -0700
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
Message-ID: <536CD388.2040708@stoneleaf.us>

On 05/08/2014 02:02 PM, Paul Moore wrote:
>
> Socially, this change does not seem to be having the effect of
> persuading more package developers to host on PyPI. The stick doesn't
> appear to have worked, maybe we should be trying to find a carrot? Or
> maybe we have to accept that some developers have sound reasons for
> not hosting on PyPI and work with them to find an acceptable
> compromise? Has anyone checked what Stefan's reasons are for not
> hosting cdecimal on PyPI? Do they represent a use case that the PEP
> hasn't considered?

Well, I do host a small handful of modules on PyPI, but I can say that some of my pain points are:

   - getting a good name: the obvious ones are taken, so the search
     begins to find a name that is not taken and yet still feels at
     least somewhat appropriate:

       my OO path module ended up being called strpath (dumb name);
       eventually changed to antipathy

       my script param line parser is called scription (okay name)

       my enum backport is called enum34 (blah)

       my dbf package is called dbf (lucky lucky lucky!)

   - reputation of PyPI is not that great: anybody can upload packages,
     no curating is done, once there the name is taken for all time,
     no standard version numbering, . . . (granted, these are my
     impressions and may not be entirely valid)

Anyway, I know it's a volunteer effort I don't want to criticize those actually running it, but these are some of the 
problems that I see with the service itself.

--
~Ethan~

From mal at egenix.com  Fri May  9 15:58:39 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 09 May 2014 15:58:39 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>	<536B97E2.8070805@egenix.com>	<536BA4D5.7080208@egenix.com>	<70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>	<CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>	<D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>	<536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
Message-ID: <536CDF0F.6050407@egenix.com>

On 09.05.2014 13:44, Donald Stufft wrote:
> 
> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Donald: I don't think anyone is arguing that hosting packages on
>> PyPI is a bad thing and PyPI as a service has gotten a lot better
>> than it was a few years ago.
> 
> Didn?t mean to imply that I thought it was otherwise.
> 
>>
>> However, I find it troubling that we as Python developers are *forcing*
>> the whole Python world to put their code into PyPI.
> 
> Forcing is a strong word. As of right now we?re "forcing" you to put it onto
> PyPI if you want to install it without *any* flags to pip. 

Which is what most people expect to be able to do.

> You're more then
> capable of hosting it externally if you wish, and pip will even tell the people
> who try to install it what they need to enable in order to install it. It even
> allows people to say "I don't care about any of this crap, just make all the
> external stuff work".
> 
> Even if pip removed the external link handling all together and required you
> to do something like:
> 
>     $ pip install --extra-index-url https://example.com/our-packages/ foo
>     $ pip install --find-links https://example.com/our-packages/ foo
> 
> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
> discouraging people from not hosting on PyPI and providing incentives to doing
> that.

This is all true, but in reality, you are making externally hosted
packages second class citizens in Python land by requiring
extra options even for package listings that are perfectly safe
to download from other servers.

As mentioned before: I can understand that you want to make downloads
safe for users, but if a package is hosted on some other reliable
servers and pip can check that it's a valid package, there's little
to argue for not allowing such downloads.

>> There are plenty good reasons not to do this, and sometimes it's
>> even impossible if you want to stay legal (see the PEP for details).
> 
> I re-read the reasons, I'm not sure I really agree with most (or all?) of them
> however if people really want to do it, then there is nothing stopping them.
> It's their choice and people on the *other* side of that who are installing
> these packages also get to make a choice if they want to allow it or not.

You don't have to agree with those reasons. Fact is, they exist,
prevent package authors from uploading to PyPI and we as
Python developers should respect those reasons.

With the new infrastructure, it's far more attractive to
host packages on PyPI than it was before, so people who do
not host on PyPI will have carefully thought about this.

>> Accordingly, we should respect those reasons make it possible for
>> Python packages to live elsewhere, without having our tools put
>> those packages into a bad light or making it harder for Python
>> users to install such packages than needed.
> 
> I'm not sure what "put it into a bad light" is supposed to mean here. Is it
> about the warning? I've already removed that completely.

It's using the same unfortunate strategy that setuptools used by
requiring an option called --single-version-externally-managed
to be able to install a package without all the .pth and egg
logic (an option that pip now uses per default, BTW ;-)).

I snipped the rest of the discussion and reliability, using
unmaintained packages and projects using their own mirrors (which
should really be the standard, not an exceptional case),
because it's not really leading anywhere:

At the time we discussed the PEP, security was the main concern,
not reliability.

Now that there is a secure way to reference external distribution
files, I think we should revisit the defaults decision in the
light of the new possibilities.


BTW: The PEP mentions re-hosting tools to upload their packages
to PyPI and also says "This re-hosting tool MUST be available
before automated hosting-mode changes are announced to package
maintainers."  I am not aware of such tools.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 09 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From phd at phdru.name  Fri May  9 16:08:55 2014
From: phd at phdru.name (Oleg Broytman)
Date: Fri, 9 May 2014 16:08:55 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <536CD388.2040708@stoneleaf.us>
References: <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <536CD388.2040708@stoneleaf.us>
Message-ID: <20140509140855.GA1076@phdru.name>

On Fri, May 09, 2014 at 06:09:28AM -0700, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 05/08/2014 02:02 PM, Paul Moore wrote:
> Well, I do host a small handful of modules on PyPI, but I can say that some of my pain points are:
> 
>   - getting a good name: the obvious ones are taken, so the search
>     begins to find a name that is not taken and yet still feels at
>     least somewhat appropriate:
> 
>       my OO path module ended up being called strpath (dumb name);
>       eventually changed to antipathy
> 
>       my script param line parser is called scription (okay name)
> 
>       my enum backport is called enum34 (blah)
> 
>       my dbf package is called dbf (lucky lucky lucky!)

   For many reasons I avoid github/gitorious/bitbucket, but there is one
thing they do right -- two-levels namespaces.

   "Namespaces are one honking great idea -- let's do more of those!"

Oleg.
-- 
     Oleg Broytman            http://phdru.name/            phd at phdru.name
           Programmers don't die, they just GOSUB without RETURN.

From donald at stufft.io  Fri May  9 17:39:02 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 11:39:02 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <536CDF0F.6050407@egenix.com>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>	<536B97E2.8070805@egenix.com>	<536BA4D5.7080208@egenix.com>	<70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>	<CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>	<D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>	<536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
Message-ID: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>


On May 9, 2014, at 9:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:

> On 09.05.2014 13:44, Donald Stufft wrote:
>> 
>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> Donald: I don't think anyone is arguing that hosting packages on
>>> PyPI is a bad thing and PyPI as a service has gotten a lot better
>>> than it was a few years ago.
>> 
>> Didn?t mean to imply that I thought it was otherwise.
>> 
>>> 
>>> However, I find it troubling that we as Python developers are *forcing*
>>> the whole Python world to put their code into PyPI.
>> 
>> Forcing is a strong word. As of right now we?re "forcing" you to put it onto
>> PyPI if you want to install it without *any* flags to pip. 
> 
> Which is what most people expect to be able to do.

Sure, but sometimes it?s better to make an informed choice about things being
installed instead of having it be installed by default and sometimes it?s better
to disallow doing something at all.

Further most people don?t expect an install to touch any server host other
than PyPI. As far as I?m aware none of the other package repositories
feature this, and even with the fact we support it barely anyone even cares to
utilize it.

> 
>> You're more then
>> capable of hosting it externally if you wish, and pip will even tell the people
>> who try to install it what they need to enable in order to install it. It even
>> allows people to say "I don't care about any of this crap, just make all the
>> external stuff work".
>> 
>> Even if pip removed the external link handling all together and required you
>> to do something like:
>> 
>>    $ pip install --extra-index-url https://example.com/our-packages/ foo
>>    $ pip install --find-links https://example.com/our-packages/ foo
>> 
>> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
>> discouraging people from not hosting on PyPI and providing incentives to doing
>> that.
> 
> This is all true, but in reality, you are making externally hosted
> packages second class citizens in Python land by requiring
> extra options even for package listings that are perfectly safe
> to download from other servers.
> 
> As mentioned before: I can understand that you want to make downloads
> safe for users, but if a package is hosted on some other reliable
> servers and pip can check that it's a valid package, there's little
> to argue for not allowing such downloads.

Except the fact that the only people I?ve ever seen *happy* that people can
host packages externally are a handful (<5) of people (tbh primarily you and
Stefan) and the feedback I get from every other person around the web
in unequivocally yes please get rid of externally hosted files, they?ve done
nothing but cause me pain.

It?s not reasonable to allow a minority of users to negatively impact the majority.

> 
>>> There are plenty good reasons not to do this, and sometimes it's
>>> even impossible if you want to stay legal (see the PEP for details).
>> 
>> I re-read the reasons, I'm not sure I really agree with most (or all?) of them
>> however if people really want to do it, then there is nothing stopping them.
>> It's their choice and people on the *other* side of that who are installing
>> these packages also get to make a choice if they want to allow it or not.
> 
> You don't have to agree with those reasons. Fact is, they exist,
> prevent package authors from uploading to PyPI and we as
> Python developers should respect those reasons.

No I disagree that they are reasons at all. Half of the ones you enumerated are
nonsensical or irrelevant, the other half can be fixed by adding features to or
fixing PyPI. One or two read like reasons why someone might actually decide
not to upload something to PyPI but which that reason isn?t really all that
reasonable and finally a grand total of one or two of them look like an actual legit
reasons and it only applies to Crypto software.

I mean your reasoning included things like:

> PyPI doesn?t let you upload two files with the same name, you gave the
> example of UC2 vs UC4

  The thing being if PyPI did allow you to do that
  then we?d have no way to determine which one is the right one and half
  of the people would just get randomly failing installs because we guessed
  the wrong one.

> They don?t know it?s possible to upload to PyPI

I don?t even know how to response to this one.

> PyPI used to be unreliable but isn?t now and they don?t know about it

Nor this one.

> Not wanting people to download a package at all and instead check out
> from a VCS repo

This?ll randomly fail for people depending on if they happen to have that VCS
installed. Also I don?t see any evidence that people are actually doing this
outside of supporting a foo==dev install which pip won?t install by default either.

> Not wanting to add latency for yourself for downloading from PyPI

Instead you?re going to inflict extra latency on everyone else when you could just
run a mirror or upload the packages to your own server.

> File types that PyPI doesn?t support and which utilize a third party package
> format, PyPM being a given example. 

Seriously? Should an apt repo support external links because you can?t install RPMs
with apt?

> Wanting people to have to read something before installing the package
> because it relies on some additional setup.

This is completely nonsensical if they want people to have to read something before
they attempt to pip install it then they don?t want pip to discover the external file at all
then.

> It currently takes too long uploading that many files to
> PyPI. This causes a problem, since in order to start the upload,
> we have to register the release on PyPI, which tools will then
> immediately find. However, during the upload time, they won't
> necessarily find the right files to download and then fail.

So if this is an actual problem file a bug and we can make it so PyPI lets you upload
things as ?drafts? and then you can publish them all at once or some such thing.

> having a strong need to know the number of downloads per
> package and associated statistics such as downloads per
> country, per year/month/day/hour

If additional statistics are desired for PyPI file a bug and we can add them.

> not wanting to give up access to the download log files

This appears to be repeating the last reason with different wording.

> the PyPI uploads not being compatible to their release process

I?m not sure how it?s possible to have a release process where it isn?t
compatible to upload to an additional server. Maybe ones that don?t
currently do it but ones that are fundamentally incompatible?


Almost all of your ?reasons? read like someone who maybe had one corner
case where it might have made sense but didn?t want to admit that it was
a corner case so tried to think up any random reason they could think of
in order to make the list look longer and more impressive than it actually was.

> 
> With the new infrastructure, it's far more attractive to
> host packages on PyPI than it was before, so people who do
> not host on PyPI will have carefully thought about this.
> 
>>> Accordingly, we should respect those reasons make it possible for
>>> Python packages to live elsewhere, without having our tools put
>>> those packages into a bad light or making it harder for Python
>>> users to install such packages than needed.
>> 
>> I'm not sure what "put it into a bad light" is supposed to mean here. Is it
>> about the warning? I've already removed that completely.
> 
> It's using the same unfortunate strategy that setuptools used by
> requiring an option called --single-version-externally-managed
> to be able to install a package without all the .pth and egg
> logic (an option that pip now uses per default, BTW ;-)).
> 
> I snipped the rest of the discussion and reliability, using
> unmaintained packages and projects using their own mirrors (which
> should really be the standard, not an exceptional case),
> because it's not really leading anywhere:

Using your own mirror shouldn?t be the standard if all you?re doing
is automatically updating that mirror. It?s a hack to get around
unreliability and it should be seen of as a sign of a failure to provide
a service that people can rely on and that?s how I see it. People
depend on this service and it?s irresponsible to not treat it as a
critical piece of infrastructure.

> 
> At the time we discussed the PEP, security was the main concern,
> not reliability.

The main concern, not the only concern.

> 
> Now that there is a secure way to reference external distribution
> files, I think we should revisit the defaults decision in the
> light of the new possibilities.

Re-evaluate all you want. If they change it will be over my strenuous
objections. The feedback I get from users is overwhelmly positive.
Even the ones who get struck by needing to add extra flags tell me
how they are glad that pip and PyPI are moving towards a more
reliable model.

> 
> 
> BTW: The PEP mentions re-hosting tools to upload their packages
> to PyPI and also says "This re-hosting tool MUST be available
> before automated hosting-mode changes are announced to package
> maintainers."  I am not aware of such tools.

It appears this got missed. I?ll add it to my stack to remedy this.

> 
> --
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, May 09 2014)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/9c21c74f/attachment.sig>

From status at bugs.python.org  Fri May  9 18:07:57 2014
From: status at bugs.python.org (Python tracker)
Date: Fri,  9 May 2014 18:07:57 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20140509160757.F39D6568E0@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2014-05-02 - 2014-05-09)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4611 ( +6)
  closed 28624 (+42)
  total  33235 (+48)

Open issues with patches: 2119 


Issues opened (35)
==================

#18604: Consolidate gui available checks in test.support
http://bugs.python.org/issue18604  reopened by ned.deily

#21416: argparse should accept bytes arguments as originally passed
http://bugs.python.org/issue21416  opened by underrun

#21417: Compression level for zipfile
http://bugs.python.org/issue21417  opened by Sworddragon

#21418: Segv during call to super_init in application embedding Python
http://bugs.python.org/issue21418  opened by snoeberger

#21419: Use calloc() instead of malloc() for int << int (lshift)
http://bugs.python.org/issue21419  opened by haypo

#21420: Optimize 2 ** n: implement it as 1 << n
http://bugs.python.org/issue21420  opened by haypo

#21422: int << 0: return the number unmodified
http://bugs.python.org/issue21422  opened by haypo

#21423: concurrent.futures.ThreadPoolExecutor should accept an initial
http://bugs.python.org/issue21423  opened by andreasvc

#21424: Simplify and speed-up heaqp.nlargest()
http://bugs.python.org/issue21424  opened by rhettinger

#21425: Python 3 pipe handling breaks python mode in emacs on Windows
http://bugs.python.org/issue21425  opened by marczellm

#21427: installer not working
http://bugs.python.org/issue21427  opened by ellipso

#21428: Python suddenly cares about EOLs formats on windows
http://bugs.python.org/issue21428  opened by pygnol

#21429: Input.output error with multiprocessing
http://bugs.python.org/issue21429  opened by mikaela

#21430: Document ssl.pending()
http://bugs.python.org/issue21430  opened by shevek

#21431: 3.4.1rc1 test_pydoc fails: pydoc_data.topics.topics values are
http://bugs.python.org/issue21431  opened by ned.deily

#21434: python -3 documentation is outdated
http://bugs.python.org/issue21434  opened by berker.peksag

#21436: Consider leaving importlib.abc.Loader.load_module()
http://bugs.python.org/issue21436  opened by srittau

#21437: document that asyncio.ProactorEventLoop doesn't support SSL
http://bugs.python.org/issue21437  opened by akira

#21439: Numerous minor issues in Language Reference
http://bugs.python.org/issue21439  opened by zach.ware

#21441: Buffer Protocol Documentation Error
http://bugs.python.org/issue21441  opened by Jake.Vanderplas

#21443: asyncio logging documentation clarifications
http://bugs.python.org/issue21443  opened by hardkrash

#21445: Some asserts in test_filecmp have the wrong messages
http://bugs.python.org/issue21445  opened by Steven.Barker

#21446: Update reload fixer to use importlib instead of imp
http://bugs.python.org/issue21446  opened by berker.peksag

#21447: Intermittent asyncio.open_connection / futures.InvalidStateErr
http://bugs.python.org/issue21447  opened by ryder.lewis

#21448: Email Parser use 100% CPU
http://bugs.python.org/issue21448  opened by jader.fabiano

#21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId
http://bugs.python.org/issue21449  opened by josh.rosenberg

#21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat
http://bugs.python.org/issue21452  opened by mloskot

#21453: Support of RPM subpackages in distutils
http://bugs.python.org/issue21453  opened by vitalyisaev2

#21454: asyncio's loop.connect_read_pipe makes pipes non-blocking cont
http://bugs.python.org/issue21454  opened by John Isidore

#21455: add default backlog to socket.listen()
http://bugs.python.org/issue21455  opened by neologix

#21456: skip 2 tests in test_urllib2net.py if _ssl module not present
http://bugs.python.org/issue21456  opened by rpointel

#21457: NetBSD curses support improvements
http://bugs.python.org/issue21457  opened by wiz

#21461: Recognize -pthread
http://bugs.python.org/issue21461  opened by wiz

#21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds
http://bugs.python.org/issue21462  opened by ncoghlan

#21463: RuntimeError when URLopener.ftpcache is full
http://bugs.python.org/issue21463  opened by erik.bray



Most recent 15 issues with no replies (15)
==========================================

#21463: RuntimeError when URLopener.ftpcache is full
http://bugs.python.org/issue21463

#21461: Recognize -pthread
http://bugs.python.org/issue21461

#21453: Support of RPM subpackages in distutils
http://bugs.python.org/issue21453

#21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId
http://bugs.python.org/issue21449

#21445: Some asserts in test_filecmp have the wrong messages
http://bugs.python.org/issue21445

#21443: asyncio logging documentation clarifications
http://bugs.python.org/issue21443

#21441: Buffer Protocol Documentation Error
http://bugs.python.org/issue21441

#21439: Numerous minor issues in Language Reference
http://bugs.python.org/issue21439

#21437: document that asyncio.ProactorEventLoop doesn't support SSL
http://bugs.python.org/issue21437

#21434: python -3 documentation is outdated
http://bugs.python.org/issue21434

#21429: Input.output error with multiprocessing
http://bugs.python.org/issue21429

#21425: Python 3 pipe handling breaks python mode in emacs on Windows
http://bugs.python.org/issue21425

#21424: Simplify and speed-up heaqp.nlargest()
http://bugs.python.org/issue21424

#21418: Segv during call to super_init in application embedding Python
http://bugs.python.org/issue21418

#21417: Compression level for zipfile
http://bugs.python.org/issue21417



Most recent 15 issues waiting for review (15)
=============================================

#21463: RuntimeError when URLopener.ftpcache is full
http://bugs.python.org/issue21463

#21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds
http://bugs.python.org/issue21462

#21461: Recognize -pthread
http://bugs.python.org/issue21461

#21457: NetBSD curses support improvements
http://bugs.python.org/issue21457

#21456: skip 2 tests in test_urllib2net.py if _ssl module not present
http://bugs.python.org/issue21456

#21455: add default backlog to socket.listen()
http://bugs.python.org/issue21455

#21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat
http://bugs.python.org/issue21452

#21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId
http://bugs.python.org/issue21449

#21446: Update reload fixer to use importlib instead of imp
http://bugs.python.org/issue21446

#21445: Some asserts in test_filecmp have the wrong messages
http://bugs.python.org/issue21445

#21441: Buffer Protocol Documentation Error
http://bugs.python.org/issue21441

#21434: python -3 documentation is outdated
http://bugs.python.org/issue21434

#21423: concurrent.futures.ThreadPoolExecutor should accept an initial
http://bugs.python.org/issue21423

#21422: int << 0: return the number unmodified
http://bugs.python.org/issue21422

#21420: Optimize 2 ** n: implement it as 1 << n
http://bugs.python.org/issue21420



Top 10 most discussed issues (10)
=================================

#21233: Add *Calloc functions to CPython memory allocation API
http://bugs.python.org/issue21233  15 msgs

#21121: -Werror=declaration-after-statement is added even for extensio
http://bugs.python.org/issue21121  14 msgs

#21419: Use calloc() instead of malloc() for int << int (lshift)
http://bugs.python.org/issue21419  10 msgs

#21448: Email Parser use 100% CPU
http://bugs.python.org/issue21448   8 msgs

#20866: segfailt with os.popen and SIGPIPE
http://bugs.python.org/issue20866   6 msgs

#21083: Add get_content_disposition() to email.message.Message
http://bugs.python.org/issue21083   6 msgs

#21420: Optimize 2 ** n: implement it as 1 << n
http://bugs.python.org/issue21420   6 msgs

#21422: int << 0: return the number unmodified
http://bugs.python.org/issue21422   6 msgs

#21423: concurrent.futures.ThreadPoolExecutor should accept an initial
http://bugs.python.org/issue21423   6 msgs

#10752: build_ssl.py is relying on unreliable behaviour of os.popen
http://bugs.python.org/issue10752   5 msgs



Issues closed (42)
==================

#5717: os.defpath includes unix /bin on windows
http://bugs.python.org/issue5717  closed by tim.golden

#13030: Be more generic when identifying the Windows main dir in insta
http://bugs.python.org/issue13030  closed by tim.golden

#17338: Add length_hint parameter to list, dict,	set constructors to a
http://bugs.python.org/issue17338  closed by rhettinger

#17752: many distutils tests fail when run from the installed location
http://bugs.python.org/issue17752  closed by doko

#18154: make install failed on solaris
http://bugs.python.org/issue18154  closed by palm.kevin

#18211: -Werror=statement-after-declaration problem
http://bugs.python.org/issue18211  closed by eric.araujo

#18314: Have os.unlink remove junction points
http://bugs.python.org/issue18314  closed by tim.golden

#19414: iter(ordered_dict) yields keys not in dict in some circumstanc
http://bugs.python.org/issue19414  closed by rhettinger

#19643: shutil rmtree fails on readonly files in Windows
http://bugs.python.org/issue19643  closed by tim.golden

#20384: os.open() exception doesn't contain file name on Windows
http://bugs.python.org/issue20384  closed by tim.golden

#20642: Enhance deepcopy-ing for tuples
http://bugs.python.org/issue20642  closed by python-dev

#20737: 3.3 _thread lock.acquire() timeout and threading.Event().wait(
http://bugs.python.org/issue20737  closed by kristjan.jonsson

#20758: mimetypes initialization order
http://bugs.python.org/issue20758  closed by tim.golden

#21101: Extend the PyDict C API to handle cases where the hash value i
http://bugs.python.org/issue21101  closed by rhettinger

#21132: Failure to import win32api
http://bugs.python.org/issue21132  closed by woakesd

#21141: Don't mention Perl in Windows build output
http://bugs.python.org/issue21141  closed by zach.ware

#21157: Update imp docs for a PEP 451 world
http://bugs.python.org/issue21157  closed by brett.cannon

#21300: Docs (incorrectly) suggest email.policy.default is the default
http://bugs.python.org/issue21300  closed by r.david.murray

#21335: Update importlib.__init__ to reset _frozen_importlib's loader 
http://bugs.python.org/issue21335  closed by brett.cannon

#21350: bug in file.writelines accepting buffers
http://bugs.python.org/issue21350  closed by pitrou

#21357: Increase filecmp test coverage from 63% to 76%
http://bugs.python.org/issue21357  closed by python-dev

#21366: Document that return in finally overwrites prev value
http://bugs.python.org/issue21366  closed by zach.ware

#21375: Fix and test overflow behavior in the C version of heapq
http://bugs.python.org/issue21375  closed by rhettinger

#21392: Python on Cygwin: subprocess.call BlockingIOError: [Errno 11] 
http://bugs.python.org/issue21392  closed by dellair.jie

#21393: Python/random.c: close hCryptProv at exit
http://bugs.python.org/issue21393  closed by haypo

#21396: Python io implementation doesn't flush with write_through=True
http://bugs.python.org/issue21396  closed by pitrou

#21405: Allow using symbols from Unicode block "Superscripts and Subsc
http://bugs.python.org/issue21405  closed by rominf

#21414: Add an intersperse function to itertools
http://bugs.python.org/issue21414  closed by rhettinger

#21421: ABCs for MappingViews should declare __slots__ so subclasses a
http://bugs.python.org/issue21421  closed by rhettinger

#21426: Invisible characters in email related souce files.
http://bugs.python.org/issue21426  closed by benjamin.peterson

#21432: samba.git from git://git.samba.org/samba.git  samba.netcmd.mai
http://bugs.python.org/issue21432  closed by ned.deily

#21433: "False = True" produces segfault
http://bugs.python.org/issue21433  closed by ned.deily

#21435: Segfault in gc with cyclic trash
http://bugs.python.org/issue21435  closed by tim.peters

#21438: Document which importlib.machinery loaders don't require an ar
http://bugs.python.org/issue21438  closed by brett.cannon

#21440: Use support.rmtree in test_zipfile
http://bugs.python.org/issue21440  closed by tim.golden

#21442: Win32 compiler warning in PyBytes_Concat
http://bugs.python.org/issue21442  closed by zach.ware

#21444: __len__ can't return big numbers
http://bugs.python.org/issue21444  closed by benjamin.peterson

#21450: [Issue 13630] IDLE: Find(ed) text is not highlighted while dia
http://bugs.python.org/issue21450  closed by berker.peksag

#21451: Improve error messages for malformed JSON
http://bugs.python.org/issue21451  closed by r.david.murray

#21458: MirBSD support
http://bugs.python.org/issue21458  closed by brett.cannon

#21459: DragonFlyBSD support
http://bugs.python.org/issue21459  closed by brett.cannon

#21460: distutils: use LDFLAGS
http://bugs.python.org/issue21460  closed by wiz

From stefan at bytereef.org  Fri May  9 18:11:42 2014
From: stefan at bytereef.org (Stefan Krah)
Date: Fri, 9 May 2014 18:11:42 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
Message-ID: <20140509161142.GA8440@sleipnir.bytereef.org>

Donald, I'm out of his discussion.  I have one last request: please don't
gossip about core devs in public as long as you have commit privs:

https://botbot.me/freenode/python-requests/


Stefan Krah




From mal at egenix.com  Fri May  9 18:21:42 2014
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 09 May 2014 18:21:42 +0200
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>	<CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>	<20140508121201.GA28465@sleipnir.bytereef.org>	<98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>	<536B8929.6040407@egenix.com>	<0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>	<536B97E2.8070805@egenix.com>	<536BA4D5.7080208@egenix.com>	<70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>	<CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>	<D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>	<536C8DD3.6090601@egenix.com>	<E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>	<536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
Message-ID: <536D0096.3090405@egenix.com>

On 09.05.2014 17:39, Donald Stufft wrote:
> 
> On May 9, 2014, at 9:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> On 09.05.2014 13:44, Donald Stufft wrote:
>>>
>>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> Donald: I don't think anyone is arguing that hosting packages on
>>>> PyPI is a bad thing and PyPI as a service has gotten a lot better
>>>> than it was a few years ago.
>>>
>>> Didn?t mean to imply that I thought it was otherwise.
>>>
>>>>
>>>> However, I find it troubling that we as Python developers are *forcing*
>>>> the whole Python world to put their code into PyPI.
>>>
>>> Forcing is a strong word. As of right now we?re "forcing" you to put it onto
>>> PyPI if you want to install it without *any* flags to pip. 
>>
>> Which is what most people expect to be able to do.
> 
> Sure, but sometimes it?s better to make an informed choice about things being
> installed instead of having it be installed by default and sometimes it?s better
> to disallow doing something at all.
> 
> Further most people don?t expect an install to touch any server host other
> than PyPI. 

For some idea of "most people" that's probably true.

I'd argue that most people probably don't really care all that
much where their packages are coming from as long as they
are getting the packages registered with PyPI, the Python
package index, and can be sure that no one has tampered with
them.

But both arguments don't really count much, since it's all
speculation.

> As far as I?m aware none of the other package repositories
> feature this, and even with the fact we support it barely anyone even cares to
> utilize it.

Most Linux repos comes with a list of standard repos to include.

In Python land, Plone uses zc.buildout with a similar
default list of repos.

>>> You're more then
>>> capable of hosting it externally if you wish, and pip will even tell the people
>>> who try to install it what they need to enable in order to install it. It even
>>> allows people to say "I don't care about any of this crap, just make all the
>>> external stuff work".
>>>
>>> Even if pip removed the external link handling all together and required you
>>> to do something like:
>>>
>>>    $ pip install --extra-index-url https://example.com/our-packages/ foo
>>>    $ pip install --find-links https://example.com/our-packages/ foo
>>>
>>> We still wouldn't be forcing anyone to upload things to PyPI. We are, however,
>>> discouraging people from not hosting on PyPI and providing incentives to doing
>>> that.
>>
>> This is all true, but in reality, you are making externally hosted
>> packages second class citizens in Python land by requiring
>> extra options even for package listings that are perfectly safe
>> to download from other servers.
>>
>> As mentioned before: I can understand that you want to make downloads
>> safe for users, but if a package is hosted on some other reliable
>> servers and pip can check that it's a valid package, there's little
>> to argue for not allowing such downloads.
> 
> Except the fact that the only people I?ve ever seen *happy* that people can
> host packages externally are a handful (<5) of people (tbh primarily you and
> Stefan) and the feedback I get from every other person around the web
> in unequivocally yes please get rid of externally hosted files, they?ve done
> nothing but cause me pain.
> 
> It?s not reasonable to allow a minority of users to negatively impact the majority.

I think you're getting this wrong:

There *are* package authors who would like to host PyPI indexed
packages external to PyPI and they all have good reasons to
do so. It may well be a minority, but that's fine.

Changing the default to allow secure external downloads is *not*
impacting any majority.

At worst, it's impacting the users of those packages, but that's
really within the realm and responsibility of the package authors,
not PyPI or the infrastructure team, so I don't understand why you
are strongly objecting this.

>>>> There are plenty good reasons not to do this, and sometimes it's
>>>> even impossible if you want to stay legal (see the PEP for details).
>>>
>>> I re-read the reasons, I'm not sure I really agree with most (or all?) of them
>>> however if people really want to do it, then there is nothing stopping them.
>>> It's their choice and people on the *other* side of that who are installing
>>> these packages also get to make a choice if they want to allow it or not.
>>
>> You don't have to agree with those reasons. Fact is, they exist,
>> prevent package authors from uploading to PyPI and we as
>> Python developers should respect those reasons.
> 
> No I disagree that they are reasons at all. Half of the ones you enumerated are
> nonsensical or irrelevant, the other half can be fixed by adding features to or
> fixing PyPI. One or two read like reasons why someone might actually decide
> not to upload something to PyPI but which that reason isn?t really all that
> reasonable and finally a grand total of one or two of them look like an actual legit
> reasons and it only applies to Crypto software.

I guess continuing this discussion doesn't really make any sense.
I'm trying to respect your reasoning, but you are apparently
not respecting mine. Without such respect there's no basis for
a discussion.

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, May 09 2014)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/

From donald at stufft.io  Fri May  9 18:23:53 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 12:23:53 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140509161142.GA8440@sleipnir.bytereef.org>
References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
 <20140509161142.GA8440@sleipnir.bytereef.org>
Message-ID: <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io>


On May 9, 2014, at 12:11 PM, Stefan Krah <stefan at bytereef.org> wrote:

> Donald, I'm out of his discussion.  I have one last request: please don't
> gossip about core devs in public as long as you have commit privs:
> 
> https://botbot.me/freenode/python-requests/

I don?t really know how to respond to this. I was talking to some people I know
and I gave them a summary of what was happening. I stand by everything I
said there and I don?t think any of it was wrong or even gossip at all.

If people feel that was inappropriate then go ahead and take my commit privs. I
have them to make it easier for me to write PEPs and to update ensurepip. If
they?re going to be used as an excuse to attempt to censor me then I?d rather
not have them as I generally always speak my mind and I won?t stop doing so.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/3dc6169b/attachment.sig>

From graffatcolmingov at gmail.com  Fri May  9 18:32:34 2014
From: graffatcolmingov at gmail.com (Ian Cordasco)
Date: Fri, 9 May 2014 11:32:34 -0500
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io>
References: <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
 <20140509161142.GA8440@sleipnir.bytereef.org>
 <7F9F36A9-1CD0-4F16-A915-8B70F43A3B68@stufft.io>
Message-ID: <CAN-Kwu2a2EPVn-B1HQozOFDOfK1L5q1n4W7zoyV+LD_Hvh_YPw@mail.gmail.com>

Stefan,

If the only way you can think of to invalidate Donald's (vastly
superior) arguments is to accuse of him of "gossip", you should
probably reconsider your arguments. Looking at the conversation you
didn't actually link to
(https://botbot.me/freenode/python-requests/msg/14389415/) there is no
gossip. There are no insinuations about your character. All that is
there is a succinct description of the topic of this conversation.

Cheers,
Ian

On Fri, May 9, 2014 at 11:23 AM, Donald Stufft <donald at stufft.io> wrote:
>
> On May 9, 2014, at 12:11 PM, Stefan Krah <stefan at bytereef.org> wrote:
>
>> Donald, I'm out of his discussion.  I have one last request: please don't
>> gossip about core devs in public as long as you have commit privs:
>>
>> https://botbot.me/freenode/python-requests/
>
> I don?t really know how to respond to this. I was talking to some people I know
> and I gave them a summary of what was happening. I stand by everything I
> said there and I don?t think any of it was wrong or even gossip at all.
>
> If people feel that was inappropriate then go ahead and take my commit privs. I
> have them to make it easier for me to write PEPs and to update ensurepip. If
> they?re going to be used as an excuse to attempt to censor me then I?d rather
> not have them as I generally always speak my mind and I won?t stop doing so.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/graffatcolmingov%40gmail.com
>

From zachary.ware+pydev at gmail.com  Fri May  9 19:00:04 2014
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Fri, 9 May 2014 12:00:04 -0500
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
In-Reply-To: <CAKJDb-MWH5QHRFcNU7u0ivY9gcubhtk544YPPKd1Wifyr9V_dQ@mail.gmail.com>
References: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
 <536BDCB6.6030300@v.loewis.de>
 <CAKJDb-MWH5QHRFcNU7u0ivY9gcubhtk544YPPKd1Wifyr9V_dQ@mail.gmail.com>
Message-ID: <CAKJDb-N6wH5PUY=rtG5Xw1+ED_k3Ob5xcqFpqoXd0Czg1XmeqA@mail.gmail.com>

On Thu, May 8, 2014 at 4:20 PM, Zachary Ware
<zachary.ware+pydev at gmail.com> wrote:
> I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple
> of weeks ago (see http://bugs.python.org/issue21303), but hadn't
> gotten anything done with Tix yet.  It should just need python.mak
> updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as
> soon as I can.

Tix for 2.7 is now
http://svn.python.org/projects/external/tix-8.4.3.5.  You can build it
with this monster of a command, run from tix-8.4.3.5\win:

nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500
TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0
INSTALL_DIR=..\..\tcltk all

Use "install" instead of "all" after building to install it to
..\..\tcltk.  Set DEBUG and MACHINE as needed; DEBUG does not need to
be set if you're building Release, but MACHINE always has to be set so
that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for
64-bit).

-- 
Zach

From rdmurray at bitdance.com  Fri May  9 19:28:28 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 09 May 2014 13:28:28 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
Message-ID: <20140509172828.ECBA2250D4E@webabinitio.net>

On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft <donald at stufft.io> wrote:
> 
> On May 9, 2014, at 9:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> > On 09.05.2014 13:44, Donald Stufft wrote:
> >> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> > I snipped the rest of the discussion and reliability, using
> > unmaintained packages and projects using their own mirrors (which
> > should really be the standard, not an exceptional case),
> > because it's not really leading anywhere:
> 
> Using your own mirror shouldn???t be the standard if all you???re doing
> is automatically updating that mirror. It???s a hack to get around
> unreliability and it should be seen of as a sign of a failure to provide
> a service that people can rely on and that???s how I see it. People
> depend on this service and it???s irresponsible to not treat it as a
> critical piece of infrastructure.

I don't understand this.  Why it is our responsibility to provide a
free service for a large project to repeatedly download a set of files
they need?  Why does it not make more sense for them to download them
once, and only update their local copies when they change?  That's almost
completely orthogonal to making the service we do provide reliable.

For perspective, Gentoo requests that people only do an emerge sync at
most once a day, and if they have multiple machines to update, that they
only do one pull, and they update the rest of their infrastructure from
their local copy.

As another point of information for comparison, Gentoo downloads files
from wherever they are hosted first, and only if that fails falls back to
a Gentoo provided mirror (if I remember correctly...I think the Gentoo
mirror copy doesn't always exist?).

--David

From donald at stufft.io  Fri May  9 20:12:22 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 14:12:22 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <20140509172828.ECBA2250D4E@webabinitio.net>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
 <20140509172828.ECBA2250D4E@webabinitio.net>
Message-ID: <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io>


On May 9, 2014, at 1:28 PM, R. David Murray <rdmurray at bitdance.com> wrote:

> On Fri, 09 May 2014 11:39:02 -0400, Donald Stufft <donald at stufft.io> wrote:
>> 
>> On May 9, 2014, at 9:58 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> On 09.05.2014 13:44, Donald Stufft wrote:
>>>> On May 9, 2014, at 4:12 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>> I snipped the rest of the discussion and reliability, using
>>> unmaintained packages and projects using their own mirrors (which
>>> should really be the standard, not an exceptional case),
>>> because it's not really leading anywhere:
>> 
>> Using your own mirror shouldn?t be the standard if all you?re doing
>> is automatically updating that mirror. It?s a hack to get around
>> unreliability and it should be seen of as a sign of a failure to provide
>> a service that people can rely on and that?s how I see it. People
>> depend on this service and it?s irresponsible to not treat it as a
>> critical piece of infrastructure.
> 
> I don't understand this.  Why it is our responsibility to provide a
> free service for a large project to repeatedly download a set of files
> they need?  Why does it not make more sense for them to download them
> once, and only update their local copies when they change?  That's almost
> completely orthogonal to making the service we do provide reliable.

Well here?s the thing right. The large projects repeatedly downloading the
same set of files is a canary. If any particular project goes uninstallable on
PyPI (or if PyPI itself goes down) then nobody can install it, the people
installing things over and over every day or the people who just happened
to be installing it during that downtime. However intermittent failures and
general insatiability is going to be noticed by the projects who install things
over and over again quicker and easier and thus it becomes a lot easier
to use them as a general gauge for what the average ?uptime? is.

IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get
a handful of ?small? installers (e.g. not the big projects) in each downtime,
but a different set who are likely to shrug it off and just call treat it as the
norm even though it?s very disruptive to what they?re doing. However the
big project is highly likely to hit every single one of those downtimes and
be able to say ?wow PyPI is flaky as hell?.

To expand further on that if we assume that we want ``pip install <foo>``
to be reliable and not work sometimes and work at other times then we?re
aiming for as high as uptime as possible. PyPI gets enough traffic that
any single large project isn?t a noticeable drop in our bucket and due to the
way our caching works it actually helps us to be faster and more reliable
to have people constantly hitting packages because they?ll be in cache
and able to be served without hitting the Origin servers.

Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the
month of April we served 71.4TB of data with 877.4 million requests of
which 80.5% never made it to the actual servers that run PyPI and were
served directly out of the geo distributed CDN that sits in front of PyPI. We
are vastly better positioned to maintain a reliable infrastructure than ask
that every large project that uses Python to do the same.

The reason that it?s our responsibility for providing it is because we chose
to provide it. There isn?t a moral imperative to run PyPI, but running PyPI
badly seems like a crummy thing to do.

> 
> For perspective, Gentoo requests that people only do an emerge sync at
> most once a day, and if they have multiple machines to update, that they
> only do one pull, and they update the rest of their infrastructure from
> their local copy.

To be clear, there are other reasons to run a local mirror but I don?t think that
it?s reasonable to expect anyone who wants a reliable install using pip to
stand up their own infrastructure.

Further to this point here I?m currently working on adding caching by default
for pip so that we minimize how often different people hit PyPI and we do it
automatically and in a way that doesn?t generally require people to think about
it and that also doesn?t require them to stand up their own infra.

> 
> As another point of information for comparison, Gentoo downloads files
> from wherever they are hosted first, and only if that fails falls back to
> a Gentoo provided mirror (if I remember correctly...I think the Gentoo
> mirror copy doesn't always exist?).
> 
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/a587e0fd/attachment.sig>

From tjreedy at udel.edu  Fri May  9 22:20:32 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 09 May 2014 16:20:32 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
 unreliable [sic]
In-Reply-To: <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
 <20140509172828.ECBA2250D4E@webabinitio.net>
 <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io>
Message-ID: <lkjdap$907$1@ger.gmane.org>

On 5/9/2014 2:12 PM, Donald Stufft wrote:
>
> On May 9, 2014, at 1:28 PM, R. David Murray <rdmurray at bitdance.com> wrote:

>> I don't understand this.  Why it is our responsibility to provide a
>> free service for a large project to repeatedly download a set of files
>> they need?  Why does it not make more sense for them to download them
>> once, and only update their local copies when they change?  That's almost
>> completely orthogonal to making the service we do provide reliable.
>
> Well here?s the thing right. The large projects repeatedly downloading the
> same set of files is a canary. If any particular project goes uninstallable on
> PyPI (or if PyPI itself goes down) then nobody can install it, the people
> installing things over and over every day or the people who just happened
> to be installing it during that downtime. However intermittent failures and
> general insatiability is going to be noticed by the projects who install things
> over and over again quicker and easier and thus it becomes a lot easier
> to use them as a general gauge for what the average ?uptime? is.

I have had the same question as David, so I also appreciate your answer.

> IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get
> a handful of ?small? installers (e.g. not the big projects) in each downtime,
> but a different set who are likely to shrug it off and just call treat it as the
> norm even though it?s very disruptive to what they?re doing. However the
> big project is highly likely to hit every single one of those downtimes and
> be able to say ?wow PyPI is flaky as hell?.
>
> To expand further on that if we assume that we want ``pip install <foo>``
> to be reliable and not work sometimes and work at other times then we?re
> aiming for as high as uptime as possible. PyPI gets enough traffic that
> any single large project isn?t a noticeable drop in our bucket and due to the
> way our caching works it actually helps us to be faster and more reliable
> to have people constantly hitting packages because they?ll be in cache
> and able to be served without hitting the Origin servers.
>
> Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the
> month of April we served 71.4TB of data with 877.4 million requests of
> which 80.5% never made it to the actual servers that run PyPI and were
> served directly out of the geo distributed CDN that sits in front of PyPI. We
> are vastly better positioned to maintain a reliable infrastructure than ask
> that every large project that uses Python to do the same.

> The reason that it?s our responsibility for providing it is because we chose
> to provide it. There isn?t a moral imperative to run PyPI, but running PyPI
> badly seems like a crummy thing to do.

Agreed.

>> For perspective, Gentoo requests that people only do an emerge sync at
>> most once a day, and if they have multiple machines to update, that they
>> only do one pull, and they update the rest of their infrastructure from
>> their local copy.
>
> To be clear, there are other reasons to run a local mirror but I don?t think that
> it?s reasonable to expect anyone who wants a reliable install using pip to
> stand up their own infrastructure.

Ok, you are not saying that caching is bad, but that having everyone 
reinvent caching, and possibly doing it badly, or at least not in 
thebest way, is bad.

> Further to this point here I?m currently working on adding caching by default
> for pip so that we minimize how often different people hit PyPI and we do it
> automatically and in a way that doesn?t generally require people to think about
> it and that also doesn?t require them to stand up their own infra.

This seems like the right solution. It would sort of make each machine a 
micro-CDN node.


-- 
Terry Jan Reedy



From donald at stufft.io  Fri May  9 22:38:27 2014
From: donald at stufft.io (Donald Stufft)
Date: Fri, 9 May 2014 16:38:27 -0400
Subject: [Python-Dev] pip: cdecimal an externally hosted file and may be
	unreliable [sic]
In-Reply-To: <lkjdap$907$1@ger.gmane.org>
References: <20140506213533.GA32765@sleipnir.bytereef.org>
 <CAMpsgwa5fq-EGGSgb8W_9ms-Vzac95pOqM=MLdQrR0jp7-VOrg@mail.gmail.com>
 <20140508121201.GA28465@sleipnir.bytereef.org>
 <98A8E14A-82BE-4739-8D3C-FA06FAFE180E@stufft.io>
 <536B8929.6040407@egenix.com>
 <0B107EB9-FEC1-4110-BC9F-4448EC0DD813@stufft.io>
 <536B97E2.8070805@egenix.com> <536BA4D5.7080208@egenix.com>
 <70F0E55A-B879-437B-80DA-F87EB5CA1098@stufft.io>
 <CACac1F-Yh=8obq4f7pmQatBXxSSLcMJk1QWfkd_mup-Xm=uACw@mail.gmail.com>
 <D38AF6BD-3A04-49D5-8C14-2FB3045B179A@stufft.io>
 <536C8DD3.6090601@egenix.com>
 <E2DEABB2-77C5-400F-AF44-0AC31F4804F1@stufft.io>
 <536CDF0F.6050407@egenix.com>
 <0083071E-5014-448A-B98F-68A3E1BCA777@stufft.io>
 <20140509172828.ECBA2250D4E@webabinitio.net>
 <023F4F14-E631-474C-9B2B-00563A7550EF@stufft.io> <lkjdap$907$1@ger.gmane.org>
Message-ID: <C6C1F799-4AC3-45C4-BD77-816F33E7A6A9@stufft.io>


On May 9, 2014, at 4:20 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 5/9/2014 2:12 PM, Donald Stufft wrote:
>> 
>> On May 9, 2014, at 1:28 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> 
>>> I don't understand this.  Why it is our responsibility to provide a
>>> free service for a large project to repeatedly download a set of files
>>> they need?  Why does it not make more sense for them to download them
>>> once, and only update their local copies when they change?  That's almost
>>> completely orthogonal to making the service we do provide reliable.
>> 
>> Well here?s the thing right. The large projects repeatedly downloading the
>> same set of files is a canary. If any particular project goes uninstallable on
>> PyPI (or if PyPI itself goes down) then nobody can install it, the people
>> installing things over and over every day or the people who just happened
>> to be installing it during that downtime. However intermittent failures and
>> general insatiability is going to be noticed by the projects who install things
>> over and over again quicker and easier and thus it becomes a lot easier
>> to use them as a general gauge for what the average ?uptime? is.
> 
> I have had the same question as David, so I also appreciate your answer.
> 
>> IOW if PyPI goes unavailable for 10 minutes 5 times a day, you might get
>> a handful of ?small? installers (e.g. not the big projects) in each downtime,
>> but a different set who are likely to shrug it off and just call treat it as the
>> norm even though it?s very disruptive to what they?re doing. However the
>> big project is highly likely to hit every single one of those downtimes and
>> be able to say ?wow PyPI is flaky as hell?.
>> 
>> To expand further on that if we assume that we want ``pip install <foo>``
>> to be reliable and not work sometimes and work at other times then we?re
>> aiming for as high as uptime as possible. PyPI gets enough traffic that
>> any single large project isn?t a noticeable drop in our bucket and due to the
>> way our caching works it actually helps us to be faster and more reliable
>> to have people constantly hitting packages because they?ll be in cache
>> and able to be served without hitting the Origin servers.
>> 
>> Just for the record, PyPI gets roughly 350 req/s basically 24/7, in the
>> month of April we served 71.4TB of data with 877.4 million requests of
>> which 80.5% never made it to the actual servers that run PyPI and were
>> served directly out of the geo distributed CDN that sits in front of PyPI. We
>> are vastly better positioned to maintain a reliable infrastructure than ask
>> that every large project that uses Python to do the same.
> 
>> The reason that it?s our responsibility for providing it is because we chose
>> to provide it. There isn?t a moral imperative to run PyPI, but running PyPI
>> badly seems like a crummy thing to do.
> 
> Agreed.
> 
>>> For perspective, Gentoo requests that people only do an emerge sync at
>>> most once a day, and if they have multiple machines to update, that they
>>> only do one pull, and they update the rest of their infrastructure from
>>> their local copy.
>> 
>> To be clear, there are other reasons to run a local mirror but I don?t think that
>> it?s reasonable to expect anyone who wants a reliable install using pip to
>> stand up their own infrastructure.
> 
> Ok, you are not saying that caching is bad, but that having everyone reinvent caching, and possibly doing it badly, or at least not in thebest way, is bad.

Yea, caching isn?t in general a bad thing, and actually PyPI uses it heavily. All
access to /simple/ and /packages/ is cached for 24 hours by our CDN unless
someone uploads or deletes a file, in which case we selectively purge those
URLs from the CDN cache so that they (nearly) instantly get the updated
results.

Warehouse (aka PyPI 2.0) is designed to utilize our CDN cache even further
and I?m hoping to get our cache rate even higher using it.

> 
>> Further to this point here I?m currently working on adding caching by default
>> for pip so that we minimize how often different people hit PyPI and we do it
>> automatically and in a way that doesn?t generally require people to think about
>> it and that also doesn?t require them to stand up their own infra.
> 
> This seems like the right solution. It would sort of make each machine a micro-CDN node.

Yes, it?s bog standard HTTP stuff just like a browser does it. The major difference is we?re
limiting the maximum lifetime of a cache item in the client (pip) for the index pages but
we are not doing that for the package files themselves. This is done to prevent a misconfigured
server from causing pip to not see new versions for hours/days/weeks/years/whatever.

Additionally this change also includes making pip smarter about HTTP requests in that if
we have a stale item in the cache which as a Last-Modified or an ETag header we?ll do
a conditional GET which will hopefully be returned with an HTTP 304 Not Modified so that
we can simply refresh the stale item in the cache and use it again instead of needing to
download an entire response body again.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140509/2dddd3c9/attachment.sig>

From 4kir4.1i at gmail.com  Fri May  9 21:53:06 2014
From: 4kir4.1i at gmail.com (akira)
Date: Fri, 09 May 2014 23:53:06 +0400
Subject: [Python-Dev] should tests be thread-safe?
Message-ID: <536D3222.5090109@gmail.com>

Hi,

May tests expect that unless they themselves start a thread then there 
are no threads to worry about?

I see that some old tests are not thread-safe and I have not found it to 
be explicitly mentioned in the devguide.

I've written a test for http://bugs.python.org/issue21332 that is known 
to be non-thread-safe. Is it acceptable for new tests?


--
akira

From benjamin at python.org  Sat May 10 02:57:43 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 09 May 2014 17:57:43 -0700
Subject: [Python-Dev] pushing 2.7.7 back a week
Message-ID: <1399683463.17589.115723569.5D169976@webmail.messagingengine.com>

I'm pushing the release schedule for 2.7.7 back a week. That means the
release candidate will be next weekend (May 17). This will hopefully
save some trouble for the people who build our binaries, since 3.4.1
final is happening next weekend, too.

Regards,
Benjamin

From ncoghlan at gmail.com  Sat May 10 13:41:40 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 10 May 2014 21:41:40 +1000
Subject: [Python-Dev] should tests be thread-safe?
In-Reply-To: <536D3222.5090109@gmail.com>
References: <536D3222.5090109@gmail.com>
Message-ID: <CADiSq7fusvbmk1H2n6pSFLbS2zy5_13jhR12vaTJkfXaGJ9c4g@mail.gmail.com>

On 10 May 2014 06:53, "akira" <4kir4.1i at gmail.com> wrote:
>
> Hi,
>
> May tests expect that unless they themselves start a thread then there
are no threads to worry about?
>
> I see that some old tests are not thread-safe and I have not found it to
be explicitly mentioned in the devguide.
>
> I've written a test for http://bugs.python.org/issue21332 that is known
to be non-thread-safe. Is it acceptable for new tests?

Thread safety is desirable, but not mandatory, since there is some process
global state (e.g. the import system and the sys module in general) that
the tests sometimes need to manipulate.

Cheers,
Nick.

>
>
> --
> akira
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/65509383/attachment.html>

From brian at python.org  Sat May 10 20:10:19 2014
From: brian at python.org (Brian Curtin)
Date: Sat, 10 May 2014 13:10:19 -0500
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
In-Reply-To: <CAKJDb-N6wH5PUY=rtG5Xw1+ED_k3Ob5xcqFpqoXd0Czg1XmeqA@mail.gmail.com>
References: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
 <536BDCB6.6030300@v.loewis.de>
 <CAKJDb-MWH5QHRFcNU7u0ivY9gcubhtk544YPPKd1Wifyr9V_dQ@mail.gmail.com>
 <CAKJDb-N6wH5PUY=rtG5Xw1+ED_k3Ob5xcqFpqoXd0Czg1XmeqA@mail.gmail.com>
Message-ID: <CAD+XWwrLfes_+0Jpa_3-BoH3OVz_nfA_ZrABX=owLJeW5=WY7A@mail.gmail.com>

On Fri, May 9, 2014 at 12:00 PM, Zachary Ware
<zachary.ware+pydev at gmail.com> wrote:
> On Thu, May 8, 2014 at 4:20 PM, Zachary Ware
> <zachary.ware+pydev at gmail.com> wrote:
>> I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple
>> of weeks ago (see http://bugs.python.org/issue21303), but hadn't
>> gotten anything done with Tix yet.  It should just need python.mak
>> updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as
>> soon as I can.
>
> Tix for 2.7 is now
> http://svn.python.org/projects/external/tix-8.4.3.5.  You can build it
> with this monster of a command, run from tix-8.4.3.5\win:
>
> nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500
> TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0
> INSTALL_DIR=..\..\tcltk all
>
> Use "install" instead of "all" after building to install it to
> ..\..\tcltk.  Set DEBUG and MACHINE as needed; DEBUG does not need to
> be set if you're building Release, but MACHINE always has to be set so
> that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for
> 64-bit).

Awesome, thanks!

So I now have a fully working setup, at least for 32-bit, and have
backported the Path option from 3.3
(http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue
with win32com getting a 64-bit installer built but didn't have time to
look into it yet.

If anyone wants to try a 2.7 installer with that Path option, here's a
copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not
signed, it'll warn you about that).

From Steve.Dower at microsoft.com  Sat May 10 20:35:17 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sat, 10 May 2014 18:35:17 +0000
Subject: [Python-Dev] Tix version needed to build 2.7 Windows installer?
In-Reply-To: <CAD+XWwrLfes_+0Jpa_3-BoH3OVz_nfA_ZrABX=owLJeW5=WY7A@mail.gmail.com>
References: <CAD+XWwp9WHC7quzH+9kZ6jKJFgWa62cVtYpeRW=BFrseoAjHAA@mail.gmail.com>
 <536BDCB6.6030300@v.loewis.de>
 <CAKJDb-MWH5QHRFcNU7u0ivY9gcubhtk544YPPKd1Wifyr9V_dQ@mail.gmail.com>
 <CAKJDb-N6wH5PUY=rtG5Xw1+ED_k3Ob5xcqFpqoXd0Czg1XmeqA@mail.gmail.com>,
 <CAD+XWwrLfes_+0Jpa_3-BoH3OVz_nfA_ZrABX=owLJeW5=WY7A@mail.gmail.com>
Message-ID: <1399746936051.48735@microsoft.com>


Brian Curtin wrote:
> On Fri, May 9, 2014 at 12:00 PM, Zachary Ware
> <zachary.ware+pydev at gmail.com> wrote:
>> On Thu, May 8, 2014 at 4:20 PM, Zachary Ware
>> <zachary.ware+pydev at gmail.com> wrote:
>>> I updated the 2.7 buildbot scripts to pull in Tcl/Tk 8.5.15 a couple
>>> of weeks ago (see http://bugs.python.org/issue21303), but hadn't
>>> gotten anything done with Tix yet.  It should just need python.mak
>>> updated to point to Tcl/Tk 8.5.15; it's on my list to get fixed as
>>> soon as I can.
>>
>> Tix for 2.7 is now
>> http://svn.python.org/projects/external/tix-8.4.3.5.  You can build it
>> with this monster of a command, run from tix-8.4.3.5\win:
>>
>> nmake -f python.mak DEBUG=0 MACHINE=IX86 COMPILERFLAGS=-DWINVER=0x0500
>> TCL_DIR=..\..\tcl-8.5.15.0 TK_DIR=..\..\tk-8.5.15.0
>> INSTALL_DIR=..\..\tcltk all
>>
>> Use "install" instead of "all" after building to install it to
>> ..\..\tcltk.  Set DEBUG and MACHINE as needed; DEBUG does not need to
>> be set if you're building Release, but MACHINE always has to be set so
>> that Tix uses the right build dir for Tk (IX86 for 32-bit, AMD64 for
>> 64-bit).
> 
> Awesome, thanks!
> 
> So I now have a fully working setup, at least for 32-bit, and have
> backported the Path option from 3.3
> (http://hg.python.org/cpython/rev/a9d34685ec47). I ran into an issue
> with win32com getting a 64-bit installer built but didn't have time to
> look into it yet.

If the error is displayed immediately after the msm filename, it probably means you don't have that file. I had to pull the right ones out of our VS build directory, since I couldn't find a public version for SP1 (and apparently the QFE that updates the 32-bit MSM doesn't update the 64-bit one... guess I'll have to go look for that build now).
 
> If anyone wants to try a 2.7 installer with that Path option, here's a
> copy: http://briancurtin.com/python-dev/python-2.7.msi (this is not
> signed, it'll warn you about that).

Looks good to me, and both of my builds worked fine.

Cheers,
Steve

From gregory.szorc at gmail.com  Sat May 10 22:05:54 2014
From: gregory.szorc at gmail.com (Gregory Szorc)
Date: Sat, 10 May 2014 13:05:54 -0700
Subject: [Python-Dev] python process creation overhead
Message-ID: <536E86A2.600@gmail.com>

I was investigating speeding up Mercurial's test suite (it runs ~13,000
Python processes) and I believe I've identified CPython
process/interpreter creation and destruction as sources of significant
overhead and thus a concern for any CPython user.

Full details are at [1]. tl;dr 10-18% of CPU time in test suite
execution was spent doing the equivalent of `python -c 1` and 30-38% of
CPU time was spent doing the equivalent of `hg version` (I suspect this
is mostly dominated by module importing - and Mercurial has lazy module
importing). My measurements show CPython is significantly slower than
Perl and Ruby when it comes to process/interpreter creation/destruction.
Furthermore, Python 3 appears to be >50% slower than Python 2.

Any system spawning many Python processes will likely feel this
overhead. I'm also a contributor to Firefox's build and testing
infrastructure. Those systems collectively spawn thousands of Python
processes. While I haven't yet measured, I fear that CPython process
overhead is also contributing to significant overhead there.

While more science and knowledge should be added before definite
conclusions are reached, I'd like to ring the figurative bell about this
problem. This apparent inefficiency has me concerned about the use of
CPython in scenarios where process spawning is a common operation and
decent latency is important. This includes CLI tools, build systems, and
testing. I'm curious if others feel this is a problem and what steps can
be taken to mitigate or correct it.

[1] http://www.selenic.com/pipermail/mercurial-devel/2014-May/058854.html

Gregory Szorc
gregory.szorc at gmail.com

P.S. I'm not subscribed, so please CC me on responses.


From alex.gaynor at gmail.com  Sat May 10 23:18:22 2014
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 10 May 2014 14:18:22 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <3gR1CR0CDcz7LjY@mail.python.org>
References: <3gR1CR0CDcz7LjY@mail.python.org>
Message-ID: <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>

Hi python-dev and Raymond,

I think this change is a considerable usability regression for the
documentation. Right now the warnings about CSPRNGs are hidden in the
introductory paragraph, which users are likely to skip. I agree that
there's no need to repeat the same advice twice, but I'd much rather we
kept the ".. warning:: " version, so users are more likely to actually read
it.

Also, there's a few errors with your commit message. First, we can
reasonably assert that urandom provides an acceptable CSPRNG, mostly
because it does on every platform I'm aware of. Second, urandom is still a
psuedo-random number generator, however they are cryptographically secure,
it's not "more random". Wikipedia does a good job laying out the necessary
properties for a CSPRNG:
https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator#Requirements

Cheers,
Alex


On Sat, May 10, 2014 at 2:05 PM, raymond.hettinger <
python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/b466dc34b86e
> changeset:   90618:b466dc34b86e
> parent:      90616:ce070040e1a6
> user:        Raymond Hettinger <python at rcn.com>
> date:        Sat May 10 14:05:28 2014 -0700
> summary:
>   Remove the redundant and poorly worded warning message.
>
> The paragraph above already says, clearly and correctly, that
> "However, being completely deterministic, it is not suitable for
> all purposes, and is completely unsuitable for cryptographic purposes."
>
> Also we should make any promises about SystemRandom or os.urandom()
> being cryptographically secure (they may be, but be can't validate
> that promise).  Further, those are actual random number generators
> not psuedo-random number generators.
>
> files:
>   Doc/library/random.rst |  6 ------
>   1 files changed, 0 insertions(+), 6 deletions(-)
>
>
> diff --git a/Doc/library/random.rst b/Doc/library/random.rst
> --- a/Doc/library/random.rst
> +++ b/Doc/library/random.rst
> @@ -43,12 +43,6 @@
>  uses the system function :func:`os.urandom` to generate random numbers
>  from sources provided by the operating system.
>
> -.. warning::
> -
> -   The pseudo-random generators of this module should not be used for
> -   security purposes.  Use :func:`os.urandom` or :class:`SystemRandom` if
> -   you require a cryptographically secure pseudo-random number generator.
> -
>
>  Bookkeeping functions:
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/5beba5d4/attachment.html>

From raymond.hettinger at gmail.com  Sat May 10 23:35:38 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 10 May 2014 14:35:38 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
	and poorly worded warning message.
In-Reply-To: <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
Message-ID: <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>


On May 10, 2014, at 2:18 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:

> I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip


In the past couple of years, we've grown an unfortunate tendency
to fill the docs with big warning boxes (the subprocess docs are
an example of implicitly communicating that the module is dangerous
and unusable).

The preferred form of documentation is to be affirmatively worded,
telling how to use a tool correctly and what its known limitations are.
We save the warnings for cases of actual danger where otherwise
well informed users get tripped-up.

Tim Peters used to be around to articulate the principle that we don't write
the docs with the assumption that the users are less bright than we are
or that they can't read.

People writing applications that need to be sure should probably have
a howto guide.  I don't think they are well served by filling the module
docs with every possible way a person could make a security mistake).

If you're writing a secure on-line poker game, you really need to know
a great deal more about security than the warning message can supply.
My understanding is that actual gaming systems use seeded CSPRNGs
rather than urandom() because those systems need to be auditable.


Raymond


P.S.  The docs for the random module had a successful 20 year history
without the redundant big red warning box, so it is not really correct
to say there has been a "considerable usability regression".


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/b5f3ef2e/attachment.html>

From victor.stinner at gmail.com  Sat May 10 23:39:57 2014
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 10 May 2014 23:39:57 +0200
Subject: [Python-Dev] should tests be thread-safe?
In-Reply-To: <536D3222.5090109@gmail.com>
References: <536D3222.5090109@gmail.com>
Message-ID: <CAMpsgwaZ8cL1VO+zU70JW5HEGdFNOzoKQtKSENe3oVjikPj3Aw@mail.gmail.com>

If you need a well defined environement, run your test in a subprocess.
Depending on the random function, your test may be run with more threads.
On BSD, it changes for example which thread receives a signal. Importing
the tkinter module creates a "hidden" C thread for the Tk loop.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/38f1387c/attachment.html>

From solipsis at pitrou.net  Sat May 10 23:43:06 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 10 May 2014 23:43:06 +0200
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <536E86A2.600@gmail.com>
References: <536E86A2.600@gmail.com>
Message-ID: <20140510234306.342b2a94@fsol>


Hello,

On Sat, 10 May 2014 13:05:54 -0700
Gregory Szorc <gregory.szorc at gmail.com> wrote:
> I was investigating speeding up Mercurial's test suite (it runs ~13,000
> Python processes) and I believe I've identified CPython
> process/interpreter creation and destruction as sources of significant
> overhead and thus a concern for any CPython user.

This was recently discussed in
https://mail.python.org/pipermail/python-dev/2014-April/134028.html

To give numbers:

$ time python -c pass

real	0m0.012s
user	0m0.008s
sys	0m0.004s

$ time python -c "from mercurial import demandimport; demandimport.enable(); from mercurial.dispatch import request, dispatch; dispatch(request(['version']))"
Mercurial Distributed SCM (version 3.0-rc+2-d924e387604f)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2014 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

real	0m0.056s
user	0m0.044s
sys	0m0.012s

So, even for "hg version", most of the time is spent initializing
Mercurial and its modules, not initializing Python itself. In this
precise measurement, an infinitely fast Python startup would make "hg
version" 20% faster (and other hg commands probably even less, since
they have to load more Mercurial code and data).

I'm wondering why Mercurial couldn't have a transparent daemon mode,
where each "hg" invocation on the CLI would only load whatever
required to communicate with the persistent daemon process. It already
has a commandserver.

> P.S. I'm not subscribed, so please CC me on responses.

I would suggest following the list using the gmane NNTP gateway.

Regards

Antoine.

From victor.stinner at gmail.com  Sat May 10 23:46:23 2014
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 10 May 2014 23:46:23 +0200
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <536E86A2.600@gmail.com>
References: <536E86A2.600@gmail.com>
Message-ID: <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>

Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com> a ?crit :
> Furthermore, Python 3 appears to be >50% slower than Python 2.

Please mention the minor version. It looks like you compared 2.7 and 3.3.
Please test 3.4, we made interesting progress on the startup time.

There is still something to do, especially on OS X. Depending on the OS,
different modules are loaded and some functions are implemented differently.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/d0bd959b/attachment.html>

From solipsis at pitrou.net  Sat May 10 23:54:01 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 10 May 2014 23:54:01 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
Message-ID: <20140510235401.1b4774f2@fsol>

On Sat, 10 May 2014 14:35:38 -0700
Raymond Hettinger <raymond.hettinger at gmail.com> wrote:
> 
> In the past couple of years, we've grown an unfortunate tendency
> to fill the docs with big warning boxes (the subprocess docs are
> an example of implicitly communicating that the module is dangerous
> and unusable).
> 
> The preferred form of documentation is to be affirmatively worded,
> telling how to use a tool correctly and what its known limitations are.
> We save the warnings for cases of actual danger where otherwise
> well informed users get tripped-up.
> 
> Tim Peters used to be around to articulate the principle that we don't write
> the docs with the assumption that the users are less bright than we are
> or that they can't read.

I agree with Alex. It's not about being bright or not, it's about being
*willing* to eat walls of text. However pleasant it may be for some
people to *write* documentation, for most readers (and especially
non-native English readers, who read more slowly and more painfully
than native ones), documentation is a piece of reference that they skim
from, rather than read from start to end like a novel.

So it makes sense to single out useful information. Knowing that
the functions from the random module are not appropriate for
cryptography is (IMO) more important than knowing that

"""Python uses the Mersenne Twister as the core generator. It produces
53-bit precision floats and has a period of 2**19937-1. The underlying
implementation in C is both fast and threadsafe"""

Regards

Antoine.



From ncoghlan at gmail.com  Sun May 11 00:10:03 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 11 May 2014 08:10:03 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
Message-ID: <CADiSq7dM=MavzaHTaJ2H9xdbT6Dz1c2mF_EydcFU-00=af+QCg@mail.gmail.com>

On 11 May 2014 07:37, "Raymond Hettinger" <raymond.hettinger at gmail.com>
wrote:
>
>
> On May 10, 2014, at 2:18 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>> I think this change is a considerable usability regression for the
documentation. Right now the warnings about CSPRNGs are hidden in the
introductory paragraph, which users are likely to skip
>
>
> In the past couple of years, we've grown an unfortunate tendency
> to fill the docs with big warning boxes (the subprocess docs are
> an example of implicitly communicating that the module is dangerous
> and unusable).
>
> The preferred form of documentation is to be affirmatively worded,
> telling how to use a tool correctly and what its known limitations are.
> We save the warnings for cases of actual danger where otherwise
> well informed users get tripped-up.
>
> Tim Peters used to be around to articulate the principle that we don't
write
> the docs with the assumption that the users are less bright than we are
> or that they can't read.

That assumption has changed somewhat, as many users are now getting their
education in programming (and how computers work in general) from
introductory community workshops and the Python documentation.

This means that writing our docs based on the assumption that the reader is
already going to be a professional programmer is no longer adequate. This
is especially essential in security related areas, as even professional
programmers usually aren't sufficiently paranoid about all the ways their
software can be attacked.

As Alex notes, the short term way to eliminate the duplication here is to
keep the security warnings and drop the material from the introductory
paragraph, not go back to expecting readers to have already been alerted to
randomness related cryptographic security issues in some other context.
Readers that are already familiar with the security concerns will hopefully
recognise that they're not a Python specific problem (and may even be
appreciative of our attempts to convey the relevant knowledge to a broader
audience), while readers that aren't yet aware of them may be more likely
to account for them appropriately when writing their software. It's as much
about cultivating a more paranoid more mindset in developers in general as
it is about the contents of the specific security warning.

A "Security Considerations" section in the module documentation can be a
better way to tackle this than trying to jam everything into one warning
box, but there should still be a "Here be dragons" warning early on noting
that random *is* a potentially security sensitive module in a cryptographic
context.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/c1c29a11/attachment.html>

From donald at stufft.io  Sun May 11 00:16:08 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 10 May 2014 18:16:08 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
	and poorly worded warning message.
In-Reply-To: <CADiSq7dM=MavzaHTaJ2H9xdbT6Dz1c2mF_EydcFU-00=af+QCg@mail.gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <CADiSq7dM=MavzaHTaJ2H9xdbT6Dz1c2mF_EydcFU-00=af+QCg@mail.gmail.com>
Message-ID: <1C20E6FF-BF2A-4076-BFEC-9645E3290319@stufft.io>


On May 10, 2014, at 6:10 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 11 May 2014 07:37, "Raymond Hettinger" <raymond.hettinger at gmail.com> wrote:
> >
> >
> > On May 10, 2014, at 2:18 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> >
> >> I think this change is a considerable usability regression for the documentation. Right now the warnings about CSPRNGs are hidden in the introductory paragraph, which users are likely to skip
> >
> >
> > In the past couple of years, we've grown an unfortunate tendency
> > to fill the docs with big warning boxes (the subprocess docs are
> > an example of implicitly communicating that the module is dangerous
> > and unusable).
> >
> > The preferred form of documentation is to be affirmatively worded,
> > telling how to use a tool correctly and what its known limitations are.
> > We save the warnings for cases of actual danger where otherwise
> > well informed users get tripped-up.
> >
> > Tim Peters used to be around to articulate the principle that we don't write
> > the docs with the assumption that the users are less bright than we are
> > or that they can't read.
> 
> That assumption has changed somewhat, as many users are now getting their education in programming (and how computers work in general) from introductory community workshops and the Python documentation.
> 
> This means that writing our docs based on the assumption that the reader is already going to be a professional programmer is no longer adequate. This is especially essential in security related areas, as even professional programmers usually aren't sufficiently paranoid about all the ways their software can be attacked.
> 
> As Alex notes, the short term way to eliminate the duplication here is to keep the security warnings and drop the material from the introductory paragraph, not go back to expecting readers to have already been alerted to randomness related cryptographic security issues in some other context. Readers that are already familiar with the security concerns will hopefully recognise that they're not a Python specific problem (and may even be appreciative of our attempts to convey the relevant knowledge to a broader audience), while readers that aren't yet aware of them may be more likely to account for them appropriately when writing their software. It's as much about cultivating a more paranoid more mindset in developers in general as it is about the contents of the specific security warning.
> 
> A "Security Considerations" section in the module documentation can be a better way to tackle this than trying to jam everything into one warning box, but there should still be a "Here be dragons" warning early on noting that random *is* a potentially security sensitive module in a cryptographic context.
> 

I completely agree with Alex, Antoine, and Nick here.

I?m both an experienced Python programmer and someone who is generally aware of the security implications of various parts of software. However I appreciate when I look at documentation that explicitly calls out the ways I might screw it up, especially in a security sensitive context, but I appreciate it any context really.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/67527ca1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/67527ca1/attachment.sig>

From raymond.hettinger at gmail.com  Sun May 11 00:21:21 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 10 May 2014 15:21:21 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
	and poorly worded warning message.
In-Reply-To: <20140510235401.1b4774f2@fsol>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
Message-ID: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>


On May 10, 2014, at 2:54 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> It's not about being bright or not, it's about being
> *willing* to eat walls of text. However pleasant it may be for some
> people to *write* documentation, for most readers (and especially
> non-native English readers, who read more slowly and more painfully
> than native ones), documentation is a piece of reference that they skim
> from, rather than read from start to end like a novel.


Most users of the random module documentation are just
normal people trying to create random numbers.  People
writing secure apps with cryptographically secure random
numbers are not the primary audience.  But we have this
big red security warning that essentially says, if you read
only one thing, read this.

Before proceeding further with stamping distracting security
warnings all over the module documentation, we should look
to other languages to see what others have found necessary.
This warning does not appear anywhere else I've looked 
(MS Excel docs, Java docs, Go lang docs, etc.)

http://docs.oracle.com/javase/6/docs/api/java/util/Random.html
http://golang.org/pkg/math/rand/

Those docs are clear, concise, not preachy, and not littered
with distractions.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/ba0dde50/attachment.html>

From guido at python.org  Sun May 11 00:24:40 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 10 May 2014 15:24:40 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
Message-ID: <CAP7+vJJ_1az4-PDj4ABnsMAXDA89G=iGD73wU7WpZEumppPAbQ@mail.gmail.com>

Give it up, Raymond.

On Saturday, May 10, 2014, Raymond Hettinger <raymond.hettinger at gmail.com>
wrote:

>
> On May 10, 2014, at 2:54 PM, Antoine Pitrou <solipsis at pitrou.net<javascript:_e(%7B%7D,'cvml','solipsis at pitrou.net');>>
> wrote:
>
> It's not about being bright or not, it's about being
> *willing* to eat walls of text. However pleasant it may be for some
> people to *write* documentation, for most readers (and especially
> non-native English readers, who read more slowly and more painfully
> than native ones), documentation is a piece of reference that they skim
> from, rather than read from start to end like a novel.
>
>
> Most users of the random module documentation are just
> normal people trying to create random numbers.  People
> writing secure apps with cryptographically secure random
> numbers are not the primary audience.  But we have this
> big red security warning that essentially says, if you read
> only one thing, read this.
>
> Before proceeding further with stamping distracting security
> warnings all over the module documentation, we should look
> to other languages to see what others have found necessary.
> This warning does not appear anywhere else I've looked
> (MS Excel docs, Java docs, Go lang docs, etc.)
>
> http://docs.oracle.com/javase/6/docs/api/java/util/Random.html
> http://golang.org/pkg/math/rand/
>
> Those docs are clear, concise, not preachy, and not littered
> with distractions.
>
>
> Raymond
>
>

-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/9ffc13f4/attachment.html>

From donald at stufft.io  Sun May 11 00:33:10 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 10 May 2014 18:33:10 -0400
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
Message-ID: <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io>


On May 10, 2014, at 5:46 PM, Victor Stinner <victor.stinner at gmail.com> wrote:

> Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com> a ?crit :
> > Furthermore, Python 3 appears to be >50% slower than Python 2.
> 
> Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time.
> 
> There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently.
> 

For what it's worth pip is the same way, about half of our test suite involves
invoking (multiple) python processes. This has historically be really slow
(~30 minutes to run ~200 tests of that type). We've been able to get the wall
clock run time down by parallelizing these but the sequential time is still
really slow.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/be55f763/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/be55f763/attachment.sig>

From ezio.melotti at gmail.com  Sun May 11 00:35:28 2014
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Sun, 11 May 2014 01:35:28 +0300
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
Message-ID: <CACBhJdHN9pizB1KGwfFF_+V8bwEbYe2-qbZ_Sarb8n3axnLORA@mail.gmail.com>

Hi,

On Sun, May 11, 2014 at 12:35 AM, Raymond Hettinger
<raymond.hettinger at gmail.com> wrote:
>
> On May 10, 2014, at 2:18 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
> I think this change is a considerable usability regression for the
> documentation. Right now the warnings about CSPRNGs are hidden in the
> introductory paragraph, which users are likely to skip
>
>
> In the past couple of years, we've grown an unfortunate tendency
> to fill the docs with big warning boxes

If the problem is the scary red boxes,
http://bugs.python.org/issue13515 still has a patch to make them less
intimidating (and some interesting discussion about it).

Best Regards,
Ezio Melotti

> (the subprocess docs are
> an example of implicitly communicating that the module is dangerous
> and unusable).
>
> The preferred form of documentation is to be affirmatively worded,
> telling how to use a tool correctly and what its known limitations are.
> We save the warnings for cases of actual danger where otherwise
> well informed users get tripped-up.
>
> Tim Peters used to be around to articulate the principle that we don't write
> the docs with the assumption that the users are less bright than we are
> or that they can't read.
>
> People writing applications that need to be sure should probably have
> a howto guide.  I don't think they are well served by filling the module
> docs with every possible way a person could make a security mistake).
>
> If you're writing a secure on-line poker game, you really need to know
> a great deal more about security than the warning message can supply.
> My understanding is that actual gaming systems use seeded CSPRNGs
> rather than urandom() because those systems need to be auditable.
>
>
> Raymond
>
>
> P.S.  The docs for the random module had a successful 20 year history
> without the redundant big red warning box, so it is not really correct
> to say there has been a "considerable usability regression".

From cf.natali at gmail.com  Sun May 11 00:43:26 2014
From: cf.natali at gmail.com (=?ISO-8859-1?Q?Charles=2DFran=E7ois_Natali?=)
Date: Sat, 10 May 2014 23:43:26 +0100
Subject: [Python-Dev] should tests be thread-safe?
In-Reply-To: <CAMZNu4eFDUqij2mvE2CE6BE-iN95u7Erdu6qzX3qLAysF8WMYA@mail.gmail.com>
References: <536D3222.5090109@gmail.com>
 <CAH_1eM2YSGG2BxBhH2Ov24gdUtE3A2cO5maEjDy4S5_p3Lc-tw@mail.gmail.com>
 <CAMZNu4eFDUqij2mvE2CE6BE-iN95u7Erdu6qzX3qLAysF8WMYA@mail.gmail.com>
Message-ID: <CAH_1eM0VPZv+fze_=_oskTVk5DPr_hWYYhMFmX8ej+_FaD-k6A@mail.gmail.com>

> You might have forgotten to include Python-dev in the reply.

Indeed, adding it back!

> Thank you for the reply. I might have expressed the question poorely. I
> meant: I have a script that I know is not thread-safe but it doesn't matter
> because the test itself doesn't run any threads and the current tests are
> never(?) run in multiple threads (-j uses processes). Should this *new* test
> be fixed if e.g., there is a desire to be able to run (at least some) tests
> in multiple threads concurrently in the future?

The short answer is: no, you don't have to make you thread thread
safe, as long as it can reliably run even in the presence of
background threads (like the tkinter threads Victor mentions).

From guido at python.org  Sun May 11 00:50:14 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 10 May 2014 15:50:14 -0700
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io>
Message-ID: <CAP7+vJ+EnOLBigWVCug1OajquM33i1U6DXzqvr_ZVVReuMPjFw@mail.gmail.com>

Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python
startup time is only a tiny fraction of that (20-40 *milliseconds*).


On Sat, May 10, 2014 at 3:33 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On May 10, 2014, at 5:46 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
> Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com> a ?crit :
> > Furthermore, Python 3 appears to be >50% slower than Python 2.
>
> Please mention the minor version. It looks like you compared 2.7 and 3.3.
> Please test 3.4, we made interesting progress on the startup time.
>
> There is still something to do, especially on OS X. Depending on the OS,
> different modules are loaded and some functions are implemented differently.
>
>
> For what it's worth pip is the same way, about half of our test suite
> involves
> invoking (multiple) python processes. This has historically be really slow
> (~30 minutes to run ~200 tests of that type). We've been able to get the
> wall
> clock run time down by parallelizing these but the sequential time is still
> really slow.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/ef365915/attachment.html>

From ncoghlan at gmail.com  Sun May 11 01:01:33 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 11 May 2014 09:01:33 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
Message-ID: <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>

On 11 May 2014 08:24, "Raymond Hettinger" <raymond.hettinger at gmail.com>
wrote:
>
> Before proceeding further with stamping distracting security
> warnings all over the module documentation, we should look
> to other languages to see what others have found necessary.
> This warning does not appear anywhere else I've looked
> (MS Excel docs, Java docs, Go lang docs, etc.)
>
> http://docs.oracle.com/javase/6/docs/api/java/util/Random.html
> http://golang.org/pkg/math/rand/
>
> Those docs are clear, concise, not preachy, and not littered
> with distractions.

The fact that many (most?) programmers treat security considerations as a
distraction is a core part of the problem we're trying to address.

As you point out, most language development teams do very little to try to
educate their users about security issues. The consequences of that are
clearly visible in the world around us: when security is treated as an
optional afterthought, you get widespread deployment of insecure software.

At this point, we have two options:

* continue with the same model as everyone else, and treat security as an
optional extra users should feel free to ignore (or treat as an advanced
topic only specialists need to worry about)

* change our documentation practices to try to encourage the growth of a
security aware development community around Python, trusting that our users
will recognise that the security issues we're discussing are inherent in
the way computers work, rather than being specific to Python.

I'm obviously a strong advocate for the second path. Users aren't stupid,
they'll figure out that almost all the security concerns we're warning
about are inherent in the problem being solved, rather than being a
Python-specific issue.

Cheers,
Nick.

>
>
> Raymond
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/555103cc/attachment.html>

From stefan_ml at behnel.de  Sun May 11 01:15:31 2014
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 11 May 2014 01:15:31 +0200
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
 <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
Message-ID: <lkmbum$d9a$1@ger.gmane.org>

Nick Coghlan, 11.05.2014 01:01:
> As you point out, most language development teams do very little to try to
> educate their users about security issues. The consequences of that are
> clearly visible in the world around us: when security is treated as an
> optional afterthought, you get widespread deployment of insecure software.
> 
> At this point, we have two options:
> 
> * continue with the same model as everyone else, and treat security as an
> optional extra users should feel free to ignore (or treat as an advanced
> topic only specialists need to worry about)
> 
> * change our documentation practices to try to encourage the growth of a
> security aware development community around Python, trusting that our users
> will recognise that the security issues we're discussing are inherent in
> the way computers work, rather than being specific to Python.
> 
> I'm obviously a strong advocate for the second path. Users aren't stupid,
> they'll figure out that almost all the security concerns we're warning
> about are inherent in the problem being solved, rather than being a
> Python-specific issue.

Even if I know the problematic parts of a certain corner of software
development or just of a specific tool, I prefer reading in the
documentation that the authors of that tool are also aware of the
(potential) problems. Makes me feel more comfortable with trusting the
software.

Total +1 on keeping these little bits around.

Stefan



From donald at stufft.io  Sun May 11 01:19:50 2014
From: donald at stufft.io (Donald Stufft)
Date: Sat, 10 May 2014 19:19:50 -0400
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <CAP7+vJ+EnOLBigWVCug1OajquM33i1U6DXzqvr_ZVVReuMPjFw@mail.gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <847D8710-E102-4EAD-A627-FAC9B4238DA0@stufft.io>
 <CAP7+vJ+EnOLBigWVCug1OajquM33i1U6DXzqvr_ZVVReuMPjFw@mail.gmail.com>
Message-ID: <A957D813-5C59-4C9A-9E42-0E83279C6739@stufft.io>

Yes right, sorry I didn?t mean to imply that all that time was spent in the
Python start up time. I?ve personally never actually spent time to figure out
which part of that was slow because getting visibility inside of a
subprocess.Popen is a pain and I?m slowly trying to rewrite our tests to not
require subprocesses at all.

I was only trying to chime in with a me too here about slow subprocess tests
since our test suite ends up starting a couple thousand subprocesses as well
and those tests are definitely our slowest.

On May 10, 2014, at 6:50 PM, Guido van Rossum <guido at python.org> wrote:

> Yeah, but 200 test in 30 minutes is 9 *seconds* per test -- the Python startup time is only a tiny fraction of that (20-40 *milliseconds*).
> 
> 
> On Sat, May 10, 2014 at 3:33 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> On May 10, 2014, at 5:46 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
>> Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com> a ?crit :
>> > Furthermore, Python 3 appears to be >50% slower than Python 2.
>> 
>> Please mention the minor version. It looks like you compared 2.7 and 3.3. Please test 3.4, we made interesting progress on the startup time.
>> 
>> There is still something to do, especially on OS X. Depending on the OS, different modules are loaded and some functions are implemented differently.
>> 
> 
> For what it's worth pip is the same way, about half of our test suite involves
> invoking (multiple) python processes. This has historically be really slow
> (~30 minutes to run ~200 tests of that type). We've been able to get the wall
> clock run time down by parallelizing these but the sequential time is still
> really slow.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/37e1e4b2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/37e1e4b2/attachment.sig>

From raymond.hettinger at gmail.com  Sun May 11 01:47:30 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 10 May 2014 16:47:30 -0700
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
	and poorly worded warning message.
In-Reply-To: <lkmbum$d9a$1@ger.gmane.org>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
 <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
 <lkmbum$d9a$1@ger.gmane.org>
Message-ID: <C5F655A1-841D-42A1-A883-39EBAE67A149@gmail.com>


On May 10, 2014, at 4:15 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Total +1 on keeping these little bits around.

Since all of you want a warning, I'll add one back
but with improved wording.

I'm not all at comfortable with the wording of the second sentence.
I was the author of the SystemRandom() class and I only want
to guarantee that it provides access to the operating system's
source of random numbers. It is a bold claim to guarantee that
it is cryptographically secure (many such claims in the past have
turned-out to be false).  We don't really know what it is going to
do on a VM for example.

Also, I don't want to call SystemRandom() a pseudo-random number
generator.  It purports to be an actual random number generator
(or at least it purports to have used some real source of entropy at
some stage).  To me (the module maintainer), that is an important distinction.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140510/321007ee/attachment.html>

From tim.peters at gmail.com  Sun May 11 02:47:39 2014
From: tim.peters at gmail.com (Tim Peters)
Date: Sat, 10 May 2014 19:47:39 -0500
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <C5F655A1-841D-42A1-A883-39EBAE67A149@gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
 <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
 <lkmbum$d9a$1@ger.gmane.org> <C5F655A1-841D-42A1-A883-39EBAE67A149@gmail.com>
Message-ID: <CAExdVN=EM-WGskmjQ6RpYYaNxk=+YthHZ8JLm1owq3udvv-OBg@mail.gmail.com>

[Raymond Hettinger]
> ...
> I'm not all at comfortable with the wording of the second sentence.
> I was the author of the SystemRandom() class and I only want
> to guarantee that it provides access to the operating system's
> source of random numbers. It is a bold claim to guarantee that
> it is cryptographically secure (many such claims in the past have
> turned-out to be false).  We don't really know what it is going to
> do on a VM for example.
>
> Also, I don't want to call SystemRandom() a pseudo-random number
> generator.  It purports to be an actual random number generator
> (or at least it purports to have used some real source of entropy at
> some stage).  To me (the module maintainer), that is an important
> distinction.

It should be sufficient to say that SystemRandom() inherits all the
properties of the operating system's os.urandom() implementation, yes?
 Since Python has nothing to do with that, it's most accurate and
helpful to tell the user to refer to their OS urandom docs.

On all platforms I'm aware of (two ;-)), urandom() is in fact a
CSPRNG, not an "actual random number generator".  That's why urandom()
can get away with never blocking, potentially producing bits far
faster than the system random() can accumulate fresh entropy.

But all the hideous details don't belong in the Python docs - they
belong in the OS's urandom docs.  A phrasing I've found helpful is to
tell users that "urandom() is as secure as the people who wrote your
operating system knew how to make it".  Linux users smile then, and
Windows users groan ;-)

From ethan at stoneleaf.us  Sun May 11 07:27:36 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sat, 10 May 2014 22:27:36 -0700
Subject: [Python-Dev] Values and objects
In-Reply-To: <CABicbJ+Mg3gxzRGW7AYCvB2m4BfSbsaQj=eaooL1ueY4-RTqtQ@mail.gmail.com>
References: <235C4BFA-9770-481A-9FCF-21C3F036769C@gmail.com>
 <mailman.9710.1399408799.18130.python-list@python.org>
 <lkbigv$ban$1@speranza.aioe.org>
 <mailman.9713.1399419985.18130.python-list@python.org>
 <lkdt0u$moc$1@speranza.aioe.org> <87ppjpwafk.fsf@elektro.pacujo.net>
 <536ad8f2$0$29965$c3e8da3$5496439d@news.astraweb.com>
 <lkjitj$d0c$1@speranza.aioe.org> <87zjiqbmy5.fsf@elektro.pacujo.net>
 <536d7a7d$0$29980$c3e8da3$5496439d@news.astraweb.com>
 <9cc0ebf9-dbed-4d3d-91fc-2abb9b0103d0@googlegroups.com>
 <mailman.9841.1399689216.18130.python-list@python.org>
 <536dc3f7$0$29980$c3e8da3$5496439d@news.astraweb.com>
 <mailman.9843.1399706518.18130.python-list@python.org>
 <536decca$0$29980$c3e8da3$5496439d@news.astraweb.com>
 <CAPTjJmqynuf6giRvYP5P7SA_g2DH_MhRp-2=w72kLthsFqx74g@mail.gmail.com>
 <536E799D.6080602@stoneleaf.us>
 <CABicbJKte7pgQk03zC4L6Ee71Vfa+O_einCH1OqPDZC0H+UQVw@mail.gmail.com>
 <536E9C3A.7060706@stoneleaf.us>
 <CABicbJ+Mg3gxzRGW7AYCvB2m4BfSbsaQj=eaooL1ueY4-RTqtQ@mail.gmail.
 com>
Message-ID: <536F0A48.5040604@stoneleaf.us>

[bringing back on-list]

On 05/10/2014 07:30 PM, Devin Jeanpierre wrote:
> On Sat, May 10, 2014 at 2:38 PM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> On 05/10/2014 02:03 PM, Devin Jeanpierre wrote:
>>>
>>>
>>> spam is referring to a local variable that has not been bound. This is
>>> not an implementation detail.
>>
>> The implementation detail is that in cpython there is a spot already
>> reserved for what will be the 'spam' variable, as opposed to the module
>> level where no such spots are reserved.
>
> I don't know what a "spot reserved" would even mean in  a language
> spec, so sure. But Chris did not bring up the idea of a "reserved
> spot" in the post you were responding to, and I didn't either. Chris
> claimed that Python had unbound variables, you disagreed, and I
> pointed at the docs, which agree with Chris systematically.

Chris was claiming that the exception UnboundLocalError, with complete text of:

UnboundLocalError: local variable 'not_here' referenced before assignment

is saying that the local variable does exist before it is assigned to;  it is my contention that it does not. 
UnboundLocalError came into existence to aid in debugging a specific subclass of errors [1], and the text was chosen for 
that specific purpose.


>>> Because module level variables work differently from local variables.
>>
>> Not by language definition.  There are pythons where modifying function
>> locals works, but the working or not-working is not a language guarantee.
>
> I did make the mistake of saying "locals", when I meant
> "function-level locals". That was wrong.

Um, what other kind of locals are there?


> See https://docs.python.org/3.4/library/exceptions.html#UnboundLocalError
>
> "Raised when a reference is made to a local variable in a function or
> method, but no value has been bound to that variable."

Hmm, looks like a doc-patch is needed.  ;)

Seriously though, error messages are chosen to provide a simple and clear description that will help the user track down 
what went wrong, not for enshrining in exact detail the language semantics.  Would you really rather have:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 2, in func
UnboundLocalError: the name 'not_here' does not yet exist as you have not yet assigned anything to it so there is 
currently no variable by that name although at some point (in the future of this function, or perhaps in a branch that 
has been passed and did not execute) you will or did assign something to it so it will exist in the future of this 
function or may exist at this point in a future run of this function.

or:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "<stdin>", line 2, in func
UnboundLocalError: local variable 'not_here' referenced before assignment


--
~Ethan~


[1] The conditions being that a variable was referenced before is was created, but the text of NameError offered no help 
in figuring that out ("what do you mean it doesn't exist?!?  It's right *there*!")


From solipsis at pitrou.net  Sun May 11 13:17:48 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 11 May 2014 13:17:48 +0200
Subject: [Python-Dev] devguide: Add myself to developer log and as a
	Windows expert.
References: <3gRBWD3q7bz7Ljd@mail.python.org>
Message-ID: <20140511131748.611ce431@fsol>

On Sun, 11 May 2014 06:04:56 +0200 (CEST)
steve.dower <python-checkins at python.org> wrote:
> http://hg.python.org/devguide/rev/8d5d1f2c7378
> changeset:   698:8d5d1f2c7378
> user:        Steve Dower <steve.dower at microsoft.com>
> date:        Sat May 10 21:01:39 2014 -0700
> summary:
>   Add myself to developer log and as a Windows expert.

Welcome onboard, Steve ! Nice to have an additional Windows expert :-)

Regards

Antoine.



From stephen at xemacs.org  Sun May 11 15:34:20 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sun, 11 May 2014 22:34:20 +0900
Subject: [Python-Dev] [Python-checkins] cpython: Remove the redundant
 and poorly worded warning message.
In-Reply-To: <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
References: <3gR1CR0CDcz7LjY@mail.python.org>
 <CAFRnB2VcF6k9e7xeOp6v212=XFHWawdE4TVfkQiwAeDzd=09JQ@mail.gmail.com>
 <DD389F08-087C-458A-AB43-07ED4B089CBC@gmail.com>
 <20140510235401.1b4774f2@fsol>
 <74049A21-9986-4509-905B-DC9D9E52A7F3@gmail.com>
 <CADiSq7f4hB6bATuxP5aP=WqEaJKwPiBbJgBq0_LJ-D0BOv7DUw@mail.gmail.com>
Message-ID: <874n0wh21v.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > As you point out, most language development teams do very little to
 > try to educate their users about security issues.

That's partly because it isn't going to be terribly effective.
Security is a difficult subject, not one that's going to be usefully
treated in a couple of lines here, a couple more there.  And it is
generally an application issue, not one that is specific to individual
features.

If we're serious about this, I suggest following the RFC pattern:
*every* module's documentation should have a "Security Considerations"
section.  Probably the content will be basically the same as the
existing warning boxes, but with a consistent approach throughout the
docs it could convey the importance of always thinking about security.

 > The consequences of that are clearly visible in the world around
 > us: when security is treated as an optional afterthought,

But (FWIW) that's what warning boxes looks like to me.  An
afterthought.  Not a systematic attempt to encourage security by
teaching about secure programming.  By your own words, we are nowhere
close to a world where "a word, to the wise, is sufficient."


From Steve.Dower at microsoft.com  Sun May 11 15:57:47 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sun, 11 May 2014 13:57:47 +0000
Subject: [Python-Dev] devguide: Add myself to developer log and as
	a	Windows expert.
In-Reply-To: <20140511131748.611ce431@fsol>
References: <3gRBWD3q7bz7Ljd@mail.python.org>,<20140511131748.611ce431@fsol>
Message-ID: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>

Thanks.

For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.)

Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well.

Cheers,
Steve

Top-posted from my Windows Phone
________________________________
From: Antoine Pitrou<mailto:solipsis at pitrou.net>
Sent: ?5/?11/?2014 4:18
To: python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert.

On Sun, 11 May 2014 06:04:56 +0200 (CEST)
steve.dower <python-checkins at python.org> wrote:
> http://hg.python.org/devguide/rev/8d5d1f2c7378
> changeset:   698:8d5d1f2c7378
> user:        Steve Dower <steve.dower at microsoft.com>
> date:        Sat May 10 21:01:39 2014 -0700
> summary:
>   Add myself to developer log and as a Windows expert.

Welcome onboard, Steve ! Nice to have an additional Windows expert :-)

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/19c5fdcf/attachment.html>

From 4kir4.1i at gmail.com  Sun May 11 14:02:31 2014
From: 4kir4.1i at gmail.com (Akira Li)
Date: Sun, 11 May 2014 16:02:31 +0400
Subject: [Python-Dev] should tests be thread-safe?
References: <536D3222.5090109@gmail.com>
 <CAMpsgwaZ8cL1VO+zU70JW5HEGdFNOzoKQtKSENe3oVjikPj3Aw@mail.gmail.com>
Message-ID: <877g5sle08.fsf@gmail.com>

Victor Stinner <victor.stinner at gmail.com> writes:

> If you need a well defined environement, run your test in a subprocess.
> Depending on the random function, your test may be run with more threads.
> On BSD, it changes for example which thread receives a signal. Importing
> the tkinter module creates a "hidden" C thread for the Tk loop.

Does it mean that non-thread-safe tests can't be run using a GUI test
runner that is implemented using tkinter?


--
akira


From eliben at gmail.com  Sun May 11 17:33:15 2014
From: eliben at gmail.com (Eli Bendersky)
Date: Sun, 11 May 2014 08:33:15 -0700
Subject: [Python-Dev] devguide: Add myself to developer log and as a
 Windows expert.
In-Reply-To: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <CAF-Rda_i82VzE1++C6FGt_Hd5Nv2vKe-1=zAhAa7SDm78U7AOQ@mail.gmail.com>

On Sun, May 11, 2014 at 6:57 AM, Steve Dower <Steve.Dower at microsoft.com>wrote:

>   Thanks.
>
> For those who missed the earlier discussions, Martin v. L?wis has handed
> over responsibility for the Windows installers. It sounds like Brett Cannon
> and I are both in a position to build 2.7 right now, and I hope to simplify
> the setup required for 3.5 so that anyone can produce and test the
> installers. (Martin is going to look after the 3.4.x builds.)
>
> Obviously I'm also here to help out with Windows in general. I haven't
> quite managed to get 50% time from my employer (maybe 10%), but my
> management at least is very supportive of my participation and keen to keep
> Python running well.
>

Welcome Steve. It's awesome to have this, at least to some extent,
officially endorsed by Microsoft. Python's Windows support is markedly
better than other similar languages (Perl, Ruby), which I think contributes
a lot to its success.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/1891abb9/attachment.html>

From nad at acm.org  Sun May 11 20:04:37 2014
From: nad at acm.org (Ned Deily)
Date: Sun, 11 May 2014 11:04:37 -0700
Subject: [Python-Dev] devguide: Add myself to developer log and as
	a	Windows expert.
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <nad-B4A474.11043711052014@news.gmane.org>

In article 
<f6144319207e4269b044238e8e05abca at BLUPR03MB389.namprd03.prod.outlook.com
>,
 Steve Dower <Steve.Dower at microsoft.com> wrote:
> For those who missed the earlier discussions, Martin v. Lowis has handed over 
> responsibility for the Windows installers.

Welcome to the the release team!

--Ned

-- 
 Ned Deily,
 nad at acm.org


From ncoghlan at gmail.com  Mon May 12 00:07:22 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 12 May 2014 08:07:22 +1000
Subject: [Python-Dev] devguide: Add myself to developer log and as a
 Windows expert.
In-Reply-To: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <CADiSq7eq_gP7gkY2EuXPhjws6MecJjD0nyhmFkARCjZz5GVVAQ@mail.gmail.com>

On 11 May 2014 23:59, "Steve Dower" <Steve.Dower at microsoft.com> wrote:
>
> Thanks.

Indeed, welcome!

> For those who missed the earlier discussions, Martin v. L?wis has handed
over responsibility for the Windows installers. It sounds like Brett Cannon
and I are both in a position to build 2.7 right now,

The other BC: Brian Curtin :)

> Obviously I'm also here to help out with Windows in general. I haven't
quite managed to get 50% time from my employer (maybe 10%), but my
management at least is very supportive of my participation and keen to keep
Python running well.

Great to hear!

Cheers,
Nick.

>
> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> ________________________________
> From: Antoine Pitrou
> Sent: ?5/?11/?2014 4:18
> To: python-dev at python.org
> Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a
Windows expert.
>
> On Sun, 11 May 2014 06:04:56 +0200 (CEST)
> steve.dower <python-checkins at python.org> wrote:
> > http://hg.python.org/devguide/rev/8d5d1f2c7378
> > changeset:   698:8d5d1f2c7378
> > user:        Steve Dower <steve.dower at microsoft.com>
> > date:        Sat May 10 21:01:39 2014 -0700
> > summary:
> >   Add myself to developer log and as a Windows expert.
>
> Welcome onboard, Steve ! Nice to have an additional Windows expert :-)
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140512/88a6e5bc/attachment.html>

From raymond.hettinger at gmail.com  Mon May 12 04:15:46 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 11 May 2014 19:15:46 -0700
Subject: [Python-Dev] Patch for robotparser.py
Message-ID: <078F3CBC-98EC-44DC-A2E1-6E622BB258B4@gmail.com>

If there is anyone here with an interest in web-spiders,
it would be nice if someone else could take a look at

    http://bugs.python.org/issue21469

which addresses the risk of false positives with the robots.txt parser.


Raymond


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/b234a720/attachment.html>

From raymond.hettinger at gmail.com  Mon May 12 05:30:52 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 11 May 2014 20:30:52 -0700
Subject: [Python-Dev] devguide: Add myself to developer log and as a
	Windows expert.
In-Reply-To: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org>, <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <238BC768-3DE1-4967-81E6-08DAB945EF20@gmail.com>


On May 11, 2014, at 6:57 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:

> Thanks.
> 
> For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.)
> 
> Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well.

Welcome to the party. :-)


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140511/eebce7ba/attachment.html>

From rdmurray at bitdance.com  Mon May 12 20:43:22 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Mon, 12 May 2014 14:43:22 -0400
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-checkins=5D_cpython_=28merge_3?=
	=?utf-8?q?=2E4_-=3E_default=29=3A_Merge_3=2E4-=3Edefault=3A_asynci?=
	=?utf-8?q?o=3A_Fix_upstream_issue_168=3A_StreamReader=2Eread=28-1?=
	=?utf-8?q?=29_from?=
In-Reply-To: <3gS7nC2qJXz7LjP@mail.python.org>
References: <3gS7nC2qJXz7LjP@mail.python.org>
Message-ID: <20140512184322.A9D01250D4E@webabinitio.net>

These changes appear to have caused several builbot failures, and
there doesn't appear to be a bugs.python.org issue to report it to.
One failure example:

    http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119

test_asyncio fails similarly for me on tip.

On Mon, 12 May 2014 19:05:19 +0200, guido.van.rossum <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/2af5a52b9b87
> changeset:   90663:2af5a52b9b87
> parent:      90661:9493fdad2a75
> parent:      90662:909ea8cc86bb
> user:        Guido van Rossum <guido at python.org>
> date:        Mon May 12 10:05:04 2014 -0700
> summary:
>   Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1) from pipe may hang if data exceeds buffer limit.
> 
> files:
>   Lib/asyncio/streams.py                |  17 ++++--
>   Lib/test/test_asyncio/test_streams.py |  36 +++++++++++++++
>   2 files changed, 47 insertions(+), 6 deletions(-)
> 
> 
> diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py
> --- a/Lib/asyncio/streams.py
> +++ b/Lib/asyncio/streams.py
> @@ -419,12 +419,17 @@
>              return b''
>  
>          if n < 0:
> -            while not self._eof:
> -                self._waiter = self._create_waiter('read')
> -                try:
> -                    yield from self._waiter
> -                finally:
> -                    self._waiter = None
> +            # This used to just loop creating a new waiter hoping to
> +            # collect everything in self._buffer, but that would
> +            # deadlock if the subprocess sends more than self.limit
> +            # bytes.  So just call self.read(self._limit) until EOF.
> +            blocks = []
> +            while True:
> +                block = yield from self.read(self._limit)
> +                if not block:
> +                    break
> +                blocks.append(block)
> +            return b''.join(blocks)
>          else:
>              if not self._buffer and not self._eof:
>                  self._waiter = self._create_waiter('read')
> diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py
> --- a/Lib/test/test_asyncio/test_streams.py
> +++ b/Lib/test/test_asyncio/test_streams.py
> @@ -1,7 +1,9 @@
>  """Tests for streams.py."""
>  
>  import gc
> +import os
>  import socket
> +import sys
>  import unittest
>  from unittest import mock
>  try:
> @@ -583,6 +585,40 @@
>              server.stop()
>              self.assertEqual(msg, b"hello world!\n")
>  
> +    @unittest.skipIf(sys.platform == 'win32', "Don't have pipes")
> +    def test_read_all_from_pipe_reader(self):
> +        # See Tulip issue 168.  This test is derived from the example
> +        # subprocess_attach_read_pipe.py, but we configure the
> +        # StreamReader's limit so that twice it is less than the size
> +        # of the data writter.  Also we must explicitly attach a child
> +        # watcher to the event loop.
> +
> +        watcher = asyncio.get_child_watcher()
> +        watcher.attach_loop(self.loop)
> +
> +        code = """\
> +import os, sys
> +fd = int(sys.argv[1])
> +os.write(fd, b'data')
> +os.close(fd)
> +"""
> +        rfd, wfd = os.pipe()
> +        args = [sys.executable, '-c', code, str(wfd)]
> +
> +        pipe = open(rfd, 'rb', 0)
> +        reader = asyncio.StreamReader(loop=self.loop, limit=1)
> +        protocol = asyncio.StreamReaderProtocol(reader, loop=self.loop)
> +        transport, _ = self.loop.run_until_complete(
> +            self.loop.connect_read_pipe(lambda: protocol, pipe))
> +
> +        proc = self.loop.run_until_complete(
> +            asyncio.create_subprocess_exec(*args, pass_fds={wfd}, loop=self.loop))
> +        self.loop.run_until_complete(proc.wait())
> +
> +        os.close(wfd)
> +        data = self.loop.run_until_complete(reader.read(-1))
> +        self.assertEqual(data, b'data')
> +
>  
>  if __name__ == '__main__':
>      unittest.main()
> 
> -- 
> Repository URL: http://hg.python.org/cpython
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins

From ericsnowcurrently at gmail.com  Tue May 13 00:59:29 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 12 May 2014 16:59:29 -0600
Subject: [Python-Dev] devguide: Add myself to developer log and as a
 Windows expert.
In-Reply-To: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <CALFfu7DudvHW-vqBVWmUeFF-JRBFfOkLoYjOSUY2s0i9qMrvGw@mail.gmail.com>

On Sun, May 11, 2014 at 7:57 AM, Steve Dower <Steve.Dower at microsoft.com> wrote:
> For those who missed the earlier discussions, Martin v. L?wis has handed
> over responsibility for the Windows installers. It sounds like Brett Cannon
> and I are both in a position to build 2.7 right now, and I hope to simplify
> the setup required for 3.5 so that anyone can produce and test the
> installers. (Martin is going to look after the 3.4.x builds.)

You're a welcome addition to the group!

>
> Obviously I'm also here to help out with Windows in general. I haven't quite
> managed to get 50% time from my employer (maybe 10%), but my management at
> least is very supportive of my participation and keen to keep Python running
> well.

That's great news.  Hopefully it's a growing trend. :)

-eric

From pcmanticore at gmail.com  Tue May 13 00:16:36 2014
From: pcmanticore at gmail.com (Claudiu Popa)
Date: Tue, 13 May 2014 01:16:36 +0300
Subject: [Python-Dev] Questions regarding Windows buildbots
Message-ID: <CAMy=CLpAOMqZm6s8nAqVYd8fJLCQ_oSzwhTHCZkK=Si63bGUBw@mail.gmail.com>

Hello!

I'm working on a patch for issue bugs.python.org/issue8579 (Add
missing tests for FlushKey, LoadKey, and SaveKey in winreg). This
issue requires the SeBackupPrivilege in order to use LoadKey and
SaveKey. While acquiring the privilege
isn't very complicated using ctypes, it fails with
ERROR_NOT_ALL_ASSIGNED (1300) when the
user has Administrative privileges, but it's not an Administrator,
problem which can be eluded by
running the script with elevated privileges. This leads me to a couple
of questions:

- should a Windows test be skipped if it can't acquire a certain privilege?

- If we can acquire the privilege by elevating our process, does the
Windows buildbots have UAC
 enabled and if so, how's the notification setting configured? For
instance, elevating a process will
 trigger a new UAC window with the message "Do you want to allow the
following program from an unknown publisher to make  changes to this
computer?" on the recommended configuration, but this doesn't happen
when the configuration is set to  "Never notify".


Thank you.

From brian at python.org  Tue May 13 01:21:06 2014
From: brian at python.org (Brian Curtin)
Date: Mon, 12 May 2014 18:21:06 -0500
Subject: [Python-Dev] Questions regarding Windows buildbots
In-Reply-To: <CAMy=CLpAOMqZm6s8nAqVYd8fJLCQ_oSzwhTHCZkK=Si63bGUBw@mail.gmail.com>
References: <CAMy=CLpAOMqZm6s8nAqVYd8fJLCQ_oSzwhTHCZkK=Si63bGUBw@mail.gmail.com>
Message-ID: <CAD+XWwrViqzJ48R8W_6aOmMc8Z+TKDVMO1kuQH2BvTi4++JNAQ@mail.gmail.com>

On Mon, May 12, 2014 at 5:16 PM, Claudiu Popa <pcmanticore at gmail.com> wrote:
> Hello!
>
> I'm working on a patch for issue bugs.python.org/issue8579 (Add
> missing tests for FlushKey, LoadKey, and SaveKey in winreg). This
> issue requires the SeBackupPrivilege in order to use LoadKey and
> SaveKey. While acquiring the privilege
> isn't very complicated using ctypes, it fails with
> ERROR_NOT_ALL_ASSIGNED (1300) when the
> user has Administrative privileges, but it's not an Administrator,
> problem which can be eluded by
> running the script with elevated privileges. This leads me to a couple
> of questions:
>
> - should a Windows test be skipped if it can't acquire a certain privilege?

Yes. Check out any of the os.symlink tests - they're currently skipped
when the symlink privilege isn't held.

> - If we can acquire the privilege by elevating our process, does the
> Windows buildbots have UAC
>  enabled and if so, how's the notification setting configured? For
> instance, elevating a process will
>  trigger a new UAC window with the message "Do you want to allow the
> following program from an unknown publisher to make  changes to this
> computer?" on the recommended configuration, but this doesn't happen
> when the configuration is set to  "Never notify".

That probably depends on how each machine is setup. If they happen to
get blocked on any individual slave, we'll just have to ask the owner
to change that setting.

Currently there are no Windows build slaves running as administrator.
I used to have one but the machine died and I never replaced it. I
also said a few months ago that I would get one setup again, but that
hasn't happened yet. I can get a new machine up and running but
probably not until next week as I'm at a conference.

From gregory.szorc at gmail.com  Tue May 13 01:22:52 2014
From: gregory.szorc at gmail.com (Gregory Szorc)
Date: Mon, 12 May 2014 16:22:52 -0700
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
Message-ID: <537157CC.1070603@gmail.com>

On 5/10/2014 2:46 PM, Victor Stinner wrote:
> Le 10 mai 2014 22:51, "Gregory Szorc" <gregory.szorc at gmail.com 
> <mailto:gregory.szorc at gmail.com>> a ?crit :
>> Furthermore, Python 3 appears to be >50% slower than Python 2.
> 
> Please mention the minor version. It looks like you compared 2.7
> and 3.3. Please test 3.4, we made interesting progress on the
> startup time.
> 
> There is still something to do, especially on OS X. Depending on
> the OS, different modules are loaded and some functions are
> implemented differently.

3.4.0 does appear to be faster than 3.3.5 on Linux - `python -c ''` is
taking ~50ms (as opposed to ~60ms) on my i7-2600K. Great to see!

But 3.4.0 is still slower than 2.7.6. And all versions of CPython are
over 3x slower than Perl 5.18.2. This difference amounts to minutes of
CPU time when thousands of processes are involved. That seems
excessive to me.

Why can't Python start as quickly as Perl or Ruby?

From ericsnowcurrently at gmail.com  Tue May 13 02:13:09 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Mon, 12 May 2014 18:13:09 -0600
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default):
 Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1)
 from
In-Reply-To: <20140512184322.A9D01250D4E@webabinitio.net>
References: <3gS7nC2qJXz7LjP@mail.python.org>
 <20140512184322.A9D01250D4E@webabinitio.net>
Message-ID: <CALFfu7AEbrZPMHi78+G5tMhpY3rZUOogGH8wGr+wFewXu2Kupw@mail.gmail.com>

On Mon, May 12, 2014 at 12:43 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> These changes appear to have caused several builbot failures, and
> there doesn't appear to be a bugs.python.org issue to report it to.
> One failure example:
>
>     http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119
>
> test_asyncio fails similarly for me on tip.

Same for me on 3.4.

-eric

From dw+python-dev at hmmz.org  Tue May 13 02:19:18 2014
From: dw+python-dev at hmmz.org (dw+python-dev at hmmz.org)
Date: Tue, 13 May 2014 00:19:18 +0000
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <537157CC.1070603@gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <537157CC.1070603@gmail.com>
Message-ID: <20140513001918.GA734@k2>

On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:

> Why can't Python start as quickly as Perl or Ruby?

On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from
81ms startup to 20ms by specifying -S, which disables site.py.

Oblitering the .pth files immediately knocks 10ms off regular startup. I
guess the question isn't why Python is slower than perl, but what
aspects of site.py could be cached, reimplemented, or stripped out
entirely.  I'd personally love to see .pth support removed.


David

From guido at python.org  Tue May 13 02:26:35 2014
From: guido at python.org (Guido van Rossum)
Date: Mon, 12 May 2014 17:26:35 -0700
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.4 -> default):
 Merge 3.4->default: asyncio: Fix upstream issue 168: StreamReader.read(-1)
 from
In-Reply-To: <CALFfu7AEbrZPMHi78+G5tMhpY3rZUOogGH8wGr+wFewXu2Kupw@mail.gmail.com>
References: <3gS7nC2qJXz7LjP@mail.python.org>
 <20140512184322.A9D01250D4E@webabinitio.net>
 <CALFfu7AEbrZPMHi78+G5tMhpY3rZUOogGH8wGr+wFewXu2Kupw@mail.gmail.com>
Message-ID: <CAP7+vJK1pVs3FxW11y5wX3R2H7ptDXULiHkwfLO=_POq-+ywMQ@mail.gmail.com>

Sorry about that. I will look into it soon, but won't have time right away.
(Also I missed David's earlier message.)
On May 12, 2014 5:14 PM, "Eric Snow" <ericsnowcurrently at gmail.com> wrote:

> On Mon, May 12, 2014 at 12:43 PM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> > These changes appear to have caused several builbot failures, and
> > there doesn't appear to be a bugs.python.org issue to report it to.
> > One failure example:
> >
> >
> http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/119
> >
> > test_asyncio fails similarly for me on tip.
>
> Same for me on 3.4.
>
> -eric
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140512/9fb23fd4/attachment.html>

From ncoghlan at gmail.com  Tue May 13 05:43:20 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 13 May 2014 13:43:20 +1000
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <20140513001918.GA734@k2>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <537157CC.1070603@gmail.com> <20140513001918.GA734@k2>
Message-ID: <CADiSq7f=zVn62UQjJVOmnev33P374kU-BzK8mXMCLrcjzPFzAA@mail.gmail.com>

On 13 May 2014 10:19,  <dw+python-dev at hmmz.org> wrote:
> On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:
>
>> Why can't Python start as quickly as Perl or Ruby?
>
> On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops from
> 81ms startup to 20ms by specifying -S, which disables site.py.
>
> Oblitering the .pth files immediately knocks 10ms off regular startup. I
> guess the question isn't why Python is slower than perl, but what
> aspects of site.py could be cached, reimplemented, or stripped out
> entirely.  I'd personally love to see .pth support removed.

The startup code is currently underspecified and underdocumented, and
quite fragile as a result. It represents 20+ years of organic growth
without any systematic refactoring to simplify and streamline things.

That's what PEP 432 aims to address, and is something I now expect to
have time to get back to for Python 3.5. And yes, one thing those
changes should enable is the creation of system Python runtimes on
Linux that initialise faster than the current implementation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From me at the-compiler.org  Tue May 13 16:49:08 2014
From: me at the-compiler.org (Florian Bruhin)
Date: Tue, 13 May 2014 16:49:08 +0200
Subject: [Python-Dev] ConfigParser mangles keys with special chars
In-Reply-To: <20140425142206.GD6735@lupin>
References: <20140425142206.GD6735@lupin>
Message-ID: <20140513144907.GA6735@lupin>

* Florian Bruhin <me at the-compiler.org> [2014-04-25 16:22:06 +0200]:
> I noticed configparser does behave in a surprising way when a key
> has a special meaning in ini-format.

I've now submitted an issue here: http://bugs.python.org/issue21498

Florian

-- 
() ascii ribbon campaign - stop html mail    www.asciiribbon.org
/\ www.the-compiler.org  | I love long mails http://email.is-not-s.ms/
Blessed are the forgetful: for they get the better even of their blunders. -- 
Nietzsche 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140513/596c6de1/attachment.sig>

From gregory.szorc at gmail.com  Tue May 13 22:58:05 2014
From: gregory.szorc at gmail.com (Gregory Szorc)
Date: Tue, 13 May 2014 13:58:05 -0700
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <CADiSq7f=zVn62UQjJVOmnev33P374kU-BzK8mXMCLrcjzPFzAA@mail.gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <537157CC.1070603@gmail.com> <20140513001918.GA734@k2>
 <CADiSq7f=zVn62UQjJVOmnev33P374kU-BzK8mXMCLrcjzPFzAA@mail.gmail.com>
Message-ID: <5372875D.8080601@gmail.com>

On 5/12/2014 8:43 PM, Nick Coghlan wrote:
> On 13 May 2014 10:19,  <dw+python-dev at hmmz.org> wrote:
>> On Mon, May 12, 2014 at 04:22:52PM -0700, Gregory Szorc wrote:
>> 
>>> Why can't Python start as quickly as Perl or Ruby?
>> 
>> On my heavily abused Core 2 Macbook with 9 .pth files, 2.7 drops
>> from 81ms startup to 20ms by specifying -S, which disables
>> site.py.
>> 
>> Oblitering the .pth files immediately knocks 10ms off regular
>> startup. I guess the question isn't why Python is slower than
>> perl, but what aspects of site.py could be cached, reimplemented,
>> or stripped out entirely.  I'd personally love to see .pth
>> support removed.
> 
> The startup code is currently underspecified and underdocumented,
> and quite fragile as a result. It represents 20+ years of organic
> growth without any systematic refactoring to simplify and
> streamline things.
> 
> That's what PEP 432 aims to address, and is something I now expect
> to have time to get back to for Python 3.5. And yes, one thing
> those changes should enable is the creation of system Python
> runtimes on Linux that initialise faster than the current
> implementation.

This is terrific news and something I greatly anticipate taking
advantage of!

But the great many of us still on 2.7 likely won't see a benefit,
correct? How insane would it be for people to do things like pass -S
in the shebang and manually implement the parts of site.py that are
actually needed?

From stephen at xemacs.org  Wed May 14 04:33:42 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 14 May 2014 11:33:42 +0900
Subject: [Python-Dev] python process creation overhead
In-Reply-To: <5372875D.8080601@gmail.com>
References: <536E86A2.600@gmail.com>
 <CAMpsgwb2A5VoUMGxFrYvke2pyapxHSR0GwLOj5-qxAiydf4ocw@mail.gmail.com>
 <537157CC.1070603@gmail.com> <20140513001918.GA734@k2>
 <CADiSq7f=zVn62UQjJVOmnev33P374kU-BzK8mXMCLrcjzPFzAA@mail.gmail.com>
 <5372875D.8080601@gmail.com>
Message-ID: <87sioddr7d.fsf@uwakimon.sk.tsukuba.ac.jp>

Gregory Szorc writes:

 > But the great many of us still on 2.7 likely won't see a benefit,
 > correct? How insane would it be for people to do things like pass -S
 > in the shebang and manually implement the parts of site.py that are
 > actually needed?

Well, since it probably won't work ....<wink/>  That is to say,
site.py typically provides different facilities to different programs
-- that's why some parts of it show up as unneeded.  So you need to
carefully analyze *each* *subprocess* that you propose to invoke with
-S and determine which parts of site.py it needs.

In most cases I suspect you would better look at alternatives that
avoid invoking a subprocess per task, but instead maintains a pool of
worker processes (or threads).  You might even be able to save network
traffic or IPC by caching replies to common requests in the worker
processes, which may save more per task than process invocation.

Even where -S makes sense, I would suggest invoking "python -S"
explicitly from the parent process rather than munging the shebang in
the children.  (You do need to be careful to audit changes in the
child programs to be sure they aren't changed in ways that change
site.py requirements.  Putting -S in the shebang may help catch such
problems early.)


From bcannon at gmail.com  Wed May 14 16:20:26 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Wed, 14 May 2014 14:20:26 +0000
Subject: [Python-Dev] Where is our official policy of what platforms we do
	support?
Message-ID: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>

Over the past week or so there have been 2 patches to add support for
various UNIX OSs. Now I thought we had stopped trying to add new esoteric
OSs (e.g. I had never heard of MirOS until the patch for it came in), but I
can't find a PEP that spells out what it takes to get a platform supported (
http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
not keeping them or adding them unless you are re-adding one which
apparently just takes a volunteer).

Do we want an official policy written down in a PEP (yes, I can write it)?
Should I keep closing these patches and saying that we are not adding
support for new operating systems and be hand-wavy about it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140514/219be54c/attachment.html>

From barry at python.org  Wed May 14 16:28:20 2014
From: barry at python.org (Barry Warsaw)
Date: Wed, 14 May 2014 10:28:20 -0400
Subject: [Python-Dev] Where is our official policy of what platforms we
 do support?
In-Reply-To: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
Message-ID: <20140514102820.0e13a141@anarchist.wooz.org>

On May 14, 2014, at 02:20 PM, Brett Cannon wrote:

>Do we want an official policy written down in a PEP (yes, I can write it)?
>Should I keep closing these patches and saying that we are not adding
>support for new operating systems and be hand-wavy about it?

Yes, I think a PEP describing both policy and implementation (i.e. which
platforms we officially support) is worthwhile.  While I think you could write
a new PEP, I also think it might just make sense to co-opt PEP 11 and broaden
its scope, since as you say, removing support is only half the story.

Cheers,
-Barry

From jsbueno at python.org.br  Wed May 14 16:31:15 2014
From: jsbueno at python.org.br (Joao S. O. Bueno)
Date: Wed, 14 May 2014 11:31:15 -0300
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
Message-ID: <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>

+1 for an official policy that comes with a "permanent maintainer for
this platform required"  as part of the list
of requisites.

  js
 -><-

On 14 May 2014 11:20, Brett Cannon <bcannon at gmail.com> wrote:
> Over the past week or so there have been 2 patches to add support for
> various UNIX OSs. Now I thought we had stopped trying to add new esoteric
> OSs (e.g. I had never heard of MirOS until the patch for it came in), but I
> can't find a PEP that spells out what it takes to get a platform supported
> (http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
> not keeping them or adding them unless you are re-adding one which
> apparently just takes a volunteer).
>
> Do we want an official policy written down in a PEP (yes, I can write it)?
> Should I keep closing these patches and saying that we are not adding
> support for new operating systems and be hand-wavy about it?
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jsbueno%40python.org.br
>

From solipsis at pitrou.net  Wed May 14 16:42:47 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 14 May 2014 16:42:47 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
Message-ID: <20140514164247.18b252e5@fsol>

On Wed, 14 May 2014 14:20:26 +0000
Brett Cannon <bcannon at gmail.com> wrote:
> Over the past week or so there have been 2 patches to add support for
> various UNIX OSs. Now I thought we had stopped trying to add new esoteric
> OSs (e.g. I had never heard of MirOS until the patch for it came in), but I
> can't find a PEP that spells out what it takes to get a platform supported (
> http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
> not keeping them or adding them unless you are re-adding one which
> apparently just takes a volunteer).

OTOH you can fix a platform bug without officially supporting it. If
someone files an OpenBSD-specific patch, it may make sense to commit it
even without officially supporting OpenBSD. In practice it all depends
on how intrusive / reasonable the patch is, and whether it is working
around a platform-specific bug rather than a standards-compliant
limitation.

(we could call those "stochastically supported platforms" :-))

Regards

Antoine.



From bcannon at gmail.com  Wed May 14 16:45:20 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Wed, 14 May 2014 14:45:20 +0000
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <20140514164247.18b252e5@fsol>
Message-ID: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>

On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou <solipsis at pitrou.net>
wrote:

> On Wed, 14 May 2014 14:20:26 +0000
> Brett Cannon <bcannon at gmail.com> wrote:
> > Over the past week or so there have been 2 patches to add support for
> > various UNIX OSs. Now I thought we had stopped trying to add new esoteric
> > OSs (e.g. I had never heard of MirOS until the patch for it came in),
> but I
> > can't find a PEP that spells out what it takes to get a platform
> supported (
> > http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
> > not keeping them or adding them unless you are re-adding one which
> > apparently just takes a volunteer).
>
> OTOH you can fix a platform bug without officially supporting it. If
> someone files an OpenBSD-specific patch, it may make sense to commit it
> even without officially supporting OpenBSD. In practice it all depends
> on how intrusive / reasonable the patch is, and whether it is working
> around a platform-specific bug rather than a standards-compliant
> limitation.
>
> (we could call those "stochastically supported platforms" :-))
>

Very true, but these patches are all for e.g. configure to recognize a
specific platform by listing them in some constant. Changing code to be
more general I have no issue with since that's just good practice.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140514/b78008ca/attachment.html>

From rdmurray at bitdance.com  Wed May 14 16:56:36 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Wed, 14 May 2014 10:56:36 -0400
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
Message-ID: <20140514145650.C0E17250D4F@webabinitio.net>

On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" <jsbueno at python.org.br> wrote:
> +1 for an official policy that comes with a "permanent maintainer for
> this platform required"  as part of the list
> of requisites.
> 
>   js
>  -><-
> 
> On 14 May 2014 11:20, Brett Cannon <bcannon at gmail.com> wrote:
> > Over the past week or so there have been 2 patches to add support for
> > various UNIX OSs. Now I thought we had stopped trying to add new esoteric
> > OSs (e.g. I had never heard of MirOS until the patch for it came in), but I
> > can't find a PEP that spells out what it takes to get a platform supported
> > (http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
> > not keeping them or adding them unless you are re-adding one which
> > apparently just takes a volunteer).
> >
> > Do we want an official policy written down in a PEP (yes, I can write it)?
> > Should I keep closing these patches and saying that we are not adding
> > support for new operating systems and be hand-wavy about it?

In addition to a maintainer (who I think doesn't have to be a committer,
though that would be ideal), I think a maintained buildbot should be a
requirement for formal support.

--David

From bcannon at gmail.com  Wed May 14 17:08:25 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Wed, 14 May 2014 15:08:25 +0000
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
 <20140514145650.C0E17250D4F@webabinitio.net>
Message-ID: <CAP1=2W6R1vKwthsftRKeoeYsP89jQoyJtUFxLwdeVeEs2=fJrg@mail.gmail.com>

On Wed May 14 2014 at 11:02:50 AM, R. David Murray <rdmurray at bitdance.com>
wrote:

> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" <
> jsbueno at python.org.br> wrote:
> > +1 for an official policy that comes with a "permanent maintainer for
> > this platform required"  as part of the list
> > of requisites.
> >
> >   js
> >  -><-
> >
> > On 14 May 2014 11:20, Brett Cannon <bcannon at gmail.com> wrote:
> > > Over the past week or so there have been 2 patches to add support for
> > > various UNIX OSs. Now I thought we had stopped trying to add new
> esoteric
> > > OSs (e.g. I had never heard of MirOS until the patch for it came in),
> but I
> > > can't find a PEP that spells out what it takes to get a platform
> supported
> > > (http://legacy.python.org/dev/peps/pep-0011/ is about removing
> platforms,
> > > not keeping them or adding them unless you are re-adding one which
> > > apparently just takes a volunteer).
> > >
> > > Do we want an official policy written down in a PEP (yes, I can write
> it)?
> > > Should I keep closing these patches and saying that we are not adding
> > > support for new operating systems and be hand-wavy about it?
>
> In addition to a maintainer (who I think doesn't have to be a committer,
> though that would be ideal), I think a maintained buildbot should be a
> requirement for formal support.
>

I would think someone how is/would be a core dev and a *stable* buildbot
are requirements.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140514/cf7dace7/attachment.html>

From doko at ubuntu.com  Wed May 14 17:30:49 2014
From: doko at ubuntu.com (Matthias Klose)
Date: Wed, 14 May 2014 17:30:49 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
 do support?
In-Reply-To: <CAP1=2W6R1vKwthsftRKeoeYsP89jQoyJtUFxLwdeVeEs2=fJrg@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
 <20140514145650.C0E17250D4F@webabinitio.net>
 <CAP1=2W6R1vKwthsftRKeoeYsP89jQoyJtUFxLwdeVeEs2=fJrg@mail.gmail.com>
Message-ID: <53738C29.7050008@ubuntu.com>

Am 14.05.2014 17:08, schrieb Brett Cannon:
> On Wed May 14 2014 at 11:02:50 AM, R. David Murray <rdmurray at bitdance.com>
> wrote:
> 
>> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" <
>> jsbueno at python.org.br> wrote:
>>> +1 for an official policy that comes with a "permanent maintainer for
>>> this platform required"  as part of the list
>>> of requisites.
>>>
>>>   js
>>>  -><-
>>>
>>> On 14 May 2014 11:20, Brett Cannon <bcannon at gmail.com> wrote:
>>>> Over the past week or so there have been 2 patches to add support for
>>>> various UNIX OSs. Now I thought we had stopped trying to add new
>> esoteric
>>>> OSs (e.g. I had never heard of MirOS until the patch for it came in),
>> but I
>>>> can't find a PEP that spells out what it takes to get a platform
>> supported
>>>> (http://legacy.python.org/dev/peps/pep-0011/ is about removing
>> platforms,
>>>> not keeping them or adding them unless you are re-adding one which
>>>> apparently just takes a volunteer).
>>>>
>>>> Do we want an official policy written down in a PEP (yes, I can write
>> it)?
>>>> Should I keep closing these patches and saying that we are not adding
>>>> support for new operating systems and be hand-wavy about it?
>>
>> In addition to a maintainer (who I think doesn't have to be a committer,
>> though that would be ideal), I think a maintained buildbot should be a
>> requirement for formal support.
>>
> 
> I would think someone how is/would be a core dev and a *stable* buildbot
> are requirements.

so, are aarch64-linux-gnu and powerpc64le-linux-gnu supported?  I assume there
are no buildbots and there won't be any for a long time. Otoh various distros do
ship python on these architectures. Or are these just new architectures for an
existing platform?  If yes, then we should ask about architecture support too.
The most prominent linux example are some RTLD constants which differ across
some architectures.

  Matthias


From bcannon at gmail.com  Wed May 14 17:52:36 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Wed, 14 May 2014 15:52:36 +0000
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
 <20140514145650.C0E17250D4F@webabinitio.net>
 <CAP1=2W6R1vKwthsftRKeoeYsP89jQoyJtUFxLwdeVeEs2=fJrg@mail.gmail.com>
 <53738C29.7050008@ubuntu.com>
Message-ID: <CAP1=2W4LGLt3bW=-oF548=Vug273opxfCxRqxF=rYzpvbBQDsw@mail.gmail.com>

On Wed May 14 2014 at 11:33:27 AM, Matthias Klose <doko at ubuntu.com> wrote:

> Am 14.05.2014 17:08, schrieb Brett Cannon:
> > On Wed May 14 2014 at 11:02:50 AM, R. David Murray <
> rdmurray at bitdance.com>
> > wrote:
> >
> >> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" <
> >> jsbueno at python.org.br> wrote:
> >>> +1 for an official policy that comes with a "permanent maintainer for
> >>> this platform required"  as part of the list
> >>> of requisites.
> >>>
> >>>   js
> >>>  -><-
> >>>
> >>> On 14 May 2014 11:20, Brett Cannon <bcannon at gmail.com> wrote:
> >>>> Over the past week or so there have been 2 patches to add support for
> >>>> various UNIX OSs. Now I thought we had stopped trying to add new
> >> esoteric
> >>>> OSs (e.g. I had never heard of MirOS until the patch for it came in),
> >> but I
> >>>> can't find a PEP that spells out what it takes to get a platform
> >> supported
> >>>> (http://legacy.python.org/dev/peps/pep-0011/ is about removing
> >> platforms,
> >>>> not keeping them or adding them unless you are re-adding one which
> >>>> apparently just takes a volunteer).
> >>>>
> >>>> Do we want an official policy written down in a PEP (yes, I can write
> >> it)?
> >>>> Should I keep closing these patches and saying that we are not adding
> >>>> support for new operating systems and be hand-wavy about it?
> >>
> >> In addition to a maintainer (who I think doesn't have to be a committer,
> >> though that would be ideal), I think a maintained buildbot should be a
> >> requirement for formal support.
> >>
> >
> > I would think someone how is/would be a core dev and a *stable* buildbot
> > are requirements.
>
> so, are aarch64-linux-gnu and powerpc64le-linux-gnu supported?  I assume
> there
> are no buildbots and there won't be any for a long time. Otoh various
> distros do
> ship python on these architectures. Or are these just new architectures
> for an
> existing platform?  If yes, then we should ask about architecture support
> too.
> The most prominent linux example are some RTLD constants which differ
> across
> some architectures.
>

I consider CPU and compiler separate things. As long as we have a buildbot
covering the CPU or compiler somehow I say they are covered (and someone is
willing to help make sure they continue to work). I'm not going to say that
we need a BSD ARM buildbot and a Linux ARM machine; having *a* machine with
ARM should be enough to shake out most arch-specific issues IMO. Same goes
with compilers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140514/4b1e781c/attachment.html>

From skip at pobox.com  Wed May 14 18:04:36 2014
From: skip at pobox.com (Skip Montanaro)
Date: Wed, 14 May 2014 11:04:36 -0500
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
Message-ID: <CANc-5UxpLa9BPsOsw6z7BLz2g30F0S02GZ2_Lj=2eSw16TBgow@mail.gmail.com>

I wonder if one or more people who maintain unofficial forks on
minority platforms (OS/2, VMS, etc) could create an informational PEP
about the process (benefits and pitfalls) of that kind of effort?

Skip

From db3l.net at gmail.com  Wed May 14 23:36:30 2014
From: db3l.net at gmail.com (David Bolen)
Date: Wed, 14 May 2014 17:36:30 -0400
Subject: [Python-Dev] Questions regarding Windows buildbots
References: <CAMy=CLpAOMqZm6s8nAqVYd8fJLCQ_oSzwhTHCZkK=Si63bGUBw@mail.gmail.com>
 <CAD+XWwrViqzJ48R8W_6aOmMc8Z+TKDVMO1kuQH2BvTi4++JNAQ@mail.gmail.com>
Message-ID: <m2tx8skppd.fsf@valheru.db3l.homeip.net>

Brian Curtin <brian at python.org> writes:

> On Mon, May 12, 2014 at 5:16 PM, Claudiu Popa <pcmanticore at gmail.com> wrote:
(...)
>> - If we can acquire the privilege by elevating our process, does the
>> Windows buildbots have UAC
>>  enabled and if so, how's the notification setting configured? For
>> instance, elevating a process will
>>  trigger a new UAC window with the message "Do you want to allow the
>> following program from an unknown publisher to make  changes to this
>> computer?" on the recommended configuration, but this doesn't happen
>> when the configuration is set to  "Never notify".
>
> That probably depends on how each machine is setup. If they happen to
> get blocked on any individual slave, we'll just have to ask the owner
> to change that setting.

For reference, my Windows 7 slave (bolen-windows7) is currently
running with stock UAC settings, so I believe would prompt in such a
scenario.  The slave does run all builds with Windows error dialogs
disabled, but I doubt that covers UAC.

It's certainly no problem to disable the UAC prompting if it would
help or if that becomes a more useful setting for the buildbot
environment.  It probably fits better with my existing model of
disabling other sorts of error popups anyway, but just isn't something
we've run up against yet in the build process.

> Currently there are no Windows build slaves running as administrator.
> I used to have one but the machine died and I never replaced it. I
> also said a few months ago that I would get one setup again, but that
> hasn't happened yet. I can get a new machine up and running but
> probably not until next week as I'm at a conference.

Yes - my slaves (XP and Win7) are running under administrative
accounts, but not the literal Administrator account.  Though I'm
guessing this topic may be moot for the XP slave as it doesn't have
UAC.

Claudiu, if you've got any specific test code you'd like to have
executed on either of my slaves to see how it behaves, I'd be happy to
help.  Just contact me directly.

-- David


From ncoghlan at gmail.com  Thu May 15 04:35:16 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 15 May 2014 12:35:16 +1000
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAP1=2W4LGLt3bW=-oF548=Vug273opxfCxRqxF=rYzpvbBQDsw@mail.gmail.com>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <CAH0mxTTgyUVWpaud-pGbY0gu-8vdGcbqG4YMS_NSL1xQnFk=Ug@mail.gmail.com>
 <20140514145650.C0E17250D4F@webabinitio.net>
 <CAP1=2W6R1vKwthsftRKeoeYsP89jQoyJtUFxLwdeVeEs2=fJrg@mail.gmail.com>
 <53738C29.7050008@ubuntu.com>
 <CAP1=2W4LGLt3bW=-oF548=Vug273opxfCxRqxF=rYzpvbBQDsw@mail.gmail.com>
Message-ID: <CADiSq7dmUvqS6xFD9umLf4GWm1Exr+uXMWKiV_RfCiOvtOMxGg@mail.gmail.com>

On 15 May 2014 01:52, Brett Cannon <bcannon at gmail.com> wrote:
>
> I consider CPU and compiler separate things. As long as we have a buildbot
> covering the CPU or compiler somehow I say they are covered (and someone is
> willing to help make sure they continue to work). I'm not going to say that
> we need a BSD ARM buildbot and a Linux ARM machine; having a machine with
> ARM should be enough to shake out most arch-specific issues IMO. Same goes
> with compilers.

We also handle some of that compatibility testing over in distro land.
The feedback loop is a bit longer, but arch specific bugs are pretty
rare anyway. For example: yes, CPython runs just fine on IBM s390x
main frames, no, I'm not going to try to arrange for an s390x BuildBot
upstream because that would be painful, and abstracting away CPU
architecture issues is a key part of what the OS layer is for in the
first place :)

Anyway, +1 from me for expanding PEP 11 to also cover:

- some rules of thumb for what kind of OS specific patches are
acceptable to improve handling of unknown/unsupported OSes (e.g.
switch from explicit OS detection to equivalent feature detection,
yes, explicitly listing an unsupported OS in a conditional check, no)
- some guidelines for what's needed for a new OS to be added as
officially supported (with age and popularity likely being worth
taking into account)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From cs at zip.com.au  Thu May 15 06:01:05 2014
From: cs at zip.com.au (Cameron Simpson)
Date: Thu, 15 May 2014 14:01:05 +1000
Subject: [Python-Dev] Where is our official policy of what platforms we
 do support?
In-Reply-To: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
Message-ID: <20140515040105.GA82654@cskk.homeip.net>

On 14May2014 14:45, Brett Cannon <bcannon at gmail.com> wrote:
>On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou <solipsis at pitrou.net>
>wrote:
>> On Wed, 14 May 2014 14:20:26 +0000
>> Brett Cannon <bcannon at gmail.com> wrote:
>> > Over the past week or so there have been 2 patches to add support for
>> > various UNIX OSs. Now I thought we had stopped trying to add new esoteric
>> > OSs (e.g. I had never heard of MirOS until the patch for it came in),
>> but I
>> > can't find a PEP that spells out what it takes to get a platform
>> supported (
>> > http://legacy.python.org/dev/peps/pep-0011/ is about removing platforms,
>> > not keeping them or adding them unless you are re-adding one which
>> > apparently just takes a volunteer).
>>
>> OTOH you can fix a platform bug without officially supporting it. If
>> someone files an OpenBSD-specific patch, it may make sense to commit it
>> even without officially supporting OpenBSD. In practice it all depends
>> on how intrusive / reasonable the patch is, and whether it is working
>> around a platform-specific bug rather than a standards-compliant
>> limitation.
>>
>> (we could call those "stochastically supported platforms" :-))
>
>Very true, but these patches are all for e.g. configure to recognize a
>specific platform by listing them in some constant. Changing code to be
>more general I have no issue with since that's just good practice.

Recognition of a special platform isn't "full support", just addition of 
recognition making possible partial support for special cases. Unless that 
makes for some horrendous recognition decision tree somewhere I would have 
thought that's a pretty low bar to accept, and worth doing.

Leaving aside any bug actually fixed, it makes it much easier for someone else 
to fix a platform specific bug by making a test constant available for turning 
on whatever special mode/code is wanted.

More context on the example patch that triggered this query?

Just 2c,
Cameron Simpson <cs at zip.com.au>

From guido at python.org  Thu May 15 06:05:49 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 14 May 2014 21:05:49 -0700
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <20140515040105.GA82654@cskk.homeip.net>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
Message-ID: <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>

Main problem with rare platform support is not breaking it accidentally,
since without buildbots we won't know when it's broken. This is why we
don't make any promises.
On May 14, 2014 9:02 PM, "Cameron Simpson" <cs at zip.com.au> wrote:

> On 14May2014 14:45, Brett Cannon <bcannon at gmail.com> wrote:
>
>> On Wed May 14 2014 at 10:43:18 AM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>>
>>> On Wed, 14 May 2014 14:20:26 +0000
>>> Brett Cannon <bcannon at gmail.com> wrote:
>>> > Over the past week or so there have been 2 patches to add support for
>>> > various UNIX OSs. Now I thought we had stopped trying to add new
>>> esoteric
>>> > OSs (e.g. I had never heard of MirOS until the patch for it came in),
>>> but I
>>> > can't find a PEP that spells out what it takes to get a platform
>>> supported (
>>> > http://legacy.python.org/dev/peps/pep-0011/ is about removing
>>> platforms,
>>> > not keeping them or adding them unless you are re-adding one which
>>> > apparently just takes a volunteer).
>>>
>>> OTOH you can fix a platform bug without officially supporting it. If
>>> someone files an OpenBSD-specific patch, it may make sense to commit it
>>> even without officially supporting OpenBSD. In practice it all depends
>>> on how intrusive / reasonable the patch is, and whether it is working
>>> around a platform-specific bug rather than a standards-compliant
>>> limitation.
>>>
>>> (we could call those "stochastically supported platforms" :-))
>>>
>>
>> Very true, but these patches are all for e.g. configure to recognize a
>> specific platform by listing them in some constant. Changing code to be
>> more general I have no issue with since that's just good practice.
>>
>
> Recognition of a special platform isn't "full support", just addition of
> recognition making possible partial support for special cases. Unless that
> makes for some horrendous recognition decision tree somewhere I would have
> thought that's a pretty low bar to accept, and worth doing.
>
> Leaving aside any bug actually fixed, it makes it much easier for someone
> else to fix a platform specific bug by making a test constant available for
> turning on whatever special mode/code is wanted.
>
> More context on the example patch that triggered this query?
>
> Just 2c,
> Cameron Simpson <cs at zip.com.au>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140514/7ae1d6bf/attachment.html>

From ezio.melotti at gmail.com  Thu May 15 13:27:46 2014
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Thu, 15 May 2014 14:27:46 +0300
Subject: [Python-Dev] Summary of Python tracker Issues
In-Reply-To: <52F61713.8000800@email.de>
References: <20140207170745.ECC15568FA@psf.upfronthosting.co.za>
 <52F61713.8000800@email.de>
Message-ID: <CACBhJdEAXhYQyNLeN8k-bcFVkLdsGvo-dkt29wfXkf87cA9m0A@mail.gmail.com>

Hi,

On Sat, Feb 8, 2014 at 1:37 PM, francis <francismb at email.de> wrote:
> On 02/07/2014 06:07 PM, Python tracker wrote:
>>
>> Open issues with patches: 2045
>
>
> Has somebody done a graphic of that data againsttime?
>

You can find some charts here (it's still a work in progress though):
   http://bugs.python.org/issue?@template=stats

Best Regards,
Ezio Melotti

> Regards,
> francis
>

From skip at pobox.com  Thu May 15 15:20:03 2014
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 15 May 2014 08:20:03 -0500
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
Message-ID: <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>

On Wed, May 14, 2014 at 11:05 PM, Guido van Rossum <guido at python.org> wrote:
> Main problem with rare platform support is not breaking it accidentally,
> since without buildbots we won't know when it's broken. This is why we don't
> make any promises.

Should we (or do we) offer to run (unofficial) buildbots for
maintainers of minority platforms where possible? For example, I have
no idea if a buildbot for MirOS is even feasible, but if the guy who
submitted the patch is amenable and it is possible to run a buildbot
slave for that OS, it still might be useful to have a "one stop" place
for this. If failing, such buildbots wouldn't block a release, but
would still provide tools for people to track down the source of
breakage.

Skip

From solipsis at pitrou.net  Thu May 15 15:26:24 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 15 May 2014 15:26:24 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
Message-ID: <20140515152624.039883c7@fsol>

On Thu, 15 May 2014 08:20:03 -0500
Skip Montanaro <skip at pobox.com> wrote:

> On Wed, May 14, 2014 at 11:05 PM, Guido van Rossum <guido at python.org> wrote:
> > Main problem with rare platform support is not breaking it accidentally,
> > since without buildbots we won't know when it's broken. This is why we don't
> > make any promises.
> 
> Should we (or do we) offer to run (unofficial) buildbots for
> maintainers of minority platforms where possible? For example, I have
> no idea if a buildbot for MirOS is even feasible, but if the guy who
> submitted the patch is amenable and it is possible to run a buildbot
> slave for that OS, it still might be useful to have a "one stop" place
> for this. If failing, such buildbots wouldn't block a release, but
> would still provide tools for people to track down the source of
> breakage.

We already have such buildbots, they are in the "unstable" category.
You can browse through existing buildbots here:
https://www.python.org/dev/buildbot/

In the case of MirOS, though, I'm unsure core developers would
proactively fix failures on such a niche platform :-)

Regards

Antoine.

From skip at pobox.com  Thu May 15 16:20:20 2014
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 15 May 2014 09:20:20 -0500
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <20140515152624.039883c7@fsol>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
Message-ID: <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>

On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> We already have such buildbots, they are in the "unstable" category.
> You can browse through existing buildbots here:
> https://www.python.org/dev/buildbot/

I can't see how to distinguish "stable" from "unstable" (or to view
just the "unstable" category. What do those two categories have to do
with "supported" and "unsupported"?

Skip

From bcannon at gmail.com  Thu May 15 16:35:07 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Thu, 15 May 2014 14:35:07 +0000
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
 <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
Message-ID: <CAP1=2W5A6oTqT1w2NgcN9aEov04r2-7yMO660DQJUY_7QVoQ-g@mail.gmail.com>

On Thu May 15 2014 at 10:24:45 AM, Skip Montanaro <skip at pobox.com> wrote:

> On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > We already have such buildbots, they are in the "unstable" category.
> > You can browse through existing buildbots here:
> > https://www.python.org/dev/buildbot/
>
> I can't see how to distinguish "stable" from "unstable" (or to view
> just the "unstable" category.


Take
http://buildbot.python.org/all/waterfall?category=3.x.stable&category=3.x.unstableand
remove the category GET argument that you  don't want to see, e.g. to
only see unstable buildbots use
http://buildbot.python.org/all/waterfall?category=3.x.unstable


> What do those two categories have to do
> with "supported" and "unsupported"?
>

Antoine can give the definitive answer, but I view stable buildbots as
staying up and testing critical platforms. I.e. when I submit a patch I
make sure the stable buildbots are always green (unless it's a transient
failure) and I don't worry about the unstable ones (I view them as more
informative than necessary). Basically red stable buildbot should block a
release.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140515/22692616/attachment.html>

From skip at pobox.com  Thu May 15 16:40:33 2014
From: skip at pobox.com (Skip Montanaro)
Date: Thu, 15 May 2014 09:40:33 -0500
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CAP1=2W5A6oTqT1w2NgcN9aEov04r2-7yMO660DQJUY_7QVoQ-g@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
 <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
 <CAP1=2W5A6oTqT1w2NgcN9aEov04r2-7yMO660DQJUY_7QVoQ-g@mail.gmail.com>
Message-ID: <CANc-5UwORKDVa6_1zn2NL1Ha-XiXtCxho+h2zkGfCNfBF43H7A@mail.gmail.com>

On Thu, May 15, 2014 at 9:35 AM, Brett Cannon <bcannon at gmail.com> wrote:
> I view stable buildbots as staying up and testing critical platforms.

Would "supported" and "unsupported" (or "critical" and "optional"?)
make more sense? "Unstable" suggests "broken" to me, not "we don't
really care about these."

S

From solipsis at pitrou.net  Thu May 15 19:14:55 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 15 May 2014 19:14:55 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <CANc-5UwORKDVa6_1zn2NL1Ha-XiXtCxho+h2zkGfCNfBF43H7A@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
 <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
 <CAP1=2W5A6oTqT1w2NgcN9aEov04r2-7yMO660DQJUY_7QVoQ-g@mail.gmail.com>
 <CANc-5UwORKDVa6_1zn2NL1Ha-XiXtCxho+h2zkGfCNfBF43H7A@mail.gmail.com>
Message-ID: <20140515191455.4b96baab@fsol>

On Thu, 15 May 2014 09:40:33 -0500
Skip Montanaro <skip at pobox.com> wrote:
> On Thu, May 15, 2014 at 9:35 AM, Brett Cannon <bcannon at gmail.com> wrote:
> > I view stable buildbots as staying up and testing critical platforms.
> 
> Would "supported" and "unsupported" (or "critical" and "optional"?)
> make more sense? "Unstable" suggests "broken" to me, not "we don't
> really care about these."

I don't know who came up with these names in the first place.
However there's a slight nuance here: some platform may be supported,
but still some buildbot end up in the "unstable" category if it has
issues of its own (for example the machine has a flaky network
connection, etc.). And indeed there are Linux and Windows machines in
the "unstable" category.

Regards

Antoine.

From rdmurray at bitdance.com  Thu May 15 20:34:07 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 15 May 2014 14:34:07 -0400
Subject: [Python-Dev] Where is our official policy of what platforms we
	do support?
In-Reply-To: <20140515191455.4b96baab@fsol>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
 <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
 <CAP1=2W5A6oTqT1w2NgcN9aEov04r2-7yMO660DQJUY_7QVoQ-g@mail.gmail.com>
 <CANc-5UwORKDVa6_1zn2NL1Ha-XiXtCxho+h2zkGfCNfBF43H7A@mail.gmail.com>
 <20140515191455.4b96baab@fsol>
Message-ID: <20140515183407.EE230250D5E@webabinitio.net>

On Thu, 15 May 2014 19:14:55 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 15 May 2014 09:40:33 -0500
> Skip Montanaro <skip at pobox.com> wrote:
> > On Thu, May 15, 2014 at 9:35 AM, Brett Cannon <bcannon at gmail.com> wrote:
> > > I view stable buildbots as staying up and testing critical platforms.
> > 
> > Would "supported" and "unsupported" (or "critical" and "optional"?)
> > make more sense? "Unstable" suggests "broken" to me, not "we don't
> > really care about these."
> 
> I don't know who came up with these names in the first place.
> However there's a slight nuance here: some platform may be supported,
> but still some buildbot end up in the "unstable" category if it has
> issues of its own (for example the machine has a flaky network
> connection, etc.). And indeed there are Linux and Windows machines in
> the "unstable" category.

There's also nothing stopping us from putting a "niche platform"
buildbot into the stable group if it normally builds fine.  I suppose
it would be pretty much supported by default then, though, if it being
red was a release blocker.  But we could decide to ignore a red 'niche'
buildbot at release time; so, I think 'stable' vs 'unstable' is indeed
the most descriptive: unstable buildbots are the ones that turn red
"randomly"[*], or are always red because no one has fixed whatever
the problem is (which might be on the buildbot or in our code).

--David

[*] Yes, our stable platforms do that sometimes too, but those are test
instabilities, whereas unstable buildbots fail tests other than the known
unstable tests.

From status at bugs.python.org  Fri May 16 18:07:57 2014
From: status at bugs.python.org (Python tracker)
Date: Fri, 16 May 2014 18:07:57 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20140516160757.9498F560CB@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2014-05-09 - 2014-05-16)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4614 ( +3)
  closed 28674 (+50)
  total  33288 (+53)

Open issues with patches: 2121 


Issues opened (40)
==================

#13916: disallow the "surrogatepass" handler for non utf-* encodings
http://bugs.python.org/issue13916  reopened by haypo

#19385: dbm.dumb should be consistent when the database is closed
http://bugs.python.org/issue19385  reopened by serhiy.storchaka

#21425: Interactive interpreter doesn't flush stderr prompty
http://bugs.python.org/issue21425  reopened by pitrou

#21459: DragonFlyBSD support
http://bugs.python.org/issue21459  reopened by brett.cannon

#21465: sqlite3 Row can return duplicate keys when using adapters
http://bugs.python.org/issue21465  opened by BreamoreBoy

#21468: NNTPLib connections become corrupt after long periods of activ
http://bugs.python.org/issue21468  opened by James.Meneghello

#21471: subprocess line-buffering only works in universal newlines mod
http://bugs.python.org/issue21471  opened by pitrou

#21472: Fix wsgiref handling of absolute HTTP Request-URI
http://bugs.python.org/issue21472  opened by mouad

#21473: Idle: test startup scripts.
http://bugs.python.org/issue21473  opened by terry.reedy

#21474: Idle: updata fixwordbreaks() for unicode identifiers
http://bugs.python.org/issue21474  opened by terry.reedy

#21475: Support the Sitemap and Crawl-delay extensions in robotparser
http://bugs.python.org/issue21475  opened by rhettinger

#21476: Inconsitent behaviour between BytesParser.parse and Parser.par
http://bugs.python.org/issue21476  opened by ??ukasz.Kucharski

#21477: Idle: improve idle_test,htest
http://bugs.python.org/issue21477  opened by terry.reedy

#21478: mock calls don't propagate to parent (autospec)
http://bugs.python.org/issue21478  opened by and

#21479: Document TarFile.open() as a classmethod
http://bugs.python.org/issue21479  opened by berker.peksag

#21480: A build now requires...
http://bugs.python.org/issue21480  opened by skip.montanaro

#21481: Argpase Namespace object methods __eq__ and __ne__  raise Type
http://bugs.python.org/issue21481  opened by Joe.Borg

#21482: get_versions() in cygwinccomiler.py cannot return correct gcc 
http://bugs.python.org/issue21482  opened by 3togo

#21483: Skip os.utime() test on NFS?
http://bugs.python.org/issue21483  opened by skip.montanaro

#21484: More clarity needed about difference between "x += e" and "x =
http://bugs.python.org/issue21484  opened by Kluzniak

#21491: race condition in SocketServer.py ForkingMixIn collect_childre
http://bugs.python.org/issue21491  opened by idsvandermolen

#21493: Add test for ntpath.expanduser
http://bugs.python.org/issue21493  opened by Claudiu.Popa

#21495: Sane default for logging config
http://bugs.python.org/issue21495  opened by guettli

#21498: configparser accepts keys beginning with comment_chars when wr
http://bugs.python.org/issue21498  opened by The Compiler

#21500: Make use of the "load_tests" protocol in test_importlib packag
http://bugs.python.org/issue21500  opened by eric.snow

#21501: submitting mmap example for use in documentation
http://bugs.python.org/issue21501  opened by hudson

#21502: freeze.py not working properly with OS X framework builds
http://bugs.python.org/issue21502  opened by yjiangnan

#21503: Use test_both() consistently throughout test_importlib
http://bugs.python.org/issue21503  opened by eric.snow

#21504: can the subprocess module war using os.wait4 and so return usa
http://bugs.python.org/issue21504  opened by donald.petravick

#21505: cx_freeze multiprocessing bug
http://bugs.python.org/issue21505  opened by shivani

#21506: Windows MSI installer should mklink (symlink) python.exe to py
http://bugs.python.org/issue21506  opened by edmorley

#21507: set and frozenset constructor should use operator.length_hint 
http://bugs.python.org/issue21507  opened by lebedov

#21508: C API PyArg_ParseTuple doc is innacurate
http://bugs.python.org/issue21508  opened by Banger

#21509: json.load fails to read UTF-8 file with (BOM) Byte Order Marks
http://bugs.python.org/issue21509  opened by Kristian.Benoit

#21510: fma documentation should provide better example.
http://bugs.python.org/issue21510  opened by jayanthkoushik

#21511: Thinko in Lib/quopri.py
http://bugs.python.org/issue21511  opened by pfalcon

#21513: speed up some ipaddress properties
http://bugs.python.org/issue21513  opened by pitrou

#21514: update json module docs in light of RFC 7159 & ECMA-404
http://bugs.python.org/issue21514  opened by cvrebert

#21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile?
http://bugs.python.org/issue21515  opened by haypo

#21516: pathlib.Path(...).is_dir() crashes on some directories (Window
http://bugs.python.org/issue21516  opened by theller



Most recent 15 issues with no replies (15)
==========================================

#21514: update json module docs in light of RFC 7159 & ECMA-404
http://bugs.python.org/issue21514

#21513: speed up some ipaddress properties
http://bugs.python.org/issue21513

#21511: Thinko in Lib/quopri.py
http://bugs.python.org/issue21511

#21505: cx_freeze multiprocessing bug
http://bugs.python.org/issue21505

#21502: freeze.py not working properly with OS X framework builds
http://bugs.python.org/issue21502

#21500: Make use of the "load_tests" protocol in test_importlib packag
http://bugs.python.org/issue21500

#21498: configparser accepts keys beginning with comment_chars when wr
http://bugs.python.org/issue21498

#21493: Add test for ntpath.expanduser
http://bugs.python.org/issue21493

#21491: race condition in SocketServer.py ForkingMixIn collect_childre
http://bugs.python.org/issue21491

#21479: Document TarFile.open() as a classmethod
http://bugs.python.org/issue21479

#21477: Idle: improve idle_test,htest
http://bugs.python.org/issue21477

#21476: Inconsitent behaviour between BytesParser.parse and Parser.par
http://bugs.python.org/issue21476

#21473: Idle: test startup scripts.
http://bugs.python.org/issue21473

#21472: Fix wsgiref handling of absolute HTTP Request-URI
http://bugs.python.org/issue21472

#21468: NNTPLib connections become corrupt after long periods of activ
http://bugs.python.org/issue21468



Most recent 15 issues waiting for review (15)
=============================================

#21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile?
http://bugs.python.org/issue21515

#21513: speed up some ipaddress properties
http://bugs.python.org/issue21513

#21507: set and frozenset constructor should use operator.length_hint 
http://bugs.python.org/issue21507

#21503: Use test_both() consistently throughout test_importlib
http://bugs.python.org/issue21503

#21493: Add test for ntpath.expanduser
http://bugs.python.org/issue21493

#21479: Document TarFile.open() as a classmethod
http://bugs.python.org/issue21479

#21472: Fix wsgiref handling of absolute HTTP Request-URI
http://bugs.python.org/issue21472

#21463: RuntimeError when URLopener.ftpcache is full
http://bugs.python.org/issue21463

#21462: PEP 466: upgrade OpenSSL in the Python 2.7 Windows builds
http://bugs.python.org/issue21462

#21461: Recognize -pthread
http://bugs.python.org/issue21461

#21459: DragonFlyBSD support
http://bugs.python.org/issue21459

#21457: NetBSD curses support improvements
http://bugs.python.org/issue21457

#21456: skip 2 tests in test_urllib2net.py if _ssl module not present
http://bugs.python.org/issue21456

#21455: add default backlog to socket.listen()
http://bugs.python.org/issue21455

#21449: Replace _PyUnicode_CompareWithId with _PyUnicode_CompareWithId
http://bugs.python.org/issue21449



Top 10 most discussed issues (10)
=================================

#21425: Interactive interpreter doesn't flush stderr prompty
http://bugs.python.org/issue21425  18 msgs

#13916: disallow the "surrogatepass" handler for non utf-* encodings
http://bugs.python.org/issue13916  14 msgs

#7776: http.client.HTTPConnection tunneling is broken
http://bugs.python.org/issue7776   9 msgs

#21506: Windows MSI installer should mklink (symlink) python.exe to py
http://bugs.python.org/issue21506   9 msgs

#21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile?
http://bugs.python.org/issue21515   8 msgs

#21027: difflib new cli interface
http://bugs.python.org/issue21027   6 msgs

#21304: PEP 466: Backport hashlib.pbkdf2_hmac to Python 2.7
http://bugs.python.org/issue21304   6 msgs

#21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists
http://bugs.python.org/issue21402   6 msgs

#21455: add default backlog to socket.listen()
http://bugs.python.org/issue21455   6 msgs

#21507: set and frozenset constructor should use operator.length_hint 
http://bugs.python.org/issue21507   6 msgs



Issues closed (52)
==================

#9266: ctypes "ValueError: NULL pointer access" on Win7 x64
http://bugs.python.org/issue9266  closed by berker.peksag

#10752: build_ssl.py is relying on unreliable behaviour of os.popen
http://bugs.python.org/issue10752  closed by tim.golden

#12985: Check signed arithmetic overflow in ./configure
http://bugs.python.org/issue12985  closed by skrah

#13869: CFLAGS="-UNDEBUG" build failure
http://bugs.python.org/issue13869  closed by skrah

#14142: getlocale(LC_ALL) behavior
http://bugs.python.org/issue14142  closed by skrah

#14198: Backport parts of the new memoryview documentation
http://bugs.python.org/issue14198  closed by skrah

#16531: Allow IPNetwork to take a tuple
http://bugs.python.org/issue16531  closed by pitrou

#18062: m68k FPU precision issue
http://bugs.python.org/issue18062  closed by skrah

#18104: Idle: make human-mediated GUI tests usable
http://bugs.python.org/issue18104  closed by terry.reedy

#19655: Replace the ASDL parser carried with CPython
http://bugs.python.org/issue19655  closed by eli.bendersky

#19721: Move all test_importlib utility code into test_importlib.util
http://bugs.python.org/issue19721  closed by brett.cannon

#19775: Provide samefile() on Path objects
http://bugs.python.org/issue19775  closed by pitrou

#20776: Add tests for importlib.machinery.PathFinder
http://bugs.python.org/issue20776  closed by brett.cannon

#20826: Faster implementation to collapse consecutive ip-networks
http://bugs.python.org/issue20826  closed by pitrou

#20998: fullmatch isn't matching correctly under re.IGNORECASE
http://bugs.python.org/issue20998  closed by serhiy.storchaka

#21037: add an AddressSanitizer build option
http://bugs.python.org/issue21037  closed by neologix

#21075: fileinput should use stdin.buffer for "rb" mode
http://bugs.python.org/issue21075  closed by serhiy.storchaka

#21088: curses addch() argument position reverses in Python3.4.0
http://bugs.python.org/issue21088  closed by larry

#21156: Consider moving importlib.abc.InspectLoader.source_to_code() t
http://bugs.python.org/issue21156  closed by brett.cannon

#21226: PyImport_ExecCodeModuleObject not setting module attributes
http://bugs.python.org/issue21226  closed by eric.snow

#21237: Update Python 2/3 porting HOWTO's suggestion for dealing with 
http://bugs.python.org/issue21237  closed by brett.cannon

#21306: PEP 466: backport hmac.compare_digest
http://bugs.python.org/issue21306  closed by python-dev

#21347: Don't use a list argument together with shell=True in subproce
http://bugs.python.org/issue21347  closed by r.david.murray

#21363: io.TextIOWrapper always closes wrapped files
http://bugs.python.org/issue21363  closed by r.david.murray

#21364: Documentation Recommends Broken Pattern
http://bugs.python.org/issue21364  closed by pitrou

#21370: segfault from simple traceback.format_exc call
http://bugs.python.org/issue21370  closed by skrah

#21383: "make touch" fails when the build directory is not the source 
http://bugs.python.org/issue21383  closed by ned.deily

#21384: Windows: Make handle non inheritable by default
http://bugs.python.org/issue21384  closed by haypo

#21398: LC_CTYPE=C:  pydoc leaves terminal in an unusable state
http://bugs.python.org/issue21398  closed by haypo

#21418: Segv during call to super_init in application embedding Python
http://bugs.python.org/issue21418  closed by haypo

#21419: Use calloc() instead of malloc() for int << int (lshift)
http://bugs.python.org/issue21419  closed by haypo

#21420: Optimize 2 ** n: implement it as 1 << n
http://bugs.python.org/issue21420  closed by haypo

#21422: int << 0: return the number unmodified
http://bugs.python.org/issue21422  closed by haypo

#21424: Simplify and speed-up heaqp.nlargest()
http://bugs.python.org/issue21424  closed by rhettinger

#21452: make_buildinfo.exe with VS2013 fails due ill-formed IntDir pat
http://bugs.python.org/issue21452  closed by tim.golden

#21464: fnmatch module uses regular expression with undefined result t
http://bugs.python.org/issue21464  closed by pfalcon

#21466: General Index link to del statement is wrong
http://bugs.python.org/issue21466  closed by python-dev

#21467: IDLE icon not included in Windows installer
http://bugs.python.org/issue21467  closed by steve.dower

#21469: False positive hazards in robotparser
http://bugs.python.org/issue21469  closed by rhettinger

#21470: Better seeding for the random module
http://bugs.python.org/issue21470  closed by rhettinger

#21485: remove unnecesary .flush() calls in the asyncio subprocess cod
http://bugs.python.org/issue21485  closed by haypo

#21486: optimize v4 & v6 netmask parsing
http://bugs.python.org/issue21486  closed by pitrou

#21487: Assorted ipaddress performance improvements
http://bugs.python.org/issue21487  closed by pitrou

#21488: codecs.encode/decode documentation inconsistency
http://bugs.python.org/issue21488  closed by haypo

#21489: Switching from -OO to -O still uses cached bytecode
http://bugs.python.org/issue21489  closed by brett.cannon

#21490: Add Py_ABS and Py_STRINGIFY macros
http://bugs.python.org/issue21490  closed by haypo

#21492: email.header.decode_header sometimes returns bytes, sometimes 
http://bugs.python.org/issue21492  closed by r.david.murray

#21494: getopt error doesnot display correct error
http://bugs.python.org/issue21494  closed by eric.smith

#21496: pyvenv activate_this.py
http://bugs.python.org/issue21496  closed by vinay.sajip

#21497: faulthandler should handle sys.stderr being None gracefully
http://bugs.python.org/issue21497  closed by haypo

#21499: test_importlib incorrectly relies on <module>.__builtins__
http://bugs.python.org/issue21499  closed by eric.snow

#21512: time module becomes None after raise SystemExit
http://bugs.python.org/issue21512  closed by ezio.melotti

From bcannon at gmail.com  Fri May 16 19:51:00 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 16 May 2014 17:51:00 +0000
Subject: [Python-Dev] Update to PEP 11 to clarify garnering platform support
Message-ID: <CAP1=2W44scVkR1aqwxsvAwzQk6NS2ZHa=h4pJM1=UkBUs024Kw@mail.gmail.com>

Here is some proposed wording. Since it is more of a clarification of what
it takes to garner support -- which is just a new section -- rather than a
complete rewrite I'm including just the diff to make it easier to read the
changes.


*diff -r 49d18bb47ebc pep-0011.txt*

*--- a/pep-0011.txt Wed May 14 11:18:22 2014 -0400*

*+++ b/pep-0011.txt Fri May 16 13:48:30 2014 -0400*

@@ -2,22 +2,21 @@

 Title: Removing support for little used platforms

 Version: $Revision$

 Last-Modified: $Date$

-Author: martin at v.loewis.de (Martin von L?wis)

+Author: Martin von L?wis <martin at v.loewis.de>,

+        Brett Cannon <brett at python.org>

 Status: Active

 Type: Process

 Content-Type: text/x-rst

 Created: 07-Jul-2002

 Post-History: 18-Aug-2007

+              16-May-2014





 Abstract

 --------



-This PEP documents operating systems (platforms) which are not

-supported in Python anymore.  For some of these systems,

-supporting code might be still part of Python, but will be removed

-in a future release - unless somebody steps forward as a volunteer

-to maintain this code.

+This PEP documents how an operating system (platform) garners

+support in Python as well as documenting past support.





 Rationale

@@ -37,16 +36,53 @@

 change to the Python source code will work on all supported

 platforms.



-To reduce this risk, this PEP proposes a procedure to remove code

-for platforms with no Python users.

+To reduce this risk, this PEP specifies what is required for a

+platform to be considered supported by Python as well as providing a

+procedure to remove code for platforms with little or no Python

+users.



+Supporting platforms

+--------------------

+

+Gaining official platform support requires two things. First, a core

+developer needs to volunteer to maintain platform-specific code. This

+core developer can either already be a member of the Python

+development team or be given contributor rights on the basis of

+maintaining platform support (it is at the discretion of the Python

+development team to decide if a person is ready to have such rights

+even if it is just for supporting a specific platform).

+

+Second, a stable buildbot must be provided [2]_. This guarantees that

+platform support will not be accidentally broken by a Python core

+developer who does not have personal access to the platform. For a

+buildbot to be considered stable it requires that the machine be

+reliably up and functioning (but it is up to the Python core

+developers to decide whether to promote a buildbot to being

+considered stable).

+

+This policy does not disqualify supporting other platforms

+indirectly. Patches which are not platform-specific but still done to

+add platform support will be considered for inclusion. For example,

+if platform-independent changes were necessary in the configure

+script which was motivated to support a specific platform that would

+be accepted. Patches which add platform-specific code such as the

+name of a specific platform to the configure script will generally

+not be accepted without the platform having official support.

+

+CPU architecture and compiler support are viewed in a similar manner

+as platforms. For example, to consider the ARM architecture supported

+a buildbot running on ARM would be required along with support from

+the Python development team. In general it is not required to have

+a CPU architecture run under every possible platform in order to be

+considered supported.



 Unsupporting platforms

 ----------------------



-If a certain platform that currently has special code in it is

-deemed to be without Python users, a note must be posted in this

-PEP that this platform is no longer actively supported.  This

+If a certain platform that currently has special code in Python is

+deemed to be without Python users or lacks proper support from the

+Python development team and/or a buildbot, a note must be posted in

+this PEP that this platform is no longer actively supported.  This

 note must include:



 - the name of the system

@@ -69,8 +105,8 @@

 forward and offer maintenance.





-Resupporting platforms

-----------------------

+Re-supporting platforms

+-----------------------



 If a user of a platform wants to see this platform supported

 again, he may volunteer to maintain the platform support.  Such an

@@ -101,7 +137,7 @@

 release is made. Developers of extension modules will generally need

 to use the same Visual Studio release; they are concerned both with

 the availability of the versions they need to use, and with keeping

-the zoo of versions small. The Python source tree will keep

+the zoo of versions small. The Python source tree will keep

 unmaintained build files for older Visual Studio releases, for which

 patches will be accepted. Such build files will be removed from the

 source tree 3 years after the extended support for the compiler has

@@ -223,6 +259,7 @@

 ----------



 .. [1] http://support.microsoft.com/lifecycle/

+.. [2] http://buildbot.python.org/3.x.stable/



 Copyright

 ---------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140516/60a11a0d/attachment.html>

From ncoghlan at gmail.com  Sat May 17 07:14:00 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 17 May 2014 15:14:00 +1000
Subject: [Python-Dev] Returning None from methods that mutate object state
Message-ID: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>

During a conversation today, I realised that the convention of
returning None from methods that change an object's state isn't
captured the Programming Recommendations section of PEP 8.
Specifically, I'm referring to this behaviour:

>>> [].sort() is None
True
>>> "ABC".lower() is None
False

That's a deliberate design choice, and one that has been explained a
few times on the list when folks ask why "[].sort().reverse()" doesn't
work when "'ABC'.lower().replace('-', '_')" does.

Would it be worth adding such a note? Or is it out of scope?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Sat May 17 10:26:12 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 17 May 2014 04:26:12 -0400
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
Message-ID: <ll76fd$34u$1@ger.gmane.org>

On 5/17/2014 1:14 AM, Nick Coghlan wrote:
> During a conversation today, I realised that the convention of
> returning None from methods that change an object's state isn't
> captured the Programming Recommendations section of PEP 8.
> Specifically, I'm referring to this behaviour:
>
>>>> [].sort() is None
> True
>>>> "ABC".lower() is None
> False

When list.pop was added, the convention was changed to
"do not return the 'self' parameter"

 >>> [1].pop() is None
False

Not returning 'self' allows some mutation functions to return something 
other than 'self'.

I phrase the rule the way I did because a recursive collections can 
incidentally return itself.

 >>> L = []
 >>> L.append(L)
 >>> L.pop() is L
True

it.__next__ is another mutator that returns neither self or None.

Actually, if one regards file read and write as mutation, then returning 
None never was the rule.

It seems to me that the actual Python rule is "Don't return 'self'. If 
there is nothing useful to return (other than self), return None." I 
believe this is true whether or not self is mutated. (Of course, there 
might be an exception I have overlooked.)

> That's a deliberate design choice, and one that has been explained a
> few times on the list when folks ask why "[].sort().reverse()" doesn't
> work when "'ABC'.lower().replace('-', '_')" does.

Do people ask why sys.stdout.write(line).write('\n') does not work?

> Would it be worth adding such a note? Or is it out of scope?

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Sat May 17 10:44:28 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 17 May 2014 18:44:28 +1000
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <ll76fd$34u$1@ger.gmane.org>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <ll76fd$34u$1@ger.gmane.org>
Message-ID: <CADiSq7d7zN8Xe2j3=hp-nykHeWxihc4LmVQsJNJ6znk25u-WJQ@mail.gmail.com>

On 17 May 2014 18:26, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/17/2014 1:14 AM, Nick Coghlan wrote:
>>
>> During a conversation today, I realised that the convention of
>> returning None from methods that change an object's state isn't
>> captured the Programming Recommendations section of PEP 8.
>> Specifically, I'm referring to this behaviour:
>>
>>>>> [].sort() is None
>>
>> True
>>>>>
>>>>> "ABC".lower() is None
>>
>> False
>
>
> When list.pop was added, the convention was changed to
> "do not return the 'self' parameter"
>
>>>> [1].pop() is None
> False
>
> Not returning 'self' allows some mutation functions to return something
> other than 'self'.

That's a good point - I wasn't thinking methods like pop() when
phrasing it the way I did.

> I phrase the rule the way I did because a recursive collections can
> incidentally return itself.
>
>>>> L = []
>>>> L.append(L)
>>>> L.pop() is L
> True
>
> it.__next__ is another mutator that returns neither self or None.
>
> Actually, if one regards file read and write as mutation, then returning
> None never was the rule.

dict.pop() is a similar example. It was bad phrasing on my part,
because I was thinking of a particular subset of mutating methods (the
ones which don't have a more meaningful result that they need to
return)

> It seems to me that the actual Python rule is "Don't return 'self'. If there
> is nothing useful to return (other than self), return None." I believe this
> is true whether or not self is mutated. (Of course, there might be an
> exception I have overlooked.)

It's actually even more subtle than that - it's "Don't return self,
unless required to do so by a protocol" (specifically, __iter__ and
the _i*__ methods sometimes involve returning "self" for iterators and
types that do in-place operations respectively)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sat May 17 11:55:12 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 17 May 2014 11:55:12 +0200
Subject: [Python-Dev] Returning None from methods that mutate object
	state
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
Message-ID: <20140517115512.1e1497f1@fsol>

On Sat, 17 May 2014 15:14:00 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> During a conversation today, I realised that the convention of
> returning None from methods that change an object's state isn't
> captured the Programming Recommendations section of PEP 8.

This is more an API design guideline than a programming recommandation,
IMO. If we start putting those in PEP 8, it will end up bloating the
PEP.
(perhaps we need another PEP?)

Regards

Antoine.



From steve at pearwood.info  Sat May 17 12:44:35 2014
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 17 May 2014 20:44:35 +1000
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <ll76fd$34u$1@ger.gmane.org>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <ll76fd$34u$1@ger.gmane.org>
Message-ID: <20140517104435.GS4273@ando>

On Sat, May 17, 2014 at 04:26:12AM -0400, Terry Reedy wrote:
> On 5/17/2014 1:14 AM, Nick Coghlan wrote:
> >During a conversation today, I realised that the convention of
> >returning None from methods that change an object's state isn't
> >captured the Programming Recommendations section of PEP 8.
> >Specifically, I'm referring to this behaviour:
> >
> >>>>[].sort() is None
> >True
> >>>>"ABC".lower() is None
> >False
> 
> When list.pop was added, the convention was changed to
> "do not return the 'self' parameter"

I don't think "return self" was ever the convention. Here's Python 1.5:

[steve at ando ~]$ python1.5
Python 1.5.2 (#1, Aug 27 2012, 09:09:18)  [GCC 4.1.2 20080704 (Red Hat 
4.1.2-52)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> [].sort() is None
1
>>> [].reverse() is None
1
>>> [1, 2, 3].remove(2) is None
1
>>> {}.update({}) is None
1

(Wow. Returning 1 instead of True -- it's ages since I've seen that.)

I think the rule "return None, not self" should be best understood as 
applying only to mutator methods which *only* operate for their 
side-effects (like sorting, reversing), not those which also have an 
obvious return value:

>>> [2, 4, 6].pop()  # Still Python 1.5.
6

I don't recall any other examples from 1.5 that did this.


> it.__next__ is another mutator that returns neither self or None.
> 
> Actually, if one regards file read and write as mutation, then returning 
> None never was the rule.

I don't think we should regard reading as a mutation :-) 

Writing to a file on disk may be a side-effect, but I don't think it 
counts as a mutation of the file object. But even if we consider it as 
one, prior to 3.x writing returned None. This is from 2.7:

py> open("/dev/null", "w").write("goodbye cruel world") is None
True


> It seems to me that the actual Python rule is "Don't return 'self'. If 
> there is nothing useful to return (other than self), return None." I 
> believe this is true whether or not self is mutated. (Of course, there 
> might be an exception I have overlooked.)

The obvious exception is the iterator protocol: __iter__ of an iterator 
must return self. But that's an exceptional case.

I don't think it is fair to say that Python imposes a *rule* "don't 
return self" -- it's more of a convention for built-ins only, not a rule 
for all Python classes. Ruby prefers methods that return self and are 
therefore chainable, while Python built-ins do not.

I think it would be worth having PEP 8 make it clear that:

- built-ins, and types which share the same interface as built-ins, 
  should have mutators which return None rather than self;

- but third-party types (including classes in the standard library, if 
  necessary) may return self;

- whichever choice you make, be consistant within a single API. E.g. 
  don't have x.mutator1() return None, while x.mutator2() returns self.


-- 
Steven

From ncoghlan at gmail.com  Sat May 17 16:18:20 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 18 May 2014 00:18:20 +1000
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <20140517115512.1e1497f1@fsol>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <20140517115512.1e1497f1@fsol>
Message-ID: <CADiSq7f-1uXAH+=pJFNPwuHXpR+2Ws4HeTdkgDKWRwNP7YDakg@mail.gmail.com>

On 17 May 2014 19:56, "Antoine Pitrou" <solipsis at pitrou.net> wrote:
>
> On Sat, 17 May 2014 15:14:00 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
> > During a conversation today, I realised that the convention of
> > returning None from methods that change an object's state isn't
> > captured the Programming Recommendations section of PEP 8.
>
> This is more an API design guideline than a programming recommandation,
> IMO. If we start putting those in PEP 8, it will end up bloating the
> PEP.
> (perhaps we need another PEP?)

A new entry in https://docs.python.org/3/faq/design.html may be a
reasonable option.

It's not like it comes up *that* often, and as Nathaniel pointed out, the
main advantage of writing it down somewhere is to be able to link to it in
the future.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/08efce42/attachment.html>

From njs at pobox.com  Sat May 17 15:39:28 2014
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 17 May 2014 14:39:28 +0100
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
Message-ID: <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>

On Sat, May 17, 2014 at 6:14 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> During a conversation today, I realised that the convention of
> returning None from methods that change an object's state isn't
> captured the Programming Recommendations section of PEP 8.
> Specifically, I'm referring to this behaviour:
>
>>>> [].sort() is None
> True
>>>> "ABC".lower() is None
> False
>
> That's a deliberate design choice, and one that has been explained a
> few times on the list when folks ask why "[].sort().reverse()" doesn't
> work when "'ABC'.lower().replace('-', '_')" does.
>
> Would it be worth adding such a note? Or is it out of scope?

Numpy also has a menagerie of in-place and out-of-place methods that
follow this convention, and we also have to go round on the mailing
list from time to time explaining to well-meaning beginners why for
the in-place methods returning None is the right ("Pythonic") design.
Having a canonical explanation of this to link to would be useful.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

From tismer at stackless.com  Sat May 17 16:45:54 2014
From: tismer at stackless.com (Christian Tismer)
Date: Sat, 17 May 2014 16:45:54 +0200
Subject: [Python-Dev] devguide: Add myself to developer log and as
 a	Windows expert.
In-Reply-To: <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org>, <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <53777622.3010209@stackless.com>

On 11.05.14 15:57, Steve Dower wrote:
> Thanks.
>
> For those who missed the earlier discussions, Martin v. L?wis has
> handed over responsibility for the Windows installers. It sounds like
> Brett Cannon and I are both in a position to build 2.7 right now, and
> I hope to simplify the setup required for 3.5 so that anyone can
> produce and test the installers. (Martin is going to look after the
> 3.4.x builds.)
>
> Obviously I'm also here to help out with Windows in general. I haven't
> quite managed to get 50% time from my employer (maybe 10%), but my
> management at least is very supportive of my participation and keen to
> keep Python running well.
>

Very nice, great to read this.
Welcome from me as well!

cheers - Chris

-- 
Christian Tismer             :^)   tismer at stackless.com
Software Consulting          :     http://www.stackless.com/
Karl-Liebknecht-Str. 121     :     http://www.pydica.net/
14482 Potsdam                :     GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140517/2af60bd7/attachment.html>

From guido at python.org  Sat May 17 17:50:47 2014
From: guido at python.org (Guido van Rossum)
Date: Sat, 17 May 2014 08:50:47 -0700
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
Message-ID: <CAP7+vJLoYcQqo3KHWwdZXpoKFb8H4bOX_wbSgi6O7sVDqDB=iQ@mail.gmail.com>

Please preserve the tradition. Adding it to PEP 8 sounds good!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140517/bb7ab01f/attachment.html>

From bcannon at gmail.com  Sun May 18 03:26:19 2014
From: bcannon at gmail.com (Dr. Brett Cannon)
Date: Sun, 18 May 2014 01:26:19 +0000
Subject: [Python-Dev] devguide: Add myself to developer log and as a
 Windows expert.
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>
Message-ID: <CAP1=2W5O_Jp1v9TMHLi_O3RDtrRdzC5fwL=rr_muv9CME_0Now@mail.gmail.com>

I don't think you meant me for helping to build Windows binaries. :)

On Sunday, May 11, 2014 9:58:16 AM, Steve Dower <Steve.Dower at microsoft.com>
wrote:

Thanks.

For those who missed the earlier discussions, Martin v. L?wis has handed
over responsibility for the Windows installers. It sounds like Brett Cannon
and I are both in a position to build 2.7 right now, and I hope to simplify
the setup required for 3.5 so that anyone can produce and test the
installers. (Martin is going to look after the 3.4.x builds.)

Obviously I'm also here to help out with Windows in general. I haven't
quite managed to get 50% time from my employer (maybe 10%), but my
management at least is very supportive of my participation and keen to keep
Python running well.

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou <solipsis at pitrou.net>
Sent: ?5/?11/?2014 4:18
To: python-dev@ <python-dev at python.org>python.org <python-dev at python.org>
Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a
Windows expert.

On Sun, 11 May 2014 06:04:56 +0200 (CEST)
steve.dower <python-
<python-checkins at python.org>checkins<python-checkins at python.org>
@ <python-checkins at python.org>python.org <python-checkins at python.org>>
wrote:
> http:// <http://hg.python.org/devguide/rev/8d5d1f2c7378>hg.python.org<http://hg.python.org/devguide/rev/8d5d1f2c7378>
/ <http://hg.python.org/devguide/rev/8d5d1f2c7378>devguide<http://hg.python.org/devguide/rev/8d5d1f2c7378>
/rev/8d5d1f2c7378 <http://hg.python.org/devguide/rev/8d5d1f2c7378>
> changeset:   698:8d5d1f2c7378
> user:        Steve Dower <steve.dower <steve.dower at microsoft.com>@<steve.dower at microsoft.com>
microsoft.com <steve.dower at microsoft.com>>
> date:        Sat May 10 21:01:39 2014 -0700
> summary:
>   Add myself to developer log and as a Windows expert.

Welcome onboard, Steve ! Nice to have an additional Windows expert :-)

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev@ <Python-Dev at python.org>python.org <Python-Dev at python.org>
https <https://mail.python.org/mailman/listinfo/python-dev>://<https://mail.python.org/mailman/listinfo/python-dev>
mail.python.org <https://mail.python.org/mailman/listinfo/python-dev>
/mailman/ <https://mail.python.org/mailman/listinfo/python-dev>listinfo<https://mail.python.org/mailman/listinfo/python-dev>
/python-dev <https://mail.python.org/mailman/listinfo/python-dev>

Unsubscribe: https<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>
://<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>
mail.python.org<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>
/mailman/options/python-dev/<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>
steve.dower<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>
%40microsoft.com<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>

_______________________________________________
Python-Dev mailing list
Python-Dev@ <Python-Dev at python.org>python.org <Python-Dev at python.org>
https <https://mail.python.org/mailman/listinfo/python-dev>://<https://mail.python.org/mailman/listinfo/python-dev>
mail.python.org <https://mail.python.org/mailman/listinfo/python-dev>
/mailman/ <https://mail.python.org/mailman/listinfo/python-dev>listinfo<https://mail.python.org/mailman/listinfo/python-dev>
/python-dev <https://mail.python.org/mailman/listinfo/python-dev>
Unsubscribe: https<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
:// <https://mail.python.org/mailman/options/python-dev/brett%40python.org>
mail.python.org<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
/mailman/options/python-dev/brett%40python.org<https://mail.python.org/mailman/options/python-dev/brett%40python.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/31ff2eaa/attachment.html>

From Steve.Dower at microsoft.com  Sun May 18 04:04:02 2014
From: Steve.Dower at microsoft.com (Steve Dower)
Date: Sun, 18 May 2014 02:04:02 +0000
Subject: [Python-Dev] devguide: Add myself to developer log and as a
 Windows expert.
In-Reply-To: <CAP1=2W5O_Jp1v9TMHLi_O3RDtrRdzC5fwL=rr_muv9CME_0Now@mail.gmail.com>
References: <3gRBWD3q7bz7Ljd@mail.python.org> <20140511131748.611ce431@fsol>
 <f6144319207e4269b044238e8e05abca@BLUPR03MB389.namprd03.prod.outlook.com>,
 <CAP1=2W5O_Jp1v9TMHLi_O3RDtrRdzC5fwL=rr_muv9CME_0Now@mail.gmail.com>
Message-ID: <e443d52fe1294b848dbc82aba0942aca@BLUPR03MB389.namprd03.prod.outlook.com>

No, apologies for that. As Nick pointed out, I meant the *other* B.C. :)

Top-posted from my Windows Phone
________________________________
From: Dr. Brett Cannon<mailto:bcannon at gmail.com>
Sent: ?5/?17/?2014 18:26
To: Steve Dower<mailto:Steve.Dower at microsoft.com>; solipsis at pitrou.net<mailto:solipsis at pitrou.net>; python-dev at python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert.

I don't think you meant me for helping to build Windows binaries. :)

On Sunday, May 11, 2014 9:58:16 AM, Steve Dower <Steve.Dower at microsoft.com<mailto:Steve.Dower at microsoft.com>> wrote:

Thanks.

For those who missed the earlier discussions, Martin v. L?wis has handed over responsibility for the Windows installers. It sounds like Brett Cannon and I are both in a position to build 2.7 right now, and I hope to simplify the setup required for 3.5 so that anyone can produce and test the installers. (Martin is going to look after the 3.4.x builds.)

Obviously I'm also here to help out with Windows in general. I haven't quite managed to get 50% time from my employer (maybe 10%), but my management at least is very supportive of my participation and keen to keep Python running well.

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou<mailto:solipsis at pitrou.net>
Sent: ?5/?11/?2014 4:18
To: python-dev@<mailto:python-dev at python.org>python.org<mailto:python-dev at python.org>
Subject: Re: [Python-Dev] devguide: Add myself to developer log and as a Windows expert.

On Sun, 11 May 2014 06:04:56 +0200 (CEST)
steve.dower <python-<mailto:python-checkins at python.org>checkins<mailto:python-checkins at python.org>@<mailto:python-checkins at python.org>python.org<mailto:python-checkins at python.org>> wrote:
> http://<http://hg.python.org/devguide/rev/8d5d1f2c7378>hg.python.org<http://hg.python.org/devguide/rev/8d5d1f2c7378>/<http://hg.python.org/devguide/rev/8d5d1f2c7378>devguide<http://hg.python.org/devguide/rev/8d5d1f2c7378>/rev/8d5d1f2c7378<http://hg.python.org/devguide/rev/8d5d1f2c7378>
> changeset:   698:8d5d1f2c7378
> user:        Steve Dower <steve.dower<mailto:steve.dower at microsoft.com>@<mailto:steve.dower at microsoft.com>microsoft.com<mailto:steve.dower at microsoft.com>>
> date:        Sat May 10 21:01:39 2014 -0700
> summary:
>   Add myself to developer log and as a Windows expert.

Welcome onboard, Steve ! Nice to have an additional Windows expert :-)

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Python-Dev@<mailto:Python-Dev at python.org>python.org<mailto:Python-Dev at python.org>
https<https://mail.python.org/mailman/listinfo/python-dev>://<https://mail.python.org/mailman/listinfo/python-dev>mail.python.org<https://mail.python.org/mailman/listinfo/python-dev>/mailman/<https://mail.python.org/mailman/listinfo/python-dev>listinfo<https://mail.python.org/mailman/listinfo/python-dev>/python-dev<https://mail.python.org/mailman/listinfo/python-dev>

Unsubscribe: https<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>://<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>mail.python.org<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>/mailman/options/python-dev/<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>steve.dower<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>%40microsoft.com<https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com>

_______________________________________________
Python-Dev mailing list
Python-Dev@<mailto:Python-Dev at python.org>python.org<mailto:Python-Dev at python.org>
https<https://mail.python.org/mailman/listinfo/python-dev>://<https://mail.python.org/mailman/listinfo/python-dev>mail.python.org<https://mail.python.org/mailman/listinfo/python-dev>/mailman/<https://mail.python.org/mailman/listinfo/python-dev>listinfo<https://mail.python.org/mailman/listinfo/python-dev>/python-dev<https://mail.python.org/mailman/listinfo/python-dev>
Unsubscribe: https<https://mail.python.org/mailman/options/python-dev/brett%40python.org>://<https://mail.python.org/mailman/options/python-dev/brett%40python.org>mail.python.org<https://mail.python.org/mailman/options/python-dev/brett%40python.org>/mailman/options/python-dev/brett%40python.org<https://mail.python.org/mailman/options/python-dev/brett%40python.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/e0d70453/attachment-0001.html>

From martin at v.loewis.de  Sun May 18 15:00:54 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 18 May 2014 15:00:54 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
 do support?
In-Reply-To: <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
References: <CAP1=2W7+x43L7kNLStMGWLx+xHqroAXJGErLE7K9iDunovkWUg@mail.gmail.com>
 <20140515040105.GA82654@cskk.homeip.net>
 <CAP7+vJ+Y+uMnmqB3rTT8aG29e-ZfYPRm58eTLNmssMjX9UZgow@mail.gmail.com>
 <CANc-5UwvvU=brJ7GJZMUtkm3APCC873zjC+Fbp1XN=Y7jPyBbg@mail.gmail.com>
 <20140515152624.039883c7@fsol>
 <CANc-5Uw6XY=+zaA4Qmjyo5iKA95D1LQC7dPF6BTTpkFthmLu3A@mail.gmail.com>
Message-ID: <5378AF06.7030704@v.loewis.de>

Am 15.05.14 16:20, schrieb Skip Montanaro:
> On Thu, May 15, 2014 at 8:26 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> We already have such buildbots, they are in the "unstable" category.
>> You can browse through existing buildbots here:
>> https://www.python.org/dev/buildbot/
> 
> I can't see how to distinguish "stable" from "unstable" (or to view
> just the "unstable" category. What do those two categories have to do
> with "supported" and "unsupported"?

There are two competing definitions of "stable" vs. "unstable" when it
comes to buildbots.

1. "stable buildbot" (my preferred definition)
   * the buildbot is available most of the time, an operator will deal
     with it when it happens to go down, and the builds always complete.
   * consequentially, an unstable buildbot is one that tends to
     disconnect, and takes a long time for the operator to recover
2. "stable platform"
   * all tests are expected to pass all the time; i.e. there are no
     tests that randomly fail, and no platform-specific failures
   * thus, a failed test is an indication that a new bug has been
     introduced

When Antoine talked about "unofficial" buildbots, he was referring to
the fact that we accept nearly any buildbot offered, but put them into
the unstable category, which they often belong to by either definition
of unstable.

Regards,
Martin




From martin at v.loewis.de  Sun May 18 15:10:55 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 18 May 2014 15:10:55 +0200
Subject: [Python-Dev] Where is our official policy of what platforms we
 do support?
In-Reply-To: <20140514102820.0e13a141@anarchist.wooz.org>
References: <CAP1=2W4k6rwADrKFH61HcO00vzVq2TCzS4EUrBQkTM4zGAz+aw@mail.gmail.com>
 <20140514102820.0e13a141@anarchist.wooz.org>
Message-ID: <5378B15F.9040207@v.loewis.de>

Am 14.05.14 16:28, schrieb Barry Warsaw:
> On May 14, 2014, at 02:20 PM, Brett Cannon wrote:
> 
>> Do we want an official policy written down in a PEP (yes, I can write it)?
>> Should I keep closing these patches and saying that we are not adding
>> support for new operating systems and be hand-wavy about it?
> 
> Yes, I think a PEP describing both policy and implementation (i.e. which
> platforms we officially support) is worthwhile.  While I think you could write
> a new PEP, I also think it might just make sense to co-opt PEP 11 and broaden
> its scope, since as you say, removing support is only half the story.

I'd be careful to use the word "support". AFAIK, we do not offer
support for Python on any platform, beyond accepting bug reports in the
bug tracker. Paid support for Python is certainly offered by some
companies, but not by us.

IIUC, Brett is asking the question "What platforms is CPython supposed
to work on?". Thanks to autoconf, it is impossible to give a complete
list of such platforms. It might be that Python builds just fine, with
no changes required, on a system that we've never heard of.

So if you want to give a complete list of "supported" platforms, it
might be the question "What platforms do we care about?", in the sense
that someone would be willing to invest time if it stops working.
Again, as we do not offer paid support, this list might get misleadingly
long. I'd be personally willing to invest time looking into a lot of
problems - just not within the next 12 months (but surely some day :-)

The only reasonable positive platform list I can think of is
"What platforms do we consider release-critical?", in the sense of
not letting a release proceed if it fails to work on one of these
platforms. We've had such a list for a long time, it is:
- Microsoft Windows
- Linux
- Apple Mac OS X

Regards,
Martin


From benjamin at python.org  Mon May 19 02:49:18 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 18 May 2014 17:49:18 -0700
Subject: [Python-Dev] [RELEASED] Python 2.7.7 release candidate 1
Message-ID: <1400460558.31367.118834865.1A1F5777@webmail.messagingengine.com>

Greetings Python users,
Python 2.7.7 release candidate 1 is now available for download. Python
2.7.7 is a regularly scheduled bugfix release for the Python 2.7 series.
The 2.7.7 release contains fixes for two severe, if arcane, potential
security vulnerabilities. The first was the possibility of reading
arbitrary process memory using JSONDecoder.raw_decode. [1] (No other
json APIs are affected.) The second security issue is an integer
overflow in the strop module. [2] (If you don't know what the strop
module is, go ahead and forget it now.) This release also includes
months of accumulated normal bugfixes. All the changes in Python 2.7.7
are described in detail in the Misc/NEWS file of the source tarball. You
can view it online at

    http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS

Downloads are at

    https://python.org/download/releases/2.7.7/

This is a testing release. Assuming no horrible bugs are found, 2.7.7
final will be released in two weeks time. Please consider testing your
applications and libraries with the release candidate and reporting bugs
to

    http://bugs.python.org/

Enjoy,
Benjamin Peterson
2.7 Release Manager

[1] http://bugs.python.org/issue21529
[2] http://bugs.python.org/issue21530

From guido at python.org  Mon May 19 03:53:42 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 18 May 2014 18:53:42 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
Message-ID: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>

On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson <benjamin at python.org>wrote:

> Greetings Python users,
> Python 2.7.7 release candidate 1 is now available for download. [...]
>
>     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
>

So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/6af2eaa0/attachment.html>

From benjamin at python.org  Mon May 19 04:04:37 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 18 May 2014 19:04:37 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
Message-ID: <1400465077.15559.118851701.268719A5@webmail.messagingengine.com>

On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote:
> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson
> <benjamin at python.org>wrote:
> 
> > Greetings Python users,
> > Python 2.7.7 release candidate 1 is now available for download. [...]
> >
> >     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
> >
> 
> So what became of PEP 466? This Misc/NEWS only mentions
> hmac.compare_digest.

It didn't get completely done. Mostly of it will land in 2.7.8
presumably.

From benjamin at python.org  Mon May 19 04:06:33 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 18 May 2014 19:06:33 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
Message-ID: <1400465193.16230.118851701.51F23E6F@webmail.messagingengine.com>

On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote:
> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson
> <benjamin at python.org>wrote:
> 
> > Greetings Python users,
> > Python 2.7.7 release candidate 1 is now available for download. [...]
> >
> >     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
> >
> 
> So what became of PEP 466? This Misc/NEWS only mentions
> hmac.compare_digest.

It didn't get completely done. Mostly of it will land in 2.7.8
presumably.
--
Regards,
Benjamin


From benjamin at python.org  Mon May 19 04:06:34 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Sun, 18 May 2014 19:06:34 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
Message-ID: <1400465194.16051.118851701.6C5E6417@webmail.messagingengine.com>

On Sun, May 18, 2014, at 18:53, Guido van Rossum wrote:
> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson
> <benjamin at python.org>wrote:
> 
> > Greetings Python users,
> > Python 2.7.7 release candidate 1 is now available for download. [...]
> >
> >     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
> >
> 
> So what became of PEP 466? This Misc/NEWS only mentions
> hmac.compare_digest.

It didn't get completely done. Mostly of it will land in 2.7.8
presumably.
--
Regards,
Benjamin


From donald at stufft.io  Mon May 19 04:02:10 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 18 May 2014 22:02:10 -0400
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
Message-ID: <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>


On May 18, 2014, at 9:53 PM, Guido van Rossum <guido at python.org> wrote:

> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson <benjamin at python.org> wrote:
> Greetings Python users,
> Python 2.7.7 release candidate 1 is now available for download. [...]
> 
>     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
> 
> So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest.
> 
> -- 
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

The SSL changes were too large to get done before 2.7.7

The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don?t feel strong enough in C code to call it a proper review.

The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307)

The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305)

And that was everything from PEP 466.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/d0a989e9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/d0a989e9/attachment.sig>

From guido at python.org  Mon May 19 04:28:32 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 18 May 2014 19:28:32 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
 <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>
Message-ID: <CAP7+vJJp_5CFQvEj4C506gg3cwAnk=2YORh7eceas5aLDRyY6Q@mail.gmail.com>

Thanks for the update, Donald. Did anyone get any help from Red Hat or
other distros?


On Sun, May 18, 2014 at 7:02 PM, Donald Stufft <donald at stufft.io> wrote:

>
> On May 18, 2014, at 9:53 PM, Guido van Rossum <guido at python.org> wrote:
>
> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson <benjamin at python.org>wrote:
>
>> Greetings Python users,
>> Python 2.7.7 release candidate 1 is now available for download. [...]
>>
>>     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
>>
>
> So what became of PEP 466? This Misc/NEWS only mentions
> hmac.compare_digest.
>
> --
> --Guido van Rossum (python.org/~guido)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
>
>
> The SSL changes were too large to get done before 2.7.7
>
> The pbkdf2 has a patch sitting on the tracker (
> http://bugs.python.org/issue21304) Alex wanted someone to review before
> commit. I looked over it but I don?t feel strong enough in C code to call
> it a proper review.
>
> The guaranteed_algorithms bug depends on the pbkdf2 bug (
> http://bugs.python.org/issue21307)
>
> The os.urandom change had some argument and some concern that the related
> change in 3.4 was still new and had some bugs being ironed out so it was
> punted until 2.7.8 (http://bugs.python.org/issue21305)
>
> And that was everything from PEP 466.
>
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372
> DCFA
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/a0a89b31/attachment.html>

From donald at stufft.io  Mon May 19 04:33:30 2014
From: donald at stufft.io (Donald Stufft)
Date: Sun, 18 May 2014 22:33:30 -0400
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <CAP7+vJJp_5CFQvEj4C506gg3cwAnk=2YORh7eceas5aLDRyY6Q@mail.gmail.com>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
 <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>
 <CAP7+vJJp_5CFQvEj4C506gg3cwAnk=2YORh7eceas5aLDRyY6Q@mail.gmail.com>
Message-ID: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io>

Well I believe Alex did what he did during his work day @ Rackspace.

Distros specifically I don?t believe so, although both Alex and myself agreed that it
made sense for the SSL changes to wait until after the other changes since it was
the largest and most complicated. I think that?s the one where it makes the most
sense to try and garner help from Red Hat or the like.

Perhaps Nick knows someone at Red Hat we can poke to see if they?d be willing
to do the SSL patch :)

On May 18, 2014, at 10:28 PM, Guido van Rossum <guido at python.org> wrote:

> Thanks for the update, Donald. Did anyone get any help from Red Hat or other distros?
> 
> 
> On Sun, May 18, 2014 at 7:02 PM, Donald Stufft <donald at stufft.io> wrote:
> 
> On May 18, 2014, at 9:53 PM, Guido van Rossum <guido at python.org> wrote:
> 
>> On Sun, May 18, 2014 at 5:49 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> Greetings Python users,
>> Python 2.7.7 release candidate 1 is now available for download. [...]
>> 
>>     http://hg.python.org/cpython/raw-file/e32e3a9f3902/Misc/NEWS
>> 
>> So what became of PEP 466? This Misc/NEWS only mentions hmac.compare_digest.
>> 
>> -- 
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
> 
> The SSL changes were too large to get done before 2.7.7
> 
> The pbkdf2 has a patch sitting on the tracker (http://bugs.python.org/issue21304) Alex wanted someone to review before commit. I looked over it but I don?t feel strong enough in C code to call it a proper review.
> 
> The guaranteed_algorithms bug depends on the pbkdf2 bug (http://bugs.python.org/issue21307)
> 
> The os.urandom change had some argument and some concern that the related change in 3.4 was still new and had some bugs being ironed out so it was punted until 2.7.8 (http://bugs.python.org/issue21305)
> 
> And that was everything from PEP 466.
> 
> -----------------
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido)


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/6adec86c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/6adec86c/attachment-0001.sig>

From guido at python.org  Mon May 19 04:35:51 2014
From: guido at python.org (Guido van Rossum)
Date: Sun, 18 May 2014 19:35:51 -0700
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
 <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>
 <CAP7+vJJp_5CFQvEj4C506gg3cwAnk=2YORh7eceas5aLDRyY6Q@mail.gmail.com>
 <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io>
Message-ID: <CAP7+vJJASd3ZHbOBXZ8epChzKd7gMcL1sEJSoKv-9158g1YVXA@mail.gmail.com>

At the very least PEP 466 needs to be updated to admit the failure -- it
would be a shame if people read the PEP and assumed the promised features
actually landed in 2.7.7 (which the PEP explicitly lists).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/302d37d8/attachment.html>

From ncoghlan at gmail.com  Mon May 19 06:07:27 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 19 May 2014 14:07:27 +1000
Subject: [Python-Dev] Python 2.7.7 and PEP 466
In-Reply-To: <CAP7+vJJASd3ZHbOBXZ8epChzKd7gMcL1sEJSoKv-9158g1YVXA@mail.gmail.com>
References: <CAP7+vJKMbWHJ1YEW4qY01v0e+X5x0qAUGSz1uQ2pcY1bmmTWOA@mail.gmail.com>
 <E454B0A5-BF7D-442A-965C-03276F8E6EC3@stufft.io>
 <CAP7+vJJp_5CFQvEj4C506gg3cwAnk=2YORh7eceas5aLDRyY6Q@mail.gmail.com>
 <596E3A6A-BFF0-482A-B37C-3464D3EE44ED@stufft.io>
 <CAP7+vJJASd3ZHbOBXZ8epChzKd7gMcL1sEJSoKv-9158g1YVXA@mail.gmail.com>
Message-ID: <CADiSq7eFNZV-Uc2qzx+DhozwSmzmQxLcKt=9GUCQNgfSPR016w@mail.gmail.com>

On 19 May 2014 12:35, Guido van Rossum <guido at python.org> wrote:
> At the very least PEP 466 needs to be updated to admit the failure -- it
> would be a shame if people read the PEP and assumed the promised features
> actually landed in 2.7.7 (which the PEP explicitly lists).

Will do - I'll update that to reference the specific issues tracking
the implementation of the individual elements.

On a related note, I was also thinking about adding a new section to
the What's New in Python 2.7 doc. Specifically, a new "Security
Enhancements in Maintenance Releases" section after the existing "The
Future of Python 2.x" section. That would reference PEP 466 for
background, and then list the specific maintenance releases where
these features have been added (so just the one 2.7.7 entry for
hmac.compare_digest to start with).

I'd also add a direct link to PEP 373 (the 2.7 release schedule PEP)
from the first bullet point under "The Future of Python 2.x" section
(as well as rewording that point to better reflect the current state
of things)

Regards,
Nick.

P.S. As far as additional development resources for long term upstream
CPython maintenance go - I'm working on it (and my understanding is
that folks at other orgs are as well). Personally, I'm still in the
gap between "that's likely a good idea" and actually translating the
concept into available developer time. While Heartbleed has helped
raise awareness of the whole "What are we depending on without
committing sufficient development resources to long term maintenance?"
problem, large orgs still don't tend to move that fast :)

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From tjreedy at udel.edu  Mon May 19 06:52:58 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 19 May 2014 00:52:58 -0400
Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits,
	what do I do?
Message-ID: <llc2nn$6sd$1@ger.gmane.org>

An hour ago, I pulled recent commits to my local repository.
I edited a couple of files, wrote commit messages, and pulled again, 
nothing new. I then did the usual: commit 2.7, commit 3.4, merge to 3.5.

Problem: merge conflicts from 6 files I have never touched.
Include/patchlevel.h
Lib/distutils/__init__.py
Lib/idlelib/idlever.py
Lib/pydoc_data/topics.py
Misc/RPM/python-3.5.spec
README

I stopped at this point and ran diskcheck. I then looked at the DAG and 
noticed 5 previous 3.4 patches that were not merged into 3.5: rev 90750 
and 90751, Larry Hastings, 2 weeks ago!, rev
90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 
hours ago. From earliest to lastest:

c67a19e11a71
7c5f1b200a24
35ea333f43bd
31211947387b
854fd6eeee2f

If there was any notice on the Committer's list about not making 
commits, I did not get it. In fact, I do not remember getting anything 
on that list for at about a month. Besides that, I had no problem 3 days 
ago, and noticed the commit by Raymond, though not the absence of a merge.

In any case, what do I do now, short of deleting and re-cloning, so I do 
not make anything worse?

-- 
Terry Jan Reedy


From raymond.hettinger at gmail.com  Mon May 19 07:31:10 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 19 May 2014 06:31:10 +0100
Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits,
	what do I do?
In-Reply-To: <llc2nn$6sd$1@ger.gmane.org>
References: <llc2nn$6sd$1@ger.gmane.org>
Message-ID: <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com>


On May 19, 2014, at 5:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> I stopped at this point and ran diskcheck. I then looked at the DAG and noticed 5 previous 3.4 patches that were not merged into 3.5: rev 90750 and 90751, Larry Hastings, 2 weeks ago!, rev
> 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 hours ago. From earliest to lastest:
> 
> c67a19e11a71
> 7c5f1b200a24
> 35ea333f43bd
> 31211947387b
> 854fd6eeee2f
> 
> If there was any notice on the Committer's list about not making commits, I did not get it. In fact, I do not remember getting anything on that list for at about a month. Besides that, I had no problem 3 days ago, and noticed the commit by Raymond, though not the absence of a merge.
> 
> In any case, what do I do now, short of deleting and re-cloning, so I do not make anything worse?

I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges, all with conflicts).  

To get my repo back into a usable state, I ran "hg update --clean" and am waiting for Larry to do his cleans-up. Once his are in, I'll merge my last 3.4 commit (which should be easy since it has no conflicts).


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140519/62f946a1/attachment.html>

From tjreedy at udel.edu  Mon May 19 08:08:05 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 19 May 2014 02:08:05 -0400
Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits,
	what do I do?
In-Reply-To: <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com>
References: <llc2nn$6sd$1@ger.gmane.org>
 <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com>
Message-ID: <llc74i$ptg$1@ger.gmane.org>

On 5/19/2014 1:31 AM, Raymond Hettinger wrote:
>
> On May 19, 2014, at 5:52 AM, Terry Reedy <tjreedy at udel.edu
> <mailto:tjreedy at udel.edu>> wrote:
>
>> I stopped at this point and ran diskcheck. I then looked at the DAG
>> and noticed 5 previous 3.4 patches that were not merged into 3.5: rev
>> 90750 and 90751, Larry Hastings, 2 weeks ago!, rev
>> 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8
>> hours ago. From earliest to lastest:
>>
>> c67a19e11a71
>> 7c5f1b200a24
>> 35ea333f43bd
>> 31211947387b
>> 854fd6eeee2f
>>
>> If there was any notice on the Committer's list about not making
>> commits, I did not get it. In fact, I do not remember getting anything
>> on that list for at about a month. Besides that, I had no problem 3
>> days ago, and noticed the commit by Raymond, though not the absence of
>> a merge.
>>
>> In any case, what do I do now, short of deleting and re-cloning, so I
>> do not make anything worse?
>
> I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges,
> all with conflicts).
>
> To get my repo back into a usable state, I ran "hg update --clean"

On Win 7, in TortoiseHG Workbench, command line

py35% hg update --clean
% hg update --clean
abort: index 00changelog.i unknown format 2!
[command returned code 255 Mon May 19 02:04:15 2014]
py35%

Same at command prompt in py35 directory.

> am waiting for Larry to do his cleans-up. Once his are in, I'll merge my
> last 3.4 commit (which should be easy since it has no conflicts).


-- 
Terry Jan Reedy


From larry at hastings.org  Mon May 19 08:27:31 2014
From: larry at hastings.org (Larry Hastings)
Date: Sun, 18 May 2014 23:27:31 -0700
Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits,
 what do I do?
In-Reply-To: <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com>
References: <llc2nn$6sd$1@ger.gmane.org>
 <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com>
Message-ID: <5379A453.50404@hastings.org>

On 05/18/2014 10:31 PM, Raymond Hettinger wrote:
>
> On May 19, 2014, at 5:52 AM, Terry Reedy <tjreedy at udel.edu 
> <mailto:tjreedy at udel.edu>> wrote:
>
>> I stopped at this point and ran diskcheck. I then looked at the DAG 
>> and noticed 5 previous 3.4 patches that were not merged into 3.5: rev 
>> 90750 and 90751, Larry Hastings, 2 weeks ago!, rev
>> 90752 and 90753 by Larry a day ago, and 90755 by Raymond Hettinger 8 
>> hours ago. From earliest to lastest:
>>
>> c67a19e11a71
>> 7c5f1b200a24
>> 35ea333f43bd
>> 31211947387b
>> 854fd6eeee2f
>>
>> If there was any notice on the Committer's list about not making 
>> commits, I did not get it. In fact, I do not remember getting 
>> anything on that list for at about a month. Besides that, I had no 
>> problem 3 days ago, and noticed the commit by Raymond, though not the 
>> absence of a merge.
>>
>> In any case, what do I do now, short of deleting and re-cloning, so I 
>> do not make anything worse?
>
> I bumped into this a few hours ago (a bunch of uncommitted 3.4 merges, 
> all with conflicts).
>
> To get my repo back into a usable state, I ran "hg update --clean" and 
> am waiting for Larry to do his cleans-up. Once his are in, I'll merge 
> my last 3.4 commit (which should be easy since it has no conflicts).


I think I've got everything merged properly.  So cry havoc and let slip 
the merge of howto/sockets.rst!


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/f724f526/attachment.html>

From victor.stinner at gmail.com  Mon May 19 09:54:28 2014
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 19 May 2014 09:54:28 +0200
Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.4.1
In-Reply-To: <53799E17.7040805@hastings.org>
References: <53799E17.7040805@hastings.org>
Message-ID: <CAMpsgwZGLk921voTUMpLYgu2ZBY4v_CxKZaOhtWWasRBDft-Mg@mail.gmail.com>

It's not easy to find the changelog. I found this page:
https://docs.python.org/3.4/whatsnew/changelog.html

Victor

2014-05-19 8:00 GMT+02:00 Larry Hastings <larry at hastings.org>:
>
>
> On behalf of the Python development community and the Python 3.4 release
> team, I'm pleased to announce the availability of Python 3.4.1.  Python
> 3.4.1 has over three hundred bugfixes and other improvements over 3.4.0. One
> notable change: the version of OpenSSL bundled with the Windows installer no
> longer has the "HeartBleed" vulnerability.
>
> You can download it here:
>
> https://www.python.org/download/releases/3.4.1
>
>
>
> /arry
>
> _______________________________________________
> python-committers mailing list
> python-committers at python.org
> https://mail.python.org/mailman/listinfo/python-committers
>

From larry at hastings.org  Mon May 19 08:00:55 2014
From: larry at hastings.org (Larry Hastings)
Date: Sun, 18 May 2014 23:00:55 -0700
Subject: [Python-Dev] [RELEASED] Python 3.4.1
Message-ID: <53799E17.7040805@hastings.org>



On behalf of the Python development community and the Python 3.4 release 
team, I'm pleased to announce the availability of Python 3.4.1.  Python 
3.4.1 has over three hundred bugfixes and other improvements over 3.4.0. 
One notable change: the version of OpenSSL bundled with the Windows 
installer no longer has the "HeartBleed" vulnerability.

You can download it here:

    https://www.python.org/download/releases/3.4.1



//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140518/cda8d167/attachment.html>

From hrvoje.niksic at avl.com  Mon May 19 16:20:01 2014
From: hrvoje.niksic at avl.com (Hrvoje Niksic)
Date: Mon, 19 May 2014 16:20:01 +0200
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <ll76fd$34u$1@ger.gmane.org>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <ll76fd$34u$1@ger.gmane.org>
Message-ID: <537A1311.9010008@avl.com>

On 05/17/2014 10:26 AM, Terry Reedy wrote:
 > When list.pop was added, the convention was changed to
 > "do not return the 'self' parameter"

Do you have a reference for this? It is my understanding that the 
convention is for mutators to return None, in order to make it clear 
that the change is destructive. For example, the tutorial at 
https://docs.python.org/3.4/tutorial/datastructures.html says:

"""
You might have noticed that methods like insert, remove or sort that 
modify the list have no return value printed ? they return None. [1] 
This is a design principle for all mutable data structures in Python.
"""

Methods like list.pop and dict.pop would seem like a case of 
"practicality beats purity" because it's more convenient (and faster) to 
delete and retrieve value at the same go.


From tjreedy at udel.edu  Tue May 20 21:50:31 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 20 May 2014 15:50:31 -0400
Subject: [Python-Dev] Merge conflicts from unmerged 3.4 commits,
	what do I do?
In-Reply-To: <llc74i$ptg$1@ger.gmane.org>
References: <llc2nn$6sd$1@ger.gmane.org>
 <C613A076-F8E4-4846-9B64-2D1B57C053B5@gmail.com> <llc74i$ptg$1@ger.gmane.org>
Message-ID: <llgbml$4t7$1@ger.gmane.org>

On 5/19/2014 2:08 AM, Terry Reedy wrote:
> On 5/19/2014 1:31 AM, Raymond Hettinger wrote:

>> To get my repo back into a usable state, I ran "hg update --clean"

> py35% hg update --clean
> % hg update --clean
> abort: index 00changelog.i unknown format 2!

After exiting Workbench and rebooting, update --clean worked fine.

-- 
Terry Jan Reedy


From tjreedy at udel.edu  Tue May 20 23:47:06 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 20 May 2014 17:47:06 -0400
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <537A1311.9010008@avl.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <ll76fd$34u$1@ger.gmane.org> <537A1311.9010008@avl.com>
Message-ID: <llgih8$ods$1@ger.gmane.org>

On 5/19/2014 10:20 AM, Hrvoje Niksic wrote:
> On 05/17/2014 10:26 AM, Terry Reedy wrote:
>  > When list.pop was added, the convention was changed to
>  > "do not return the 'self' parameter"
>
> Do you have a reference for this?

I think the fact that Guido accepted, in 2000, my 1999 proposal, with 
generalizations, at Tim Peter's suggestion, and that other pop methods 
and iterator.__next__ methods have been added since, speaks for itself.

However, here is the reference to my original PEP-like post and the 
subsequent discussion. I consider the subthread about the convention as 
the most important part of the discussion, as I considered it the most 
salient objection to the proposal.

https://groups.google.com/forum/#!topic/comp.lang.python/SKJq3S2ZYmg

Mike Meyer said "While I personally like the idea of a .pop method for 
lists, it seems to violate one of the design principles that Guido uses: 
that methods either modify objects, or return values from/about it, but 
not both."

John (Max) Skaller, in his second response, noted that Alex Stepanov 
used the same principle when designing STL, but the C++ modified his 
design for convenience and efficiency.

In my first response, I suggested that if a convention prevents writing 
a proper stack, queue, or deque class, all of which are computer science 
standards, or if a convention prevents pairing a inverse with a 
function, as is standard in mathematics, then the convention should be 
re-examined.

In Guido's first response he settled the particular question by saying 
"To implement a stack, one would need to add a list.pop() primitive
(and no, I'm not against this particular one on the basis of any
principle)."  At the time, he was just more inclined "to leave the list 
data type alone and implement a stack as a class that uses a list for 
its representation."

That covers half the thread.

 > It is my understanding that the
> convention is for mutators to return None,

I would say that the convention was to return nothing, which in Python 
means accepting the default return of None. In a language that allowed 
procedure methods, there would literally be no return. Python's 
interactive read-eval-print loop does not print None because it usually 
represents a non-return.

 > in order to make it clear that the change is destructive.

and in particular, to disallow chaining by not returning 'self'.

 > For example, the tutorial at
> https://docs.python.org/3.4/tutorial/datastructures.html says:
>
> """
> You might have noticed that methods like insert, remove or sort that
> modify the list have no return value printed ? they return None. [1]
> This is a design principle for all mutable data structures in Python.
> """

Good catch. This only needs 'only' inserted before 'modify' to become 
true again. I opened http://bugs.python.org/issue21545
Note that the point of the comment is to explain the apparent absence of 
a return in the r.e.p. loop, the same as in a language with no-return 
procedures.

 >>> a.sort()
 >>>

-- 
Terry Jan Reedy



From chris.barker at noaa.gov  Tue May 20 18:30:47 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 20 May 2014 09:30:47 -0700
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
Message-ID: <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>

>
> >>>> [].sort() is None
> > True
> >>>> "ABC".lower() is None
> > False
> >
> > That's a deliberate design choice, and one that has been explained a
> > few times on the list when folks ask why "[].sort().reverse()" doesn't
> > work when "'ABC'.lower().replace('-', '_')" does.
> >
> > Would it be worth adding such a note? Or is it out of scope?
>

Is there a reference anywhere as to *why* the convention in Python is to do
it that way?

Personally, I often miss the ability to chain operations on mutable
objects, but I can only imagine that that design decision was made for good
reason. However, as I teach Python, I find I have nothing to say other than
"that's the way it's done in Python".

I note in a way that there is a bit of backwards logic:

immutable objects **have** to return something, so they naturally chain.
It's almost as if the philosophy of the language is that you don't want
chaining behavior, and it's really just a side effect of immutables that
you get it with them.

I suppose the argument could be that for mutable objects, returning None is
an indicator that you are a) working with an mutable object, and b) that
the method changes the internal state. But the .pop() example demonstrates
that a method can both return something meaningful, and change internal
state, so I'm not sure it's really a distinction worth making.

So again: is there a clear explanation of the logic behind the convention
posted somewhere?

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140520/3a1f56b2/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed May 21 00:29:50 2014
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 21 May 2014 10:29:50 +1200
Subject: [Python-Dev] Returning None from methods that mutate
	object	state
In-Reply-To: <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
 <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
Message-ID: <537BD75E.7090101@canterbury.ac.nz>

Chris Barker wrote:
> Personally, I often miss the ability to chain operations on mutable 
> objects, but I can only imagine that that design decision was made for 
> good reason. However, as I teach Python, I find I have nothing to say 
> other than "that's the way it's done in Python".

Python has better ways of doing many of the things that
method chaining is used for in other languages. In Java,
for example, I often see it used as a somewhat klunky
workaround for the lack of keyword arguments when
initialising objects.

Other than that, it seems to be mainly for stuffing
multiple operations into one line, which is not something
the Python style generally goes in for.

> I suppose the argument could be that for mutable objects, returning None 
> is an indicator that you are a) working with an mutable object, and b) 
> that the method changes the internal state. But the .pop() example 
> demonstrates that a method can both return something meaningful, and 
> change internal state, so I'm not sure it's really a distinction worth 
> making.

It's not just about mutating the object, it's about a
mutating method with a name that could also plausibly
be the name of a non-mutating method. The canonical
example is sort(), which, if you didn't already know,
could equally well be mutating or non-mutating. In
those cases, I think it's worth making it difficult
to get wrong.

This doesn't apply so much to pop(), which sounds
much more like a mutating method than a non-mutating
one, so it's less likely you'll make a mistake.

-- 
Greg

From rdmurray at bitdance.com  Wed May 21 01:50:14 2014
From: rdmurray at bitdance.com (R. David Murray)
Date: Tue, 20 May 2014 19:50:14 -0400
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
 <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
Message-ID: <20140520235015.813F6250D4D@webabinitio.net>

On Tue, 20 May 2014 09:30:47 -0700, Chris Barker <chris.barker at noaa.gov> wrote:
> >
> > >>>> [].sort() is None
> > > True
> > >>>> "ABC".lower() is None
> > > False
> > >
> > > That's a deliberate design choice, and one that has been explained a
> > > few times on the list when folks ask why "[].sort().reverse()" doesn't
> > > work when "'ABC'.lower().replace('-', '_')" does.
> > >
> > > Would it be worth adding such a note? Or is it out of scope?
> >
> 
> Is there a reference anywhere as to *why* the convention in Python is to do
> it that way?
> 
> Personally, I often miss the ability to chain operations on mutable
> objects, but I can only imagine that that design decision was made for good
> reason. However, as I teach Python, I find I have nothing to say other than
> "that's the way it's done in Python".
> 
> I note in a way that there is a bit of backwards logic:
> 
> immutable objects **have** to return something, so they naturally chain.
> It's almost as if the philosophy of the language is that you don't want
> chaining behavior, and it's really just a side effect of immutables that
> you get it with them.
> 
> I suppose the argument could be that for mutable objects, returning None is
> an indicator that you are a) working with an mutable object, and b) that
> the method changes the internal state. But the .pop() example demonstrates
> that a method can both return something meaningful, and change internal
> state, so I'm not sure it's really a distinction worth making.
> 
> So again: is there a clear explanation of the logic behind the convention
> posted somewhere?

I think it is exactly about not encouraging chaining of mutations, but I
could be wrong :)

--David

From tjreedy at udel.edu  Wed May 21 01:56:37 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 20 May 2014 19:56:37 -0400
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
 <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
Message-ID: <llgq44$6qc$1@ger.gmane.org>

On 5/20/2014 12:30 PM, Chris Barker wrote:
>      >>>> [].sort() is None
>      > True
>      >>>> "ABC".lower() is None
>      > False

> Is there a reference anywhere as to *why* the convention in Python is to
> do it that way?

In short, reducing bugs induced by mutation of aliased objects. 
Functional languages evade the problem by prohibiting mutation 
(sometimes at the cost of inefficiency).

In an alternate universe, the example above might become

 >>> a = []; a.sort() is a
True
 >>> a = "ABC"' a.lower() is a
False

As I suggested earlier, having pure mutation methods not return anything 
made is easy to suggest a mutation + non-self return method, list.pop. 
If all mutation methods had previously returned 'self', there might have 
been disagreement over whether the item return should augment or replace 
the self return. Before you say the latter, consider the inconsistency 
of only sometimes returning self and the potential consistency between

 >>> most, last = 'a b c'.rsplit(maxsplit=1)
 >>> most, last
('a b', 'c')

 >>> most, last = [0, 1, 2].pop()
 >>> most, last
([0, 1], 2)

One could also consider first, rest pairings.

-- 
Terry Jan Reedy


From guido at python.org  Wed May 21 03:40:10 2014
From: guido at python.org (Guido van Rossum)
Date: Tue, 20 May 2014 18:40:10 -0700
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <537BD75E.7090101@canterbury.ac.nz>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
 <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
 <537BD75E.7090101@canterbury.ac.nz>
Message-ID: <CAP7+vJKkCtMZWJntFU8xSEDCEVC+Sy_HXAXsW-i=+iNESJCUEQ@mail.gmail.com>

On Tuesday, May 20, 2014, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:

> Chris Barker wrote:
>
>> Personally, I often miss the ability to chain operations on mutable
>> objects, but I can only imagine that that design decision was made for good
>> reason. However, as I teach Python, I find I have nothing to say other than
>> "that's the way it's done in Python".
>>
>
> Python has better ways of doing many of the things that
> method chaining is used for in other languages. In Java,
> for example, I often see it used as a somewhat klunky
> workaround for the lack of keyword arguments when
> initialising objects.
>
> Other than that, it seems to be mainly for stuffing
> multiple operations into one line, which is not something
> the Python style generally goes in for.
>
>  I suppose the argument could be that for mutable objects, returning None
>> is an indicator that you are a) working with an mutable object, and b) that
>> the method changes the internal state. But the .pop() example demonstrates
>> that a method can both return something meaningful, and change internal
>> state, so I'm not sure it's really a distinction worth making.
>>
>
> It's not just about mutating the object, it's about a
> mutating method with a name that could also plausibly
> be the name of a non-mutating method. The canonical
> example is sort(), which, if you didn't already know,
> could equally well be mutating or non-mutating. In
> those cases, I think it's worth making it difficult
> to get wrong.


This is the key reason, by far the most important.


> This doesn't apply so much to pop(), which sounds
> much more like a mutating method than a non-mutating
> one, so it's less likely you'll make a mistake.
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>


-- 
--Guido van Rossum (on iPad)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140520/e7c6c2bc/attachment.html>

From chris.barker at noaa.gov  Thu May 22 01:33:54 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Wed, 21 May 2014 16:33:54 -0700
Subject: [Python-Dev] Returning None from methods that mutate object
	state
In-Reply-To: <llgq44$6qc$1@ger.gmane.org>
References: <CADiSq7dsS97pfKsRCY3dTiN2ih4NCWy1DCZGAaWvBKu6bj8KJA@mail.gmail.com>
 <CAPJVwB=NXaYYFwvyqDPsaSuhRU1VDNz1bF+D6YMSGojDTHMN9Q@mail.gmail.com>
 <CALGmxELQhWyFWvnRD+Mv7ZsjNAxg3Zi+3WXxyMCa+UWiKM6r0Q@mail.gmail.com>
 <llgq44$6qc$1@ger.gmane.org>
Message-ID: <CALGmxE+20KnLJ_EB08uFHkf5PE4pKET67+wL7DMBKJ0nqRAcYQ@mail.gmail.com>

Thanks all,

Now I need to try to sum this all up to present to my students. ;-)

-Chris





On Tue, May 20, 2014 at 4:56 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 5/20/2014 12:30 PM, Chris Barker wrote:
>
>>      >>>> [].sort() is None
>>      > True
>>      >>>> "ABC".lower() is None
>>      > False
>>
>
>  Is there a reference anywhere as to *why* the convention in Python is to
>> do it that way?
>>
>
> In short, reducing bugs induced by mutation of aliased objects. Functional
> languages evade the problem by prohibiting mutation (sometimes at the cost
> of inefficiency).
>
> In an alternate universe, the example above might become
>
> >>> a = []; a.sort() is a
> True
> >>> a = "ABC"' a.lower() is a
> False
>
> As I suggested earlier, having pure mutation methods not return anything
> made is easy to suggest a mutation + non-self return method, list.pop. If
> all mutation methods had previously returned 'self', there might have been
> disagreement over whether the item return should augment or replace the
> self return. Before you say the latter, consider the inconsistency of only
> sometimes returning self and the potential consistency between
>
> >>> most, last = 'a b c'.rsplit(maxsplit=1)
> >>> most, last
> ('a b', 'c')
>
> >>> most, last = [0, 1, 2].pop()
> >>> most, last
> ([0, 1], 2)
>
> One could also consider first, rest pairings.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140521/ef559359/attachment.html>

From Chandra.Srinivasan at synopsys.com  Fri May 23 17:49:31 2014
From: Chandra.Srinivasan at synopsys.com (Chandra Srinivasan)
Date: Fri, 23 May 2014 15:49:31 +0000
Subject: [Python-Dev] Unexpected increase of globals() refcount
Message-ID: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com>

Hi,
I ran the following code in the Python interpreter and am trying to determine if the behavior I see is expected:

import sys
print sys.getrefcount(globals())
class Foo(object):
   def __init__(self):
      pass
print sys.getrefcount(globals())

The first print statement above prints '4' and the second one prints '5'. However, if I remove the __init__ method from the class, the refcount stays the same.

If I change the above code like this, the ref count stays the same:

import gc
import sys
print sys.getrefcount(globals())
class Foo(object):
   def __init__(self):
      pass
del Foo
while gc.collect():
  pass
print sys.getrefcount(globals())

Can you let me know if this is a bug in the Python interpreter?

-Chandra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140523/d6ee9a18/attachment.html>

From benjamin at python.org  Fri May 23 18:03:03 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 23 May 2014 09:03:03 -0700
Subject: [Python-Dev] Unexpected increase of globals() refcount
In-Reply-To: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com>
References: <8E464094D6501844A38D24D47ACEC221EC935B45@us01wembx1.internal.synopsys.com>
Message-ID: <1400860983.12094.120843053.24A2D612@webmail.messagingengine.com>

On Fri, May 23, 2014, at 8:49, Chandra Srinivasan wrote:
> Hi,
> I ran the following code in the Python interpreter and am trying to
> determine if the behavior I see is expected:
> 
> import sys
> print sys.getrefcount(globals())
> class Foo(object):
>    def __init__(self):
>       pass
> print sys.getrefcount(globals())
> 
> The first print statement above prints '4' and the second one prints '5'.
> However, if I remove the __init__ method from the class, the refcount
> stays the same.
> 
> If I change the above code like this, the ref count stays the same:
> 
> import gc
> import sys
> print sys.getrefcount(globals())
> class Foo(object):
>    def __init__(self):
>       pass
> del Foo
> while gc.collect():
>   pass
> print sys.getrefcount(globals())
> 
> Can you let me know if this is a bug in the Python interpreter?

No, functions hold a reference to the globals of the module they are
defined in.

From status at bugs.python.org  Fri May 23 18:07:56 2014
From: status at bugs.python.org (Python tracker)
Date: Fri, 23 May 2014 18:07:56 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20140523160756.50AD656A31@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2014-05-16 - 2014-05-23)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4622 ( +8)
  closed 28710 (+36)
  total  33332 (+44)

Open issues with patches: 2129 


Issues opened (26)
==================

#21518: Expose RegUnloadKey in winreg
http://bugs.python.org/issue21518  opened by Claudiu.Popa

#21519: IDLE : Bug in keybinding validity check
http://bugs.python.org/issue21519  opened by sahutd

#21520: Erroneous zipfile test failure if the string 'bad' appears in 
http://bugs.python.org/issue21520  opened by larry

#21521: Tkinter + OSX + Spaces : Multiple file dialogues created
http://bugs.python.org/issue21521  opened by rovf

#21524: Allowing to pass pathlib.Path object in mimetypes.guess_type f
http://bugs.python.org/issue21524  opened by jorispilot

#21526: Support new booleans in Tkinter
http://bugs.python.org/issue21526  opened by serhiy.storchaka

#21527: concurrent.futures.ThreadPoolExecutor does not use a default v
http://bugs.python.org/issue21527  opened by Claudiu.Popa

#21531: Sending a zero-length UDP packet to asyncore invokes handle_cl
http://bugs.python.org/issue21531  opened by Tony.Gedge

#21532: 2.7.7rc1 msi is lacking libpython27.a
http://bugs.python.org/issue21532  opened by loewis

#21533: built-in types dict docs - construct dict from iterable, not i
http://bugs.python.org/issue21533  opened by wolma

#21534: 404 on documentation download links
http://bugs.python.org/issue21534  opened by zach.ware

#21536: extension built with a shared python cannot be loaded with a s
http://bugs.python.org/issue21536  opened by pitrou

#21539: pathlib's Path.mkdir() should allow for "mkdir -p" functionali
http://bugs.python.org/issue21539  opened by garrison

#21541: Provide configure option --with-ssl for compilation with custo
http://bugs.python.org/issue21541  opened by machyniak

#21546: int('\0') gives wrong error message
http://bugs.python.org/issue21546  opened by Kurt.Rose

#21547: '!s' formatting documentation bug
http://bugs.python.org/issue21547  opened by Joshua.Landau

#21548: pydoc -k IndexError on empty docstring
http://bugs.python.org/issue21548  opened by Dima.Tisnek

#21549: Add the members parameter for TarFile.list()
http://bugs.python.org/issue21549  opened by serhiy.storchaka

#21550: Add Python implementation of the tar utility
http://bugs.python.org/issue21550  opened by serhiy.storchaka

#21552: String length overflow in Tkinter
http://bugs.python.org/issue21552  opened by serhiy.storchaka

#21555: gcmodule.c could use pytime.h
http://bugs.python.org/issue21555  opened by pitrou

#21556: try to use hashtable in pickle
http://bugs.python.org/issue21556  opened by neologix

#21557: os.popen & os.system lack shell-related security warnings
http://bugs.python.org/issue21557  opened by cvrebert

#21558: Fix a typo in the contextlib docs
http://bugs.python.org/issue21558  opened by berker.peksag

#21559: OverflowError should not happen for integer operations
http://bugs.python.org/issue21559  opened by theme

#21560: gzip.write changes trailer ISIZE field before type checking - 
http://bugs.python.org/issue21560  opened by wolma



Most recent 15 issues with no replies (15)
==========================================

#21559: OverflowError should not happen for integer operations
http://bugs.python.org/issue21559

#21558: Fix a typo in the contextlib docs
http://bugs.python.org/issue21558

#21557: os.popen & os.system lack shell-related security warnings
http://bugs.python.org/issue21557

#21552: String length overflow in Tkinter
http://bugs.python.org/issue21552

#21548: pydoc -k IndexError on empty docstring
http://bugs.python.org/issue21548

#21533: built-in types dict docs - construct dict from iterable, not i
http://bugs.python.org/issue21533

#21526: Support new booleans in Tkinter
http://bugs.python.org/issue21526

#21524: Allowing to pass pathlib.Path object in mimetypes.guess_type f
http://bugs.python.org/issue21524

#21521: Tkinter + OSX + Spaces : Multiple file dialogues created
http://bugs.python.org/issue21521

#21514: update json module docs in light of RFC 7159 & ECMA-404
http://bugs.python.org/issue21514

#21505: cx_freeze multiprocessing bug
http://bugs.python.org/issue21505

#21502: freeze.py not working properly with OS X framework builds
http://bugs.python.org/issue21502

#21500: Make use of the "load_tests" protocol in test_importlib packag
http://bugs.python.org/issue21500

#21498: configparser accepts keys beginning with comment_chars when wr
http://bugs.python.org/issue21498

#21493: Add test for ntpath.expanduser
http://bugs.python.org/issue21493



Most recent 15 issues waiting for review (15)
=============================================

#21560: gzip.write changes trailer ISIZE field before type checking - 
http://bugs.python.org/issue21560

#21558: Fix a typo in the contextlib docs
http://bugs.python.org/issue21558

#21556: try to use hashtable in pickle
http://bugs.python.org/issue21556

#21555: gcmodule.c could use pytime.h
http://bugs.python.org/issue21555

#21552: String length overflow in Tkinter
http://bugs.python.org/issue21552

#21549: Add the members parameter for TarFile.list()
http://bugs.python.org/issue21549

#21541: Provide configure option --with-ssl for compilation with custo
http://bugs.python.org/issue21541

#21539: pathlib's Path.mkdir() should allow for "mkdir -p" functionali
http://bugs.python.org/issue21539

#21533: built-in types dict docs - construct dict from iterable, not i
http://bugs.python.org/issue21533

#21527: concurrent.futures.ThreadPoolExecutor does not use a default v
http://bugs.python.org/issue21527

#21526: Support new booleans in Tkinter
http://bugs.python.org/issue21526

#21519: IDLE : Bug in keybinding validity check
http://bugs.python.org/issue21519

#21518: Expose RegUnloadKey in winreg
http://bugs.python.org/issue21518

#21515: Use Linux O_TMPFILE flag in tempfile.TemporaryFile?
http://bugs.python.org/issue21515

#21513: speed up some ipaddress properties
http://bugs.python.org/issue21513



Top 10 most discussed issues (10)
=================================

#14776: Add SystemTap static markers
http://bugs.python.org/issue14776   8 msgs

#20584: On FreeBSD, signal.NSIG is smaller than biggest signal value
http://bugs.python.org/issue20584   7 msgs

#21516: pathlib.Path(...).is_dir() crashes on some directories (Window
http://bugs.python.org/issue21516   6 msgs

#21556: try to use hashtable in pickle
http://bugs.python.org/issue21556   6 msgs

#21304: PEP 466: Backport hashlib.pbkdf2_hmac to Python 2.7
http://bugs.python.org/issue21304   5 msgs

#20611: socket.create_connection() doesn't handle EINTR properly
http://bugs.python.org/issue20611   4 msgs

#21145: Add the @cached_property decorator
http://bugs.python.org/issue21145   4 msgs

#21511: Thinko in Lib/quopri.py
http://bugs.python.org/issue21511   4 msgs

#21518: Expose RegUnloadKey in winreg
http://bugs.python.org/issue21518   4 msgs

#21560: gzip.write changes trailer ISIZE field before type checking - 
http://bugs.python.org/issue21560   4 msgs



Issues closed (35)
==================

#7776: http.client.HTTPConnection tunneling is broken
http://bugs.python.org/issue7776  closed by larry

#9673: Entry Widget Not Editable under Windows XP
http://bugs.python.org/issue9673  closed by terry.reedy

#10744: ctypes arrays have incorrect buffer information (PEP-3118)
http://bugs.python.org/issue10744  closed by python-dev

#19866: tests aifc, sunau and wave failures on a fresh Win64 installat
http://bugs.python.org/issue19866  closed by python-dev

#20620: Update the min()/max() docs for the new default argument
http://bugs.python.org/issue20620  closed by rhettinger

#20635: Fix the grid geometry manager and add tests for geometry manag
http://bugs.python.org/issue20635  closed by serhiy.storchaka

#21027: difflib new cli interface
http://bugs.python.org/issue21027  closed by rhettinger

#21198: Minor tarfile documentation bug
http://bugs.python.org/issue21198  closed by rhettinger

#21213: Memory bomb by incorrect custom serializer to json.dumps
http://bugs.python.org/issue21213  closed by rhettinger

#21362: concurrent.futures does not validate that max_workers is prope
http://bugs.python.org/issue21362  closed by bquinlan

#21430: Document ssl.pending()
http://bugs.python.org/issue21430  closed by pitrou

#21455: add default backlog to socket.listen()
http://bugs.python.org/issue21455  closed by neologix

#21479: Document TarFile.open() as a classmethod
http://bugs.python.org/issue21479  closed by rhettinger

#21484: More clarity needed about difference between "x += e" and "x =
http://bugs.python.org/issue21484  closed by rhettinger

#21495: Sane default for logging config
http://bugs.python.org/issue21495  closed by terry.reedy

#21503: Use test_both() consistently throughout test_importlib
http://bugs.python.org/issue21503  closed by eric.snow

#21507: memory used by frozenset created from set differs from that of
http://bugs.python.org/issue21507  closed by rhettinger

#21517: installer Python default setting fails with mac Python Launche
http://bugs.python.org/issue21517  closed by ned.deily

#21522: Add more tkinter tests
http://bugs.python.org/issue21522  closed by serhiy.storchaka

#21523: quadratic-time compilation in the number of 'and' or 'or' expr
http://bugs.python.org/issue21523  closed by pitrou

#21525: Accept lists in Tkinter
http://bugs.python.org/issue21525  closed by serhiy.storchaka

#21528: Fix a number of typos in the documentation
http://bugs.python.org/issue21528  closed by dstufft

#21529: JSON module: reading arbitrary process memory
http://bugs.python.org/issue21529  closed by benjamin.peterson

#21530: Integer overflow in strop
http://bugs.python.org/issue21530  closed by benjamin.peterson

#21535: test_license_exists_at_url fails with 3.4.1, wrong/unexpected 
http://bugs.python.org/issue21535  closed by ned.deily

#21537: functools.lru_cache does not cache exceptions
http://bugs.python.org/issue21537  closed by rhettinger

#21538: plistlib unable to load iOS7 Safari History.plist
http://bugs.python.org/issue21538  closed by slo.sleuth

#21540: PEP 8 should recommend "is not" and "not in"
http://bugs.python.org/issue21540  closed by barry

#21542: pprint doesn't work well for counters, sometimes shows them li
http://bugs.python.org/issue21542  closed by rhettinger

#21543: json library fails to serialize objects such as datetime
http://bugs.python.org/issue21543  closed by r.david.murray

#21544: Spam
http://bugs.python.org/issue21544  closed by berker.peksag

#21545: Tutorial: examples and comment about mutation methods
http://bugs.python.org/issue21545  closed by terry.reedy

#21551: Behavior of word boundaries in regexes unexpected
http://bugs.python.org/issue21551  closed by r.david.murray

#21553: Behaviour of modules depends on how they where imported
http://bugs.python.org/issue21553  closed by eric.snow

#21554: Possible Error in "Brief Tour of the Standard Library"
http://bugs.python.org/issue21554  closed by rhettinger

From herbertg at gmx.at  Sat May 24 11:55:23 2014
From: herbertg at gmx.at (Herbert Griebel)
Date: Sat, 24 May 2014 11:55:23 +0200
Subject: [Python-Dev] [RELEASED] Python 3.4.1
Message-ID: <trinity-4f8f9747-d8ce-42d4-8033-ccfad4356cd7-1400925323834@3capp-gmx-bs32>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140524/0964f160/attachment.html>

From herbertg at gmx.at  Sat May 24 12:15:37 2014
From: herbertg at gmx.at (Herbert Griebel)
Date: Sat, 24 May 2014 12:15:37 +0200
Subject: [Python-Dev] [RELEASED] Python 3.4.1
In-Reply-To: <53799E17.7040805@hastings.org>
References: <53799E17.7040805@hastings.org>
Message-ID: <53807149.5090801@gmx.at>

Found the issue. To reproduce the problem install Python 3.4.0, then rename the install 
folder (e.g. C:\Python34 to C:\Python34x) and install Python 3.4.1.

From martin at v.loewis.de  Sat May 24 19:23:47 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 24 May 2014 19:23:47 +0200
Subject: [Python-Dev] [RELEASED] Python 3.4.1
In-Reply-To: <53807149.5090801@gmx.at>
References: <53799E17.7040805@hastings.org> <53807149.5090801@gmx.at>
Message-ID: <5380D5A3.6060806@v.loewis.de>

Am 24.05.14 12:15, schrieb Herbert Griebel:
> Found the issue. To reproduce the problem install Python 3.4.0, then
> rename the install folder (e.g. C:\Python34 to C:\Python34x) and install
> Python 3.4.1.

Please understand that installation of 3.4.1 attempts uninstallation of
3.4 first. Without testing, my guess is that it is the pip
uninstallation which fails, due to python.exe not being there anymore.

If you want to diagnose this further, run the installer with

msiexec /i python-3.4.1.msi /l*v python.log

Either study the log file yourself, or post it to bugs.python.org.

Regards,
Martin


From Nikolaus at rath.org  Sat May 24 21:35:42 2014
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Sat, 24 May 2014 12:35:42 -0700
Subject: [Python-Dev] Commit-ready patches in need of review
Message-ID: <8738fzgeb5.fsf@vostro.rath.org>

Hello,

While my last appeal resulted in quite some commits (thanks!), I still
have some more commit-ready patches waiting for review.  It'd be great
if some people could find time to take a look:

* http://bugs.python.org/issue1738 (filecmp.dircmp does exact match
  only)
  
  Reviewed patch, rebased on current hg tip
  
* http://bugs.python.org/issue20177 (Derby #8: Convert 28 sites to
  Argument Clinic across 2 files)

  I only wrote the patch for one file because I'd like to have feedback
  before tackling the second. However, the patches are independent so
  unless there are other problems this is ready for commit.
  
* http://bugs.python.org/issue15955 (gzip, bz2, lzma: add option to
  limit output size)
  
* http://bugs.python.org/issue20578 (BufferedIOBase.readinto1 is
  missing)

  
Best,
Nikolaus


-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             ?Time flies like an arrow, fruit flies like a Banana.?

From storchaka at gmail.com  Tue May 27 10:58:51 2014
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 27 May 2014 11:58:51 +0300
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <3gcW0S2PBfz7LjV@mail.python.org>
References: <3gcW0S2PBfz7LjV@mail.python.org>
Message-ID: <lm1k3a$fs$1@ger.gmane.org>

26.05.14 10:59, raymond.hettinger ???????(??):
> +        result = [(elem, i) for i, elem in zip(range(n), it)]

Perhaps it is worth to add simple comment explaining why this is not 
equivalent to just list(zip(it, range(n))). Otherwise it can be 
unintentionally "optimized" in future.



From rosuav at gmail.com  Tue May 27 11:05:56 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 27 May 2014 19:05:56 +1000
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <lm1k3a$fs$1@ger.gmane.org>
References: <3gcW0S2PBfz7LjV@mail.python.org>
	<lm1k3a$fs$1@ger.gmane.org>
Message-ID: <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>

On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 26.05.14 10:59, raymond.hettinger ???????(??):
>>
>> +        result = [(elem, i) for i, elem in zip(range(n), it)]
>
>
> Perhaps it is worth to add simple comment explaining why this is not
> equivalent to just list(zip(it, range(n))). Otherwise it can be
> unintentionally "optimized" in future.
>

Where is the difference? I'm very much puzzled now. My first thought
was based on differing-length iterables in zip, but the docs say it
stops at the shortest of its args.

ChrisA

From murman at gmail.com  Tue May 27 13:45:01 2014
From: murman at gmail.com (Michael Urman)
Date: Tue, 27 May 2014 06:45:01 -0500
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
References: <3gcW0S2PBfz7LjV@mail.python.org> <lm1k3a$fs$1@ger.gmane.org>
 <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
Message-ID: <CAOpBPYX81Q3GhoioTSiEj=8254wUKRnRe7DBN_51Ubi_KxHs6g@mail.gmail.com>

On Tue, May 27, 2014 at 4:05 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 26.05.14 10:59, raymond.hettinger ???????(??):
>>>
>>> +        result = [(elem, i) for i, elem in zip(range(n), it)]
>>
>>
>> Perhaps it is worth to add simple comment explaining why this is not
>> equivalent to just list(zip(it, range(n))). Otherwise it can be
>> unintentionally "optimized" in future.
>>
>
> Where is the difference? I'm very much puzzled now. My first thought
> was based on differing-length iterables in zip, but the docs say it
> stops at the shortest of its args.

Due to how zip stops, it leaves the longer iterable in different places:

>>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it)
[(0, 'a'), (1, 'b'), (2, 'c')]
'd'
>>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it)
[('a', 0), ('b', 1), ('c', 2)]
'e'

This seems like a potentially nasty gotcha, but I'm unclear what real
use cases would be impacted.

Michael

From taleinat at gmail.com  Tue May 27 14:44:33 2014
From: taleinat at gmail.com (Tal Einat)
Date: Tue, 27 May 2014 15:44:33 +0300
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <CAOpBPYX81Q3GhoioTSiEj=8254wUKRnRe7DBN_51Ubi_KxHs6g@mail.gmail.com>
References: <3gcW0S2PBfz7LjV@mail.python.org> <lm1k3a$fs$1@ger.gmane.org>
 <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
 <CAOpBPYX81Q3GhoioTSiEj=8254wUKRnRe7DBN_51Ubi_KxHs6g@mail.gmail.com>
Message-ID: <CALWZvp7UC1kPgMSxgw6p3Ns73P_UzL5eLopLHViPSq48p1GHuw@mail.gmail.com>

On Tue, May 27, 2014 at 2:45 PM, Michael Urman <murman at gmail.com> wrote:
> On Tue, May 27, 2014 at 4:05 AM, Chris Angelico <rosuav at gmail.com> wrote:
>> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>> 26.05.14 10:59, raymond.hettinger ???????(??):
>>>>
>>>> +        result = [(elem, i) for i, elem in zip(range(n), it)]
>>>
>>>
>>> Perhaps it is worth to add simple comment explaining why this is not
>>> equivalent to just list(zip(it, range(n))). Otherwise it can be
>>> unintentionally "optimized" in future.
>>>
>>
>> Where is the difference? I'm very much puzzled now. My first thought
>> was based on differing-length iterables in zip, but the docs say it
>> stops at the shortest of its args.
>
> Due to how zip stops, it leaves the longer iterable in different places:
>
>>>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it)
> [(0, 'a'), (1, 'b'), (2, 'c')]
> 'd'
>>>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it)
> [('a', 0), ('b', 1), ('c', 2)]
> 'e'
>
> This seems like a potentially nasty gotcha, but I'm unclear what real
> use cases would be impacted.

If that's the issue, a comment explaining it should definitely be added.

We could also use islice(enumerate(it), n)) to avoid the confusion.

- Tal

From j.wielicki at sotecware.net  Tue May 27 12:01:35 2014
From: j.wielicki at sotecware.net (Jonas Wielicki)
Date: Tue, 27 May 2014 12:01:35 +0200
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
References: <3gcW0S2PBfz7LjV@mail.python.org>	<lm1k3a$fs$1@ger.gmane.org>
 <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
Message-ID: <5384627F.7090508@sotecware.net>

On 27.05.2014 11:05, Chris Angelico wrote:
> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> 26.05.14 10:59, raymond.hettinger ???????(??):
>>>
>>> +        result = [(elem, i) for i, elem in zip(range(n), it)]
>>
>>
>> Perhaps it is worth to add simple comment explaining why this is not
>> equivalent to just list(zip(it, range(n))). Otherwise it can be
>> unintentionally "optimized" in future.
>>
> 
> Where is the difference? I'm very much puzzled now. My first thought
> was based on differing-length iterables in zip, but the docs say it
> stops at the shortest of its args.

Nice puzzle. (elem, i) is not (i, elem), though. Thats really hard to
see when not looking at it closely.

> 
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/j.wielicki%40sotecware.net
> 


From raymond.hettinger at gmail.com  Tue May 27 23:54:35 2014
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Tue, 27 May 2014 14:54:35 -0700
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <lm1k3a$fs$1@ger.gmane.org>
References: <3gcW0S2PBfz7LjV@mail.python.org> <lm1k3a$fs$1@ger.gmane.org>
Message-ID: <25554AEF-3E8C-4D91-9945-BB0705E79CCC@gmail.com>


On May 27, 2014, at 1:58 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:

> Perhaps it is worth to add simple comment explaining why this is not equivalent to just list(zip(it, range(n))). Otherwise it can be unintentionally "optimized" in future.

FWIW, that is covered by the test cases.
If you substitute list(zip(it, range(n))), the tests fail.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140527/391d90cf/attachment.html>

From rob.cliffe at btinternet.com  Wed May 28 01:23:55 2014
From: rob.cliffe at btinternet.com (Rob Cliffe)
Date: Wed, 28 May 2014 00:23:55 +0100
Subject: [Python-Dev] cpython: Minor clean-ups for heapq.
In-Reply-To: <CALWZvp7UC1kPgMSxgw6p3Ns73P_UzL5eLopLHViPSq48p1GHuw@mail.gmail.com>
References: <3gcW0S2PBfz7LjV@mail.python.org> <lm1k3a$fs$1@ger.gmane.org>
 <CAPTjJmrj4JUpBoxtCkUOyxR5u24QcH+7thmEskybLneK2MV7kg@mail.gmail.com>
 <CAOpBPYX81Q3GhoioTSiEj=8254wUKRnRe7DBN_51Ubi_KxHs6g@mail.gmail.com>
 <CALWZvp7UC1kPgMSxgw6p3Ns73P_UzL5eLopLHViPSq48p1GHuw@mail.gmail.com>
Message-ID: <53851E8B.2060600@btinternet.com>


On 27/05/2014 13:44, Tal Einat wrote:
> On Tue, May 27, 2014 at 2:45 PM, Michael Urman <murman at gmail.com> wrote:
>> On Tue, May 27, 2014 at 4:05 AM, Chris Angelico <rosuav at gmail.com> wrote:
>>> On Tue, May 27, 2014 at 6:58 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>>>> 26.05.14 10:59, raymond.hettinger ???????(??):
>>>>> +        result = [(elem, i) for i, elem in zip(range(n), it)]
>>>>
>>>> Perhaps it is worth to add simple comment explaining why this is not
>>>> equivalent to just list(zip(it, range(n))). Otherwise it can be
>>>> unintentionally "optimized" in future.
>>>>
>>> Where is the difference? I'm very much puzzled now. My first thought
>>> was based on differing-length iterables in zip, but the docs say it
>>> stops at the shortest of its args.
>> Due to how zip stops, it leaves the longer iterable in different places:
>>
>>>>> it = iter(string.ascii_letters); list(zip(range(3), it)); next(it)
>> [(0, 'a'), (1, 'b'), (2, 'c')]
>> 'd'
>>>>> it = iter(string.ascii_letters); list(zip(it, range(3))); next(it)
>> [('a', 0), ('b', 1), ('c', 2)]
>> 'e'
>>
>> This seems like a potentially nasty gotcha, but I'm unclear what real
>> use cases would be impacted.
> If that's the issue, a comment explaining it should definitely be added.
+1.  FWIW I couldn't see the difference between the 2 codings.  I was 
relieved to see that the zip() doc says
"The left-to-right evaluation order of the iterables is guaranteed."  
Even so, I don't think the reliance on this is obvious.
Rob Cliffe
>
> We could also use islice(enumerate(it), n)) to avoid the confusion.
>
> - Tal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com
>
>
> -----
> No virus found in this message.
> Checked by AVG - www.avg.com
> Version: 2014.0.4592 / Virus Database: 3950/7568 - Release Date: 05/26/14


From michael.haubenwallner at ssi-schaefer.com  Wed May 28 16:51:19 2014
From: michael.haubenwallner at ssi-schaefer.com (Michael Haubenwallner)
Date: Wed, 28 May 2014 16:51:19 +0200
Subject: [Python-Dev] use cases for "python-config" versus "pkg-config
	python"
Message-ID: <5385F7E7.9090408@ssi-schaefer.com>

Hello!

Stumbling over problems on AIX (Modules/python.exp not found) building libxml2 as python module
let me wonder about the intended use-cases for 'python-config' and 'pkg-config python'.

FWIW, I can see these distinct use cases here, and I'm kindly asking if I got them right:

* Build an application containing a python interpreter (like python$EXE itself):
  + link against libpython.so
  + re-export symbols from libpython.so for python-modules (platform-specific)
  + This is similar to build against any other library, thus
  = 'python.pc' is installed (for 'pkg-config python').

* Build a python-module (like build/lib.<platform>-<pyver>/*.so):
  + no need to link against libpython.so, instead
  + expect symbols from libpython.so to be available at runtime, platform-specific either as
  + undefined symbols at build-time (Linux, others), or
  + a list of symbols to import from "the main executable" (AIX)
  + This is specific to python-modules, thus
  = 'python-config' is installed.

Thank you!
/haubi/

From brian at python.org  Wed May 28 18:49:52 2014
From: brian at python.org (Brian Curtin)
Date: Wed, 28 May 2014 11:49:52 -0500
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
Message-ID: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>

On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I don't think we have recent download numbers since the Website
> overhaul (do we?), but Python 3 isn't an "experimental concept
> language" anymore (it hasn't been since 3.3 or 3.2, I'd say).

Using the old logs, which are still good through 2013, I've found the following:

The first year of a release series (month of final release month + 12mos):
2.6.x - 10.3 Million
2.7.x - 10.26M
3.2.x - 5.84M
3.3.x - 13.1M

2013 downloads (out of 34.79M across all possible versions):
2.6.x - 1.9M
2.7.x - 14.3M
3.2.x - 1.03M
3.3.x - 13.85M

3.3 had a big first year of availability (Oct '12-'13), and throughout
2013 it represented 48% of those versions listed above.

From brian at python.org  Wed May 28 19:57:49 2014
From: brian at python.org (Brian Curtin)
Date: Wed, 28 May 2014 12:57:49 -0500
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
Message-ID: <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>

On May 28, 2014 12:49 PM, "Brian Curtin" <brian at python.org> wrote:
>
> On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou <solipsis at pitrou.net>
wrote:
> > I don't think we have recent download numbers since the Website
> > overhaul (do we?), but Python 3 isn't an "experimental concept
> > language" anymore (it hasn't been since 3.3 or 3.2, I'd say).
>
> Using the old logs, which are still good through 2013, I've found the
following:
>
> The first year of a release series (month of final release month + 12mos):
> 2.6.x - 10.3 Million
> 2.7.x - 10.26M
> 3.2.x - 5.84M
> 3.3.x - 13.1M
>
> 2013 downloads (out of 34.79M across all possible versions):
> 2.6.x - 1.9M
> 2.7.x - 14.3M
> 3.2.x - 1.03M
> 3.3.x - 13.85M
>
> 3.3 had a big first year of availability (Oct '12-'13), and throughout
> 2013 it represented 48% of those versions listed above.

Sorry for not being explicit: these are download counts for Windows
installers.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140528/61456200/attachment.html>

From guido at python.org  Wed May 28 20:10:00 2014
From: guido at python.org (Guido van Rossum)
Date: Wed, 28 May 2014 11:10:00 -0700
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
Message-ID: <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>

Is the Windows/Mac ratio still 70/30, with Linux in the single digits?


On Wed, May 28, 2014 at 10:57 AM, Brian Curtin <brian at python.org> wrote:

> On May 28, 2014 12:49 PM, "Brian Curtin" <brian at python.org> wrote:
> >
> > On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> > > I don't think we have recent download numbers since the Website
> > > overhaul (do we?), but Python 3 isn't an "experimental concept
> > > language" anymore (it hasn't been since 3.3 or 3.2, I'd say).
> >
> > Using the old logs, which are still good through 2013, I've found the
> following:
> >
> > The first year of a release series (month of final release month +
> 12mos):
> > 2.6.x - 10.3 Million
> > 2.7.x - 10.26M
> > 3.2.x - 5.84M
> > 3.3.x - 13.1M
> >
> > 2013 downloads (out of 34.79M across all possible versions):
> > 2.6.x - 1.9M
> > 2.7.x - 14.3M
> > 3.2.x - 1.03M
> > 3.3.x - 13.85M
> >
> > 3.3 had a big first year of availability (Oct '12-'13), and throughout
> > 2013 it represented 48% of those versions listed above.
>
> Sorry for not being explicit: these are download counts for Windows
> installers.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140528/79ee5ada/attachment.html>

From eliben at gmail.com  Wed May 28 22:05:48 2014
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 28 May 2014 13:05:48 -0700
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
Message-ID: <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>

On Wed, May 28, 2014 at 11:10 AM, Guido van Rossum <guido at python.org> wrote:

> Is the Windows/Mac ratio still 70/30, with Linux in the single digits?
>
>
Most Linux installs go through package managers which don't count here, no?

Eli


>
> On Wed, May 28, 2014 at 10:57 AM, Brian Curtin <brian at python.org> wrote:
>
>> On May 28, 2014 12:49 PM, "Brian Curtin" <brian at python.org> wrote:
>> >
>> > On Fri, Apr 18, 2014 at 1:31 PM, Antoine Pitrou <solipsis at pitrou.net>
>> wrote:
>> > > I don't think we have recent download numbers since the Website
>> > > overhaul (do we?), but Python 3 isn't an "experimental concept
>> > > language" anymore (it hasn't been since 3.3 or 3.2, I'd say).
>> >
>> > Using the old logs, which are still good through 2013, I've found the
>> following:
>> >
>> > The first year of a release series (month of final release month +
>> 12mos):
>> > 2.6.x - 10.3 Million
>> > 2.7.x - 10.26M
>> > 3.2.x - 5.84M
>> > 3.3.x - 13.1M
>> >
>> > 2013 downloads (out of 34.79M across all possible versions):
>> > 2.6.x - 1.9M
>> > 2.7.x - 14.3M
>> > 3.2.x - 1.03M
>> > 3.3.x - 13.85M
>> >
>> > 3.3 had a big first year of availability (Oct '12-'13), and throughout
>> > 2013 it represented 48% of those versions listed above.
>>
>> Sorry for not being explicit: these are download counts for Windows
>> installers.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/eliben%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140528/1c3f3867/attachment.html>

From brian at python.org  Wed May 28 23:02:58 2014
From: brian at python.org (Brian Curtin)
Date: Wed, 28 May 2014 16:02:58 -0500
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
 <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
Message-ID: <CAD+XWwozRbfEJENP2wjED1kTJXWkfRpNziBaM=7+98GhaRpCpw@mail.gmail.com>

On May 28, 2014 4:06 PM, "Eli Bendersky" <eliben at gmail.com> wrote:
>
>
>
>
> On Wed, May 28, 2014 at 11:10 AM, Guido van Rossum <guido at python.org>
wrote:
>>
>> Is the Windows/Mac ratio still 70/30, with Linux in the single digits?
>>
>
> Most Linux installs go through package managers which don't count here,
no?

I'll have to run something for the other non-Windows files, but the single
digit Linux downloads he meant are the tarballs. We'll (probably) never
know the true counts in the Linux world because of how pervasive Python is
within basically every distro, but that's also likely the case on Mac.

With Windows, since you must download Python it to use it, the numbers we
see are probably the most useful on their own. I'm giving a talk at PyCon
Russia that covers some of these numbers, so I'll probably try to dig up
more and turn it into a blog post.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140528/cc8e8340/attachment.html>

From victor.stinner at gmail.com  Wed May 28 23:27:59 2014
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 28 May 2014 23:27:59 +0200
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
 <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
Message-ID: <CAMpsgwZDJNnZd0Z99uLnxUpi_hHt8PavQOF-wBoLaH4bVTTjgQ@mail.gmail.com>

2014-05-28 22:05 GMT+02:00 Eli Bendersky <eliben at gmail.com>:
> Most Linux installs go through package managers which don't count here, no?

For Debian, there is the "popcorn" project which provides some statistics:

http://qa.debian.org/popcon.php?package=python2.6
http://qa.debian.org/popcon.php?package=python2.7
http://qa.debian.org/popcon.php?package=python3.2
http://qa.debian.org/popcon.php?package=python3.3
http://qa.debian.org/popcon.php?package=python3.4

It looks like python2.6 is installed more often than python2.7 ! (if
you look at the "Inst" column, not in the "Recent" column.)

Python 3.4 recently became the default python3 package on Debian
Unstable ("sid"). Debian Stable (Wheezy) still uses Python 3.2. I
guess that Debian Testing uses Python 3.3.

Victor

From rosuav at gmail.com  Wed May 28 23:37:57 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 29 May 2014 07:37:57 +1000
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAMpsgwZDJNnZd0Z99uLnxUpi_hHt8PavQOF-wBoLaH4bVTTjgQ@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
 <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
 <CAMpsgwZDJNnZd0Z99uLnxUpi_hHt8PavQOF-wBoLaH4bVTTjgQ@mail.gmail.com>
Message-ID: <CAPTjJmrnJzByohMptqiE2Xq=DB9kwU2M-Ueu1j7vB7MMhunusw@mail.gmail.com>

On Thu, May 29, 2014 at 7:27 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2014-05-28 22:05 GMT+02:00 Eli Bendersky <eliben at gmail.com>:
>> Most Linux installs go through package managers which don't count here, no?
>
> For Debian, there is the "popcorn" project which provides some statistics:
>
> http://qa.debian.org/popcon.php?package=python2.6
> http://qa.debian.org/popcon.php?package=python2.7
> http://qa.debian.org/popcon.php?package=python3.2
> http://qa.debian.org/popcon.php?package=python3.3
> http://qa.debian.org/popcon.php?package=python3.4
>
> It looks like python2.6 is installed more often than python2.7 ! (if
> you look at the "Inst" column, not in the "Recent" column.)
>
> Python 3.4 recently became the default python3 package on Debian
> Unstable ("sid"). Debian Stable (Wheezy) still uses Python 3.2. I
> guess that Debian Testing uses Python 3.3.

2.6 is the default in oldstable (Squeeze), so every Debian system that
hasn't upgraded from there is likely to be counted as a 2.6 and not a
2.7. (And Python 2 is a dependency of all sorts of things, so it's
likely to get installed.) Wheezy ships 2.6.8, and I seem to have both
that and 2.7.3 installed, so it's possible that even though 2.7 is the
default, a lot of 2.6 installations are happening.

Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3'
package pulling in 3.3. That may change before Jessie becomes stable,
I don't know.

ChrisA

From barry at python.org  Thu May 29 01:05:50 2014
From: barry at python.org (Barry Warsaw)
Date: Wed, 28 May 2014 19:05:50 -0400
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <CAPTjJmrnJzByohMptqiE2Xq=DB9kwU2M-Ueu1j7vB7MMhunusw@mail.gmail.com>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
 <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
 <CAMpsgwZDJNnZd0Z99uLnxUpi_hHt8PavQOF-wBoLaH4bVTTjgQ@mail.gmail.com>
 <CAPTjJmrnJzByohMptqiE2Xq=DB9kwU2M-Ueu1j7vB7MMhunusw@mail.gmail.com>
Message-ID: <20140528190550.096f16ee@limelight.wooz.org>

On May 29, 2014, at 07:37 AM, Chris Angelico wrote:

>Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3'
>package pulling in 3.3. That may change before Jessie becomes stable,
>I don't know.

It already has:

https://lists.debian.org/debian-python/2014/05/msg00162.html

The question remains whether Jessie will drop Python 3.3, but I expect it to.

Cheers,
-Barry

From rosuav at gmail.com  Thu May 29 01:37:44 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 29 May 2014 09:37:44 +1000
Subject: [Python-Dev] Download Counts (was: Language Summit notes)
In-Reply-To: <20140528190550.096f16ee@limelight.wooz.org>
References: <CAD+XWwr3tyhmC04oWLthxrsFEV04XCXkR3ho0mnUyswKFPgk-A@mail.gmail.com>
 <CAD+XWwoeav869XHtsAqB_sMqCm0jigfwQxZd7eAOTejwR4=GyQ@mail.gmail.com>
 <CAP7+vJ+5GbpjdGOn8szcz0NAVM0uXzxj=vcwZn4QO+LECvGTeg@mail.gmail.com>
 <CAF-Rda86FRixz9saHvfMQ6LtDH4q3vqkJDXk+yckJE+CDQPzEA@mail.gmail.com>
 <CAMpsgwZDJNnZd0Z99uLnxUpi_hHt8PavQOF-wBoLaH4bVTTjgQ@mail.gmail.com>
 <CAPTjJmrnJzByohMptqiE2Xq=DB9kwU2M-Ueu1j7vB7MMhunusw@mail.gmail.com>
 <20140528190550.096f16ee@limelight.wooz.org>
Message-ID: <CAPTjJmrG6epg82nJdgL+dTp_BWLm_w-_Ykvtbr18v9FK2KNrfw@mail.gmail.com>

On Thu, May 29, 2014 at 9:05 AM, Barry Warsaw <barry at python.org> wrote:
> On May 29, 2014, at 07:37 AM, Chris Angelico wrote:
>
>>Debian Testing (Jessie) ships both 3.3 and 3.4, with the 'python3'
>>package pulling in 3.3. That may change before Jessie becomes stable,
>>I don't know.
>
> It already has:
>
> https://lists.debian.org/debian-python/2014/05/msg00162.html
>
> The question remains whether Jessie will drop Python 3.3, but I expect it to.

Well, there you go! :) I was almost going to say that my information
was up to 24 hours out of date, because I hadn't run daily updates
yet, and it turns out it was exactly that wrong :)

ChrisA

From glyph at twistedmatrix.com  Thu May 29 00:26:38 2014
From: glyph at twistedmatrix.com (Glyph Lefkowitz)
Date: Wed, 28 May 2014 15:26:38 -0700
Subject: [Python-Dev] Language Summit Follow-Up
Message-ID: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>

At the language summit, Alex and I volunteered to put together some recommendations on what changes could be made to Python (the language) in order to facilitate a smoother transition from Python 2 to Python 3.  One of the things that motivated this was the (surprising, to us) consideration that features like ensurepip might be added to the future versions of the 2.7 installers from python.org.

The specific motivations for writing this are:

Library maintainers have a rapidly expanding matrix that requires an increasing number of branches to satisfy.
People with large corporate codebases absolutely cannot port all at once.

If you don't have perfect test coverage then you can't make any progress.  So these changes are intended to make porting from python 2 to python 3 more guided and incremental.  We believe that these attributes are necessary.

We would like to stress that we don't believe anything on this list is as important as the continuing efforts that everyone in the broader ecosystem is making.  If you just want to ease the transition by working on anything at all, the best use of your time right now is porting https://warehouse.python.org/project/MySQL-python/ to Python 3. :)

Nevertheless there are some things that the language and CPython could do.

Unfortunately we had to reject any proposal that involved new __future__ imports, since unknown __future__ imports are un-catchable SyntaxErrors.

Here are some ideas for Python 2.7+.

Add ensurepip to the installers.  Having pip reliably available increases the availability of libraries that help with porting, and will generally strengthen the broader ecosystem in the (increasingly long) transition period.
Add some warnings about python 3 compatibility.
It should at least be possible to get a warning for every single implicit string coercion.
Old-style classes.
Old-style division.
Print statements.
Old-style exception syntax.
buffer().
bytes(memoryview(b'abc'))
Importing old locations from the stdlib (see point 4.)
Long integer syntax.
Use of variables beyond the lifetime of an 'except Exception as e' block or a list comprehension.
Backport 'yield from' to allow people to use Tulip and Tulip-compatible code, and to facilitate the development of Tulip-friendly libraries and a Tulip ecosystem.  A robust Tulip ecosystem requires the participation of people who are not yet using Python 3.
Add aliases for the renamed modules in the stdlib.  This will allow people to "just write python 3" in a lot more circumstances.
(re-)Enable warnings by default, including enabling -3 warnings.  Right now all warnings are silent by default, which greatly reduces discoverability of future compatibility issues.  I hope it's not controversial to say that most new Python code is still being written against Python 2.7 today; if people are writing that code in such a way that it's not 3-friendly, it should be a more immediately noticeable issue.
Get rid of 2to3. Particularly, of any discussion of using 2to3 in the documentation.  More than one very experienced, well-known Python developer in this discussion has told me that they thought 2to3 was the blessed way to port their code, and it's no surprise that they think so, given that the first technique <https://docs.python.org/3/howto/pyporting.html> mentions is still 2to3.  We should replace 2to3 with something like <https://github.com/mitsuhiko/python-modernize>. 2to3 breaks your code on python 2, and doesn't necessarily get it running on python 3.  A more conservative approach that reduced the amount of work to get your code 2/3 compatible but was careful to leave everything working would be a lot more effective.
Add a new 'bytes' type that actually behaves like the Python 3 bytes type (bytes(5)).

We have rejected any changes for Python 3.5, simply because of the extremely long time to get those features into users hands.  Any changes for Python 3 that we're proposing would need to get into a 3.4.x release, so that, for example, they can make their way into Ubuntu 14.04 LTS.

Here are some ideas for Python 3.4.x:

Usage of Python2 style syntax (for example, a print statement) or stdlib module names (for example, 'import urllib2') should result in a specific, informative warning, not a generic SyntaxError/ImportError.  This will really help new users.
Add 'unicode' back as an alias for 'str'.  Just today I was writing some documentation where I had to resort to some awkward encoding tricks just to get a bytes object out without explaining the whole 2/3 dichotomy in some unrelated prose.

We'd like to thank all the individuals who gave input and feedback in creating this list.

-glyph & Alex Gaynor

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140528/9446de55/attachment-0001.html>

From ericsnowcurrently at gmail.com  Thu May 29 05:51:41 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Wed, 28 May 2014 21:51:41 -0600
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <CALFfu7A46ec7QrZ3USMRO6-7wNuP5Xqh+xdFFjZAkerrMw_pAg@mail.gmail.com>

Thanks for for putting this together.

On Wed, May 28, 2014 at 4:26 PM, Glyph Lefkowitz
<glyph at twistedmatrix.com> wrote:
> At the language summit, Alex and I volunteered to put together some
> recommendations on what changes could be made to Python (the language) in
> order to facilitate a smoother transition from Python 2 to Python 3.  One of
> the things that motivated this was the (surprising, to us) consideration
> that features like ensurepip might be added to the future versions of the
> 2.7 installers from python.org.
>
> The specific motivations for writing this are:
>
> 1. Library maintainers have a rapidly expanding matrix that requires an
> increasing number of branches to satisfy.
> 2. People with large corporate codebases absolutely cannot port all at once.
>
>
> If you don't have perfect test coverage then you can't make any progress.
> So these changes are intended to make porting from python 2 to python 3 more
> guided and incremental.  We believe that these attributes are necessary.

One of the parallel discussions that came out of the language summit
centers on basically the same idea:

https://mail.python.org/pipermail/python-dev/2014-April/133986.html

The key points are that there would be a separate tool that wraps the
2.7 interpreter and enforces the 2.7/3.x subset language but only for
source files that have are tagged (a la the coding cookie).  So there
would be no need to make changes to Python itself.  Some have also
suggested this could also be accomplished via a pylint plugin.  I'm
working on an implementation called (for now) "pymigrate" that should
eventually be available via the cheeseshop.  A lot of what you've
written here is really useful.

> Here are some ideas for Python 2.7+.
>
> 1. Add ensurepip to the installers.  Having pip reliably available increases
> the availability of libraries that help with porting, and will generally
> strengthen the broader ecosystem in the (increasingly long) transition
> period.

My understanding is that this is already the plan.

> 2. Add some warnings about python 3 compatibility.
>
>     1. It should at least be possible to get a warning for every single implicit
> string coercion.
>     2. Old-style classes.
>     3. Old-style division.
>     4. Print statements.
>     5. Old-style exception syntax.
>     6. buffer().
>     7. bytes(memoryview(b'abc'))
>     8. Importing old locations from the stdlib (see point 4.)
>     9. Long integer syntax.
>   10. Use of variables beyond the lifetime of an 'except Exception as e' block or
> a list comprehension.

+1 to improving the coverage of the -3 option in 2.7.

>
> 3. Backport 'yield from' to allow people to use Tulip and Tulip-compatible
> code, and to facilitate the development of Tulip-friendly libraries and a
> Tulip ecosystem.  A robust Tulip ecosystem requires the participation of
> people who are not yet using Python 3.

What about Trellis?  Regardless, adding yield from to 2.7 is going to
be a tough sell.

> 4. Add aliases for the renamed modules in the stdlib.  This will allow people
> to "just write python 3" in a lot more circumstances.

I like this, but it seems to me to be a better fit for something like
six or even pymigrate.

> 5. (re-)Enable warnings by default, including enabling -3 warnings.  Right now
> all warnings are silent by default, which greatly reduces discoverability of
> future compatibility issues.

This is also something that fits better with a separate tool.

> We have rejected any changes for Python 3.5, simply because of the extremely
> long time to get those features into users hands.  Any changes for Python 3
> that we're proposing would need to get into a 3.4.x release

Good luck with that one! <wink>

-eric

From benjamin at python.org  Thu May 29 06:50:15 2014
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 28 May 2014 21:50:15 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <1401339015.20566.122748037.2A7D52C6@webmail.messagingengine.com>



On Wed, May 28, 2014, at 15:26, Glyph Lefkowitz wrote:
> Add some warnings about python 3 compatibility.
> It should at least be possible to get a warning for every single implicit
> string coercion.
> Old-style classes.
> Old-style division.

Fun fact: This can be achieved with the -Qwarn command line option.

From songofacandy at gmail.com  Thu May 29 07:22:31 2014
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 29 May 2014 14:22:31 +0900
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <CAEfz+TzdtrFhx0NWo+c7AZEKeHtdaoJYa0gy7zard6Y4k8mEBQ@mail.gmail.com>

> We would like to stress that we don't believe anything on this list is as
> important as the continuing efforts that everyone in the broader ecosystem
> is making.  If you just want to ease the transition by working on anything
> at all, the best use of your time right now is porting
> https://warehouse.python.org/project/MySQL-python/ to Python 3. :)
>

I've did it.

https://github.com/PyMySQL/mysqlclient-python
https://pypi.python.org/pypi/mysqlclient


-- 
INADA Naoki  <songofacandy at gmail.com>

From tjreedy at udel.edu  Thu May 29 08:53:19 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 29 May 2014 02:53:19 -0400
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CAEfz+TzdtrFhx0NWo+c7AZEKeHtdaoJYa0gy7zard6Y4k8mEBQ@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CAEfz+TzdtrFhx0NWo+c7AZEKeHtdaoJYa0gy7zard6Y4k8mEBQ@mail.gmail.com>
Message-ID: <lm6lh4$url$1@ger.gmane.org>

On 5/29/2014 1:22 AM, INADA Naoki wrote:
>> We would like to stress that we don't believe anything on this list is as
>> important as the continuing efforts that everyone in the broader ecosystem
>> is making.  If you just want to ease the transition by working on anything
>> at all, the best use of your time right now is porting
>> https://warehouse.python.org/project/MySQL-python/ to Python 3. :)
>>
>
> I've did it.
>
> https://github.com/PyMySQL/mysqlclient-python
> https://pypi.python.org/pypi/mysqlclient

I don't think that most people interpret
Programming Language :: Python
as indicating support for Python 3. So I suggest adding
Programming Language :: Python :: 3
or even the more specific 3.3 and 3.4

-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Thu May 29 13:43:07 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 29 May 2014 21:43:07 +1000
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>

On 29 May 2014 08:26, Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:

Thanks for the write-up!

> Here are some ideas for Python 2.7+.
>
> Add ensurepip to the installers.  Having pip reliably available increases
> the availability of libraries that help with porting, and will generally
> strengthen the broader ecosystem in the (increasingly long) transition
> period.

Agreed on this one - I only dropped it from 453 because I didn't think
I could continue to push for it and still make the beta deadline for
3.4, not because I changed my mind. This mainly needs a champion to
write the PEP to describe the exact details of what would be
backported (the tricky parts of PEP 453 relating to pyvenv don't
apply) and to actually do the work. Bundling the Python Launcher with
the Windows installer would also be desirable.

> Add some warnings about python 3 compatibility.
>
> It should at least be possible to get a warning for every single implicit
> string coercion.
> Old-style classes.
> Old-style division.
> Print statements.
> Old-style exception syntax.
> buffer().
> bytes(memoryview(b'abc'))
> Importing old locations from the stdlib (see point 4.)
> Long integer syntax.
> Use of variables beyond the lifetime of an 'except Exception as e' block or
> a list comprehension.

A few more for the "missing -3 warning" list:

    - calling str.encode
    - calling unicode.decode
    - returning str from str.decode
    - returning unicode from unicode.encode

The relevant TypeErrors in 3.x show where the latter two warnings need
to go in 2.7.

More controversially: warn for any calls to Py2 str methods that
aren't present on Py3 bytes.

> Backport 'yield from' to allow people to use Tulip and Tulip-compatible
> code, and to facilitate the development of Tulip-friendly libraries and a
> Tulip ecosystem.  A robust Tulip ecosystem requires the participation of
> people who are not yet using Python 3.

Given that the target audience here is our more conservative user
base, Tulip/Trollius backends for Twisted and gevent seem more
conducive to that than supporting Tulip's native coroutine syntax.
That said, it's a pure additive change without any significant
backwards compatibility implications, so I personally wouldn't object
if Guido said yes.

> Add aliases for the renamed modules in the stdlib.  This will allow people
> to "just write python 3" in a lot more circumstances.

This is what the module aliasing options provided by python-future are
designed to do - it's much closer to "just write Python 3" than six is
(although that comes at the price of being a bit more magical).

The python-future experience also shows there's a potential backwards
compatibility issue here - depending on exactly how you do it, you can
end up conflicting with other libraries trying to do the same thing.

> (re-)Enable warnings by default, including enabling -3 warnings.  Right now
> all warnings are silent by default, which greatly reduces discoverability of
> future compatibility issues.  I hope it's not controversial to say that most
> new Python code is still being written against Python 2.7 today; if people
> are writing that code in such a way that it's not 3-friendly, it should be a
> more immediately noticeable issue.

The reasons these were turned off haven't changed, and passing -3
already enables DeprecationWarnings generally (along with -Qwarn).
However, several of the warnings mentioned in the list above are
currently missing entirely, so they won't show up regardless of the
warning state.

> Get rid of 2to3. Particularly, of any discussion of using 2to3 in the
> documentation.  More than one very experienced, well-known Python developer
> in this discussion has told me that they thought 2to3 was the blessed way to
> port their code, and it's no surprise that they think so, given that the
> first technique <https://docs.python.org/3/howto/pyporting.html> mentions is
> still 2to3.  We should replace 2to3 with something like
> <https://github.com/mitsuhiko/python-modernize>. 2to3 breaks your code on
> python 2, and doesn't necessarily get it running on python 3.  A more
> conservative approach that reduced the amount of work to get your code 2/3
> compatible but was careful to leave everything working would be a lot more
> effective.

+1, although I think python-future provides a better "subset language"
experience than modernize+six does, and provides a 3->common subset
converter in addition to a 2->common subset (see
http://python-future.org/faq.html#relationship-between-python-future-and-other-compatibility-tools
for more on the difference between the two approaches).

The futurize script also has the nice property of supporting two
different stages, with "--stage1" being a pure syntactic update - no
new runtime dependencies needed.

"Big bang" migrations are highly unlikely to be the right choice for
anyone, and while the code produced by python-future will still have
some quirks (e.g. avoiding direct use of any mapping methods as PEP
469 ended up recommending, plus a bunch of import noise at the start
of the file)

> Add a new 'bytes' type that actually behaves like the Python 3 bytes type
> (bytes(5)).

This is available in python-future.

> We have rejected any changes for Python 3.5, simply because of the extremely
> long time to get those features into users hands.  Any changes for Python 3
> that we're proposing would need to get into a 3.4.x release, so that, for
> example, they can make their way into Ubuntu 14.04 LTS.
>
> Here are some ideas for Python 3.4.x:
>
> Usage of Python2 style syntax (for example, a print statement) or stdlib
> module names (for example, 'import urllib2') should result in a specific,
> informative warning, not a generic SyntaxError/ImportError.  This will
> really help new users.
> Add 'unicode' back as an alias for 'str'.  Just today I was writing some
> documentation where I had to resort to some awkward encoding tricks just to
> get a bytes object out without explaining the whole 2/3 dichotomy in some
> unrelated prose.

While I realise it's out of scope for your "near term changes" list,
there are still a few changes I think are worth exploring for the
Python 3.5+ time frame:

- the return of binary interpolation is already approved (but isn't
implemented yet)
- various other improvements to the Py3 binary data model (such as the
ideas in PEP 467)
- PEP 432 (cleaning up the startup sequence), or something along those
lines, has now been identified as a prerequisite for decoupling Python
3 from the locale encoding on Linux (and hence letting us more easily
ignore the entirely-inappropriate-for-the-21st-century C locale)
- I'd like to see someone take a serious tilt at proposing "NAME
<args>" call statement support at the language level. IPython already
supports that style of invocation for arbitrary callables (so print
statements still appear work even in Python 3 if you use IPython
rather than the default REPL or direct script execution), and there's
a proof of concept patch at http://bugs.python.org/issue18788 that
shows CPython's parser is able to handle the idea (see my final
comment for additional restrictions a full proposal would need)

For that last point, my interest is as much educational as it is in
easing the transition from Python 2. The parentheses in "print('Hello
world!')" mean introducing the idea of function calls early to explain
how it works, while being able to omit them makes it easier to gloss
over the distinction between statements and function calls initially
and then cover it later after the basics of flow control have been
nailed down.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From bcannon at gmail.com  Thu May 29 16:49:55 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Thu, 29 May 2014 14:49:55 +0000
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
	(8391f99208f6): sum=181
References: <E1WpvIz-0002JF-Tb@vds2544.sivit.org>
Message-ID: <CAP1=2W4J=mVuZ6YSMvYvjDq-7RW-Xjjq9ZtOsC8F_7BbdHQ9Ew@mail.gmail.com>

I think the memory leak was caused by
http://hg.python.org/cpython/rev/7d20e30bd540 because
http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets
the 'res' variable and then overwrites it unconditionally w/o PY_DECREF
beforehand.

On Thu May 29 2014 at 4:02:17 AM, <solipsis at pitrou.net> wrote:

> results for 8391f99208f6 on branch "default"
> --------------------------------------------
>
> test_collections leaked [0, 2, 4] references, sum=6
> test_collections leaked [0, 1, 2] memory blocks, sum=3
> test_functools leaked [0, 0, 3] memory blocks, sum=3
> test_importlib leaked [5, 5, 5] references, sum=15
> test_pkgutil leaked [1, 1, 1] references, sum=3
> test_site leaked [0, 0, 2] references, sum=2
> test_site leaked [0, 0, 2] memory blocks, sum=2
> test_zipimport leaked [47, 47, 47] references, sum=141
> test_zipimport_support leaked [2, 2, 2] references, sum=6
>
>
> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> https://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/6da0f838/attachment.html>

From bcannon at gmail.com  Thu May 29 17:10:44 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Thu, 29 May 2014 15:10:44 +0000
Subject: [Python-Dev]  Language Summit Follow-Up
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <CAP1=2W5O7wDw_ugDFWm_6NnQBy+hW1qE6EhT5rKtxWUm9KghLQ@mail.gmail.com>

On Wed May 28 2014 at 10:14:39 PM, Glyph Lefkowitz <glyph at twistedmatrix.com>
wrote:

> At the language summit, Alex and I volunteered to put together some
> recommendations on what changes could be made to Python (the language) in
> order to facilitate a smoother transition from Python 2 to Python 3.  One
> of the things that motivated this was the (surprising, to us) consideration
> that features like ensurepip might be added to the future versions of the
> 2.7 installers from python.org.
>
> The specific motivations for writing this are:
>
>
>    1. Library maintainers have a rapidly expanding matrix that requires
>    an increasing number of branches to satisfy.
>    2. People with large corporate codebases absolutely cannot port all at
>    once.
>
>
> If you don't have perfect test coverage then you can't make any progress.
>  So these changes are intended to make porting from python 2 to python 3
> more guided and incremental.  We believe that these attributes are
> necessary.
>
> We would like to stress that we don't believe anything on this list is as
> important as the continuing efforts that everyone in the broader ecosystem
> is making.  If you just want to ease the transition by working on anything
> at all, the best use of your time right now is porting
> https://warehouse.python.org/project/MySQL-python/ to Python 3. :)
>
> Nevertheless there are some things that the language and CPython could do.
>
> Unfortunately we had to reject any proposal that involved new __future__
> imports, since unknown __future__ imports are un-catchable SyntaxErrors.
>
> Here are some ideas for Python 2.7+.
>
[SNIP]

>
>    1. Get rid of 2to3. Particularly, of any discussion of using 2to3 in
>    the documentation.  More than one very experienced, well-known Python
>    developer in this discussion has told me that they thought 2to3 was the
>    blessed way to port their code, and it's no surprise that they think so,
>    given that the first technique <
>    https://docs.python.org/3/howto/pyporting.html> mentions is still
>    2to3.  We should replace 2to3 with something like <
>    https://github.com/mitsuhiko/python-modernize>. 2to3 breaks your code
>    on python 2, and doesn't necessarily get it running on python 3.  A more
>    conservative approach that *reduced* the amount of work to get your
>    code 2/3 compatible but was careful to leave everything working would be a
>    lot more effective.
>
>
Two things. One is the HOWTO mentions 2to3 *only* when you are dropping
Python 3 support and since that makes your life the easiest it's listed as
the first option. You will notice point 2 in the TL;DR is be source
compatible if keeping Python 2 compatibility. The main part of the doc is
also all about source compatibility and using 2to3 while maintaining
compatibility is relegated to the Alternatives section at the end (heck,
2to3 is mentioned only 3 times in that entire document, one of which is in
a code comment).

Two, is how do you see python-future fit into this (I know others have
brought it up but we have two different approaches now to keeping source
compatibility that vary in magic and thus smoothing out edges)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/53fcd4d8/attachment.html>

From tjreedy at udel.edu  Thu May 29 18:30:14 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 29 May 2014 12:30:14 -0400
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <lm7nas$tp2$1@ger.gmane.org>

On 5/28/2014 6:26 PM, Glyph Lefkowitz wrote:

> I hope it's
> not controversial to say that most new Python code is still being
> written against Python 2.7 today;

Given that Python 3 downloads now outnumber Python 2 downloads, I think 
'most' might be an overstatement. But I think it a moot point.

 > if people are writing that code in such a way that it's not
 > 3-friendly, it should be a more immediately noticeable issue.

If the truth were, conservatively, 1/4 of new Python code in 2.7, or 
even less, I would still be in favor of making 3-friendly 2.7 code easier.

This is also important for the separate codebase approach, as in the 
stdlib. Just last week, I got a rejected chunk when backporting because 
a 2.7 idlelib module uses 'file(...' instead of 'open(...'.

-- 
Terry Jan Reedy


From ericsnowcurrently at gmail.com  Thu May 29 18:34:04 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 29 May 2014 10:34:04 -0600
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
	(8391f99208f6): sum=181
In-Reply-To: <CAP1=2W4J=mVuZ6YSMvYvjDq-7RW-Xjjq9ZtOsC8F_7BbdHQ9Ew@mail.gmail.com>
References: <E1WpvIz-0002JF-Tb@vds2544.sivit.org>
 <CAP1=2W4J=mVuZ6YSMvYvjDq-7RW-Xjjq9ZtOsC8F_7BbdHQ9Ew@mail.gmail.com>
Message-ID: <CALFfu7Cq3P4o7y1Bn7QiQ+eJQeHHWeXUd2Lzi4Vfvjdn3MKjbg@mail.gmail.com>

Good catch.  I'll look into it.

-eric

On Thu, May 29, 2014 at 8:49 AM, Brett Cannon <bcannon at gmail.com> wrote:
> I think the memory leak was caused by
> http://hg.python.org/cpython/rev/7d20e30bd540 because
> http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the
> 'res' variable and then overwrites it unconditionally w/o PY_DECREF
> beforehand.
>
>
> On Thu May 29 2014 at 4:02:17 AM, <solipsis at pitrou.net> wrote:
>>
>> results for 8391f99208f6 on branch "default"
>> --------------------------------------------
>>
>> test_collections leaked [0, 2, 4] references, sum=6
>> test_collections leaked [0, 1, 2] memory blocks, sum=3
>> test_functools leaked [0, 0, 3] memory blocks, sum=3
>> test_importlib leaked [5, 5, 5] references, sum=15
>> test_pkgutil leaked [1, 1, 1] references, sum=3
>> test_site leaked [0, 0, 2] references, sum=2
>> test_site leaked [0, 0, 2] memory blocks, sum=2
>> test_zipimport leaked [47, 47, 47] references, sum=141
>> test_zipimport_support leaked [2, 2, 2] references, sum=6
>>
>>
>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> https://mail.python.org/mailman/listinfo/python-checkins

From ericsnowcurrently at gmail.com  Thu May 29 20:46:11 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 29 May 2014 12:46:11 -0600
Subject: [Python-Dev] [Python-checkins] Daily reference leaks
	(8391f99208f6): sum=181
In-Reply-To: <CALFfu7Cq3P4o7y1Bn7QiQ+eJQeHHWeXUd2Lzi4Vfvjdn3MKjbg@mail.gmail.com>
References: <E1WpvIz-0002JF-Tb@vds2544.sivit.org>
 <CAP1=2W4J=mVuZ6YSMvYvjDq-7RW-Xjjq9ZtOsC8F_7BbdHQ9Ew@mail.gmail.com>
 <CALFfu7Cq3P4o7y1Bn7QiQ+eJQeHHWeXUd2Lzi4Vfvjdn3MKjbg@mail.gmail.com>
Message-ID: <CALFfu7D3aojrGXYnybV=h2j=hxVeS1xumTt0L1HnmOczwjpCxQ@mail.gmail.com>

Fixed!  (test_site and test_functools are still leaking sporadically,
but it looks unrelated to the import.c leak).

-eric

On Thu, May 29, 2014 at 10:34 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Good catch.  I'll look into it.
>
> -eric
>
> On Thu, May 29, 2014 at 8:49 AM, Brett Cannon <bcannon at gmail.com> wrote:
>> I think the memory leak was caused by
>> http://hg.python.org/cpython/rev/7d20e30bd540 because
>> http://hg.python.org/cpython/file/0eedac3d0b0a/Python/import.c#l903 sets the
>> 'res' variable and then overwrites it unconditionally w/o PY_DECREF
>> beforehand.
>>
>>
>> On Thu May 29 2014 at 4:02:17 AM, <solipsis at pitrou.net> wrote:
>>>
>>> results for 8391f99208f6 on branch "default"
>>> --------------------------------------------
>>>
>>> test_collections leaked [0, 2, 4] references, sum=6
>>> test_collections leaked [0, 1, 2] memory blocks, sum=3
>>> test_functools leaked [0, 0, 3] memory blocks, sum=3
>>> test_importlib leaked [5, 5, 5] references, sum=15
>>> test_pkgutil leaked [1, 1, 1] references, sum=3
>>> test_site leaked [0, 0, 2] references, sum=2
>>> test_site leaked [0, 0, 2] memory blocks, sum=2
>>> test_zipimport leaked [47, 47, 47] references, sum=141
>>> test_zipimport_support leaked [2, 2, 2] references, sum=6
>>>
>>>
>>> Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R',
>>> '3:3:/home/antoine/cpython/refleaks/reflogDdbYzY', '-x']
>>> _______________________________________________
>>> Python-checkins mailing list
>>> Python-checkins at python.org
>>> https://mail.python.org/mailman/listinfo/python-checkins

From josiah.carlson at gmail.com  Fri May 30 02:34:15 2014
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 29 May 2014 17:34:15 -0700
Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for
	subprocess.popen
In-Reply-To: <CADcB0yisH5pfb2Ms2jBgFApq_1_qNnFyVZCDip3_MqD+H5a2TQ@mail.gmail.com>
References: <20140325231947.5ec9ad63@fsol>
 <CAMpsgwYAQX=ATP+RgUrdWz_WrbzMNeof+Y8QsaC-=PBPkeJdKQ@mail.gmail.com>
 <CADcB0yi8xdpKEzRgLjGYuj6OGgVOQxrjLTE7eWjOHK7MQudZzA@mail.gmail.com>
 <CAMpsgwYNHwCZ6cAmjC-BshfiVzJi9dpZ+EdWOWR8jsH6Y+PK0g@mail.gmail.com>
 <CADcB0yjLN-c0FRfk1nJEA8xn6QsMDM8CO_A7133Gjv0shRRH-A@mail.gmail.com>
 <lh2m4v$3an$1@ger.gmane.org>
 <CADcB0yhaU7aAO5=53xfPk4MiU8xxwEuhrG6ki2bjHdt89sC7Eg@mail.gmail.com>
 <CACac1F_oYD3KO9Wen_EcDb1=MvxQ_m+jncT0ZO754gUwsw5C0g@mail.gmail.com>
 <CADcB0yjJoG7C=XK2p1_bVKsSCFU7-mAmDVcvNG2nM4bWLuNaCw@mail.gmail.com>
 <CAP7+vJ+eteEA1wUFRsj29NsxqSK7W9-CHDJEMYH4AFioxG0vtA@mail.gmail.com>
 <CADcB0yix8abmQ2_0NLatt-VdVOnsbk_Zn6ByLR0vWiUHxgTfOg@mail.gmail.com>
 <CADcB0yisH5pfb2Ms2jBgFApq_1_qNnFyVZCDip3_MqD+H5a2TQ@mail.gmail.com>
Message-ID: <CADcB0ygGCcY+wYdG6rUEum=sobV1q+CJmH+0GqQkGEvUcB7fMQ@mail.gmail.com>

Pinging this thread 2 months later with a progress/status update.

To those that have reviewed, commented, helped, or otherwise pushed this
along, which includes (but is not limited to) Richard Oudkerk, eryksun,
Giampaolo Rodola, thank you.


The short version:
As far as I can tell, the patch is ready:
http://bugs.python.org/issue1191964

What is available:
There are docs, tests, and obviously the functionality. Some code was moved
from asyncio/windows_utils.py (which has a separate issue here:
https://code.google.com/p/tulip/issues/detail?id=170). The API was changed
slightly from what was proposed by Guido:

sent = Popen.write_nonblocking(input, timeout=0)
data = Popen.read_nonblocking(bufsize=4096)
data = Popen.read_stderr_nonblocking(bufsize=4096)

As a bonus feature, Windows communicate() calls no longer spawn worker
threads, and instead use overlapped IO.


I'm bringing this back up to python-dev to offer a slightly wider audience
for commentary/concerns, and hopefully to get a stamp of approval that it
is ready.

Thank you,
 - Josiah




On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson <josiah.carlson at gmail.com>
wrote:

> I've got a patch with partial tests and documentation that I'm holding off
> on upload because I believe there should be a brief discussion.
>
> Long story short, Windows needs a thread to handle writing in a
> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>
> If you'd like to continue following this issue and participate in the
> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>
>  - Josiah
>
>
>
> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson <josiah.carlson at gmail.com
> > wrote:
>
>>
>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
>>> josiah.carlson at gmail.com> wrote:
>>>
>>>>
>>>> If it makes you feel any better, I spent an hour this morning building
>>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and
>>>> not even using any part of asyncio (the Windows bits are in msvcrt and
>>>> _winapi). It works in Python 3.3+. You can see it here:
>>>> http://pastebin.com/0LpyQtU5
>>>>
>>>
>>> Seeing this makes *me* feel better. I think it's reasonable to add (some
>>> variant) of that to the subprocess module in Python 3.5. It also belongs in
>>> the Activestate cookbook. And no, the asyncio module hasn't made it
>>> obsolete.
>>>
>>
>> Cool.
>>
>>  Josiah, you sound upset about the whole thing -- to the point of
>>> writing unintelligible sentences and passive-aggressive digs at everyone
>>> reading this list. I'm sorry that something happened that led you feel that
>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote
>>> that code snippet -- it is completely clear what you want and why you want
>>> it, and also what should happen next (a few rounds of code review on the
>>> tracker).
>>>
>>
>> I'm not always a prat. Something about python-dev brings out parts of me
>> that I thought I had discarded from my personality years ago. Toss in a bit
>> of needing to re-explain ideas that I've been trying to explain for almost
>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>> pretty much why I won't be rejoining the party here regularly, you are all
>> better off without me commenting on 95% of threads like I used to.
>>
>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>> was when I spend time on this list. That's *my* issue, not yours. That I
>> spent any time redirecting my frustration towards you is BS, and if I could
>> take back the email I sent just before getting Guido's, I would.
>>
>> I would advise everyone to write it off as the ramblings of a
>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>> accurate. :)
>>
>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
>>> of example code. It doesn't specify the proposed interface, it just
>>> describes in way too many sentences how it should work, and contains a
>>> whole lot of references to various rants from which the reader is
>>> apparently meant to become enlightened. I don't know which of the three
>>> authors *really* wrote it, and I don't want to know -- I think the PEP is
>>> irrelevant to the proposed feature, which is of "put it in the bug tracker
>>> and work from there" category -- presumably the PEP was written based on
>>> the misunderstanding that having a PEP would make acceptance of the patch
>>> easier, or because during an earlier bikeshedding round someone said
>>> "please write a PEP" (someone always says that). I propose to scrap the PEP
>>> (set the status to Withdrawn) and just work on adding the methods to the
>>> subprocess module.
>>>
>>
>> I'm not going to argue. The first I read it was 2-3 days ago.
>>
>>  If it were me, I'd define three methods, with longer names to clarify
>>> what they do, e.g.
>>>
>>> proc.write_nonblocking(data)
>>> data = proc.read_nonblocking()
>>> data = proc.read_stderr_nonblocking()
>>>
>>
>> Easily doable.
>>
>> I.e. add _nonblocking to the method names to clarify that they may return
>>> '' when there's nothing available, and have a separate method for reading
>>> stderr instead of a flag. And I'd wonder if there should be an unambiguous
>>> way to detect EOF or whether the caller should just check for
>>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable
>>> when the other end is closed, and then the write() will fail. But maybe I
>>> forget.)
>>>
>>> But that's all bikeshedding and it can happen on the tracker or directly
>>> on the list just as easily; I don't see the need for a PEP.
>>>
>>
>>  Sounds good.
>>
>>  - Josiah
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/d6a1e8d9/attachment-0001.html>

From josiah.carlson at gmail.com  Fri May 30 02:35:21 2014
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 29 May 2014 17:35:21 -0700
Subject: [Python-Dev] Status of PEP 3145 - Asynchronous I/O for
	subprocess.popen
In-Reply-To: <CADcB0ygGCcY+wYdG6rUEum=sobV1q+CJmH+0GqQkGEvUcB7fMQ@mail.gmail.com>
References: <20140325231947.5ec9ad63@fsol>
 <CAMpsgwYAQX=ATP+RgUrdWz_WrbzMNeof+Y8QsaC-=PBPkeJdKQ@mail.gmail.com>
 <CADcB0yi8xdpKEzRgLjGYuj6OGgVOQxrjLTE7eWjOHK7MQudZzA@mail.gmail.com>
 <CAMpsgwYNHwCZ6cAmjC-BshfiVzJi9dpZ+EdWOWR8jsH6Y+PK0g@mail.gmail.com>
 <CADcB0yjLN-c0FRfk1nJEA8xn6QsMDM8CO_A7133Gjv0shRRH-A@mail.gmail.com>
 <lh2m4v$3an$1@ger.gmane.org>
 <CADcB0yhaU7aAO5=53xfPk4MiU8xxwEuhrG6ki2bjHdt89sC7Eg@mail.gmail.com>
 <CACac1F_oYD3KO9Wen_EcDb1=MvxQ_m+jncT0ZO754gUwsw5C0g@mail.gmail.com>
 <CADcB0yjJoG7C=XK2p1_bVKsSCFU7-mAmDVcvNG2nM4bWLuNaCw@mail.gmail.com>
 <CAP7+vJ+eteEA1wUFRsj29NsxqSK7W9-CHDJEMYH4AFioxG0vtA@mail.gmail.com>
 <CADcB0yix8abmQ2_0NLatt-VdVOnsbk_Zn6ByLR0vWiUHxgTfOg@mail.gmail.com>
 <CADcB0yisH5pfb2Ms2jBgFApq_1_qNnFyVZCDip3_MqD+H5a2TQ@mail.gmail.com>
 <CADcB0ygGCcY+wYdG6rUEum=sobV1q+CJmH+0GqQkGEvUcB7fMQ@mail.gmail.com>
Message-ID: <CADcB0yhy+OnzaVUOruHGiXT04fKBBvunSH6KtwHBhizTkM5_NA@mail.gmail.com>

And as I was writing the "thank you" to folks, I hit send too early. Also
thank you to Victor Stinner, Guido, Terry Reedy, and everyone else on this
thread :)

 - Josiah


On Thu, May 29, 2014 at 5:34 PM, Josiah Carlson <josiah.carlson at gmail.com>
wrote:

> Pinging this thread 2 months later with a progress/status update.
>
> To those that have reviewed, commented, helped, or otherwise pushed this
> along, which includes (but is not limited to) Richard Oudkerk, eryksun,
> Giampaolo Rodola, thank you.
>
>
> The short version:
> As far as I can tell, the patch is ready:
> http://bugs.python.org/issue1191964
>
> What is available:
> There are docs, tests, and obviously the functionality. Some code was
> moved from asyncio/windows_utils.py (which has a separate issue here:
> https://code.google.com/p/tulip/issues/detail?id=170). The API was
> changed slightly from what was proposed by Guido:
>
> sent = Popen.write_nonblocking(input, timeout=0)
> data = Popen.read_nonblocking(bufsize=4096)
> data = Popen.read_stderr_nonblocking(bufsize=4096)
>
> As a bonus feature, Windows communicate() calls no longer spawn worker
> threads, and instead use overlapped IO.
>
>
> I'm bringing this back up to python-dev to offer a slightly wider audience
> for commentary/concerns, and hopefully to get a stamp of approval that it
> is ready.
>
> Thank you,
>  - Josiah
>
>
>
>
> On Sat, Mar 29, 2014 at 11:58 PM, Josiah Carlson <josiah.carlson at gmail.com
> > wrote:
>
>> I've got a patch with partial tests and documentation that I'm holding
>> off on upload because I believe there should be a brief discussion.
>>
>> Long story short, Windows needs a thread to handle writing in a
>> non-blocking fashion, regardless of the use of asyncio or plain subprocess.
>>
>> If you'd like to continue following this issue and participate in the
>> discussion, I'll see you over on http://bugs.python.org/issue1191964 .
>>
>>  - Josiah
>>
>>
>>
>> On Fri, Mar 28, 2014 at 11:35 AM, Josiah Carlson <
>> josiah.carlson at gmail.com> wrote:
>>
>>>
>>> On Fri, Mar 28, 2014 at 10:46 AM, Guido van Rossum <guido at python.org>
>>> wrote:
>>>
>>>> On Fri, Mar 28, 2014 at 9:45 AM, Josiah Carlson <
>>>> josiah.carlson at gmail.com> wrote:
>>>>
>>>>>
>>>>> If it makes you feel any better, I spent an hour this morning building
>>>>> a 2-function API for Linux and Windows, both tested, not using ctypes, and
>>>>> not even using any part of asyncio (the Windows bits are in msvcrt and
>>>>> _winapi). It works in Python 3.3+. You can see it here:
>>>>> http://pastebin.com/0LpyQtU5
>>>>>
>>>>
>>>> Seeing this makes *me* feel better. I think it's reasonable to add
>>>> (some variant) of that to the subprocess module in Python 3.5. It also
>>>> belongs in the Activestate cookbook. And no, the asyncio module hasn't made
>>>> it obsolete.
>>>>
>>>
>>> Cool.
>>>
>>>  Josiah, you sound upset about the whole thing -- to the point of
>>>> writing unintelligible sentences and passive-aggressive digs at everyone
>>>> reading this list. I'm sorry that something happened that led you feel that
>>>> way (if you indeed feel upset or frustrated) but I'm glad that you wrote
>>>> that code snippet -- it is completely clear what you want and why you want
>>>> it, and also what should happen next (a few rounds of code review on the
>>>> tracker).
>>>>
>>>
>>> I'm not always a prat. Something about python-dev brings out parts of me
>>> that I thought I had discarded from my personality years ago. Toss in a bit
>>> of needing to re-explain ideas that I've been trying to explain for almost
>>> 9 years? Frustration + formerly discarded personality traits = uck. That's
>>> pretty much why I won't be rejoining the party here regularly, you are all
>>> better off without me commenting on 95% of threads like I used to.
>>>
>>> Victor, I'm sorry for being a jerk. It's hard for me to not be the guy I
>>> was when I spend time on this list. That's *my* issue, not yours. That I
>>> spent any time redirecting my frustration towards you is BS, and if I could
>>> take back the email I sent just before getting Guido's, I would.
>>>
>>> I would advise everyone to write it off as the ramblings of a
>>> surprisingly young, angry old man. Or call me an a-hole. Both are pretty
>>> accurate. :)
>>>
>>> But that PEP? It's just a terrible PEP. It doesn't contain a single line
>>>> of example code. It doesn't specify the proposed interface, it just
>>>> describes in way too many sentences how it should work, and contains a
>>>> whole lot of references to various rants from which the reader is
>>>> apparently meant to become enlightened. I don't know which of the three
>>>> authors *really* wrote it, and I don't want to know -- I think the PEP is
>>>> irrelevant to the proposed feature, which is of "put it in the bug tracker
>>>> and work from there" category -- presumably the PEP was written based on
>>>> the misunderstanding that having a PEP would make acceptance of the patch
>>>> easier, or because during an earlier bikeshedding round someone said
>>>> "please write a PEP" (someone always says that). I propose to scrap the PEP
>>>> (set the status to Withdrawn) and just work on adding the methods to the
>>>> subprocess module.
>>>>
>>>
>>> I'm not going to argue. The first I read it was 2-3 days ago.
>>>
>>>  If it were me, I'd define three methods, with longer names to clarify
>>>> what they do, e.g.
>>>>
>>>> proc.write_nonblocking(data)
>>>> data = proc.read_nonblocking()
>>>> data = proc.read_stderr_nonblocking()
>>>>
>>>
>>> Easily doable.
>>>
>>> I.e. add _nonblocking to the method names to clarify that they may
>>>> return '' when there's nothing available, and have a separate method for
>>>> reading stderr instead of a flag. And I'd wonder if there should be an
>>>> unambiguous way to detect EOF or whether the caller should just check for
>>>> proc.stdout.closed. (And what for stdin? IIRC it actually becomes writable
>>>> when the other end is closed, and then the write() will fail. But maybe I
>>>> forget.)
>>>>
>>>> But that's all bikeshedding and it can happen on the tracker or
>>>> directly on the list just as easily; I don't see the need for a PEP.
>>>>
>>>
>>>  Sounds good.
>>>
>>>  - Josiah
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140529/aca9db64/attachment.html>

From status at bugs.python.org  Fri May 30 18:07:58 2014
From: status at bugs.python.org (Python tracker)
Date: Fri, 30 May 2014 18:07:58 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20140530160758.17F8056A46@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2014-05-23 - 2014-05-30)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    4635 (+13)
  closed 28750 (+40)
  total  33385 (+53)

Open issues with patches: 2127 


Issues opened (41)
==================

#16638: support multi-line docstring signatures in IDLE calltips
http://bugs.python.org/issue16638  reopened by rhettinger

#21561: help() on enum34 enumeration class creates only a dummy docume
http://bugs.python.org/issue21561  opened by andymaier

#21563: Segv during call to builtin_execfile in application embedding 
http://bugs.python.org/issue21563  opened by snoeberger

#21564: Declaration of EVP_MD_CTX causes crash when switching between 
http://bugs.python.org/issue21564  opened by Ryan.Calhoun

#21566: make use of the new default socket.listen() backlog argument
http://bugs.python.org/issue21566  opened by neologix

#21567: cannot create multipart alternative message with us-ascii char
http://bugs.python.org/issue21567  opened by Viktor.Sz??pe

#21568: Win32 pip doesn't use relative path to found site-packages.
http://bugs.python.org/issue21568  opened by ??????.???

#21569: PEP 466: Python 2.7 What's New preamble changes
http://bugs.python.org/issue21569  opened by ncoghlan

#21571: Python build should check CPATH, C_INCLUDE_PATH for module dep
http://bugs.python.org/issue21571  opened by JanKanis

#21572: Use generic license web page rather than requiring release-spe
http://bugs.python.org/issue21572  opened by ned.deily

#21573: Clean up turtle.py code formatting
http://bugs.python.org/issue21573  opened by jesstess

#21574: Port image types detections from PIL to the imghdr module
http://bugs.python.org/issue21574  opened by serhiy.storchaka

#21576: Overwritten (custom) uuid inside dictionary
http://bugs.python.org/issue21576  opened by beta990

#21577: Help for ImportError should show a more useful signature.
http://bugs.python.org/issue21577  opened by eric.snow

#21578: Misleading error message when ImportError called with invalid 
http://bugs.python.org/issue21578  opened by eric.snow

#21579: Python 3.4: tempfile.close attribute does not work
http://bugs.python.org/issue21579  opened by mmarkk

#21580: PhotoImage(data=...) apparently has to be UTF-8 or Base-64 enc
http://bugs.python.org/issue21580  opened by vadmium

#21581: Consider dropping importlib.abc.Loader.create_module()
http://bugs.python.org/issue21581  opened by brett.cannon

#21582: use support.catpured context managers - test_asyncore
http://bugs.python.org/issue21582  opened by diana

#21583: use support.catpured_stderr context manager - test_logging
http://bugs.python.org/issue21583  opened by diana

#21585: Run Tkinter tests with wantobjects=False
http://bugs.python.org/issue21585  opened by serhiy.storchaka

#21588: Idle: make editor title bar user configurable
http://bugs.python.org/issue21588  opened by terry.reedy

#21590: Systemtap and DTrace support
http://bugs.python.org/issue21590  opened by bkabrda

#21591: "exec(a, b, c)" not the same as "exec a in b, c" in nested fun
http://bugs.python.org/issue21591  opened by Robert.Jordens

#21592: Make statistics.median run in linear time
http://bugs.python.org/issue21592  opened by Thomas.Dybdahl.Ahle

#21593: Clarify re.search documentation first match
http://bugs.python.org/issue21593  opened by Joshua.Landau

#21594: asyncio.create_subprocess_exec raises OSError
http://bugs.python.org/issue21594  opened by Sebastian.Kreft.Deezer

#21595: Creating many subprocess generates lots of internal BlockingIO
http://bugs.python.org/issue21595  opened by Sebastian.Kreft.Deezer

#21596: asyncio.wait fails when futures list is empty
http://bugs.python.org/issue21596  opened by Sebastian.Kreft.Deezer

#21597: Allow turtledemo code pane to get wider.
http://bugs.python.org/issue21597  opened by terry.reedy

#21599: Argument transport in attach and detach method in Server class
http://bugs.python.org/issue21599  opened by vajrasky

#21600: mock.patch.stopall doesn't work with patch.dict to sys.modules
http://bugs.python.org/issue21600  opened by kakuma

#21601: Cancel method for Asyncio Task is not documented
http://bugs.python.org/issue21601  opened by vajrasky

#21603: IDLE SaveAs drops the extension in the prompted filename
http://bugs.python.org/issue21603  opened by rhettinger

#21604: Misleading 2to3 fixer name in documentation: standard_error
http://bugs.python.org/issue21604  opened by wluebbe

#21605: Add tests for Tkinter images
http://bugs.python.org/issue21605  opened by serhiy.storchaka

#21608: logging.handlers.HTTPHandler.setFormatter() has no effect
http://bugs.python.org/issue21608  opened by bk2zsto

#21610: load_module not closing opened files
http://bugs.python.org/issue21610  opened by mattip

#21611: int() docstring - unclear what number is
http://bugs.python.org/issue21611  opened by and

#21612: IDLE should not open multiple instances of one file
http://bugs.python.org/issue21612  opened by irdb

#21613: Installer for mac doesn't store the installation location
http://bugs.python.org/issue21613  opened by ustinov



Most recent 15 issues with no replies (15)
==========================================

#21612: IDLE should not open multiple instances of one file
http://bugs.python.org/issue21612

#21611: int() docstring - unclear what number is
http://bugs.python.org/issue21611

#21605: Add tests for Tkinter images
http://bugs.python.org/issue21605

#21604: Misleading 2to3 fixer name in documentation: standard_error
http://bugs.python.org/issue21604

#21601: Cancel method for Asyncio Task is not documented
http://bugs.python.org/issue21601

#21600: mock.patch.stopall doesn't work with patch.dict to sys.modules
http://bugs.python.org/issue21600

#21599: Argument transport in attach and detach method in Server class
http://bugs.python.org/issue21599

#21597: Allow turtledemo code pane to get wider.
http://bugs.python.org/issue21597

#21596: asyncio.wait fails when futures list is empty
http://bugs.python.org/issue21596

#21593: Clarify re.search documentation first match
http://bugs.python.org/issue21593

#21591: "exec(a, b, c)" not the same as "exec a in b, c" in nested fun
http://bugs.python.org/issue21591

#21582: use support.catpured context managers - test_asyncore
http://bugs.python.org/issue21582

#21577: Help for ImportError should show a more useful signature.
http://bugs.python.org/issue21577

#21568: Win32 pip doesn't use relative path to found site-packages.
http://bugs.python.org/issue21568

#21564: Declaration of EVP_MD_CTX causes crash when switching between 
http://bugs.python.org/issue21564



Most recent 15 issues waiting for review (15)
=============================================

#21610: load_module not closing opened files
http://bugs.python.org/issue21610

#21608: logging.handlers.HTTPHandler.setFormatter() has no effect
http://bugs.python.org/issue21608

#21605: Add tests for Tkinter images
http://bugs.python.org/issue21605

#21601: Cancel method for Asyncio Task is not documented
http://bugs.python.org/issue21601

#21599: Argument transport in attach and detach method in Server class
http://bugs.python.org/issue21599

#21595: Creating many subprocess generates lots of internal BlockingIO
http://bugs.python.org/issue21595

#21585: Run Tkinter tests with wantobjects=False
http://bugs.python.org/issue21585

#21583: use support.catpured_stderr context manager - test_logging
http://bugs.python.org/issue21583

#21582: use support.catpured context managers - test_asyncore
http://bugs.python.org/issue21582

#21580: PhotoImage(data=...) apparently has to be UTF-8 or Base-64 enc
http://bugs.python.org/issue21580

#21578: Misleading error message when ImportError called with invalid 
http://bugs.python.org/issue21578

#21572: Use generic license web page rather than requiring release-spe
http://bugs.python.org/issue21572

#21566: make use of the new default socket.listen() backlog argument
http://bugs.python.org/issue21566

#21564: Declaration of EVP_MD_CTX causes crash when switching between 
http://bugs.python.org/issue21564

#21563: Segv during call to builtin_execfile in application embedding 
http://bugs.python.org/issue21563



Top 10 most discussed issues (10)
=================================

#21477: Idle: improve idle_test.htest
http://bugs.python.org/issue21477  18 msgs

#20383: Add a keyword-only spec argument to types.ModuleType
http://bugs.python.org/issue20383  13 msgs

#17172: Add turtledemo to IDLE menu
http://bugs.python.org/issue17172  10 msgs

#20611: socket.create_connection() doesn't handle EINTR properly
http://bugs.python.org/issue20611   9 msgs

#20689: socket.AddressFamily is absent in pydoc output
http://bugs.python.org/issue20689   7 msgs

#21579: Python 3.4: tempfile.close attribute does not work
http://bugs.python.org/issue21579   7 msgs

#3015: tkinter with wantobjects=False has been broken for some time
http://bugs.python.org/issue3015   6 msgs

#21235: importlib's spec module create algorithm is not exposed
http://bugs.python.org/issue21235   6 msgs

#8243: curses writing to window's bottom right position raises: `_cur
http://bugs.python.org/issue8243   5 msgs

#21561: help() on enum34 enumeration class creates only a dummy docume
http://bugs.python.org/issue21561   5 msgs



Issues closed (39)
==================

#1641: asyncore delayed calls feature
http://bugs.python.org/issue1641  closed by haypo

#7094: Add alternate float formatting styles to new-style formatting.
http://bugs.python.org/issue7094  closed by eric.smith

#8743: set() operators don't work with collections.Set instances
http://bugs.python.org/issue8743  closed by rhettinger

#10203: sqlite3.Row doesn't support sequence protocol
http://bugs.python.org/issue10203  closed by serhiy.storchaka

#13742: Add a key parameter (like sorted) to heapq.merge
http://bugs.python.org/issue13742  closed by rhettinger

#14315: zipfile.ZipFile() unable to open zip File with a short extra h
http://bugs.python.org/issue14315  closed by gregory.p.smith

#14710: pkgutil.get_loader and find_loader fail when module is missing
http://bugs.python.org/issue14710  closed by brett.cannon

#15246: Line coverage for collectionts.abc.Set
http://bugs.python.org/issue15246  closed by rhettinger

#16774: Additional recipes for itertools docs
http://bugs.python.org/issue16774  closed by rhettinger

#19048: itertools.tee doesn't have a __sizeof__ method
http://bugs.python.org/issue19048  closed by rhettinger

#19925: Add unit test for spwd module
http://bugs.python.org/issue19925  closed by serhiy.storchaka

#20197: Support WebP image format detection in imghdr module
http://bugs.python.org/issue20197  closed by serhiy.storchaka

#21137: Better repr for threading.Lock()
http://bugs.python.org/issue21137  closed by rhettinger

#21343: os.path.relpath returns inconsistent types
http://bugs.python.org/issue21343  closed by serhiy.storchaka

#21376: asyncio docs refer to wrong TimeoutError
http://bugs.python.org/issue21376  closed by haypo

#21402: tkinter.ttk._val_or_dict assumes tkinter._default_root exists
http://bugs.python.org/issue21402  closed by serhiy.storchaka

#21434: python -3 documentation is outdated
http://bugs.python.org/issue21434  closed by python-dev

#21439: Numerous minor issues in Language Reference
http://bugs.python.org/issue21439  closed by rhettinger

#21454: asyncio's loop.connect_read_pipe makes pipes non-blocking cont
http://bugs.python.org/issue21454  closed by haypo

#21481: Argpase Namespace object methods __eq__ and __ne__  raise Type
http://bugs.python.org/issue21481  closed by rhettinger

#21493: Add test for ntpath.expanduser
http://bugs.python.org/issue21493  closed by serhiy.storchaka

#21513: speed up some ipaddress properties
http://bugs.python.org/issue21513  closed by pitrou

#21534: 404 on documentation download links
http://bugs.python.org/issue21534  closed by benjamin.peterson

#21546: int('\0') gives wrong error message
http://bugs.python.org/issue21546  closed by terry.reedy

#21555: gcmodule.c could use pytime.h
http://bugs.python.org/issue21555  closed by pitrou

#21558: Fix a typo in the contextlib docs
http://bugs.python.org/issue21558  closed by rhettinger

#21562: curses getsxy() should be curses getxy() in https://docs.pytho
http://bugs.python.org/issue21562  closed by jcea

#21565: multiprocessing: use contex-manager protocol for synchronizati
http://bugs.python.org/issue21565  closed by neologix

#21570: String being confused with datetime.datetime object.
http://bugs.python.org/issue21570  closed by brandon

#21575: list.sort() should show arguments in tutorial
http://bugs.python.org/issue21575  closed by rhettinger

#21584: Unraised overflow error in sqlite3.Row indexing
http://bugs.python.org/issue21584  closed by serhiy.storchaka

#21586: Minor error "How To Sockets"
http://bugs.python.org/issue21586  closed by python-dev

#21587: Tabs in source
http://bugs.python.org/issue21587  closed by python-dev

#21589: Use better idiom in unittest example
http://bugs.python.org/issue21589  closed by ezio.melotti

#21598: Is __getitem__ and __len__ implementations enough to make a us
http://bugs.python.org/issue21598  closed by santa4nt

#21602: smtplib.py socket.create_connection() also doesn't handle EINT
http://bugs.python.org/issue21602  closed by neologix

#21606: No visual feedback when entering japanese Characters in Entry 
http://bugs.python.org/issue21606  closed by ned.deily

#21607: results of `zip` are displayed as '<zip object at 0xxxxxx>
http://bugs.python.org/issue21607  closed by SilentGhost

#21609: Documentation for datetime.datetime uses microseconds instead 
http://bugs.python.org/issue21609  closed by Miquel.Garcia

From chris.barker at noaa.gov  Fri May 30 18:46:52 2014
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 30 May 2014 09:46:52 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
Message-ID: <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>

On Thu, May 29, 2014 at 4:43 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> For that last point, my interest is as much educational as it is in
> easing the transition from Python 2. The parentheses in "print('Hello
> world!')" mean introducing the idea of function calls early to explain
> how it works, while being able to omit them makes it easier to gloss
> over the distinction between statements and function calls initially
> and then cover it later after the basics of flow control have been
> nailed down.
>

I've been doing a lot of intro-to-python teaching lately (py2 so far...),
and I understand this desire. IN fact, a lot of notes point to the fact
that python's "hello world" is simply : print "hello world", rather than
what is required in some languages to do something that simple.

However, I also believe that when teaching it's better to introduce the
"right way" to do something up front, rather than a "beginners' way", then
later say, well, you really SHOULD do it this other way... So if we want
our students to use print as a function, we might a well start them off
that way. Saying that their very first easy program is:

print("hello world")

is fine -- they don't have to know or understand what a function call is --
they simply copy the syntax. And frankly, we get to simple function calls,
VERY early in the program -- you can't really do anything without them...

In fact, in my latest class, we've made an effort to introduce
forward-thinking up front, even before we explain quite what it all means:

use u"a string" to make a string

you write a class like:

class C(object):
   ...

before we talk about subclassing, or what "object" is...

just my $0.02

-Chris












> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/07016708/attachment.html>

From wizzat at gmail.com  Fri May 30 19:40:18 2014
From: wizzat at gmail.com (Mark Roberts)
Date: Fri, 30 May 2014 10:40:18 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <lm7nas$tp2$1@ger.gmane.org>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <lm7nas$tp2$1@ger.gmane.org>
Message-ID: <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>

On Thu, May 29, 2014 at 9:30 AM, Terry Reedy <tjreedy at udel.edu> wrote:

> On 5/28/2014 6:26 PM, Glyph Lefkowitz wrote:
>
>  I hope it's
>> not controversial to say that most new Python code is still being
>> written against Python 2.7 today;
>>
>
> Given that Python 3 downloads now outnumber Python 2 downloads, I think
> 'most' might be an overstatement. But I think it a moot point.



How can you determine that Python3 downloads actually outnumber Python2
downloads?  It seems that looking at Windows downloads (as I saw in a
thread earlier this month) is fallacious at absolute best.  Linux and OSX
both ship with Python2, and most downloads happen via individual package
management tools.  Even including Python3.4 as a default Python3 doesn't
mean that it's the default Python on the system.

>From my perspective as an engineer and library maintainer: Pypi seems to
indicate an overwhelming number of Python2 usage, and so do the job
requisitions that I've seen.  Furthermore, even Python based libraries for
cutting edge technologies are still written in Python2 and later converted
to Python3.  I just don't understand how we (as a community) can make the
assertion that most "new Python" is written in Python 3.

What I'd really like to see is a Python 2.8 that makes sufficient changes
to Python 2 that writing libraries which cross the boundary between 2 and 3
is relatively easy instead of a painful nightmarish chore.  Because when
push comes to shove, Python 2 support is still infinitely more important
than Python 3.

-Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/37f33bfd/attachment.html>

From rosuav at gmail.com  Fri May 30 19:51:00 2014
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 31 May 2014 03:51:00 +1000
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <lm7nas$tp2$1@ger.gmane.org>
 <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
Message-ID: <CAPTjJmrrRPRfLH77uUH4_gfv6=939guw6wDk2q88cVWYE1fi-Q@mail.gmail.com>

On Sat, May 31, 2014 at 3:40 AM, Mark Roberts <wizzat at gmail.com> wrote:
> What I'd really like to see is a Python 2.8 that makes sufficient changes to
> Python 2 that writing libraries which cross the boundary between 2 and 3 is
> relatively easy instead of a painful nightmarish chore.  Because when push
> comes to shove, Python 2 support is still infinitely more important than
> Python 3.

That's of absolutely ZERO value to anyone who has to maintain support
for Python 2.3 or even 2.7. Will this hypothetical 2.8 run, unchanged,
all code written for 2.7? If not, all you've done is add a third
branch to maintain, and solved nothing; and if it will, how much can
you really change?

ChrisA

From breamoreboy at yahoo.co.uk  Fri May 30 20:06:51 2014
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Fri, 30 May 2014 19:06:51 +0100
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <lm7nas$tp2$1@ger.gmane.org>
 <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
Message-ID: <lmahbt$i5h$1@ger.gmane.org>

On 30/05/2014 18:40, Mark Roberts wrote:
>
> What I'd really like to see is a Python 2.8 that makes sufficient
> changes to Python 2 that writing libraries which cross the boundary
> between 2 and 3 is relatively easy instead of a painful nightmarish
> chore.  Because when push comes to shove, Python 2 support is still
> infinitely more important than Python 3.
>
> -Mark
>

What is the point of flogging a horse that's been dead for so long that 
it's already down to a skeleton?

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com



From ethan at stoneleaf.us  Fri May 30 19:01:42 2014
From: ethan at stoneleaf.us (Ethan Furman)
Date: Fri, 30 May 2014 10:01:42 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
 <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
Message-ID: <5388B976.8050305@stoneleaf.us>

On 05/30/2014 09:46 AM, Chris Barker wrote:
> On Thu, May 29, 2014 at 4:43 AM, Nick Coghlan wrote:
>>
>> For that last point, my interest is as much educational as it is in
>> easing the transition from Python 2. The parentheses in "print('Hello
>> world!')" mean introducing the idea of function calls early to explain
>> how it works, while being able to omit them makes it easier to gloss
>> over the distinction between statements and function calls initially
>> and then cover it later after the basics of flow control have been
>> nailed down.

> However, I also believe that when teaching it's better to introduce the "right way" to do something up front, rather
> than a "beginners' way", then later say, well, you really SHOULD do it this other way...

+1

Function calls are not that hard to understand.  Anybody who has ever asked someone to do something for them should have 
a basic grasp of the nature of a function call.

--
~Ethan~

From ericsnowcurrently at gmail.com  Fri May 30 21:39:12 2014
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Fri, 30 May 2014 13:39:12 -0600
Subject: [Python-Dev] [Python-checkins] cpython: Issue #20383: Introduce
	importlib.util.module_from_spec().
In-Reply-To: <3ggFND0tdbz7SSy@mail.python.org>
References: <3ggFND0tdbz7SSy@mail.python.org>
Message-ID: <CALFfu7B0ZOmyBrgXs=aWRoi_GJ0v1R09gXE8y0ZJjNxq_AqkLQ@mail.gmail.com>

On Fri, May 30, 2014 at 12:55 PM, brett.cannon
<python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/b26d021081d2
> changeset:   90915:b26d021081d2
> parent:      90913:69011f6ce573
> user:        Brett Cannon <brett at python.org>
> date:        Fri May 30 14:55:29 2014 -0400
> summary:
>   Issue #20383: Introduce importlib.util.module_from_spec().
>
> diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
> --- a/Lib/importlib/_bootstrap.py
> +++ b/Lib/importlib/_bootstrap.py
> @@ -581,20 +581,19 @@
>      return loader
>
>
> -def _load_module_shim(self, fullname):
> +def _load_module_shim(spec, fullname):
>      """Load the specified module into sys.modules and return it.

This should have stayed "self", no?

-eric

From bcannon at gmail.com  Fri May 30 22:28:46 2014
From: bcannon at gmail.com (Brett Cannon)
Date: Fri, 30 May 2014 20:28:46 +0000
Subject: [Python-Dev] [Python-checkins] cpython: Issue #20383: Introduce
	importlib.util.module_from_spec().
References: <3ggFND0tdbz7SSy@mail.python.org>
 <CALFfu7B0ZOmyBrgXs=aWRoi_GJ0v1R09gXE8y0ZJjNxq_AqkLQ@mail.gmail.com>
Message-ID: <CAP1=2W7Wvoi6NpUyTXAs21wbyYnMdA=LOh3Bcx=wDv0hofFKSg@mail.gmail.com>

On Fri May 30 2014 at 3:39:47 PM, Eric Snow <ericsnowcurrently at gmail.com>
wrote:

> On Fri, May 30, 2014 at 12:55 PM, brett.cannon
> <python-checkins at python.org> wrote:
> > http://hg.python.org/cpython/rev/b26d021081d2
> > changeset:   90915:b26d021081d2
> > parent:      90913:69011f6ce573
> > user:        Brett Cannon <brett at python.org>
> > date:        Fri May 30 14:55:29 2014 -0400
> > summary:
> >   Issue #20383: Introduce importlib.util.module_from_spec().
> >
> > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
> > --- a/Lib/importlib/_bootstrap.py
> > +++ b/Lib/importlib/_bootstrap.py
> > @@ -581,20 +581,19 @@
> >      return loader
> >
> >
> > -def _load_module_shim(self, fullname):
> > +def _load_module_shim(spec, fullname):
> >      """Load the specified module into sys.modules and return it.
>
> This should have stayed "self", no?
>
>
Yes. Fixed with an added comment to explain why since it isn't obvious in
isolation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/23a36f6b/attachment-0001.html>

From solipsis at pitrou.net  Fri May 30 22:47:31 2014
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 30 May 2014 22:47:31 +0200
Subject: [Python-Dev] Language Summit Follow-Up
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
Message-ID: <20140530224731.1abe5ede@fsol>

On Wed, 28 May 2014 15:26:38 -0700
Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> Backport 'yield from' to allow people to use Tulip and Tulip-compatible code, and to facilitate the development of Tulip-friendly libraries and a Tulip ecosystem.  A robust Tulip ecosystem requires the participation of people who are not yet using Python 3.

I was wondering whether you were trolling or not on this one.
From a quality assurance point of view, adding major features to a
bugfix branch is extremely destructive, so I'm strongly -1 on it.

> Get rid of 2to3. Particularly, of any discussion of using 2to3 in the documentation.  More than one very experienced, well-known Python developer in this discussion has told me that they thought 2to3 was the blessed way to port their code, and it's no surprise that they think so, given that the first technique <https://docs.python.org/3/howto/pyporting.html> mentions is still 2to3.

2to3 is certainly fine if you are porting to 3.x without looking to
keep your code 2.x-compatible. Until there's a better alternative, of
course.
So what we should do is better explain the choice (if you want to port
your code to 3.x, use 2to3; if you want to maintain dual-compatible
code, use six or something similar).

Regards

Antoine.



From guido at python.org  Sat May 31 00:39:22 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 May 2014 15:39:22 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <20140530224731.1abe5ede@fsol>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <20140530224731.1abe5ede@fsol>
Message-ID: <CAP7+vJ+PGnA-As3hdsT=bjbdyZOm6dsB4Rz6ugdBvoSkq_Epog@mail.gmail.com>

2to3 is poorly named. With different fixers it is a fine tool for
converting 2-only code to 2-and-3 straddling code. Even when using six, you
need to do things like convert print statements to print() calls with
future import, use 'as' in except clauses, and so on.


On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Wed, 28 May 2014 15:26:38 -0700
> Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> > Backport 'yield from' to allow people to use Tulip and Tulip-compatible
> code, and to facilitate the development of Tulip-friendly libraries and a
> Tulip ecosystem.  A robust Tulip ecosystem requires the participation of
> people who are not yet using Python 3.
>
> I was wondering whether you were trolling or not on this one.
> From a quality assurance point of view, adding major features to a
> bugfix branch is extremely destructive, so I'm strongly -1 on it.
>
> > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the
> documentation.  More than one very experienced, well-known Python developer
> in this discussion has told me that they thought 2to3 was the blessed way
> to port their code, and it's no surprise that they think so, given that the
> first technique <https://docs.python.org/3/howto/pyporting.html> mentions
> is still 2to3.
>
> 2to3 is certainly fine if you are porting to 3.x without looking to
> keep your code 2.x-compatible. Until there's a better alternative, of
> course.
> So what we should do is better explain the choice (if you want to port
> your code to 3.x, use 2to3; if you want to maintain dual-compatible
> code, use six or something similar).
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/b4ca43bb/attachment.html>

From ncoghlan at gmail.com  Sat May 31 03:46:33 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 31 May 2014 11:46:33 +1000
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CAP7+vJ+PGnA-As3hdsT=bjbdyZOm6dsB4Rz6ugdBvoSkq_Epog@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <20140530224731.1abe5ede@fsol>
 <CAP7+vJ+PGnA-As3hdsT=bjbdyZOm6dsB4Rz6ugdBvoSkq_Epog@mail.gmail.com>
Message-ID: <CADiSq7eZ14va5jQmA2uU=V-rizY-UrPCKgOnadyXZWToUONCmA@mail.gmail.com>

On 31 May 2014 08:42, "Guido van Rossum" <guido at python.org> wrote:
>
> 2to3 is poorly named. With different fixers it is a fine tool for
converting 2-only code to 2-and-3 straddling code. Even when using six, you
need to do things like convert print statements to print() calls with
future import, use 'as' in except clauses, and so on.

Both modernize & futurize build on lib2to3 to do the heavy lifting - they
don't reinvent that wheel, they just change which fixes are applied by
default and add a few more of their own.

However, even if going straight to Python 3, I suspect folks would still be
better off doing something like:

* futurize --stage1 (migrates to 2.6+, but adds no new runtime dependencies
- the output should be fairly idiomatic Python 2.6/7 code)
* 2to3 (wholesale migration to Python 3)

The trick, though, is that the granularity of that approach is at the
process level - the entire application must be converted at once. You also
can't start your own migration until all your dependencies are available in
Python 3.

By contrast, migration via the common subset can be incremental and
opportunistic:

* the granularity of migration is individual modules, rather than entire
processes, so you can make a low risk gradual transition, even without a
comprehensive regression test suite
* you initially stay on Python 2, so you can start whenever is convenient
for you, rather than having to wait for all your dependencies

Cheers,
Nick.

>
>
> On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net>
wrote:
>>
>> On Wed, 28 May 2014 15:26:38 -0700
>> Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
>> > Backport 'yield from' to allow people to use Tulip and
Tulip-compatible code, and to facilitate the development of Tulip-friendly
libraries and a Tulip ecosystem.  A robust Tulip ecosystem requires the
participation of people who are not yet using Python 3.
>>
>> I was wondering whether you were trolling or not on this one.
>> From a quality assurance point of view, adding major features to a
>> bugfix branch is extremely destructive, so I'm strongly -1 on it.
>>
>> > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the
documentation.  More than one very experienced, well-known Python developer
in this discussion has told me that they thought 2to3 was the blessed way
to port their code, and it's no surprise that they think so, given that the
first technique <https://docs.python.org/3/howto/pyporting.html> mentions
is still 2to3.
>>
>> 2to3 is certainly fine if you are porting to 3.x without looking to
>> keep your code 2.x-compatible. Until there's a better alternative, of
>> course.
>> So what we should do is better explain the choice (if you want to port
>> your code to 3.x, use 2to3; if you want to maintain dual-compatible
>> code, use six or something similar).
>>
>> Regards
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140531/67faf9b1/attachment.html>

From ncoghlan at gmail.com  Sat May 31 04:05:06 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 31 May 2014 12:05:06 +1000
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
 <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
Message-ID: <CADiSq7dkAYpvTQux9BE4jmSVj6kz9YAOswsRCxwXzDr6RX7wYA@mail.gmail.com>

On 31 May 2014 02:47, "Chris Barker" <chris.barker at noaa.gov> wrote:
> However, I also believe that when teaching it's better to introduce the
"right way" to do something up front, rather than a "beginners' way", then
later say, well, you really SHOULD do it this other way... So if we want
our students to use print as a function, we might a well start them off
that way. Saying that their very first easy program is:
>
> print("hello world")
>
> is fine -- they don't have to know or understand what a function call is
-- they simply copy the syntax. And frankly, we get to simple function
calls, VERY early in the program -- you can't really do anything without
them...

OK, that's a fair criticism of that particular argument, so I'll stop using
it. There are others, though:

* improve consistency between IPython's existing implied call support and
vanilla Python 3.5+
* retroactively make a lot of Python 2 examples also valid for Python 3.5+
* ease the migration to Python 3 for ad hoc utility scripts (which often
use print heavily)
* reduce the syntactic noise in migration patches (although large
applications tend to use print less than scripts do)
* largely eliminate one of the *human* barriers to migration from Python 2
(forgetting the parens for print at the interactive console and in ad hoc
scripts)

I already have too many other things on my todo list to work this up into a
full PEP, but the proof of concept (along with IPython's existing support)
shows there's no *technical* barrier to adding the feature. It's a question
of whether or not we *want* to blur the boundary between simple statements
and simple imperative function calls like that.

Cheers,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140531/510ed181/attachment.html>

From ncoghlan at gmail.com  Sat May 31 04:32:51 2014
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 31 May 2014 12:32:51 +1000
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <lm7nas$tp2$1@ger.gmane.org>
 <CACbEM1dweMWqusUL_LsuseTFGauiArwycvzhhUWu2OzkWh5NVQ@mail.gmail.com>
Message-ID: <CADiSq7e1AtNaFXPM_y7X-1LBfRXMEHMD3sUPBE2uJ6R-=4TR6A@mail.gmail.com>

On 31 May 2014 03:42, "Mark Roberts" <wizzat at gmail.com> wrote:
>
> What I'd really like to see is a Python 2.8 that makes sufficient changes
to Python 2 that writing libraries which cross the boundary between 2 and 3
is relatively easy instead of a painful nightmarish chore.

That's what projects like python-future are for. Python 2.8 wouldn't help,
since most folks still target 2.6 for compatibility with stable Linux
platforms like RHEL, CentOS, Debian Stable & Ubuntu LTS.

This is a key point folks often miss in these discussions: even putting
Python 3 aside entirely, the migration of the overall ecosystem from Python
2.6 to Python *2.7* is not yet finished. RHEL 7 (which uses 2.7 as the
system Python) is currently only available as a release candidate, and the
same necessarily holds true for its downstream rebuilds like CentOS. We saw
this happen with Python 2.4 as well: for a lot of library developers, the
day CentOS 6 finally landed (with Python 2.6 as the system Python) was the
day they finally decided to drop support for Python 2.4.

It's that slow adoption cycle for new feature releases *within* the Python
2 series that means the effort that would be needed to create a Python 2.8
release is better put into tools and utilities that people can use *now*
(like PyPI backports of standard library modules, python-future and the
"pymigrate" utility Steve Dower suggested and Eric Snow has started working
on), tweaks to 2.7 itself (like PEP 466 and a possible future backport of
the ensurepip changes) and Python 3 changes that improve both Python 3
*and* the subset it shares with Python 2 (like the restoration of binary
interpolation support approved for 3.5).

Cheers,
Nick.

P.S. I've written more about adoption cycles for new Python versions at
http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#wouldn-t-a-python-2-8-release-help-ease-the-transition
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140531/8ac898d6/attachment.html>

From tjreedy at udel.edu  Sat May 31 05:32:04 2014
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 30 May 2014 23:32:04 -0400
Subject: [Python-Dev] Updating  turtle.py
Message-ID: <lmbifr$drl$1@ger.gmane.org>

I have two areas of questions about updating turtle.py. First the module 
itself, then a turtle tracker issue versus code cleanup policies.

A. Unlike most stdlib modules, turtle is copyrighted and licensed by an 
individual.
'''
# turtle.py: a Tkinter based turtle graphics module for Python
# Version 1.1b - 4. 5. 2009
# Copyright (C) 2006 - 2010  Gregor Lingl
# email: glingl at aon.at
'''
I am not sure what the copyright covers other than the exact text 
contributed, with updates, by Gregor. It certainly does not cover the 
API and whatever code he copied from the previous version (unless that 
was also by him, and I have no idea how much he copied when 
reimplementing). I don't think it should cover additions made by others 
either. Should there be another line to cover these?
'''
# Permission is granted to anyone to use this software for any purpose,
# including commercial applications, and to alter it and redistribute it
# freely, subject to the following restrictions:
#
# 1. The origin of this software must not be misrepresented; you must not
#    claim that you wrote the original software. If you use this software
#    in a product, an acknowledgment in the product documentation would be
#    appreciated but is not required.
# 2. Altered source versions must be plainly marked as such, and must not be
#    misrepresented as being the original software.
# 3. This notice may not be removed or altered from any source distribution.
'''
As to point 2, the source has been altered a bit (by others) but it is 
not marked as such. How should it be?
'''
_ver = "turtle 1.1b- - for Python 3.1   -  4. 5. 2009"
'''
Obsolete; delete or alter (how)?

When this replaced the previous turtle.py, it was considered 'owned' by 
Gregor in that changes had to go through him. However, he became 
inactive soon after and maintenance ceased. There has been only one 
turtle-specific code change that I know of (by Ned Daily, a month ago, 
for OSX. So is turtle.py unpatchable by anyone or fair game for anyone?

A particular example: Gregor added intermediate layers to isolate turtle 
from tkinter. (He had a plan to add other backends, but I believe he 
abandoned that.) If someone wanted to reduce the layering and make the 
code easier to understand and maintain, while speeding execution, would 
that be allowed now?


B. Lets assuming that turtle.py is, at least to some degree, fair game 
for fixes and enhancements. PSF Python PyLadies (Jessica Keller, Lynn 
Root) are participating in the 2014 GNOME Outreach Program for Women 
(OPW) https://wiki.python.org/moin/OPW/2014 . One of the projects 
(bottem of that page) is Graphical Python, in particular Turtle.

A few days ago, Jessica posted
http://bugs.python.org/issue21573 Clean up turtle.py code formatting
"Lib/turtle.py has some code formatting issues. Let's clean them up to 
make the module easier to read as interns start working on it this 
summer." She want to follow cleanup with code examination, fixes, and 
enhancements.

Responding today, I cautioned that clean-up only patches, such as she 
apparently would like to start with, are not in favor. But I cannot say 
how the policy applies to her proposal. Does the 'promise' of later work 
on the code make preliminary clean-up ok?

Since she only marked the issue for 3.5, I also cautioned that 3.5-only 
cleanups would make fixing bugs in other issues harder. Is the code 
clean-up policy the same for all branches?

Test? you ask?. There are apparently no unittests for turtle. On the 
other hand, turtledemo tests the overall function of the module and of 
many (most) of the top-level turtle functions.

-- 
Terry Jan Reedy


From guido at python.org  Sat May 31 05:38:14 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 May 2014 20:38:14 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CADiSq7dkAYpvTQux9BE4jmSVj6kz9YAOswsRCxwXzDr6RX7wYA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
 <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
 <CADiSq7dkAYpvTQux9BE4jmSVj6kz9YAOswsRCxwXzDr6RX7wYA@mail.gmail.com>
Message-ID: <CAP7+vJL8iEpe0SeLwdg5pHzN==_F=43Z7Qy_ifr5x+=x72E_fQ@mail.gmail.com>

On Fri, May 30, 2014 at 7:05 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> I already have too many other things on my todo list to work this up into
> a full PEP, but the proof of concept (along with IPython's existing
> support) shows there's no *technical* barrier to adding the feature.
>

That doesn't follow. A PoC and even IPython can ignore all kinds of
*technical* issues (e.g. giving code that's valid today a different
meaning). Please drop this idea or at least take it out of this thread and
focus on what *can* and *should* be done, in particular adding warnings
with good hints about what went wrong when Python 3 encounters what appears
to be a print statement.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/23879861/attachment-0001.html>

From guido at python.org  Sat May 31 05:39:15 2014
From: guido at python.org (Guido van Rossum)
Date: Fri, 30 May 2014 20:39:15 -0700
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CADiSq7eZ14va5jQmA2uU=V-rizY-UrPCKgOnadyXZWToUONCmA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <20140530224731.1abe5ede@fsol>
 <CAP7+vJ+PGnA-As3hdsT=bjbdyZOm6dsB4Rz6ugdBvoSkq_Epog@mail.gmail.com>
 <CADiSq7eZ14va5jQmA2uU=V-rizY-UrPCKgOnadyXZWToUONCmA@mail.gmail.com>
Message-ID: <CAP7+vJK26+jshmbHEJR5gfJL76C4Xd+2sqwRarVn_i0K6mt-Vg@mail.gmail.com>

Right. The point is that the current HOWTO gives information that is not
useful for most people who are tasked with a port.


On Fri, May 30, 2014 at 6:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
> On 31 May 2014 08:42, "Guido van Rossum" <guido at python.org> wrote:
> >
> > 2to3 is poorly named. With different fixers it is a fine tool for
> converting 2-only code to 2-and-3 straddling code. Even when using six, you
> need to do things like convert print statements to print() calls with
> future import, use 'as' in except clauses, and so on.
>
> Both modernize & futurize build on lib2to3 to do the heavy lifting - they
> don't reinvent that wheel, they just change which fixes are applied by
> default and add a few more of their own.
>
> However, even if going straight to Python 3, I suspect folks would still
> be better off doing something like:
>
> * futurize --stage1 (migrates to 2.6+, but adds no new runtime
> dependencies - the output should be fairly idiomatic Python 2.6/7 code)
> * 2to3 (wholesale migration to Python 3)
>
> The trick, though, is that the granularity of that approach is at the
> process level - the entire application must be converted at once. You also
> can't start your own migration until all your dependencies are available in
> Python 3.
>
> By contrast, migration via the common subset can be incremental and
> opportunistic:
>
> * the granularity of migration is individual modules, rather than entire
> processes, so you can make a low risk gradual transition, even without a
> comprehensive regression test suite
> * you initially stay on Python 2, so you can start whenever is convenient
> for you, rather than having to wait for all your dependencies
>
> Cheers,
> Nick.
>
> >
> >
> > On Fri, May 30, 2014 at 1:47 PM, Antoine Pitrou <solipsis at pitrou.net>
> wrote:
> >>
> >> On Wed, 28 May 2014 15:26:38 -0700
> >> Glyph Lefkowitz <glyph at twistedmatrix.com> wrote:
> >> > Backport 'yield from' to allow people to use Tulip and
> Tulip-compatible code, and to facilitate the development of Tulip-friendly
> libraries and a Tulip ecosystem.  A robust Tulip ecosystem requires the
> participation of people who are not yet using Python 3.
> >>
> >> I was wondering whether you were trolling or not on this one.
> >> From a quality assurance point of view, adding major features to a
> >> bugfix branch is extremely destructive, so I'm strongly -1 on it.
> >>
> >> > Get rid of 2to3. Particularly, of any discussion of using 2to3 in the
> documentation.  More than one very experienced, well-known Python developer
> in this discussion has told me that they thought 2to3 was the blessed way
> to port their code, and it's no surprise that they think so, given that the
> first technique <https://docs.python.org/3/howto/pyporting.html> mentions
> is still 2to3.
> >>
> >> 2to3 is certainly fine if you are porting to 3.x without looking to
> >> keep your code 2.x-compatible. Until there's a better alternative, of
> >> course.
> >> So what we should do is better explain the choice (if you want to port
> >> your code to 3.x, use 2to3; if you want to maintain dual-compatible
> >> code, use six or something similar).
> >>
> >> Regards
> >>
> >> Antoine.
> >>
> >>
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido)
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
> >
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140530/4c2ff7a7/attachment.html>

From stephen at xemacs.org  Sat May 31 10:09:19 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 31 May 2014 17:09:19 +0900
Subject: [Python-Dev]  Updating  turtle.py
In-Reply-To: <lmbifr$drl$1@ger.gmane.org>
References: <lmbifr$drl$1@ger.gmane.org>
Message-ID: <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp>

Terry Reedy writes:

 > As to point 2, the source has been altered a bit (by others) but it is 
 > not marked as such. How should it be?

I would suggest adding

"""
Based on turtle 1.1b for Python 3.1 (4.5.2009) by Gregor Lingl.
This is a revised version including changes from the Python community.
Change history is available from the Python repository:
<URL: ...>.
"""

Including the URL is questionable as updates are likely to be
overlooked if the repo should ever move.  Including the first sentence
is a matter of taste.

 > '''
 > _ver = "turtle 1.1b- - for Python 3.1   -  4. 5. 2009"
 > '''
 > Obsolete; delete or alter (how)?

Delete definition and references, or replace with more generic
wording, I think.  E.g., '_ver = "turtle for Python 3"'.  It's a pain
to maintain as is.  It's not information for determining copyright, so
is covered by Gregor's permissive license.

 > When this replaced the previous turtle.py, it was considered 'owned' by 
 > Gregor in that changes had to go through him. However, he became 
 > inactive soon after and maintenance ceased. There has been only one 
 > turtle-specific code change that I know of (by Ned Daily, a month ago, 
 > for OSX. So is turtle.py unpatchable by anyone or fair game for anyone?

Legally, it's fair game.  Socially, it's a matter of project policy.

AFAICT Python policy is that someone should ask Gregor (a precedent is
the Fredrik Lundh/ElementTree case).  AIUI, there's been a five-year
span since Gregor's been active, so I would think it's basically a
matter of courtesy.  Most likely he's not interested in returning as
maintainer, or he can't be contacted with reasonable effort.  Then
it's open to anyone.

If he's interested in maintaining control but obstructive toward third
party contributions, that's messy but in the end the PSF, or its
delegates, as owners of the repo have legal control, and social
legitimacy to exercise it as seems best for the project.  If the PSF
does "go over Gregor's head" to open up the file in trunk for
modifications, the wording above might want to change to "This fork
includes changes from ...".

An alternative would be to fork into a new module with a different
name.  That can be done at any time, with the only downsides being
redundancy and potential bad will between Python and Gregor Lingl.

 > B. Lets assuming that turtle.py is, at least to some degree, fair game 
 > for fixes and enhancements. PSF Python PyLadies (Jessica Keller, Lynn 
 > Root) are participating in the 2014 GNOME Outreach Program for Women 
 > (OPW) https://wiki.python.org/moin/OPW/2014 . One of the projects 
 > (bottem of that page) is Graphical Python, in particular Turtle.

I don't think there is any issue at all here.  Legally there's no
barrier at all to working on Turtle or pushing changes anywhere.
Socially, there's no barrier to anything but pushing to release
branches (including trunk).  To the extent that OPW (GSoC, etc)
requires integration with the parent project, they can push to a
hg.python.org branch pending clarification of the maintainership
issue.

Best of all, you've identified volunteers for searching for Gregor!
<wink/>

IMHO YMMV

Steve

From stephen at xemacs.org  Sat May 31 10:18:59 2014
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Sat, 31 May 2014 17:18:59 +0900
Subject: [Python-Dev] Language Summit Follow-Up
In-Reply-To: <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
References: <632D86DB-3802-4C0D-B8AD-8C8CC87108A3@twistedmatrix.com>
 <CADiSq7ep8ipkssY6P3GjQ5N5jJ-v-41Uyd3wnTy_SwP8PvGhtA@mail.gmail.com>
 <CALGmxEKG13HEFVch1UQdZg7qQDwoZxxm2nt=54U1nZ68QUYXFA@mail.gmail.com>
Message-ID: <87wqd25pjg.fsf@uwakimon.sk.tsukuba.ac.jp>

Chris Barker writes:

 > that way. Saying that their very first easy program is:
 > 
 > print("hello world")
 > 
 > is fine

I have had similar experience on a small scale.

Also I've been teaching R recently.  The students who know Python
(Python 3, we don't have backward compatibility issues in our work)
got the concept that all "real work" in R is done by functions,
immediately.  Those who don't, have trouble with the concept that
"help" and "q" (for "quit") need parentheses to get them to work.

Unfortunately R doesn't have Python's Easter Eggs, so they get a code
dump rather than help when they type the bare names.

From martin at v.loewis.de  Sat May 31 20:05:55 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 31 May 2014 20:05:55 +0200
Subject: [Python-Dev] Updating  turtle.py
In-Reply-To: <lmbifr$drl$1@ger.gmane.org>
References: <lmbifr$drl$1@ger.gmane.org>
Message-ID: <538A1A03.4020405@v.loewis.de>

Am 31.05.14 05:32, schrieb Terry Reedy:
> I have two areas of questions about updating turtle.py. First the module
> itself, then a turtle tracker issue versus code cleanup policies.
> 
> A. Unlike most stdlib modules, turtle is copyrighted and licensed by an
> individual.
> '''
> # turtle.py: a Tkinter based turtle graphics module for Python
> # Version 1.1b - 4. 5. 2009
> # Copyright (C) 2006 - 2010  Gregor Lingl
> # email: glingl at aon.at
> '''
> I am not sure what the copyright covers other than the exact text
> contributed, with updates, by Gregor. It certainly does not cover the
> API and whatever code he copied from the previous version (unless that
> was also by him, and I have no idea how much he copied when
> reimplementing). I don't think it should cover additions made by others
> either. Should there be another line to cover these?

He should provide a contributor form, covering his past contributions.
Would you like to contact him about this?

Adding a license up-front (as you propose) is counter-productive; the
author may not agree to your specific licensing terms. If he was
unwilling to agree to the contributor form (which I doubt, knowing
him personally), the only option would be to remove the code from the
distribution.

> _ver = "turtle 1.1b- - for Python 3.1   -  4. 5. 2009"
> '''
> Obsolete; delete or alter (how)?

Delete. It's a private variable, and the true version number is
maintained by Mercurial, and not in the code itself (or else is the
version of Python it ships with).

> A particular example: Gregor added intermediate layers to isolate turtle
> from tkinter. (He had a plan to add other backends, but I believe he
> abandoned that.) If someone wanted to reduce the layering and make the
> code easier to understand and maintain, while speeding execution, would
> that be allowed now?

It would be good if there would be a new maintainer of the module. Maybe
Gregor would be willing to reactivate his contributions when asked,
maybe he would be willing to hand it over to somebody else. A maintainer
would have the ultimate say in architectural changes.

Without a maintainer, I'd rather not make architectural changes.

> Responding today, I cautioned that clean-up only patches, such as she
> apparently would like to start with, are not in favor. 

I would not say that. I recall that I asked Gregor to make a number of
style changes before he submitted the code, and eventually agreed to the
code when I thought it was "good enough". However, continuing on that
path sounds reasonable to me.

It is the mixing of clean-up patches with functional changes that is not
in favor.

> Since she only marked the issue for 3.5, I also cautioned that 3.5-only
> cleanups would make fixing bugs in other issues harder. Is the code
> clean-up policy the same for all branches?

I don't think that we should be taken hostage by merging restrictions
of the DVCS - we switched to the DVCS precisely with the promise that
merging would be easier. Given the number of bug fixes that the turtle
module has seen, I'd suggest that it is less work to restrict cleanup
to 3.5, and then deal with any forward-porting of bug fixing when it
actually happens.

Regards,
Martin


From martin at v.loewis.de  Sat May 31 20:09:30 2014
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 31 May 2014 20:09:30 +0200
Subject: [Python-Dev] Updating  turtle.py
In-Reply-To: <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <lmbifr$drl$1@ger.gmane.org>
 <87y4xi5pzk.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <538A1ADA.2060805@v.loewis.de>

Am 31.05.14 10:09, schrieb Stephen J. Turnbull:
> AFAICT Python policy is that someone should ask Gregor (a precedent is
> the Fredrik Lundh/ElementTree case).  AIUI, there's been a five-year
> span since Gregor's been active, so I would think it's basically a
> matter of courtesy.  Most likely he's not interested in returning as
> maintainer, or he can't be contacted with reasonable effort. 

I would not be so sure about this, see

http://python4kids.net/turtle.html

Regards,
Martin