From tjreedy at udel.edu Thu Nov 1 02:20:01 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 31 Oct 2012 21:20:01 -0400 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <20121031202807.13E8B25016F@webabinitio.net> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <20121031202807.13E8B25016F@webabinitio.net> Message-ID: <k6sioi$930$1@ger.gmane.org> On 10/31/2012 4:28 PM, R. David Murray wrote: >"local variable referenced before assignment" *is* > a pointer to the concept of when global variables become local...perhaps > there is a better wording, do you have a suggestion? The current wording is an exact, concise, description of the problem. Rather than tinkering with the wording, I think a more general solution might be a new HOWTO: Understanding exception messages. It could have alphabetically sorted entries for exceptions and messages that people find problematic. UnboundLocalError local variable referenced before assignment Problem: You used a local name 'x' in an expression before you assigned it a value. So the interpreter could not compute the value of the expression. Remember that assignment makes a name local unless it is declared global or nonlocal. Remedy: If you intend 'x' to refer to a glocal or nonlocal name, add the necessary global or nonlocal declaration. If you intend -- Terry Jan Reedy From rosuav at gmail.com Thu Nov 1 02:45:44 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Nov 2012 12:45:44 +1100 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <k6sioi$930$1@ger.gmane.org> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <20121031202807.13E8B25016F@webabinitio.net> <k6sioi$930$1@ger.gmane.org> Message-ID: <CAPTjJmoGujs43QKU3MdxcpYqNRQLxnbK6gPcqruvZF2qd=NLag@mail.gmail.com> On Thu, Nov 1, 2012 at 12:20 PM, Terry Reedy <tjreedy at udel.edu> wrote: > The current wording is an exact, concise, description of the problem. Rather > than tinkering with the wording, I think a more general solution might be a > new HOWTO: Understanding exception messages. It could have alphabetically > sorted entries for exceptions and messages that people find problematic. > > UnboundLocalError > > local variable referenced before assignment > ... > Remedy: If you intend 'x' to refer to a glocal or nonlocal name, add the > necessary global or nonlocal declaration. If you intend +1. Can this be tied in with help(UnboundLocalError) perhaps? At the moment, that produces a huge screed of details that aren't particularly helpful to a novice (all its methods etc), and only a one-line explanation "Local name referenced but not bound to a value.". If that could be shortened and expanded on, it'd be a logical place to point people. "You got an error you don't understand? Go to the interactive interpreter and type help(NameOfError) - that should tell you what it is." ChrisA From tjreedy at udel.edu Thu Nov 1 06:49:51 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 01 Nov 2012 01:49:51 -0400 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <k6sioi$930$1@ger.gmane.org> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <20121031202807.13E8B25016F@webabinitio.net> <k6sioi$930$1@ger.gmane.org> Message-ID: <k6t2ih$f1g$1@ger.gmane.org> On 10/31/2012 9:20 PM, Terry Reedy wrote: > On 10/31/2012 4:28 PM, R. David Murray wrote: >> "local variable referenced before assignment" *is* >> a pointer to the concept of when global variables become local...perhaps >> there is a better wording, do you have a suggestion? > > The current wording is an exact, concise, description of the problem. > Rather than tinkering with the wording, I think a more general solution > might be a new HOWTO: Understanding exception messages. It could have > alphabetically sorted entries for exceptions and messages that people > find problematic. > > UnboundLocalError > local variable referenced before assignment > > Problem: You used a local name 'x' in an expression before you assigned > it a value. So the interpreter could not compute the value of the > expression. Remember that assignment makes a name local unless it is > declared global or nonlocal. > > Remedy: If you intend 'x' to refer to a glocal or nonlocal name, add the > necessary global or nonlocal declaration. If you intend <left out> 'x' to be local, rearrange your code to give it a value before you use it. -- Terry Jan Reedy From rob.cliffe at btinternet.com Thu Nov 1 09:36:51 2012 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 01 Nov 2012 08:36:51 +0000 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <CAPTjJmoGujs43QKU3MdxcpYqNRQLxnbK6gPcqruvZF2qd=NLag@mail.gmail.com> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <20121031202807.13E8B25016F@webabinitio.net> <k6sioi$930$1@ger.gmane.org> <CAPTjJmoGujs43QKU3MdxcpYqNRQLxnbK6gPcqruvZF2qd=NLag@mail.gmail.com> Message-ID: <509234A3.8030507@btinternet.com> On 01/11/2012 01:45, Chris Angelico wrote: > On Thu, Nov 1, 2012 at 12:20 PM, Terry Reedy <tjreedy at udel.edu> wrote: >> The current wording is an exact, concise, description of the problem. Rather >> than tinkering with the wording, I think a more general solution might be a >> new HOWTO: Understanding exception messages. It could have alphabetically >> sorted entries for exceptions and messages that people find problematic. >> >> UnboundLocalError >> >> local variable referenced before assignment >> ... >> Remedy: If you intend 'x' to refer to a glocal or nonlocal name, add the >> necessary global or nonlocal declaration. If you intend > +1. Can this be tied in with help(UnboundLocalError) perhaps? At the > moment, that produces a huge screed of details that aren't > particularly helpful to a novice (all its methods etc), and only a > one-line explanation "Local name referenced but not bound to a > value.". If that could be shortened and expanded on, it'd be a logical > place to point people. "You got an error you don't understand? Go to > the interactive interpreter and type help(NameOfError) - that should > tell you what it is." > > ChrisA An excellent idea IMO. +1 Rob Cliffe > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com > > From andrew.svetlov at gmail.com Fri Nov 2 15:46:58 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 2 Nov 2012 16:46:58 +0200 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu Message-ID: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> Hi. There are issue for subject: http://bugs.python.org/issue1207589 It has patches, implemented well and committed. I've pushed it to versions 2.7, 3.2, 3.3 and 3.4. My thoughts are: the issue is not brand new subject but improvement for current IDLE functionality. Added code cannot break any existing program/library I hope, it's related to humans who use IDLE only. It is desirable feature and probably IDLE users will be ok with new items in context menu even they are still 2.7 users. There are another opinion: it is new feature, not a bug, and the patch should be applied to 3.4 only. If you look on discussion for issue (http://bugs.python.org/issue1207589) you can see we cannot make decision, votes are split. I want to raise the question on this mailing list (as Terry Reedy suggested) to ask for community opinion. Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121102/9027db22/attachment.html> From ncoghlan at gmail.com Fri Nov 2 16:16:12 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Nov 2012 01:16:12 +1000 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> Message-ID: <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> On Sat, Nov 3, 2012 at 12:46 AM, Andrew Svetlov <andrew.svetlov at gmail.com> wrote: > Hi. There are issue for subject: http://bugs.python.org/issue1207589 > It has patches, implemented well and committed. > I've pushed it to versions 2.7, 3.2, 3.3 and 3.4. > > My thoughts are: the issue is not brand new subject but improvement for > current IDLE functionality. > Added code cannot break any existing program/library I hope, it's related to > humans who use IDLE only. > It is desirable feature and probably IDLE users will be ok with new items in > context menu even they are still 2.7 users. > > There are another opinion: it is new feature, not a bug, and the patch > should be applied to 3.4 only. > > If you look on discussion for issue (http://bugs.python.org/issue1207589) > you can see we cannot make decision, votes are split. > > I want to raise the question on this mailing list (as Terry Reedy suggested) > to ask for community opinion. The status quo is that IDLE is covered by the "no new features in maintenance releases" rule along with the rest of the standard library. Now, it may be *unreasonable* that this is so, and changing it would help improve IDLE as a tool. The way to resolve a proposal like that is to put it forward as a PEP, and explain the rationale for treating IDLE differently. A PEP also makes it possible to state exactly which modules are being proposed for exemption from the no-new-features rule. The fact that many (most?) distro packaging teams split idle out into a separate package from their base python installation may be a point in such a proposal's favour (e.g. in Fedora, there's a separate "python-tools" package, so the main python package doesn't need to depend on tkinter). Until such a PEP has been submitted and approved, any feature additions on maintenance branches should be reverted. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rdmurray at bitdance.com Fri Nov 2 16:34:58 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 02 Nov 2012 11:34:58 -0400 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> Message-ID: <20121102153459.0107B250174@webabinitio.net> On Sat, 03 Nov 2012 01:16:12 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Sat, Nov 3, 2012 at 12:46 AM, Andrew Svetlov > <andrew.svetlov at gmail.com> wrote: > > Hi. There are issue for subject: http://bugs.python.org/issue1207589 [...] > The status quo is that IDLE is covered by the "no new features in > maintenance releases" rule along with the rest of the standard > library. Now, it may be *unreasonable* that this is so, and changing > it would help improve IDLE as a tool. The way to resolve a proposal > like that is to put it forward as a PEP, and explain the rationale for > treating IDLE differently. A PEP also makes it possible to state > exactly which modules are being proposed for exemption from the > no-new-features rule. In this particular instance we are not looking to exempt the entire module, just this changeset (because it does not change callable code). Exempting IDLE in general is an interesting idea, but is not the immediate question. --David From status at bugs.python.org Fri Nov 2 18:07:26 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 2 Nov 2012 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20121102170726.DC5181CA74@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-10-26 - 2012-11-02) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3804 (-19) closed 24361 (+77) total 28165 (+58) Open issues with patches: 1714 Issues opened (29) ================== #9742: Python 2.7: math module fails to build on Solaris 9 http://bugs.python.org/issue9742 reopened by mark.dickinson #15478: UnicodeDecodeError on OSError on Windows with undecodable (byt http://bugs.python.org/issue15478 reopened by skrah #16218: Python launcher does not support non ascii characters http://bugs.python.org/issue16218 reopened by jcea #16333: Trailing whitespace in json dump when using indent http://bugs.python.org/issue16333 opened by zach.mathew #16334: Faster unicode-escape and raw-unicode-escape codecs http://bugs.python.org/issue16334 opened by serhiy.storchaka #16335: Integer overflow in unicode-escape decoder http://bugs.python.org/issue16335 opened by serhiy.storchaka #16336: Check input in surrogatepass error handler http://bugs.python.org/issue16336 opened by serhiy.storchaka #16338: pysnmp/asyncore - timeout ineffective? http://bugs.python.org/issue16338 opened by Trenton.Craig #16339: Document "exec(stmt, global_dict, local_dict)" form in Python http://bugs.python.org/issue16339 opened by mark.dickinson #16346: readline problem http://bugs.python.org/issue16346 opened by mathieu37 #16347: configure.ac patch http://bugs.python.org/issue16347 opened by cavallo71 #16349: Document whether it's safe to use bytes for struct format stri http://bugs.python.org/issue16349 opened by takluyver #16350: zlib.Decompress.decompress() after EOF discards existing value http://bugs.python.org/issue16350 opened by nadeem.vawda #16353: add function to os module for getting path to default shell http://bugs.python.org/issue16353 opened by chris.jerdonek #16354: Remember python version choice on docs.python.org http://bugs.python.org/issue16354 opened by wichert #16355: inspect.getcomments() does not work in the interactive shell http://bugs.python.org/issue16355 opened by marco.buttu #16357: SSLSocket created from SSLContext.wrap_socket doesn't include http://bugs.python.org/issue16357 opened by mcjeff #16360: argparse: comma in metavar causes assertion failure when forma http://bugs.python.org/issue16360 opened by bgamari #16361: HTTPS/TLS Problem in Python 3.3 http://bugs.python.org/issue16361 opened by pventura #16367: io.FileIO.readall() is not 64-bit safe on Windows http://bugs.python.org/issue16367 opened by haypo #16376: wrong type for wintypes.BYTE http://bugs.python.org/issue16376 opened by techtonik #16378: venv.EnvBuilder docstring inconsistencies http://bugs.python.org/issue16378 opened by bfroehle #16379: SQLite error code not exposed to python http://bugs.python.org/issue16379 opened by torsten #16381: Introduce option to force the interpreter to exit upon MemoryE http://bugs.python.org/issue16381 opened by ctheune #16382: Better warnings exception for bad category http://bugs.python.org/issue16382 opened by pelson #16383: Python 3.3 Permission Error with User Library on Windows http://bugs.python.org/issue16383 opened by jimp02 #16384: import.c doesn't handle EOFError from PyMarshal_Read* http://bugs.python.org/issue16384 opened by syeberman #16385: evaluating literal dict with repeated keys gives no warnings/e http://bugs.python.org/issue16385 opened by Albert.Ferras #16386: imp.find_module does not specify registry key it searches on w http://bugs.python.org/issue16386 opened by dhgmgn Most recent 15 issues with no replies (15) ========================================== #16386: imp.find_module does not specify registry key it searches on w http://bugs.python.org/issue16386 #16384: import.c doesn't handle EOFError from PyMarshal_Read* http://bugs.python.org/issue16384 #16382: Better warnings exception for bad category http://bugs.python.org/issue16382 #16379: SQLite error code not exposed to python http://bugs.python.org/issue16379 #16376: wrong type for wintypes.BYTE http://bugs.python.org/issue16376 #16360: argparse: comma in metavar causes assertion failure when forma http://bugs.python.org/issue16360 #16353: add function to os module for getting path to default shell http://bugs.python.org/issue16353 #16347: configure.ac patch http://bugs.python.org/issue16347 #16346: readline problem http://bugs.python.org/issue16346 #16339: Document "exec(stmt, global_dict, local_dict)" form in Python http://bugs.python.org/issue16339 #16334: Faster unicode-escape and raw-unicode-escape codecs http://bugs.python.org/issue16334 #16321: Move eq.h out of stringlib http://bugs.python.org/issue16321 #16320: Establish order in bytes/string dependencies http://bugs.python.org/issue16320 #16287: Sporadic test_utime() failures on Solaris http://bugs.python.org/issue16287 #16283: ctypes.util.find_library does not find all DLLs anymore http://bugs.python.org/issue16283 Most recent 15 issues waiting for review (15) ============================================= #16386: imp.find_module does not specify registry key it searches on w http://bugs.python.org/issue16386 #16382: Better warnings exception for bad category http://bugs.python.org/issue16382 #16381: Introduce option to force the interpreter to exit upon MemoryE http://bugs.python.org/issue16381 #16367: io.FileIO.readall() is not 64-bit safe on Windows http://bugs.python.org/issue16367 #16357: SSLSocket created from SSLContext.wrap_socket doesn't include http://bugs.python.org/issue16357 #16350: zlib.Decompress.decompress() after EOF discards existing value http://bugs.python.org/issue16350 #16336: Check input in surrogatepass error handler http://bugs.python.org/issue16336 #16335: Integer overflow in unicode-escape decoder http://bugs.python.org/issue16335 #16334: Faster unicode-escape and raw-unicode-escape codecs http://bugs.python.org/issue16334 #16333: Trailing whitespace in json dump when using indent http://bugs.python.org/issue16333 #16329: mimetypes does not support webm type http://bugs.python.org/issue16329 #16327: subprocess.Popen leaks file descriptors on os.fork() failure http://bugs.python.org/issue16327 #16326: distutils build_ext fails to set library_dirs in 2.7.2 on Linu http://bugs.python.org/issue16326 #16324: MIMEText __init__ does not support Charset instance http://bugs.python.org/issue16324 #16323: Wrong C API documentation for locale encoding http://bugs.python.org/issue16323 Top 10 most discussed issues (10) ================================= #16248: Security bug in tkinter allows for untrusted, arbitrary code e http://bugs.python.org/issue16248 19 msgs #16381: Introduce option to force the interpreter to exit upon MemoryE http://bugs.python.org/issue16381 14 msgs #15478: UnicodeDecodeError on OSError on Windows with undecodable (byt http://bugs.python.org/issue15478 12 msgs #16312: New command line option supported by unittest.main() for runni http://bugs.python.org/issue16312 10 msgs #16355: inspect.getcomments() does not work in the interactive shell http://bugs.python.org/issue16355 9 msgs #16385: evaluating literal dict with repeated keys gives no warnings/e http://bugs.python.org/issue16385 9 msgs #9742: Python 2.7: math module fails to build on Solaris 9 http://bugs.python.org/issue9742 8 msgs #16218: Python launcher does not support non ascii characters http://bugs.python.org/issue16218 8 msgs #16350: zlib.Decompress.decompress() after EOF discards existing value http://bugs.python.org/issue16350 6 msgs #2005: posixmodule expects sizeof(pid_t/gid_t/uid_t) <= sizeof(long) http://bugs.python.org/issue2005 5 msgs Issues closed (71) ================== #3636: Managing dual 2.x and 3.0 installations on Windows http://bugs.python.org/issue3636 closed by eric.araujo #5210: zlib does not indicate end of compressed stream properly http://bugs.python.org/issue5210 closed by nadeem.vawda #7098: g formatting for decimal types should always strip trailing ze http://bugs.python.org/issue7098 closed by mark.dickinson #8996: Add a default role to allow writing bare `len` instead of :fun http://bugs.python.org/issue8996 closed by eric.araujo #9017: doctest option flag to enable/disable some chunk of doctests? http://bugs.python.org/issue9017 closed by ezio.melotti #9530: integer undefined behaviors http://bugs.python.org/issue9530 closed by mark.dickinson #10189: SyntaxError: no binding for nonlocal doesn't contain a useful http://bugs.python.org/issue10189 closed by python-dev #10836: urllib.request.urlretrieve calls URLError with 3 args http://bugs.python.org/issue10836 closed by orsenthil #12890: cgitb displays <p> tags when executed in text mode http://bugs.python.org/issue12890 closed by r.david.murray #13701: Remove Decimal Python 2.3 Compatibility http://bugs.python.org/issue13701 closed by mark.dickinson #14570: Document json "sort_keys" parameter properly http://bugs.python.org/issue14570 closed by asvetlov #14616: subprocess docs should mention pipes.quote/shlex.quote http://bugs.python.org/issue14616 closed by asvetlov #14625: Faster utf-32 decoder http://bugs.python.org/issue14625 closed by python-dev #14675: make distutils.ccompiler.CCompiler an abstract class http://bugs.python.org/issue14675 closed by eric.araujo #14893: Tutorial: Add function annotation example to function tutorial http://bugs.python.org/issue14893 closed by asvetlov #14897: struct.pack raises unexpected error message http://bugs.python.org/issue14897 closed by petri.lehtinen #14900: cProfile does not take its result headers as sort arguments http://bugs.python.org/issue14900 closed by asvetlov #15148: shutil.which() docstring could be clearer http://bugs.python.org/issue15148 closed by r.david.murray #15746: test_winsound bombing out on 2003 buildslave http://bugs.python.org/issue15746 closed by trent #15889: regrtest --start option raises AttributeError in many scenario http://bugs.python.org/issue15889 closed by r.david.murray #15957: README.txt points to broken "contributing" url in python wiki http://bugs.python.org/issue15957 closed by eric.araujo #15961: Missing return value in ``system_message`` http://bugs.python.org/issue15961 closed by eric.araujo #16086: tp_flags: Undefined behaviour with 32 bits long http://bugs.python.org/issue16086 closed by haypo #16107: distutils2.version doesn't str() "1.0.post1" correctly http://bugs.python.org/issue16107 closed by eric.araujo #16197: Several small errors in winreg documentation http://bugs.python.org/issue16197 closed by brian.curtin #16228: JSON crashes during encoding resized lists http://bugs.python.org/issue16228 closed by pitrou #16230: select.select crashes on resized lists http://bugs.python.org/issue16230 closed by pitrou #16239: PEP8 arithmetic operator examples http://bugs.python.org/issue16239 closed by gvanrossum #16243: Add example for inspect module doc http://bugs.python.org/issue16243 closed by asvetlov #16250: URLError invoked with reason as filename http://bugs.python.org/issue16250 closed by orsenthil #16268: dir(closure) does not find __dir__ http://bugs.python.org/issue16268 closed by benjamin.peterson #16271: weird dual behavior with changing __qualname__; different valu http://bugs.python.org/issue16271 closed by python-dev #16300: Windows raises ValueError for file:///file/not/exists instead http://bugs.python.org/issue16300 closed by orsenthil #16301: localhost() and thishost() in urllib/request.py http://bugs.python.org/issue16301 closed by orsenthil #16307: multiprocess.pool.map_async callables not working http://bugs.python.org/issue16307 closed by hynek #16316: Support xz compression in mimetypes module http://bugs.python.org/issue16316 closed by nadeem.vawda #16325: PEP??8 refers to reference to PEP 8 http://bugs.python.org/issue16325 closed by ezio.melotti #16330: Use surrogate-related macros http://bugs.python.org/issue16330 closed by serhiy.storchaka #16331: Add a version switcher to python docs site http://bugs.python.org/issue16331 closed by loewis #16332: Use new exception handling syntax in Subprocess documentation http://bugs.python.org/issue16332 closed by ezio.melotti #16337: typo in documentation regarding "Bytearray Objects", chapter 4 http://bugs.python.org/issue16337 closed by ezio.melotti #16340: Decoding error due to byte-compiling venv scripts at install t http://bugs.python.org/issue16340 closed by python-dev #16341: In examples, "except:" should use new syntax http://bugs.python.org/issue16341 closed by asvetlov #16342: setup.py not compiling with NDEBUG in non-debug builds http://bugs.python.org/issue16342 closed by brett.cannon #16343: PyUnicode_FromFormatV() doesn't support utf-8 text http://bugs.python.org/issue16343 closed by chris.jerdonek #16344: Traceback Internationalization Proposal http://bugs.python.org/issue16344 closed by rhettinger #16345: dict.fromkeys() assumes 'self' is empty http://bugs.python.org/issue16345 closed by python-dev #16348: Decimal.remainder_near documentation incorrect. http://bugs.python.org/issue16348 closed by mark.dickinson #16351: Add a function to get GC statistics http://bugs.python.org/issue16351 closed by pitrou #16352: Error call http://bugs.python.org/issue16352 closed by r.david.murray #16356: cjson dose not decode \/ properly http://bugs.python.org/issue16356 closed by christian.heimes #16358: Enhancement for mmap_read: Consistency with standard file read http://bugs.python.org/issue16358 closed by r.david.murray #16359: can't print figures 08 and 09 http://bugs.python.org/issue16359 closed by ezio.melotti #16362: _LegalCharsPatt in cookies.py includes illegal characters http://bugs.python.org/issue16362 closed by r.david.murray #16363: super cannot invoke descriptors http://bugs.python.org/issue16363 closed by r.david.murray #16364: datetime.strftime and locale.getdefaultlocale conflict on Wind http://bugs.python.org/issue16364 closed by jwfang #16365: IDLE for Windows 8 http://bugs.python.org/issue16365 closed by loewis #16366: _handleError not very informative http://bugs.python.org/issue16366 closed by python-dev #16368: error when logging message http://bugs.python.org/issue16368 closed by r.david.murray #16369: Global PyTypeObjects not initialized with PyType_Ready(...) http://bugs.python.org/issue16369 closed by python-dev #16370: Mention Py_SetProgramName in example for very high level embed http://bugs.python.org/issue16370 closed by asvetlov #16371: typo in ctypes http://bugs.python.org/issue16371 closed by asvetlov #16372: Initialization strange behavior http://bugs.python.org/issue16372 closed by mark.dickinson #16373: Recursion error comparing set() and collections.Set instances http://bugs.python.org/issue16373 closed by asvetlov #16374: ConfigParser: Passing a semicolon as a value http://bugs.python.org/issue16374 closed by lukasz.langa #16375: Warning in Parser/grammar1.c http://bugs.python.org/issue16375 closed by benjamin.peterson #16377: Fix bisect unittest http://bugs.python.org/issue16377 closed by asvetlov #16380: true/proper subset http://bugs.python.org/issue16380 closed by asvetlov #16387: test_cmd_line_script failures http://bugs.python.org/issue16387 closed by skrah #16388: Urllib screws up capitalization in "User-Agent" HTTP Header http://bugs.python.org/issue16388 closed by r.david.murray #1207589: IDLE: Right Click Context Menu http://bugs.python.org/issue1207589 closed by asvetlov From tjreedy at udel.edu Fri Nov 2 18:29:47 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 02 Nov 2012 13:29:47 -0400 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> Message-ID: <k70vuu$qi7$1@ger.gmane.org> On 11/2/2012 11:16 AM, Nick Coghlan wrote: > On Sat, Nov 3, 2012 at 12:46 AM, Andrew Svetlov > <andrew.svetlov at gmail.com> wrote: >> Hi. There are issue for subject: http://bugs.python.org/issue1207589 >> It has patches, implemented well and committed. >> I've pushed it to versions 2.7, 3.2, 3.3 and 3.4. >> >> My thoughts are: the issue is not brand new subject but improvement for >> current IDLE functionality. >> Added code cannot break any existing program/library I hope, it's related to >> humans who use IDLE only. >> It is desirable feature and probably IDLE users will be ok with new items in >> context menu even they are still 2.7 users. >> >> There are another opinion: it is new feature, not a bug, and the patch >> should be applied to 3.4 only. >> >> If you look on discussion for issue (http://bugs.python.org/issue1207589) >> you can see we cannot make decision, votes are split. >> >> I want to raise the question on this mailing list (as Terry Reedy suggested) >> to ask for community opinion. > > The status quo is that IDLE is covered by the "no new features in > maintenance releases" rule along with the rest of the standard > library. That may be your opinion, but I disagree that the situation is so clear. In what PEP (or other document) is the above stated. IDLE has previously been treated here as an exception to the normal rules. Two years ago, we debated *dropping* IDLE from the distribution -- beginning with the next release. We would not have had such a discussion for any normal library module as simply removing a module before Py 4 is against policy. > Now, it may be *unreasonable* that this is so, and changing > it would help improve IDLE as a tool. If it is 'unreasonable', then perhaps it is not so. > The way to resolve a proposal > like that is to put it forward as a PEP, and explain the rationale for > treating IDLE differently. A PEP seems like overkill to me. The matter is a rule clarification, not an enhancement proposal. The rationale is simple. 1. The reason for the no-new-features rule does not apply to user interface features, certainly not to the right-click context menu. 2. Users will prefer consistency, especially in something like right-click behavior (or search/replace boxes, etc). 3. It is often unclear whether a particular change is a fix or an enhancement. I would suggest that a) in many cases neither word really applies and b) in such cases, given 1) and 2) above, it is not worth the effort to force fit a change into either category. For instance, the existence of a right-click context menu is not mentioned in the sketchy Library manual chapter for IDLE. So there can neither be consistency nor inconsistency between current behavior and the non-existent doc entry. Hence, there is no objective standard for classifying a change, and hence there is disagreement. Since http://bugs.python.org/issue1207589 brings IDLE in line with external standards, I consider it a bug fix. Actually, I consider it *both* a bug fix *and* and enhancement, but a bug fix for the purpose of deciding where to apply it (given that someone, Andrew, was willing to go though the effort of applying it everywhere). Even features that are documented as to existence are not specified. The following is typical. "Find... Open a search dialog box with many options" There have been or still are proposed changes to Find or Replace that could be classified either way, depending on whether, in the absence of any specification, one is inclined to make 'bug fix' or 'enhancement' the default choice. > A PEP also makes it possible to state > exactly which modules are being proposed for exemption from the > no-new-features rule. Since the proposal two years ago was to delete the entire idlelib/* package, I would start with the entire package. If and when possibly generally useful modules (perhaps idlelib/rpc.py -- remote procedure calls) are documented for general use, they should be versioned. But then they should perhaps be moved elsewhere. > The fact that many (most?) distro packaging teams split idle out into > a separate package from their base python installation may be a point > in such a proposal's favour (e.g. in Fedora, there's a separate > "python-tools" package, so the main python package doesn't need to > depend on tkinter). > > Until such a PEP has been submitted and approved, any feature > additions on maintenance branches should be reverted. And who shall decide what change is what? -- Terry Jan Reedy From nad at acm.org Fri Nov 2 18:31:47 2012 From: nad at acm.org (Ned Deily) Date: Fri, 02 Nov 2012 10:31:47 -0700 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> <20121102153459.0107B250174@webabinitio.net> Message-ID: <nad-E3721F.10314602112012@news.gmane.org> In article <20121102153459.0107B250174 at webabinitio.net>, "R. David Murray" <rdmurray at bitdance.com> wrote: > On Sat, 03 Nov 2012 01:16:12 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote: > > On Sat, Nov 3, 2012 at 12:46 AM, Andrew Svetlov > > <andrew.svetlov at gmail.com> wrote: > > > Hi. There are issue for subject: http://bugs.python.org/issue1207589 > [...] > > The status quo is that IDLE is covered by the "no new features in > > maintenance releases" rule along with the rest of the standard > > library. Now, it may be *unreasonable* that this is so, and changing > > it would help improve IDLE as a tool. The way to resolve a proposal > > like that is to put it forward as a PEP, and explain the rationale for > > treating IDLE differently. A PEP also makes it possible to state > > exactly which modules are being proposed for exemption from the > > no-new-features rule. > In this particular instance we are not looking to exempt the entire > module, just this changeset (because it does not change callable code). > > Exempting IDLE in general is an interesting idea, but is not the immediate > question. Also, as Roger Serwy has pointed out in the issue, the change also can affect third-party IDLE extensions but he thinks the backport is still worthwhile. Since the discussion has progressed primarily on the issue tracker and the python-dev interest is probably limited, I would suggest keeping the discussion over there rather than both here and there. -- Ned Deily, nad at acm.org From nad at acm.org Fri Nov 2 18:47:11 2012 From: nad at acm.org (Ned Deily) Date: Fri, 02 Nov 2012 10:47:11 -0700 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> <k70vuu$qi7$1@ger.gmane.org> Message-ID: <nad-FF4D5B.10471102112012@news.gmane.org> In article <k70vuu$qi7$1 at ger.gmane.org>, Terry Reedy <tjreedy at udel.edu> wrote: > For instance, the existence of a right-click context menu is not > mentioned in the sketchy Library manual chapter for IDLE. Actually, it is documented as of a few weeks ago (see the changes for Issue10405). And the issue under discussion here updated both the manual and the IDLE help file. http://docs.python.org/2/library/idle.html#edit-context-menu -- Ned Deily, nad at acm.org From pjenvey at underboss.org Fri Nov 2 19:16:25 2012 From: pjenvey at underboss.org (Philip Jenvey) Date: Fri, 2 Nov 2012 11:16:25 -0700 Subject: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) In-Reply-To: <CAP1=2W74+-dqx8=SOD0QBThRuGY_B9ed4wmDLq_mo0x3_1W1eg@mail.gmail.com> References: <CAP1=2W74+-dqx8=SOD0QBThRuGY_B9ed4wmDLq_mo0x3_1W1eg@mail.gmail.com> Message-ID: <3E792B3D-BEA9-4E9F-9102-7F18FD3771E4@underboss.org> On Oct 26, 2012, at 12:14 PM, Brett Cannon wrote: > > Worst benchmark is nosite_startup, best is telco. The benchmarks people might want to analyze (i.e. more than 20% slower in Python 3.3) are mako_v2, threaded_count, normal_startup, iterative_count, pathlib, formatted_logging, and simple_logging. > > ### mako_v2 ### > Min: 0.083660 -> 0.243323: 2.91x slower > Avg: 0.084634 -> 0.247875: 2.93x slower > Significant (t=-821.55) > Stddev: 0.00193 -> 0.00400: 2.0737x larger > Timeline: b'http://tinyurl.com/98n9fab' So Mike Bayer and I narrowed down mako_v2's slowness to use of an inline re This: http://www.makotemplates.org/trac/changeset/c1468b12f115ac9e469150ce24ea042aeae5e270 brings it down to around: ### mako_v2 ### Min: 0.087608 -> 0.066748: 1.31x faster Avg: 0.091348 -> 0.071224: 1.28x faster Significant (t=26.10) Stddev: 0.00312 -> 0.00447: 1.4340x larger Timeline: http://tinyurl.com/as2zedo The culprit is the lru_cache on re._compile_typed. Notice functools' numbers from the profiler: http://paste.ofcode.org/yZRKnJfTsHesFR8hMWfc7f Mike also noticed that the mako fix above does nothing to 2.7's numbers. -- Philip Jenvey From brett at python.org Fri Nov 2 19:42:55 2012 From: brett at python.org (Brett Cannon) Date: Fri, 2 Nov 2012 14:42:55 -0400 Subject: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) In-Reply-To: <3E792B3D-BEA9-4E9F-9102-7F18FD3771E4@underboss.org> References: <CAP1=2W74+-dqx8=SOD0QBThRuGY_B9ed4wmDLq_mo0x3_1W1eg@mail.gmail.com> <3E792B3D-BEA9-4E9F-9102-7F18FD3771E4@underboss.org> Message-ID: <CAP1=2W4VMMDCmQk-7VKEyK3zFCq9R8bsK3pv6hV69sCLA0MQsA@mail.gmail.com> Issue filed for the performance issue: http://bugs.python.org/issue16390 With that change and running on tip of Mako on my laptop now reports 1.25x slower which is *much* better than it was. This performance issue might also explain why all of the regex compilation benchmarks are worse under Python 3.3 by a decent margin. On Fri, Nov 2, 2012 at 2:16 PM, Philip Jenvey <pjenvey at underboss.org> wrote: > lru_cache on re._compile_typed -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121102/40045126/attachment.html> From ericsnowcurrently at gmail.com Fri Nov 2 21:05:51 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 2 Nov 2012 14:05:51 -0600 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20121102170726.DC5181CA74@psf.upfronthosting.co.za> References: <20121102170726.DC5181CA74@psf.upfronthosting.co.za> Message-ID: <CALFfu7A+dx5kVxmW-c-5d-egQeqad_ihUE+-JgaEpBLz=hm3tQ@mail.gmail.com> On Fri, Nov 2, 2012 at 11:07 AM, Python tracker <status at bugs.python.org> wrote: > > ACTIVITY SUMMARY (2012-10-26 - 2012-11-02) > Python tracker at http://bugs.python.org/ > > To view or respond to any of the issues listed below, click on the issue. > Do NOT respond to this message. > > Issues counts and deltas: > open 3804 (-19) wow! > closed 24361 (+77) > total 28165 (+58) > > Open issues with patches: 1714 From andrew.svetlov at gmail.com Fri Nov 2 21:24:38 2012 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Fri, 2 Nov 2012 22:24:38 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <CALFfu7A+dx5kVxmW-c-5d-egQeqad_ihUE+-JgaEpBLz=hm3tQ@mail.gmail.com> References: <20121102170726.DC5181CA74@psf.upfronthosting.co.za> <CALFfu7A+dx5kVxmW-c-5d-egQeqad_ihUE+-JgaEpBLz=hm3tQ@mail.gmail.com> Message-ID: <CAL3CFcVcfLqo2H=mGSoCVJ2pmuNdzKDt5Dj7zz9UM-yD1gN63A@mail.gmail.com> On Fri, Nov 2, 2012 at 10:05 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > > On Fri, Nov 2, 2012 at 11:07 AM, Python tracker <status at bugs.python.org> wrote: > > > > ACTIVITY SUMMARY (2012-10-26 - 2012-11-02) > > Python tracker at http://bugs.python.org/ > > > > To view or respond to any of the issues listed below, click on the issue. > > Do NOT respond to this message. > > > > Issues counts and deltas: > > open 3804 (-19) > > wow! > Aha! > > > closed 24361 (+77) > > total 28165 (+58) > > > > Open issues with patches: 1714 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121102/07b3a71d/attachment.html> From ncoghlan at gmail.com Sat Nov 3 03:04:57 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Nov 2012 12:04:57 +1000 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <k70vuu$qi7$1@ger.gmane.org> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> <k70vuu$qi7$1@ger.gmane.org> Message-ID: <CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com> On Sat, Nov 3, 2012 at 3:29 AM, Terry Reedy <tjreedy at udel.edu> wrote: >> The way to resolve a proposal >> >> like that is to put it forward as a PEP, and explain the rationale for >> treating IDLE differently. > > > A PEP seems like overkill to me. The matter is a rule clarification, not an > enhancement proposal. The rationale is simple. If you don't want to run the risk of needing to rehash this discussion every time an IDLE feature addition is made in maintenance branches, write the rules down in a PEP. I don't actually care all that much about IDLE (except in an abstract "I know other people care about it" sense), but I *do* care about developers being able to understand the rules about whether or not a feature addition is permissible in a maintenance branch. That kind of thing *cannot* be left to "oh, this change is OK, this change isn't, but you needed to be around for this discussion that happened 5 years ago in order to understand why", because it *wastes everybody's time* explaining the unwritten rules to new developers. If there is a part of the code base where new features are permitted in maintenance branches, write them down, get agreement on them, and if anyone complains "but that's a new feature on a maintenance branch", point them to the PEP that says its OK. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Sat Nov 3 04:10:23 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 02 Nov 2012 23:10:23 -0400 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> <k70vuu$qi7$1@ger.gmane.org> <CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com> Message-ID: <50948B1F.1030000@udel.edu> On 11/2/2012 10:04 PM, Nick Coghlan wrote: > On Sat, Nov 3, 2012 at 3:29 AM, Terry Reedy <tjreedy at udel.edu> wrote: >>> The way to resolve a proposal >>> >>> like that is to put it forward as a PEP, and explain the rationale for >>> treating IDLE differently. >> >> >> A PEP seems like overkill to me. The matter is a rule clarification, not an >> enhancement proposal. The rationale is simple. > > If you don't want to run the risk of needing to rehash this discussion > every time an IDLE feature addition is made in maintenance branches, > write the rules down in a PEP. [snip reasons] OK, I am convinced an info PEP would be a good idea. -- Terry From ncoghlan at gmail.com Sat Nov 3 10:28:02 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Nov 2012 19:28:02 +1000 Subject: [Python-Dev] Cut/Copy/Paste items in IDLE right click context menu In-Reply-To: <50948B1F.1030000@udel.edu> References: <CAL3CFcX6d1OJeit9iBBzGTTvo0-gvOWKB=wLTDmPNFZ4FOjqkQ@mail.gmail.com> <CADiSq7dX3NbOvMuMnYFrMMwOPRn+Sf_rmsUO0L9LnovhTQkyyw@mail.gmail.com> <k70vuu$qi7$1@ger.gmane.org> <CADiSq7eEN3nSxbrouZ5VgyQC7cKwgTjCM78jQz26NxKjU7ipoA@mail.gmail.com> <50948B1F.1030000@udel.edu> Message-ID: <CADiSq7fYXqXO+QWG6eJdxkiCtJLQDdWVJk6e11pv2PssyNBkUA@mail.gmail.com> On Sat, Nov 3, 2012 at 1:10 PM, Terry Reedy <tjreedy at udel.edu> wrote: > [snip reasons] > OK, I am convinced an info PEP would be a good idea. Unless anyone objects, I'm happy to be BDFL-delegate for such a PEP. I mainly want to ensure we clearly fence off "IDLE-the-application", with (in effect) a 6 month release cycle synchronised across versions, from the rest of the standard library. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 3 10:38:55 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Nov 2012 19:38:55 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #15814: Use hash function that is compatible with the equality In-Reply-To: <3XtV7q58c6zRMy@mail.python.org> References: <3XtV7q58c6zRMy@mail.python.org> Message-ID: <CADiSq7fSaz6uxQeoQeyi0aukH3kLifUAmSpd1ZRJe2KedXD5DQ@mail.gmail.com> On Sat, Nov 3, 2012 at 3:07 AM, stefan.krah <python-checkins at python.org> wrote: > + # equality-hash invariant > + x = ndarray(list(range(12)), shape=[12], format='B') > + a = memoryview(nd) > + > + y = ndarray(list(range(12)), shape=[12], format='b') > + b = memoryview(nd) > + > + z = ndarray(list(bytes(chr(x), 'latin-1') for x in range(12)), > + shape=[12], format='c') > + c = memoryview(nd) > + > + if (a == b): > + self.assertEqual(hash(a), hash(b)) > + > + if (a == c): > + self.assertEqual(hash(a), hash(c)) > + > + if (b == c): > + self.assertEqual(hash(b), hash(c)) These checks could do with a comment explaining why the if statements are needed (I'm assuming something to do with memory order). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan at bytereef.org Sat Nov 3 10:59:23 2012 From: stefan at bytereef.org (Stefan Krah) Date: Sat, 3 Nov 2012 10:59:23 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #15814: Use hash function that is compatible with the equality In-Reply-To: <CADiSq7fSaz6uxQeoQeyi0aukH3kLifUAmSpd1ZRJe2KedXD5DQ@mail.gmail.com> References: <3XtV7q58c6zRMy@mail.python.org> <CADiSq7fSaz6uxQeoQeyi0aukH3kLifUAmSpd1ZRJe2KedXD5DQ@mail.gmail.com> Message-ID: <20121103095923.GA30827@sleipnir.bytereef.org> Nick Coghlan <ncoghlan at gmail.com> wrote: > > + if (b == c): > > + self.assertEqual(hash(b), hash(c)) > > These checks could do with a comment explaining why the if statements > are needed (I'm assuming something to do with memory order). The checks aren't needed; they were supposed to spell out the equality-hash relationship. But that isn't obvious, so I'll add a comment or remove them. Stefan Krah From solipsis at pitrou.net Sat Nov 3 11:17:48 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Nov 2012 11:17:48 +0100 Subject: [Python-Dev] [Python-checkins] cpython (3.3): Issue #15814: Use hash function that is compatible with the equality References: <3XtV7q58c6zRMy@mail.python.org> <CADiSq7fSaz6uxQeoQeyi0aukH3kLifUAmSpd1ZRJe2KedXD5DQ@mail.gmail.com> <20121103095923.GA30827@sleipnir.bytereef.org> Message-ID: <20121103111748.1100c73e@pitrou.net> On Sat, 3 Nov 2012 10:59:23 +0100 Stefan Krah <stefan at bytereef.org> wrote: > Nick Coghlan <ncoghlan at gmail.com> wrote: > > > + if (b == c): > > > + self.assertEqual(hash(b), hash(c)) > > > > These checks could do with a comment explaining why the if statements > > are needed (I'm assuming something to do with memory order). > > The checks aren't needed; they were supposed to spell out the equality-hash > relationship. Better use assertEqual(), then. Regards Antoine. From techtonik at gmail.com Sat Nov 3 12:29:57 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 14:29:57 +0300 Subject: [Python-Dev] Using Wiki for Python development coordination Message-ID: <CAPkN8xLkxbV8083NTr=+b_fXpxwJYuiNwihmW43NNZgd2Sm99g@mail.gmail.com> I don't have access to modify the front page. http://wiki.python.org/moin/FrontPage To me it lacks one more section concerning help with core development infrastructure. Python Core Development Python Website <http://wiki.python.org/moin/PythonWebsite> The entrypoint for everything concerning the language Bug Tracker <http://wiki.python.org/moin/TrackerDevelopment> Roundup and code review services we use Package Index <http://wiki.python.org/moin/CheeseShopDev> Some building blocks for your own projects (including frameworks for database, GUI, Web programming) -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121103/e85fb688/attachment.html> From fijall at gmail.com Sat Nov 3 15:48:25 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 3 Nov 2012 16:48:25 +0200 Subject: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) In-Reply-To: <CAP1=2W4VMMDCmQk-7VKEyK3zFCq9R8bsK3pv6hV69sCLA0MQsA@mail.gmail.com> References: <CAP1=2W74+-dqx8=SOD0QBThRuGY_B9ed4wmDLq_mo0x3_1W1eg@mail.gmail.com> <3E792B3D-BEA9-4E9F-9102-7F18FD3771E4@underboss.org> <CAP1=2W4VMMDCmQk-7VKEyK3zFCq9R8bsK3pv6hV69sCLA0MQsA@mail.gmail.com> Message-ID: <CAK5idxSzr19zuWEd34EY0YscMBWnG-BU+W3vFd9MSb3ORPiNVg@mail.gmail.com> On Fri, Nov 2, 2012 at 8:42 PM, Brett Cannon <brett at python.org> wrote: > Issue filed for the performance issue: http://bugs.python.org/issue16390 > > With that change and running on tip of Mako on my laptop now reports 1.25x > slower which is much better than it was. This performance issue might also > explain why all of the regex compilation benchmarks are worse under Python > 3.3 by a decent margin. > > On Fri, Nov 2, 2012 at 2:16 PM, Philip Jenvey <pjenvey at underboss.org> wrote: >> >> lru_cache on re._compile_typed I would like to warn you about modifying benchmarks like this (or frameworks). Why is it relevant anyway? From paul at boddie.org.uk Sat Nov 3 15:54:45 2012 From: paul at boddie.org.uk (Paul Boddie) Date: Sat, 3 Nov 2012 15:54:45 +0100 Subject: [Python-Dev] Using Wiki for Python development coordination In-Reply-To: <CAPkN8xLkxbV8083NTr=+b_fXpxwJYuiNwihmW43NNZgd2Sm99g@mail.gmail.com> References: <CAPkN8xLkxbV8083NTr=+b_fXpxwJYuiNwihmW43NNZgd2Sm99g@mail.gmail.com> Message-ID: <201211031554.46007.paul@boddie.org.uk> On Saturday 03 November 2012 12:29:57 anatoly techtonik wrote: > I don't have access to modify the front page. > http://wiki.python.org/moin/FrontPage Yes, access was restricted a while ago because of spamming. > To me it lacks one more section concerning help with core development > infrastructure. > > Python Core Development > > Python Website <http://wiki.python.org/moin/PythonWebsite> > > > The entrypoint for everything concerning the language > > > Bug Tracker <http://wiki.python.org/moin/TrackerDevelopment> > > > Roundup and code review services we use > > > Package Index <http://wiki.python.org/moin/CheeseShopDev> > > > Some building blocks for your own projects (including frameworks for > database, GUI, Web programming) I'll admit that the current content is just a reformatted version of what was there before, but tidied up and making better use of vertical space, and it could be improved. Certainly, there isn't a core development section, so perhaps I'll just add what you suggest. I know how you dislike the Moin markup, Anatoly. ;-) Paul From techtonik at gmail.com Sat Nov 3 16:53:57 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Sat, 3 Nov 2012 18:53:57 +0300 Subject: [Python-Dev] Using Wiki for Python development coordination In-Reply-To: <201211031554.46007.paul@boddie.org.uk> References: <CAPkN8xLkxbV8083NTr=+b_fXpxwJYuiNwihmW43NNZgd2Sm99g@mail.gmail.com> <201211031554.46007.paul@boddie.org.uk> Message-ID: <CAPkN8xJZVGW0qU+3x8wtUAct8uXG552ukXdi=fvtt57joosQXg@mail.gmail.com> On Sat, Nov 3, 2012 at 5:54 PM, Paul Boddie <paul at boddie.org.uk> wrote: > > > I'll admit that the current content is just a reformatted version of what was > there before, but tidied up and making better use of vertical space, and it > could be improved. Certainly, there isn't a core development section, so > perhaps I'll just add what you suggest. Thanks! That brings visibility of that rather critical section up to the highest point possible. =) > I know how you dislike the Moin > markup, Anatoly. ;-) Yes, I even think about further enhancing "Using this Wiki" part to erase references to Moin help pages. They are a little bit misleading to me and if somebody needs help on Moin - there is a side menu. I'd like to change it from: ---- Feel free to add more useful stuff (see HelpContents and HelpOnEditing to learn how), but do us a favour and do tests in the WikiSandBox if you're not accustomed to Wiki technologies. If you're new to Wikis, please read WikiWikiWeb. See WikiGuidelines for details of the policies and rules governing this Wiki. See SiteImprovements for a discussion of improvements to this Wiki and other related sites. See RecentChanges for a history, available in RSS format. To see pages which need writing, take a look at DesiredPages. To report problems or to help out, please contact the python.org maintainers. ---- to ---- This Wiki is community place to gather and organize all things about Python. Feel free to exercise your editorial skills and expertise to improve it into a useful knowledge base and up-to-date reference on different matters. There are some WikiGuidelines if you feel lost. SiteImprovements contains TODO list with various tasks you can help with. Check RecentChanges frequently to read the news (also available in RSS). In case of emergency please contact the python.org maintainers, or pop up in pydotorg-www to say "help". ---- From brett at python.org Sat Nov 3 17:29:18 2012 From: brett at python.org (Brett Cannon) Date: Sat, 3 Nov 2012 12:29:18 -0400 Subject: [Python-Dev] Python 3.3 vs. Python 2.7 benchmark results (again, but this time more solid numbers) In-Reply-To: <CAK5idxSzr19zuWEd34EY0YscMBWnG-BU+W3vFd9MSb3ORPiNVg@mail.gmail.com> References: <CAP1=2W74+-dqx8=SOD0QBThRuGY_B9ed4wmDLq_mo0x3_1W1eg@mail.gmail.com> <3E792B3D-BEA9-4E9F-9102-7F18FD3771E4@underboss.org> <CAP1=2W4VMMDCmQk-7VKEyK3zFCq9R8bsK3pv6hV69sCLA0MQsA@mail.gmail.com> <CAK5idxSzr19zuWEd34EY0YscMBWnG-BU+W3vFd9MSb3ORPiNVg@mail.gmail.com> Message-ID: <CAP1=2W4MY31QTEPRzkEcYwW7uhsxgcytv9eSy2LhoegFQYio0Q@mail.gmail.com> I'm not modifying any benchmark or framework. At best I will replace Mako 0.7.2 with Mako 0.7.3 in the benchmark suite since no one is historically recording the mako_v2 benchmark yet and it should be running with the newest version until we set it in stone. On Sat, Nov 3, 2012 at 10:48 AM, Maciej Fijalkowski <fijall at gmail.com>wrote: > On Fri, Nov 2, 2012 at 8:42 PM, Brett Cannon <brett at python.org> wrote: > > Issue filed for the performance issue: http://bugs.python.org/issue16390 > > > > With that change and running on tip of Mako on my laptop now reports > 1.25x > > slower which is much better than it was. This performance issue might > also > > explain why all of the regex compilation benchmarks are worse under > Python > > 3.3 by a decent margin. > > > > On Fri, Nov 2, 2012 at 2:16 PM, Philip Jenvey <pjenvey at underboss.org> > wrote: > >> > >> lru_cache on re._compile_typed > > I would like to warn you about modifying benchmarks like this (or > frameworks). Why is it relevant anyway? > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121103/fd85810c/attachment.html> From solipsis at pitrou.net Sat Nov 3 18:13:43 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 3 Nov 2012 18:13:43 +0100 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding References: <3Xv06h2mWtzMCY@mail.python.org> Message-ID: <20121103181343.27e690d7@pitrou.net> On Sat, 3 Nov 2012 13:37:48 +0100 (CET) andrew.svetlov <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/95d1adf144ee > changeset: 80187:95d1adf144ee > user: Andrew Svetlov <andrew.svetlov at gmail.com> > date: Sat Nov 03 14:37:37 2012 +0200 > summary: > Issue #16218: skip test if filesystem doesn't support required encoding > > files: > Lib/test/test_cmd_line_script.py | 7 ++++++- > 1 files changed, 6 insertions(+), 1 deletions(-) > > > diff --git a/Lib/test/test_cmd_line_script.py b/Lib/test/test_cmd_line_script.py > --- a/Lib/test/test_cmd_line_script.py > +++ b/Lib/test/test_cmd_line_script.py > @@ -366,7 +366,12 @@ > def test_non_utf8(self): > # Issue #16218 > with temp_dir() as script_dir: > - script_basename = '\udcf1\udcea\udcf0\udce8\udcef\udcf2' > + script_basename = '\u0441\u043a\u0440\u0438\u043f\u0442' Why exactly did you change the tested name here? Regards Antoine. From chris.jerdonek at gmail.com Sat Nov 3 18:36:28 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 3 Nov 2012 10:36:28 -0700 Subject: [Python-Dev] [Python-checkins] peps: Revert PEP 430 to Final. In-Reply-To: <3XqHjw6HZ6zR4V@mail.python.org> References: <3XqHjw6HZ6zR4V@mail.python.org> Message-ID: <CAOTb1wf6U_bEBK5oJ2hdBD4+KPd+02t+MZzGs75WNM23ie0hRw@mail.gmail.com> On Sun, Oct 28, 2012 at 5:07 AM, nick.coghlan <python-checkins at python.org> wrote: > > http://hg.python.org/peps/rev/1ccf682bdfc9 > changeset: 4575:1ccf682bdfc9 > user: Nick Coghlan <ncoghlan at gmail.com> > date: Sun Oct 28 22:06:46 2012 +1000 > summary: > Revert PEP 430 to Final. > ... > -To help ensure future stability even of links to the in-development version, > -the ``/dev/`` subpath will be redirected to the appropriate version specific > -subpath (currently ``/3.4/``). > ... > * ``http://docs.python.org/x/*`` > * ``http://docs.python.org/x.y/*`` > +* ``http://docs.python.org/dev/*`` > * ``http://docs.python.org/release/x.y.z/*`` > * ``http://docs.python.org/devguide`` > ... > +The ``/dev/`` URL means the documentation for the default branch in source > +control. I noticed on an older issue in the tracker that the following deep, in-development link is broken: http://docs.python.org/dev/2.6/library/multiprocessing.html (in comment http://bugs.python.org/issue4711#msg78151 ) Can something be done to preserve those deep links, or is it not worth worrying about? I'm not sure how prevalent such links are or when they were valid (and when they stopped working, assuming they once did). --Chris From chris.jerdonek at gmail.com Sat Nov 3 20:36:36 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 3 Nov 2012 12:36:36 -0700 Subject: [Python-Dev] [Python-checkins] peps: Revert PEP 430 to Final. In-Reply-To: <50955FE9.3000400@udel.edu> References: <3XqHjw6HZ6zR4V@mail.python.org> <CAOTb1wf6U_bEBK5oJ2hdBD4+KPd+02t+MZzGs75WNM23ie0hRw@mail.gmail.com> <50955FE9.3000400@udel.edu> Message-ID: <CAOTb1wdTD6nY5y8CP8EmGPbeimxpPqeFFpbhqOj7uVAEp6iZxg@mail.gmail.com> On Sat, Nov 3, 2012 at 11:18 AM, Terry Reedy <tjreedy at udel.edu> wrote: > > On 11/3/2012 1:36 PM, Chris Jerdonek wrote: >> >> I noticed on an older issue in the tracker that the following deep, >> in-development link is broken: >> >> http://docs.python.org/dev/2.6/library/multiprocessing.html >> >> (in comment http://bugs.python.org/issue4711#msg78151 ) >> >> Can something be done to preserve those deep links, or is it not worth >> worrying about? I'm not sure how prevalent such links are or when >> they were valid (and when they stopped working, assuming they once >> did). > > > I'd say not bother. The message was posted 2008-12-11 and the link was > already obsolete when Ezio answered 2009-07-01, when he gave a permanent > one. Fortunately, the OP gave a .png screenshot showing the problem. > > Any doc link in the tracker becomes obsolete in a sense as soon as the > problem noted is fixed. So tracker messages should state the problem > independently of the link. (After the docs are rebuilt tonight, > http://docs.python.org/library/multiprocessing.html > will no longer exhibit the problem either ;-). Though this link will still exhibit it (which may be what it would redirect to): http://docs.python.org/2.6/library/multiprocessing.html So a doc link in the tracker won't become obsolete if we wait long enough to fix the issue. :) --Chris From techtonik at gmail.com Mon Nov 5 08:31:26 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 5 Nov 2012 10:31:26 +0300 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? Message-ID: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> >From http://bugs.python.org/issue16410 Subj? Aren't there any modules in stdlib that access system API through ctypes? My arguments for ctypes: 1. doesn't require compilation 2. easier to maintain (no C/toolchain knowledge/ownership needed) 3. pure Python is impossible to exploit (unlike pure C) 4. eating your own dogfood helps to make modules complete and notice such silly/critical/timewasting/drivesmemad errors as http://bugs.python.org/issue16376 a few years earlier Maybe it could even help to make ctypes faster (through some caching mechanizm). -- anatoly t. From ronaldoussoren at mac.com Mon Nov 5 10:32:18 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 05 Nov 2012 10:32:18 +0100 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> Message-ID: <81A6AADC-39DE-4B46-B6F2-D545ED3C4C72@mac.com> On 5 Nov, 2012, at 8:31, anatoly techtonik <techtonik at gmail.com> wrote: > From http://bugs.python.org/issue16410 > Subj? > > Aren't there any modules in stdlib that access system API through ctypes? uuid uses ctypes to access platform APIs. > > My arguments for ctypes: > 1. doesn't require compilation > 2. easier to maintain (no C/toolchain knowledge/ownership needed) > 3. pure Python is impossible to exploit (unlike pure C) That's not not quite true, python code that uses ctypes can still cause buffer overflows and the like when you aren't careful, and adds new failure modes (such as incorrect specification of a C function prototype in the Python code that calls that C function) Ronald From victor.stinner at gmail.com Mon Nov 5 12:36:00 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 5 Nov 2012 12:36:00 +0100 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> Message-ID: <CAMpsgwZqW1SzJWhFCTj6W0K+=eN2uVQ4dzGc2m-seUa9hLPPWg@mail.gmail.com> I'm not sure that ctypes is always available (available on all platforms). Victor From rdmurray at bitdance.com Mon Nov 5 12:49:15 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 05 Nov 2012 06:49:15 -0500 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> Message-ID: <20121105114915.5818225006A@webabinitio.net> On Mon, 05 Nov 2012 10:31:26 +0300, anatoly techtonik <techtonik at gmail.com> wrote: > Aren't there any modules in stdlib that access system API through ctypes? No. This is a policy. Changing that policy would require a PEP (*after* a discussion on python-ideas...which has probably already happened in the past, I would guess). --David From dickinsm at gmail.com Mon Nov 5 13:08:47 2012 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 5 Nov 2012 12:08:47 +0000 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? Message-ID: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> In Python 2, the 'exec' statement supports 'exec'-ing a (statement, globals, locals) tuple: >>> exec("print 2", {}, {}) 2 This isn't currently documented at: http://docs.python.org/reference/simple_stmts.html#the-exec-statement. It's easy to fix the docs, but in doing so we'd effectively be blessing this form of exec as an official part of the language. Do people think it's acceptable to add this to the docs, or are there good reasons for the 'exec tuple' form of the exec statement to remain an undocumented feature? See also http://bugs.python.org/issue16339. Mark From ncoghlan at gmail.com Mon Nov 5 13:22:26 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 5 Nov 2012 22:22:26 +1000 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? In-Reply-To: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> References: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> Message-ID: <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> On Mon, Nov 5, 2012 at 10:08 PM, Mark Dickinson <dickinsm at gmail.com> wrote: > In Python 2, the 'exec' statement supports 'exec'-ing a (statement, > globals, locals) tuple: > >>>> exec("print 2", {}, {}) > 2 > > This isn't currently documented at: > http://docs.python.org/reference/simple_stmts.html#the-exec-statement. > It's easy to fix the docs, but in doing so we'd effectively be > blessing this form of exec as an official part of the language. Do > people think it's acceptable to add this to the docs, or are there > good reasons for the 'exec tuple' form of the exec statement to remain > an undocumented feature? > > See also http://bugs.python.org/issue16339. If you can find an existing test for it, then definitely (although the fact it didn't previously work on Jython suggests there may not be one). If there's no test, then I'd still be in favour of making it official both by testing *and* documenting it, for the forward compatibility benefits you mention on the tracker. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From dickinsm at gmail.com Mon Nov 5 14:09:55 2012 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 5 Nov 2012 13:09:55 +0000 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? In-Reply-To: <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> References: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> Message-ID: <CAAu3qLX_qyiCAtnOzrqceuHj_CmxdM4m=73cwbM8P6tfvqrU+A@mail.gmail.com> On Mon, Nov 5, 2012 at 12:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > If you can find an existing test for it, then definitely (although the > fact it didn't previously work on Jython suggests there may not be > one). I don't see any *direct* tests for this feature, though there are a couple of tests that just happen to use that form of 'exec' (test_pkg.py, test_module.py, test_pep263,py). > If there's no test, then I'd still be in favour of making it official > both by testing *and* documenting it, for the forward compatibility > benefits you mention on the tracker. Sounds good to me. I'll add a note to the issue about tests. Mark From fijall at gmail.com Mon Nov 5 14:38:23 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 5 Nov 2012 15:38:23 +0200 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> Message-ID: <CAK5idxQeTySE6w5KMYu_9y2_b_94PLurtuNzGQ0xSTCNrLOOkQ@mail.gmail.com> On Mon, Nov 5, 2012 at 9:31 AM, anatoly techtonik <techtonik at gmail.com> wrote: > From http://bugs.python.org/issue16410 > Subj? > > Aren't there any modules in stdlib that access system API through ctypes? > > My arguments for ctypes: > 1. doesn't require compilation > 2. easier to maintain (no C/toolchain knowledge/ownership needed) > 3. pure Python is impossible to exploit (unlike pure C) > 4. eating your own dogfood helps to make modules complete and notice > such silly/critical/timewasting/drivesmemad errors as > http://bugs.python.org/issue16376 a few years earlier > > Maybe it could even help to make ctypes faster (through some caching mechanizm). > -- > anatoly t. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com Hi anatoly. ctypes comes with it's own set of problems that manifest themselves more or less depending what sort of libary have you tried to wrap. Have you ever tried to use it seriously? The list of my personal issues is available here: http://morepypy.blogspot.com/2012/06/release-01-of-cffi.html The main problem is API vs ABI and the robustness of checks. I would not recommend using ctypes for any of the sdtlib (we actually tried in pypy, it turned out a bit awful). Cheers, fijal From catch-all at masklinn.net Mon Nov 5 15:14:17 2012 From: catch-all at masklinn.net (Xavier Morel) Date: Mon, 5 Nov 2012 15:14:17 +0100 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <81A6AADC-39DE-4B46-B6F2-D545ED3C4C72@mac.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> <81A6AADC-39DE-4B46-B6F2-D545ED3C4C72@mac.com> Message-ID: <B3F74D2B-F87B-41D1-9478-AA022A969DE5@masklinn.net> On 2012-11-05, at 10:32 , Ronald Oussoren wrote: >> My arguments for ctypes: >> 1. doesn't require compilation >> 2. easier to maintain (no C/toolchain knowledge/ownership needed) >> 3. pure Python is impossible to exploit (unlike pure C) > > That's not not quite true, python code that uses ctypes can still cause buffer overflows and the like Such as segfaulting the interpreter. I seem to reliably segfault everything every time I try to use ctypes. From sturla at molden.no Mon Nov 5 15:47:05 2012 From: sturla at molden.no (Sturla Molden) Date: Mon, 05 Nov 2012 15:47:05 +0100 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <B3F74D2B-F87B-41D1-9478-AA022A969DE5@masklinn.net> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> <81A6AADC-39DE-4B46-B6F2-D545ED3C4C72@mac.com> <B3F74D2B-F87B-41D1-9478-AA022A969DE5@masklinn.net> Message-ID: <5097D169.7060304@molden.no> On 05.11.2012 15:14, Xavier Morel wrote: > Such as segfaulting the interpreter. I seem to reliably segfault > everything every time I try to use ctypes. You can do that with C extensions too, by the way. Apart from that, dependency on ABI is more annoying to maintain across platforms than dependency on API. Function calls with ctypes are also very slow. For C extensions in the stdlib, Cython might be a better choice then ctypes. ctypes might be a good choice if you are to use a DLL on your own computer. Because then you only have one ABI to worry about. Not so for Python's standard library. Sturla From guido at python.org Mon Nov 5 15:57:30 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 5 Nov 2012 06:57:30 -0800 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? In-Reply-To: <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> References: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> Message-ID: <CAP7+vJKJOH-58tXxLZ9J4jULcqFVMD1xwq=8RudxZqSQEGQsmw@mail.gmail.com> +1. This probably fell through the cracks because I changed my mind on this. The function form was old, and I wanted to root it out in favor fo the statement. But then I changed my mind for py3k. Good idea to document and add tests in 2.x. --Guido On Monday, November 5, 2012, Nick Coghlan wrote: > On Mon, Nov 5, 2012 at 10:08 PM, Mark Dickinson <dickinsm at gmail.com<javascript:;>> > wrote: > > In Python 2, the 'exec' statement supports 'exec'-ing a (statement, > > globals, locals) tuple: > > > >>>> exec("print 2", {}, {}) > > 2 > > > > This isn't currently documented at: > > http://docs.python.org/reference/simple_stmts.html#the-exec-statement. > > It's easy to fix the docs, but in doing so we'd effectively be > > blessing this form of exec as an official part of the language. Do > > people think it's acceptable to add this to the docs, or are there > > good reasons for the 'exec tuple' form of the exec statement to remain > > an undocumented feature? > > > > See also http://bugs.python.org/issue16339. > > If you can find an existing test for it, then definitely (although the > fact it didn't previously work on Jython suggests there may not be > one). > > If there's no test, then I'd still be in favour of making it official > both by testing *and* documenting it, for the forward compatibility > benefits you mention on the tracker. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com <javascript:;> | Brisbane, > Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org <javascript:;> > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121105/901ba3dd/attachment.html> From g.brandl at gmx.net Mon Nov 5 19:04:34 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 05 Nov 2012 19:04:34 +0100 Subject: [Python-Dev] cpython (3.3): Add examples for opener argument of open (#13424). In-Reply-To: <3XvCTv2Q38zPjt@mail.python.org> References: <3XvCTv2Q38zPjt@mail.python.org> Message-ID: <k78v3i$n6$1@ger.gmane.org> Am 03.11.2012 22:10, schrieb eric.araujo: > http://hg.python.org/cpython/rev/17b094c08600 > changeset: 80219:17b094c08600 > branch: 3.3 > parent: 80214:e6d0951f412a > user: ?ric Araujo <merwok at netwok.org> > date: Sat Nov 03 17:06:52 2012 -0400 > summary: > Add examples for opener argument of open (#13424). > > Patch by Guillaume Pratte. > > files: > Doc/library/functions.rst | 30 +++++++++++++++++++++++++++ > Doc/library/io.rst | 3 ++ > Misc/ACKS | 1 + > 3 files changed, 34 insertions(+), 0 deletions(-) > > > diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst > --- a/Doc/library/functions.rst > +++ b/Doc/library/functions.rst > @@ -935,6 +935,36 @@ > :mod:`os.open` as *opener* results in functionality similar to passing > ``None``). > > + The following example is an alternative implementation for opening files > + for exclusive writing. If we did not have support for the ``'x'`` mode, > + we could implement it with this opener:: > + > + >>> import os > + >>> def open_exclusive(path, mode): > + ... return os.open(path, mode | os.O_CREAT | os.O_EXCL) > + ... > + >>> filename = 'spam.txt' > + >>> fp = open(filename, 'w', opener=open_exclusive) > + >>> fp2 = open(filename, 'w', opener=open_exclusive) > + Traceback (most recent call last): > + ... > + FileExistsError: [Errno 17] File exists: 'spam.txt' > + > + This other example uses the :ref:`dir_fd` parameter of the > + :func:`os.open` function to open a file relative to a given directory:: Please heed your Sphinx warnings: the :ref:`dir_fd` needs a link caption, since it can't autogenerate one (the dir_fd anchor does not point to a heading). Georg From greg.ewing at canterbury.ac.nz Mon Nov 5 22:51:01 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 06 Nov 2012 10:51:01 +1300 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? In-Reply-To: <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> References: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> Message-ID: <509834C5.8060302@canterbury.ac.nz> Nick Coghlan wrote: > On Mon, Nov 5, 2012 at 10:08 PM, Mark Dickinson <dickinsm at gmail.com> wrote: > >>In Python 2, the 'exec' statement supports 'exec'-ing a (statement, >>globals, locals) tuple: If this is a deliberate feature, I'd guess it's because exec is a statement rather than a function in Python 2, so you can't use * to pass a tuple of arguments to it. -- Greg From chris at simplistix.co.uk Tue Nov 6 07:18:07 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Tue, 06 Nov 2012 06:18:07 +0000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> Message-ID: <5098AB9F.2010806@simplistix.co.uk> Hi All, I bumped into this using Michael Foord's Mock library. It feels like a bug to me, but thought I'd ask here before logging one in the tracker in case people know that we won't be able to fix it: On 05/11/2012 13:43, Michael Foord wrote: > >>>>>> class Foo(object): >>> ... def __setattr__(s, k, v): >>> ... print k >>> ... >>>>>> f = Foo() >>>>>> >>>>>> f.g = f.x = 3 >>> g >>> x >> >> Hmm, that's definitely counter-intuitive. Worth raising on python-dev? > > Well, I guess "it's always been that way", so changing it would be backwards incompatible. Feel free to raise it though. :-) Here's the actual problem I had: >>>>>>> mock = Mock() >>>>>>> inst = mock.Popen.return_value = mock.Popen_instance = Mock(spec=Popen) >>> >>> >>> Here the *return_value* is set first. So when mock.Popen_instance is assigned it already has a parent! (Setting a mock as the return value of another mock also establishes the child / parent relationship.) >>> >>> This is why calls to the instance are then not recorded in "method_calls". cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From jeanpierreda at gmail.com Tue Nov 6 09:54:08 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 6 Nov 2012 03:54:08 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <5098AB9F.2010806@simplistix.co.uk> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> Message-ID: <CABicbJKY0-P90rgdXfb3xi5sgs9ysH2iYg6HqfXpbuV9jNxCcQ@mail.gmail.com> On Tue, Nov 6, 2012 at 1:18 AM, Chris Withers <chris at simplistix.co.uk> wrote: > Hi All, > > I bumped into this using Michael Foord's Mock library. > It feels like a bug to me, but thought I'd ask here before logging one in > the tracker in case people know that we won't be able to fix it: > > On 05/11/2012 13:43, Michael Foord wrote: >> >> >>>>>>> class Foo(object): >>>> >>>> ... def __setattr__(s, k, v): >>>> ... print k >>>> ... >>>>>>> >>>>>>> f = Foo() >>>>>>> >>>>>>> f.g = f.x = 3 >>>> >>>> g >>>> x >>> It's terrible and counter-intuitive, but it is documented. See: http://docs.python.org/2/reference/simple_stmts.html#assignment-statements Notice that it proclaims that target lists are assigned to from left to right. That's the behavior you've noticed. On the other hand, one might easily misread this piece of documentation, which implies right to left assignment: http://docs.python.org/2/reference/expressions.html#evaluation-order But the last note doesn't look normative to me. (I'm not a dev, just trying to be helpful.) -- Devin From tjreedy at udel.edu Tue Nov 6 10:24:37 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 06 Nov 2012 04:24:37 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <5098AB9F.2010806@simplistix.co.uk> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> Message-ID: <k7al0r$iom$1@ger.gmane.org> On 11/6/2012 1:18 AM, Chris Withers wrote: > Hi All, > > I bumped into this using Michael Foord's Mock library. > It feels like a bug to me, but thought I'd ask here before logging one > in the tracker in case people know that we won't be able to fix it: > > On 05/11/2012 13:43, Michael Foord wrote: >> >>>>>>> class Foo(object): >>>> ... def __setattr__(s, k, v): >>>> ... print k >>>> ... >>>>>>> f = Foo() >>>>>>> >>>>>>> f.g = f.x = 3 >>>> g >>>> x >>> >>> Hmm, that's definitely counter-intuitive. Worth raising on python-dev? >> >> Well, I guess "it's always been that way", so changing it would be >> backwards incompatible. Feel free to raise it though. :-) You are expecting a chained assignment a = b = c to be parsed as a = (b = c), as in C and as it should be if assignment is an expression. But in Python it is not. The right association would be meaningless since (b = c) has no value to be assigned to a. Python actually parses the chained assignment as (a =) (b =) c. The relevant grammar production is "assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)". The explanation after the grammar begins with this sentence. "An assignment statement evaluates the expression list (remember that this can be a single expression or a comma-separated list, the latter yielding a tuple) and assigns the single resulting object to each of the target lists, from left to right." In other words, in my example, c is assigned to a then b. Please do not report documented behavior on the tracker as a bug (and don't encourage people to). If you think the above is not clear enough, you *can* suggest improvement. Perhaps add an example and explanation such as a = b, (c,d), *e, f = 1, (2, 3), 4, 5, 6 "The tuple on the right is first assigned to a and then unpacked to b, c, d, e, and f, giving them the values 1, 2, 3, [4, 5], and 6." -- Terry Jan Reedy From ncoghlan at gmail.com Tue Nov 6 13:01:14 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 6 Nov 2012 22:01:14 +1000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <k7al0r$iom$1@ger.gmane.org> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> Message-ID: <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> As noted, it's really only counterintuitive if your intuition is primed to expect C style right to left chained assignments. Python, on the other hand, is able to preserve primarily left to right evaluation in this case with only the far right hand expression needing to be evaluated out of order. One example that can really make the intended behaviour clear: *a = *b = iter(range(3)) a ends up as (0,1,2), b ends up as (), because the first assignment consumes the entire iterable. My actual advice, though? If the order of assignment really matters, use multiple assignment statements rather than relying on readers knowing the assignment order. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121106/e9260b3d/attachment.html> From trent at snakebite.org Tue Nov 6 15:35:00 2012 From: trent at snakebite.org (Trent Nelson) Date: Tue, 6 Nov 2012 09:35:00 -0500 Subject: [Python-Dev] ctypes is not an acceptable implementation strategy for modules in the standard library? In-Reply-To: <CAMpsgwZqW1SzJWhFCTj6W0K+=eN2uVQ4dzGc2m-seUa9hLPPWg@mail.gmail.com> References: <CAPkN8xJCTs=9meZSLEkP=wJ05vNUyx+TnDGQ3z30t0uCfSDmFQ@mail.gmail.com> <CAMpsgwZqW1SzJWhFCTj6W0K+=eN2uVQ4dzGc2m-seUa9hLPPWg@mail.gmail.com> Message-ID: <20121106143458.GA86328@snakebite.org> On Mon, Nov 05, 2012 at 03:36:00AM -0800, Victor Stinner wrote: > I'm not sure that ctypes is always available (available on all platforms). Indeed. Every non-x86 Snakebite platform has pretty serious issues with ctypes. I spent a morning looking into one platform, Solaris 10 on SPARC, and, after building the latest libffi from scratch, it failed more tests than that box chews amps. Trent. From rob.cliffe at btinternet.com Tue Nov 6 16:02:31 2012 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 06 Nov 2012 15:02:31 +0000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> Message-ID: <50992687.2000505@btinternet.com> On 06/11/2012 12:01, Nick Coghlan wrote: > > As noted, it's really only counterintuitive if your intuition is > primed to expect C style right to left chained assignments. > > Python, on the other hand, is able to preserve primarily left to right > evaluation in this case with only the far right hand expression > needing to be evaluated out of order. > > > It strikes me that a really intuitive language (at least for Westerners who read left-to-right) would write assignments as expression --> target and then the order of assignment in expression -> target1 -> target2 could be the natural left-to-right one. [Sorry, this is more appropriate to Python-ideas, but I couldn't resist adding my 2c.] Rob Cliffe > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/rob.cliffe%40btinternet.com -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121106/2792fc1f/attachment.html> From merwok at netwok.org Tue Nov 6 16:12:54 2012 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 06 Nov 2012 10:12:54 -0500 Subject: [Python-Dev] cpython (3.3): Add examples for opener argument of open (#13424). In-Reply-To: <k78v3i$n6$1@ger.gmane.org> References: <3XvCTv2Q38zPjt@mail.python.org> <k78v3i$n6$1@ger.gmane.org> Message-ID: <509928F6.7080906@netwok.org> Hi, Le 05/11/2012 13:04, Georg Brandl a ?crit : > Please heed your Sphinx warnings: the :ref:`dir_fd` needs a link caption, since > it can't autogenerate one (the dir_fd anchor does not point to a heading). Okay. I hadn?t noticed it because I was using my system sphinx-build instead of a local one and there were many warnings related (I think) to pyspecific extensions not found. Will fix (and also simplify the examples per the reviews I got). Cheers From python at mrabarnett.plus.com Tue Nov 6 16:32:23 2012 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 06 Nov 2012 15:32:23 +0000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <50992687.2000505@btinternet.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <50992687.2000505@btinternet.com> Message-ID: <50992D87.20007@mrabarnett.plus.com> On 2012-11-06 15:02, Rob Cliffe wrote: > > On 06/11/2012 12:01, Nick Coghlan wrote: >> >> As noted, it's really only counterintuitive if your intuition is >> primed to expect C style right to left chained assignments. >> >> Python, on the other hand, is able to preserve primarily left to right >> evaluation in this case with only the far right hand expression >> needing to be evaluated out of order. >> > It strikes me that a really intuitive language (at least for Westerners > who read left-to-right) would write assignments as > expression --> target > and then the order of assignment in > expression -> target1 -> target2 > could be the natural left-to-right one. That would make augmented assignment more difficult. For example, how would you write the equivalent of "x -= y"? > [Sorry, this is more appropriate to Python-ideas, but I couldn't resist > adding my 2c.] > Rob Cliffe From guido at python.org Tue Nov 6 16:37:45 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Nov 2012 07:37:45 -0800 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> Message-ID: <CAP7+vJLv7Q_nCCxas3t94uj2xm+2WY6Fo2gm7Xa0iPOHvE-wpQ@mail.gmail.com> +1 to what Nick said. And I thought about this carefully when designing the language. It's not a bug. The note about assignment RHS being evaluated before LHS is normative -- you just have to interpret RHS as "after the *last* '=' symbol". Assignment itself is *not* an expression. On Tue, Nov 6, 2012 at 4:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > As noted, it's really only counterintuitive if your intuition is primed to > expect C style right to left chained assignments. > > Python, on the other hand, is able to preserve primarily left to right > evaluation in this case with only the far right hand expression needing to > be evaluated out of order. > > One example that can really make the intended behaviour clear: > > *a = *b = iter(range(3)) > > a ends up as (0,1,2), b ends up as (), because the first assignment consumes > the entire iterable. > > My actual advice, though? If the order of assignment really matters, use > multiple assignment statements rather than relying on readers knowing the > assignment order. > > Cheers, > Nick. > > -- > Sent from my phone, thus the relative brevity :) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Tue Nov 6 17:14:38 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 06 Nov 2012 18:14:38 +0200 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> Message-ID: <k7bd1m$i5b$1@ger.gmane.org> On 06.11.12 14:01, Nick Coghlan wrote: > Python, on the other hand, is able to preserve primarily left to right > evaluation in this case with only the far right hand expression needing > to be evaluated out of order. I'm surprised, but it is really so. >>> {}[print('foo')] = print('bar') bar foo I was expecting "foo" before "bar". Another counterintuitive (and possible wrong) example: >>> {print('foo'): print('bar')} bar foo {None: None} From rdmurray at bitdance.com Tue Nov 6 17:26:55 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 06 Nov 2012 11:26:55 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <k7bd1m$i5b$1@ger.gmane.org> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> Message-ID: <20121106162655.A858625006B@webabinitio.net> On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka <storchaka at gmail.com> wrote: > Another counterintuitive (and possible wrong) example: > > >>> {print('foo'): print('bar')} > bar > foo > {None: None} http://bugs.python.org/issue11205 --David From ned at nedbatchelder.com Tue Nov 6 18:58:53 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 06 Nov 2012 12:58:53 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <20121106162655.A858625006B@webabinitio.net> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> Message-ID: <50994FDD.8060906@nedbatchelder.com> On 11/6/2012 11:26 AM, R. David Murray wrote: > On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka <storchaka at gmail.com> wrote: >> Another counterintuitive (and possible wrong) example: >> >> >>> {print('foo'): print('bar')} >> bar >> foo >> {None: None} > http://bugs.python.org/issue11205 This seems to me better left undefined, since there's hardly ever a need to know the precise evaluation sequence between keys and values, and retaining some amount of "unspecified" to allow for implementation flexibility is a good thing. --Ned. > > --David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > From jeanpierreda at gmail.com Tue Nov 6 19:19:56 2012 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Tue, 6 Nov 2012 13:19:56 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <50994FDD.8060906@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> Message-ID: <CABicbJK+_3aFmcfegELAt=Kgd1n89jzW13ukHL3gn=woiAqUUQ@mail.gmail.com> On Nov 6, 2012 1:05 PM, "Ned Batchelder" <ned at nedbatchelder.com> wrote: > > On 11/6/2012 11:26 AM, R. David Murray wrote: >> >> On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka <storchaka at gmail.com> wrote: >>> >>> Another counterintuitive (and possible wrong) example: >>> >>> >>> {print('foo'): print('bar')} >>> bar >>> foo >>> {None: None} >> >> http://bugs.python.org/issue11205 > > > This seems to me better left undefined, since there's hardly ever a need to know the precise evaluation sequence between keys and values, and retaining some amount of "unspecified" to allow for implementation flexibility is a good thing. "Left undefined"? The behavior was defined, but CPython didn't follow the defined behaviour. --Devin (phone) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121106/3f9e0091/attachment.html> From ned at nedbatchelder.com Tue Nov 6 20:00:17 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 06 Nov 2012 14:00:17 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CABicbJK+_3aFmcfegELAt=Kgd1n89jzW13ukHL3gn=woiAqUUQ@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CABicbJK+_3aFmcfegELAt=Kgd1n89jzW13ukHL3gn=woiAqUUQ@mail.gmail.com> Message-ID: <50995E41.5080907@nedbatchelder.com> On 11/6/2012 1:19 PM, Devin Jeanpierre wrote: > > > On Nov 6, 2012 1:05 PM, "Ned Batchelder" <ned at nedbatchelder.com > <mailto:ned at nedbatchelder.com>> wrote: > > > > On 11/6/2012 11:26 AM, R. David Murray wrote: > >> > >> On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka > <storchaka at gmail.com <mailto:storchaka at gmail.com>> wrote: > >>> > >>> Another counterintuitive (and possible wrong) example: > >>> > >>> >>> {print('foo'): print('bar')} > >>> bar > >>> foo > >>> {None: None} > >> > >> http://bugs.python.org/issue11205 > > > > > > This seems to me better left undefined, since there's hardly ever a > need to know the precise evaluation sequence between keys and values, > and retaining some amount of "unspecified" to allow for implementation > flexibility is a good thing. > > "Left undefined"? The behavior was defined, but CPython didn't follow > the defined behaviour. > I would change the reference manual to leave it undefined. Clearly not many people have been bothered by the fact that CPython implemented it "wrong". If someone really needs to control whether the keys or values are evaluated first, they shouldn't use a dict literal. --Ned. > --Devin (phone) > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121106/0a5b1d93/attachment.html> From greg.ewing at canterbury.ac.nz Tue Nov 6 22:30:58 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Nov 2012 10:30:58 +1300 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <50992D87.20007@mrabarnett.plus.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <50992687.2000505@btinternet.com> <50992D87.20007@mrabarnett.plus.com> Message-ID: <50998192.9080907@canterbury.ac.nz> MRAB wrote: > That would make augmented assignment more difficult. For example, how > would you write the equivalent of "x -= y"? SUBTRACT x FROM y. CLOSE POST WITH SMILEY. -- Greg From guido at python.org Tue Nov 6 23:12:40 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 6 Nov 2012 14:12:40 -0800 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <50994FDD.8060906@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> Message-ID: <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> On Tue, Nov 6, 2012 at 9:58 AM, Ned Batchelder <ned at nedbatchelder.com> wrote: > On 11/6/2012 11:26 AM, R. David Murray wrote: >> >> On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka <storchaka at gmail.com> >> wrote: >>> >>> Another counterintuitive (and possible wrong) example: >>> >>> >>> {print('foo'): print('bar')} >>> bar >>> foo >>> {None: None} >> >> http://bugs.python.org/issue11205 > > > This seems to me better left undefined, since there's hardly ever a need to > know the precise evaluation sequence between keys and values, and retaining > some amount of "unspecified" to allow for implementation flexibility is a > good thing. Maybe. Do note that Python tries to be *different* than your average C++ standard and actually prescribes evaluation orders in most cases. The idea being that the kind of optimizations that C++ compilers get to do by moving evaluation order around aren't worth it in Python anyway, and it's better for the user if there are no arbitrary differences in this area between Python implementations. Note that I didn't say "no surprises" -- the post that started this thread shows that surprises are still possible. -- --Guido van Rossum (python.org/~guido) From jiuc at yahoo.cn Wed Nov 7 06:24:11 2012 From: jiuc at yahoo.cn (=?utf-8?B?6YKT5Lmd56Wl?=) Date: Wed, 7 Nov 2012 13:24:11 +0800 (CST) Subject: [Python-Dev] =?utf-8?b?5Zue5aSN77yaIFB5dGhvbi1EZXYgRGlnZXN0LCBW?= =?utf-8?q?ol_112=2C_Issue_8?= In-Reply-To: <mailman.13042.1352237459.27097.python-dev@python.org> References: <mailman.13042.1352237459.27097.python-dev@python.org> Message-ID: <1352265851.72557.YahooMailNeo@web92103.mail.cnh.yahoo.com> I hava some question about Object which "inital" need ,but return "object" in mongo shell. So? ?BSONElement initial = p["initial"]; ????????????if ( initial.type() != Object ) { ????????????????errmsg = "initial has to be an object"; ????????????????return false; ????????????} ?"initial.type() != Object " will be found ,so "group" command will not run forever. What can'i do Mybe this is my fase, please give me some advise. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/2dbd62f2/attachment.html> From storchaka at gmail.com Wed Nov 7 08:49:43 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 07 Nov 2012 09:49:43 +0200 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <50995E41.5080907@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CABicbJK+_3aFmcfegELAt=Kgd1n89jzW13ukHL3gn=woiAqUUQ@mail.gmail.com> <50995E41.5080907@nedbatchelder.com> Message-ID: <k7d3qm$sqi$1@ger.gmane.org> On 06.11.12 21:00, Ned Batchelder wrote: > If someone really needs to control whether the keys or values > are evaluated first, they shouldn't use a dict literal. Not only a dict literal. >>> {print('foo'): print('bar') for x in [1]} bar foo {None: None} From amauryfa at gmail.com Wed Nov 7 09:48:22 2012 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 7 Nov 2012 09:48:22 +0100 Subject: [Python-Dev] =?gb2312?b?u9i4tKO6IFB5dGhvbi1EZXYgRGlnZXN0LCBWb2wg?= =?gb2312?b?MTEyLCBJc3N1ZSA4?= In-Reply-To: <1352265851.72557.YahooMailNeo@web92103.mail.cnh.yahoo.com> References: <mailman.13042.1352237459.27097.python-dev@python.org> <1352265851.72557.YahooMailNeo@web92103.mail.cnh.yahoo.com> Message-ID: <CAGmFidZSqfvLXA8wL=v6_and0P=adLX0-VqRuhftper=27vnYQ@mail.gmail.com> Hi, 2012/11/7 ??? <jiuc at yahoo.cn>: > I hava some question about Object which "inital" need ,but return "object" > in mongo shell. So This is the python-dev mailing list, which deals with development of the Python language. You should ask your question on some MongoDB forum. > > BSONElement initial = p["initial"]; > if ( initial.type() > != Object ) { > errmsg = "initial has to be an object"; > return false; > } > "initial.type() != Object " will be found ,so "group" command will not run > forever. What can'i do > Mybe this is my fase, please give me some advise. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/amauryfa%40gmail.com > -- Amaury Forgeot d'Arc From techtonik at gmail.com Wed Nov 7 11:57:05 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 7 Nov 2012 13:57:05 +0300 Subject: [Python-Dev] urlretrieve regression no.2, now in Python 3.3 Message-ID: <CAPkN8xJtZ71gB3LdgvmQAy0cxKnwsCc7yMhpKj7c4Mb4T8jgmQ@mail.gmail.com> urlretrieve has a callback parameter, which takes function with the following prototype: def callback(block_number, block_size, total_size): pass Where block_size was constant and block_size*block_number gave an exact number of transferred bytes. Recent change in Python 3.3 changed the semantic of block_size and broke my `wget` package. http://bugs.python.org/issue16409 Can anybody tell me if I can wait for the fix in 3.3.1 or should implement a long-term workaround myself? -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/99b06a29/attachment.html> From techtonik at gmail.com Wed Nov 7 12:02:08 2012 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 7 Nov 2012 14:02:08 +0300 Subject: [Python-Dev] Fwd: [issue16424] regression: os.path.split('//hostname/foo/bar.txt') In-Reply-To: <1352285169.2.0.24678892452.issue16424@psf.upfronthosting.co.za> References: <1352268933.45.0.882120414377.issue16424@psf.upfronthosting.co.za> <1352285169.2.0.24678892452.issue16424@psf.upfronthosting.co.za> Message-ID: <CAPkN8xJCm2FNJYVSrcvFP+zWZ=YOf1kpMnOHrO1dLptLidYB-A@mail.gmail.com> Forwarding to python-dev. Does everybody agree that the following behavior is not a regression, but a feature of os.path.split()? Python 3: >>> import os.path as osp >>> osp.split('//hostname/foo/') ('//hostname/foo/', '') Python 2: >>> osp.split('//hostname/foo/') ('//hostname/foo', '') But Python 3 again: >>> osp.split('//hostname/foo/bar/') ('//hostname/foo/bar', '') http://bugs.python.org/issue16424 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/d9794b0b/attachment.html> From ned at nedbatchelder.com Wed Nov 7 13:13:00 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 07 Nov 2012 07:13:00 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> Message-ID: <509A504C.10807@nedbatchelder.com> On 11/6/2012 5:12 PM, Guido van Rossum wrote: > On Tue, Nov 6, 2012 at 9:58 AM, Ned Batchelder <ned at nedbatchelder.com> wrote: >> On 11/6/2012 11:26 AM, R. David Murray wrote: >>> On Tue, 06 Nov 2012 18:14:38 +0200, Serhiy Storchaka <storchaka at gmail.com> >>> wrote: >>>> Another counterintuitive (and possible wrong) example: >>>> >>>> >>> {print('foo'): print('bar')} >>>> bar >>>> foo >>>> {None: None} >>> http://bugs.python.org/issue11205 >> >> This seems to me better left undefined, since there's hardly ever a need to >> know the precise evaluation sequence between keys and values, and retaining >> some amount of "unspecified" to allow for implementation flexibility is a >> good thing. > Maybe. Do note that Python tries to be *different* than your average > C++ standard and actually prescribes evaluation orders in most cases. > The idea being that the kind of optimizations that C++ compilers get > to do by moving evaluation order around aren't worth it in Python > anyway, and it's better for the user if there are no arbitrary > differences in this area between Python implementations. Note that I > didn't say "no surprises" -- the post that started this thread shows > that surprises are still possible. > I think it's unfortunate that the current patch in the referenced bug ( http://bugs.python.org/issue11205 ) fixes the "problem" by adding one more bytecode to the compiled Python. The other alternative seems to be changing the meaning of two opcodes. Neither of these are great alternatives. I know we don't promise to maintain backward compatibility in the meanings of opcodes, but I'll bet actual code will be broken by changing the meaning of STORE_MAP and MAP_ADD. Slowing down dict displays (just slightly) to get this right seems unfortunate also. If the bug report is accurate, CPython and the reference manual have disagreed since Python 2.5, and many of us are now surprised to hear it, which means there can't have been much broken code. I understand the point about C compilers having more opportunities to take advantage of "undefined" in the spec, but Python implementations are getting cleverer about how to implement Python semantics as well, perhaps we should have some faith in their future cleverness and give them even a little bit of leeway. I don't imagine this little bit will actually be useful to them, but making this undefined will already help CPython by avoiding either an extra bytecode or a change in the opcodes. There are plenty of places where different Python implementations differ, and even careful observers had different ideas about how keys and values were ordered in dict displays ("I thought it was like a function call," vs, "I thought it was like an assignment"). We've gone out of our way to maintain backward compatibility with the implemented behavior before (ordering of dict keys, for example). The simplest change to make here is to update the reference and keep the implementation. --Ned. From ulrich.eckhardt at dominolaser.com Wed Nov 7 14:57:57 2012 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Wed, 07 Nov 2012 14:57:57 +0100 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <5091A31D.4000904@pearwood.info> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> Message-ID: <509A68E5.6020403@dominolaser.com> Am 31.10.2012 23:15, schrieb Steven D'Aprano: > On 01/11/12 06:57, anatoly techtonik wrote: > [...] >> UnboundLocalError: local variable 'FONT_NAMES' referenced before >> assignment [...] >> I wonder if this message can be improved with a >> pointer to the concept on when global variables become local? > > If you have a suggestion for an improved message, please tell us. I'll take a shot, since I was also bitten by this when trying to learn Python. The important point is that some code earlier or later in that function does an assignment, so this location should be referenced in the error. How about: "UnboundLocalError: Local variable 'FONT_NAMES' (created on line 11) referenced before assignment." What I don't really like is the term "created". Maybe "implicitly created on line 11"? Or "implied by line 11"? Or how about "Local variable FONT_NAMES (implied by line 11) doesn't refer to an object", to avoid the multiple interpretations of the term "assignment"? =just my 2cc= Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstra???e 75a, 22547 Hamburg, Deutschland Gesch???ftsf???hrer: Hans Robert Dapprich, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.dominolaser.com ************************************************************************************** Diese E-Mail einschlie???lich s???mtlicher Anh???nge ist nur f???r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf???nger sein sollten. Die E-Mail ist in diesem Fall zu l???schen und darf weder gelesen, weitergeleitet, ver???ffentlicht oder anderweitig benutzt werden. E-Mails k???nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ???nderungen enthalten. Domino Laser GmbH ist f???r diese Folgen nicht verantwortlich. ************************************************************************************** From ncoghlan at gmail.com Wed Nov 7 15:11:41 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Nov 2012 00:11:41 +1000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509A504C.10807@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> Message-ID: <CADiSq7cB3w+h=_-CMT_9YN65urQ+VXRxf7J3m=3KKvUPsFE3qg@mail.gmail.com> On Wed, Nov 7, 2012 at 10:13 PM, Ned Batchelder <ned at nedbatchelder.com> wrote: > There are plenty of places where different Python implementations differ, > and even careful observers had different ideas about how keys and values > were ordered in dict displays ("I thought it was like a function call," vs, > "I thought it was like an assignment"). We've gone out of our way to > maintain backward compatibility with the implemented behavior before > (ordering of dict keys, for example). The simplest change to make here is > to update the reference and keep the implementation. "The implementation is right, the docs are wrong" sounds good to me, as it's easy to justify the out of order evaluation in terms of the equivalent item assignment statements: x = {a:b, c:d} vs x = {} x[a] = b x[c] = d That relationship is quite logical given that (ignoring namespace details) dict construction from a display [1] pretty much does the equivalent of: result = {} for key_expr, val_expr in display_entries: result[eval(key_expr)] = eval(val_expr) This comment [2] from the dict comprehension implementation makes it explicit that the behaviour of the equivalent Python item assignment code was taken to be normative. [1] http://hg.python.org/cpython/file/default/Python/compile.c#l3319 [2] http://hg.python.org/cpython/file/default/Python/compile.c#l3020 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From sam.partington at gmail.com Wed Nov 7 15:15:34 2012 From: sam.partington at gmail.com (Sam Partington) Date: Wed, 7 Nov 2012 14:15:34 +0000 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <509A68E5.6020403@dominolaser.com> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> <509A68E5.6020403@dominolaser.com> Message-ID: <CABuPkmThPu929Gek0g85UOb+AoabFTv-7TGcMrrLpAnaeZq+XQ@mail.gmail.com> On 7 November 2012 13:57, Ulrich Eckhardt <ulrich.eckhardt at dominolaser.com> wrote: > Am 31.10.2012 23:15, schrieb Steven D'Aprano: > I'll take a shot, since I was also bitten by this when trying to learn > Python. The important point is that some code earlier or later in that > function does an assignment, so this location should be referenced in the > error. +1 > How about: > > "UnboundLocalError: Local variable 'FONT_NAMES' (created on > line 11) referenced before assignment." > > What I don't really like is the term "created". Maybe "implicitly created on > line 11"? Or "implied by line 11"? Or how about "Local variable FONT_NAMES > (implied by line 11) doesn't refer to an object", to avoid the multiple > interpretations of the term "assignment"? Why not make use of the well defined words already there in the error message, but at the line number "UnboundLocalError: Local variable 'FONT_NAMES' referenced before it has been assigned to on line 11." Sam From rosuav at gmail.com Wed Nov 7 15:16:01 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Nov 2012 01:16:01 +1100 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7cB3w+h=_-CMT_9YN65urQ+VXRxf7J3m=3KKvUPsFE3qg@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CADiSq7cB3w+h=_-CMT_9YN65urQ+VXRxf7J3m=3KKvUPsFE3qg@mail.gmail.com> Message-ID: <CAPTjJmqJXATbW8mfC9VO5fCdeFbH8E-30y1POcDv0ohVBGTOwQ@mail.gmail.com> On Thu, Nov 8, 2012 at 1:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > "The implementation is right, the docs are wrong" sounds good to me, > as it's easy to justify the out of order evaluation in terms of the > equivalent item assignment statements: What do other Pythons than CPython do currently? Or is it "The reference implementation is right, the docs are wrong"? ChrisA From skip at pobox.com Wed Nov 7 15:16:57 2012 From: skip at pobox.com (Skip Montanaro) Date: Wed, 7 Nov 2012 08:16:57 -0600 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <509A68E5.6020403@dominolaser.com> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> <509A68E5.6020403@dominolaser.com> Message-ID: <CANc-5Uyhs0w0VQ-hBAC2nyE0gsbCek=4vR3=H6KdzmKw15wFzA@mail.gmail.com> > How about: > > "UnboundLocalError: Local variable 'FONT_NAMES' (created on > line 11) referenced before assignment." > > What I don't really like is the term "created". Maybe "implicitly created > on line 11"? Or "implied by line 11"? Or how about "Local variable > FONT_NAMES (implied by line 11) doesn't refer to an object", to avoid the > multiple interpretations of the term "assignment"? It's been a long (long) while since I looked at the virtual machine implementation, but my guess is that at the point where the error is detected the system won't know what line number corresponds to the assignment. There might also be multiple assignments. How would you know which one to pick? As for a better word than "created", I would use "assigned." Skip From lrekucki at gmail.com Wed Nov 7 15:19:55 2012 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Wed, 7 Nov 2012 15:19:55 +0100 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAPTjJmqJXATbW8mfC9VO5fCdeFbH8E-30y1POcDv0ohVBGTOwQ@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CADiSq7cB3w+h=_-CMT_9YN65urQ+VXRxf7J3m=3KKvUPsFE3qg@mail.gmail.com> <CAPTjJmqJXATbW8mfC9VO5fCdeFbH8E-30y1POcDv0ohVBGTOwQ@mail.gmail.com> Message-ID: <CAEZs-ELt5-LZ0kBobQF7jaopZw+yuADp3AVCxySxhZZOne9juQ@mail.gmail.com> On 7 November 2012 15:16, Chris Angelico <rosuav at gmail.com> wrote: > On Thu, Nov 8, 2012 at 1:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> "The implementation is right, the docs are wrong" sounds good to me, >> as it's easy to justify the out of order evaluation in terms of the >> equivalent item assignment statements: > > What do other Pythons than CPython do currently? Or is it "The > reference implementation is right, the docs are wrong"? PyPy and IronPython are the same as CPython. Only Jython (both 2.5 and 2.7a) follows the docs. Regards, ?ukasz Rekucki From rdmurray at bitdance.com Wed Nov 7 15:20:19 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 07 Nov 2012 09:20:19 -0500 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <509A68E5.6020403@dominolaser.com> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> <509A68E5.6020403@dominolaser.com> Message-ID: <20121107142019.BD46025006F@webabinitio.net> On Wed, 07 Nov 2012 14:57:57 +0100, Ulrich Eckhardt <ulrich.eckhardt at dominolaser.com> wrote: > Am 31.10.2012 23:15, schrieb Steven D'Aprano: > > On 01/11/12 06:57, anatoly techtonik wrote: > > [...] > >> UnboundLocalError: local variable 'FONT_NAMES' referenced before > >> assignment > [...] > >> I wonder if this message can be improved with a > >> pointer to the concept on when global variables become local? > > > > If you have a suggestion for an improved message, please tell us. > > I'll take a shot, since I was also bitten by this when trying to learn > Python. The important point is that some code earlier or later in that > function does an assignment, so this location should be referenced in > the error. > > How about: > > "UnboundLocalError: Local variable 'FONT_NAMES' (created on > line 11) referenced before assignment." > > What I don't really like is the term "created". Maybe "implicitly > created on line 11"? Or "implied by line 11"? Or how about "Local > variable FONT_NAMES (implied by line 11) doesn't refer to an object", to > avoid the multiple interpretations of the term "assignment"? The problem is that when Python executes the statement that raises the error it has no idea where the assignment was done. All it knows is that the variable is local. Keeping track of the location of every assignment that makes a variable local and writing it in to the .pyc file is a very non-trivial change to how the Python bytecode compiler works, I think, and probably not worth the overhead in order to improve this error message. (And note that raising an error at compile time would change Python's semantics.) --David From guido at python.org Wed Nov 7 15:54:32 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Nov 2012 06:54:32 -0800 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509A504C.10807@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> Message-ID: <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> On Wed, Nov 7, 2012 at 4:13 AM, Ned Batchelder <ned at nedbatchelder.com> wrote: > I think it's unfortunate that the current patch in the referenced bug ( > http://bugs.python.org/issue11205 ) fixes the "problem" by adding one more > bytecode to the compiled Python. The other alternative seems to be changing > the meaning of two opcodes. Neither of these are great alternatives. I know > we don't promise to maintain backward compatibility in the meanings of > opcodes, but I'll bet actual code will be broken by changing the meaning of > STORE_MAP and MAP_ADD. Slowing down dict displays (just slightly) to get > this right seems unfortunate also. I agree this would be unfortunate and I recommend that you add this to the bug. I agree that we should be *very* conservative in changing the meaning of existing opcodes (adding new one is a different story). But it was fixed before (http://bugs.python.org/issue448679 has a simple fix that doesn't change opcode meanings and was applied in 2001) -- what happened? > If the bug report is accurate, CPython and the reference manual have > disagreed since Python 2.5, and many of us are now surprised to hear it, > which means there can't have been much broken code. Give that it was discussed before and fixed before, I think the intent is clear: we should fix the code, not the docs. > I understand the point about C compilers having more opportunities to take > advantage of "undefined" in the spec, but Python implementations are getting > cleverer about how to implement Python semantics as well, perhaps we should > have some faith in their future cleverness and give them even a little bit > of leeway. Hm. I really don't think that is a good development for Python to compromise in the area of expression evaluation order where side effects are involved. A good compiler should be able to detect the absence of potential side effects. E.g. it might reorder when only constants and simple variable references are present, or (in the case of a JIT, which has run-time type information) when it knows enough about the types of the operands involved to determine that operations like getattr or getitem are guaranteed side-effect-free. > I don't imagine this little bit will actually be useful to them, > but making this undefined will already help CPython by avoiding either an > extra bytecode or a change in the opcodes. I haven't looked at the proposed fixes, but I think correctness is more important than saving an extra bytecode (OTOH keeping the set of opcodes the same trumps both). I can't imagine that this extra opcode will be significant in many cases. > There are plenty of places where different Python implementations differ, > and even careful observers had different ideas about how keys and values > were ordered in dict displays ("I thought it was like a function call," vs, > "I thought it was like an assignment"). We've gone out of our way to > maintain backward compatibility with the implemented behavior before > (ordering of dict keys, for example). Not sure what you're referencing here -- didn't we just start randomizing hashing? > The simplest change to make here is > to update the reference and keep the implementation. In this particular case I disagree. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Nov 7 16:01:59 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Nov 2012 01:01:59 +1000 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <509A68E5.6020403@dominolaser.com> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> <509A68E5.6020403@dominolaser.com> Message-ID: <CADiSq7e6FWbccEB-NOHx-6ZUmG+AVnJVpnpYfkTu6Q1SFaNCAA@mail.gmail.com> On Wed, Nov 7, 2012 at 11:57 PM, Ulrich Eckhardt <ulrich.eckhardt at dominolaser.com> wrote: > How about: > > "UnboundLocalError: Local variable 'FONT_NAMES' (created on > line 11) referenced before assignment." > > What I don't really like is the term "created". Maybe "implicitly created on > line 11"? Or "implied by line 11"? Or how about "Local variable FONT_NAMES > (implied by line 11) doesn't refer to an object", to avoid the multiple > interpretations of the term "assignment"? Unfortunately, we don't track the information we would need in order to emit that kind of error message. However, you did give me an idea: I believe the compiler is actually in a position to emit SyntaxWarning for functions that have a high chance of triggering UnboundLocalError when called. With output pointing to *both* problematic lines, beginners should stand a better chance of figuring out what is going on. (http://bugs.python.org/issue16429) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Wed Nov 7 16:06:15 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Nov 2012 02:06:15 +1100 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> Message-ID: <CAPTjJmr6bjdXVDS5D=eZbgsa3cbE3ZOxot1poz78bYOoUddKWQ@mail.gmail.com> On Thu, Nov 8, 2012 at 1:54 AM, Guido van Rossum <guido at python.org> wrote: > On Wed, Nov 7, 2012 at 4:13 AM, Ned Batchelder <ned at nedbatchelder.com> wrote: >> We've gone out of our way to >> maintain backward compatibility with the implemented behavior before >> (ordering of dict keys, for example). > > Not sure what you're referencing here -- didn't we just start > randomizing hashing? Hash randomization could have been quietly introduced as a security fix and backported to old versions. But since applications might have depended on dict iteration order, it wasn't - old versions of Python won't randomize hashes unless you set an environment variable. That's taking care of old code, even when that old code is depending on non-spec behaviour. ChrisA From ncoghlan at gmail.com Wed Nov 7 16:12:37 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 8 Nov 2012 01:12:37 +1000 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> Message-ID: <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> On Thu, Nov 8, 2012 at 12:54 AM, Guido van Rossum <guido at python.org> wrote: > On Wed, Nov 7, 2012 at 4:13 AM, Ned Batchelder <ned at nedbatchelder.com> wrote: >> If the bug report is accurate, CPython and the reference manual have >> disagreed since Python 2.5, and many of us are now surprised to hear it, >> which means there can't have been much broken code. > > Give that it was discussed before and fixed before, I think the intent > is clear: we should fix the code, not the docs. Almost certainly, it was broken in the migration to the AST compiler and there was no regression test to pick up the change. > I haven't looked at the proposed fixes, but I think correctness is > more important than saving an extra bytecode (OTOH keeping the set of > opcodes the same trumps both). I can't imagine that this extra opcode > will be significant in many cases. Since you've indicated the implementation is in the wrong here and you also want to preserve opcode semantics, I think Skip's patch is correct, but also needs to be applied to dict comprehensions (now we have them). The extra bytecode is only ROT_TWO, which is one of the cheapest we have kicking around :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Wed Nov 7 16:17:41 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Nov 2012 07:17:41 -0800 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> Message-ID: <CAP7+vJKnN9DDH=vTGs0kQSBwiFA72F8gu6aiOXX-+P4ikEF-jQ@mail.gmail.com> Ok, somebody go for it! (Also please refer to my pronouncement in the bug -- I've gotta run.) On Wed, Nov 7, 2012 at 7:12 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Thu, Nov 8, 2012 at 12:54 AM, Guido van Rossum <guido at python.org> > wrote: > > On Wed, Nov 7, 2012 at 4:13 AM, Ned Batchelder <ned at nedbatchelder.com> > wrote: > >> If the bug report is accurate, CPython and the reference manual have > >> disagreed since Python 2.5, and many of us are now surprised to hear it, > >> which means there can't have been much broken code. > > > > Give that it was discussed before and fixed before, I think the intent > > is clear: we should fix the code, not the docs. > > Almost certainly, it was broken in the migration to the AST compiler > and there was no regression test to pick up the change. > > > I haven't looked at the proposed fixes, but I think correctness is > > more important than saving an extra bytecode (OTOH keeping the set of > > opcodes the same trumps both). I can't imagine that this extra opcode > > will be significant in many cases. > > Since you've indicated the implementation is in the wrong here and you > also want to preserve opcode semantics, I think Skip's patch is > correct, but also needs to be applied to dict comprehensions (now we > have them). The extra bytecode is only ROT_TWO, which is one of the > cheapest we have kicking around :) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/f71543c4/attachment.html> From storchaka at gmail.com Wed Nov 7 18:08:25 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 07 Nov 2012 19:08:25 +0200 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> Message-ID: <k7e4ih$idp$1@ger.gmane.org> On 07.11.12 17:12, Nick Coghlan wrote: > Since you've indicated the implementation is in the wrong here and you > also want to preserve opcode semantics, I think Skip's patch is > correct, but also needs to be applied to dict comprehensions (now we > have them). The extra bytecode is only ROT_TWO, which is one of the > cheapest we have kicking around :) Not only to dict comprehensions, but also to item assignments. It will be weird if a dict comprehension and a plain loop will be inconsistent. From tjreedy at udel.edu Wed Nov 7 20:08:33 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Nov 2012 14:08:33 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> Message-ID: <k7ebjk$lrl$1@ger.gmane.org> On 11/7/2012 9:54 AM, Guido van Rossum wrote: > Hm. I really don't think that is a good development for Python to > compromise in the area of expression evaluation order where side > effects are involved. I agreee. I think Python's simple left to right evaluation order is one of its virtues. > A good compiler should be able to detect the > absence of potential side effects. E.g. it might reorder when only > constants and simple variable references are present, or (in the case > of a JIT, which has run-time type information) when it knows enough > about the types of the operands involved to determine that operations > like getattr or getitem are guaranteed side-effect-free. I call this the 'as-if' rule: the compiler can take shortcuts if the result is 'as-if' no shortcut. -- Terry Jan Reedy From tjreedy at udel.edu Wed Nov 7 20:19:06 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Nov 2012 14:19:06 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <CAP7+vJKnN9DDH=vTGs0kQSBwiFA72F8gu6aiOXX-+P4ikEF-jQ@mail.gmail.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <CAP7+vJKnN9DDH=vTGs0kQSBwiFA72F8gu6aiOXX-+P4ikEF-jQ@mail.gmail.com> Message-ID: <k7ec7e$roa$1@ger.gmane.org> On 11/7/2012 10:17 AM, Guido van Rossum wrote: > Ok, somebody go for it! (Also please refer to my pronouncement in the > bug -- I've gotta run.) Done. http://bugs.python.org/issue11205?@ok_message=msg 175120 -- Terry Jan Reedy From ned at nedbatchelder.com Wed Nov 7 22:39:28 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 07 Nov 2012 16:39:28 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <k7e4ih$idp$1@ger.gmane.org> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> Message-ID: <509AD510.5060200@nedbatchelder.com> On 11/7/2012 12:08 PM, Serhiy Storchaka wrote: > On 07.11.12 17:12, Nick Coghlan wrote: >> Since you've indicated the implementation is in the wrong here and you >> also want to preserve opcode semantics, I think Skip's patch is >> correct, but also needs to be applied to dict comprehensions (now we >> have them). The extra bytecode is only ROT_TWO, which is one of the >> cheapest we have kicking around :) > > Not only to dict comprehensions, but also to item assignments. It > will be weird if a dict comprehension and a plain loop will be > inconsistent. > > Just to be clear: the reference guide says that the behavior *SHOULD BE* (but is not yet) this: Python 3.3.0 >>> {print("a"):print("b")} a b {None: None} >>> d = {} >>> d[print("a")] = print("b") b a >>> Is this or is this not "weird" to you? --Ned. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/2474e3be/attachment.html> From guido at python.org Wed Nov 7 23:03:07 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 7 Nov 2012 14:03:07 -0800 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509AD510.5060200@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> <509AD510.5060200@nedbatchelder.com> Message-ID: <CAP7+vJL6t6dKk+zOw9jMaH8AQ0JsK4GF2iR6Q4=uMmeUdSjBJQ@mail.gmail.com> The dict display is considered an *expression* and thus must follow the L2R rule. The assignment is explicitly covered by the R2L rule for assignments (only). Weird or not, those are the rules, and I don't want to change them. On Wed, Nov 7, 2012 at 1:39 PM, Ned Batchelder <ned at nedbatchelder.com> wrote: > > On 11/7/2012 12:08 PM, Serhiy Storchaka wrote: > > On 07.11.12 17:12, Nick Coghlan wrote: > > Since you've indicated the implementation is in the wrong here and you > also want to preserve opcode semantics, I think Skip's patch is > correct, but also needs to be applied to dict comprehensions (now we > have them). The extra bytecode is only ROT_TWO, which is one of the > cheapest we have kicking around :) > > > Not only to dict comprehensions, but also to item assignments. It will be > weird if a dict comprehension and a plain loop will be inconsistent. > > > Just to be clear: the reference guide says that the behavior *SHOULD BE* > (but is not yet) this: > > Python 3.3.0 >>>> {print("a"):print("b")} > a > b > {None: None} >>>> d = {} >>>> d[print("a")] = print("b") > b > a >>>> > > Is this or is this not "weird" to you? > > --Ned. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ned%40nedbatchelder.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Wed Nov 7 23:11:40 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Nov 2012 17:11:40 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509AD510.5060200@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> <509AD510.5060200@nedbatchelder.com> Message-ID: <k7emb0$okb$1@ger.gmane.org> On 11/7/2012 4:39 PM, Ned Batchelder wrote: > Just to be clear: the reference guide says that the behavior *SHOULD BE* > (but is not yet) this: > > Python 3.3.0 > >>> {print("a"):print("b")} > a > b > {None: None} > >>> d = {} > >>> d[print("a")] = print("b") > b > a > >>> > > Is this or is this not "weird" to you? Not weird. Expressions and assignment targets are each consistently evaluated left to right (except as *necessarily* alter by precedence), with expressions evaluated before targets. What is weird -- to me ;-) -- is using side-effects in either example above. -- Terry Jan Reedy From alexandre at peadrop.com Wed Nov 7 23:21:28 2012 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Wed, 7 Nov 2012 14:21:28 -0800 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <20121103181343.27e690d7@pitrou.net> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> Message-ID: <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> The Unicode code points in the U+DC00-DFFF<http://www.unicode.org/charts/PDF/UDC00.pdf> range (low surrogate area) can't be encoded in UTF-8. Quoting from RFC 3629<http://tools.ietf.org/html/rfc3629> : *The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF, which are reserved for use with the UTF-16 encoding form (as surrogate pairs) and do not directly represent characters.* It looks like this test was doing something specific with regards to this. So, I am curious as well about this change. On Sat, Nov 3, 2012 at 10:13 AM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sat, 3 Nov 2012 13:37:48 +0100 (CET) > andrew.svetlov <python-checkins at python.org> wrote: > > http://hg.python.org/cpython/rev/95d1adf144ee > > changeset: 80187:95d1adf144ee > > user: Andrew Svetlov <andrew.svetlov at gmail.com> > > date: Sat Nov 03 14:37:37 2012 +0200 > > summary: > > Issue #16218: skip test if filesystem doesn't support required encoding > > > > files: > > Lib/test/test_cmd_line_script.py | 7 ++++++- > > 1 files changed, 6 insertions(+), 1 deletions(-) > > > > > > diff --git a/Lib/test/test_cmd_line_script.py > b/Lib/test/test_cmd_line_script.py > > --- a/Lib/test/test_cmd_line_script.py > > +++ b/Lib/test/test_cmd_line_script.py > > @@ -366,7 +366,12 @@ > > def test_non_utf8(self): > > # Issue #16218 > > with temp_dir() as script_dir: > > - script_basename = '\udcf1\udcea\udcf0\udce8\udcef\udcf2' > > + script_basename = '\u0441\u043a\u0440\u0438\u043f\u0442' > > Why exactly did you change the tested name here? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/alexandre%40peadrop.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121107/0c9cbc34/attachment-0001.html> From steve at pearwood.info Wed Nov 7 23:29:27 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 08 Nov 2012 09:29:27 +1100 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509AD510.5060200@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> <509AD510.5060200@nedbatchelder.com> Message-ID: <509AE0C7.5080304@pearwood.info> On 08/11/12 08:39, Ned Batchelder wrote: > Just to be clear: the reference guide says that the behavior *SHOULD BE* (but is not yet) this: > > Python 3.3.0 >> >> {print("a"):print("b")} > a > b > {None: None} That was the behaviour of Python 2.4: py> def pr(x): ... print x ... py> {pr(1): pr(2), pr(3): pr(4)} 1 2 3 4 {None: None} 2.5 changed to the behaviour seen now, that is, it prints 2 1 4 3 in that order. >> >> d = {} >> >> d[print("a")] = print("b") > b > a > > Is this or is this not "weird" to you? Not weird to me. The first case has no assignment, so it operates left to right without exception. The second case has an assignment, so it operates left to right with a single exception, the right hand side of the assignment is evaluated before the left hand side(s). This gives a single, intuitive[1] order of evaluation (left to right), with the fewest number of exceptions necessary[2]. Using Python 2.4 again: py> d = {} py> d[pr(1)] = d[pr(2)] = d[pr(3)] = pr(4) is pr(5) 4 5 1 2 3 [1] Well, intuitive to those whose native language reads left to right. [2] I assume it is necessary. -- Steven From tjreedy at udel.edu Wed Nov 7 23:42:58 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 07 Nov 2012 17:42:58 -0500 Subject: [Python-Dev] urlretrieve regression no.2, now in Python 3.3 In-Reply-To: <CAPkN8xJtZ71gB3LdgvmQAy0cxKnwsCc7yMhpKj7c4Mb4T8jgmQ@mail.gmail.com> References: <CAPkN8xJtZ71gB3LdgvmQAy0cxKnwsCc7yMhpKj7c4Mb4T8jgmQ@mail.gmail.com> Message-ID: <k7eo5m$6pc$1@ger.gmane.org> On 11/7/2012 5:57 AM, anatoly techtonik wrote: > urlretrieve has a callback parameter, which takes function with the > following prototype: > > def callback(block_number, block_size, total_size): > pass > > Where block_size was constant and block_size*block_number gave an exact > number of transferred bytes. The 3.2 and 3.3 docs both say "The third argument, if present, is a hook function that will be called once on establishment of the network connection and once after each block read thereafter. The hook will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. The third argument may be -1 on older FTP servers which do not return a file size in response to a retrieval request." The word 'constant' does not appear. The product is still the same. > Recent change in Python 3.3 changed the semantic of block_size and broke > my `wget` package. http://bugs.python.org/issue16409 The only change is that blocksize is now reported as 0 before any blocks have been transmitted. It is a side-effect of commits for http://bugs.python.org/issue10050. -- Terry Jan Reedy From victor.stinner at gmail.com Wed Nov 7 23:47:13 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 7 Nov 2012 23:47:13 +0100 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> Message-ID: <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> 2012/11/7 Alexandre Vassalotti <alexandre at peadrop.com>: > The Unicode code points in the U+DC00-DFFF range (low surrogate area) can't > be encoded in UTF-8. Quoting from RFC 3629: > > The definition of UTF-8 prohibits encoding character numbers between U+D800 > and U+DFFF, which are reserved for use with the UTF-16 encoding form (as > surrogate pairs) and do not directly represent characters. > > > It looks like this test was doing something specific with regards to this. > So, I am curious as well about this change. os.fsencode() uses the surrogateescape error handler (PEP 393) on UNIX. >>> os.fsencode('\udcf1\udcea\udcf0\udce8\udcef\udcf2') b'\xf1\xea\xf0\xe8\xef\xf2' I replaced this arbitrary string (and other similar constant strings) with support.FS_NONASCII which is more portable (should be available on all locale encodings... except ASCII) and documented. I rewrote test_cmd_line_script.test_non_ascii() (and other tests) in Python 3.4 to use support.FS_NONASCII. This change should improve code coverage on heterogeneous environments. Victor From rdmurray at bitdance.com Thu Nov 8 00:08:45 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 07 Nov 2012 18:08:45 -0500 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> Message-ID: <20121107230845.7A7D225006F@webabinitio.net> On Wed, 07 Nov 2012 23:47:13 +0100, Victor Stinner <victor.stinner at gmail.com> wrote: > 2012/11/7 Alexandre Vassalotti <alexandre at peadrop.com>: > > The Unicode code points in the U+DC00-DFFF range (low surrogate area) can't > > be encoded in UTF-8. Quoting from RFC 3629: > > > > The definition of UTF-8 prohibits encoding character numbers between U+D800 > > and U+DFFF, which are reserved for use with the UTF-16 encoding form (as > > surrogate pairs) and do not directly represent characters. > > > > > > It looks like this test was doing something specific with regards to this. > > So, I am curious as well about this change. > > os.fsencode() uses the surrogateescape error handler (PEP 393) on UNIX. > > >>> os.fsencode('\udcf1\udcea\udcf0\udce8\udcef\udcf2') > b'\xf1\xea\xf0\xe8\xef\xf2' > > I replaced this arbitrary string (and other similar constant strings) > with support.FS_NONASCII which is more portable (should be available > on all locale encodings... except ASCII) and documented. > > I rewrote test_cmd_line_script.test_non_ascii() (and other tests) in > Python 3.4 to use support.FS_NONASCII. > > This change should improve code coverage on heterogeneous environments. Alexandre's point was that the string did not appear to be arbitrary, but rather appeared to specifically be a string containing surrogates. Is this not the case? --David From greg at krypto.org Thu Nov 8 00:48:06 2012 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 7 Nov 2012 15:48:06 -0800 Subject: [Python-Dev] urlretrieve regression no.2, now in Python 3.3 In-Reply-To: <k7eo5m$6pc$1@ger.gmane.org> References: <CAPkN8xJtZ71gB3LdgvmQAy0cxKnwsCc7yMhpKj7c4Mb4T8jgmQ@mail.gmail.com> <k7eo5m$6pc$1@ger.gmane.org> Message-ID: <CAGE7PN+3Sr4pgw_TD5JeLSqRhxb=PXBfFU-7ZPLbRtj4t5EcKA@mail.gmail.com> On Wed, Nov 7, 2012 at 2:42 PM, Terry Reedy <tjreedy at udel.edu> wrote: > On 11/7/2012 5:57 AM, anatoly techtonik wrote: >> >> urlretrieve has a callback parameter, which takes function with the >> following prototype: >> >> def callback(block_number, block_size, total_size): >> pass >> >> Where block_size was constant and block_size*block_number gave an exact >> number of transferred bytes. > > > The 3.2 and 3.3 docs both say "The third argument, if present, is a hook > function that will be called once on establishment of the network connection > and once after each block read thereafter. The hook will be passed three > arguments; a count of blocks transferred so far, a block size in bytes, and > the total size of the file. The third argument may be -1 on older FTP > servers which do not return a file size in response to a retrieval request." > > The word 'constant' does not appear. The product is still the same. > > >> Recent change in Python 3.3 changed the semantic of block_size and broke >> my `wget` package. http://bugs.python.org/issue16409 > > > The only change is that blocksize is now reported as 0 before any blocks > have been transmitted. It is a side-effect of commits for > http://bugs.python.org/issue10050. two changes. the first block size is 0. the last block size is the number of bytes read. It is a trivial two line change to undo this and I think it would be wise. It is difficult to even find documentation on what the arguments to that callback mean. Look up the urlretrieve() function docs in 2.7 and 3.3 and it doesn't tell you there or refer you to anywhere else to find out. I'm in favor of restoring and documenting the old behavior for this legacy urlretrieve API to make existing code being ported from 2.7 not break. -gps > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org From ned at nedbatchelder.com Thu Nov 8 02:11:21 2012 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 07 Nov 2012 20:11:21 -0500 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <k7emb0$okb$1@ger.gmane.org> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> <509AD510.5060200@nedbatchelder.com> <k7emb0$okb$1@ger.gmane.org> Message-ID: <509B06B9.6000406@nedbatchelder.com> On 11/7/2012 5:11 PM, Terry Reedy wrote: > On 11/7/2012 4:39 PM, Ned Batchelder wrote: > >> Just to be clear: the reference guide says that the behavior *SHOULD BE* >> (but is not yet) this: >> >> Python 3.3.0 >> >>> {print("a"):print("b")} >> a >> b >> {None: None} >> >>> d = {} >> >>> d[print("a")] = print("b") >> b >> a >> >>> >> >> Is this or is this not "weird" to you? > > Not weird. Expressions and assignment targets are each consistently > evaluated left to right (except as *necessarily* alter by precedence), > with expressions evaluated before targets. > Sorry, I should have been clearer: I was asking about weird not to say, "This is weird and should be changed!", but to get clarification from Serhiy about his statement, " It will be weird if a dict comprehension and a plain loop will be inconsistent." I honestly didn't know which behavior he considered inconsistent and therefore weird. --Ned. From stefan_ml at behnel.de Thu Nov 8 13:47:06 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 13:47:06 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation Message-ID: <k7g9k9$8bl$1@ger.gmane.org> Hi, I suspect that this will be put into a proper PEP at some point, but I'd like to bring this up for discussion first. This came out of issues 13429 and 16392. http://bugs.python.org/issue13429 http://bugs.python.org/issue16392 Stefan The problem =========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it's not trivial to correctly add the module to sys.modules either. We specifically run into this for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The proposal ============ I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple C level setup). Optionally, after creating the module (and this is the new part), the module init code can register a C callback function that will be called after setting up the module. Step 2: The shared library importer receives the module instance from the module init function, adds __file__, __path__, __package__ and friends to the module dict, and then checks for the callback. If non-NULL, it calls it to continue the module initialisation by user code. The callback ============ The callback is defined as follows:: int (*PyModule_init_callback)(PyObject* the_module, PyModuleInitContext* context) "PyModuleInitContext" is a struct that is meant mostly for making the callback more future proof by allowing additional parameters to be passed in. For now, I can see a use case for the following fields:: struct PyModuleInitContext { char* module_name; char* qualified_module_name; } Both names are encoded in UTF-8. As for the file path, I consider it best to retrieve it from the module's __file__ attribute as a Python string object to reduce filename encoding problems. Note that this struct argument is not strictly required, but given that this proposal would have been much simpler if the module init function had accepted such an argument in the first place, I consider it a good idea not to let this chance pass by again. The registration of the callback uses a new C-API function: int PyModule_SetInitFunction(PyObject* module, PyModule_init_callback callback) The function name uses "Set" instead of "Register" to make it clear that there is only one such function per module. An alternative would be a new module creation function "PyModule_Create3()" that takes the callback as third argument, in addition to what "PyModule_Create2()" accepts. This would require users to explicitly pass in the (second) version argument, which might be considered only a minor issue. Implementation ============== The implementation requires local changes to the extension module importer and a new C-API function. In order to store the callback, it should use a new field in the module object struct. Open questions ============== It is not clear how extensions should be handled that register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case, although not the import file path. Alternatively, the import machinery could use a stack to remember for which modules a callback was registered during the last init function call, set up all of them and then call their callbacks. It's not clear if this meets the intention of the user. Alternatives ============ 1) It would be possible to make extension modules optionally export another symbol, e.g. "PyInit2_modulename", that the shared library loader would call in addition to the required function "PyInit_modulename". This would remove the need for a new API that registers the above callback. The drawback is that it also makes it easier to write broken code because a Python version or implementation that does not support this second symbol would simply not call it, without error. The new C-API function would let the build fail instead if it is not supported. 2) The callback could be made available as a Python function in the module dict, thus also removing the need for an explicit registration API. However, this approach would add overhead to both sides, the importer code and the user provided module init code, as it would require additional dictionary handling and the implementation of a one-time Python function in user code. It would also suffer from the problem that missing support in the runtime would pass silently. 3) The callback could be registered statically in the PyModuleDef struct by adding a new field. This is not trivial to do in a backwards compatible way because the struct would grow longer without explicit initialisation by existing user code. Extending PyModuleDef_HEAD_INIT might be possible but would still break at least binary compatibility. 4) Pass a new context argument into the module init function that contains all information necessary to properly and completely set up the module at creation time. This would provide a much simpler and cleaner solution than the proposed solution. However, it will not be possible before Python 4 as it breaks backwards compatibility with all existing extension modules at both the source and binary level. From mal at egenix.com Thu Nov 8 14:01:27 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 08 Nov 2012 14:01:27 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <k7g9k9$8bl$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> Message-ID: <509BAD27.2030906@egenix.com> On 08.11.2012 13:47, Stefan Behnel wrote: > Hi, > > I suspect that this will be put into a proper PEP at some point, but I'd > like to bring this up for discussion first. This came out of issues 13429 > and 16392. > > http://bugs.python.org/issue13429 > > http://bugs.python.org/issue16392 > > Stefan > > > The problem > =========== > > Python modules and extension modules are not being set up in the same way. > For Python modules, the module is created and set up first, then the module > code is being executed. For extensions, i.e. shared libraries, the module > init function is executed straight away and does both the creation and > initialisation. This means that it knows neither the __file__ it is being > loaded from nor its package (i.e. its FQMN). This hinders relative imports > and resource loading. In Py3, it's also not being added to sys.modules, > which means that a (potentially transitive) re-import of the module will > really try to reimport it and thus run into an infinite loop when it > executes the module init function again. And without the FQMN, it's not > trivial to correctly add the module to sys.modules either. > > We specifically run into this for Cython generated modules, for which it's > not uncommon that the module init code has the same level of complexity as > that of any 'regular' Python module. Also, the lack of a FQMN and correct > file path hinders the compilation of __init__.py modules, i.e. packages, > especially when relative imports are being used at module init time. > > The proposal > ============ > > ... [callbacks] ... > > Alternatives > ============ > ... > 3) The callback could be registered statically in the PyModuleDef struct by > adding a new field. This is not trivial to do in a backwards compatible way > because the struct would grow longer without explicit initialisation by > existing user code. Extending PyModuleDef_HEAD_INIT might be possible but > would still break at least binary compatibility. I think the above is the cleaner approach than the callback mechanism. There's no problem in adding new slots to the end of the PyModuleDef struct - we've been doing that for years in many other structs :-) All you have to do is bump the Python API version number. (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 08 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From stefan_ml at behnel.de Thu Nov 8 14:20:27 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 14:20:27 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <509BAD27.2030906@egenix.com> References: <k7g9k9$8bl$1@ger.gmane.org> <509BAD27.2030906@egenix.com> Message-ID: <k7gbir$psq$1@ger.gmane.org> M.-A. Lemburg, 08.11.2012 14:01: > On 08.11.2012 13:47, Stefan Behnel wrote: >> I suspect that this will be put into a proper PEP at some point, but I'd >> like to bring this up for discussion first. This came out of issues 13429 >> and 16392. >> >> http://bugs.python.org/issue13429 >> >> http://bugs.python.org/issue16392 >> >> Stefan >> >> >> The problem >> =========== >> >> Python modules and extension modules are not being set up in the same way. >> For Python modules, the module is created and set up first, then the module >> code is being executed. For extensions, i.e. shared libraries, the module >> init function is executed straight away and does both the creation and >> initialisation. This means that it knows neither the __file__ it is being >> loaded from nor its package (i.e. its FQMN). This hinders relative imports >> and resource loading. In Py3, it's also not being added to sys.modules, >> which means that a (potentially transitive) re-import of the module will >> really try to reimport it and thus run into an infinite loop when it >> executes the module init function again. And without the FQMN, it's not >> trivial to correctly add the module to sys.modules either. >> >> We specifically run into this for Cython generated modules, for which it's >> not uncommon that the module init code has the same level of complexity as >> that of any 'regular' Python module. Also, the lack of a FQMN and correct >> file path hinders the compilation of __init__.py modules, i.e. packages, >> especially when relative imports are being used at module init time. >> >> The proposal >> ============ >> >> ... [callbacks] ... >> >> Alternatives >> ============ >> ... >> 3) The callback could be registered statically in the PyModuleDef struct by >> adding a new field. This is not trivial to do in a backwards compatible way >> because the struct would grow longer without explicit initialisation by >> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but >> would still break at least binary compatibility. > > I think the above is the cleaner approach than the callback mechanism. Oh, definitely. > There's no problem in adding new slots to the end of the PyModuleDef struct > - we've been doing that for years in many other structs :-) > > All you have to do is bump the Python API version number. > > (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) The difference is that this specific struct is provided by user code and (typically) initialised statically. There is no guarantee that user code that does not expect the additional field will initialise it to 0. Failing that, I don't see how we could trust its value in any way. Stefan From stefan_ml at behnel.de Thu Nov 8 14:51:54 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 14:51:54 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <k7gbir$psq$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> <509BAD27.2030906@egenix.com> <k7gbir$psq$1@ger.gmane.org> Message-ID: <k7gddo$b2v$1@ger.gmane.org> Stefan Behnel, 08.11.2012 14:20: > M.-A. Lemburg, 08.11.2012 14:01: >> On 08.11.2012 13:47, Stefan Behnel wrote: >>> I suspect that this will be put into a proper PEP at some point, but I'd >>> like to bring this up for discussion first. This came out of issues 13429 >>> and 16392. >>> >>> http://bugs.python.org/issue13429 >>> >>> http://bugs.python.org/issue16392 >>> >>> Stefan >>> >>> >>> The problem >>> =========== >>> >>> Python modules and extension modules are not being set up in the same way. >>> For Python modules, the module is created and set up first, then the module >>> code is being executed. For extensions, i.e. shared libraries, the module >>> init function is executed straight away and does both the creation and >>> initialisation. This means that it knows neither the __file__ it is being >>> loaded from nor its package (i.e. its FQMN). This hinders relative imports >>> and resource loading. In Py3, it's also not being added to sys.modules, >>> which means that a (potentially transitive) re-import of the module will >>> really try to reimport it and thus run into an infinite loop when it >>> executes the module init function again. And without the FQMN, it's not >>> trivial to correctly add the module to sys.modules either. >>> >>> We specifically run into this for Cython generated modules, for which it's >>> not uncommon that the module init code has the same level of complexity as >>> that of any 'regular' Python module. Also, the lack of a FQMN and correct >>> file path hinders the compilation of __init__.py modules, i.e. packages, >>> especially when relative imports are being used at module init time. >>> >>> The proposal >>> ============ >>> >>> ... [callbacks] ... >>> >>> Alternatives >>> ============ >>> ... >>> 3) The callback could be registered statically in the PyModuleDef struct by >>> adding a new field. This is not trivial to do in a backwards compatible way >>> because the struct would grow longer without explicit initialisation by >>> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but >>> would still break at least binary compatibility. >> >> I think the above is the cleaner approach than the callback mechanism. > > Oh, definitely. > > >> There's no problem in adding new slots to the end of the PyModuleDef struct >> - we've been doing that for years in many other structs :-) >> >> All you have to do is bump the Python API version number. >> >> (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) > > The difference is that this specific struct is provided by user code and > (typically) initialised statically. There is no guarantee that user code > that does not expect the additional field will initialise it to 0. Failing > that, I don't see how we could trust its value in any way. Hmm - you're actually right. In C, uninitialised fields in a static struct are set to 0 automatically. Same case as the type structs. That makes your objection perfectly valid. I'll rewrite and shorten the proposal. Thanks! Stefan From stefan_ml at behnel.de Thu Nov 8 15:32:34 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 15:32:34 +0100 Subject: [Python-Dev] Update - Re: Make extension module initialisation more like Python module initialisation In-Reply-To: <k7g9k9$8bl$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> Message-ID: <k7gfq0$177$1@ger.gmane.org> Hi, here's an updated proposal, adopting Marc-Andre's improvement that uses a new field in the PyModuleDef struct to register the callback. Note that this change no longer keeps up binary compatibility, which may or may not be acceptable for Python 3.4. Stefan The problem =========== Python modules and extension modules are not being set up in the same way. For Python modules, the module is created and set up first, then the module code is being executed. For extensions, i.e. shared libraries, the module init function is executed straight away and does both the creation and initialisation. This means that it knows neither the __file__ it is being loaded from nor its package (i.e. its FQMN). This hinders relative imports and resource loading. In Py3, it's also not being added to sys.modules, which means that a (potentially transitive) re-import of the module will really try to reimport it and thus run into an infinite loop when it executes the module init function again. And without the FQMN, it's not trivial to correctly add the module to sys.modules either. We specifically run into this for Cython generated modules, for which it's not uncommon that the module init code has the same level of complexity as that of any 'regular' Python module. Also, the lack of a FQMN and correct file path hinders the compilation of __init__.py modules, i.e. packages, especially when relative imports are being used at module init time. The proposal ============ I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple C level setup). Additionally, and this is the new part, the module init code can register a C callback function in its PyModuleDef struct that will be called after setting up the module. Step 2: The shared library importer receives the module instance from the module init function, adds __file__, __path__, __package__ and friends to the module dict, and then checks for the callback. If non-NULL, it calls it to continue the module initialisation by user code. The callback ============ The callback is defined as follows:: int (*PyModule_init_callback)(PyObject* the_module, PyModuleInitContext* context) "PyModuleInitContext" is a struct that is meant mostly for making the callback more future proof by allowing additional parameters to be passed in. For now, I can see a use case for the following fields:: struct PyModuleInitContext { char* module_name; char* qualified_module_name; } Both names are encoded in UTF-8. As for the file path, I consider it best to retrieve it from the module's __file__ attribute as a Python string object to reduce filename encoding problems. Note that this struct argument is not strictly required (it could be a simple "inquiry" function), but given that this proposal would have been much simpler if the module init function had accepted such an argument in the first place, I consider it a good idea not to let this chance pass by again. The counter arguments would be "keep it simple" and "we already pass in the whole module (and its dict) anyway". Up for debate! The registration of the callback uses a new field "m_init" in the PyModuleDef struct:: typedef struct PyModuleDef{ PyModuleDef_Base m_base; const char* m_name; const char* m_doc; Py_ssize_t m_size; PyMethodDef *m_methods; inquiry m_reload; traverseproc m_traverse; inquiry m_clear; freefunc m_free; /* --- original fields up to here */ PyModule_init_callback m_init; /* post-setup init callback */ } PyModuleDef; Implementation ============== The implementation requires local changes to the extension module importer and a new field in the PyModuleDef struct. Open questions ============== It is not clear how extensions should be handled that register more than one module in their module init function, e.g. compiled packages. One possibility would be to leave the setup to the user, who would have to know all FQMNs anyway in this case, although not the import file path. Alternatively, the import machinery could use a stack to remember for which modules a callback was registered during the last init function call, set up all of them and then call their callbacks. It's not clear if this meets the intention of the user. It's not guaranteed that all of these modules will be related to the module that registered them, in the sense that they should receive the same setup. The best way to fix this correctly might be to make users pass the setup explicitly into the module creation functions in Python 4 (see alternatives below), so that the setup and sys.modules registration can happen directly at this point. Alternatives ============ 1) It would be possible to make extension modules optionally export another symbol, e.g. "PyInit2_modulename", that the shared library loader would call in addition to the required function "PyInit_modulename". This would keep up binary compatibility. The drawback is that it also makes it easier to write broken code because a Python version or implementation that does not support this second symbol would simply not call it, without error. The new struct field would let the build fail instead if it is not supported. 2) The callback could be made available as a Python function in the module dict, thus also removing the need for an explicit registration API. However, this approach would add overhead to both sides, the importer code and the user provided module init code, as it would require additional dictionary handling and the implementation of a one-time Python function in user code. It would also suffer from the problem that missing support in the runtime would pass silently. 3) The original proposal used a new C-API function to register the callback explicitly, as opposed to extending the PyModuleDef struct. This has the advantage of keeping up binary compatibility with existing Py3.3 extensions. It has the disadvantage of adding another indirection to the setup procedure where a static function pointer would suffice. 4) Pass a new context argument into the module init function that contains all information necessary to properly and completely set up the module at creation time. This would provide a much simpler and cleaner solution than the proposed solution. However, it will not be possible before Python 4 as it breaks backwards compatibility with all existing extension modules at both the source and binary level. From brett at python.org Thu Nov 8 15:41:30 2012 From: brett at python.org (Brett Cannon) Date: Thu, 8 Nov 2012 09:41:30 -0500 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <k7g9k9$8bl$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> Message-ID: <CAP1=2W6W1LwY00B+ZJMFTrBcH2_iQ=MGghgxn=nNMc7h0c4sww@mail.gmail.com> On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Hi, > > I suspect that this will be put into a proper PEP at some point, but I'd > like to bring this up for discussion first. This came out of issues 13429 > and 16392. > > http://bugs.python.org/issue13429 > > http://bugs.python.org/issue16392 > > Stefan > > > The problem > =========== > > Python modules and extension modules are not being set up in the same way. > For Python modules, the module is created and set up first, then the module > code is being executed. For extensions, i.e. shared libraries, the module > init function is executed straight away and does both the creation and > initialisation. This means that it knows neither the __file__ it is being > loaded from nor its package (i.e. its FQMN). This hinders relative imports > and resource loading. In Py3, it's also not being added to sys.modules, > which means that a (potentially transitive) re-import of the module will > really try to reimport it and thus run into an infinite loop when it > executes the module init function again. And without the FQMN, it's not > trivial to correctly add the module to sys.modules either. > > We specifically run into this for Cython generated modules, for which it's > not uncommon that the module init code has the same level of complexity as > that of any 'regular' Python module. Also, the lack of a FQMN and correct > file path hinders the compilation of __init__.py modules, i.e. packages, > especially when relative imports are being used at module init time. > Or to put it another way, importlib doesn't give you a nice class to inherit from which will handle all of the little details of creating a blank module (or fetching from sys.modules if you are reloading), setting __file__, __cached__, __package__, __name__, __loader__, and (optionally) __path__ for you, and then cleaning up if something goes wrong. It's a pain to do all of this yourself and to get all the details right (i.e. there's a reason that @importlib.util.module_for_loader exists). > > The proposal > ============ > > I propose to split the extension module initialisation into two steps in > Python 3.4, in a backwards compatible way. > > Step 1: The current module init function can be reduced to just creating > the module instance and returning it (and potentially doing some simple C > level setup). Optionally, after creating the module (and this is the new > part), the module init code can register a C callback function that will be > called after setting up the module. > Why even bother with the module creation? Why can't Python do that as well and then call the callback? > > Step 2: The shared library importer receives the module instance from the > module init function, adds __file__, __path__, __package__ and friends to > the module dict, and then checks for the callback. If non-NULL, it calls it > to continue the module initialisation by user code. > The callback > ============ > > The callback is defined as follows:: > > int (*PyModule_init_callback)(PyObject* the_module, > PyModuleInitContext* context) > > "PyModuleInitContext" is a struct that is meant mostly for making the > callback more future proof by allowing additional parameters to be passed > in. For now, I can see a use case for the following fields:: > > struct PyModuleInitContext { > char* module_name; > char* qualified_module_name; > } > > Both names are encoded in UTF-8. As for the file path, I consider it best > to retrieve it from the module's __file__ attribute as a Python string > object to reduce filename encoding problems. > > Note that this struct argument is not strictly required, but given that > this proposal would have been much simpler if the module init function had > accepted such an argument in the first place, I consider it a good idea not > to let this chance pass by again. > > The registration of the callback uses a new C-API function: > > int PyModule_SetInitFunction(PyObject* module, > PyModule_init_callback callback) > > The function name uses "Set" instead of "Register" to make it clear that > there is only one such function per module. > > An alternative would be a new module creation function "PyModule_Create3()" > that takes the callback as third argument, in addition to what > "PyModule_Create2()" accepts. This would require users to explicitly pass > in the (second) version argument, which might be considered only a minor > issue. > > Implementation > ============== > > The implementation requires local changes to the extension module importer > and a new C-API function. In order to store the callback, it should use a > new field in the module object struct. > > Open questions > ============== > > It is not clear how extensions should be handled that register more than > one module in their module init function, e.g. compiled packages. One > possibility would be to leave the setup to the user, who would have to know > all FQMNs anyway in this case, although not the import file path. > Alternatively, the import machinery could use a stack to remember for which > modules a callback was registered during the last init function call, set > up all of them and then call their callbacks. It's not clear if this meets > the intention of the user. > > Alternatives > ============ > > 1) It would be possible to make extension modules optionally export another > symbol, e.g. "PyInit2_modulename", that the shared library loader would > call in addition to the required function "PyInit_modulename". This would > remove the need for a new API that registers the above callback. The > drawback is that it also makes it easier to write broken code because a > Python version or implementation that does not support this second symbol > would simply not call it, without error. The new C-API function would let > the build fail instead if it is not supported. > An alternative to the alternative is that if the PyInit2 function exists it's called instead of the the PyInit function, and then the PyInit function is nothing more than a single line function call (or whatever the absolute bare minimum is) into some helper that calls the PyInit2 call properly for backwards ABI compatibility (i.e. passes in whatever details are lost by the indirection in function call). That provides an eventual upgrade path of dropping PyInit and moving over to PyInit2. -Brett > > 2) The callback could be made available as a Python function in the module > dict, thus also removing the need for an explicit registration API. > However, this approach would add overhead to both sides, the importer code > and the user provided module init code, as it would require additional > dictionary handling and the implementation of a one-time Python function in > user code. It would also suffer from the problem that missing support in > the runtime would pass silently. > > 3) The callback could be registered statically in the PyModuleDef struct by > adding a new field. This is not trivial to do in a backwards compatible way > because the struct would grow longer without explicit initialisation by > existing user code. Extending PyModuleDef_HEAD_INIT might be possible but > would still break at least binary compatibility. > > 4) Pass a new context argument into the module init function that contains > all information necessary to properly and completely set up the module at > creation time. This would provide a much simpler and cleaner solution than > the proposed solution. However, it will not be possible before Python 4 as > it breaks backwards compatibility with all existing extension modules at > both the source and binary level. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121108/dccfb542/attachment-0001.html> From stefan_ml at behnel.de Thu Nov 8 16:00:20 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 16:00:20 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <CAP1=2W6W1LwY00B+ZJMFTrBcH2_iQ=MGghgxn=nNMc7h0c4sww@mail.gmail.com> References: <k7g9k9$8bl$1@ger.gmane.org> <CAP1=2W6W1LwY00B+ZJMFTrBcH2_iQ=MGghgxn=nNMc7h0c4sww@mail.gmail.com> Message-ID: <k7ghe3$h0g$1@ger.gmane.org> Hi Brett, thanks for the feedback. Brett Cannon, 08.11.2012 15:41: > On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: >> I propose to split the extension module initialisation into two steps in >> Python 3.4, in a backwards compatible way. >> >> Step 1: The current module init function can be reduced to just creating >> the module instance and returning it (and potentially doing some simple C >> level setup). Optionally, after creating the module (and this is the new >> part), the module init code can register a C callback function that will be >> called after setting up the module. > > Why even bother with the module creation? Why can't Python do that as well > and then call the callback? > > >> Step 2: The shared library importer receives the module instance from the >> module init function, adds __file__, __path__, __package__ and friends to >> the module dict, and then checks for the callback. If non-NULL, it calls it >> to continue the module initialisation by user code. > [...] > An alternative to the alternative is that if the PyInit2 function exists > it's called instead of the the PyInit function, and then the PyInit > function is nothing more than a single line function call (or whatever the > absolute bare minimum is) into some helper that calls the PyInit2 call > properly for backwards ABI compatibility (i.e. passes in whatever details > are lost by the indirection in function call). That provides an eventual > upgrade path of dropping PyInit and moving over to PyInit2. In that case, you'd have to export the PyModuleDef descriptor as well, because that's what tells CPython how the module behaves and what to do with it to set it up properly (e.g. allocate module state space on the heap). In fact, if the module init function became a field in the descriptor, it would be enough (taking backwards compatibility aside) if *only* the descriptor was exported and used by the module loader. With the caveat that this might kill some less common but not necessarily illegitimate use cases that do more than just creating and initialising a single module... Stefan From brett at python.org Thu Nov 8 16:06:50 2012 From: brett at python.org (Brett Cannon) Date: Thu, 8 Nov 2012 10:06:50 -0500 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <k7ghe3$h0g$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> <CAP1=2W6W1LwY00B+ZJMFTrBcH2_iQ=MGghgxn=nNMc7h0c4sww@mail.gmail.com> <k7ghe3$h0g$1@ger.gmane.org> Message-ID: <CAP1=2W5qGqn_ig94LUwcK6u6XYR4RpBMpCuyVx4LH_zCd4=QPQ@mail.gmail.com> On Thu, Nov 8, 2012 at 10:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > Hi Brett, > > thanks for the feedback. > > Brett Cannon, 08.11.2012 15:41: > > On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: > >> I propose to split the extension module initialisation into two steps in > >> Python 3.4, in a backwards compatible way. > >> > >> Step 1: The current module init function can be reduced to just creating > >> the module instance and returning it (and potentially doing some simple > C > >> level setup). Optionally, after creating the module (and this is the new > >> part), the module init code can register a C callback function that > will be > >> called after setting up the module. > > > > Why even bother with the module creation? Why can't Python do that as > well > > and then call the callback? > > > > > >> Step 2: The shared library importer receives the module instance from > the > >> module init function, adds __file__, __path__, __package__ and friends > to > >> the module dict, and then checks for the callback. If non-NULL, it > calls it > >> to continue the module initialisation by user code. > > [...] > > An alternative to the alternative is that if the PyInit2 function exists > > it's called instead of the the PyInit function, and then the PyInit > > function is nothing more than a single line function call (or whatever > the > > absolute bare minimum is) into some helper that calls the PyInit2 call > > properly for backwards ABI compatibility (i.e. passes in whatever details > > are lost by the indirection in function call). That provides an eventual > > upgrade path of dropping PyInit and moving over to PyInit2. > > In that case, you'd have to export the PyModuleDef descriptor as well, > because that's what tells CPython how the module behaves and what to do > with it to set it up properly (e.g. allocate module state space on the > heap). > True. > > In fact, if the module init function became a field in the descriptor, it > would be enough (taking backwards compatibility aside) if *only* the > descriptor was exported and used by the module loader. > > Also true. > With the caveat that this might kill some less common but not necessarily > illegitimate use cases that do more than just creating and initialising a > single module... > You mean creating another module in the init function? That's fine, but that should be a call to __import__ anyway and that should handle things properly. Else you are circumventing the import system and you can do everything from scratch. I don't see why this would stop you from doing anything you want, it just simplifies the common case. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121108/3f6c9db5/attachment.html> From stefan_ml at behnel.de Thu Nov 8 16:30:05 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 08 Nov 2012 16:30:05 +0100 Subject: [Python-Dev] Make extension module initialisation more like Python module initialisation In-Reply-To: <CAP1=2W5qGqn_ig94LUwcK6u6XYR4RpBMpCuyVx4LH_zCd4=QPQ@mail.gmail.com> References: <k7g9k9$8bl$1@ger.gmane.org> <CAP1=2W6W1LwY00B+ZJMFTrBcH2_iQ=MGghgxn=nNMc7h0c4sww@mail.gmail.com> <k7ghe3$h0g$1@ger.gmane.org> <CAP1=2W5qGqn_ig94LUwcK6u6XYR4RpBMpCuyVx4LH_zCd4=QPQ@mail.gmail.com> Message-ID: <k7gj5s$2n1$1@ger.gmane.org> Brett Cannon, 08.11.2012 16:06: > On Thu, Nov 8, 2012 at 10:00 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > >> Hi Brett, >> >> thanks for the feedback. >> >> Brett Cannon, 08.11.2012 15:41: >>> On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: >>>> I propose to split the extension module initialisation into two steps in >>>> Python 3.4, in a backwards compatible way. >>>> >>>> Step 1: The current module init function can be reduced to just creating >>>> the module instance and returning it (and potentially doing some simple >> C >>>> level setup). Optionally, after creating the module (and this is the new >>>> part), the module init code can register a C callback function that >> will be >>>> called after setting up the module. >>> >>> Why even bother with the module creation? Why can't Python do that as >> well >>> and then call the callback? >>> >>> >>>> Step 2: The shared library importer receives the module instance from >> the >>>> module init function, adds __file__, __path__, __package__ and friends >> to >>>> the module dict, and then checks for the callback. If non-NULL, it >> calls it >>>> to continue the module initialisation by user code. >>> [...] >>> An alternative to the alternative is that if the PyInit2 function exists >>> it's called instead of the the PyInit function, and then the PyInit >>> function is nothing more than a single line function call (or whatever >> the >>> absolute bare minimum is) into some helper that calls the PyInit2 call >>> properly for backwards ABI compatibility (i.e. passes in whatever details >>> are lost by the indirection in function call). That provides an eventual >>> upgrade path of dropping PyInit and moving over to PyInit2. >> >> In that case, you'd have to export the PyModuleDef descriptor as well, >> because that's what tells CPython how the module behaves and what to do >> with it to set it up properly (e.g. allocate module state space on the >> heap). > > True. > >> In fact, if the module init function became a field in the descriptor, it >> would be enough (taking backwards compatibility aside) if *only* the >> descriptor was exported and used by the module loader. > > Also true. > >> With the caveat that this might kill some less common but not necessarily >> illegitimate use cases that do more than just creating and initialising a >> single module... > > You mean creating another module in the init function? That's fine, but > that should be a call to __import__ anyway and that should handle things > properly. Ok. > Else you are circumventing the import system and you can do > everything from scratch. I guess I'd be ok with putting that burden on users in this case. > I don't see why this would stop you from doing > anything you want, it just simplifies the common case. The only problematic case I see here would be a module that calculates the size of its state space at init time, e.g. based on some platform specifics or environment parameters, anything from the platform specific size of some data type to the runtime configured number of OpenMP threads. That would make the PyModuleDef a compile time static thing - not sure if that's currently required. Stefan From ncoghlan at gmail.com Thu Nov 8 16:55:15 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 9 Nov 2012 01:55:15 +1000 Subject: [Python-Dev] Update - Re: Make extension module initialisation more like Python module initialisation In-Reply-To: <k7gfq0$177$1@ger.gmane.org> References: <k7g9k9$8bl$1@ger.gmane.org> <k7gfq0$177$1@ger.gmane.org> Message-ID: <CADiSq7cKArRVpmV4Ah_31d-xXeQfY_Yny-64FLdLe2TjqTndxQ@mail.gmail.com> On Fri, Nov 9, 2012 at 12:32 AM, Stefan Behnel <stefan_ml at behnel.de> wrote: > here's an updated proposal, adopting Marc-Andre's improvement that uses a > new field in the PyModuleDef struct to register the callback. Note that > this change no longer keeps up binary compatibility, which may or may not > be acceptable for Python 3.4. > It's not acceptable, as PyModuleDef is part of PEP 384's stable ABI. All such public structures are locked at their original size. 3) The original proposal used a new C-API function to register the callback > explicitly, as opposed to extending the PyModuleDef struct. This has the > advantage of keeping up binary compatibility with existing Py3.3 > extensions. It has the disadvantage of adding another indirection to the > setup procedure where a static function pointer would suffice. > Module initialisation is (and must be) part of the stable ABI. Indirection (especially through Python) is a *good* thing, as, ideally, any new interfaces should be defined in a way that doesn't increase the maintenance burden for the stable ABI. I don't agree that the use of a new init API can fail silently, so long as it completely *replaces* the old API, rather than being an addition. That way, since you won't be defining the *old* init function at all, old versions will correctly refuse to load your module. So I propose that we simply *fix* extension module loading to work the same way as everything else: the loader creates the module object, and passes it in to a new init function to be fully populated. __file__ and __name__ would be passed in as preinitialised module attributes. The existing PyModule_Create functions would be complemented by a PyModule_SetDef function which allowed a PyModuleDef to be configured on a pre-existing module. Extension modules that wanted to create multiple Python modules would still be free to do so - it would just be up to the extension initialisation code to call PyModule_Create to construct them and set __file__ based on the __file__ of the passed in module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121109/d54e8fc5/attachment.html> From tseaver at palladion.com Thu Nov 8 17:18:32 2012 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 08 Nov 2012 11:18:32 -0500 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python Message-ID: <k7gm2f$ujh$1@ger.gmane.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 While porting the BTrees package (split out from ZODB) to Python3., my first step is to get the pure-Python reference implementation working: in order to do that, I need a way to check that objects used as keys are *not* using the comparison semantics inherited from the base 'object' class, because those semantics rely on properties which are not stable across the lifetime of the (persisted / restored) key. The existing C code does something like:: static int check_argument_cmp(PyObject *arg) { if (arg->ob_type->tp_richcompare == NULL && arg->ob_type->tp_compare == ((PyTypeObject *)object_)->ob_type->tp_compare) { PyErr_SetString(PyExc_TypeError, "Object has default comparison"); return 0; } return 1; } Unless I'm mistaken, there is no way to do the equivalent from pure Python.. I tried a couple of approximations which relied on non-API attributes (I"m looking at out, methodwrapper.__objclass__), but without much success (and I need the code to work on PyPy / Jython, too). Am I missing something? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCb21gACgkQ+gerLs4ltQ4jBACfV0lQaQ2eW2vhAtWunLUsPQWM esEAoMYeeQvlJVnckaBg4HM19LoxPIWB =+d+0 -----END PGP SIGNATURE----- From guido at python.org Thu Nov 8 18:38:46 2012 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Nov 2012 09:38:46 -0800 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <k7gm2f$ujh$1@ger.gmane.org> References: <k7gm2f$ujh$1@ger.gmane.org> Message-ID: <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> Well, the default behavior has changed to raise an exception when using <, <=, >, >=; i.e., inequalities do not have a default implementation at all. Perhaps that is enough for your purpose? == and != still compare by pointer, but you're primarily interested in ordering her, right? On Thu, Nov 8, 2012 at 8:18 AM, Tres Seaver <tseaver at palladion.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > While porting the BTrees package (split out from ZODB) to Python3., my > first step is to get the pure-Python reference implementation working: > in order to do that, I need a way to check that objects used as keys are > *not* using the comparison semantics inherited from the base 'object' > class, because those semantics rely on properties which are not stable > across the lifetime of the (persisted / restored) key. > > The existing C code does something like:: > > static int > check_argument_cmp(PyObject *arg) > { > if (arg->ob_type->tp_richcompare == NULL > && arg->ob_type->tp_compare == > ((PyTypeObject *)object_)->ob_type->tp_compare) > { > PyErr_SetString(PyExc_TypeError, "Object has default comparison"); > return 0; > } > return 1; > } > > Unless I'm mistaken, there is no way to do the equivalent from pure > Python.. I tried a couple of approximations which relied on non-API > attributes (I"m looking at out, methodwrapper.__objclass__), but without > much success (and I need the code to work on PyPy / Jython, too). > > Am I missing something? > > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlCb21gACgkQ+gerLs4ltQ4jBACfV0lQaQ2eW2vhAtWunLUsPQWM > esEAoMYeeQvlJVnckaBg4HM19LoxPIWB > =+d+0 > -----END PGP SIGNATURE----- > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Thu Nov 8 18:45:28 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 08 Nov 2012 19:45:28 +0200 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <20121107230845.7A7D225006F@webabinitio.net> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> <20121107230845.7A7D225006F@webabinitio.net> Message-ID: <k7gr3v$fmq$1@ger.gmane.org> On 08.11.12 01:08, R. David Murray wrote: > Alexandre's point was that the string did not appear to be arbitrary, > but rather appeared to specifically be a string containing surrogates. > Is this not the case? My intention was testing with filename which cannot be decoded as UTF-8 in strict mode. I agree that testing with name which is encodable in locale encoding can be useful too, but now the test has no effect on UTF-8 locale. From tseaver at palladion.com Thu Nov 8 18:55:47 2012 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 08 Nov 2012 12:55:47 -0500 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> References: <k7gm2f$ujh$1@ger.gmane.org> <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> Message-ID: <509BF223.1040408@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/08/2012 12:38 PM, Guido van Rossum wrote: > Well, the default behavior has changed to raise an exception when > using <, <=, >, >=; i.e., inequalities do not have a default > implementation at all. Perhaps that is enough for your purpose? == > and != still compare by pointer, but you're primarily interested in > ordering her, right? The b-search algorithm needs ordering on keys, but also meaningful (non-identity) equality. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCb8iMACgkQ+gerLs4ltQ6WOACgwERmICWr80qnEoOVVLQtFwTH ttAAnAt4An0dPjaRuZyJlDAUGzH0CS7B =u2mG -----END PGP SIGNATURE----- From tseaver at palladion.com Thu Nov 8 18:55:47 2012 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 08 Nov 2012 12:55:47 -0500 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> References: <k7gm2f$ujh$1@ger.gmane.org> <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> Message-ID: <509BF223.1040408@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/08/2012 12:38 PM, Guido van Rossum wrote: > Well, the default behavior has changed to raise an exception when > using <, <=, >, >=; i.e., inequalities do not have a default > implementation at all. Perhaps that is enough for your purpose? == > and != still compare by pointer, but you're primarily interested in > ordering her, right? The b-search algorithm needs ordering on keys, but also meaningful (non-identity) equality. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCb8iMACgkQ+gerLs4ltQ6WOACgwERmICWr80qnEoOVVLQtFwTH ttAAnAt4An0dPjaRuZyJlDAUGzH0CS7B =u2mG -----END PGP SIGNATURE----- From guido at python.org Thu Nov 8 18:59:35 2012 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Nov 2012 09:59:35 -0800 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <509BF223.1040408@palladion.com> References: <k7gm2f$ujh$1@ger.gmane.org> <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> <509BF223.1040408@palladion.com> Message-ID: <CAP7+vJLnW9g7XtgeUUyv=+FiHwzaB_a=qRTa+Oj5mPF5SOH9jA@mail.gmail.com> In Python, you can write if C.__eq__ == object.__eq__: print('class C does not override __eq__') Does that help? On Thu, Nov 8, 2012 at 9:55 AM, Tres Seaver <tseaver at palladion.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 11/08/2012 12:38 PM, Guido van Rossum wrote: >> Well, the default behavior has changed to raise an exception when >> using <, <=, >, >=; i.e., inequalities do not have a default >> implementation at all. Perhaps that is enough for your purpose? == >> and != still compare by pointer, but you're primarily interested in >> ordering her, right? > > The b-search algorithm needs ordering on keys, but also meaningful > (non-identity) equality. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iEYEARECAAYFAlCb8iMACgkQ+gerLs4ltQ6WOACgwERmICWr80qnEoOVVLQtFwTH > ttAAnAt4An0dPjaRuZyJlDAUGzH0CS7B > =u2mG > -----END PGP SIGNATURE----- -- --Guido van Rossum (python.org/~guido) From storchaka at gmail.com Thu Nov 8 19:06:15 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 08 Nov 2012 20:06:15 +0200 Subject: [Python-Dev] chained assignment weirdity In-Reply-To: <509B06B9.6000406@nedbatchelder.com> References: <508EA6E3.5090201@simplistix.co.uk> <D13B9C12-0DEC-495F-9805-F5DFF964E524@voidspace.org.uk> <50937B1B.2080404@simplistix.co.uk> <5B9E75D2-5953-4772-BB44-E567AA3B2672@voidspace.org.uk> <5098AB9F.2010806@simplistix.co.uk> <k7al0r$iom$1@ger.gmane.org> <CADiSq7fhGORQF24EiPXnZUJJTP9WxfVDYWFkb1rJWW76aUqaow@mail.gmail.com> <k7bd1m$i5b$1@ger.gmane.org> <20121106162655.A858625006B@webabinitio.net> <50994FDD.8060906@nedbatchelder.com> <CAP7+vJKqW9UE338y=tjBAC93wTX7XoGA+VgN87j4CYJ1r9Htuw@mail.gmail.com> <509A504C.10807@nedbatchelder.com> <CAP7+vJLKAuWMxcym5oQEq_HJ8yLY-Xg_H_cqEoBnq2eD09FY5w@mail.gmail.com> <CADiSq7f=ZFB6ATMwzWa0V9+46YXc4Wzh7+jfs7Cv_mHv-bFruQ@mail.gmail.com> <k7e4ih$idp$1@ger.gmane.org> <509AD510.5060200@nedbatchelder.com> <k7emb0$okb$1@ger.gmane.org> <509B06B9.6000406@nedbatchelder.com> Message-ID: <k7gsau$r7s$1@ger.gmane.org> On 08.11.12 03:11, Ned Batchelder wrote: > Sorry, I should have been clearer: I was asking about weird not to say, > "This is weird and should be changed!", but to get clarification from > Serhiy about his statement, " It will be weird if a dict comprehension > and a plain loop will be inconsistent." I honestly didn't know which > behavior he considered inconsistent and therefore weird. I was referring to two of the most popular idioms to dynamically create a dict. d = {} for x in a: d[k(x)] = v(x) d = {k(x): v(x) for x in a} For now these methods are consistent. I agree that the use of the side effects here is not a sensible idea, but when such effects occur by accident, it will cause a surprise. From alexandre at peadrop.com Thu Nov 8 19:10:42 2012 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Thu, 8 Nov 2012 10:10:42 -0800 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <k7gr3v$fmq$1@ger.gmane.org> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> <20121107230845.7A7D225006F@webabinitio.net> <k7gr3v$fmq$1@ger.gmane.org> Message-ID: <CANcUUee6-Tt=4gdPZa6Kq3H3sdGzCXzOp0eeDC452sL1cDbURQ@mail.gmail.com> On Thu, Nov 8, 2012 at 9:45 AM, Serhiy Storchaka <storchaka at gmail.com>wrote: > My intention was testing with filename which cannot be decoded as UTF-8 in > strict mode. I agree that testing with name which is encodable in locale > encoding can be useful too, but now the test has no effect on UTF-8 locale. So should we change the test back? Or just change the test name? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121108/10342bfa/attachment.html> From storchaka at gmail.com Thu Nov 8 19:21:19 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 08 Nov 2012 20:21:19 +0200 Subject: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding In-Reply-To: <CANcUUee6-Tt=4gdPZa6Kq3H3sdGzCXzOp0eeDC452sL1cDbURQ@mail.gmail.com> References: <3Xv06h2mWtzMCY@mail.python.org> <20121103181343.27e690d7@pitrou.net> <CANcUUedKG+P4c+=SxxCG7_GVRJkNmSnnYXtuB=DYeQ2m+P5hLg@mail.gmail.com> <CAMpsgwYys2dWU0S+32TM1Kyq=u14ne50hZRg5myvezR=_5sRXQ@mail.gmail.com> <20121107230845.7A7D225006F@webabinitio.net> <k7gr3v$fmq$1@ger.gmane.org> <CANcUUee6-Tt=4gdPZa6Kq3H3sdGzCXzOp0eeDC452sL1cDbURQ@mail.gmail.com> Message-ID: <k7gt76$40n$1@ger.gmane.org> On 08.11.12 20:10, Alexandre Vassalotti wrote: > So should we change the test back? Or just change the test name? No. I also missed some details. The issue is still not closed. From tseaver at palladion.com Thu Nov 8 20:51:54 2012 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 08 Nov 2012 14:51:54 -0500 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <CAP7+vJLnW9g7XtgeUUyv=+FiHwzaB_a=qRTa+Oj5mPF5SOH9jA@mail.gmail.com> References: <k7gm2f$ujh$1@ger.gmane.org> <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> <509BF223.1040408@palladion.com> <CAP7+vJLnW9g7XtgeUUyv=+FiHwzaB_a=qRTa+Oj5mPF5SOH9jA@mail.gmail.com> Message-ID: <509C0D5A.5090403@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/08/2012 12:59 PM, Guido van Rossum wrote: > In Python, you can write > > if C.__eq__ == object.__eq__: print('class C does not override > __eq__') > > Does that help? That works in Python3, but not in Python2 -- methodwrappers don't compare equal the same way slotwrappers do :(. I still need to straddle (back to Python 2.6. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCcDVoACgkQ+gerLs4ltQ49kwCgtBJX1oyi5cudZeE4LBEdcAmO aTAAn0jL56d3eK/sDUD3G7zzZ/ZdQH4W =NG86 -----END PGP SIGNATURE----- From tseaver at palladion.com Thu Nov 8 21:31:51 2012 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 08 Nov 2012 15:31:51 -0500 Subject: [Python-Dev] Detecting tp_compare / tp_richcompare from Python In-Reply-To: <509C0D5A.5090403@palladion.com> References: <k7gm2f$ujh$1@ger.gmane.org> <CAP7+vJJ3qDNDH=em2Q_dqE9C=Lu4fDwmNi1yFb+nRefaYeDdQg@mail.gmail.com> <509BF223.1040408@palladion.com> <CAP7+vJLnW9g7XtgeUUyv=+FiHwzaB_a=qRTa+Oj5mPF5SOH9jA@mail.gmail.com> <509C0D5A.5090403@palladion.com> Message-ID: <509C16B7.20608@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11/08/2012 02:51 PM, Tres Seaver wrote: > On 11/08/2012 12:59 PM, Guido van Rossum wrote: >> In Python, you can write > >> if C.__eq__ == object.__eq__: print('class C does not override >> __eq__') > >> Does that help? > > That works in Python3, but not in Python2 -- methodwrappers don't > compare equal the same way slotwrappers do :(. I still need to > straddle (back to Python 2.6. I think I'm going to need two separate checkers: - - In Py3k, use your check above for both '__eq__' and '__lt__', e.g: def check_comparable(key); klass = type(key) if klass.__eq__ == object.__eq__ or klass.__lt__ == object.__lt__: raise TypeError('Incomparable') - - In Python2, I think I can just check that the instance (not its class) has either ('__lt__' and '__eq__') *or* '__cmp__', e.g.: def check_comparable(key); if not ((getattr(key, '__eq__', None) is not None and getattr(key, '__lt__', None) is not None) or getattr(key, '__cmp__', None) is not None): raise TypeError('Incomparable') Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCcFrcACgkQ+gerLs4ltQ6JDgCeJpDmo70H9Xgz3preNHVvaTl5 dYkAoNS+fpSXwZiaD2J2ONyQZ2mPIcFC =OiMY -----END PGP SIGNATURE----- From g.brandl at gmx.net Thu Nov 8 21:43:19 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 08 Nov 2012 21:43:19 +0100 Subject: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon In-Reply-To: <3XwsjZ4sTWzLs9@mail.python.org> References: <3XwsjZ4sTWzLs9@mail.python.org> Message-ID: <k7h5h6$frq$1@ger.gmane.org> On 11/06/2012 02:56 PM, tim.golden wrote: > http://hg.python.org/cpython/rev/dafca4714298 > changeset: 80273:dafca4714298 > user: Tim Golden <mail at timgolden.me.uk> > date: Tue Nov 06 13:50:42 2012 +0000 > summary: > issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon > > files: > Doc/library/glob.rst | 11 ++-- > Lib/glob.py | 65 ++++++++++++++++++++++-------- > Lib/test/test_glob.py | 64 +++++++++++++++++++++++++++++- > Misc/NEWS | 3 + > 4 files changed, 118 insertions(+), 25 deletions(-) > > > diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst > --- a/Doc/library/glob.rst > +++ b/Doc/library/glob.rst > @@ -13,10 +13,10 @@ > > The :mod:`glob` module finds all the pathnames matching a specified pattern > according to the rules used by the Unix shell. No tilde expansion is done, but > -``*``, ``?``, and character ranges expressed with ``[]`` will be correctly > -matched. This is done by using the :func:`os.listdir` and > -:func:`fnmatch.fnmatch` functions in concert, and not by actually invoking a > -subshell. (For tilde and shell variable expansion, use > +``*``, ``?``, character ranges expressed with ``[]`` and list of options > +expressed with ``{}`` will be correctly matched. This is done by using the > +:func:`os.listdir` and :func:`fnmatch.fnmatch` functions in concert, and not by > +actually invoking a subshell. (For tilde and shell variable expansion, use > :func:`os.path.expanduser` and :func:`os.path.expandvars`.) Needs a versionchanged. In any case, brace expansion is not part of globbing (see the bash or zsh manuals) because it does not generate valid file names, and it is a non-POSIX expansion of some shells. Are you sure it should be put into the glob module? (Not speaking of the backward incompatibility it creates.) Georg From g.brandl at gmx.net Thu Nov 8 21:46:00 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 08 Nov 2012 21:46:00 +0100 Subject: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon In-Reply-To: <k7h5h6$frq$1@ger.gmane.org> References: <3XwsjZ4sTWzLs9@mail.python.org> <k7h5h6$frq$1@ger.gmane.org> Message-ID: <k7h5m7$frq$2@ger.gmane.org> On 11/08/2012 09:43 PM, Georg Brandl wrote: > > Needs a versionchanged. > > In any case, brace expansion is not part of globbing (see the bash or zsh > manuals) because it does not generate valid file names, and it is a non-POSIX > expansion of some shells. Are you sure it should be put into the glob module? > (Not speaking of the backward incompatibility it creates.) Sorry, just saw the reversion. Never mind, my concerns have been addressed. Georg From mail at timgolden.me.uk Thu Nov 8 21:52:25 2012 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 08 Nov 2012 20:52:25 +0000 Subject: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon In-Reply-To: <k7h5h6$frq$1@ger.gmane.org> References: <3XwsjZ4sTWzLs9@mail.python.org> <k7h5h6$frq$1@ger.gmane.org> Message-ID: <509C1B89.7080005@timgolden.me.uk> On 08/11/2012 20:43, Georg Brandl wrote: > On 11/06/2012 02:56 PM, tim.golden wrote: >> http://hg.python.org/cpython/rev/dafca4714298 >> changeset: 80273:dafca4714298 >> user: Tim Golden <mail at timgolden.me.uk> >> date: Tue Nov 06 13:50:42 2012 +0000 >> summary: >> issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon >> >> files: >> Doc/library/glob.rst | 11 ++-- >> Lib/glob.py | 65 ++++++++++++++++++++++-------- >> Lib/test/test_glob.py | 64 +++++++++++++++++++++++++++++- >> Misc/NEWS | 3 + >> 4 files changed, 118 insertions(+), 25 deletions(-) >> >> >> diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst >> --- a/Doc/library/glob.rst >> +++ b/Doc/library/glob.rst >> @@ -13,10 +13,10 @@ >> >> The :mod:`glob` module finds all the pathnames matching a specified pattern >> according to the rules used by the Unix shell. No tilde expansion is done, but >> -``*``, ``?``, and character ranges expressed with ``[]`` will be correctly >> -matched. This is done by using the :func:`os.listdir` and >> -:func:`fnmatch.fnmatch` functions in concert, and not by actually invoking a >> -subshell. (For tilde and shell variable expansion, use >> +``*``, ``?``, character ranges expressed with ``[]`` and list of options >> +expressed with ``{}`` will be correctly matched. This is done by using the >> +:func:`os.listdir` and :func:`fnmatch.fnmatch` functions in concert, and not by >> +actually invoking a subshell. (For tilde and shell variable expansion, use >> :func:`os.path.expanduser` and :func:`os.path.expandvars`.) > > Needs a versionchanged. > > In any case, brace expansion is not part of globbing (see the bash or zsh > manuals) because it does not generate valid file names, and it is a non-POSIX > expansion of some shells. Are you sure it should be put into the glob module? > (Not speaking of the backward incompatibility it creates.) I backed it out very soon afterwards, Georg. It had had some (quite a bit of) discussion on the issue, but I'd messed up the patch somehow and the backwards compat issue was raised pretty much immediately by Serhiy. So I pulled the commit. TJG Insofar as From chris at simplistix.co.uk Fri Nov 9 10:57:22 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Nov 2012 09:57:22 +0000 Subject: [Python-Dev] problems building python2.7 Message-ID: <509CD382.4080609@simplistix.co.uk> Hi All, I wanted to run the unit tests before checking in the patch for http://bugs.python.org/issue16441, even though it's a trivial change, so I was trying to follow the instructions at: http://docs.python.org/devguide/ I'm on MacOS, so following the "unix" instructions did: ./configure --with-pydebug && make -j2 This appears to have worked, given the end of the output: Python build finished, but the necessary bits to build these modules were not found: _bsddb dl gdbm imageop linuxaudiodev ossaudiodev readline spwd sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. running build_scripts creating build/scripts-2.7 copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/pydoc -> build/scripts-2.7 copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/idle -> build/scripts-2.7 copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/2to3 -> build/scripts-2.7 copying and adjusting /Users/chris/LocalHG/cpython/Lib/smtpd.py -> build/scripts-2.7 changing mode of build/scripts-2.7/pydoc from 644 to 755 changing mode of build/scripts-2.7/idle from 644 to 755 changing mode of build/scripts-2.7/2to3 from 644 to 755 changing mode of build/scripts-2.7/smtpd.py from 644 to 755 [65930 refs] However, I can't find the python it's built... I thought I'd be clever and try: buzzkill:cpython chris$ cat build/scripts-2.7/2to3 #!/usr/local/bin/python2.7 ... There is a python there, but it's a symlink put in place around a year ago. So, where should I look for the built python? Okay, so regardless, my change is only to the stdlib, so I thought I'd try the test instructions anyway: /usr/local/bin/python2.7 -m test -j3 gives: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named test.__main__; 'test' is a package and cannot be directly executed Now, if this had worked, would it have discovered the right gzip tests, or is it going to run whatever came with the python2.7 distro the binary comes from? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From hansmu at xs4all.nl Fri Nov 9 11:37:15 2012 From: hansmu at xs4all.nl (Hans Mulder) Date: Fri, 09 Nov 2012 11:37:15 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CD382.4080609@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> Message-ID: <k7imcq$8f0$1@ger.gmane.org> On 9/11/12 10:57:22, Chris Withers wrote: > Hi All, > > I wanted to run the unit tests before checking in the patch for > http://bugs.python.org/issue16441, even though it's a trivial change, so > I was trying to follow the instructions at: > > http://docs.python.org/devguide/ > > I'm on MacOS, so following the "unix" instructions did: > > ./configure --with-pydebug && make -j2 > > This appears to have worked, given the end of the output: > > Python build finished, but the necessary bits to build these modules > were not found: > _bsddb dl gdbm > imageop linuxaudiodev ossaudiodev > readline spwd sunaudiodev > To find the necessary bits, look in setup.py in detect_modules() for the > module's name. > > running build_scripts > creating build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/pydoc > -> build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/idle -> > build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/2to3 -> > build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Lib/smtpd.py -> > build/scripts-2.7 > changing mode of build/scripts-2.7/pydoc from 644 to 755 > changing mode of build/scripts-2.7/idle from 644 to 755 > changing mode of build/scripts-2.7/2to3 from 644 to 755 > changing mode of build/scripts-2.7/smtpd.py from 644 to 755 > [65930 refs] > > However, I can't find the python it's built... On my system, it's in the current directory. It's called "python.exe", to avoid a name clash with the "Python" subdirectory. > I thought I'd be clever and try: > > buzzkill:cpython chris$ cat build/scripts-2.7/2to3 > #!/usr/local/bin/python2.7 > ... Presumable, it would be installed in /usr/local/bin if I'd tried "make install". > There is a python there, but it's a symlink put in place around a year ago. Same here. > So, where should I look for the built python? In the current directory: $ ./python.exe Python 2.7.3+ (2.7:8b181c75792f, Nov 9 2012, 11:26:59) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> Hope this helps, -- HansM From fuzzyman at voidspace.org.uk Fri Nov 9 11:52:06 2012 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 9 Nov 2012 10:52:06 +0000 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CD382.4080609@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> Message-ID: <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> On 9 Nov 2012, at 09:57, Chris Withers <chris at simplistix.co.uk> wrote: > Hi All, > > I wanted to run the unit tests before checking in the patch for http://bugs.python.org/issue16441, even though it's a trivial change, so I was trying to follow the instructions at: > > http://docs.python.org/devguide/ > > I'm on MacOS, so following the "unix" instructions did: > > ./configure --with-pydebug && make -j2 > > This appears to have worked, given the end of the output: > > Python build finished, but the necessary bits to build these modules were not found: > _bsddb dl gdbm > imageop linuxaudiodev ossaudiodev > readline spwd sunaudiodev > To find the necessary bits, look in setup.py in detect_modules() for the module's name. > > running build_scripts > creating build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/pydoc -> build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/idle -> build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Tools/scripts/2to3 -> build/scripts-2.7 > copying and adjusting /Users/chris/LocalHG/cpython/Lib/smtpd.py -> build/scripts-2.7 > changing mode of build/scripts-2.7/pydoc from 644 to 755 > changing mode of build/scripts-2.7/idle from 644 to 755 > changing mode of build/scripts-2.7/2to3 from 644 to 755 > changing mode of build/scripts-2.7/smtpd.py from 644 to 755 > [65930 refs] > > However, I can't find the python it's built... It should be python.exe (yes really). After the build you should be able to do: ./python.exe Michael > I thought I'd be clever and try: > > buzzkill:cpython chris$ cat build/scripts-2.7/2to3 > #!/usr/local/bin/python2.7 > ... > > There is a python there, but it's a symlink put in place around a year ago. > > So, where should I look for the built python? > > Okay, so regardless, my change is only to the stdlib, so I thought I'd try the test instructions anyway: > > /usr/local/bin/python2.7 -m test -j3 gives: > > /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named test.__main__; 'test' is a package and cannot be directly executed > > Now, if this had worked, would it have discovered the right gzip tests, or is it going to run whatever came with the python2.7 distro the binary comes from? > > Chris > > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From chris at simplistix.co.uk Fri Nov 9 11:57:39 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Nov 2012 10:57:39 +0000 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> Message-ID: <509CE1A3.2000202@simplistix.co.uk> On 09/11/2012 10:52, Michael Foord wrote: > >> However, I can't find the python it's built... > > It should be python.exe (yes really). Hah! Should http://docs.python.org/devguide/ be updated to reflect this or does this only affect Mac OS? (or should we correct the build so it doesn't spit out a .exe?) > After the build you should be able to do: > > ./python.exe Unfortunately, still: buzzkill:cpython chris$ ./python.exe -m test -j3 /Users/chris/LocalHG/cpython/python.exe: No module named test.__main__; 'test' is a package and cannot be directly executed [18856 refs] Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From hansmu at xs4all.nl Fri Nov 9 12:09:26 2012 From: hansmu at xs4all.nl (Hans Mulder) Date: Fri, 09 Nov 2012 12:09:26 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CE1A3.2000202@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> Message-ID: <k7io94$o92$1@ger.gmane.org> On 9/11/12 11:57:39, Chris Withers wrote: > On 09/11/2012 10:52, Michael Foord wrote: >> >>> However, I can't find the python it's built... >> >> It should be python.exe (yes really). > > Hah! Should http://docs.python.org/devguide/ be updated to reflect this > or does this only affect Mac OS? (or should we correct the build so it > doesn't spit out a .exe?) It only affects MacOS. On other Unix system, the build process spits out an executable named "python". On MacOS that's not possible, because that name would clash with the subdirectory named "Python". >> After the build you should be able to do: >> >> ./python.exe > > Unfortunately, still: > > buzzkill:cpython chris$ ./python.exe -m test -j3 > /Users/chris/LocalHG/cpython/python.exe: No module named test.__main__; > 'test' is a package and cannot be directly executed > [18856 refs] How about "make test" ? Hope this helps, -- HansM From petri at digip.org Fri Nov 9 12:10:42 2012 From: petri at digip.org (Petri Lehtinen) Date: Fri, 9 Nov 2012 13:10:42 +0200 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CE1A3.2000202@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> Message-ID: <20121109111042.GG4265@p29> Chris Withers wrote: > On 09/11/2012 10:52, Michael Foord wrote: > > > >>However, I can't find the python it's built... > > > >It should be python.exe (yes really). > > Hah! Should http://docs.python.org/devguide/ be updated to reflect > this or does this only affect Mac OS? (or should we correct the > build so it doesn't spit out a .exe?) The filesystem on Mac OS X is case-insensitive by default, so plain "python" would clash with the "Python" directory. > >After the build you should be able to do: > > > > ./python.exe > > Unfortunately, still: > > buzzkill:cpython chris$ ./python.exe -m test -j3 > /Users/chris/LocalHG/cpython/python.exe: No module named > test.__main__; 'test' is a package and cannot be directly executed > [18856 refs] You're running the tests for Python 2, so the corret way to do it is: ./python.exe -m test.regrtest See http://docs.python.org/devguide/runtests.html#running . From hansmu at xs4all.nl Fri Nov 9 12:54:24 2012 From: hansmu at xs4all.nl (Hans Mulder) Date: Fri, 09 Nov 2012 12:54:24 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <k7io94$o92$1@ger.gmane.org> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <k7io94$o92$1@ger.gmane.org> Message-ID: <k7iqte$eq5$1@ger.gmane.org> On 9/11/12 12:09:26, Hans Mulder wrote: > On 9/11/12 11:57:39, Chris Withers wrote: >> On 09/11/2012 10:52, Michael Foord wrote: >>> >>>> However, I can't find the python it's built... >>> >>> It should be python.exe (yes really). >> >> Hah! Should http://docs.python.org/devguide/ be updated to reflect this >> or does this only affect Mac OS? (or should we correct the build so it >> doesn't spit out a .exe?) > > It only affects MacOS. On other Unix system, the build process spits > out an executable named "python". On MacOS that's not possible, > because that name would clash with the subdirectory named "Python". > >>> After the build you should be able to do: >>> >>> ./python.exe >> >> Unfortunately, still: >> >> buzzkill:cpython chris$ ./python.exe -m test -j3 >> /Users/chris/LocalHG/cpython/python.exe: No module named test.__main__; >> 'test' is a package and cannot be directly executed >> [18856 refs] > > How about "make test" ? > > > Hope this helps, > > -- HansM I tried "make test", and I got: test test_urllib failed -- Traceback (most recent call last): File "/Users/hans/python/cpython/cpython-2.7/Lib/test/test_urllib.py", line 235, in test_missing_localfile fp.close() UnboundLocalError: local variable 'fp' referenced before assignment Is this a bug in the test suite, or a regression in 2.7.3+ ? This is on MacOS 10.6.5, using a checkout from hg.python.org from earlier today. -- HansM From ronaldoussoren at mac.com Fri Nov 9 12:54:17 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Fri, 09 Nov 2012 12:54:17 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CE1A3.2000202@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> Message-ID: <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> On 9 Nov, 2012, at 11:57, Chris Withers <chris at simplistix.co.uk> wrote: > On 09/11/2012 10:52, Michael Foord wrote: >> >>> However, I can't find the python it's built... >> >> It should be python.exe (yes really). > > Hah! Should http://docs.python.org/devguide/ be updated to reflect this or does this only affect Mac OS? (or should we correct the build so it doesn't spit out a .exe?) How else should it be named, given that the default filesystem on OSX machines is not case sensitive. The binary cannot be named "python" because that clashes with the directory named "Python" in the source and build tres. Ronald From chris at simplistix.co.uk Fri Nov 9 12:57:54 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Nov 2012 11:57:54 +0000 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> Message-ID: <509CEFC2.9020205@simplistix.co.uk> On 09/11/2012 11:54, Ronald Oussoren wrote: > > On 9 Nov, 2012, at 11:57, Chris Withers<chris at simplistix.co.uk> wrote: > >> On 09/11/2012 10:52, Michael Foord wrote: >>> >>>> However, I can't find the python it's built... >>> >>> It should be python.exe (yes really). >> >> Hah! Should http://docs.python.org/devguide/ be updated to reflect this or does this only affect Mac OS? (or should we correct the build so it doesn't spit out a .exe?) > > How else should it be named, given that the default filesystem on OSX machines is not case sensitive. The binary cannot be named "python" because that clashes with the directory named "Python" in the source and build tres. Maybe python.bin, or put it in the ./bin folder? I saw python.exe, but my brain went "oh, .exe, that's windows, I'll ignore that then, since I'm on Mac OS" Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Fri Nov 9 15:11:26 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 09 Nov 2012 14:11:26 +0000 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <k7iqte$eq5$1@ger.gmane.org> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <k7io94$o92$1@ger.gmane.org> <k7iqte$eq5$1@ger.gmane.org> Message-ID: <509D0F0E.7080306@simplistix.co.uk> On 09/11/2012 11:54, Hans Mulder wrote: > I tried "make test", and I got: > > test test_urllib failed -- Traceback (most recent call last): > File "/Users/hans/python/cpython/cpython-2.7/Lib/test/test_urllib.py", > line 235, in test_missing_localfile > fp.close() > UnboundLocalError: local variable 'fp' referenced before assignment > > Is this a bug in the test suite, or a regression in 2.7.3+ ? > > This is on MacOS 10.6.5, using a checkout from hg.python.org > from earlier today. I'm on MacOS 10.7.5 and got: 356 tests OK. 35 tests skipped: test_al test_bsddb test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses test_dl test_epoll test_gdb test_gdbm test_gl test_imageop test_imgfile test_largefile test_linuxaudiodev test_msilib test_ossaudiodev test_readline test_smtpnet test_socketserver test_startfile test_sunaudiodev test_timeout test_tk test_ttk_guionly test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 4 skips unexpected on darwin: test_dl test_readline test_tk test_ttk_guionly Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From hansmu at xs4all.nl Fri Nov 9 16:44:00 2012 From: hansmu at xs4all.nl (Hans Mulder) Date: Fri, 09 Nov 2012 16:44:00 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509D0F0E.7080306@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <k7io94$o92$1@ger.gmane.org> <k7iqte$eq5$1@ger.gmane.org> <509D0F0E.7080306@simplistix.co.uk> Message-ID: <k7j8bv$98f$1@ger.gmane.org> On 9/11/12 15:11:26, Chris Withers wrote: > On 09/11/2012 11:54, Hans Mulder wrote: >> I tried "make test", and I got: >> >> test test_urllib failed -- Traceback (most recent call last): >> File "/Users/hans/python/cpython/cpython-2.7/Lib/test/test_urllib.py", >> line 235, in test_missing_localfile >> fp.close() >> UnboundLocalError: local variable 'fp' referenced before assignment >> >> Is this a bug in the test suite, or a regression in 2.7.3+ ? >> >> This is on MacOS 10.6.5, using a checkout from hg.python.org >> from earlier today. > > I'm on MacOS 10.7.5 and got: > > 356 tests OK. > 35 tests skipped: > test_al test_bsddb test_bsddb3 test_cd test_cl test_codecmaps_cn > test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr > test_codecmaps_tw test_curses test_dl test_epoll test_gdb > test_gdbm test_gl test_imageop test_imgfile test_largefile > test_linuxaudiodev test_msilib test_ossaudiodev test_readline > test_smtpnet test_socketserver test_startfile test_sunaudiodev > test_timeout test_tk test_ttk_guionly test_urllib2net > test_urllibnet test_winreg test_winsound test_zipfile64 > 4 skips unexpected on darwin: > test_dl test_readline test_tk test_ttk_guionly I looked into it, and the problem is this bit of code (line 230-235): try: self.assertTrue(os.path.exists(tmp_file)) fp = urllib.urlopen(tmp_fileurl) finally: os.close(fd) fp.close() Due to a misconfiguration, urllib.thishost() raises an IOError on my laptop. This causes urllib.urlopen to raise an exception, and the name fp is never bound, so that the fp.close() in the finally clause raises an UnboundLocalError, masking the problem in urlopen. A quick fix would be: try: self.assertTrue(os.path.exists(tmp_file)) fp = urllib.urlopen(tmp_fileurl) fp.close() finally: os.close(fd) That way, the .close is only attempted if the open succeeds. -- HansM From tjreedy at udel.edu Fri Nov 9 16:48:06 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 09 Nov 2012 10:48:06 -0500 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CE1A3.2000202@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> Message-ID: <k7j8jr$83t$1@ger.gmane.org> On 11/9/2012 5:57 AM, Chris Withers wrote: > On 09/11/2012 10:52, Michael Foord wrote: >> >>> However, I can't find the python it's built... >> >> It should be python.exe (yes really). > > Hah! Should http://docs.python.org/devguide/ be updated to reflect this > or does this only affect Mac OS? The devguide could be updated to briefly note the Mac specific variation. -- Terry Jan Reedy From rdmurray at bitdance.com Fri Nov 9 17:07:13 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 09 Nov 2012 11:07:13 -0500 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <k7j8bv$98f$1@ger.gmane.org> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <k7io94$o92$1@ger.gmane.org> <k7iqte$eq5$1@ger.gmane.org> <509D0F0E.7080306@simplistix.co.uk> <k7j8bv$98f$1@ger.gmane.org> Message-ID: <20121109160714.4327E25003E@webabinitio.net> On Fri, 09 Nov 2012 16:44:00 +0100, Hans Mulder <hansmu at xs4all.nl> wrote: > I looked into it, and the problem is this bit of code (line 230-235): > > try: > self.assertTrue(os.path.exists(tmp_file)) > fp = urllib.urlopen(tmp_fileurl) > finally: > os.close(fd) > fp.close() > > Due to a misconfiguration, urllib.thishost() raises an IOError on my > laptop. This causes urllib.urlopen to raise an exception, and the > name fp is never bound, so that the fp.close() in the finally clause > raises an UnboundLocalError, masking the problem in urlopen. > > A quick fix would be: > > try: > self.assertTrue(os.path.exists(tmp_file)) > fp = urllib.urlopen(tmp_fileurl) > fp.close() > finally: > os.close(fd) > > That way, the .close is only attempted if the open succeeds. Could you open an issue for this on the tracker, please? That way we won't forget to fix it. --David From status at bugs.python.org Fri Nov 9 18:07:26 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 9 Nov 2012 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20121109170726.C1B0C1CBD9@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-11-02 - 2012-11-09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3806 ( +2) closed 24418 (+57) total 28224 (+59) Open issues with patches: 1657 Issues opened (42) ================== #13424: Add examples for open???s new opener argument http://bugs.python.org/issue13424 reopened by eric.araujo #16350: zlib.Decompress.decompress() after EOF discards existing value http://bugs.python.org/issue16350 reopened by serhiy.storchaka #16389: re._compiled_typed's lru_cache causes significant degradation http://bugs.python.org/issue16389 opened by pjenvey #16391: add "terminator" ctor argument to logging.StreamHandlers deriv http://bugs.python.org/issue16391 opened by nikicat #16392: import crashes on circular imports in ext modules http://bugs.python.org/issue16392 opened by scoder #16394: Reducing tee() memory footprint http://bugs.python.org/issue16394 opened by cami #16395: Documentation claims that PySequence_Fast returns a tuple, whe http://bugs.python.org/issue16395 opened by sfllaw #16396: Importing ctypes.wintypes on Linux gives a traceback http://bugs.python.org/issue16396 opened by techtonik #16397: UserString doesn't combine nicely with strings http://bugs.python.org/issue16397 opened by Jorge.Cardona #16398: deque.rotate() could be much faster http://bugs.python.org/issue16398 opened by sfllaw #16399: argparse: append action with default list adds to list instead http://bugs.python.org/issue16399 opened by Markus.Amalthea.Magnuson #16400: update default PyPI behavior in docs re: listing versions http://bugs.python.org/issue16400 opened by chris.jerdonek #16401: mention PKG-INFO in the documentation http://bugs.python.org/issue16401 opened by chris.jerdonek #16403: update distutils docs to say that maintainer replaces author http://bugs.python.org/issue16403 opened by chris.jerdonek #16404: Uses of PyLong_FromLong that don't check for errors http://bugs.python.org/issue16404 opened by nedbat #16405: Explain how to set up the whitespace commit hook locally http://bugs.python.org/issue16405 opened by ncoghlan #16406: move the "Uploading Packages" section to distutils/packageinde http://bugs.python.org/issue16406 opened by chris.jerdonek #16408: FD leaks in zipfile http://bugs.python.org/issue16408 opened by Eric.Busboom #16409: urlretrieve regression: first call returns block size as 0 http://bugs.python.org/issue16409 opened by techtonik #16411: zlib.Decompress.decompress() retains pointer to input buffer w http://bugs.python.org/issue16411 opened by nadeem.vawda #16413: Non cross-platform behavior of os.path.split http://bugs.python.org/issue16413 opened by techtonik #16416: Mac OS X: don't use the locale encoding but UTF-8 to encode an http://bugs.python.org/issue16416 opened by haypo #16418: argparse with many choices can generate absurdly long usage me http://bugs.python.org/issue16418 opened by Juraj.Variny #16421: importlib.machinery.ExtensionFileLoader cannot load several mo http://bugs.python.org/issue16421 opened by eudoxos #16422: Decimal constants should be the same for py & c module version http://bugs.python.org/issue16422 opened by yselivanov #16423: urllib data URL recipe http://bugs.python.org/issue16423 opened by panzi #16424: regression: os.path.split('//hostname/foo/bar.txt') http://bugs.python.org/issue16424 opened by techtonik #16425: minidom replaceChild(new_child, old_child) removes new_child e http://bugs.python.org/issue16425 opened by Martin.Kugler #16427: Faster hash implementation http://bugs.python.org/issue16427 opened by serhiy.storchaka #16428: turtle with compound shape doesn't get clicks http://bugs.python.org/issue16428 opened by pythonick #16429: Emit SyntaxWarning for code that risks UnboundLocalError http://bugs.python.org/issue16429 opened by ncoghlan #16434: SocketServer call shutdown in the wrong way http://bugs.python.org/issue16434 opened by Erik.G??nther #16436: Link directly to set and frozenset in built-in function docs http://bugs.python.org/issue16436 opened by fossilet #16437: issubclass doc improvement http://bugs.python.org/issue16437 opened by fossilet #16438: Numeric operator predecence confusing http://bugs.python.org/issue16438 opened by fossilet #16442: PATH_MAX vs MAXPATHLEN vs pathconf(..., _PC_PATH_MAX). http://bugs.python.org/issue16442 opened by trent #16443: Add docstrings to regular expression match objects http://bugs.python.org/issue16443 opened by rhettinger #16444: Use support.TESTFN_UNDECODABLE on UNIX http://bugs.python.org/issue16444 opened by haypo #16445: SEGFAULT when deleting Exception.message http://bugs.python.org/issue16445 opened by amaury.forgeotdarc #16446: pdb raises BdbQuit on 'quit' when started with set_trace http://bugs.python.org/issue16446 opened by xdegaye #16447: SEGFAULT when setting type.__name__ http://bugs.python.org/issue16447 opened by amaury.forgeotdarc #16432: Template strings documentation in Python 3 refers to % substit http://bugs.python.org/issue16432 opened by andrewsg Most recent 15 issues with no replies (15) ========================================== #16446: pdb raises BdbQuit on 'quit' when started with set_trace http://bugs.python.org/issue16446 #16442: PATH_MAX vs MAXPATHLEN vs pathconf(..., _PC_PATH_MAX). http://bugs.python.org/issue16442 #16429: Emit SyntaxWarning for code that risks UnboundLocalError http://bugs.python.org/issue16429 #16428: turtle with compound shape doesn't get clicks http://bugs.python.org/issue16428 #16427: Faster hash implementation http://bugs.python.org/issue16427 #16425: minidom replaceChild(new_child, old_child) removes new_child e http://bugs.python.org/issue16425 #16423: urllib data URL recipe http://bugs.python.org/issue16423 #16416: Mac OS X: don't use the locale encoding but UTF-8 to encode an http://bugs.python.org/issue16416 #16406: move the "Uploading Packages" section to distutils/packageinde http://bugs.python.org/issue16406 #16403: update distutils docs to say that maintainer replaces author http://bugs.python.org/issue16403 #16400: update default PyPI behavior in docs re: listing versions http://bugs.python.org/issue16400 #16398: deque.rotate() could be much faster http://bugs.python.org/issue16398 #16386: imp.find_module does not specify registry key it searches on w http://bugs.python.org/issue16386 #16382: Better warnings exception for bad category http://bugs.python.org/issue16382 #16379: SQLite error code not exposed to python http://bugs.python.org/issue16379 Most recent 15 issues waiting for review (15) ============================================= #16447: SEGFAULT when setting type.__name__ http://bugs.python.org/issue16447 #16445: SEGFAULT when deleting Exception.message http://bugs.python.org/issue16445 #16444: Use support.TESTFN_UNDECODABLE on UNIX http://bugs.python.org/issue16444 #16427: Faster hash implementation http://bugs.python.org/issue16427 #16423: urllib data URL recipe http://bugs.python.org/issue16423 #16421: importlib.machinery.ExtensionFileLoader cannot load several mo http://bugs.python.org/issue16421 #16416: Mac OS X: don't use the locale encoding but UTF-8 to encode an http://bugs.python.org/issue16416 #16411: zlib.Decompress.decompress() retains pointer to input buffer w http://bugs.python.org/issue16411 #16408: FD leaks in zipfile http://bugs.python.org/issue16408 #16395: Documentation claims that PySequence_Fast returns a tuple, whe http://bugs.python.org/issue16395 #16392: import crashes on circular imports in ext modules http://bugs.python.org/issue16392 #16389: re._compiled_typed's lru_cache causes significant degradation http://bugs.python.org/issue16389 #16386: imp.find_module does not specify registry key it searches on w http://bugs.python.org/issue16386 #16382: Better warnings exception for bad category http://bugs.python.org/issue16382 #16381: Introduce option to force the interpreter to exit upon MemoryE http://bugs.python.org/issue16381 Top 10 most discussed issues (10) ================================= #14621: Hash function is not randomized properly http://bugs.python.org/issue14621 36 msgs #16218: Python launcher does not support unicode characters http://bugs.python.org/issue16218 32 msgs #16353: add function to os module for getting path to default shell http://bugs.python.org/issue16353 32 msgs #9584: Allow curly brace expansion http://bugs.python.org/issue9584 21 msgs #14794: slice.indices raises OverflowError http://bugs.python.org/issue14794 18 msgs #16392: import crashes on circular imports in ext modules http://bugs.python.org/issue16392 17 msgs #16389: re._compiled_typed's lru_cache causes significant degradation http://bugs.python.org/issue16389 16 msgs #16421: importlib.machinery.ExtensionFileLoader cannot load several mo http://bugs.python.org/issue16421 15 msgs #10030: Patch for zip decryption speedup http://bugs.python.org/issue10030 12 msgs #13349: Non-informative error message in index() and remove() function http://bugs.python.org/issue13349 8 msgs Issues closed (54) ================== #4318: optparse: formatting of help text/descriptions http://bugs.python.org/issue4318 closed by eric.araujo #4711: Wide literals in the table of contents overflow in documentati http://bugs.python.org/issue4711 closed by ezio.melotti #5765: stack overflow evaluating eval("()" * 30000) http://bugs.python.org/issue5765 closed by ncoghlan #5894: Lookup of localised language name by ISO 639 language code and http://bugs.python.org/issue5894 closed by christian.heimes #6709: It's possible to create TryExcept with no handlers http://bugs.python.org/issue6709 closed by benjamin.peterson #7317: Display full tracebacks when an error occurs asynchronously http://bugs.python.org/issue7317 closed by asvetlov #8271: str.decode('utf8', 'replace') -- conformance with Unicode 5.2. http://bugs.python.org/issue8271 closed by ezio.melotti #8401: Strange behavior of bytearray slice assignment http://bugs.python.org/issue8401 closed by ezio.melotti #9109: absolute import cleanups for Python 3 http://bugs.python.org/issue9109 closed by akuchling #10358: Doc styles for print should only use dark colors http://bugs.python.org/issue10358 closed by ezio.melotti #10385: Mark up "subprocess" as module in its doc http://bugs.python.org/issue10385 closed by ezio.melotti #11166: No exit when daemon thread is running. http://bugs.python.org/issue11166 closed by akuchling #11313: Speed up default encode()/decode() http://bugs.python.org/issue11313 closed by ezio.melotti #11481: The copy module already uses copyreg http://bugs.python.org/issue11481 closed by ezio.melotti #11631: Python 2.7.1 64bit, Win7 64bit problem to read and write with http://bugs.python.org/issue11631 closed by christian.heimes #11842: slice.indices with negative step and default stop http://bugs.python.org/issue11842 closed by mark.dickinson #12759: "(?P=)" input for Tools/scripts/redemo.py raises unnhandled ex http://bugs.python.org/issue12759 closed by ezio.melotti #13301: the script Tools/i18n/msgfmt.py allows arbitrary code executio http://bugs.python.org/issue13301 closed by ezio.melotti #15001: segmentation fault with del sys.modules['__main__'] http://bugs.python.org/issue15001 closed by hynek #15076: Sometimes couldn't import os, shown 'import site' failed, use http://bugs.python.org/issue15076 closed by akuchling #15165: test_email: failure on Windows http://bugs.python.org/issue15165 closed by python-dev #15641: Clean up importlib for Python 3.4 http://bugs.python.org/issue15641 closed by asvetlov #15814: memoryview: equality-hash invariant http://bugs.python.org/issue15814 closed by skrah #15837: Added test to test_random.py testing Random.shuffle http://bugs.python.org/issue15837 closed by pitrou #16048: Tutorial-classes-remarks: replace paragraph http://bugs.python.org/issue16048 closed by ezio.melotti #16152: Trailing whitespace makes tokenize.generate_tokens pathologica http://bugs.python.org/issue16152 closed by ezio.melotti #16183: ZipExtFile object close without file handle closed http://bugs.python.org/issue16183 closed by python-dev #16284: concurrent.futures ThreadPoolExecutor keeps unnecessary refere http://bugs.python.org/issue16284 closed by asvetlov #16304: re: Match Objects always have a boolean value of True http://bugs.python.org/issue16304 closed by ezio.melotti #16309: "PYTHONPATH=" different from no PYTHONPATH at all http://bugs.python.org/issue16309 closed by asvetlov #16311: Use _PyUnicodeWriter API in text decoders http://bugs.python.org/issue16311 closed by python-dev #16336: Check input in surrogatepass error handler http://bugs.python.org/issue16336 closed by ezio.melotti #16338: pysnmp/asyncore - timeout ineffective? http://bugs.python.org/issue16338 closed by neologix #16354: Remember python version choice on docs.python.org http://bugs.python.org/issue16354 closed by georg.brandl #16385: evaluating literal dict with repeated keys gives no warnings/e http://bugs.python.org/issue16385 closed by benjamin.peterson #16390: re compilation slow in Python 3.3 due to functools.lru_cache o http://bugs.python.org/issue16390 closed by pjenvey #16393: typo in pkgutil.py http://bugs.python.org/issue16393 closed by ezio.melotti #16402: range slicing swallows exceptions from __index__ methods. http://bugs.python.org/issue16402 closed by mark.dickinson #16407: Python 2.7.3: test_sys fails when building under 10.7.5 with d http://bugs.python.org/issue16407 closed by ned.deily #16410: rewrite winsound with ctypes http://bugs.python.org/issue16410 closed by loewis #16412: test_shutil: 2 failures related to symlinks http://bugs.python.org/issue16412 closed by pitrou #16414: Add support.NONASCII to test non-ASCII characters http://bugs.python.org/issue16414 closed by haypo #16415: eventlet.monkey_patch() breaks subprocess.Popen on Windows http://bugs.python.org/issue16415 closed by r.david.murray #16417: Integer converted in tuple without request http://bugs.python.org/issue16417 closed by r.david.murray #16419: email.message._headers is a list http://bugs.python.org/issue16419 closed by ezio.melotti #16420: PEP 249 (DB-API 2.0) converted to reStructuredText http://bugs.python.org/issue16420 closed by lemburg #16426: RoundUp signature removal test (please ignore) http://bugs.python.org/issue16426 closed by lemburg #16430: re.match blocking and taking 100% CPU http://bugs.python.org/issue16430 closed by serhiy.storchaka #16431: CDecimal disregards subclass passed into __new__ when value ar http://bugs.python.org/issue16431 closed by skrah #16433: unittest.TestCase.assertNotEqual has incorrect docstring. http://bugs.python.org/issue16433 closed by ezio.melotti #16435: Python 3 doc link to Python 2 FAQ http://bugs.python.org/issue16435 closed by asvetlov #16439: Code not collapsed correctly http://bugs.python.org/issue16439 closed by asvetlov #16440: Exception type http://bugs.python.org/issue16440 closed by ezio.melotti #16441: range usage in gzip module leads to excessive memory usage. http://bugs.python.org/issue16441 closed by cjw296 From chris.jerdonek at gmail.com Fri Nov 9 18:48:18 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 9 Nov 2012 09:48:18 -0800 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <509CE1A3.2000202@simplistix.co.uk> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> Message-ID: <CAOTb1wcOwCencss7EuPDSdfzM0EGdCHaaFr2fuO6OkQteLkZOA@mail.gmail.com> On Fri, Nov 9, 2012 at 2:57 AM, Chris Withers <chris at simplistix.co.uk> wrote: > On 09/11/2012 10:52, Michael Foord wrote: >> >> It should be python.exe (yes really). > > Hah! Should http://docs.python.org/devguide/ be updated to reflect this or > does this only affect Mac OS? (or should we correct the build so it doesn't > spit out a .exe?) That link already mentions this information in the top "Quick Start" section: "3. Run the tests with ./python -m test -j3 (use ./python.exe on most Mac OS X systems and PCbuild\python_d.exe on Windows; replace test with test.regrtest for 2.7)." But it would probably be good to repeat this information in certain locations, for example in the more detailed sections on building and running tests. --Chris From solipsis at pitrou.net Fri Nov 9 20:20:50 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 9 Nov 2012 20:20:50 +0100 Subject: [Python-Dev] problems building python2.7 References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> <509CEFC2.9020205@simplistix.co.uk> Message-ID: <20121109202050.346dadaa@pitrou.net> On Fri, 09 Nov 2012 11:57:54 +0000 Chris Withers <chris at simplistix.co.uk> wrote: > On 09/11/2012 11:54, Ronald Oussoren wrote: > > > > On 9 Nov, 2012, at 11:57, Chris Withers<chris at simplistix.co.uk> wrote: > > > >> On 09/11/2012 10:52, Michael Foord wrote: > >>> > >>>> However, I can't find the python it's built... > >>> > >>> It should be python.exe (yes really). > >> > >> Hah! Should http://docs.python.org/devguide/ be updated to reflect this or does this only affect Mac OS? (or should we correct the build so it doesn't spit out a .exe?) > > > > How else should it be named, given that the default filesystem on OSX machines is not case sensitive. The binary cannot be named "python" because that clashes with the directory named "Python" in the source and build tres. > > Maybe python.bin, or put it in the ./bin folder? > > I saw python.exe, but my brain went "oh, .exe, that's windows, I'll > ignore that then, since I'm on Mac OS" Or just re-wire your brain. Regards Antoine. From larry at hastings.org Fri Nov 9 22:05:50 2012 From: larry at hastings.org (Larry Hastings) Date: Fri, 09 Nov 2012 19:05:50 -0200 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> Message-ID: <509D702E.10605@hastings.org> On 11/09/2012 09:54 AM, Ronald Oussoren wrote: > On 9 Nov, 2012, at 11:57, Chris Withers <chris at simplistix.co.uk> wrote: >> Hah! Should http://docs.python.org/devguide/ be updated to reflect this or does this only affect Mac OS? (or should we correct the build so it doesn't spit out a .exe?) > How else should it be named, given that the default filesystem on OSX machines is not case sensitive. The binary cannot be named "python" because that clashes with the directory named "Python" in the source and build tres. Have we considered renaming the directory instead? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121109/d3c17846/attachment.html> From nad at acm.org Fri Nov 9 23:32:18 2012 From: nad at acm.org (Ned Deily) Date: Fri, 09 Nov 2012 14:32:18 -0800 Subject: [Python-Dev] problems building python2.7 References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <59B407E8-B1B7-464C-A0D3-5B692953763A@mac.com> <509D702E.10605@hastings.org> Message-ID: <nad-D5F982.14321809112012@news.gmane.org> In article <509D702E.10605 at hastings.org>, Larry Hastings <larry at hastings.org> wrote: > On 11/09/2012 09:54 AM, Ronald Oussoren wrote: > > On 9 Nov, 2012, at 11:57, Chris Withers <chris at simplistix.co.uk> wrote: > >> Hah! Should http://docs.python.org/devguide/ be updated to reflect this or > >> does this only affect Mac OS? (or should we correct the build so it > >> doesn't spit out a .exe?) > > How else should it be named, given that the default filesystem on OSX > > machines is not case sensitive. The binary cannot be named "python" because > > that clashes with the directory named "Python" in the source and build > > tres. > > Have we considered renaming the directory instead? That's another solution. Another is to use the optional case-sensitive variant of HFS+ on OS X. But, really, this is a very minor one-time issue facing people doing development on Python itself. Normal users or Python don't run iit from a source directory, they run it from an installed location (make install) and then there is no such problem. It's not worth changing at this point. -- Ned Deily, nad at acm.org From hansmu at xs4all.nl Sat Nov 10 15:46:47 2012 From: hansmu at xs4all.nl (Hans Mulder) Date: Sat, 10 Nov 2012 15:46:47 +0100 Subject: [Python-Dev] problems building python2.7 In-Reply-To: <20121109160714.4327E25003E@webabinitio.net> References: <509CD382.4080609@simplistix.co.uk> <999BF3A1-7456-4194-ADF7-0C68D59A8D16@voidspace.org.uk> <509CE1A3.2000202@simplistix.co.uk> <k7io94$o92$1@ger.gmane.org> <k7iqte$eq5$1@ger.gmane.org> <509D0F0E.7080306@simplistix.co.uk> <k7j8bv$98f$1@ger.gmane.org> <20121109160714.4327E25003E@webabinitio.net> Message-ID: <k7lpcm$3vc$1@ger.gmane.org> On 9/11/12 17:07:13, R. David Murray wrote: > On Fri, 09 Nov 2012 16:44:00 +0100, Hans Mulder <hansmu at xs4all.nl> wrote: >> I looked into it, and the problem is this bit of code (line 230-235): >> >> try: >> self.assertTrue(os.path.exists(tmp_file)) >> fp = urllib.urlopen(tmp_fileurl) >> finally: >> os.close(fd) >> fp.close() >> >> Due to a misconfiguration, urllib.thishost() raises an IOError on my >> laptop. This causes urllib.urlopen to raise an exception, and the >> name fp is never bound, so that the fp.close() in the finally clause >> raises an UnboundLocalError, masking the problem in urlopen. >> >> A quick fix would be: >> >> try: >> self.assertTrue(os.path.exists(tmp_file)) >> fp = urllib.urlopen(tmp_fileurl) >> fp.close() >> finally: >> os.close(fd) >> >> That way, the .close is only attempted if the open succeeds. > > Could you open an issue for this on the tracker, please? That > way we won't forget to fix it. Done: issue 16450. -- HansM From nadeem.vawda at gmail.com Sat Nov 10 22:54:23 2012 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Sat, 10 Nov 2012 22:54:23 +0100 Subject: [Python-Dev] [Python-checkins] cpython (merge 3.3 -> default): Fixes issue #16409: The reporthook callback made by the legacy In-Reply-To: <3XzWwq3HNVzPt0@mail.python.org> References: <3XzWwq3HNVzPt0@mail.python.org> Message-ID: <CANF4RM=wYFEvHiGphj27bUxOFcXyYMw9DTFzuwKEeD5j3PmdDQ@mail.gmail.com> This change appears to have broken test_urllib. See, for example: http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/4774/steps/test/logs/stdio - Nadeem -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121110/74206ebe/attachment.html> From tismer at stackless.com Sun Nov 11 21:31:08 2012 From: tismer at stackless.com (Christian Tismer) Date: Sun, 11 Nov 2012 21:31:08 +0100 Subject: [Python-Dev] Setting project home path the best way Message-ID: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> Hi friends, I have a project that has its root somewhere on my machine. This project has many folders and contains quite some modules. There is a common root of the module tree, and I want to use - either absolute imports - relative imports with '.' Problem: - I want to run any module inside the heirarchy from the command-line - this should work, regardless what my 'cwd' is - this should work with or without virtualenv. So far, things work fine with virtualenv, because sys.executable is in the project module tree. Without virtualenv, this is not so. But I hate to make settings like PYTHONPATH, because these are not permanent. . Question: How should I define my project root dir in a unique way, without setting an environment variable? What is the lest intrusive way to spell that? Reason: I'd like to make things work correctly and unambigously when I call a script inside the module heirarchy. Things are not fixed: there exist many checkouts In the file system, and each should know where to search its home/root in the tree. Is this elegantly possible to deduce from the actually executed script file? Cheers - chris Sent from my Ei4Steve From tismer at stackless.com Sun Nov 11 21:53:40 2012 From: tismer at stackless.com (Christian Tismer) Date: Sun, 11 Nov 2012 21:53:40 +0100 Subject: [Python-Dev] Relatwd: py3 import strategy (was: Setting project home path the best way( In-Reply-To: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> Message-ID: <9A3375A4-D61D-4938-983C-553CFDDAFC29@stackless.com> Once again on this: With the introduction of module folders without __init__, it has become even harder to deduce a sensible project root Dir without relying on a global settings file. Abd as you probably agree, such files are a bad idea. Any global is bad, also in the file system. Guido, this is another reason why I dislike the absence of __init__. : Rhere is no longer an indicator that pretty clearly defines the root of my module heirarchy. Cheers, hoping for enlightment - chris Sent from my Ei4Steve On Nov 11, 2012, at 21:31, Christian Tismer <tismer at stackless.com> wrote: > Hi friends, > > I have a project that has its root somewhere on my machine. > This project has many folders and contains quite some modules. > > There is a common root of the module tree, and I want to use > - either absolute imports > - relative imports with '.' > > Problem: > > - I want to run any module inside the heirarchy from the command-line > > - this should work, regardless what my 'cwd' is > > - this should work with or without virtualenv. > > So far, things work fine with virtualenv, because sys.executable is in the project module tree. > > Without virtualenv, this is not so. But I hate to make settings like PYTHONPATH, because these are not permanent. . > > Question: > > How should I define my project root dir in a unique way, without setting an environment variable? > What is the lest intrusive way to spell that? > > Reason: > > I'd like to make things work correctly and unambigously when I call a script > inside the module heirarchy. Things are not fixed: there exist many checkouts > In the file system, and each should know where to search its home/root in the tree. > > Is this elegantly possible to deduce from the actually executed script file? > > Cheers - chris > > Sent from my Ei4Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/tismer%40stackless.com From ericsnowcurrently at gmail.com Sun Nov 11 22:45:23 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 11 Nov 2012 14:45:23 -0700 Subject: [Python-Dev] Relatwd: py3 import strategy (was: Setting project home path the best way( In-Reply-To: <9A3375A4-D61D-4938-983C-553CFDDAFC29@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <9A3375A4-D61D-4938-983C-553CFDDAFC29@stackless.com> Message-ID: <CALFfu7AGJOaAu=z7x1e-GWwmaWo+G_HOFsAR=_G7x5X70t57MQ@mail.gmail.com> On Sun, Nov 11, 2012 at 1:53 PM, Christian Tismer <tismer at stackless.com>wrote: > Once again on this: > > With the introduction of module folders > without __init__, it has become even harder to deduce a sensible project > root > Dir without relying on a global settings file. Abd as you probably agree, > such files are a bad idea. Any global is bad, also in the file system. > > Guido, this is another reason why I dislike the absence of __init__. : > > Rhere is no longer an indicator that pretty clearly defines the root of my > module heirarchy. > This came up during the discussions leading up to PEP 420, particularly regarding the impact on PEP 395. My thoughts at the time on addressing the problem: http://mail.python.org/pipermail/import-sig/2012-March/000438.html -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121111/da19ec0f/attachment.html> From tismer at stackless.com Sun Nov 11 22:59:34 2012 From: tismer at stackless.com (Christian Tismer) Date: Sun, 11 Nov 2012 22:59:34 +0100 Subject: [Python-Dev] Relatwd: py3 import strategy In-Reply-To: <CALFfu7AGJOaAu=z7x1e-GWwmaWo+G_HOFsAR=_G7x5X70t57MQ@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <9A3375A4-D61D-4938-983C-553CFDDAFC29@stackless.com> <CALFfu7AGJOaAu=z7x1e-GWwmaWo+G_HOFsAR=_G7x5X70t57MQ@mail.gmail.com> Message-ID: <50A01FC6.7050803@stackless.com> Hi Eric, On 11.11.12 22:45, Eric Snow wrote: > On Sun, Nov 11, 2012 at 1:53 PM, Christian Tismer > <tismer at stackless.com <mailto:tismer at stackless.com>> wrote: > > Once again on this: > > With the introduction of module folders > without __init__, it has become even harder to deduce a sensible > project root > Dir without relying on a global settings file. Abd as you probably > agree, such files are a bad idea. Any global is bad, also in the > file system. > > Guido, this is another reason why I dislike the absence of __init__. : > > Rhere is no longer an indicator that pretty clearly defines the > root of my module heirarchy. > > > This came up during the discussions leading up to PEP 420, > particularly regarding the impact on PEP 395. My thoughts at the time > on addressing the problem: > > http://mail.python.org/pipermail/import-sig/2012-March/000438.html > I do not quite understand what you want to propose: """ * there is no __init__.py in the current directory (in the case where there was one adjacent to the original module) * current directory is on sys.path * setup.py is in the current directory """ Are you saying that this is a way for me to get at the project root? And can this be done with a one-liner, or would I need a module for that, where the cat would bite its tail, because that module would not be found, easily? Please tell me in a simpler way what you think ;-) ciao - Chris -- Christian Tismer :^) <mailto:tismer at stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121111/efc52972/attachment.html> From ncoghlan at gmail.com Mon Nov 12 01:38:40 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Nov 2012 10:38:40 +1000 Subject: [Python-Dev] Setting project home path the best way In-Reply-To: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> Message-ID: <CADiSq7c8XiH1sOT6q53H=vScMbgeq1L3x2-OVYEmj0X8Gvfx+A@mail.gmail.com> The only way I know how to do it is to have my cwd set to the directory I want on sys.path, then use -m for script execution (using a separate shell session for anything where I want a different working directory). I don't know of any way to handle a varying cwd without manipulating the path directly through either virtualenv or PYTHONPATH. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121112/06a15458/attachment.html> From tismer at stackless.com Mon Nov 12 13:48:57 2012 From: tismer at stackless.com (Christian Tismer) Date: Mon, 12 Nov 2012 13:48:57 +0100 Subject: [Python-Dev] Setting project home path the best way SOLVED In-Reply-To: <CADiSq7c8XiH1sOT6q53H=vScMbgeq1L3x2-OVYEmj0X8Gvfx+A@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <CADiSq7c8XiH1sOT6q53H=vScMbgeq1L3x2-OVYEmj0X8Gvfx+A@mail.gmail.com> Message-ID: <50A0F039.4010401@stackless.com> Hi Nick, Holger, this is a crazy fault of mine, see the end below. I wrote: > I have a project that has its root somewhere on my machine. > This project has many folders and contains quite some modules. > > There is a common root of the module tree, and I want to use > - either absolute imports > - relative imports with '.' > > Problem: > > - I want to run any module inside the heirarchy from the command-line > > - this should work, regardless what my 'cwd' is > > - this should work with or without virtualenv. > > So far, things work fine with virtualenv, because sys.executable is in the project module tree. > > Without virtualenv, this is not so. But I hate to make settings like PYTHONPATH, because these are not permanent. . > > Question: > > How should I define my project root dir in a unique way, without setting an environment variable? > What is the lest intrusive way to spell that? > > Reason: > > I'd like to make things work correctly and unambigously when I call a script > inside the module heirarchy. Things are not fixed: there exist many checkouts > In the file system, and each should know where to search its home/root in the tree. > > Is this elegantly possible to deduce from the actually executed script file? > Nick wrote: On 12.11.12 01:38, Nick Coghlan wrote: > > The only way I know how to do it is to have my cwd set to the > directory I want on sys.path, then use -m for script execution (using > a separate shell session for anything where I want a different working > directory). > > I don't know of any way to handle a varying cwd without manipulating > the path directly through either virtualenv or PYTHONPATH. > > Cheers, > Nick. > > -- > Sent from my phone, thus the relative brevity :) > Holger wrote (private message): > ich selbst nutze seit langer Zeit setuptools, genauer: > > python setup.py develop > > z.B. in pytest, execnet, tox. Das ist wie "setup.py install" nur dass > das paket "inplace" installiert wird. Aber vermutlich kennst du das schon?! So here is the crazy story: I'm actually using setuptools for my tiffany project with setup.py develop My real work is about the Pydica project which is a lot more. There I'm playing sys.path tricks, using virtualenv and the fact that I have full control over the local site-packages, and use some tricky .pth scripts to set everything up. But it needs virtualenv. Believe it or not: *I was not able to realize that all my problems are already solved* if I change Pydica and build a proper setup using setuptools. I needed to bug python-dev with this, and finally got the right hint from Holger Krekel who is absolutely right here - no point in playing tricks, because this is all solved. And Pydica is about to be prepared for PyPI, anyway. My blocker was probably that Pydica is very much in development, and I did not think right about it. The project does not need to be ready in order to use setuptools! My apologies, and many thanks again! This solved a Gordian Knot. It is great when people help me to leave a dead-lock in my brain :-) cheers - Chris -- Christian Tismer :^) <mailto:tismer at stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From jmatejek at suse.cz Mon Nov 12 16:30:41 2012 From: jmatejek at suse.cz (=?UTF-8?B?SmFuIE1hdMSbamVr?=) Date: Mon, 12 Nov 2012 16:30:41 +0100 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <5091A31D.4000904@pearwood.info> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> Message-ID: <50A11621.6080201@suse.cz> Dne 31.10.2012 23:15, Steven D'Aprano napsal(a): >> I wonder if this message can be improved with a >> pointer to the concept on when global variables become local? > > If you have a suggestion for an improved message, please tell us. Or raise > an issue on the bug tracker. I believe the "human problem" here is that the one tends to gloss over "local variable VARNAME", because it describes VARNAME and you already think you know what that is, so you don't stop to think about it. The following would be better in this regard, IMO: "Variable VARNAME is local in FUNCNAME, but doesn't have a value at line 123" regards m. From solipsis at pitrou.net Mon Nov 12 23:08:59 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Nov 2012 23:08:59 +0100 Subject: [Python-Dev] cpython: Issue #16416: OS data are now always encoded/decoded to/from References: <3Y0mFY3pjgzQ8T@mail.python.org> Message-ID: <20121112230859.10cf9c98@pitrou.net> On Mon, 12 Nov 2012 23:03:45 +0100 (CET) victor.stinner <python-checkins at python.org> wrote: > > +- Issue #16416: OS data are now always encoded/decoded to/from > + UTF-8/surrogateescape, instead of the locale encoding (which may be ASCII if > + no locale environment variable is set), to avoid inconsistencies with > + os.fsencode() and os.fsdecode() functions which are already using > + UTF-8/surrogateescape. I believe you mean "OS X", not "OS". Regards Antoine. From mike_mp at zzzcomputing.com Mon Nov 12 23:15:07 2012 From: mike_mp at zzzcomputing.com (Michael Bayer) Date: Mon, 12 Nov 2012 17:15:07 -0500 Subject: [Python-Dev] Okay to document the "exec tuple" form of the exec statement in Python 2.7? In-Reply-To: <509834C5.8060302@canterbury.ac.nz> References: <CAAu3qLXJNOgLSnHxcTdFPYrfc2eCkRvKsKotw7LH=nzU2Mixpw@mail.gmail.com> <CADiSq7fuYDPJawjKejJn6rn6aYpydZGfYyJv5SxoOUVom=i4hQ@mail.gmail.com> <509834C5.8060302@canterbury.ac.nz> Message-ID: <FC5B9D06-CE61-4881-8B50-9D8BD0D912DB@zzzcomputing.com> On Nov 5, 2012, at 4:51 PM, Greg Ewing wrote: > Nick Coghlan wrote: >> On Mon, Nov 5, 2012 at 10:08 PM, Mark Dickinson <dickinsm at gmail.com> wrote: >>> In Python 2, the 'exec' statement supports 'exec'-ing a (statement, >>> globals, locals) tuple: > > If this is a deliberate feature, I'd guess it's because exec > is a statement rather than a function in Python 2, so you > can't use * to pass a tuple of arguments to it. any chance, if this function is documented, someone can make it very clear what the cross-compatibility implications are with the Python 2.x form of exec() and the Python 3.x form ? I'm having a hard time demonstrating the difference to myself, though the winning answer to his StackOverflow question http://stackoverflow.com/questions/6561482/why-did-python-3-changes-to-exec-break-this-code seems to suggest the behavior of symbol tables has changed. > > -- > Greg > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mike_mp%40zzzcomputing.com > From v+python at g.nevcal.com Tue Nov 13 00:52:15 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 12 Nov 2012 15:52:15 -0800 Subject: [Python-Dev] Improve error message "UnboundLocalError: local variable referenced before assignment" In-Reply-To: <50A11621.6080201@suse.cz> References: <CAPkN8x+ne38WqqqutwknQb=YRVrxtkYodorZXTg6L0hR-3kc5A@mail.gmail.com> <5091A31D.4000904@pearwood.info> <50A11621.6080201@suse.cz> Message-ID: <50A18BAF.9000105@g.nevcal.com> On 11/12/2012 7:30 AM, Jan Mat?jek wrote: > > I believe the "human problem" here is that the one tends to gloss over > "local variable VARNAME", because it describes VARNAME and you already > think you know what that is, so you don't stop to think about it. > > The following would be better in this regard, IMO: > > "Variable VARNAME is local in FUNCNAME, but doesn't have a value at > line 123" +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121112/d30fa5d7/attachment.html> From ncoghlan at gmail.com Tue Nov 13 03:04:33 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Nov 2012 12:04:33 +1000 Subject: [Python-Dev] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAPZV6o8=Uk5zn+f7kGCs6cFQnq50Nd46yOpu9FMwu4p6ctbE4A@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> Message-ID: <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth <dholth at gmail.com> wrote: > I think Metadata 1.3 is done. Who would like to czar? > (Apologies for the belated reply, it's been a busy few weeks) I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated with some additional rationale based on Ronald's comments later in this thread, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/287b4dca/attachment.html> From martin at v.loewis.de Tue Nov 13 10:51:04 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 10:51:04 +0100 Subject: [Python-Dev] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> Message-ID: <50A21808.8040909@v.loewis.de> Am 13.11.12 03:04, schrieb Nick Coghlan: > On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth <dholth at gmail.com > <mailto:dholth at gmail.com>> wrote: > > I think Metadata 1.3 is done. Who would like to czar? > > (Apologies for the belated reply, it's been a busy few weeks) > > I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > with some additional rationale based on Ronald's comments later in this > thread, though. For the record, I'm still -1 on PEP 427, because of the signature issues. The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot readily be used to verify the integrity of an archive - the whole point of these technologies is to do exactly that. The FAQ is entirely silent on why it is not using a more standard signature algorithm such as ECDSA. It explains why it uses Ed25519, but ignores that the very same rationale would apply to ECDSA as well; plus that would be one of the standard JWS algorithms. In addition, the FAQ claims that the format is designed to introduce cryptopgraphy that is actually used, yet leaves the issue of key distribution alone (except that pointing out that you can put them into requires.txt - a file that doesn't seem to be specified anywhere). Regards, Martin From mal at egenix.com Tue Nov 13 11:26:50 2012 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 13 Nov 2012 11:26:50 +0100 Subject: [Python-Dev] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A21808.8040909@v.loewis.de> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> Message-ID: <50A2206A.40804@egenix.com> On 13.11.2012 10:51, "Martin v. L?wis" wrote: > Am 13.11.12 03:04, schrieb Nick Coghlan: >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth <dholth at gmail.com >> <mailto:dholth at gmail.com>> wrote: >> >> I think Metadata 1.3 is done. Who would like to czar? >> >> (Apologies for the belated reply, it's been a busy few weeks) >> >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated >> with some additional rationale based on Ronald's comments later in this >> thread, though. > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > readily be used to verify the integrity of an archive - the whole > point of these technologies is to do exactly that. > > The FAQ is entirely silent on why it is not using a more standard > signature algorithm such as ECDSA. It explains why it uses Ed25519, > but ignores that the very same rationale would apply to ECDSA as well; > plus that would be one of the standard JWS algorithms. > > In addition, the FAQ claims that the format is designed to introduce > cryptopgraphy that is actually used, yet leaves the issue of key > distribution alone (except that pointing out that you can put them > into requires.txt - a file that doesn't seem to be specified anywhere). I agree with Martin. If the point is to "to protect against cryptography that is not used", then not using the de-facto standard in signing open source distribution files, which today is PGP/GPG, misses that point :-) Note that signing such distribution files can be handled outside of the wheel format PEP. It just way to complex and out of scope for the wheel format itself. Also note that PGP/GPG and the other signing tools work well on any distribution file. There's really no need to build these into the format itself. It's a good idea to check integrity, but that can be done using hashes. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 13 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Nov 13 11:39:52 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Nov 2012 10:39:52 +0000 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <CACac1F_ja5ZFCegEiGeXH2=np06_v0uzoX+hZnsFcAfP9WVrXQ@mail.gmail.com> On 13 November 2012 10:26, M.-A. Lemburg <mal at egenix.com> wrote: > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > I agree as well. For me, the main reason for cryptography not being used is key distribution. Sure, I have a signed file, but without a key what's the point? And if I'm creating a file, why sign it if I don't know how to securely publish my key? So inventing a new signing infrastructure without a key distribution process doesn't encourage me to use crypto at all... > It's a good idea to check integrity, but that can be done using > hashes. > +1 hashing is fine, and I don't have any problem with the hashing aspects of the PEP. Maybe the signing aspects could be deferred to a subsequent PEP, to be thrashed out separately? I know Daniel has a strong interest in the signing aspect, so I'm reluctant to suggest just dropping it, but I'd rather it not be a showstopper for the rest of the current proposal. Paul. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/c3fbb4db/attachment.html> From kristjan at ccpgames.com Tue Nov 13 14:41:22 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 13 Nov 2012 13:41:22 +0000 Subject: [Python-Dev] externals? Message-ID: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> This may be a silly question, but haven't the python externals been moved to HG yet? I usually work on cpython without bothering with the externals, but I found today that I needed them. On Windows this is a bit of a bother. And I've thrown away all my SVN stuff... K -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/30868ba3/attachment.html> From dholth at gmail.com Tue Nov 13 16:00:26 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 10:00:26 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> On Tue, Nov 13, 2012 at 5:26 AM, M.-A. Lemburg <mal at egenix.com> wrote: > On 13.11.2012 10:51, "Martin v. L?wis" wrote: > > Am 13.11.12 03:04, schrieb Nick Coghlan: > >> On Mon, Oct 29, 2012 at 4:47 AM, Daniel Holth <dholth at gmail.com > >> <mailto:dholth at gmail.com>> wrote: > >> > >> I think Metadata 1.3 is done. Who would like to czar? > >> > >> (Apologies for the belated reply, it's been a busy few weeks) > >> > >> I'm happy to be BDFL delegate for these. I'd like to see PEP 425 updated > >> with some additional rationale based on Ronald's comments later in this > >> thread, though. > > > > For the record, I'm still -1 on PEP 427, because of the signature issues. > > > > The FAQ in the PEP is incorrect in claiming PGP or X.509 cannot > > readily be used to verify the integrity of an archive - the whole > > point of these technologies is to do exactly that. > > > > The FAQ is entirely silent on why it is not using a more standard > > signature algorithm such as ECDSA. It explains why it uses Ed25519, > > but ignores that the very same rationale would apply to ECDSA as well; > > plus that would be one of the standard JWS algorithms. > > > > In addition, the FAQ claims that the format is designed to introduce > > cryptopgraphy that is actually used, yet leaves the issue of key > > distribution alone (except that pointing out that you can put them > > into requires.txt - a file that doesn't seem to be specified anywhere). > > I agree with Martin. If the point is to "to protect against cryptography > that is not used", then not using the de-facto standard in signing > open source distribution files, which today is PGP/GPG, misses that > point :-) > > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. > > It's a good idea to check integrity, but that can be done using > hashes. PGP-on-pypi is the very definition of cryptography that isn't used. I'm willing to go ahead and move any mention of signing algorithms into a separate PEP, leaving only the basic manifest hash vs. file contents verification under the auspices of this PEP. Is the rest of the wheel specification, explaining how packages are actually produced and installed, clear? I want to remove distutils from the standard library. If that happens then we might want a secure way to install it from pypi. One way would be to include the public key used to sign distutils in Python's own signature-verifying bootstrap wheel installer, never mind whether it used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? https://www.updateframework.com/wiki/SecuringPythonPackageManagement -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/9b99dd27/attachment.html> From benjamin at python.org Tue Nov 13 16:04:17 2012 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 13 Nov 2012 10:04:17 -0500 Subject: [Python-Dev] externals? In-Reply-To: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> Their still in svn as far I know. 2012/11/13 Kristj?n Valur J?nsson <kristjan at ccpgames.com>: > This may be a silly question, but haven?t the python externals been moved to > HG yet? > > I usually work on cpython without bothering with the externals, but I found > today that I needed them. On Windows this is a bit of a bother. And I?ve > thrown away all my SVN stuff... > > K > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/benjamin%40python.org > -- Regards, Benjamin From dirkjan at ochtman.nl Tue Nov 13 16:04:50 2012 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 13 Nov 2012 16:04:50 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> Message-ID: <CAKmKYaBS+xKO+Ht+jYPRe0HRJHjw3D0iO_SjGhRtdfc5D5YdLA@mail.gmail.com> On Tue, Nov 13, 2012 at 4:00 PM, Daniel Holth <dholth at gmail.com> wrote: > I'm willing to go ahead and move any mention of signing algorithms into a > separate PEP, leaving only the basic manifest hash vs. file contents > verification under the auspices of this PEP. >From the discussion so far, that seems like the smart thing to do. Cheers, Dirkjan From benjamin at python.org Tue Nov 13 16:05:23 2012 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 13 Nov 2012 10:05:23 -0500 Subject: [Python-Dev] externals? In-Reply-To: <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> Message-ID: <CAPZV6o-dKAVM=T2KrActxhiwAaJ+eeZgVo1hd1aOGn4j_FzFGA@mail.gmail.com> 2012/11/13 Benjamin Peterson <benjamin at python.org>: > Their still in svn as far I know. s/Their/They're -- Regards, Benjamin From ronaldoussoren at mac.com Tue Nov 13 16:10:30 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 13 Nov 2012 16:10:30 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> Message-ID: <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> On 13 Nov, 2012, at 16:00, Daniel Holth <dholth at gmail.com> wrote: > > I want to remove distutils from the standard library. Why? Distutils may not be perfect, but is usable for basic packages. It could even be enhanced to support these peps and be even more useable, although patches for that would run into the self-imposed freeze of distutils development. Ronald From dholth at gmail.com Tue Nov 13 17:35:13 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 11:35:13 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: <CAG8k2+6i6WyBxDY5GWaBCHH0507-6jByZ775u5dRDZiMU_s1Cg@mail.gmail.com> Setuptools! You would avoid 75% of pypi. It is nonsense to pretend that setuptools is not a significant packaging innovation. Its main flaw is that it is based on distutils, a non-extensible design. distutils2 is a lot of setuptools and distutils code with the plug-ability taken out. Perhaps I should say that I would like distutils to become as relevant to packaging as the cgi module is to web development. It is not a short-term goal. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/fbacf1bd/attachment.html> From dholth at gmail.com Tue Nov 13 17:37:13 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 11:37:13 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAKmKYaBS+xKO+Ht+jYPRe0HRJHjw3D0iO_SjGhRtdfc5D5YdLA@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <CAKmKYaBS+xKO+Ht+jYPRe0HRJHjw3D0iO_SjGhRtdfc5D5YdLA@mail.gmail.com> Message-ID: <CAG8k2+63xGz-p=3MOt=Gj8jT14_=raeR0S7E+L_Zp5hb_5YuYA@mail.gmail.com> The signatures section is now just: +If JSON web signatures are used, one or more JSON Web Signature JSON +Serialization (JWS-JS) signatures may be stored in a file RECORD.jws +adjacent to RECORD. JWS is used to sign RECORD by including the SHA-256 +hash of RECORD as the JWS payload:: { "hash": "sha256=ADD-r2urObZHcxBW3Cr-vDCu5RJwT4CaRTHiFmbcIYY" } +If RECORD.p7s is used, it must contain a PKCS#7 format signature of +RECORD. + +A wheel installer may assume that the signature has already been checked +against RECORD, and only must verify the hashes in RECORD against the +extracted file contents. FAQ +Why does wheel include attached signatures? + Attached signatures are more convenient than detached signatures + because they travel with the archive. Since only the individual files + are signed, the archive can be recompressed without invalidating + the signature, or individual files can be verified without having + to download the whole archive. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/f91629d1/attachment.html> From a.cavallo at cavallinux.eu Tue Nov 13 17:31:05 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 13 Nov 2012 17:31:05 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <44174.1352824265@cavallinux.eu> +1 Distutils is good enough: it could be better but for what is required (essentially copying files and creating packages installers) is fine. The only corner case is an absolute pain in the neck is in the cross compile scenario. Currently I don't have *any* need for "auto" tools (setuptools and descendants). As an example I always advice to avoid like a plague anything that depends on setuptools... with all the due respect I think is the poor's developer attempt to play the sys admin game. Even virtual env is a poor work around on the python interpreter not being relocatable (as in a portable app fashion). thanks On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren at mac.com wrote: > > On 13 Nov, 2012, at 16:00, Daniel Holth <dholth at gmail > .com> wrote: > > > I want to remove distutils from the standard > library. > Why? Distutils may not be perfect, but is usable for basic packages. It > could even be enhanced to support these peps and be even more useable, > although patches for that would run into the self-imposed freeze of > distutils development. > Ronald > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From a.cavallo at cavallinux.eu Tue Nov 13 17:43:22 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Tue, 13 Nov 2012 17:43:22 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <29996.1352825002@cavallinux.eu> I'll give you that number(?) but ... mercurial, docutils, jinjia2 pygments, sphinx, lxml, nose, cherrypy, django, pyqt ... all they don't need/use setuptools: that 25% left is quite an interesting field to play in. If setuptools was "significant packaging innovation" do you think people wouldn't have embraced already? Allow me to call *that* a nonsense. thanks ps. my experience is on the field, so please give me the credit of many years of experience if I'm say I'm not that keen on "auto" tools: in this kiss rules. On Tue 13/11/12 17:35, "Daniel Holth" dholth at gmail.com wrote: > Setuptools! You would avoid 75% of pypi. It is nonsense to pretend > that setuptools is not a significant packaging innovation. Its main > flaw is that it is based on distutils, a non-extensible design. > distutils2 is a lot of setuptools and distutils code with the > plug-ability taken out. > > Perhaps I should say that I would like distutils to become as > relevant to packaging as the cgi module is to web development. It is > not a short-term goal. > > From fijall at gmail.com Tue Nov 13 17:45:52 2012 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 13 Nov 2012 17:45:52 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: <CAK5idxQvY9WJk3GN6Bzyp=Y+UHK9ataG2R0cK_UZtiuho0QWmw@mail.gmail.com> On Tue, Nov 13, 2012 at 5:31 PM, <a.cavallo at cavallinux.eu> wrote: > > +1 > Distutils is good enough: it could be better but for what is required > (essentially copying files and creating packages installers) is fine. The only > corner case is an absolute pain in the neck is in the cross compile scenario. I think you should qualify all such statements like "distutils is good enough for *me*" and ideally also describe why exactly it's good enough for you. Such blank statement does not bring anything to the discussion (for example I could reply with "distutils is clearly not good enough" and we're both right and we did not get anywhere). For example distutils is absolutely *untestable* which makes it very far from good enough for me. Cheers, fijal From donald.stufft at gmail.com Tue Nov 13 17:49:45 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 13 Nov 2012 11:49:45 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <44174.1352824265@cavallinux.eu> References: <44174.1352824265@cavallinux.eu> Message-ID: <046C6552C46E4A8B907B9D5A1CC97EBD@gmail.com> Distutils is not good enough for the vast majority of people. Even for what it does, it does not do it well. It is a library that is user hostile and buggy. It was a fine first revision of packaging, but the Python community needs something better. On Tuesday, November 13, 2012 at 11:31 AM, a.cavallo at cavallinux.eu wrote: > > +1 > Distutils is good enough: it could be better but for what is required > (essentially copying files and creating packages installers) is fine. The only > corner case is an absolute pain in the neck is in the cross compile scenario. > > Currently I don't have *any* need for "auto" tools (setuptools and descendants). > As an example I always advice to avoid like a plague anything that depends on > setuptools... with all the due respect I think is the poor's developer attempt to > play the sys admin game. > > Even virtual env is a poor work around on the python interpreter not being > relocatable (as in a portable app fashion). > > thanks > > > On Tue 13/11/12 16:10, "Ronald Oussoren" ronaldoussoren at mac.com (mailto:ronaldoussoren at mac.com) wrote: > > > > On 13 Nov, 2012, at 16:00, Daniel Holth <dholth at gmail > > .com> wrote: > > > > > I want to remove distutils from the standard > > library. > > Why? Distutils may not be perfect, but is usable for basic packages. It > > could even be enhanced to support these peps and be even more useable, > > although patches for that would run into the self-imposed freeze of > > distutils development. > > Ronald > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > > http://mail.python.org/mailman/listinfo/distutils-sig > > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org (mailto:Distutils-SIG at python.org) > http://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/cc37ae5e/attachment-0001.html> From martin at v.loewis.de Tue Nov 13 18:07:05 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:07:05 +0100 Subject: [Python-Dev] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A2206A.40804@egenix.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <20121018211014.73d56057@pitrou.net> <CAG8k2+6oALhohqOePU-yJYBA=Hr=FjqzGJJ1g5Ah3pjR+fCuGA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> Message-ID: <50A27E39.8080809@v.loewis.de> Am 13.11.12 11:26, schrieb M.-A. Lemburg: > Note that signing such distribution files can be handled outside > of the wheel format PEP. It just way to complex and out of scope > for the wheel format itself. Also note that PGP/GPG and the other > signing tools work well on any distribution file. There's really no > need to build these into the format itself. And even if the desire is to include the signature in the distribution (as is common for code-signing - you want the signature in the executable, the jar file, etc), then it's still possible to include, say, a PGP signature inside the file, which may well be a signature to the manuscript, preserving the rest of the signing procedure. Regards, Martin From martin at v.loewis.de Tue Nov 13 18:23:35 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:23:35 +0100 Subject: [Python-Dev] [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> Message-ID: <50A28217.5060301@v.loewis.de> > I want to remove distutils from the standard library. If that happens > then we might want a secure way to install it from pypi. One way would > be to include the public key used to sign distutils in Python's own > signature-verifying bootstrap wheel installer, never mind whether it > used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? > https://www.updateframework.com/wiki/SecuringPythonPackageManagement It depends on the threat model - whose definition is key to any security discussion. I'd say that providing the CA certificate of the CA, and to use https for downloading, should be enough. Alternatively, if the threat is that somebody may have hacked PyPI, then hard-code the hash (SHA-3 if you are paranoid) in the Python distribution, and rely on downloading a specific version from PyPI. OTOH, I'm -1 on removing the code from Python in a way that it may come back through downloading. Instead, it is much easier to keep it included. Regards, Martin From martin at v.loewis.de Tue Nov 13 18:32:11 2012 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Nov 2012 18:32:11 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <CAK5idxQvY9WJk3GN6Bzyp=Y+UHK9ataG2R0cK_UZtiuho0QWmw@mail.gmail.com> References: <44174.1352824265@cavallinux.eu> <CAK5idxQvY9WJk3GN6Bzyp=Y+UHK9ataG2R0cK_UZtiuho0QWmw@mail.gmail.com> Message-ID: <50A2841B.5020006@v.loewis.de> Am 13.11.12 17:45, schrieb Maciej Fijalkowski: > For example distutils is absolutely *untestable* which makes it very > far from good enough for me. I never had issues with testing distutils applications. I use "python setup.py sdist", then unpack the resulting source package, install it, and run its test - if that passes, the packaging was successful. Regards, Martin From dholth at gmail.com Tue Nov 13 18:41:22 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 13 Nov 2012 12:41:22 -0500 Subject: [Python-Dev] [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <50A28217.5060301@v.loewis.de> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAPZV6o9nC+7CsOhvE5pyp_u_cKoaKhGzPJXNgFoTDqNkbkJZkg@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <50A28217.5060301@v.loewis.de> Message-ID: <CAG8k2+4zSEPPjymYcOs9WGVEdRLfL66AJew-MQ9rppONzMWi=w@mail.gmail.com> On Tue, Nov 13, 2012 at 12:23 PM, "Martin v. L?wis" <martin at v.loewis.de>wrote: > I want to remove distutils from the standard library. If that happens >> then we might want a secure way to install it from pypi. One way would >> be to include the public key used to sign distutils in Python's own >> signature-verifying bootstrap wheel installer, never mind whether it >> used ECDSA or RSA or Ed25519. Do you have a better idea? TUF? >> https://www.updateframework.**com/wiki/**SecuringPythonPackageManagemen** >> t <https://www.updateframework.com/wiki/SecuringPythonPackageManagement> >> > > It depends on the threat model - whose definition is key to any security > discussion. > > I'd say that providing the CA certificate of the CA, and to use https > for downloading, should be enough. > > Alternatively, if the threat is that somebody may have hacked PyPI, > then hard-code the hash (SHA-3 if you are paranoid) in the Python > distribution, and rely on downloading a specific version from PyPI. > > OTOH, I'm -1 on removing the code from Python in a way that it may > come back through downloading. Instead, it is much easier to keep > it included. > It is a long term goal. It would be more practical to discourage distutils and encourage users to look elsewhere (Bento) for a beautifully designed build system. The short term goal is just to standardize a way to install packages without having a build system, which is what wheel is for apart from the practical goal of simply installing lxml in a reasonable amount of time. I think Metadata 1.2 (PEP 426) is ready to be accepted. The compatibility tags (PEP 425) need some additional text in the discussion section, any contributors for https://bitbucket.org/dholth/pep425tags/ ? Wheel (PEP 427) might mention that version 1.0 of the spec is only concerned with losslessly representing existing (setuptools & distutils) packages without trying to add too many new features apart from a standardized .egg substitute. distutils itself is not testable. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/d49b03e7/attachment.html> From barry at python.org Tue Nov 13 21:43:30 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Nov 2012 15:43:30 -0500 Subject: [Python-Dev] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CAG8k2+4zSEPPjymYcOs9WGVEdRLfL66AJew-MQ9rppONzMWi=w@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <50A28217.5060301@v.loewis.de> <CAG8k2+4zSEPPjymYcOs9WGVEdRLfL66AJew-MQ9rppONzMWi=w@mail.gmail.com> Message-ID: <20121113154330.0b1d0e2c@resist.wooz.org> I'm just beginning to review these PEPs and the threads, with an OS vendor packager's eye. Let me start with one small suggestion for PEP 425. From the FAQ: Q. Who will maintain the registry of abbreviated implementations? A. New two-letter abbreviations can be requested on the python-dev mailing list. As a rule of thumb, abbreviations are reserved for the current 4 most prominent implementations. I think the PEP should explicitly say that it will be the definitive keeper of the abbreviations. The request can be made on python-dev, and once approved, PEP 425 will be updated with the new abbreviation. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20121113/43123fcd/attachment.pgp> From solipsis at pitrou.net Tue Nov 13 21:43:32 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 13 Nov 2012 21:43:32 +0100 Subject: [Python-Dev] cpython (3.2): add gc_collects to weakref tests References: <3Y1L3H4Qv8zRNf@mail.python.org> Message-ID: <20121113214332.575108be@pitrou.net> On Tue, 13 Nov 2012 21:26:51 +0100 (CET) philip.jenvey <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/a580cf4ab940 > changeset: 80418:a580cf4ab940 > branch: 3.2 > parent: 80397:8a28c974f903 > user: Philip Jenvey <pjenvey at underboss.org> > date: Tue Nov 13 12:26:31 2012 -0800 > summary: > add gc_collects to weakref tests > > files: > Lib/test/test_exceptions.py | 3 +++ > 1 files changed, 3 insertions(+), 0 deletions(-) While this is necessary for non-refcounted implementations, it does relax the requirement quite a bit for CPython (absence of reference cycles can be a significant feature). I think perhaps gc_collect() should be called only for non-CPython implementations. Regards Antoine. From ronaldoussoren at mac.com Wed Nov 14 08:23:43 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Nov 2012 08:23:43 +0100 Subject: [Python-Dev] [Catalog-sig] [Distutils] accept the wheel PEPs 425, 426, 427 In-Reply-To: <20121113172114.40b0ca4a@pitrou.net> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> Message-ID: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> On 13 Nov, 2012, at 17:21, Antoine Pitrou <solipsis at pitrou.net> wrote: > Le Tue, 13 Nov 2012 16:10:30 +0100, > Ronald Oussoren <ronaldoussoren at mac.com> a ?crit : >> >> On 13 Nov, 2012, at 16:00, Daniel Holth <dholth at gmail.com> wrote: >>> >>> I want to remove distutils from the standard library. >> >> Why? Distutils may not be perfect, but is usable for basic packages. >> It could even be enhanced to support these peps and be even more >> useable, although patches for that would run into the self-imposed >> freeze of distutils development. > > It wouldn't be totally unreasonable to start a project named > "distutils2" to build the next-generation distutils library. > > Oops :-) Or carefully enhance distutils itself. Rewriting distutils in one go seems to be too ambitious with the limited resources available to do so. Ronald From ronaldoussoren at mac.com Wed Nov 14 08:32:34 2012 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 14 Nov 2012 08:32:34 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 In-Reply-To: <50A2841B.5020006@v.loewis.de> References: <44174.1352824265@cavallinux.eu> <CAK5idxQvY9WJk3GN6Bzyp=Y+UHK9ataG2R0cK_UZtiuho0QWmw@mail.gmail.com> <50A2841B.5020006@v.loewis.de> Message-ID: <EE91BA7A-A77A-41B9-9554-F1E68814EB8F@mac.com> On 13 Nov, 2012, at 18:32, Martin v. L?wis <martin at v.loewis.de> wrote: > Am 13.11.12 17:45, schrieb Maciej Fijalkowski: >> For example distutils is absolutely *untestable* which makes it very >> far from good enough for me. > > I never had issues with testing distutils applications. I use > "python setup.py sdist", then unpack the resulting source package, > install it, and run its test - if that passes, the packaging was > successful. I'm pretty sure he means testing distutils itself. I'm not convinced that distutils is untestable. A major problem with distutils is that its API is barely documented, which effectly makes all of it public API (simular to asyncore). IIRC that's the main reason why distutils is frozen right now: with a lot of changes to distutils people started to complain that this could break there there distutils scripts. The lack of a specification also makes testing harder, as it is unclear what should be tested. Ronald From chris at simplistix.co.uk Wed Nov 14 10:12:05 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 14 Nov 2012 09:12:05 +0000 Subject: [Python-Dev] performance of {} versus dict() Message-ID: <50A36065.9010704@simplistix.co.uk> Hi All, A colleague pointed me at Doug's excellent article here: http://www.doughellmann.com/articles/misc/dict-performance/index.html ...which made me a little sad, I suspect I'm not the only one who finds: a_dict = dict( x = 1, y = 2, z = 3, ... ) ...easier to read than: a_dict = { 'x':1, 'y':2, 'z':3, ... } What can we do to speed up the former case? Here's comparison for different versions of CPython: $ python2.5 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 2.96 2.49 2.47 2.42 2.42 1000000 loops, best of 5: 2.42 usec per loop $ python2.5 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 1.69 1.71 1.68 1.68 1.68 1000000 loops, best of 5: 1.68 usec per loop $ python2.6 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 2.41 2.41 2.42 2.44 2.41 1000000 loops, best of 5: 2.41 usec per loop $ python2.6 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 1.51 1.51 1.52 1.51 1.51 1000000 loops, best of 5: 1.51 usec per loop $ python2.7 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 2.32 2.31 2.31 2.32 2.31 1000000 loops, best of 5: 2.31 usec per loop $ python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 1.49 1.49 1.77 1.76 1.55 1000000 loops, best of 5: 1.49 usec per loop So, not the 6 times headline figure that Doug quotes, but certainly a difference. Can someone with Python 3 handy compare there too? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From ulrich.eckhardt at dominolaser.com Wed Nov 14 10:30:27 2012 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Wed, 14 Nov 2012 10:30:27 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <50A364B3.5070807@dominolaser.com> Am 14.11.2012 10:12, schrieb Chris Withers: > Can someone with Python 3 handy compare there too? C:\Python27\python -m timeit -n 1000000 -r 5 -v "dict(a=1, b=2, c=3, d=4, e=5, f=6, g=7)" raw times: 0.918 0.924 0.922 0.928 0.926 1000000 loops, best of 5: 0.918 usec per loop C:\Python27\python -m timeit -n 1000000 -r 5 -v "{'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7}" raw times: 0.48 0.49 0.482 0.496 0.497 1000000 loops, best of 5: 0.48 usec per loop C:\Python32\python -m timeit -n 1000000 -r 5 -v "dict(a=1, b=2, c=3, d=4, e=5, f=6, g=7)" raw times: 0.898 0.891 0.892 0.899 0.891 1000000 loops, best of 5: 0.891 usec per loop C:\Python32\python -m timeit -n 1000000 -r 5 -v "{'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7}" raw times: 0.444 0.463 0.461 0.464 0.461 1000000 loops, best of 5: 0.444 usec per loop C:\Python32_64\python -m timeit -n 1000000 -r 5 -v "dict(a=1, b=2, c=3, d=4, e=5, f=6, g=7)" raw times: 0.908 0.923 0.927 0.912 0.923 1000000 loops, best of 5: 0.908 usec per loop C:\Python32_64\python -m timeit -n 1000000 -r 5 -v "{'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7}" raw times: 0.484 0.446 0.501 0.45 0.442 1000000 loops, best of 5: 0.442 usec per loop C:\Python33_64\python -m timeit -n 1000000 -r 5 -v "dict(a=1, b=2, c=3, d=4, e=5, f=6, g=7)" raw times: 1.02 1.07 1.03 1.11 1.07 1000000 loops, best of 5: 1.02 usec per loop C:\Python33_64\python -m timeit -n 1000000 -r 5 -v "{'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7}" raw times: 0.444 0.449 0.455 0.452 0.46 1000000 loops, best of 5: 0.444 usec per loop Tested on Win7/64 bit. Python 2.7 is the 32-bit version, 3.2 is installed as 32-bit and 64-bit versions and 3.3 only as 64-bit version. In any case, the difference is even a bit stronger than what you experience and it seems that all versions perform roughly similar. Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstra???e 75a, 22547 Hamburg, Deutschland Gesch???ftsf???hrer: Hans Robert Dapprich, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.dominolaser.com ************************************************************************************** Diese E-Mail einschlie???lich s???mtlicher Anh???nge ist nur f???r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf???nger sein sollten. Die E-Mail ist in diesem Fall zu l???schen und darf weder gelesen, weitergeleitet, ver???ffentlicht oder anderweitig benutzt werden. E-Mails k???nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ???nderungen enthalten. Domino Laser GmbH ist f???r diese Folgen nicht verantwortlich. ************************************************************************************** From chris at simplistix.co.uk Wed Nov 14 11:00:59 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 14 Nov 2012 10:00:59 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CADJO0eLD0PXLFSK37w5xqe9bN-OuiM0kega_Z_yhWdxBJnb2NQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CADJO0eLD0PXLFSK37w5xqe9bN-OuiM0kega_Z_yhWdxBJnb2NQ@mail.gmail.com> Message-ID: <50A36BDB.3010202@simplistix.co.uk> On 14/11/2012 09:58, Merlijn van Deen wrote: > On 14 November 2012 10:12, Chris Withers <chris at simplistix.co.uk> wrote: >> ...which made me a little sad > > Why did it make you sad? dict() takes 0.2?s, {} takes 0.04?s. In other > words: you can run dict() _five million_ times per second, and {} > twenty-five million times per second. That is 'a lot' and 'a lot'. It > also means you are unlikely to notice the difference in real-world > code. Just use the one you feel is clearer in the situation, and don't > worry about micro-optimalization. I'm inclined to agree, but it makes me sad for two reasons: - it's something that people get hung up on, for better or worse. (if it wasn't, Doug wouldn't have written his article) - it can make a difference, for example setting up a dict with many keys at the core of a type loop. Without looking at implementation, they should logically perform the same... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From martin at v.loewis.de Wed Nov 14 11:11:39 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Wed, 14 Nov 2012 11:11:39 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> Zitat von Chris Withers <chris at simplistix.co.uk>: > a_dict = dict( > x = 1, > y = 2, > z = 3, > ... > ) > What can we do to speed up the former case? It should be possible to special-case it. Rather than creating a new dictionary from scratch, one could try to have the new dictionary the same size as the original one, and copy all entries. I also wonder whether the PyArg_ValidateKeywordArguments call is really necessary: if this is not a proper keyword dictionary, dict creation could still proceed in a reasonable way. I don't know how much this would gain, though. You still have to create two dictionary objects. For a better speedup, try def xdict(**kwds): return kwds (possibly written in C for even more speed) Regards, Martin From solipsis at pitrou.net Wed Nov 14 11:48:09 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Nov 2012 11:48:09 +0100 Subject: [Python-Dev] performance of {} versus dict() References: <50A36065.9010704@simplistix.co.uk> <CADJO0eLD0PXLFSK37w5xqe9bN-OuiM0kega_Z_yhWdxBJnb2NQ@mail.gmail.com> <50A36BDB.3010202@simplistix.co.uk> Message-ID: <20121114114809.71b76936@pitrou.net> Le Wed, 14 Nov 2012 10:00:59 +0000, Chris Withers <chris at simplistix.co.uk> a ?crit : > On 14/11/2012 09:58, Merlijn van Deen wrote: > > On 14 November 2012 10:12, Chris Withers <chris at simplistix.co.uk> > > wrote: > >> ...which made me a little sad > > > > Why did it make you sad? dict() takes 0.2?s, {} takes 0.04?s. In > > other words: you can run dict() _five million_ times per second, > > and {} twenty-five million times per second. That is 'a lot' and 'a > > lot'. It also means you are unlikely to notice the difference in > > real-world code. Just use the one you feel is clearer in the > > situation, and don't worry about micro-optimalization. > > I'm inclined to agree, but it makes me sad for two reasons: > > - it's something that people get hung up on, for better or worse. (if > it wasn't, Doug wouldn't have written his article) > > - it can make a difference, for example setting up a dict with many > keys at the core of a type loop. Well, please post examples of *real-world* use cases where it makes a difference. Otherwise, you're asking us to add hacks to the implementation just to make you feel good, which is quite unacceptable. Regards Antoine. From pete.alex.harris at gmail.com Wed Nov 14 12:00:46 2012 From: pete.alex.harris at gmail.com (Peter Harris) Date: Wed, 14 Nov 2012 11:00:46 +0000 Subject: [Python-Dev] Python-Dev Digest, Vol 112, Issue 23 In-Reply-To: <mailman.14373.1352887265.27097.python-dev@python.org> References: <mailman.14373.1352887265.27097.python-dev@python.org> Message-ID: <CANYapPP9QS4fBMbh9vyc5z_k+sD8+p8vdBmXJQpB9-w-aHyRvg@mail.gmail.com> Chris Withers wrote: > On 14/11/2012 09:58, Merlijn van Deen wrote: > > On 14 November 2012 10:12, Chris Withers <chris at simplistix.co.uk> wrote: > >> ...which made me a little sad > > > > Why did it make you sad? dict() takes 0.2?s, {} takes 0.04?s. In other > > words: you can run dict() _five million_ times per second, and {} > > twenty-five million times per second. That is 'a lot' and 'a lot'. It > > also means you are unlikely to notice the difference in real-world > > code. Just use the one you feel is clearer in the situation, and don't > > worry about micro-optimalization. > > I'm inclined to agree, but it makes me sad for two reasons: > > - it's something that people get hung up on, for better or worse. (if it > wasn't, Doug wouldn't have written his article) > > - it can make a difference, for example setting up a dict with many keys > at the core of a type loop. > > Without looking at implementation, they should logically perform the > same... > > Well, without looking at the implementation, you could form any opinion you like about how they should perform. Still you could speculate that dict() will require a builtins name lookup, and that the process of passing keyword arguments might itself involve constructing a dictionary, so must inherently take at least a little longer than compiling a {} literal. Peter -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/e458572c/attachment.html> From steve at pearwood.info Wed Nov 14 12:21:03 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Nov 2012 22:21:03 +1100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36BDB.3010202@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <CADJO0eLD0PXLFSK37w5xqe9bN-OuiM0kega_Z_yhWdxBJnb2NQ@mail.gmail.com> <50A36BDB.3010202@simplistix.co.uk> Message-ID: <50A37E9F.2020407@pearwood.info> On 14/11/12 21:00, Chris Withers wrote: > On 14/11/2012 09:58, Merlijn van Deen wrote: >> On 14 November 2012 10:12, Chris Withers <chris at simplistix.co.uk> wrote: >>> ...which made me a little sad >> >> Why did it make you sad? dict() takes 0.2?s, {} takes 0.04?s. In other >> words: you can run dict() _five million_ times per second, and {} >> twenty-five million times per second. That is 'a lot' and 'a lot'. It >> also means you are unlikely to notice the difference in real-world >> code. Just use the one you feel is clearer in the situation, and don't >> worry about micro-optimalization. > > I'm inclined to agree, but it makes me sad for two reasons: > > - it's something that people get hung up on, for better or worse. (if it > wasn't, Doug wouldn't have written his article) People get hung up on all sorts of things. I would hate to think we would complicate the implementation to pander to pointless micro-optimization. I'm sure that there are many far more important things than this. > - it can make a difference, for example setting up a dict with many keys >at the core of a type loop. Ah yes, the semi-mythical "tight loop". I've never come across one of these tight loops that creates a dict with many keys in a tight loop, and then apparently fails to actually use it for anything useful. For if it did, surely the actual work done with the dict is going to outweigh the setup cost for all but the most trivial applications. I find it hard to get uptight about a small inefficiency in trivial applications that don't do much. Show me a non-toy use-case where creating dicts is an actual bottleneck, and I'll revise my position. > Without looking at implementation, they should logically perform the same... I disagree. Calling dict() has to do a name lookup, and then a function call. That alone is almost 2.5 times as expensive as creating a dict literal on my machine: [steve at ando ~]$ python3.3 -m timeit "d = {}" 10000000 loops, best of 3: 0.17 usec per loop [steve at ando ~]$ python3.3 -m timeit "d = dict()" 1000000 loops, best of 3: 0.416 usec per loop Then you have the function call itself, which engages the argument parsing mechanism, which does more work than dict literal syntax. For example, it checks for duplicate keyword arguments, while dict literals happily accept duplicate keys. It's hardly a surprise that dict() is slower than {}. -- Steven From dholth at gmail.com Wed Nov 14 13:04:57 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Nov 2012 07:04:57 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> Message-ID: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> On Nov 14, 2012, at 2:23 AM, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > > On 13 Nov, 2012, at 17:21, Antoine Pitrou <solipsis at pitrou.net> wrote: > >> Le Tue, 13 Nov 2012 16:10:30 +0100, >> Ronald Oussoren <ronaldoussoren at mac.com> a ?crit : >>> >>> On 13 Nov, 2012, at 16:00, Daniel Holth <dholth at gmail.com> wrote: >>>> >>>> I want to remove distutils from the standard library. >>> >>> Why? Distutils may not be perfect, but is usable for basic packages. >>> It could even be enhanced to support these peps and be even more >>> useable, although patches for that would run into the self-imposed >>> freeze of distutils development. >> >> It wouldn't be totally unreasonable to start a project named >> "distutils2" to build the next-generation distutils library. >> >> Oops :-) > > Or carefully enhance distutils itself. Rewriting distutils in one go seems > to be too ambitious with the limited resources available to do so. That has been tried already (setuptools, distribute, distutils2). Instead, try bento (http://cournape.github.com/Bento/). Hilariously everyone I've showed it to is immediately put off by the indentation based syntax (who would use such a thing?) but the project has a few years of effort behind it and is well designed. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/659c2e4c/attachment.html> From rosuav at gmail.com Wed Nov 14 14:18:38 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Nov 2012 00:18:38 +1100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> On Wed, Nov 14, 2012 at 8:12 PM, Chris Withers <chris at simplistix.co.uk> wrote: > I suspect I'm not the only one who finds: > > a_dict = dict( > x = 1, > y = 2, > z = 3, > ... > ) > > ...easier to read than: > > a_dict = { > 'x':1, > 'y':2, > 'z':3, > ... > } > > What can we do to speed up the former case? Perhaps an alternative question: What can be done to make the latter less unpalatable? I personally prefer dict literal syntax to a dict constructor call, but no doubt there are a number of people who feel as you do. In what way(s) do you find the literal syntax less readable, and can some simple (and backward-compatible) enhancements help that? I've seen criticisms (though I don't recall where) of Python, comparing it to JavaScript/ECMAScript, that complain of the need to quote the keys. IMO this is a worthwhile downside, as it allows you to use variables as the keys, rather than requiring (effectively) literal strings. But it does make a dict literal that much more "noisy" than the constructor. ChrisA From benjamin at python.org Wed Nov 14 14:43:18 2012 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 14 Nov 2012 08:43:18 -0500 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> Message-ID: <CAPZV6o90yUeyixQKnGzKvykd9xZ2M2Z3SY5k8bjgJ4teD27-vA@mail.gmail.com> 2012/11/14 <martin at v.loewis.de>: > > Zitat von Chris Withers <chris at simplistix.co.uk>: > > >> a_dict = dict( >> x = 1, >> y = 2, >> z = 3, >> ... >> ) > > >> What can we do to speed up the former case? > > > It should be possible to special-case it. Rather than creating > a new dictionary from scratch, one could try to have the new dictionary > the same size as the original one, and copy all entries. > > I also wonder whether the PyArg_ValidateKeywordArguments call is really > necessary: if this is not a proper keyword dictionary, dict creation > could still proceed in a reasonable way. In the common case PyArg_ValidateKeywordArguments should be a simple check. -- Regards, Benjamin From stefan_ml at behnel.de Wed Nov 14 15:35:37 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 14 Nov 2012 15:35:37 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> Message-ID: <k80a7n$66f$1@ger.gmane.org> Chris Angelico, 14.11.2012 14:18: > On Wed, Nov 14, 2012 at 8:12 PM, Chris Withers wrote: >> I suspect I'm not the only one who finds: >> >> a_dict = dict( >> x = 1, >> y = 2, >> z = 3, >> ... >> ) >> >> ...easier to read than: >> >> a_dict = { >> 'x':1, >> 'y':2, >> 'z':3, >> ... >> } >> >> What can we do to speed up the former case? > > Perhaps an alternative question: What can be done to make the latter > less unpalatable? I personally prefer dict literal syntax to a dict > constructor call, but no doubt there are a number of people who feel > as you do. In what way(s) do you find the literal syntax less > readable, and can some simple (and backward-compatible) enhancements > help that? > > I've seen criticisms (though I don't recall where) of Python, > comparing it to JavaScript/ECMAScript, that complain of the need to > quote the keys. IMO this is a worthwhile downside, as it allows you to > use variables as the keys, rather than requiring (effectively) literal > strings. But it does make a dict literal that much more "noisy" than > the constructor. If that bothers you in a specific case, I recommend using the constructor instead of a literal. Stefan From p.f.moore at gmail.com Wed Nov 14 15:38:11 2012 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Nov 2012 14:38:11 +0000 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> Message-ID: <CACac1F9js9=R8Wk_+4sX+NEj6GVEUU-HbRQ7VfO5iKJcewtd=g@mail.gmail.com> On 14 November 2012 12:04, Daniel Holth <dholth at gmail.com> wrote: > That has been tried already (setuptools, distribute, distutils2). Instead, > try bento (http://cournape.github.com/Bento/). > > Hilariously everyone I've showed it to is immediately put off by the > indentation based syntax (who would use such a thing?) but the project has > a few years of effort behind it and is well designed. > > Ironically, given the thread, it doesn't support the metadata PEPs, so packages installed with bento aren't visible to pip, etc. Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/927aa213/attachment.html> From dholth at gmail.com Wed Nov 14 16:17:23 2012 From: dholth at gmail.com (Daniel Holth) Date: Wed, 14 Nov 2012 10:17:23 -0500 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs 425, 426, 427 In-Reply-To: <CACac1F9js9=R8Wk_+4sX+NEj6GVEUU-HbRQ7VfO5iKJcewtd=g@mail.gmail.com> References: <CAG8k2+5CjZ48tjGSoykhu_30Ui5jx0eFuQNqEK3hiD_MyEFcvA@mail.gmail.com> <CAG8k2+7krh3R5_C0Emn3mb1sy+072iUKCRS6X0N4VdwkJr27SA@mail.gmail.com> <CAG8k2+5YLwk1RGG2CqDnMQkijvPJJYHCO0SwEoJKVkak2vpTDw@mail.gmail.com> <87k3un5fwh.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+6Ck+zACr6eAOZGAShWWtFeVyqcWy=vTLG-PojvsZ+TZA@mail.gmail.com> <87a9vh694r.fsf@uwakimon.sk.tsukuba.ac.jp> <CAG8k2+4WAqNxd7pZ=h==WHvSq=f8UrizzUWPV70-rAC6bwQ_Cg@mail.gmail.com> <87626457t6.fsf@uwakimon.sk.tsukuba.ac.jp> <CADiSq7fgsz49X4X09L4QJPA+Gft17K1SmAUpwbnjKMj-4Hu34w@mail.gmail.com> <CAG8k2+68O-uca_k_=n9FiyDNo7DSNkg3nECCza_nVt9L4_LT2A@mail.gmail.com> <CAG8k2+6dtWoy5_dX2gVVQwyrt7d3X3pZxz-ZioELGMMoqMkTZA@mail.gmail.com> <CADiSq7e7xxyoiZZNtJeH8Vafw5vG+EWDh3SC7htC916FaCrOhw@mail.gmail.com> <50A21808.8040909@v.loewis.de> <50A2206A.40804@egenix.com> <CAG8k2+5uBpq7FO99QbH4yvt-MLXSAH3uoDfjxTttUHe3CqsdWQ@mail.gmail.com> <55555795-0D06-4358-B4CE-E6E5DBBCA296@mac.com> <20121113172114.40b0ca4a@pitrou.net> <2A6F78E2-65CF-4925-8DE0-09EA570A7239@mac.com> <404A3CEE-7F9B-4611-B12C-E45BD780BDC8@gmail.com> <CACac1F9js9=R8Wk_+4sX+NEj6GVEUU-HbRQ7VfO5iKJcewtd=g@mail.gmail.com> Message-ID: <CAG8k2+6iVW2GSnZkuS-r1f_TMPXAUg-LF+=V47kijG5T97p=WA@mail.gmail.com> Well, you can build eggs with Bento, and I have a patch that allows it to build wheels, in both cases it will produce pip-compatible metadata. The Bento author has his own informed opinions about the way packaging should work which do not necessarily include the packaging PEPs. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/0ea1bae0/attachment.html> From dreamingforward at gmail.com Wed Nov 14 17:12:54 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 10:12:54 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> On Wed, Nov 14, 2012 at 3:12 AM, Chris Withers <chris at simplistix.co.uk> wrote: > Hi All, > > A colleague pointed me at Doug's excellent article here: > ...which made me a little sad, I suspect I'm not the only one who finds: > > a_dict = dict( > x = 1, > y = 2, > z = 3, > ... > ) > > ...easier to read than: > > a_dict = { > 'x':1, > 'y':2, > 'z':3, > ... > } Hey, it makes me a little sad that dict breaks convention by allowing the use of unquoted characters (which everywhere else looks like variable names) just for a silly typing optimization. mark From phd at phdru.name Wed Nov 14 17:20:55 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Nov 2012 20:20:55 +0400 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> Message-ID: <20121114162055.GA18900@iskra.aviel.ru> On Wed, Nov 14, 2012 at 10:12:54AM -0600, Mark Adam <dreamingforward at gmail.com> wrote: > On Wed, Nov 14, 2012 at 3:12 AM, Chris Withers <chris at simplistix.co.uk> wrote: > > Hi All, > > > > A colleague pointed me at Doug's excellent article here: > > ...which made me a little sad, I suspect I'm not the only one who finds: > > > > a_dict = dict( > > x = 1, > > y = 2, > > z = 3, > > ... > > ) > > > > ...easier to read than: > > > > a_dict = { > > 'x':1, > > 'y':2, > > 'z':3, > > ... > > } > > Hey, it makes me a little sad that dict breaks convention by allowing > the use of unquoted characters (which everywhere else looks like > variable names) just for a silly typing optimization. It doesn't. It's a call (function call or or a class instantiation) and it's not dict-specific: function(a=1, b=None)... Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From storchaka at gmail.com Wed Nov 14 17:23:16 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 14 Nov 2012 18:23:16 +0200 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <k80ghr$6ic$1@ger.gmane.org> On 14.11.12 11:12, Chris Withers wrote: > ....which made me a little sad, I suspect I'm not the only one who finds: > > a_dict = dict( > x = 1, > y = 2, > z = 3, > ... > ) > > ....easier to read than: > > a_dict = { > 'x':1, > 'y':2, > 'z':3, > ... > } PEP 8 recommends: a_dict = dict( x=1, y=2, z=3, ... ) and a_dict = { 'x': 1, 'y': 2, 'z': 3, ... } From shibturn at gmail.com Wed Nov 14 17:42:39 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 14 Nov 2012 16:42:39 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <k80ghr$6ic$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <k80ghr$6ic$1@ger.gmane.org> Message-ID: <k80hm2$fs8$1@ger.gmane.org> On 14/11/2012 4:23pm, Serhiy Storchaka wrote: > PEP 8 recommends: > > a_dict = dict( > x=1, > y=2, > z=3, > ... > ) > > and > > a_dict = { > 'x': 1, > 'y': 2, > 'z': 3, > ... > } In which section? I can't see such a recommendation. -- Richard From brian at python.org Wed Nov 14 18:00:47 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Nov 2012 11:00:47 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> Message-ID: <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> On Wed, Nov 14, 2012 at 10:12 AM, Mark Adam <dreamingforward at gmail.com> wrote: > On Wed, Nov 14, 2012 at 3:12 AM, Chris Withers <chris at simplistix.co.uk> wrote: >> Hi All, >> >> A colleague pointed me at Doug's excellent article here: >> ...which made me a little sad, I suspect I'm not the only one who finds: >> >> a_dict = dict( >> x = 1, >> y = 2, >> z = 3, >> ... >> ) >> >> ...easier to read than: >> >> a_dict = { >> 'x':1, >> 'y':2, >> 'z':3, >> ... >> } > > Hey, it makes me a little sad that dict breaks convention by allowing > the use of unquoted characters (which everywhere else looks like > variable names) just for a silly typing optimization. What convention and typing optimization is this? I hope you aren't suggesting it should be dict("x"=1) or dict("x":1)? From python-dev at masklinn.net Wed Nov 14 18:02:43 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 14 Nov 2012 18:02:43 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <k80hm2$fs8$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <k80ghr$6ic$1@ger.gmane.org> <k80hm2$fs8$1@ger.gmane.org> Message-ID: <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> On 2012-11-14, at 17:42 , Richard Oudkerk wrote: > On 14/11/2012 4:23pm, Serhiy Storchaka wrote: >> PEP 8 recommends: >> >> a_dict = dict( >> x=1, >> y=2, >> z=3, >> ... >> ) >> >> and >> >> a_dict = { >> 'x': 1, >> 'y': 2, >> 'z': 3, >> ... >> } > > In which section? I can't see such a recommendation. Whitespace in Expressions and Statements > Other Recommendations 3rd bullet: ? Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value. Yes: def complex(real, imag=0.0): return magic(r=real, i=imag) No: def complex(real, imag = 0.0): return magic(r = real, i = imag) ? From dreamingforward at gmail.com Wed Nov 14 18:08:32 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 11:08:32 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> References: <50A36065.9010704@simplistix.co.uk> <k80ghr$6ic$1@ger.gmane.org> <k80hm2$fs8$1@ger.gmane.org> <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> Message-ID: <CAMjeLr-n2BssxMc6SsyKo2yu_yjbSk8z3_wYhHsuWLnmhKCeSw@mail.gmail.com> On Wed, Nov 14, 2012 at 11:02 AM, Xavier Morel <python-dev at masklinn.net> wrote: > > On 2012-11-14, at 17:42 , Richard Oudkerk wrote: > >> On 14/11/2012 4:23pm, Serhiy Storchaka wrote: >>> PEP 8 recommends: >>> >>> a_dict = dict( >>> x=1, >>> y=2, >>> z=3, >>> ... >>> ) >>> >>> and >>> >>> a_dict = { >>> 'x': 1, >>> 'y': 2, >>> 'z': 3, >>> ... >>> } >> >> In which section? I can't see such a recommendation. > > Whitespace in Expressions and Statements > Other Recommendations > > 3rd bullet: > > ? > Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value. > > Yes: > > def complex(real, imag=0.0): > return magic(r=real, i=imag) > > No: > > def complex(real, imag = 0.0): > return magic(r = real, i = imag) That's not a recommendation to use the **kwargs style. mark From dreamingforward at gmail.com Wed Nov 14 18:10:15 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 11:10:15 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> Message-ID: <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> On Wed, Nov 14, 2012 at 11:00 AM, Brian Curtin <brian at python.org> wrote: > On Wed, Nov 14, 2012 at 10:12 AM, Mark Adam <dreamingforward at gmail.com> wrote: >> On Wed, Nov 14, 2012 at 3:12 AM, Chris Withers <chris at simplistix.co.uk> wrote: >>> Hi All, >>> >>> A colleague pointed me at Doug's excellent article here: >>> ...which made me a little sad, I suspect I'm not the only one who finds: >>> >>> a_dict = dict( >>> x = 1, >>> y = 2, >>> z = 3, >>> ... >>> ) >>> >>> ...easier to read than: >>> >>> a_dict = { >>> 'x':1, >>> 'y':2, >>> 'z':3, >>> ... >>> } >> >> Hey, it makes me a little sad that dict breaks convention by allowing >> the use of unquoted characters (which everywhere else looks like >> variable names) just for a silly typing optimization. > > What convention and typing optimization is this? I hope you aren't > suggesting it should be dict("x"=1) or dict("x":1)? Try the canonical {'x':1}. Only dict allows the special initialization above. Other collections require an iterable. I'm guessing **kwargs initialization was only used because it is so simple to implement, but that's not necessarily a heuristic for good language design. mark From rdmurray at bitdance.com Wed Nov 14 18:27:46 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 14 Nov 2012 12:27:46 -0500 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> Message-ID: <20121114172746.B08492500F1@webabinitio.net> On Wed, 14 Nov 2012 11:10:15 -0600, Mark Adam <dreamingforward at gmail.com> wrote: > On Wed, Nov 14, 2012 at 11:00 AM, Brian Curtin <brian at python.org> wrote: > > On Wed, Nov 14, 2012 at 10:12 AM, Mark Adam <dreamingforward at gmail.com> wrote: > >> On Wed, Nov 14, 2012 at 3:12 AM, Chris Withers <chris at simplistix.co.uk> wrote: > >>> Hi All, > >>> > >>> A colleague pointed me at Doug's excellent article here: > >>> ...which made me a little sad, I suspect I'm not the only one who finds: > >>> > >>> a_dict = dict( > >>> x = 1, > >>> y = 2, > >>> z = 3, > >>> ... > >>> ) > >>> > >>> ...easier to read than: > >>> > >>> a_dict = { > >>> 'x':1, > >>> 'y':2, > >>> 'z':3, > >>> ... > >>> } > >> > >> Hey, it makes me a little sad that dict breaks convention by allowing > >> the use of unquoted characters (which everywhere else looks like > >> variable names) just for a silly typing optimization. > > > > What convention and typing optimization is this? I hope you aren't > > suggesting it should be dict("x"=1) or dict("x":1)? > > Try the canonical {'x':1}. Only dict allows the special > initialization above. Other collections require an iterable. I'm guessing > **kwargs initialization was only used because it is so simple to > implement, but that's not necessarily a heuristic for good language design. Maybe it's not good design, but I'll bet you that if it didn't do that, there would be lots of instances of this scattered around various codebases: def makedict(**kw): return kw --David From dreamingforward at gmail.com Wed Nov 14 18:57:48 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 11:57:48 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <20121114172746.B08492500F1@webabinitio.net> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <20121114172746.B08492500F1@webabinitio.net> Message-ID: <CAMjeLr-eroPBXTsi1wWtKDYZ6Z9wCjwCqL4k7B4K015RuadU4A@mail.gmail.com> On Wed, Nov 14, 2012 at 11:27 AM, R. David Murray <rdmurray at bitdance.com> wrote: > Maybe it's not good design, but I'll bet you that if it didn't do that, > there would be lots of instances of this scattered around various > codebases: > > def makedict(**kw): > return kw Now that's a good solution and probably solves the OP speed problem. mark From shibturn at gmail.com Wed Nov 14 19:01:53 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 14 Nov 2012 18:01:53 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> References: <50A36065.9010704@simplistix.co.uk> <k80ghr$6ic$1@ger.gmane.org> <k80hm2$fs8$1@ger.gmane.org> <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> Message-ID: <k80mam$u1n$1@ger.gmane.org> On 14/11/2012 5:02pm, Xavier Morel wrote: >> In which section? I can't see such a recommendation. > > Whitespace in Expressions and Statements > Other Recommendations > > 3rd bullet: > > ? > Don't use spaces around the = sign when used to indicate a keyword argument or a default parameter value. Oops, I did not even notice that difference. I thought Serhiy was talking about indenting the closing ')' and '}'. -- Richard From python-dev at masklinn.net Wed Nov 14 19:08:44 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 14 Nov 2012 19:08:44 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr-n2BssxMc6SsyKo2yu_yjbSk8z3_wYhHsuWLnmhKCeSw@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <k80ghr$6ic$1@ger.gmane.org> <k80hm2$fs8$1@ger.gmane.org> <2081DD91-D5E9-4C59-B495-C7E163436821@masklinn.net> <CAMjeLr-n2BssxMc6SsyKo2yu_yjbSk8z3_wYhHsuWLnmhKCeSw@mail.gmail.com> Message-ID: <DA00908F-A304-4EB2-A9B0-F0D0C64CF920@masklinn.net> On 2012-11-14, at 18:08 , Mark Adam wrote: > > That's not a recommendation to use the **kwargs style. And nobody said it was. It's a recommendation to not put spaces around the equals sign when using keyword arguments which is the correction Serhiy applied to the original code (along with adding a space after the colon in the literal dict, also a PEP8 recommendation). From catch-all at masklinn.net Wed Nov 14 19:12:25 2012 From: catch-all at masklinn.net (Xavier Morel) Date: Wed, 14 Nov 2012 19:12:25 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> Message-ID: <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> On 2012-11-14, at 18:10 , Mark Adam wrote: > > Try the canonical {'x':1}. Only dict allows the special > initialization above. Other collections require an iterable. Other collections don't have a choice, because it would often be ambiguous. Dicts do not have that issue. > I'm guessing > **kwargs initialization was only used because it is so simple to > implement, but that's not necessarily a heuristic for good language design. In this case it very much is, it permits easily merging two dicts in a single expression or cloning-with-replacement. It also mirrors the signature of dict.update which I think is a Good Thing. From dreamingforward at gmail.com Wed Nov 14 19:54:32 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 12:54:32 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> Message-ID: <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> On Wed, Nov 14, 2012 at 12:12 PM, Xavier Morel <catch-all at masklinn.net> wrote: > On 2012-11-14, at 18:10 , Mark Adam wrote: >> >> Try the canonical {'x':1}. Only dict allows the special >> initialization above. Other collections require an iterable. > > Other collections don't have a choice, because it would often be > ambiguous. Dicts do not have that issue. mkay.... >> I'm guessing >> **kwargs initialization was only used because it is so simple to >> implement, but that's not necessarily a heuristic for good language design. > > In this case it very much is, it permits easily merging two dicts in a > single expression or cloning-with-replacement. It also mirrors the > signature of dict.update which I think is a Good Thing. Merging of two dicts is done with dict.update. How do you do it on initialization? This doesn't make sense. mark From a.cavallo at cavallinux.eu Wed Nov 14 13:15:02 2012 From: a.cavallo at cavallinux.eu (a.cavallo at cavallinux.eu) Date: Wed, 14 Nov 2012 13:15:02 +0100 Subject: [Python-Dev] [Distutils] [Catalog-sig] accept the wheel PEPs425, 426, 427 Message-ID: <9326.1352895302@cavallinux.eu> Mmmm, interesting point and worth a discussion about different roles (developer, system admin, final user etc.) having different needs. I believe distutils is used as tool primarily (setup.py bdist_rpm/msi to create installable objects, setup.py bdist_sdist to manage the source code etc.): this complicates the landscape. Some developer has expectation (wrong IMHO) to replace a whole lot of tools with it (downloaders/installers/package managers/etc.): this is an uphill struggle against already widely deployed systems (dpkg/yum/even windows has something!). I think Tarek did some work in the past and the result is visible in the "The Hitchhiker?s Guide to Packaging" but I've no idea where it went in the end ... only is left is a punch-in-the-eye page (http://www.python.org/doc) :D These days I'm stuck with the old KISS approach and I write a setup.py to create "packages" and a makefile to create doc, run tests etc. I'm fairly happy with that. Thanks On Wed 14/11/12 08:32, "Ronald Oussoren" ronaldoussoren at mac.com wrote: > > I'm not convinced that distutils is untestable. A major problem with > distutils is that its API is barely documented, which effectly makes > all of it public API (simular to asyncore). IIRC that's the main > reason why distutils is frozen right now: with a lot of changes to distutils > people started to complain that this could break there there distutils > scripts. > The lack of a specification also makes testing harder, as it is > unclear what should be tested. > > Ronald > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > http://mail.python.org/mailman/listinfo/distutils-sig > > From catch-all at masklinn.net Wed Nov 14 20:37:02 2012 From: catch-all at masklinn.net (Xavier Morel) Date: Wed, 14 Nov 2012 20:37:02 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> Message-ID: <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> On 2012-11-14, at 19:54 , Mark Adam wrote: > > Merging of two dicts is done with dict.update. No, dict.update merges one dict (or two) into a third one. > How do you do it on > initialization? This doesn't make sense. dict(d1, **d2) From phd at phdru.name Wed Nov 14 14:25:00 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Nov 2012 17:25:00 +0400 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> Message-ID: <20121114132500.GA14633@iskra.aviel.ru> On Thu, Nov 15, 2012 at 12:18:38AM +1100, Chris Angelico <rosuav at gmail.com> wrote: > On Wed, Nov 14, 2012 at 8:12 PM, Chris Withers <chris at simplistix.co.uk> wrote: > > I suspect I'm not the only one who finds: > > > > a_dict = dict( > > x = 1, > > y = 2, > > z = 3, > > ... > > ) > > > > ...easier to read than: > > > > a_dict = { > > 'x':1, > > 'y':2, > > 'z':3, > > ... > > } > > > > What can we do to speed up the former case? > > Perhaps an alternative question: What can be done to make the latter > less unpalatable? I personally prefer dict literal syntax to a dict > constructor call, but no doubt there are a number of people who feel > as you do. In what way(s) do you find the literal syntax less > readable, and can some simple (and backward-compatible) enhancements > help that? > > I've seen criticisms (though I don't recall where) of Python, > comparing it to JavaScript/ECMAScript, that complain of the need to > quote the keys. IMO this is a worthwhile downside, as it allows you to > use variables as the keys, rather than requiring (effectively) literal > strings. But it does make a dict literal that much more "noisy" than > the constructor. On the other had it's more powerful. You can write {'class': 'foo'} but cannot dict(class='bar'). {1: '1'} but not dict(1='1'). Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From dreamingforward at gmail.com Wed Nov 14 21:53:11 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 14:53:11 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> Message-ID: <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> wrote: > On 2012-11-14, at 19:54 , Mark Adam wrote: >> >> Merging of two dicts is done with dict.update. > > No, dict.update merges one dict (or two) into a third one. No. I think you need to read the docs. >> How do you do it on >> initialization? This doesn't make sense. > > dict(d1, **d2) That's not valid syntax is it? mark From solipsis at pitrou.net Wed Nov 14 21:58:30 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Nov 2012 21:58:30 +0100 Subject: [Python-Dev] performance of {} versus dict() References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> Message-ID: <20121114215830.674acb49@pitrou.net> On Wed, 14 Nov 2012 14:53:11 -0600 Mark Adam <dreamingforward at gmail.com> wrote: > On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> wrote: > > On 2012-11-14, at 19:54 , Mark Adam wrote: > >> > >> Merging of two dicts is done with dict.update. > > > > No, dict.update merges one dict (or two) into a third one. > > No. I think you need to read the docs. > > >> How do you do it on > >> initialization? This doesn't make sense. > > > > dict(d1, **d2) > > That's not valid syntax is it? Why don't you try it for yourself: >>> d1 = {1:2} >>> d2 = {3:4} >>> dict(d1, **d2) {1: 2, 3: 4} From python at mrabarnett.plus.com Wed Nov 14 22:20:33 2012 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Nov 2012 21:20:33 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> Message-ID: <50A40B21.2090206@mrabarnett.plus.com> On 2012-11-14 20:53, Mark Adam wrote: > On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> wrote: >> On 2012-11-14, at 19:54 , Mark Adam wrote: >>> >>> Merging of two dicts is done with dict.update. >> >> No, dict.update merges one dict (or two) into a third one. > > No. I think you need to read the docs. > >>> How do you do it on >>> initialization? This doesn't make sense. >> >> dict(d1, **d2) > > That's not valid syntax is it? > No. You can have dict(d1) and dict(**d2), but not dict(d1, **d2). From python at mrabarnett.plus.com Wed Nov 14 22:24:14 2012 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Nov 2012 21:24:14 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40B21.2090206@mrabarnett.plus.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> <50A40B21.2090206@mrabarnett.plus.com> Message-ID: <50A40BFE.1010500@mrabarnett.plus.com> On 2012-11-14 21:20, MRAB wrote: > On 2012-11-14 20:53, Mark Adam wrote: >> On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> wrote: >>> On 2012-11-14, at 19:54 , Mark Adam wrote: >>>> >>>> Merging of two dicts is done with dict.update. >>> >>> No, dict.update merges one dict (or two) into a third one. >> >> No. I think you need to read the docs. >> >>>> How do you do it on >>>> initialization? This doesn't make sense. >>> >>> dict(d1, **d2) >> >> That's not valid syntax is it? >> > No. > > You can have dict(d1) and dict(**d2), but not dict(d1, **d2). > Oops, wrong! :-( (I see now where I went wrong...) From brian at python.org Wed Nov 14 22:24:20 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Nov 2012 15:24:20 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40B21.2090206@mrabarnett.plus.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> <50A40B21.2090206@mrabarnett.plus.com> Message-ID: <CAD+XWwrG4GNAT8FYaF7Tmfojnqmfrj6i=qwmKKCVo6gqqZ_vmQ@mail.gmail.com> On Wed, Nov 14, 2012 at 3:20 PM, MRAB <python at mrabarnett.plus.com> wrote: > On 2012-11-14 20:53, Mark Adam wrote: >> >> On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> >> wrote: >>> >>> On 2012-11-14, at 19:54 , Mark Adam wrote: >>>> >>>> >>>> Merging of two dicts is done with dict.update. >>> >>> >>> No, dict.update merges one dict (or two) into a third one. >> >> >> No. I think you need to read the docs. >> >>>> How do you do it on >>>> initialization? This doesn't make sense. >>> >>> >>> dict(d1, **d2) >> >> >> That's not valid syntax is it? >> > No. > > You can have dict(d1) and dict(**d2), but not dict(d1, **d2). Yes you can. From ethan at stoneleaf.us Wed Nov 14 22:27:15 2012 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 14 Nov 2012 13:27:15 -0800 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40B21.2090206@mrabarnett.plus.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> <50A40B21.2090206@mrabarnett.plus.com> Message-ID: <50A40CB3.7010801@stoneleaf.us> MRAB wrote: > On 2012-11-14 20:53, Mark Adam wrote: >> On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> >> wrote: >>> On 2012-11-14, at 19:54 , Mark Adam wrote: >>>> >>>> Merging of two dicts is done with dict.update. >>> >>> No, dict.update merges one dict (or two) into a third one. >> >> No. I think you need to read the docs. >> >>>> How do you do it on >>>> initialization? This doesn't make sense. >>> >>> dict(d1, **d2) >> >> That's not valid syntax is it? >> > No. > > You can have dict(d1) and dict(**d2), but not dict(d1, **d2). To (mis-)quote Antoine: >--> d1 = {1:2} >--> d2 = {'3':4} >--> dict(d1, **d2) > {1: 2, '3': 4} Apparently it is valid syntax. Just make sure you keys for the ** operator are valid strings. :) ~Ethan~ From bwmaister at gmail.com Wed Nov 14 22:40:37 2012 From: bwmaister at gmail.com (Brandon W Maister) Date: Wed, 14 Nov 2012 16:40:37 -0500 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40CB3.7010801@stoneleaf.us> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> <50A40B21.2090206@mrabarnett.plus.com> <50A40CB3.7010801@stoneleaf.us> Message-ID: <CANNZFEns1i1s96USM8gVePJWFOU5eYkwVAQo-TYh1ug1QG=Yxg@mail.gmail.com> > > To (mis-)quote Antoine: > >--> d1 = {1:2} > >--> d2 = {'3':4} > >--> dict(d1, **d2) > > {1: 2, '3': 4} > > Apparently it is valid syntax. Just make sure you keys for the ** > operator are valid strings. :) > > or not: >>> dict(**{'not a valid identifier': True, 1: True}) {1: True, 'not a valid identifier': True} brandon -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/41b790a0/attachment.html> From brian at python.org Wed Nov 14 22:43:46 2012 From: brian at python.org (Brian Curtin) Date: Wed, 14 Nov 2012 15:43:46 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CANNZFEns1i1s96USM8gVePJWFOU5eYkwVAQo-TYh1ug1QG=Yxg@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> <50A40B21.2090206@mrabarnett.plus.com> <50A40CB3.7010801@stoneleaf.us> <CANNZFEns1i1s96USM8gVePJWFOU5eYkwVAQo-TYh1ug1QG=Yxg@mail.gmail.com> Message-ID: <CAD+XWwo5Dmw7jaKjXBFvivoBbq59tmSgw66oT_krx2jLCC9cQQ@mail.gmail.com> On Wed, Nov 14, 2012 at 3:40 PM, Brandon W Maister <bwmaister at gmail.com> wrote: > >> >> To (mis-)quote Antoine: >> >--> d1 = {1:2} >> >--> d2 = {'3':4} >> >--> dict(d1, **d2) >> > {1: 2, '3': 4} >> >> Apparently it is valid syntax. Just make sure you keys for the ** >> operator are valid strings. :) >> > > or not: > >>>> dict(**{'not a valid identifier': True, 1: True}) > {1: True, 'not a valid identifier': True} Just because the string says it's not valid doesn't mean it's not valid. Anyway, can this thread go to python-ideas or python-list now? From python-dev at masklinn.net Wed Nov 14 23:01:32 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Wed, 14 Nov 2012 23:01:32 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <21EFD6BC-574D-425C-8F1F-54033D098530@masklinn.net> <CAMjeLr8AkBbhO-kgLNib9=73WdE2JFdkij76yPckGBB7NkGiyQ@mail.gmail.com> Message-ID: <B96CBAA5-D55B-49F8-A848-D4F0DE38CA93@masklinn.net> On 2012-11-14, at 21:53 , Mark Adam wrote: > On Wed, Nov 14, 2012 at 1:37 PM, Xavier Morel <catch-all at masklinn.net> wrote: >> On 2012-11-14, at 19:54 , Mark Adam wrote: >>> >>> Merging of two dicts is done with dict.update. >> >> No, dict.update merges one dict (or two) into a third one. > > No. I think you need to read the docs. I know what the docs say. dict.update requires an existing dict and (as mutator methods usually do in Python) doesn't return anything. Thus it merges a dict (or two) into a third one (the subject of the call). >>> How do you do it on >>> initialization? This doesn't make sense. >> >> dict(d1, **d2) > > That's not valid syntax is it? Of course it is, why would it not be? From greg.ewing at canterbury.ac.nz Wed Nov 14 22:40:48 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Nov 2012 10:40:48 +1300 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> Message-ID: <50A40FE0.9050109@canterbury.ac.nz> Chris Angelico wrote: > Perhaps an alternative question: What can be done to make the latter > less unpalatable? * We could introduce a new syntax such as {a = 1, b = 2}. * If the compiler were allowed to recognise builtins, it could turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically. -- Greg From chris at simplistix.co.uk Wed Nov 14 23:37:46 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 14 Nov 2012 22:37:46 +0000 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> Message-ID: <50A41D3A.9040904@simplistix.co.uk> On 14/11/2012 10:11, martin at v.loewis.de wrote: > > Zitat von Chris Withers <chris at simplistix.co.uk>: > >> a_dict = dict( >> x = 1, >> y = 2, >> z = 3, >> ... >> ) > >> What can we do to speed up the former case? > > It should be possible to special-case it. Rather than creating > a new dictionary from scratch, one could try to have the new dictionary > the same size as the original one, and copy all entries. Indeed, Doug, what are your views on this? Also, did you have a real-world example where this speed difference was causing you a problem? > I don't know how much this would gain, though. You still have to > create two dictionary objects. For a better speedup, try > > def xdict(**kwds): > return kwds Hah, good call, this trumps both of the other options: $ python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 1.45 1.45 1.44 1.45 1.45 1000000 loops, best of 5: 1.44 usec per loop $ python2.6 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 2.37 2.36 2.36 2.37 2.37 1000000 loops, best of 5: 2.36 usec per loop$ python2.6 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 0.548 0.533 0.55 0.577 0.539 1000000 loops, best of 5: 0.533 usec per loop For the naive observer (ie: me!), why is that? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Wed Nov 14 23:41:28 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 14 Nov 2012 22:41:28 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40FE0.9050109@canterbury.ac.nz> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> Message-ID: <50A41E18.1050102@simplistix.co.uk> On 14/11/2012 21:40, Greg Ewing wrote: > * If the compiler were allowed to recognise builtins, it could > turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically. That would be my naive suggestion, I am prepared to be shot down in flames ;-) Would be even more awesome if it could end up with the magical performance of "def md(**kw): return kw"... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Wed Nov 14 23:43:52 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Wed, 14 Nov 2012 22:43:52 +0000 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <50A41D3A.9040904@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> Message-ID: <50A41EA8.9040502@simplistix.co.uk> On 14/11/2012 22:37, Chris Withers wrote: > On 14/11/2012 10:11, martin at v.loewis.de wrote: >> def xdict(**kwds): >> return kwds > > Hah, good call, this trumps both of the other options: > > $ python2.7 -m timeit -n 1000000 -r 5 -v > "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" > raw times: 1.45 1.45 1.44 1.45 1.45 > 1000000 loops, best of 5: 1.44 usec per loop > $ python2.6 -m timeit -n 1000000 -r 5 -v > 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 2.37 2.36 2.36 2.37 2.37 > 1000000 loops, best of 5: 2.36 usec per loop$ python2.6 -m timeit -n > 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 0.548 0.533 0.55 0.577 0.539 > 1000000 loops, best of 5: 0.533 usec per loop Before anyone shoots me, yes, wrong python for two of them: $ python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 1.49 1.49 1.5 1.49 1.48 1000000 loops, best of 5: 1.48 usec per loop $ python2.7 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 2.35 2.36 2.41 2.42 2.35 1000000 loops, best of 5: 2.35 usec per loop $ python2.7 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 0.507 0.515 0.516 0.529 0.524 1000000 loops, best of 5: 0.507 usec per loop Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From python at mrabarnett.plus.com Wed Nov 14 23:51:37 2012 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Nov 2012 22:51:37 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A40FE0.9050109@canterbury.ac.nz> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> Message-ID: <50A42079.4060908@mrabarnett.plus.com> On 2012-11-14 21:40, Greg Ewing wrote: > Chris Angelico wrote: >> Perhaps an alternative question: What can be done to make the latter >> less unpalatable? > > * We could introduce a new syntax such as {a = 1, b = 2}. > > * If the compiler were allowed to recognise builtins, it could > turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically. > That would be a transformation of the AST, although it assumes that 'dict' hasn't been rebound. Should there be the option of a warning if a builtin is rebound? Or the option of the transformation plus a warning if the builtin is rebound? From donald.stufft at gmail.com Thu Nov 15 00:00:58 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Wed, 14 Nov 2012 18:00:58 -0500 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A42079.4060908@mrabarnett.plus.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A42079.4060908@mrabarnett.plus.com> Message-ID: <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> $ pypy -m timeit 'dict()' 1000000000 loops, best of 3: 0.000811 usec per loop $ pypy -m timeit '{}' 1000000000 loops, best of 3: 0.000809 usec per loop $ pypy -m timeit 'def md(**kw): return kw; md()' 100000000 loops, best of 3: 0.0182 usec per loop $ pypy -m timeit -s 'def md(**kw): return kw' 'md()' 1000000000 loops, best of 3: 0.00136 usec per loop If the difference between dict() and {} is hurting your code why are you still using CPython. On Wednesday, November 14, 2012 at 5:51 PM, MRAB wrote: > On 2012-11-14 21:40, Greg Ewing wrote: > > Chris Angelico wrote: > > > Perhaps an alternative question: What can be done to make the latter > > > less unpalatable? > > > > > > > > > * We could introduce a new syntax such as {a = 1, b = 2}. > > > > * If the compiler were allowed to recognise builtins, it could > > turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically. > > > > That would be a transformation of the AST, although it assumes that > 'dict' hasn't been rebound. > > Should there be the option of a warning if a builtin is rebound? Or the > option of the transformation plus a warning if the builtin is rebound? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121114/4aed86cd/attachment-0001.html> From python-dev at masklinn.net Thu Nov 15 00:03:45 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Thu, 15 Nov 2012 00:03:45 +0100 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <50A41EA8.9040502@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> Message-ID: <F70A48A3-C13A-4814-BE3C-6BE43CBDAAD9@masklinn.net> On 2012-11-14, at 23:43 , Chris Withers wrote: > On 14/11/2012 22:37, Chris Withers wrote: >> On 14/11/2012 10:11, martin at v.loewis.de wrote: >>> def xdict(**kwds): >>> return kwds >> >> Hah, good call, this trumps both of the other options: >> >> $ python2.7 -m timeit -n 1000000 -r 5 -v >> "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" >> raw times: 1.45 1.45 1.44 1.45 1.45 >> 1000000 loops, best of 5: 1.44 usec per loop >> $ python2.6 -m timeit -n 1000000 -r 5 -v >> 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' >> raw times: 2.37 2.36 2.36 2.37 2.37 >> 1000000 loops, best of 5: 2.36 usec per loop$ python2.6 -m timeit -n >> 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' >> raw times: 0.548 0.533 0.55 0.577 0.539 >> 1000000 loops, best of 5: 0.533 usec per loop > > Before anyone shoots me, yes, wrong python for two of them: > > $ python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" > raw times: 1.49 1.49 1.5 1.49 1.48 > 1000000 loops, best of 5: 1.48 usec per loop > > $ python2.7 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 2.35 2.36 2.41 2.42 2.35 > 1000000 loops, best of 5: 2.35 usec per loop > > $ python2.7 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 0.507 0.515 0.516 0.529 0.524 > 1000000 loops, best of 5: 0.507 usec per loop The last one is kind-of weird, it seems to be greatly advantaged by the local lookup: > python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" raw times: 0.676 0.683 0.682 0.698 0.691 1000000 loops, best of 5: 0.676 usec per loop > python2.7 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 1.64 1.66 1.4 1.44 1.44 1000000 loops, best of 5: 1.4 usec per loop > python2.7 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 0.188 0.203 0.201 0.195 0.202 1000000 loops, best of 5: 0.188 usec per loop > python2.7 -m timeit -n 1000000 -r 5 -v -s 'def md(**kw): return kw' 'md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' raw times: 0.871 0.864 0.863 0.889 0.871 1000000 loops, best of 5: 0.863 usec per loop From steve at pearwood.info Thu Nov 15 00:36:24 2012 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Nov 2012 10:36:24 +1100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> Message-ID: <50A42AF8.1070807@pearwood.info> On 15/11/12 05:54, Mark Adam wrote: > Merging of two dicts is done with dict.update. How do you do it on > initialization? This doesn't make sense. Frequently. my_prefs = dict(default_prefs, setting=True, another_setting=False) Notice that I'm not merging one dict into another, but merging two dicts into a third. (Well, technically, one of the two comes from keyword arguments rather than an actual dict, but the principle is the same.) The Python 1.5 alternative was: my_prefs = {} my_prefs.update(default_prefs) my_prefs['setting'] = True my_prefs['another_setting'] = False Blah, I'm so glad I don't have to write Python 1.5 code any more. Even using copy only saves a line: my_prefs = default_prefs.copy() my_prefs['setting'] = True my_prefs['another_setting'] = False -- Steven From lukas.lueg at gmail.com Thu Nov 15 00:38:38 2012 From: lukas.lueg at gmail.com (Lukas Lueg) Date: Thu, 15 Nov 2012 00:38:38 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A42AF8.1070807@pearwood.info> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> Message-ID: <CAJF-kY=3EUAt3U2PZTzcjWNEHYa4WVVg6Gkzo2kmC_gBYf1cvA@mail.gmail.com> Notice that {'x':1} and dict(x=1) are different beasts: The first one compiles directly to BUILD_MAP. The second one loads a reference to 'dict' from globals() and calls the constructor. The two are not the same. 2012/11/15 Steven D'Aprano <steve at pearwood.info> > On 15/11/12 05:54, Mark Adam wrote: > > Merging of two dicts is done with dict.update. How do you do it on >> initialization? This doesn't make sense. >> > > Frequently. > > my_prefs = dict(default_prefs, setting=True, another_setting=False) > > > Notice that I'm not merging one dict into another, but merging two dicts > into a third. > > (Well, technically, one of the two comes from keyword arguments rather > than an actual dict, but the principle is the same.) > > The Python 1.5 alternative was: > > my_prefs = {} > my_prefs.update(default_prefs) > my_prefs['setting'] = True > my_prefs['another_setting'] = False > > > Blah, I'm so glad I don't have to write Python 1.5 code any more. Even > using copy only saves a line: > > my_prefs = default_prefs.copy() > my_prefs['setting'] = True > my_prefs['another_setting'] = False > > > > > -- > Steven > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev> > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > lukas.lueg%40gmail.com<http://mail.python.org/mailman/options/python-dev/lukas.lueg%40gmail.com> > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121115/8316bda0/attachment.html> From rosuav at gmail.com Thu Nov 15 00:40:20 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Nov 2012 10:40:20 +1100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A42AF8.1070807@pearwood.info> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> Message-ID: <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> On Thu, Nov 15, 2012 at 10:36 AM, Steven D'Aprano <steve at pearwood.info> wrote: > On 15/11/12 05:54, Mark Adam wrote: > >> Merging of two dicts is done with dict.update. How do you do it on >> initialization? This doesn't make sense. > > > Frequently. > > my_prefs = dict(default_prefs, setting=True, another_setting=False) > > > Notice that I'm not merging one dict into another, but merging two dicts > into a third. Side point: Wouldn't it be quite logical to support dict addition? >>> {"a":1}+{"b":2} Traceback (most recent call last): File "<pyshell#59>", line 1, in <module> {"a":1}+{"b":2} TypeError: unsupported operand type(s) for +: 'dict' and 'dict' It would make sense for this to result in {"a":1,"b":2}. ChrisA From tjreedy at udel.edu Thu Nov 15 00:47:50 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Nov 2012 18:47:50 -0500 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A36065.9010704@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> Message-ID: <k81ajk$m1v$1@ger.gmane.org> On 11/14/2012 4:12 AM, Chris Withers wrote: To somewhat paraphrase: ''' I prefer 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' to "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}". I am sad that the former takes +-2 times as long to run (in 2.7). Is the difference about the same in 3.x? What can we do to speed up the former case? ''' My responses, trying not to duplicate others. 1. Visual preference depends on the viewer. I prefer the dict display, perhaps because I am more accustomed to it. 2. The two types of expressions have overlapping but distinct use cases. This differences include that dict can be wrapped or replaced, while displays cannot. 3. a) 3.x has dict comprehensions. How do they stack up? b) If one were really initializing multiple dicts with the same starting items, and one were really concerned about speed, should one calculate the common base dict just once and then copy? Win7 64 with 3.3.0: >>> repeat("dict(a=1, b=2, c=3, d=4, e=5)") [0.6200045004915467, 0.6212762582470646, 0.6114683222573376] >>> repeat("{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}") [0.27170026972208233, 0.2594874604131968, 0.25977058559879584] >>> repeat("d.copy()", "d={'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}") [0.25768296004457625, 0.243041299387869, 0.2421860830290825] >>> repeat("{str(i):i for i in range(10)}") [4.914327732926495, 4.874041570524014, 4.871596119002334] >>> repeat("{'0':0, '1':1, '2':2, '3':3, '4':4, '5':5, '6':6, '7':7, '8':8, '9':9}") [0.5207065648769458, 0.5000415004344632, 0.49980294978922757] >>> repeat("d.copy()", "d={'0':0, '1':1, '2':2, '3':3, '4':4, '5':5, '6':6, '7':7, '8':8, '9':9}") [0.571671864980317, 0.5516194699132484, 0.5514937389677925] Assuming no overlooked errors in the above... a) Dict comprehensions are much slower than calls, which makes the calls look good by comparison. b) Copying is not worthwhile. 4. There are about 3000 issues on the tracker. Nearly all are worth more attention than this ;-). -- Terry Jan Reedy From dreamingforward at gmail.com Thu Nov 15 00:52:17 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 17:52:17 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> Message-ID: <CAMjeLr-J06yb_tQDTu=qt5SkRq+sHLTfZjzMY4dj3e7gVvtiCQ@mail.gmail.com> On Wed, Nov 14, 2012 at 5:40 PM, Chris Angelico <rosuav at gmail.com> wrote: > On Thu, Nov 15, 2012 at 10:36 AM, Steven D'Aprano <steve at pearwood.info> wrote: >> On 15/11/12 05:54, Mark Adam wrote: >> Notice that I'm not merging one dict into another, but merging two dicts >> into a third. > > Side point: Wouldn't it be quite logical to support dict addition? > Yes, but then you'd be in my old argument that dict should inherit from set. From phd at phdru.name Wed Nov 14 14:25:00 2012 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Nov 2012 17:25:00 +0400 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> Message-ID: <20121114132500.GA14633@iskra.aviel.ru> On Thu, Nov 15, 2012 at 12:18:38AM +1100, Chris Angelico <rosuav at gmail.com> wrote: > On Wed, Nov 14, 2012 at 8:12 PM, Chris Withers <chris at simplistix.co.uk> wrote: > > I suspect I'm not the only one who finds: > > > > a_dict = dict( > > x = 1, > > y = 2, > > z = 3, > > ... > > ) > > > > ...easier to read than: > > > > a_dict = { > > 'x':1, > > 'y':2, > > 'z':3, > > ... > > } > > > > What can we do to speed up the former case? > > Perhaps an alternative question: What can be done to make the latter > less unpalatable? I personally prefer dict literal syntax to a dict > constructor call, but no doubt there are a number of people who feel > as you do. In what way(s) do you find the literal syntax less > readable, and can some simple (and backward-compatible) enhancements > help that? > > I've seen criticisms (though I don't recall where) of Python, > comparing it to JavaScript/ECMAScript, that complain of the need to > quote the keys. IMO this is a worthwhile downside, as it allows you to > use variables as the keys, rather than requiring (effectively) literal > strings. But it does make a dict literal that much more "noisy" than > the constructor. On the other had it's more powerful. You can write {'class': 'foo'} but cannot dict(class='bar'). {1: '1'} but not dict(1='1'). Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From stephen at xemacs.org Thu Nov 15 03:28:45 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 15 Nov 2012 11:28:45 +0900 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> Message-ID: <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > >>> {"a":1}+{"b":2} > It would make sense for this to result in {"a":1,"b":2}. The test is not "does this sometimes make sense?" It's "does this ever result in nonsense, and if so, do we care?" Here, addition is usually commutative. Should {'a':1}+{'a':2} be the same as, or different from, {'a':2}+{'a':1}, or should it be an error? From rosuav at gmail.com Thu Nov 15 03:35:46 2012 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Nov 2012 13:35:46 +1100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAPTjJmrzq11TrkTfVvAJoVw_HOCKHC9o56Y4zsjAY-Z=9qRTUQ@mail.gmail.com> On Thu, Nov 15, 2012 at 1:28 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Chris Angelico writes: > > > >>> {"a":1}+{"b":2} > > > It would make sense for this to result in {"a":1,"b":2}. > > The test is not "does this sometimes make sense?" It's "does this > ever result in nonsense, and if so, do we care?" > > Here, addition is usually commutative. Should {'a':1}+{'a':2} be the > same as, or different from, {'a':2}+{'a':1}, or should it be an error? >>> "a"+"b" 'ab' >>> "b"+"a" 'ba' I would say that the two dictionary examples are equally allowed to give different results - that they should be equivalent to (shallow) copy followed by update(), but possibly more efficiently. ChrisA From dreamingforward at gmail.com Thu Nov 15 04:24:07 2012 From: dreamingforward at gmail.com (Mark Adam) Date: Wed, 14 Nov 2012 21:24:07 -0600 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CAMjeLr_fftJbBtiWkezSuge7cEdzVFWQk2C9z2rL2-54OKOh=Q@mail.gmail.com> On Wed, Nov 14, 2012 at 8:28 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > Chris Angelico writes: > > > >>> {"a":1}+{"b":2} > > > It would make sense for this to result in {"a":1,"b":2}. > > The test is not "does this sometimes make sense?" It's "does this > ever result in nonsense, and if so, do we care?" > > Here, addition is usually commutative. Should {'a':1}+{'a':2} be the > same as, or different from, {'a':2}+{'a':1}, or should it be an error? Easy: dict should have a (user substitutable) collision function that is called in these cases. This would allow significant functionality with practically no cost. In addition, it could be implemented in such a way as to offer significant speedups (when using dict.update for example) over any possible hand-written substitutes (since it's only run on key collisions and otherwise uses an underlying loop coded in C). mark From stephen at xemacs.org Thu Nov 15 04:48:22 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 15 Nov 2012 12:48:22 +0900 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmrzq11TrkTfVvAJoVw_HOCKHC9o56Y4zsjAY-Z=9qRTUQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> <CAPTjJmrzq11TrkTfVvAJoVw_HOCKHC9o56Y4zsjAY-Z=9qRTUQ@mail.gmail.com> Message-ID: <87zk2j7chl.fsf@uwakimon.sk.tsukuba.ac.jp> Chris Angelico writes: > On Thu, Nov 15, 2012 at 1:28 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote: > > Chris Angelico writes: > > > > > >>> {"a":1}+{"b":2} > > > > > It would make sense for this to result in {"a":1,"b":2}. > > > > The test is not "does this sometimes make sense?" It's "does this > > ever result in nonsense, and if so, do we care?" > > > > Here, addition is usually commutative. Should {'a':1}+{'a':2} be the > > same as, or different from, {'a':2}+{'a':1}, or should it be an error? > > >>> "a"+"b" > 'ab' > >>> "b"+"a" > 'ba' > > I would say that the two dictionary examples are equally allowed to > give different results - that they should be equivalent to (shallow) > copy followed by update(), but possibly more efficiently. I wouldn't. A string is a sequence of uninterpreted letters, and necessarily ordered. In fact, that's about all you can say about strings in general. I would prefer that concatenation be expressed by juxtaposition, but that's troublesome for machine parsing (especially error recovery). My intuition is elastic enough to admit exceptional cases where the essential ordered nature of the objects being "added" is more important than the customary interpretation of the operator symbol, so interpreting string addition as concatenation doesn't bother me. Furthermore, in string addition both operands affect the result in proportion to their content, though differently. Dictionaries aren't ordered, and their "elements" have structure (key-value pairs). It would definitely bother me if dictionary addition weren't commutative, and it's worse that an operand affects the outcome in an all-or-nothing way. Also, "update" is more appropriately expressed by an extended assignment operator. Defining "+" in terms of "+=" as you propose just doesn't seem right to me. From stephen at xemacs.org Thu Nov 15 05:11:24 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 15 Nov 2012 13:11:24 +0900 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAMjeLr_fftJbBtiWkezSuge7cEdzVFWQk2C9z2rL2-54OKOh=Q@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> <CAMjeLr_fftJbBtiWkezSuge7cEdzVFWQk2C9z2rL2-54OKOh=Q@mail.gmail.com> Message-ID: <87vcd77bf7.fsf@uwakimon.sk.tsukuba.ac.jp> Mark Adam writes: > Easy: dict should have a (user substitutable) collision function that > is called in these cases. "I smell overengineering." > This would allow significant functionality with practically no > cost. We already have that functionality if we want it; just define an appropriate mapping class. I don't need or want it, so I can ignore it, but I suspect to get anywhere with this proposal you're going to need to show that this "significant functionality" needs to be in syntax. From martin at v.loewis.de Thu Nov 15 05:19:31 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 15 Nov 2012 05:19:31 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A41E18.1050102@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A41E18.1050102@simplistix.co.uk> Message-ID: <20121115051931.Horde.liS5Mdjz9kRQpG1TeJazG-A@webmail.df.eu> Zitat von Chris Withers <chris at simplistix.co.uk>: > On 14/11/2012 21:40, Greg Ewing wrote: >> * If the compiler were allowed to recognise builtins, it could >> turn dict(a = 1, b = 2) into {'a':1, 'b':2} automatically. > > That would be my naive suggestion, I am prepared to be shot down in > flames ;-) In general, special-casing builtins in the compiler is not possible in Python. You cannot know statically that 'dict' really refers to the builtin. Something may shadow the name at run-time, making dict refer to some other callable. Regards, Martin From martin at v.loewis.de Thu Nov 15 05:27:23 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 15 Nov 2012 05:27:23 +0100 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <50A41EA8.9040502@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> Message-ID: <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> Zitat von Chris Withers <chris at simplistix.co.uk>: > $ python2.7 -m timeit -n 1000000 -r 5 -v > "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" > raw times: 1.49 1.49 1.5 1.49 1.48 > 1000000 loops, best of 5: 1.48 usec per loop > > $ python2.7 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 2.35 2.36 2.41 2.42 2.35 > 1000000 loops, best of 5: 2.35 usec per loop > > $ python2.7 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; > md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 0.507 0.515 0.516 0.529 0.524 > 1000000 loops, best of 5: 0.507 usec per loop > > For the naive observer (ie: me!), why is that? It's faster than calling dict() because the dict code will create a second dictionary, and discard the keywords dictionary. It's (probably) faster than the dictionary display, because the {} byte code builds the dictionary one-by-one, whereas the keywords dictionary is built in a single step (taking all keys and values from the evaluation stack). Regards, Martin From martin at v.loewis.de Thu Nov 15 06:04:57 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Thu, 15 Nov 2012 06:04:57 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <CAPTjJmrzq11TrkTfVvAJoVw_HOCKHC9o56Y4zsjAY-Z=9qRTUQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAMjeLr8jGwCmYU9mokr-nOM06-RyFJNbuD=z_rdfhYa_Hg8gbA@mail.gmail.com> <CAD+XWwpv8DjVXK6uq_9dUPXMYzzYJtbtX-Y-QUMtqT47u2m4NQ@mail.gmail.com> <CAMjeLr-+cDgvC3fujn99G8VJ1XrnZq4FgGFmPfSEDCkwFOuSxw@mail.gmail.com> <8C81B6EB-05FE-4C0E-BC6C-23400BBEE403@masklinn.net> <CAMjeLr9R4jks6A1vmAnWtuZ6=Gq6hM287bW_X86Yxgp5VGfCXQ@mail.gmail.com> <50A42AF8.1070807@pearwood.info> <CAPTjJmpJ66sndb4aZ52WiZWa2dv6iGg4Ozua26=YqjOQC113hQ@mail.gmail.com> <874nkr8uqq.fsf@uwakimon.sk.tsukuba.ac.jp> <CAPTjJmrzq11TrkTfVvAJoVw_HOCKHC9o56Y4zsjAY-Z=9qRTUQ@mail.gmail.com> Message-ID: <20121115060457.Horde.ugXBbKGZi1VQpHf5nyWTVWA@webmail.df.eu> Zitat von Chris Angelico <rosuav at gmail.com>: > On Thu, Nov 15, 2012 at 1:28 PM, Stephen J. Turnbull > <stephen at xemacs.org> wrote: >> Chris Angelico writes: >> >> > >>> {"a":1}+{"b":2} >> >> > It would make sense for this to result in {"a":1,"b":2}. >> >> The test is not "does this sometimes make sense?" It's "does this >> ever result in nonsense, and if so, do we care?" >> >> Here, addition is usually commutative. Should {'a':1}+{'a':2} be the >> same as, or different from, {'a':2}+{'a':1}, or should it be an error? > >>>> "a"+"b" > 'ab' >>>> "b"+"a" > 'ba' > > I would say that the two dictionary examples are equally allowed to > give different results - that they should be equivalent to (shallow) > copy followed by update(), but possibly more efficiently. Can this be moved to python-ideas, please? Regards, Martin From stefan_ml at behnel.de Thu Nov 15 07:32:41 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 15 Nov 2012 07:32:41 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A42079.4060908@mrabarnett.plus.com> <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> Message-ID: <k822a7$mmd$1@ger.gmane.org> Donald Stufft, 15.11.2012 00:00: > $ pypy -m timeit 'dict()' > 1000000000 loops, best of 3: 0.000811 usec per loop > > $ pypy -m timeit '{}' > 1000000000 loops, best of 3: 0.000809 usec per loop > > $ pypy -m timeit 'def md(**kw): return kw; md()' > 100000000 loops, best of 3: 0.0182 usec per loop > > $ pypy -m timeit -s 'def md(**kw): return kw' 'md()' > 1000000000 loops, best of 3: 0.00136 usec per loop Yep, I really like the fact that optimisers can fold stupid benchmarks into no-ops. I wonder why it fails so badly in the latter two cases, though. You should bring that to the attention of the PyPy developers, they might want to fix it. Stefan From chris at simplistix.co.uk Thu Nov 15 08:14:42 2012 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 15 Nov 2012 07:14:42 +0000 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <k822a7$mmd$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A42079.4060908@mrabarnett.plus.com> <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> <k822a7$mmd$1@ger.gmane.org> Message-ID: <50A49662.6030308@simplistix.co.uk> On 15/11/2012 06:32, Stefan Behnel wrote: > Donald Stufft, 15.11.2012 00:00: >> $ pypy -m timeit 'dict()' >> 1000000000 loops, best of 3: 0.000811 usec per loop >> >> $ pypy -m timeit '{}' >> 1000000000 loops, best of 3: 0.000809 usec per loop >> >> $ pypy -m timeit 'def md(**kw): return kw; md()' >> 100000000 loops, best of 3: 0.0182 usec per loop >> >> $ pypy -m timeit -s 'def md(**kw): return kw' 'md()' >> 1000000000 loops, best of 3: 0.00136 usec per loop > > Yep, I really like the fact that optimisers can fold stupid benchmarks into > no-ops. I wonder why it fails so badly in the latter two cases, though. You > should bring that to the attention of the PyPy developers, they might want > to fix it. Agreed, but Donald, please try with a bunch of keys rather than an empty dict... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From stefan_ml at behnel.de Thu Nov 15 08:22:57 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 15 Nov 2012 08:22:57 +0100 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <50A49662.6030308@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A42079.4060908@mrabarnett.plus.com> <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> <k822a7$mmd$1@ger.gmane.org> <50A49662.6030308@simplistix.co.uk> Message-ID: <k8258f$fbv$1@ger.gmane.org> Chris Withers, 15.11.2012 08:14: > On 15/11/2012 06:32, Stefan Behnel wrote: >> Donald Stufft, 15.11.2012 00:00: >>> $ pypy -m timeit 'dict()' >>> 1000000000 loops, best of 3: 0.000811 usec per loop >>> >>> $ pypy -m timeit '{}' >>> 1000000000 loops, best of 3: 0.000809 usec per loop >>> >>> $ pypy -m timeit 'def md(**kw): return kw; md()' >>> 100000000 loops, best of 3: 0.0182 usec per loop >>> >>> $ pypy -m timeit -s 'def md(**kw): return kw' 'md()' >>> 1000000000 loops, best of 3: 0.00136 usec per loop >> >> Yep, I really like the fact that optimisers can fold stupid benchmarks into >> no-ops. I wonder why it fails so badly in the latter two cases, though. You >> should bring that to the attention of the PyPy developers, they might want >> to fix it. > > Agreed, but Donald, please try with a bunch of keys rather than an empty > dict... Right. If that makes a difference, it's another bug. Stefan From storchaka at gmail.com Thu Nov 15 08:24:21 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 15 Nov 2012 09:24:21 +0200 Subject: [Python-Dev] performance of {} versus dict() In-Reply-To: <k81ajk$m1v$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <k81ajk$m1v$1@ger.gmane.org> Message-ID: <k825bl$gou$1@ger.gmane.org> On 15.11.12 01:47, Terry Reedy wrote: > 4. There are about 3000 issues on the tracker. Nearly all are worth more > attention than this ;-). This is the best conclusion of this thread. From lrekucki at gmail.com Thu Nov 15 10:22:50 2012 From: lrekucki at gmail.com (=?UTF-8?Q?=C5=81ukasz_Rekucki?=) Date: Thu, 15 Nov 2012 10:22:50 +0100 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <CAEZs-EKsvkhLVj=sryNdxOQTKkPHvgZGARTt6Aps-nvNc2i_sQ@mail.gmail.com> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <F70A48A3-C13A-4814-BE3C-6BE43CBDAAD9@masklinn.net> <CAEZs-EKsvkhLVj=sryNdxOQTKkPHvgZGARTt6Aps-nvNc2i_sQ@mail.gmail.com> Message-ID: <CAEZs-E+AMsB3ySFo4yJjrsz-feBTTkmZTFEnNQ8RHs8FV8zEzQ@mail.gmail.com> Hi, I posted this (by accident) off the list: > On 2012-11-14, at 23:43 , Chris Withers wrote: > >> On 14/11/2012 22:37, Chris Withers wrote: >>> On 14/11/2012 10:11, martin at v.loewis.de wrote: >>>> def xdict(**kwds): >>>> return kwds >>> >>> Hah, good call, this trumps both of the other options: >>> >>> 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' >>> raw times: 0.548 0.533 0.55 0.577 0.539 >>> 1000000 loops, best of 5: 0.533 usec per loop No, this just doesn't execute the right code: >>> def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7) ... >>> import dis >>> dis.dis(md) 1 0 LOAD_FAST 0 (kw) 3 RETURN_VALUE 4 LOAD_GLOBAL 0 (md) 7 LOAD_CONST 1 ('a') 10 LOAD_CONST 2 (1) 13 LOAD_CONST 3 ('b') 16 LOAD_CONST 4 (2) 19 LOAD_CONST 5 ('c') 22 LOAD_CONST 6 (3) 25 LOAD_CONST 7 ('d') 28 LOAD_CONST 8 (4) 31 LOAD_CONST 9 ('e') 34 LOAD_CONST 10 (5) 37 LOAD_CONST 11 ('f') 40 LOAD_CONST 12 (6) 43 LOAD_CONST 13 ('g') 46 LOAD_CONST 14 (7) 49 CALL_FUNCTION 1792 52 POP_TOP Also: Python 3.2.3 (default, Apr 11 2012, 07:12:16) [MSC v.1500 64 bit (AMD64)] on win 32 Type "help", "copyright", "credits" or "license" for more information. >>> dict({1: "foo"}, **{frozenset([2]): "bar"}) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: keyword arguments must be strings While: Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> dict({1: "foo"}, **{2: "bar"}) {1: 'foo', 2: 'bar'} >>> dict({1: "foo"}, **{frozenset([2]): "bar"}) {1: 'foo', frozenset([2]): 'bar'} If you're worrying about global lookup, you should stop (in this case): $ py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(): return dict()" "xdict()" raw times: 0.477 0.47 0.468 0.473 0.469 1000000 loops, best of 5: 0.468 usec per loop $ py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=dict): return dict()" "xdict()" raw times: 0.451 0.45 0.451 0.45 0.449 1000000 loops, best of 5: 0.449 usec per loop $ py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=lambda **kw: kw): return dict()" "xdict()" raw times: 0.433 0.434 0.435 0.435 0.431 1000000 loops, best of 5: 0.431 usec per loop $ py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=dict): return {}" "xdict()" raw times: 0.276 0.279 0.279 0.277 0.275 1000000 loops, best of 5: 0.275 usec per loop And using non-empty dicts doesn't change much and the first one is roughly the sum of the latter two (as expected): C:\Users\lrekucki>py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=dict): return dict(a=1, b=2, c=3, d=4, e=5, f=6)" "xdict()" raw times: 1.72 1.71 1.71 1.71 1.71 1000000 loops, best of 5: 1.71 usec per loop C:\Users\lrekucki>py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=lambda **kw: kw): return dict(a=1, b=2, c=3, d=4, e=5, f=6)" "xdict()" raw times: 1.01 1.01 1.01 1.01 1.01 1000000 loops, best of 5: 1.01 usec per loop C:\Users\lrekucki>py -3.3 -m timeit -n 1000000 -r 5 -v -s "def xdict(dict=dict): return {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6}" "xdict()" raw times: 0.744 0.736 0.735 0.733 0.733 1000000 loops, best of 5: 0.733 usec per loop I hope that this helps move python-dev's focus to some more useful discussion. -- ?ukasz Rekucki From kristjan at ccpgames.com Thu Nov 15 10:17:39 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 15 Nov 2012 09:17:39 +0000 Subject: [Python-Dev] Setting project home path the best way In-Reply-To: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. (I always thought that this "special" execution mode, hardwired in, was a bit odd, and suggested that this could be made a function of pep405) Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: site.py sitecustomize.py sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. K > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian Tismer > Sent: 11. n?vember 2012 20:31 > To: python-dev at python.org > Subject: [Python-Dev] Setting project home path the best way > > Hi friends, > > I have a project that has its root somewhere on my machine. > This project has many folders and contains quite some modules. > > There is a common root of the module tree, and I want to use > - either absolute imports > - relative imports with '.' > > Problem: > > - I want to run any module inside the heirarchy from the command-line > > - this should work, regardless what my 'cwd' is > > - this should work with or without virtualenv. > > So far, things work fine with virtualenv, because sys.executable is in the > project module tree. > > Without virtualenv, this is not so. But I hate to make settings like > PYTHONPATH, because these are not permanent. . > > Question: > > How should I define my project root dir in a unique way, without setting an > environment variable? > What is the lest intrusive way to spell that? > > Reason: > > I'd like to make things work correctly and unambigously when I call a script > inside the module heirarchy. Things are not fixed: there exist many > checkouts In the file system, and each should know where to search its > home/root in the tree. > > Is this elegantly possible to deduce from the actually executed script file? > > Cheers - chris > > Sent from my Ei4Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com From kristjan at ccpgames.com Thu Nov 15 10:20:09 2012 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Thu, 15 Nov 2012 09:20:09 +0000 Subject: [Python-Dev] externals? In-Reply-To: <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> Okay, that means I need to re-install svn, cool. But I should mention that this needs to be mentioned in the core development FAQs, setting up and so forth. There is no mention of it there. Perhaps the unix makefiles get the proper version, but a windows developer has to fetch those externals manually. Also, is there any reason to keep this in svn? Why not check this in to HG, we need not worry about history, etc. K > -----Original Message----- > From: Benjamin Peterson [mailto:benjamin at python.org] > Sent: 13. n?vember 2012 15:04 > To: Kristj?n Valur J?nsson > Cc: Python-Dev (python-dev at python.org) > Subject: Re: [Python-Dev] externals? > > Their still in svn as far I know. > > 2012/11/13 Kristj?n Valur J?nsson <kristjan at ccpgames.com>: > > This may be a silly question, but haven?t the python externals been > > moved to HG yet? > > > > I usually work on cpython without bothering with the externals, but I > > found today that I needed them. On Windows this is a bit of a bother. > > And I?ve thrown away all my SVN stuff... > > > > K > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python- > dev/benjamin%40python.or > > g > > > > > > -- > Regards, > Benjamin From greg.ewing at canterbury.ac.nz Thu Nov 15 11:48:27 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Nov 2012 23:48:27 +1300 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> Message-ID: <50A4C87B.7010401@canterbury.ac.nz> martin at v.loewis.de wrote: > It's faster than calling dict() because the dict code will > create a second dictionary, and discard the keywords dictionary. Perhaps in the case where dict() is called with keyword args only, it could just return the passed-in keyword dictionary instead of creating another one? -- Greg From sdl.web at gmail.com Thu Nov 15 12:16:39 2012 From: sdl.web at gmail.com (Leo) Date: Thu, 15 Nov 2012 19:16:39 +0800 Subject: [Python-Dev] [BUG] Trailing spaces in pretty-printed JSON References: <m2k3uuri19.fsf@gmail.com> <33E0EAD5-8BEE-4574-ABEC-A342BDADA34D@masklinn.net> Message-ID: <m24nkrjeug.fsf@gmail.com> On 2012-10-14 02:06 +0800, Xavier Morel wrote: > 1. Why didn't you report that on the tracker? Reported: http://bugs.python.org/issue16476 From roundup-admin at psf.upfronthosting.co.za Thu Nov 15 15:30:56 2012 From: roundup-admin at psf.upfronthosting.co.za (Python tracker) Date: Thu, 15 Nov 2012 14:30:56 +0000 Subject: [Python-Dev] Failed issue tracker submission Message-ID: <20121115143056.404191CCF3@psf.upfronthosting.co.za> An unexpected error occurred during the processing of your message. The tracker administrator is being notified. -------------- next part -------------- Return-Path: <python-dev at python.org> X-Original-To: report at bugs.python.org Delivered-To: roundup+tracker at psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 8F7E41C98F for <report at bugs.python.org>; Thu, 15 Nov 2012 15:30:55 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3Y2Q3g2PtxzRjv for <report at bugs.python.org>; Thu, 15 Nov 2012 15:30:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1352989855; bh=X+inBqLoRVfh7Gtof4lWQ3GKcOPuRIb4f6aeDmPB+Z0=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: Subject:Message-Id:Date; b=bj7svLDU57Chaz18mDOtCNtAs4y0V8RO1R/xoKH8KOELcUKhqVHFdJPfiOKurVwc3 ZvP/i9O0oKNrXvACnjFnTi9Z9x8sW7kWvMxNKpWo4wQPwrXPn1RDhtRoIFZ9l98sbt P8zJS8xwXS592hm1od1ZadbKgEaSw2BERQ6c1zsI= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 15 Nov 2012 15:30:55 +0100 Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) by mail.python.org (Postfix) with ESMTP for <report at bugs.python.org>; Thu, 15 Nov 2012 15:30:54 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 From: python-dev at python.org To: report at bugs.python.org Subject: [issue16144] Message-Id: <3Y2Q3g2PtxzRjv at mail.python.org> Date: Thu, 15 Nov 2012 15:30:55 +0100 (CET) TmV3IGNoYW5nZXNldCBhOGNhMTQ5ODNhYjEgYnkgQW5kcmV3IFN2ZXRsb3YgaW4gYnJhbmNoICcz LjMnOgpJc3N1ZSAjMTYxNDQ6IEZpeCBtaXNsZWFkaW5nIHNlbnRlbmNlIGluIHJlZmVyZW5jZS9p bXBvcnQuCmh0dHA6Ly9oZy5weXRob24ub3JnL2NweXRob24vcmV2L2E4Y2ExNDk4M2FiMQoKCk5l dyBjaGFuZ2VzZXQgOTk2MWEwZGFmY2M3IGJ5IEFuZHJldyBTdmV0bG92IGluIGJyYW5jaCAnZGVm YXVsdCc6Ck1lcmdlIGlzc3VlICMxNjE0NDogRml4IG1pc2xlYWRpbmcgc2VudGVuY2UgaW4gcmVm ZXJlbmNlL2ltcG9ydC4KaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvOTk2MWEwZGFm Y2M3Cg== From stefan_ml at behnel.de Thu Nov 15 15:58:46 2012 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 15 Nov 2012 15:58:46 +0100 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <50A4C87B.7010401@canterbury.ac.nz> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> <50A4C87B.7010401@canterbury.ac.nz> Message-ID: <k82vv5$naj$1@ger.gmane.org> Greg Ewing, 15.11.2012 11:48: > martin at v.loewis.de wrote: >> It's faster than calling dict() because the dict code will >> create a second dictionary, and discard the keywords dictionary. > > Perhaps in the case where dict() is called with keyword > args only, it could just return the passed-in keyword > dictionary instead of creating another one? This should work as long as this still creates a copy of d at some point: d = {...} dict(**d) Stefan From alex.gaynor at gmail.com Thu Nov 15 17:18:42 2012 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 15 Nov 2012 16:18:42 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?performance_of_=7B=7D_versus_dict=28=29?= References: <50A36065.9010704@simplistix.co.uk> <CAPTjJmqcvN14+DM743jF3Ut+sq4YJL=OKpXONNEnhzyYjr+2gQ@mail.gmail.com> <50A40FE0.9050109@canterbury.ac.nz> <50A42079.4060908@mrabarnett.plus.com> <37EDB7B5B89649AC9E0AA19C89A8C87E@gmail.com> <k822a7$mmd$1@ger.gmane.org> <50A49662.6030308@simplistix.co.uk> <k8258f$fbv$1@ger.gmane.org> Message-ID: <loom.20121115T171734-858@post.gmane.org> Stefan Behnel <stefan_ml <at> behnel.de> writes: > Right. If that makes a difference, it's another bug. > > Stefan > > It's fixed, with, I will note, fewer lines of code than many messages in this thread: https://bitbucket.org/pypy/pypy/changeset/c30cb1dcb7a9adc32548fd14274e4995 Alex From tjreedy at udel.edu Thu Nov 15 17:21:53 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 15 Nov 2012 11:21:53 -0500 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <k82vv5$naj$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> <50A4C87B.7010401@canterbury.ac.nz> <k82vv5$naj$1@ger.gmane.org> Message-ID: <k834r0$6v8$1@ger.gmane.org> On 11/15/2012 9:58 AM, Stefan Behnel wrote: > Greg Ewing, 15.11.2012 11:48: >> martin at v.loewis.de wrote: >>> It's faster than calling dict() because the dict code will >>> create a second dictionary, and discard the keywords dictionary. >> >> Perhaps in the case where dict() is called with keyword >> args only, it could just return the passed-in keyword >> dictionary instead of creating another one? > > This should work as long as this still creates a copy of d at some point: > > d = {...} > dict(**d) I was thinking that CPython could check the ref count of the input keyword dict to determine whether it is newly created and can be returned or is pre-existing and must be copied. But it seems not so. >>> def d(**x): return sys.getrefcount(x) >>> import sys >>> d(a = 3) 2 >>> d(**{'a': 3}) 2 >>> b = {'a': 3} >>> d(**b) 2 I was expecting 3 for the last one. -- Terry Jan Reedy From shibturn at gmail.com Thu Nov 15 17:45:42 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 15 Nov 2012 16:45:42 +0000 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <k834r0$6v8$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> <50A4C87B.7010401@canterbury.ac.nz> <k82vv5$naj$1@ger.gmane.org> <k834r0$6v8$1@ger.gmane.org> Message-ID: <k8367r$itn$1@ger.gmane.org> On 15/11/2012 4:21pm, Terry Reedy wrote: > I was thinking that CPython could check the ref count of the input > keyword dict to determine whether it is newly created and can be > returned or is pre-existing and must be copied. But it seems not so. > > >>> def d(**x): return sys.getrefcount(x) > > >>> import sys > >>> d(a = 3) > 2 > >>> d(**{'a': 3}) > 2 > >>> b = {'a': 3} > >>> d(**b) > 2 > > I was expecting 3 for the last one. Isn't it always newly created? >>> def f(**x): return x ... >>> b = {'a':3} >>> b is f(**b) False -- Richard From roundup-admin at psf.upfronthosting.co.za Thu Nov 15 19:24:19 2012 From: roundup-admin at psf.upfronthosting.co.za (Python tracker) Date: Thu, 15 Nov 2012 18:24:19 +0000 Subject: [Python-Dev] Failed issue tracker submission Message-ID: <20121115182419.950971CE11@psf.upfronthosting.co.za> An unexpected error occurred during the processing of your message. The tracker administrator is being notified. -------------- next part -------------- Return-Path: <python-dev at python.org> X-Original-To: report at bugs.python.org Delivered-To: roundup+tracker at psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [IPv6:2001:888:2000:d::a6]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 5B78D1CC04 for <report at bugs.python.org>; Thu, 15 Nov 2012 19:24:19 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3Y2WDz1BQtzRhX for <report at bugs.python.org>; Thu, 15 Nov 2012 19:24:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1353003859; bh=GDDVQWyvA249FMc1twsoY4VfCXH15D/J8oGHrHuD8Og=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: Subject:Message-Id:Date; b=sMId1I1AgUhjGpLVNulgs6JADkCmjcEvAyX34oy66KWe8KvbSTNxig9wD53MNGnso vUA4DbpD2v2oRy/Udvmgn2Tqap/lZpo/RPxjT+eivdsvTWOpbFUE8401pNmG/f316p 4wis2s3IIGJv1jpw9e4fuBwbCi5iSDyB3HYYaKjI= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 15 Nov 2012 19:24:19 +0100 Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) by mail.python.org (Postfix) with ESMTP for <report at bugs.python.org>; Thu, 15 Nov 2012 19:24:18 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 From: python-dev at python.org To: report at bugs.python.org Subject: [issue16481] Message-Id: <3Y2WDz1BQtzRhX at mail.python.org> Date: Thu, 15 Nov 2012 19:24:19 +0100 (CET) TmV3IGNoYW5nZXNldCBjNTc0Y2U3OGNkNjEgYnkgUmljaGFyZCBPdWRrZXJrIGluIGJyYW5jaCAn My4zJzoKSXNzdWUgIzE2NDgxOiBtdWx0aXByb2Nlc3Npbmcgbm8gbG9uZ2VyIGxlYWtzIHByb2Nl c3MgaGFuZGxlcyBvbiBXaW5kb3dzLgpodHRwOi8vaGcucHl0aG9uLm9yZy9jcHl0aG9uL3Jldi9j NTc0Y2U3OGNkNjEKCgpOZXcgY2hhbmdlc2V0IGNiNjEyYzVmMzBjYiBieSBSaWNoYXJkIE91ZGtl cmsgaW4gYnJhbmNoICdkZWZhdWx0JzoKSXNzdWUgIzE2NDgxOiBNZXJnZQpodHRwOi8vaGcucHl0 aG9uLm9yZy9jcHl0aG9uL3Jldi9jYjYxMmM1ZjMwY2IK From roundup-admin at psf.upfronthosting.co.za Thu Nov 15 21:58:51 2012 From: roundup-admin at psf.upfronthosting.co.za (Python tracker) Date: Thu, 15 Nov 2012 20:58:51 +0000 Subject: [Python-Dev] Failed issue tracker submission Message-ID: <20121115205851.ADB351CDAD@psf.upfronthosting.co.za> An unexpected error occurred during the processing of your message. The tracker administrator is being notified. -------------- next part -------------- Return-Path: <python-dev at python.org> X-Original-To: report at bugs.python.org Delivered-To: roundup+tracker at psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [82.94.164.166]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id 728881C884 for <report at bugs.python.org>; Thu, 15 Nov 2012 21:58:51 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3Y2ZgH1WglzRlk for <report at bugs.python.org>; Thu, 15 Nov 2012 21:58:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1353013131; bh=OUB+ieZrhXLf00fS3ScDXJOs2fZdncpLHhVH8Mq4aG8=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: Subject:Message-Id:Date; b=bxgo73yXi3M9v/34ihpVbL175HI0avNqRJ9ZZPR8RJ/Ebf+s6CLSY6KSFvEqsey6o k9VOuwgxEoJ6EPYG9b80rF0TfIkaPS/ly2PkD0oDeL2MONpJ/T+mCqEYegdK16I1mO QP53t81gJuW1rACwgBViWQfoGh0NTebNRHzjE2zw= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 15 Nov 2012 21:58:51 +0100 Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) by mail.python.org (Postfix) with ESMTP for <report at bugs.python.org>; Thu, 15 Nov 2012 21:58:50 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 From: python-dev at python.org To: report at bugs.python.org Subject: [issue16469] Message-Id: <3Y2ZgH1WglzRlk at mail.python.org> Date: Thu, 15 Nov 2012 21:58:51 +0100 (CET) TmV3IGNoYW5nZXNldCBhMmI1NGI2ZDk3NTkgYnkgTWFyayBEaWNraW5zb24gaW4gYnJhbmNoICdk ZWZhdWx0JzoKSXNzdWUgIzE2NDY5OiBGcmFjdGlvbihmbG9hdCgnbmFuJykpIGFuZCBGcmFjdGlv bihmbG9hdCgnaW5mJykpIG5vdyByYWlzZSBWYWx1ZUVycm9yIGFuZCBPdmVyZmxvd0Vycm9yIChy ZXNwLiksIG5vdCBUeXBlRXJyb3IuCmh0dHA6Ly9oZy5weXRob24ub3JnL2NweXRob24vcmV2L2Ey YjU0YjZkOTc1OQo= From tismer at stackless.com Thu Nov 15 22:43:31 2012 From: tismer at stackless.com (Christian Tismer) Date: Thu, 15 Nov 2012 22:43:31 +0100 Subject: [Python-Dev] Setting project home path the best way In-Reply-To: <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <2D5A4C65-85AD-463C-9C6B-263C4C36EFAA@stackless.com> Hi Kristjan, does that mean that your scheme simply works, without any config step necessary after I did my checkout? This would in fact be an interesting alternative to Python setup.py develop but I'm not sure if this is the same scheme on windows and Os X. Getting this part right was rather tricky, and I fear this is still an issue. Right now I think to just force my users to run the install step, since it is quite accepted in general. Still, I'd love to see a way with no action needed at all: write yout your structure, and it works as-is. Seems to be impossible without tricks. Cheers - chris Sent from my Ei4Steve On Nov 15, 2012, at 10:17, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. > (I always thought that this "special" execution mode, hardwired in, was a bit odd, and suggested that this could be made a function of pep405) > Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: > site.py > sitecustomize.py > > sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. > site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. > > K > > >> -----Original Message----- >> From: Python-Dev [mailto:python-dev- >> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian Tismer >> Sent: 11. n?vember 2012 20:31 >> To: python-dev at python.org >> Subject: [Python-Dev] Setting project home path the best way >> >> Hi friends, >> >> I have a project that has its root somewhere on my machine. >> This project has many folders and contains quite some modules. >> >> There is a common root of the module tree, and I want to use >> - either absolute imports >> - relative imports with '.' >> >> Problem: >> >> - I want to run any module inside the heirarchy from the command-line >> >> - this should work, regardless what my 'cwd' is >> >> - this should work with or without virtualenv. >> >> So far, things work fine with virtualenv, because sys.executable is in the >> project module tree. >> >> Without virtualenv, this is not so. But I hate to make settings like >> PYTHONPATH, because these are not permanent. . >> >> Question: >> >> How should I define my project root dir in a unique way, without setting an >> environment variable? >> What is the lest intrusive way to spell that? >> >> Reason: >> >> I'd like to make things work correctly and unambigously when I call a script >> inside the module heirarchy. Things are not fixed: there exist many >> checkouts In the file system, and each should know where to search its >> home/root in the tree. >> >> Is this elegantly possible to deduce from the actually executed script file? >> >> Cheers - chris >> >> Sent from my Ei4Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python- >> dev/kristjan%40ccpgames.com > > <winmail.dat> From greg.ewing at canterbury.ac.nz Thu Nov 15 23:39:44 2012 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Nov 2012 11:39:44 +1300 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <k82vv5$naj$1@ger.gmane.org> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> <50A41EA8.9040502@simplistix.co.uk> <20121115052723.Horde.smTDANjz9kRQpG8rXpwzJKA@webmail.df.eu> <50A4C87B.7010401@canterbury.ac.nz> <k82vv5$naj$1@ger.gmane.org> Message-ID: <50A56F30.5000608@canterbury.ac.nz> Stefan Behnel wrote: > This should work as long as this still creates a copy of d at some point: > > d = {...} > dict(**d) It will -- the implementation of the function call opcode always creates a new keyword dict for passing to the called function. -- Greg From tismer at stackless.com Fri Nov 16 00:10:09 2012 From: tismer at stackless.com (Christian Tismer) Date: Fri, 16 Nov 2012 00:10:09 +0100 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> Hi guys, I am bored of installing things. Bored of things that happen to not work for some minor reasons. Reasons that are not immediately obvious. Things that don't work because some special case was not handled. Things that compile for half an hour and then complain that something is not as expected. May it be a compiler, a library, a command like pip or easy-install, a system like macports or homebrew, virtualenv, whatsoever. These things are all great if they work. When they do not work, the user is in some real trouble. And he reads hundreds Of blogs and sites and emails, which all answer a bit of slightly related questions, but all in all - This is not how Python should work !! I am really bored and exhausted and annoyed by those packages which Pretend to make my life eadier, but they don't really. Something is really missing. I want something that is easy to use in all cases, also if it fails. Son't get me wrong, I like things like pip or virtualenv or homebrew. I just think they have to be rewritten completely. They have the wrong assumption that things work! The opposite should be the truth: by default, things go wrong. Correctness is very fragile. I am thinking of a better concept that is harder to break. I thin to design a setup tool that is much more checking itself and does not trust in any assumption. Scenario: After hours and hours, I find how to modify setup.py to function almost correctly for PySide. This was ridiculously hard to do! Settings for certain directories, included and stuff are not checked when they could be, but after compiling a lot of things! After a lot of tries and headaches, I find out that virtualenv barfs on a simple link like ./.Python, the executable, when switching from stock Python to a different (homebrew) version!! This was obviously never tested well, so it frustrates me quite a lot. I could fill a huge list full of complaints like that if I had time. But I don't. Instead, I think installation scripts are generally still wrong by concept today and need to be written in a different way. I would like to start a task force and some sprints about improving this situation. My goal is some unbreakable system of building blocks that are self-contained with no dependencies, that have a defined interface to talk to, and that know themselves completely by introspection. They should not work because they happen to work around all known defects, but by design and control. Whoever is interested to work with me on this is hereby honestly welcomed! Cheers - chris Sent from my Ei4Steve On Nov 15, 2012, at 10:17, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. > (I always thought that this "special" execution mode, hardwired in, was a bit odd, and suggested that this could be made a function of pep405) > Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: > site.py > sitecustomize.py > > sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. > site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. > > K > > >> -----Original Message----- >> From: Python-Dev [mailto:python-dev- >> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian Tismer >> Sent: 11. n?vember 2012 20:31 >> To: python-dev at python.org >> Subject: [Python-Dev] Setting project home path the best way >> >> Hi friends, >> >> I have a project that has its root somewhere on my machine. >> This project has many folders and contains quite some modules. >> >> There is a common root of the module tree, and I want to use >> - either absolute imports >> - relative imports with '.' >> >> Problem: >> >> - I want to run any module inside the heirarchy from the command-line >> >> - this should work, regardless what my 'cwd' is >> >> - this should work with or without virtualenv. >> >> So far, things work fine with virtualenv, because sys.executable is in the >> project module tree. >> >> Without virtualenv, this is not so. But I hate to make settings like >> PYTHONPATH, because these are not permanent. . >> >> Question: >> >> How should I define my project root dir in a unique way, without setting an >> environment variable? >> What is the lest intrusive way to spell that? >> >> Reason: >> >> I'd like to make things work correctly and unambigously when I call a script >> inside the module heirarchy. Things are not fixed: there exist many >> checkouts In the file system, and each should know where to search its >> home/root in the tree. >> >> Is this elegantly possible to deduce from the actually executed script file? >> >> Cheers - chris >> >> Sent from my Ei4Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python- >> dev/kristjan%40ccpgames.com > > <winmail.dat> From doug.hellmann at gmail.com Fri Nov 16 00:17:19 2012 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Thu, 15 Nov 2012 18:17:19 -0500 Subject: [Python-Dev] performance of {} versus dict(), de fmd(**kw): return kw trumps all ; -) In-Reply-To: <50A41D3A.9040904@simplistix.co.uk> References: <50A36065.9010704@simplistix.co.uk> <20121114111139.Horde.PXapOElCcOxQo25burhGAmA@webmail.df.eu> <50A41D3A.9040904@simplistix.co.uk> Message-ID: <386E45A9-3B0A-43CE-9197-23F89FA70B98@gmail.com> On Nov 14, 2012, at 5:37 PM, Chris Withers wrote: > On 14/11/2012 10:11, martin at v.loewis.de wrote: >> >> Zitat von Chris Withers <chris at simplistix.co.uk>: >> >>> a_dict = dict( >>> x = 1, >>> y = 2, >>> z = 3, >>> ... >>> ) >> >>> What can we do to speed up the former case? >> >> It should be possible to special-case it. Rather than creating >> a new dictionary from scratch, one could try to have the new dictionary >> the same size as the original one, and copy all entries. > > Indeed, Doug, what are your views on this? Also, did you have a real-world example where this speed difference was causing you a problem? No, not particularly. I noticed people using dict() and wondered what impact it might have in a general case. > >> I don't know how much this would gain, though. You still have to >> create two dictionary objects. For a better speedup, try >> >> def xdict(**kwds): >> return kwds > > Hah, good call, this trumps both of the other options: > > $ python2.7 -m timeit -n 1000000 -r 5 -v "{'a':1,'b':2,'c':3,'d':4,'e':5,'f':6,'g':7}" > raw times: 1.45 1.45 1.44 1.45 1.45 > 1000000 loops, best of 5: 1.44 usec per loop > $ python2.6 -m timeit -n 1000000 -r 5 -v 'dict(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 2.37 2.36 2.36 2.37 2.37 > 1000000 loops, best of 5: 2.36 usec per loop$ python2.6 -m timeit -n 1000000 -r 5 -v 'def md(**kw): return kw; md(a=1,b=2,c=3,d=4,e=5,f=6,g=7)' > raw times: 0.548 0.533 0.55 0.577 0.539 > 1000000 loops, best of 5: 0.533 usec per loop > > For the naive observer (ie: me!), why is that? > > Chris > > -- > Simplistix - Content Management, Batch Processing & Python Consulting > - http://www.simplistix.co.uk From roundup-admin at psf.upfronthosting.co.za Fri Nov 16 03:39:50 2012 From: roundup-admin at psf.upfronthosting.co.za (Python tracker) Date: Fri, 16 Nov 2012 02:39:50 +0000 Subject: [Python-Dev] Failed issue tracker submission Message-ID: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> An unexpected error occurred during the processing of your message. The tracker administrator is being notified. -------------- next part -------------- Return-Path: <python-dev at python.org> X-Original-To: report at bugs.python.org Delivered-To: roundup+tracker at psf.upfronthosting.co.za Received: from mail.python.org (mail.python.org [IPv6:2001:888:2000:d::a6]) by psf.upfronthosting.co.za (Postfix) with ESMTPS id DFD211C98E for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) Received: from albatross.python.org (localhost [127.0.0.1]) by mail.python.org (Postfix) with ESMTP id 3Y2kDj4Tk8zRb6 for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; t=1353033589; bh=H41Haza/UPvj9B16qvXQtqjlb22jRmazur238MqAPOE=; h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: Subject:Message-Id:Date; b=yz1YIEtIYuM3H9V1Ok0YJ/SDvU8xtO+cFOApgDH8jPRdxRS6z/CSdZ96bSY1cNFM7 4xxKY3LiUXipe439DPwEjxZr0HcsfX+2JdR4Pzf3OZmdopV6PEl8/twaIFoTHQMy8u dTwpZ7G4OJKCc2IeCq4e5Bl/TEvQvVT9NO8eoa3k= Received: from localhost (HELO mail.python.org) (127.0.0.1) by albatross.python.org with SMTP; 16 Nov 2012 03:39:49 +0100 Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) by mail.python.org (Postfix) with ESMTP for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 From: python-dev at python.org To: report at bugs.python.org Subject: [issue15894] Message-Id: <3Y2kDj4Tk8zRb6 at mail.python.org> Date: Fri, 16 Nov 2012 03:39:49 +0100 (CET) TmV3IGNoYW5nZXNldCBiZDg1MzMxMWZmZTAgYnkgQnJldHQgQ2Fubm9uIGluIGJyYW5jaCAnZGVm YXVsdCc6Cklzc3VlICMxNTg5NDogRG9jdW1lbnQgd2h5IHdlIGRvbid0IHdvcnJ5IGFib3V0IHJl LWFjcXVpcmluZyB0aGUKaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvYmQ4NTMzMTFm ZmUwCg== From dholth at gmail.com Fri Nov 16 04:22:27 2012 From: dholth at gmail.com (Daniel Holth) Date: Thu, 15 Nov 2012 22:22:27 -0500 Subject: [Python-Dev] Setting project home path the best way In-Reply-To: <2D5A4C65-85AD-463C-9C6B-263C4C36EFAA@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2D5A4C65-85AD-463C-9C6B-263C4C36EFAA@stackless.com> Message-ID: <72466DBD-CFF8-4FAD-8878-46984675A705@gmail.com> Are you familiar with executing directories having __main__.py as python scripts? Daniel Holth On Nov 15, 2012, at 4:43 PM, Christian Tismer <tismer at stackless.com> wrote: > Hi Kristjan, > > does that mean that your scheme simply works, without any config step > necessary after I did my checkout? > This would in fact be an interesting alternative to > > Python setup.py develop > > but I'm not sure if this is the same scheme on windows and Os X. > > Getting this part right was rather tricky, and I fear this is still an issue. > > Right now I think to just force my users to run the install step, since it is quite > accepted in general. > > Still, I'd love to see a way with no action needed at all: write yout your structure, > and it works as-is. Seems to be impossible without tricks. > > Cheers - chris > > Sent from my Ei4Steve > > On Nov 15, 2012, at 10:17, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > >> When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. >> (I always thought that this "special" execution mode, hardwired in, was a bit odd, and suggested that this could be made a function of pep405) >> Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: >> site.py >> sitecustomize.py >> >> sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. >> site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. >> >> K >> >> >>> -----Original Message----- >>> From: Python-Dev [mailto:python-dev- >>> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian Tismer >>> Sent: 11. n?vember 2012 20:31 >>> To: python-dev at python.org >>> Subject: [Python-Dev] Setting project home path the best way >>> >>> Hi friends, >>> >>> I have a project that has its root somewhere on my machine. >>> This project has many folders and contains quite some modules. >>> >>> There is a common root of the module tree, and I want to use >>> - either absolute imports >>> - relative imports with '.' >>> >>> Problem: >>> >>> - I want to run any module inside the heirarchy from the command-line >>> >>> - this should work, regardless what my 'cwd' is >>> >>> - this should work with or without virtualenv. >>> >>> So far, things work fine with virtualenv, because sys.executable is in the >>> project module tree. >>> >>> Without virtualenv, this is not so. But I hate to make settings like >>> PYTHONPATH, because these are not permanent. . >>> >>> Question: >>> >>> How should I define my project root dir in a unique way, without setting an >>> environment variable? >>> What is the lest intrusive way to spell that? >>> >>> Reason: >>> >>> I'd like to make things work correctly and unambigously when I call a script >>> inside the module heirarchy. Things are not fixed: there exist many >>> checkouts In the file system, and each should know where to search its >>> home/root in the tree. >>> >>> Is this elegantly possible to deduce from the actually executed script file? >>> >>> Cheers - chris >>> >>> Sent from my Ei4Steve >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> http://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: http://mail.python.org/mailman/options/python- >>> dev/kristjan%40ccpgames.com >> >> <winmail.dat> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com From phd at phdru.name Fri Nov 16 06:01:35 2012 From: phd at phdru.name (Oleg Broytman) Date: Fri, 16 Nov 2012 09:01:35 +0400 Subject: [Python-Dev] Generally bored by installation In-Reply-To: <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> Message-ID: <20121116050135.GA7247@iskra.aviel.ru> Hello, Christian! On Fri, Nov 16, 2012 at 12:10:09AM +0100, Christian Tismer <tismer at stackless.com> wrote: > I am bored of installing things. > Bored of things that happen to not work for some minor reasons. > Reasons that are not immediately obvious. > Things that don't work because some special case was not handled. [a hot rant skipped] Christian, I feel your pain. Everyone does, I am sure. Who among us never cursed that dratted computers? Those dirty rotten things that always hang due to bugs in drivers or whatnot, require constant cleaning, mending, upgrading, repairing, debugging. Condemn that magical mouse gestures, be cursed those magical incantations. Who in his normal mind would ever think of writing things like exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o -type f -exec grep -I "$@" '{}' \+ 2>/dev/null ???!!! Unfortunately, there is no escape. Remember that classical book "The Mythical Man-Month"? Quoting Brooks, "There is no silver bullet". Things are becoming better but will never be absolutely perfect. > I would like to start a task force and some sprints about improving this > situation. > My goal is some unbreakable system of building blocks that are self-contained with no dependencies, that have a defined interface to talk to, and that know themselves completely by introspection. > > They should not work because they happen to work around all known defects, but by design and control. > > Whoever is interested to work with me on this is hereby honestly welcomed! I am sorry to disappoint you but I don't believe this could work. Many single developers and teams -- small teams and huge companies, commercial and non-commercial -- have tried the approach and failed. Sometimes things work fine, some small projects look perfect... in isolation... But every now and then they failed, especially when combined. Try to connect a perfect project that measures distances in inches with another perfect project that that measures distances in centimeters -- and poof! your perfect satellite failed and dropped off of the sky. This will always happen. There is always pain for both users and developers. And still people are not afraid to create bigger and more complex projects -- and often they succeed. The only sane approach in my humble opinion is evolution. Similar to biological evolution. And biological evolution means huge birth rate, adapting and huge death rate. So write a lot of code, adapt it to your user's need and suffer a lot of pain -- code death and all that. Try to lessen user's pain, but never expect a piece of software to become perfect once, for all and forever. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tismer at stackless.com Fri Nov 16 12:32:46 2012 From: tismer at stackless.com (Christian Tismer) Date: Fri, 16 Nov 2012 12:32:46 +0100 Subject: [Python-Dev] Setting project home path the best way In-Reply-To: <72466DBD-CFF8-4FAD-8878-46984675A705@gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2D5A4C65-85AD-463C-9C6B-263C4C36EFAA@stackless.com> <72466DBD-CFF8-4FAD-8878-46984675A705@gmail.com> Message-ID: <50A6245E.2030205@stackless.com> Hi Daniel, no, I was not aware of this. I just read it up on http://sayspy.blogspot.de/2010/03/various-ways-of-distributing-python.html Yeah, thank you very much for this hint, very useful! ;-) cheers - Chris On 16.11.12 04:22, Daniel Holth wrote: > Are you familiar with executing directories having __main__.py as python scripts? > > Daniel Holth > > On Nov 15, 2012, at 4:43 PM, Christian Tismer <tismer at stackless.com> wrote: > >> Hi Kristjan, >> >> does that mean that your scheme simply works, without any config step >> necessary after I did my checkout? >> This would in fact be an interesting alternative to >> >> Python setup.py develop >> >> but I'm not sure if this is the same scheme on windows and Os X. >> >> Getting this part right was rather tricky, and I fear this is still an issue. >> >> Right now I think to just force my users to run the install step, since it is quite >> accepted in general. >> >> Still, I'd love to see a way with no action needed at all: write yout your structure, >> and it works as-is. Seems to be impossible without tricks. >> >> Cheers - chris >> >> Sent from my Ei4Steve >> >> On Nov 15, 2012, at 10:17, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: >> >>> When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. >>> (I always thought that this "special" execution mode, hardwired in, was a bit odd, and suggested that this could be made a function of pep405) >>> Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: >>> site.py >>> sitecustomize.py >>> >>> sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. >>> site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. >>> >>> K >>> >>> >>>> -----Original Message----- >>>> From: Python-Dev [mailto:python-dev- >>>> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian Tismer >>>> Sent: 11. n?vember 2012 20:31 >>>> To: python-dev at python.org >>>> Subject: [Python-Dev] Setting project home path the best way >>>> >>>> Hi friends, >>>> >>>> I have a project that has its root somewhere on my machine. >>>> This project has many folders and contains quite some modules. >>>> >>>> There is a common root of the module tree, and I want to use >>>> - either absolute imports >>>> - relative imports with '.' >>>> >>>> Problem: >>>> >>>> - I want to run any module inside the heirarchy from the command-line >>>> >>>> - this should work, regardless what my 'cwd' is >>>> >>>> - this should work with or without virtualenv. >>>> >>>> So far, things work fine with virtualenv, because sys.executable is in the >>>> project module tree. >>>> >>>> Without virtualenv, this is not so. But I hate to make settings like >>>> PYTHONPATH, because these are not permanent. . >>>> >>>> Question: >>>> >>>> How should I define my project root dir in a unique way, without setting an >>>> environment variable? >>>> What is the lest intrusive way to spell that? >>>> >>>> Reason: >>>> >>>> I'd like to make things work correctly and unambigously when I call a script >>>> inside the module heirarchy. Things are not fixed: there exist many >>>> checkouts In the file system, and each should know where to search its >>>> home/root in the tree. >>>> >>>> Is this elegantly possible to deduce from the actually executed script file? >>>> >>>> Cheers - chris >>>> >>>> Sent from my Ei4Steve >>>> _______________________________________________ >>>> Python-Dev mailing list >>>> Python-Dev at python.org >>>> http://mail.python.org/mailman/listinfo/python-dev >>>> Unsubscribe: http://mail.python.org/mailman/options/python- >>>> dev/kristjan%40ccpgames.com >>> <winmail.dat> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com -- Christian Tismer :^) <mailto:tismer at stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From trent at snakebite.org Fri Nov 16 13:12:52 2012 From: trent at snakebite.org (Trent Nelson) Date: Fri, 16 Nov 2012 07:12:52 -0500 Subject: [Python-Dev] externals? In-Reply-To: <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <20121116121251.GA13953@snakebite.org> On Thu, Nov 15, 2012 at 01:20:09AM -0800, Kristj?n Valur J?nsson wrote: > Perhaps the unix makefiles get the proper version, but a windows developer has to fetch those externals manually. Pro-tip: if you're developing on Windows, you're mad if you don't prime your dev env with Tools\buildbot\externals.bat. It takes care of *everything*. I wish every proprietary UNIX system we support had something similar. > Also, is there any reason to keep this in svn? I think it's more a case of there being no tangible benefit (and numerous drawbacks) to switching it to hg. I personally have no need for a local hg repo with 30 different Tcl/Tk histories in it. Subversion's good for this sort of use case. The externals repo gets committed to maybe, what, 3-4 times a year? > Why not check this in to HG, we need not worry about history, etc. Who are these mystical people worried about history? ;-) Trent. From eliben at gmail.com Fri Nov 16 14:34:37 2012 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 16 Nov 2012 05:34:37 -0800 Subject: [Python-Dev] Generally bored by installation In-Reply-To: <20121116050135.GA7247@iskra.aviel.ru> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> Message-ID: <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> > Christian, I feel your pain. Everyone does, I am sure. Who among us > never cursed that dratted computers? Those dirty rotten things that > always hang due to bugs in drivers or whatnot, require constant > cleaning, mending, upgrading, repairing, debugging. Condemn that magical > mouse gestures, be cursed those magical incantations. Who in his normal > mind would ever think of writing things like > > exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name > .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o > -type f -exec grep -I "$@" '{}' \+ 2>/dev/null > > ???!!! > > Use pss ;-) [http://pypi.python.org/pypi/pss] Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121116/3bc7dbf5/attachment.html> From phd at phdru.name Fri Nov 16 15:25:45 2012 From: phd at phdru.name (Oleg Broytman) Date: Fri, 16 Nov 2012 18:25:45 +0400 Subject: [Python-Dev] pss (was: Generally bored by installation) In-Reply-To: <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> Message-ID: <20121116142545.GA19390@iskra.aviel.ru> On Fri, Nov 16, 2012 at 05:34:37AM -0800, Eli Bendersky <eliben at gmail.com> wrote: > > exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name > > .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o > > -type f -exec grep -I "$@" '{}' \+ 2>/dev/null > > > Use pss ;-) [http://pypi.python.org/pypi/pss] Hardly. I am proficient in find+grep. And pss is underdocumented. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From status at bugs.python.org Fri Nov 16 18:07:26 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 16 Nov 2012 18:07:26 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20121116170726.825591CC04@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-11-09 - 2012-11-16) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3801 ( -5) closed 24464 (+46) total 28265 (+41) Open issues with patches: 1660 Issues opened (27) ================== #13863: import.c sometimes generates incorrect timestamps on Windows + http://bugs.python.org/issue13863 reopened by eric.snow #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 opened by HansM #16451: Remove duplication between slice_indices and compute_slice_ind http://bugs.python.org/issue16451 opened by mark.dickinson #16455: sys.getfilesystemencoding() is not the locale encoding on Free http://bugs.python.org/issue16455 opened by haypo #16458: subprocess.py throw "The handle is invalid" error on duplicati http://bugs.python.org/issue16458 opened by kartlee05 #16461: wave module: wrong integer format http://bugs.python.org/issue16461 opened by ckern #16462: smtpd should return greeting http://bugs.python.org/issue16462 opened by mike.a #16463: test_timeout failure on the RHEL buildbot http://bugs.python.org/issue16463 opened by pitrou #16464: urllib.request: opener not resetting content-length http://bugs.python.org/issue16464 opened by terry.reedy #16465: dict creation performance regression http://bugs.python.org/issue16465 opened by phihag #16466: register command forgets password if no config file is created http://bugs.python.org/issue16466 opened by techtonik #16470: Backport set and dictionary comprehensions in tutorial to 2.7 http://bugs.python.org/issue16470 opened by fossilet #16471: upgrade to sphinx 1.1 http://bugs.python.org/issue16471 opened by chris.jerdonek #16472: Distutils+mingw links agains msvcr90, while python27.dll is li http://bugs.python.org/issue16472 opened by eudoxos #16473: Minor difference in decoding quoted-printable text http://bugs.python.org/issue16473 opened by aleperalta #16474: More code coverage for imp module http://bugs.python.org/issue16474 opened by tenuki #16475: Support object instancing and recursion in marshal http://bugs.python.org/issue16475 opened by kristjan.jonsson #16477: tarfile fails to close file handles in case of exception http://bugs.python.org/issue16477 opened by ssam #16480: pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bi http://bugs.python.org/issue16480 opened by Marco.Amadori #16482: pdb.set_trace() clobbering traceback on error http://bugs.python.org/issue16482 opened by akaptur #16483: Make int(float('inf')) raise ValueError rather than OverflowEr http://bugs.python.org/issue16483 opened by mark.dickinson #16484: Missing/broken documentation redirect for http://docs.python.o http://bugs.python.org/issue16484 opened by mgedmin #16485: FD leaks in aifc module http://bugs.python.org/issue16485 opened by serhiy.storchaka #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 opened by serhiy.storchaka #16487: Allow ssl certificates to be speficfied from memory rather tha http://bugs.python.org/issue16487 opened by kristjan.jonsson #16488: Add context manager support to epoll object http://bugs.python.org/issue16488 opened by serhiy.storchaka #16468: argparse only supports iterable choices http://bugs.python.org/issue16468 opened by chris.jerdonek Most recent 15 issues with no replies (15) ========================================== #16488: Add context manager support to epoll object http://bugs.python.org/issue16488 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16484: Missing/broken documentation redirect for http://docs.python.o http://bugs.python.org/issue16484 #16482: pdb.set_trace() clobbering traceback on error http://bugs.python.org/issue16482 #16480: pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bi http://bugs.python.org/issue16480 #16472: Distutils+mingw links agains msvcr90, while python27.dll is li http://bugs.python.org/issue16472 #16471: upgrade to sphinx 1.1 http://bugs.python.org/issue16471 #16464: urllib.request: opener not resetting content-length http://bugs.python.org/issue16464 #16463: test_timeout failure on the RHEL buildbot http://bugs.python.org/issue16463 #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 #16446: pdb raises BdbQuit on 'quit' when started with set_trace http://bugs.python.org/issue16446 #16429: Emit SyntaxWarning for code that risks UnboundLocalError http://bugs.python.org/issue16429 #16428: turtle with compound shape doesn't get clicks http://bugs.python.org/issue16428 #16425: minidom replaceChild(new_child, old_child) removes new_child e http://bugs.python.org/issue16425 #16406: move the "Uploading Packages" section to distutils/packageinde http://bugs.python.org/issue16406 Most recent 15 issues waiting for review (15) ============================================= #16488: Add context manager support to epoll object http://bugs.python.org/issue16488 #16487: Allow ssl certificates to be speficfied from memory rather tha http://bugs.python.org/issue16487 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16485: FD leaks in aifc module http://bugs.python.org/issue16485 #16483: Make int(float('inf')) raise ValueError rather than OverflowEr http://bugs.python.org/issue16483 #16475: Support object instancing and recursion in marshal http://bugs.python.org/issue16475 #16474: More code coverage for imp module http://bugs.python.org/issue16474 #16473: Minor difference in decoding quoted-printable text http://bugs.python.org/issue16473 #16470: Backport set and dictionary comprehensions in tutorial to 2.7 http://bugs.python.org/issue16470 #16466: register command forgets password if no config file is created http://bugs.python.org/issue16466 #16465: dict creation performance regression http://bugs.python.org/issue16465 #16455: sys.getfilesystemencoding() is not the locale encoding on Free http://bugs.python.org/issue16455 #16451: Remove duplication between slice_indices and compute_slice_ind http://bugs.python.org/issue16451 #16447: SEGFAULT when setting type.__name__ http://bugs.python.org/issue16447 #16445: SEGFAULT when deleting Exception.message http://bugs.python.org/issue16445 Top 10 most discussed issues (10) ================================= #16465: dict creation performance regression http://bugs.python.org/issue16465 14 msgs #8865: select.poll is not thread safe http://bugs.python.org/issue8865 13 msgs #16418: argparse with many choices can generate absurdly long usage me http://bugs.python.org/issue16418 13 msgs #16427: Faster hash implementation http://bugs.python.org/issue16427 13 msgs #16444: Use support.TESTFN_UNDECODABLE on UNIX http://bugs.python.org/issue16444 11 msgs #16218: Python launcher does not support unicode characters http://bugs.python.org/issue16218 10 msgs #14631: Instance methods and WeakRefs don't mix. http://bugs.python.org/issue14631 8 msgs #13349: Non-informative error message in index() and remove() function http://bugs.python.org/issue13349 7 msgs #16416: Mac OS X: don't use the locale encoding but UTF-8 to encode an http://bugs.python.org/issue16416 7 msgs #16458: subprocess.py throw "The handle is invalid" error on duplicati http://bugs.python.org/issue16458 7 msgs Issues closed (41) ================== #7620: Vim syntax highlight http://bugs.python.org/issue7620 closed by brett.cannon #9084: vimrc: use matchall() instead of ":match" to allow multiple ma http://bugs.python.org/issue9084 closed by brett.cannon #9535: Pending signals are inherited by child processes http://bugs.python.org/issue9535 closed by gregory.p.smith #9893: Removing the Misc/Vim/ files http://bugs.python.org/issue9893 closed by brett.cannon #12428: functools test coverage http://bugs.python.org/issue12428 closed by pitrou #12907: Update test coverage devguide page http://bugs.python.org/issue12907 closed by brett.cannon #12908: Update dev-in-a-box for new coverage steps http://bugs.python.org/issue12908 closed by brett.cannon #14215: http://www.python.org/dev/peps/ title is python.org http://bugs.python.org/issue14215 closed by georg.brandl #14396: Popen wait() doesn't handle spurious wakeups http://bugs.python.org/issue14396 closed by gregory.p.smith #15677: Gzip/zlib allows for compression level=0 http://bugs.python.org/issue15677 closed by nadeem.vawda #15835: HP-UX build needs to be tweaked to pick up PATH_MAX http://bugs.python.org/issue15835 closed by skrah #15880: os.path.split() and long UNC names http://bugs.python.org/issue15880 closed by serhiy.storchaka #15894: _PyImport_ReInitLock() doesn't check return value of PyThread_ http://bugs.python.org/issue15894 closed by brett.cannon #16140: subprocess.Popen the os.close calls in _execute_child can rais http://bugs.python.org/issue16140 closed by gregory.p.smith #16144: misleading sentence in reference/import http://bugs.python.org/issue16144 closed by asvetlov #16290: PyComplex_AsCComplex should allow __complex__ to return float. http://bugs.python.org/issue16290 closed by mark.dickinson #16327: subprocess.Popen leaks file descriptors on os.fork() failure http://bugs.python.org/issue16327 closed by gregory.p.smith #16357: SSLSocket created from SSLContext.wrap_socket doesn't include http://bugs.python.org/issue16357 closed by pitrou #16378: venv.EnvBuilder docstring inconsistencies http://bugs.python.org/issue16378 closed by python-dev #16391: add "terminator" ctor argument to logging.StreamHandlers deriv http://bugs.python.org/issue16391 closed by vinay.sajip #16397: UserString doesn't combine nicely with strings http://bugs.python.org/issue16397 closed by Jorge.Cardona #16400: update default PyPI behavior in docs re: listing versions http://bugs.python.org/issue16400 closed by chris.jerdonek #16409: urlretrieve regression: first call returns block size as 0 http://bugs.python.org/issue16409 closed by gregory.p.smith #16411: zlib.Decompress.decompress() retains pointer to input buffer w http://bugs.python.org/issue16411 closed by nadeem.vawda #16424: regression: os.path.split('//hostname/foo/bar.txt') http://bugs.python.org/issue16424 closed by jcea #16436: Link directly to set and frozenset in built-in function docs http://bugs.python.org/issue16436 closed by chris.jerdonek #16448: 'module' object has no attribute 'font' when "import tkinter" http://bugs.python.org/issue16448 closed by r.david.murray #16449: RotatingFileHandler rollover doesn't work on QNX shmem filesys http://bugs.python.org/issue16449 closed by vinay.sajip #16452: Add support for weak reference to bound methods http://bugs.python.org/issue16452 closed by sfeltman #16453: Inconsistent dead weakref equality http://bugs.python.org/issue16453 closed by pitrou #16454: Mostly for discussion: _winapi as builtin for bootstrapping di http://bugs.python.org/issue16454 closed by sbt #16456: UnicodeDecodeError thrown for 'encode' operation on string http://bugs.python.org/issue16456 closed by haypo #16457: Allow operator 'getter' methods to take a list and return a tu http://bugs.python.org/issue16457 closed by r.david.murray #16459: sys.stdout.write printing length of string http://bugs.python.org/issue16459 closed by firatozgul #16460: Strange results for floor division ("//") with non-integer div http://bugs.python.org/issue16460 closed by mark.dickinson #16469: Exceptions raised by Fraction() different from those raised by http://bugs.python.org/issue16469 closed by mark.dickinson #16476: Trailing spaces in pretty-printed JSON http://bugs.python.org/issue16476 closed by ezio.melotti #16478: Fix division in tabnanny http://bugs.python.org/issue16478 closed by ezio.melotti #16479: Can't install Python 3 on Redhat Linux, make failed http://bugs.python.org/issue16479 closed by pitrou #16481: process handle leak on windows in multiprocessing http://bugs.python.org/issue16481 closed by sbt #16467: frozen importlib required for extending Python interpreter not http://bugs.python.org/issue16467 closed by atuining From vinay_sajip at yahoo.co.uk Fri Nov 16 18:18:02 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 16 Nov 2012 17:18:02 +0000 (UTC) Subject: [Python-Dev] Generally bored by installation References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> Message-ID: <loom.20121116T181532-233@post.gmane.org> Eli Bendersky <eliben <at> gmail.com> writes: > Use pss [http://pypi.python.org/pypi/pss] How does it compare with grin? [http://pypi.python.org/pypi/grin] On the face of it, they both do the same job. Regards, Vinay Sajip From chris.jerdonek at gmail.com Fri Nov 16 21:47:52 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 16 Nov 2012 12:47:52 -0800 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> Message-ID: <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> I filed an issue in the meta tracker about e-mails like the below that are sent to python-dev. The issue link is here: http://psf.upfronthosting.co.za/roundup/meta/issue492 --Chris On Thu, Nov 15, 2012 at 6:39 PM, Python tracker <roundup-admin at psf.upfronthosting.co.za> wrote: > > > An unexpected error occurred during the processing > of your message. The tracker administrator is being > notified. > > Return-Path: <python-dev at python.org> > X-Original-To: report at bugs.python.org > Delivered-To: roundup+tracker at psf.upfronthosting.co.za > Received: from mail.python.org (mail.python.org [IPv6:2001:888:2000:d::a6]) > by psf.upfronthosting.co.za (Postfix) with ESMTPS id DFD211C98E > for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) > Received: from albatross.python.org (localhost [127.0.0.1]) > by mail.python.org (Postfix) with ESMTP id 3Y2kDj4Tk8zRb6 > for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901; > t=1353033589; bh=H41Haza/UPvj9B16qvXQtqjlb22jRmazur238MqAPOE=; > h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To: > Subject:Message-Id:Date; > b=yz1YIEtIYuM3H9V1Ok0YJ/SDvU8xtO+cFOApgDH8jPRdxRS6z/CSdZ96bSY1cNFM7 > 4xxKY3LiUXipe439DPwEjxZr0HcsfX+2JdR4Pzf3OZmdopV6PEl8/twaIFoTHQMy8u > dTwpZ7G4OJKCc2IeCq4e5Bl/TEvQvVT9NO8eoa3k= > Received: from localhost (HELO mail.python.org) (127.0.0.1) > by albatross.python.org with SMTP; 16 Nov 2012 03:39:49 +0100 > Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72]) > by mail.python.org (Postfix) with ESMTP > for <report at bugs.python.org>; Fri, 16 Nov 2012 03:39:49 +0100 (CET) > MIME-Version: 1.0 > Content-Type: text/plain; charset="utf-8" > Content-Transfer-Encoding: base64 > From: python-dev at python.org > To: report at bugs.python.org > Subject: [issue15894] > Message-Id: <3Y2kDj4Tk8zRb6 at mail.python.org> > Date: Fri, 16 Nov 2012 03:39:49 +0100 (CET) > > TmV3IGNoYW5nZXNldCBiZDg1MzMxMWZmZTAgYnkgQnJldHQgQ2Fubm9uIGluIGJyYW5jaCAnZGVm > YXVsdCc6Cklzc3VlICMxNTg5NDogRG9jdW1lbnQgd2h5IHdlIGRvbid0IHdvcnJ5IGFib3V0IHJl > LWFjcXVpcmluZyB0aGUKaHR0cDovL2hnLnB5dGhvbi5vcmcvY3B5dGhvbi9yZXYvYmQ4NTMzMTFm > ZmUwCg== > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com > From eliben at gmail.com Fri Nov 16 22:15:26 2012 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 16 Nov 2012 13:15:26 -0800 Subject: [Python-Dev] Generally bored by installation In-Reply-To: <loom.20121116T181532-233@post.gmane.org> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <loom.20121116T181532-233@post.gmane.org> Message-ID: <CAF-Rda--Mi=J-L7MA1Ffo-1Jf3S=bBO9aoSVJfpmH+B+frQw9g@mail.gmail.com> On Fri, Nov 16, 2012 at 9:18 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote: > Eli Bendersky <eliben <at> gmail.com> writes: > > > Use pss [http://pypi.python.org/pypi/pss] > > How does it compare with grin? [http://pypi.python.org/pypi/grin] > > On the face of it, they both do the same job. > They're quite similar. pss is more close to ack, in the sense that it knows how to identify files belonging to various categories/languages and lets you easily guide the search this way. And some others bells and whistles. But generally, both are better than using the horrible "find | xargs grep" incantation and all its variations. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121116/0c24a7dc/attachment.html> From eliben at gmail.com Fri Nov 16 22:20:17 2012 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 16 Nov 2012 13:20:17 -0800 Subject: [Python-Dev] pss (was: Generally bored by installation) In-Reply-To: <20121116142545.GA19390@iskra.aviel.ru> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <20121116142545.GA19390@iskra.aviel.ru> Message-ID: <CAF-Rda9K4ZFS6UsUH7tD6AxbtfubGcy=F7bkcemcQ4D7DY9ufQ@mail.gmail.com> On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman <phd at phdru.name> wrote: > On Fri, Nov 16, 2012 at 05:34:37AM -0800, Eli Bendersky <eliben at gmail.com> > wrote: > > > exec find . \( -type d \( -name CVS -o -name .svn -o -name .hg -o -name > > > .git -o -path ./.mozilla/\*/Cache/\* -o -path ./tmp/\* \) -prune \) -o > > > -type f -exec grep -I "$@" '{}' \+ 2>/dev/null > > > > > Use pss ;-) [http://pypi.python.org/pypi/pss] > > Hardly. I am proficient in find+grep. And pss is underdocumented. > > Even reading https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined with the built-in help? If anything is missing, by all means let me know and I'll gladly add it. [Sorry for the offtopic, folks, promise it's short] Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121116/e256874c/attachment-0001.html> From phd at phdru.name Fri Nov 16 22:37:45 2012 From: phd at phdru.name (Oleg Broytman) Date: Sat, 17 Nov 2012 01:37:45 +0400 Subject: [Python-Dev] Generally bored by installation In-Reply-To: <CAF-Rda--Mi=J-L7MA1Ffo-1Jf3S=bBO9aoSVJfpmH+B+frQw9g@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <loom.20121116T181532-233@post.gmane.org> <CAF-Rda--Mi=J-L7MA1Ffo-1Jf3S=bBO9aoSVJfpmH+B+frQw9g@mail.gmail.com> Message-ID: <20121116213745.GA28111@iskra.aviel.ru> On Fri, Nov 16, 2012 at 01:15:26PM -0800, Eli Bendersky <eliben at gmail.com> wrote: > But generally, both are better than using the horrible "find | xargs grep" > incantation and all its variations. In what way they are better? I found 'find' to be quite capable, and find -exec grep '{}' \+ to be quite fast. In my opinion all those small utilities are pets of their authors and they solve their problems quite good... but not mine. GNU utilities are used by a huge number of people and solve much wider problems much better. Though not necessary in a much more elegant way ;-) Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Fri Nov 16 22:40:04 2012 From: phd at phdru.name (Oleg Broytman) Date: Sat, 17 Nov 2012 01:40:04 +0400 Subject: [Python-Dev] pss In-Reply-To: <CAF-Rda9K4ZFS6UsUH7tD6AxbtfubGcy=F7bkcemcQ4D7DY9ufQ@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <20121116142545.GA19390@iskra.aviel.ru> <CAF-Rda9K4ZFS6UsUH7tD6AxbtfubGcy=F7bkcemcQ4D7DY9ufQ@mail.gmail.com> Message-ID: <20121116214004.GB28111@iskra.aviel.ru> On Fri, Nov 16, 2012 at 01:20:17PM -0800, Eli Bendersky <eliben at gmail.com> wrote: > On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman <phd at phdru.name> wrote: > > And pss is underdocumented. > > > Even reading https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined Compare your page with man find and man grep Compare amount of text, levels of details, number of options, examples... Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From guido at python.org Fri Nov 16 23:35:13 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Nov 2012 14:35:13 -0800 Subject: [Python-Dev] pss In-Reply-To: <20121116214004.GB28111@iskra.aviel.ru> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <20121116142545.GA19390@iskra.aviel.ru> <CAF-Rda9K4ZFS6UsUH7tD6AxbtfubGcy=F7bkcemcQ4D7DY9ufQ@mail.gmail.com> <20121116214004.GB28111@iskra.aviel.ru> Message-ID: <CAP7+vJJYbbRW_s1PZW=pWhgHiPn3F=tYvHCm4+kiHCaf7fJdig@mail.gmail.com> Oleg, this is inappropriate for python-dev. It's also a bit uncivilized and unkind. On Fri, Nov 16, 2012 at 1:40 PM, Oleg Broytman <phd at phdru.name> wrote: > On Fri, Nov 16, 2012 at 01:20:17PM -0800, Eli Bendersky <eliben at gmail.com> wrote: >> On Fri, Nov 16, 2012 at 6:25 AM, Oleg Broytman <phd at phdru.name> wrote: >> > And pss is underdocumented. >> > >> Even reading https://bitbucket.org/eliben/pss/wiki/Usage#!usage-samplescombined > > Compare your page with > > man find > > and > > man grep > > Compare amount of text, levels of details, number of options, > examples... > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Nov 16 23:41:11 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 16 Nov 2012 23:41:11 +0100 Subject: [Python-Dev] Failed issue tracker submission References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> Message-ID: <20121116234111.7ef55ee8@pitrou.net> On Fri, 16 Nov 2012 12:47:52 -0800 Chris Jerdonek <chris.jerdonek at gmail.com> wrote: > I filed an issue in the meta tracker about e-mails like the below that > are sent to python-dev. The issue link is here: > > http://psf.upfronthosting.co.za/roundup/meta/issue492 These e-mails are just because someone mentioned the wrong issue number in a commit message, I think. Regards Antoine. From guido at python.org Sat Nov 17 00:36:03 2012 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Nov 2012 15:36:03 -0800 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <20121116234111.7ef55ee8@pitrou.net> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> Message-ID: <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> But python-dev seems a poor place to spam with those errors. On Fri, Nov 16, 2012 at 2:41 PM, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Fri, 16 Nov 2012 12:47:52 -0800 > Chris Jerdonek <chris.jerdonek at gmail.com> wrote: >> I filed an issue in the meta tracker about e-mails like the below that >> are sent to python-dev. The issue link is here: >> >> http://psf.upfronthosting.co.za/roundup/meta/issue492 > > These e-mails are just because someone mentioned the wrong issue number > in a commit message, I think. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Sat Nov 17 00:47:33 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sat, 17 Nov 2012 00:47:33 +0100 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> Message-ID: <20121117004733.Horde.GGeAbLuWis5QptCVhIcgUVA@webmail.df.eu> Zitat von Guido van Rossum <guido at python.org>: > But python-dev seems a poor place to spam with those errors. It's certainly not deliberate that they are sent. However, there are too few people interested in working on fixing that so that it remains unfixed. Even finding out why they are sent to python-dev is tricky. Regards, Martin From solipsis at pitrou.net Sat Nov 17 01:01:52 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 17 Nov 2012 01:01:52 +0100 Subject: [Python-Dev] Failed issue tracker submission References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> <20121117004733.Horde.GGeAbLuWis5QptCVhIcgUVA@webmail.df.eu> Message-ID: <20121117010152.7ade2b38@pitrou.net> On Sat, 17 Nov 2012 00:47:33 +0100 martin at v.loewis.de wrote: > > Zitat von Guido van Rossum <guido at python.org>: > > > But python-dev seems a poor place to spam with those errors. > > It's certainly not deliberate that they are sent. However, there > are too few people interested in working on fixing that so that > it remains unfixed. Even finding out why they are sent to python-dev > is tricky. I think it's because I configured the dummy "python-dev" user that way: http://bugs.python.org/user13902 Regards Antoine. From phd at phdru.name Sat Nov 17 01:34:08 2012 From: phd at phdru.name (Oleg Broytman) Date: Sat, 17 Nov 2012 04:34:08 +0400 Subject: [Python-Dev] pss In-Reply-To: <CAP7+vJJYbbRW_s1PZW=pWhgHiPn3F=tYvHCm4+kiHCaf7fJdig@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <20121116050135.GA7247@iskra.aviel.ru> <CAF-Rda92uB8vKtErhQx9T10D2MqQ-RNxNoAKDGH2B=+g51tV2A@mail.gmail.com> <20121116142545.GA19390@iskra.aviel.ru> <CAF-Rda9K4ZFS6UsUH7tD6AxbtfubGcy=F7bkcemcQ4D7DY9ufQ@mail.gmail.com> <20121116214004.GB28111@iskra.aviel.ru> <CAP7+vJJYbbRW_s1PZW=pWhgHiPn3F=tYvHCm4+kiHCaf7fJdig@mail.gmail.com> Message-ID: <20121117003408.GB2381@iskra.aviel.ru> On Fri, Nov 16, 2012 at 02:35:13PM -0800, Guido van Rossum <guido at python.org> wrote: > Oleg, this is inappropriate for python-dev. It's also a bit > uncivilized and unkind. Shame on me! Please, everyone, take my sincerest apology! I'm stopping now. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From victor.stinner at gmail.com Sat Nov 17 02:13:25 2012 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 17 Nov 2012 02:13:25 +0100 Subject: [Python-Dev] Register-based VM for CPython Message-ID: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> Hi, I'm still looking for the best approach to speedup CPython. I found the following article comparing a stack-based VM to a register-based VM which announces a speed up between 20% and 40% (depending if the instruction dispatch uses a dummy switch or computed goto): http://static.usenix.org/events/vee05/full_papers/p153-yunhe.pdf Yunhe Shi, David Gregg, Andrew Beatty, M. Anton Ertl, 2005 I tried to implement such register-based instructions for CPython in the following fork of Python 3.4: http://hg.python.org/sandbox/registervm/ Contact me if you are interested and/or if you would like to contribute! -- My implementation is not finished and still experimental. It is only enabled explictly by calling registervm.optimize_func(func) where registervm is a new module. This function rewrites the bytecode of the function inplace. I chose this approach because it was simple to implement, it is simple to compare stack-based and register-based instructions, and it is also the approach used in the paper :-) The major drawback of the register approach (at least of my implementation) is that it changes the lifetime of objects. Newly created objects are only "destroyed" at the exit of the function, whereas the stack-based VM destroys "immediatly" objects (thanks to the reference counter). PyPy has similar issues with its different garbage collector. On Linux (with computed goto in ceval.c), there are some interesting speedups in pybench, up to 25% faster. Such speedup can be explained with the bytecode. For example, "c = a + b" is compiled to: LOAD_FAST 'a' LOAD_FAST 'b' BINARY_ADD STORE_FAST 'c' The optimized bytecode becomes: BINARY_ADD_REG 'c', 'a', 'b' So 4 operations are replaced with only 1 operation. So instruction dispatching has a cost, measurable in this microbenchmark. My implementation is incomplete: if you see "PUSH_REG" or "POP_REG" in the assembler, your code will be slower than the original bytecode. There is no register allocator and the optimizer doesn't understand the control flow, so I expect more performance improvment with a more complete implementation. Some optimizations are also disabled by default because the implementation is buggy. The main changes of registervm are done in Python/ceval.c (implement new instructions) and Lib/registervm.py (the new module rewriting the bytecode). Objects/frameobject.c is also modified to allocate registers. See registervm documentation for more information: http://hg.python.org/sandbox/registervm/file/tip/REGISTERVM.txt -- The WPython project is similar to my work (except that it does not use registers). It tries also to reduce the overhead of instruction dispatch by using more complex instructions. http://code.google.com/p/wpython/ Using registers instead of a stack allow to implement more optimizations (than WPython). For example, it's possible to load constants outside loops and merge "duplicate constant loads". I also implemented more aggressive and experimental optimizations (disabled by default) which may break applications: move loads of attributes and globals outside of loops, and replace binary operations with inplace operations. For example, "x=[]; for ...: x.append(...)" is optimized to something like "x=[]; x_append=x.append; for ...: x_append(...)", and "x = x + 1" is replaced with "x += 1". Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121117/3b6d4290/attachment.html> From rdmurray at bitdance.com Sat Nov 17 07:06:06 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 17 Nov 2012 01:06:06 -0500 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <20121117010152.7ade2b38@pitrou.net> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> <20121117004733.Horde.GGeAbLuWis5QptCVhIcgUVA@webmail.df.eu> <20121117010152.7ade2b38@pitrou.net> Message-ID: <20121117060607.68142250129@webabinitio.net> On Sat, 17 Nov 2012 01:01:52 +0100, Antoine Pitrou <solipsis at pitrou.net> wrote: > On Sat, 17 Nov 2012 00:47:33 +0100 > martin at v.loewis.de wrote: > > > > Zitat von Guido van Rossum <guido at python.org>: > > > > > But python-dev seems a poor place to spam with those errors. > > > > It's certainly not deliberate that they are sent. However, there > > are too few people interested in working on fixing that so that > > it remains unfixed. Even finding out why they are sent to python-dev > > is tricky. > > I think it's because I configured the dummy "python-dev" user that way: > http://bugs.python.org/user13902 I'm pretty sure it's because python-dev is the 'from' address used when the messages are sent...and the configuration of that user is what allows them to be accepted. I suggest changing the from address and the account configuration to tracker-discuss at python.org instead. --David From chris.jerdonek at gmail.com Sat Nov 17 07:35:38 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 16 Nov 2012 22:35:38 -0800 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <20121117060607.68142250129@webabinitio.net> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> <20121117004733.Horde.GGeAbLuWis5QptCVhIcgUVA@webmail.df.eu> <20121117010152.7ade2b38@pitrou.net> <20121117060607.68142250129@webabinitio.net> Message-ID: <CAOTb1wdbaSWFnKkf2N5kzrddjbJgvCE1=5KgmNJ6v+eBqhESRA@mail.gmail.com> On Fri, Nov 16, 2012 at 10:06 PM, R. David Murray <rdmurray at bitdance.com> wrote: > On Sat, 17 Nov 2012 01:01:52 +0100, Antoine Pitrou <solipsis at pitrou.net> wrote: >> On Sat, 17 Nov 2012 00:47:33 +0100 >> martin at v.loewis.de wrote: >> > >> > Zitat von Guido van Rossum <guido at python.org>: >> > >> > > But python-dev seems a poor place to spam with those errors. >> > >> > It's certainly not deliberate that they are sent. However, there >> > are too few people interested in working on fixing that so that >> > it remains unfixed. Even finding out why they are sent to python-dev >> > is tricky. >> >> I think it's because I configured the dummy "python-dev" user that way: >> http://bugs.python.org/user13902 > > I'm pretty sure it's because python-dev is the 'from' address > used when the messages are sent...and the configuration of > that user is what allows them to be accepted. > > I suggest changing the from address and the account configuration > to tracker-discuss at python.org instead. Above and on the meta tracker issue I filed, it sounds like Martin is saying that the e-mails should never be going to python-dev (or tracker-discuss) in the first place -- due to a roundup bug and because the e-mail is already being sent to the submitter and the roundup admins. Thus, changing the from address would only mitigate the problem and not be fixing the root cause. --Chris From solipsis at pitrou.net Sat Nov 17 11:00:03 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 17 Nov 2012 11:00:03 +0100 Subject: [Python-Dev] cpython (2.7): Closes #16461: Wave library should be able to deal with 4GB wav files, and References: <3Y3LGW5lKLzRXK@mail.python.org> Message-ID: <20121117110003.7086e779@pitrou.net> On Sat, 17 Nov 2012 03:43:31 +0100 (CET) jesus.cea <python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/b8ece33ce27d > changeset: 80455:b8ece33ce27d > branch: 2.7 > parent: 80439:457c0c9c7893 > user: Jesus Cea <jcea at jcea.es> > date: Sat Nov 17 03:38:17 2012 +0100 > summary: > Closes #16461: Wave library should be able to deal with 4GB wav files, and sample rate of 44100 Hz. Any reason you didn't add any tests? Regards Antoine. From arigo at tunes.org Sat Nov 17 11:17:40 2012 From: arigo at tunes.org (Armin Rigo) Date: Sat, 17 Nov 2012 11:17:40 +0100 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> Message-ID: <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> Hi Victor, On Sat, Nov 17, 2012 at 2:13 AM, Victor Stinner <victor.stinner at gmail.com> wrote: > The major drawback of the register approach (at least of my implementation) > is that it changes the lifetime of objects. Newly created objects are only > "destroyed" at the exit of the function, whereas the stack-based VM destroys > "immediatly" objects (thanks to the reference counter). PyPy has similar > issues with its different garbage collector. That is not strictly correct. PyPy, Jython and IronPython have non-immediate destructors, but as far as I can tell they all avoid to keep objects alive for an unbounded amount of time. This important difference is visible if the function calls other code that takes a long while to run: in your approach, the objects created by the function itself will stay alive for the whole duration, while the other interpreters will all release them soon after they are not referenced any more --- not instantly like CPython but still soon. A bient?t, Armin. From chris.jerdonek at gmail.com Sat Nov 17 19:47:02 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 17 Nov 2012 10:47:02 -0800 Subject: [Python-Dev] Split unicodeobject.c into subfiles? In-Reply-To: <20121004234657.Horde.McWZBML8999QbgPRvu5ECIA@webmail.df.eu> References: <CAMpsgwajAtvqiBa7ruD3rX39BC7ZCgNuyq6vza0yZ77t6icx5A@mail.gmail.com> <20121004234657.Horde.McWZBML8999QbgPRvu5ECIA@webmail.df.eu> Message-ID: <CAOTb1wdbkGkK+pQ-qyVr3RBDxL-LBhs+Wp6D+yYcv9viH9JL_w@mail.gmail.com> [Apologies for resurrecting a few-weeks old thread.] On Thu, Oct 4, 2012 at 2:46 PM, <martin at v.loewis.de> wrote: > > Zitat von Victor Stinner <victor.stinner at gmail.com>: > >> I only see one argument against such refactoring: it will be harder to >> backport/forwardport bugfixes. > > I'm opposed for a different reason: I think it will be *harder* to maintain. > The amount of code will not be reduced, but now you also need to guess what > file some piece of functionality may be in. Instead of having my text editor > (Emacs) search in one file, it will have to search across multiple files - > but not across all open buffers, but only some of them (since I will have > many other source files open as well). > > I really fail to see what problem people have with large source files. > What is it that you want to do that can be done easier if it's multiple > files? One thing is browse or link to such code files on the web (e.g. from within a tracker comment or from within our online documentation). For example, today I was unable to open the following page from within a browser to link to one of its lines on a tracker comment: http://hg.python.org/cpython/file/27c20650aeab/Objects/unicodeobject.c My laptop's fan simply turns on and the page hangs indefinitely while loading. I don't think this point was ever mentioned. --Chris From rosuav at gmail.com Sat Nov 17 19:55:02 2012 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 18 Nov 2012 05:55:02 +1100 Subject: [Python-Dev] Split unicodeobject.c into subfiles? In-Reply-To: <CAOTb1wdbkGkK+pQ-qyVr3RBDxL-LBhs+Wp6D+yYcv9viH9JL_w@mail.gmail.com> References: <CAMpsgwajAtvqiBa7ruD3rX39BC7ZCgNuyq6vza0yZ77t6icx5A@mail.gmail.com> <20121004234657.Horde.McWZBML8999QbgPRvu5ECIA@webmail.df.eu> <CAOTb1wdbkGkK+pQ-qyVr3RBDxL-LBhs+Wp6D+yYcv9viH9JL_w@mail.gmail.com> Message-ID: <CAPTjJmot3xJtsVNHbZomWR9c=KiM_neUTH2OToHvH8TroFq+wg@mail.gmail.com> On Sun, Nov 18, 2012 at 5:47 AM, Chris Jerdonek <chris.jerdonek at gmail.com> wrote: > On Thu, Oct 4, 2012 at 2:46 PM, <martin at v.loewis.de> wrote: >> I really fail to see what problem people have with large source files. >> What is it that you want to do that can be done easier if it's multiple >> files? > > One thing is browse or link to such code files on the web (e.g. from > within a tracker comment or from within our online documentation). > For example, today I was unable to open the following page from within > a browser to link to one of its lines on a tracker comment: > > http://hg.python.org/cpython/file/27c20650aeab/Objects/unicodeobject.c > > My laptop's fan simply turns on and the page hangs indefinitely while loading. Curious. This sounds like a web browser issue - I can pull it up in either Chrome or Firefox on Windows on my 2GHz/2GB RAM laptop with a visible pause, but not more than half a second. However, this page is rather more significant, and is affected equally by the file size: http://hg.python.org/cpython/annotate/27c20650aeab/Objects/unicodeobject.c ChrisA From chris.jerdonek at gmail.com Sat Nov 17 20:16:21 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 17 Nov 2012 11:16:21 -0800 Subject: [Python-Dev] Split unicodeobject.c into subfiles? In-Reply-To: <CAPTjJmot3xJtsVNHbZomWR9c=KiM_neUTH2OToHvH8TroFq+wg@mail.gmail.com> References: <CAMpsgwajAtvqiBa7ruD3rX39BC7ZCgNuyq6vza0yZ77t6icx5A@mail.gmail.com> <20121004234657.Horde.McWZBML8999QbgPRvu5ECIA@webmail.df.eu> <CAOTb1wdbkGkK+pQ-qyVr3RBDxL-LBhs+Wp6D+yYcv9viH9JL_w@mail.gmail.com> <CAPTjJmot3xJtsVNHbZomWR9c=KiM_neUTH2OToHvH8TroFq+wg@mail.gmail.com> Message-ID: <CAOTb1wd0q8LfGjwrsyHRZq9RGGNj_BxZFUyqydNFbBeKuacS0Q@mail.gmail.com> On Sat, Nov 17, 2012 at 10:55 AM, Chris Angelico <rosuav at gmail.com> wrote: > On Sun, Nov 18, 2012 at 5:47 AM, Chris Jerdonek > <chris.jerdonek at gmail.com> wrote: >> On Thu, Oct 4, 2012 at 2:46 PM, <martin at v.loewis.de> wrote: >>> I really fail to see what problem people have with large source files. >>> What is it that you want to do that can be done easier if it's multiple >>> files? >> >> One thing is browse or link to such code files on the web (e.g. from >> within a tracker comment or from within our online documentation). >> For example, today I was unable to open the following page from within >> a browser to link to one of its lines on a tracker comment: >> >> http://hg.python.org/cpython/file/27c20650aeab/Objects/unicodeobject.c >> >> My laptop's fan simply turns on and the page hangs indefinitely while loading. > > Curious. This sounds like a web browser issue - I can pull it up in > either Chrome or Firefox on Windows on my 2GHz/2GB RAM laptop with a > visible pause, but not more than half a second. I'm also using Chrome and on a fairly new Mac. Perhaps. I tried again and it froze up several open *.python.org tabs (mail.python.org, bugs.python.org, etc). However, later it worked as you describe. The problem seems sporadic. --Chris From kristjan at ccpgames.com Sat Nov 17 22:52:44 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Sat, 17 Nov 2012 21:52:44 +0000 Subject: [Python-Dev] externals? In-Reply-To: <20121116121251.GA13953@snakebite.org> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> Message-ID: <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> Thanks for your pro-tip. Might I suggest that it ought to go into the dev FAQ? Along with an explanation that a windows dev has to have SVN installed too, just for the laughs? I think there might be a benefit to moving at least the current externals to a separate HG repository. We could easily have multiple branches in that repo reflecting the required externals for each version under active HG development). There is an inherent drawback in having to rely on two different RCS to fetch the necessary stuff, imho. K -----Original Message----- From: Trent Nelson [mailto:trent at snakebite.org] Sent: 16. n?vember 2012 12:13 To: Kristj?n Valur J?nsson Cc: Benjamin Peterson; Python-Dev (python-dev at python.org) Subject: Re: [Python-Dev] externals? On Thu, Nov 15, 2012 at 01:20:09AM -0800, Kristj?n Valur J?nsson wrote: > Perhaps the unix makefiles get the proper version, but a windows developer has to fetch those externals manually. Pro-tip: if you're developing on Windows, you're mad if you don't prime your dev env with Tools\buildbot\externals.bat. It takes care of *everything*. I wish every proprietary UNIX system we support had something similar. > Also, is there any reason to keep this in svn? I think it's more a case of there being no tangible benefit (and numerous drawbacks) to switching it to hg. I personally have no need for a local hg repo with 30 different Tcl/Tk histories in it. Subversion's good for this sort of use case. The externals repo gets committed to maybe, what, 3-4 times a year? > Why not check this in to HG, we need not worry about history, etc. Who are these mystical people worried about history? ;-) Trent. From kristjan at ccpgames.com Sat Nov 17 22:48:03 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Sat, 17 Nov 2012 21:48:03 +0000 Subject: [Python-Dev] need reviewers for #16475 and #16487 Message-ID: <EFE3877620384242A686D52278B7CCD329DE035B@RKV-IT-EXCH103.ccp.ad.local> Hello there. I'd like to have some pair of eyes on a couple of patches i?ve submitted. http://bugs.python.org/issue16475 This fixes a regression in marshal between 2.x and 3.x, reinstating string reuse and internment support. In addition, it generalizes string reuse to all objects, allowing for data optimizations to be made on code objects before marshaling. This straighforward extension considerably enhances the utility of the marshal module as a low-cost data serialization tool. http://bugs.python.org/issue16487 This allows ssl contexts to be initialized with certificates from memory, rather than having to rely on the openssl performing its own file IO to read them. This allows clients and servers that have their certificates deployed e.g. from a db or verbatim in a module, to use ssl without having to resort to temporary disk files and physical IO. Both of these patches are bourne out of work performed at CCP. The former comes from work on marshal in order to support our own code object optimizer, which helps save memory on the PS3. The second comes from us supporting isolated embedded python servers and clients and not wanting to complicate things with unnecessary temporary files for storing credidentials that are obtained from elsewhere. Both were, of course, 2.7 modifications, that I have now ported to 3.4 for the benefit of the python community. K -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121117/e389ba20/attachment.html> From martin at v.loewis.de Sat Nov 17 23:47:57 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sat, 17 Nov 2012 23:47:57 +0100 Subject: [Python-Dev] externals? In-Reply-To: <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> Zitat von Kristj?n Valur J?nsson <kristjan at ccpgames.com>: > Thanks for your pro-tip. Might I suggest that it ought to go into > the dev FAQ? Along with an explanation that a windows dev has to > have SVN installed too, just for the laughs? > I think there might be a benefit to moving at least the current > externals to a separate HG repository. We could easily have > multiple branches in that repo reflecting the required externals for > each version under active HG development). Feel free to work on this. Consider that using hg may significantly increase the amount of network traffic, since the repo(s) will contain multiple versions, when only one specific version is needed. When working on this, try to come up with other automated download procedures, e.g. ones that do not involve revision control and have support built into Windows. Regards, Martin From solipsis at pitrou.net Sun Nov 18 00:05:31 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Nov 2012 00:05:31 +0100 Subject: [Python-Dev] Failed issue tracker submission In-Reply-To: <CAOTb1wdbaSWFnKkf2N5kzrddjbJgvCE1=5KgmNJ6v+eBqhESRA@mail.gmail.com> References: <20121116023950.2EC2B1CDAD@psf.upfronthosting.co.za> <CAOTb1wcaUsB8-ZOTGgAdSJHg519iMMxm8=PU-i3DVXhS43a_bQ@mail.gmail.com> <20121116234111.7ef55ee8@pitrou.net> <CAP7+vJKG3a7T_gV4_kkJoi4WWtsi90BJQCS+XBy_MrA-DsMioA@mail.gmail.com> <20121117004733.Horde.GGeAbLuWis5QptCVhIcgUVA@webmail.df.eu> <20121117010152.7ade2b38@pitrou.net> <20121117060607.68142250129@webabinitio.net> <CAOTb1wdbaSWFnKkf2N5kzrddjbJgvCE1=5KgmNJ6v+eBqhESRA@mail.gmail.com> Message-ID: <20121118000531.6cf0b00b@pitrou.net> On Fri, 16 Nov 2012 22:35:38 -0800 Chris Jerdonek <chris.jerdonek at gmail.com> wrote: > > > > I'm pretty sure it's because python-dev is the 'from' address > > used when the messages are sent...and the configuration of > > that user is what allows them to be accepted. > > > > I suggest changing the from address and the account configuration > > to tracker-discuss at python.org instead. > > Above and on the meta tracker issue I filed, it sounds like Martin is > saying that the e-mails should never be going to python-dev (or > tracker-discuss) in the first place -- due to a roundup bug and > because the e-mail is already being sent to the submitter and the > roundup admins. Thus, changing the from address would only mitigate > the problem and not be fixing the root cause. So, for the record, the "From" address has been changed to tracker-discuss at python.org, so python-dev shouldn't be spammed anymore. Cheers Antoine. From guido at python.org Sun Nov 18 01:05:04 2012 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Nov 2012 16:05:04 -0800 Subject: [Python-Dev] need reviewers for #16475 and #16487 In-Reply-To: <EFE3877620384242A686D52278B7CCD329DE035B@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DE035B@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CAP7+vJ+f+csADFCSpZFvUxhVV=X_pu5jGsHDc2yJaz1wYcuoZw@mail.gmail.com> On Sat, Nov 17, 2012 at 1:48 PM, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > Hello there. > > I?d like to have some pair of eyes on a couple of patches i?ve submitted. Sorry, can't help you with these, but I've got a language nit... > http://bugs.python.org/issue16475 > > This fixes a regression in marshal between 2.x and 3.x, reinstating string > reuse and internment support. In addition, it generalizes string reuse to It's not internment -- that means imprisonment. The term we use is interning. (The dictionary will tell you that means imprisonment too -- but it's long been used as the name for this particular technique. Internment has not.) > all objects, allowing for data optimizations to be made on code objects > before marshaling. This straighforward extension considerably enhances the > utility of the marshal module as a low-cost data serialization tool. > > > > http://bugs.python.org/issue16487 > > This allows ssl contexts to be initialized with certificates from memory, > rather than having to rely on the openssl performing its own file IO to read > them. This allows clients and servers that have their certificates > deployed e.g. from a db or verbatim in a module, to use ssl without having > to resort to temporary disk files and physical IO. > > > > Both of these patches are bourne out of work performed at CCP. The former > comes from work on marshal in order to support our own code object > optimizer, which helps save memory on the PS3. The second comes from us > supporting isolated embedded python servers and clients and not wanting to > complicate things with unnecessary temporary files for storing credidentials > that are obtained from elsewhere. > > > > Both were, of course, 2.7 modifications, that I have now ported to 3.4 for > the benefit of the python community. > > > > K > > > > > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From g.brandl at gmx.net Sun Nov 18 08:22:23 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 18 Nov 2012 08:22:23 +0100 Subject: [Python-Dev] externals? In-Reply-To: <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> Message-ID: <k8a2at$tg5$1@ger.gmane.org> Am 17.11.2012 23:47, schrieb martin at v.loewis.de: > > Zitat von Kristj?n Valur J?nsson <kristjan at ccpgames.com>: > >> Thanks for your pro-tip. Might I suggest that it ought to go into >> the dev FAQ? Along with an explanation that a windows dev has to >> have SVN installed too, just for the laughs? >> I think there might be a benefit to moving at least the current >> externals to a separate HG repository. We could easily have >> multiple branches in that repo reflecting the required externals for >> each version under active HG development). > > Feel free to work on this. Consider that using hg may significantly > increase the amount of network traffic, since the repo(s) will contain > multiple versions, when only one specific version is needed. > > When working on this, try to come up with other automated download > procedures, e.g. ones that do not involve revision control and have > support built into Windows. One way would be to use one hg repo per version, and (maybe, if needed) a master repo that has them as subrepos. Whoever needs externals can download the repo as a zipfile and unpack it (both of which is possible with batteries only). Note that in this scenario, hg is used mostly in order to avoid another service (such as rsync), and for developer convenience. Georg From arigo at tunes.org Sun Nov 18 10:00:43 2012 From: arigo at tunes.org (Armin Rigo) Date: Sun, 18 Nov 2012 10:00:43 +0100 Subject: [Python-Dev] externals? In-Reply-To: <k8a2at$tg5$1@ger.gmane.org> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> Message-ID: <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> Hi, On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl <g.brandl at gmx.net> wrote: > One way would be to use one hg repo per version, and (maybe, if needed) > a master repo that has them as subrepos. Or have all versions in the same repo as usual (with branches), but have hg subrepos point to different repos: ones extracted from the main repo by containing only the correct branch. But it might be a bit delicate to pull this off. (hg clone takes a "-r" option and copies only things needed for the given revision or branch, but apparently we can't pass this option automatically to the cloning of subrepos. (Maybe it points out that subrepos are a hack best done without altogether, which is what we did in pypy.)) A bient?t, Armin. From g.brandl at gmx.net Sun Nov 18 10:04:24 2012 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 18 Nov 2012 10:04:24 +0100 Subject: [Python-Dev] externals? In-Reply-To: <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> Message-ID: <k8a8a7$3bf$1@ger.gmane.org> Am 18.11.2012 10:00, schrieb Armin Rigo: > Hi, > > On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl <g.brandl at gmx.net> wrote: >> One way would be to use one hg repo per version, and (maybe, if needed) >> a master repo that has them as subrepos. > > Or have all versions in the same repo as usual (with branches), but > have hg subrepos point to different repos: ones extracted from the > main repo by containing only the correct branch. But it might be a > bit delicate to pull this off. (hg clone takes a "-r" option and > copies only things needed for the given revision or branch, but > apparently we can't pass this option automatically to the cloning of > subrepos. (Maybe it points out that subrepos are a hack best done > without altogether, which is what we did in pypy.)) Yep. Anyway, if every external version goes into a branch, then we don't need subrepos anyway. That is a better idea than mine. Since you can use (e.g.) "hg clone -r tk-8.5" or download a tarball specific to a branch, nobody should need to get the whole externals history on clone. Georg From kristjan at ccpgames.com Sun Nov 18 11:54:57 2012 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Sun, 18 Nov 2012 10:54:57 +0000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> Yes! For many years I have been very frustrated by the install-centric nature of python. I am biased, of course, by the fact that I am developing an application where python is embedded, an application that needs to run out of the box. A developer may have many many versions (branches) of the application on his drive, and each needs to work separately. We have managed to isolate things, by patching python (and contributing that patch) to override the default library seach path (and ignore environment paths) when python is started up thogh the api. All well and good. But recently we have started in increasing amount to use external libraries and packages and we have been introduced to the dependency hell that is public python packages. In this install-centric world, developers reference huge external packages without a second thought, which cause large dependency trees. Using a simple tool may require whole HTTP frameworks to be downloaded. What is worse is when there are versioning conflicts between those dependencies. I don't have a well formed solution in mind, but I would see it desirable to have a way for someone to release his package with all its dependencies as a self-contained and isolated unit. E.g. if package foo.py relies on functionality from version 1.7 of bar.py, then that functionality could be bottled up for foo?s exclusive usage. Another package, baz.py, could then also make use of bar, but version 1.8. The two bar versions would be isolated. Perhaps this is just a pipedream. Even unpossible. But it doesn't harm to try to think about better ways to do things. K -----Original Message----- From: Christian Tismer [mailto:tismer at stackless.com] Sent: 15. n?vember 2012 23:10 To: Kristj?n Valur J?nsson Cc: python-dev at python.org Subject: Generally boared by installation (Re: [Python-Dev] Setting project home path the best way) Hi guys, I am bored of installing things. Bored of things that happen to not work for some minor reasons. Reasons that are not immediately obvious. Things that don't work because some special case was not handled. Things that compile for half an hour and then complain that something is not as expected. May it be a compiler, a library, a command like pip or easy-install, a system like macports or homebrew, virtualenv, whatsoever. These things are all great if they work. When they do not work, the user is in some real trouble. And he reads hundreds Of blogs and sites and emails, which all answer a bit of slightly related questions, but all in all - This is not how Python should work !! I am really bored and exhausted and annoyed by those packages which Pretend to make my life eadier, but they don't really. Something is really missing. I want something that is easy to use in all cases, also if it fails. Son't get me wrong, I like things like pip or virtualenv or homebrew. I just think they have to be rewritten completely. They have the wrong assumption that things work! The opposite should be the truth: by default, things go wrong. Correctness is very fragile. I am thinking of a better concept that is harder to break. I thin to design a setup tool that is much more checking itself and does not trust in any assumption. Scenario: After hours and hours, I find how to modify setup.py to function almost correctly for PySide. This was ridiculously hard to do! Settings for certain directories, included and stuff are not checked when they could be, but after compiling a lot of things! After a lot of tries and headaches, I find out that virtualenv barfs on a simple link like ./.Python, the executable, when switching from stock Python to a different (homebrew) version!! This was obviously never tested well, so it frustrates me quite a lot. I could fill a huge list full of complaints like that if I had time. But I don't. Instead, I think installation scripts are generally still wrong by concept today and need to be written in a different way. I would like to start a task force and some sprints about improving this situation. My goal is some unbreakable system of building blocks that are self-contained with no dependencies, that have a defined interface to talk to, and that know themselves completely by introspection. They should not work because they happen to work around all known defects, but by design and control. Whoever is interested to work with me on this is hereby honestly welcomed! Cheers - chris Sent from my Ei4Steve On Nov 15, 2012, at 10:17, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > When python is being run from a compile environment, it detects this by looking for "Lib" folders in directories above the one containing the executable. > (I always thought that this "special" execution mode, hardwired in, > was a bit odd, and suggested that this could be made a function of pep405) Anyway, keeping your executable as part of the tree is the trick I use, and to make things nice I put right next to it: > site.py > sitecustomize.py > > sitecustomize.py is where you would put the logic to set sys.path by walking up the hierarchy and finding the proper root. > site.py is there to merely import sitecustomize.py, in case a site.py is not found in all the default places python looks. > > K > > >> -----Original Message----- >> From: Python-Dev [mailto:python-dev- >> bounces+kristjan=ccpgames.com at python.org] On Behalf Of Christian >> bounces+Tismer >> Sent: 11. n?vember 2012 20:31 >> To: python-dev at python.org >> Subject: [Python-Dev] Setting project home path the best way >> >> Hi friends, >> >> I have a project that has its root somewhere on my machine. >> This project has many folders and contains quite some modules. >> >> There is a common root of the module tree, and I want to use >> - either absolute imports >> - relative imports with '.' >> >> Problem: >> >> - I want to run any module inside the heirarchy from the command-line >> >> - this should work, regardless what my 'cwd' is >> >> - this should work with or without virtualenv. >> >> So far, things work fine with virtualenv, because sys.executable is >> in the project module tree. >> >> Without virtualenv, this is not so. But I hate to make settings like >> PYTHONPATH, because these are not permanent. . >> >> Question: >> >> How should I define my project root dir in a unique way, without >> setting an environment variable? >> What is the lest intrusive way to spell that? >> >> Reason: >> >> I'd like to make things work correctly and unambigously when I call a >> script inside the module heirarchy. Things are not fixed: there exist >> many checkouts In the file system, and each should know where to >> search its home/root in the tree. >> >> Is this elegantly possible to deduce from the actually executed script file? >> >> Cheers - chris >> >> Sent from my Ei4Steve >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python- >> dev/kristjan%40ccpgames.com > > <winmail.dat> From ncoghlan at gmail.com Sun Nov 18 12:24:49 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Nov 2012 21:24:49 +1000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> On Sun, Nov 18, 2012 at 8:54 PM, Kristj?n Valur J?nsson < kristjan at ccpgames.com> wrote: > I don't have a well formed solution in mind, but I would see it desirable > to have a way for someone to release his package with all its dependencies > as a self-contained and isolated unit. E.g. if package foo.py relies on > functionality from version 1.7 of bar.py, then that functionality could be > bottled up for foo?s exclusive usage. > Another package, baz.py, could then also make use of bar, but version 1.8. > The two bar versions would be isolated. > > Perhaps this is just a pipedream. Even unpossible. But it doesn't harm > to try to think about better ways to do things. > > Easily bundling dependencies is a key principle behind the ability to execute directories and zipfiles that contain a top level __main__.py file that was added back in 2.6 (although the zipfile version doesn't play nicely with extension modules). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121118/e0da2c51/attachment.html> From kristjan at ccpgames.com Sun Nov 18 12:18:45 2012 From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=) Date: Sun, 18 Nov 2012 11:18:45 +0000 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DE0CF6@RKV-IT-EXCH103.ccp.ad.local> Interesting work indeed. From profiling CPython it has long been clear to me that enormous gains can be made by making instruction dispatching faster. A huge amount of time is spent in the evaluation loop. I have also been making small inroads to offline bytecode optimization. Identifying common patterns and introducing special opcodes to deal with them. Obviously using register addressing makes such an approach more effective. (Working with code objects is fun and exciting, btw, and the reason for my patch http://bugs.python.org/issue16475) K From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Victor Stinner Sent: 17. n?vember 2012 01:13 To: Python Dev Subject: [Python-Dev] Register-based VM for CPython The WPython project is similar to my work (except that it does not use registers). It tries also to reduce the overhead of instruction dispatch by using more complex instructions. http://code.google.com/p/wpython/ Using registers instead of a stack allow to implement more optimizations (than WPython). For example, it's possible to load constants outside loops and merge "duplicate constant loads". I also implemented more aggressive and experimental optimizations (disabled by default) which may break applications: move loads of attributes and globals outside of loops, and replace binary operations with inplace operations. For example, "x=[]; for ...: x.append(...)" is optimized to something like "x=[]; x_append=x.append; for ...: x_append(...)", and "x = x + 1" is replaced with "x += 1". Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121118/04c05c17/attachment.html> From solipsis at pitrou.net Sun Nov 18 12:41:56 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Nov 2012 12:41:56 +0100 Subject: [Python-Dev] Register-based VM for CPython References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> Message-ID: <20121118124156.1e2651a0@pitrou.net> On Sat, 17 Nov 2012 11:17:40 +0100 Armin Rigo <arigo at tunes.org> wrote: > Hi Victor, > > On Sat, Nov 17, 2012 at 2:13 AM, Victor Stinner > <victor.stinner at gmail.com> wrote: > > The major drawback of the register approach (at least of my implementation) > > is that it changes the lifetime of objects. Newly created objects are only > > "destroyed" at the exit of the function, whereas the stack-based VM destroys > > "immediatly" objects (thanks to the reference counter). PyPy has similar > > issues with its different garbage collector. > > That is not strictly correct. PyPy, Jython and IronPython have > non-immediate destructors, but as far as I can tell they all avoid to > keep objects alive for an unbounded amount of time. This important > difference is visible if the function calls other code that takes a > long while to run: in your approach, the objects created by the > function itself will stay alive for the whole duration, while the > other interpreters will all release them soon after they are not > referenced any more --- not instantly like CPython but still soon. Agreed with Armin. Also, I would point out that the reference counting behaviour is an important feature of *C*Python (to the point that we have test cases checking against reference cycles), so we can't break it nilly-willy. Regards Antoine. From martin at v.loewis.de Sun Nov 18 13:18:43 2012 From: martin at v.loewis.de (martin at v.loewis.de) Date: Sun, 18 Nov 2012 13:18:43 +0100 Subject: [Python-Dev] externals? In-Reply-To: <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> Message-ID: <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> Zitat von Armin Rigo <arigo at tunes.org>: > Hi, > > On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl <g.brandl at gmx.net> wrote: >> One way would be to use one hg repo per version, and (maybe, if needed) >> a master repo that has them as subrepos. > > Or have all versions in the same repo as usual (with branches), but > have hg subrepos point to different repos: ones extracted from the > main repo by containing only the correct branch. But it might be a > bit delicate to pull this off. (hg clone takes a "-r" option and > copies only things needed for the given revision or branch, but > apparently we can't pass this option automatically to the cloning of > subrepos. (Maybe it points out that subrepos are a hack best done > without altogether, which is what we did in pypy.)) I'd like to stress that we don't need any versioning here. wget and tar would be sufficient, except that it's Windows, so we have neither wget nor tar. However, including a PowerShell script may be an option; most developers will have PowerShell already on their system. AFAICT, PowerShell can do HTTP downloads and extract zip files. Regards, Martin From storchaka at gmail.com Sun Nov 18 14:58:33 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 18 Nov 2012 15:58:33 +0200 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> Message-ID: <k8apia$28k$1@ger.gmane.org> On 17.11.12 03:13, Victor Stinner wrote: > The major drawback of the register approach (at least of my > implementation) is that it changes the lifetime of objects. Newly > created objects are only "destroyed" at the exit of the function, > whereas the stack-based VM destroys "immediatly" objects (thanks to the > reference counter). It should not be a problem. Just register instructions should clear input registers if they are not binded to named local variables. I.e. "a = b + c * d" should be compiled to: BINARY_MUL_REG R1, 'c', 'd' BINARY_ADD_REG 'a', 'b', R1 # R1 cleared From benjamin at python.org Sun Nov 18 15:37:57 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 Nov 2012 09:37:57 -0500 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <20121118124156.1e2651a0@pitrou.net> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> <20121118124156.1e2651a0@pitrou.net> Message-ID: <CAPZV6o-BFQmA_+O_s+10cbnzk5pocS2agAajf19AEi6d18pQ0A@mail.gmail.com> 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: > Also, I would point out that the reference counting behaviour is an > important feature of *C*Python (to the point that we have test cases > checking against reference cycles), so we can't break it nilly-willy. The tests about reference cycles are just tests that garbage is collected, a Python language feature. Those aren't technically CPython specific. -- Regards, Benjamin From solipsis at pitrou.net Sun Nov 18 16:11:34 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Nov 2012 16:11:34 +0100 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <CAPZV6o-BFQmA_+O_s+10cbnzk5pocS2agAajf19AEi6d18pQ0A@mail.gmail.com> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> <20121118124156.1e2651a0@pitrou.net> <CAPZV6o-BFQmA_+O_s+10cbnzk5pocS2agAajf19AEi6d18pQ0A@mail.gmail.com> Message-ID: <20121118161134.2e7de294@pitrou.net> On Sun, 18 Nov 2012 09:37:57 -0500 Benjamin Peterson <benjamin at python.org> wrote: > 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: > > Also, I would point out that the reference counting behaviour is an > > important feature of *C*Python (to the point that we have test cases > > checking against reference cycles), so we can't break it nilly-willy. > > The tests about reference cycles are just tests that garbage is > collected, a Python language feature. Those aren't technically CPython > specific. We do have tests which check reference cycles are not created. Regards Antoine. From benjamin at python.org Sun Nov 18 17:27:32 2012 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 18 Nov 2012 11:27:32 -0500 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <20121118161134.2e7de294@pitrou.net> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> <20121118124156.1e2651a0@pitrou.net> <CAPZV6o-BFQmA_+O_s+10cbnzk5pocS2agAajf19AEi6d18pQ0A@mail.gmail.com> <20121118161134.2e7de294@pitrou.net> Message-ID: <CAPZV6o9u+EYHMJDYMWw3wF=f5w8sHC6qwDmNp21KkUyBYJbvRw@mail.gmail.com> 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: > On Sun, 18 Nov 2012 09:37:57 -0500 > Benjamin Peterson <benjamin at python.org> wrote: > >> 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: >> > Also, I would point out that the reference counting behaviour is an >> > important feature of *C*Python (to the point that we have test cases >> > checking against reference cycles), so we can't break it nilly-willy. >> >> The tests about reference cycles are just tests that garbage is >> collected, a Python language feature. Those aren't technically CPython >> specific. > > We do have tests which check reference cycles are not created. Oh, those. Those are implementation details. -- Regards, Benjamin From brian at python.org Sun Nov 18 18:05:45 2012 From: brian at python.org (Brian Curtin) Date: Sun, 18 Nov 2012 11:05:45 -0600 Subject: [Python-Dev] externals? In-Reply-To: <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> Message-ID: <CAD+XWwrf_Qedv+_+-T+Y=Wj9waY5ud0dFX-L8Nu9aZ+QCLvVQg@mail.gmail.com> On Sun, Nov 18, 2012 at 6:18 AM, <martin at v.loewis.de> wrote: > > Zitat von Armin Rigo <arigo at tunes.org>: > > >> Hi, >> >> On Sun, Nov 18, 2012 at 8:22 AM, Georg Brandl <g.brandl at gmx.net> wrote: >>> >>> One way would be to use one hg repo per version, and (maybe, if needed) >>> a master repo that has them as subrepos. >> >> >> Or have all versions in the same repo as usual (with branches), but >> have hg subrepos point to different repos: ones extracted from the >> main repo by containing only the correct branch. But it might be a >> bit delicate to pull this off. (hg clone takes a "-r" option and >> copies only things needed for the given revision or branch, but >> apparently we can't pass this option automatically to the cloning of >> subrepos. (Maybe it points out that subrepos are a hack best done >> without altogether, which is what we did in pypy.)) > > > I'd like to stress that we don't need any versioning here. wget and > tar would be sufficient, except that it's Windows, so we have neither > wget nor tar. However, including a PowerShell script may be an option; > most developers will have PowerShell already on their system. AFAICT, > PowerShell can do HTTP downloads and extract zip files. I would hope we can just write a simple Python script to do this, rather than require PowerShell. I'm 99.99999% certain anyone building Python on Windows will already have a version of Python installed. Plus, they're going to need it anyway to build OpenSSL (see PCbuild/build_ssl.py and the references to it in VS projects). From solipsis at pitrou.net Sun Nov 18 18:31:49 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 18 Nov 2012 18:31:49 +0100 Subject: [Python-Dev] Register-based VM for CPython In-Reply-To: <CAPZV6o9u+EYHMJDYMWw3wF=f5w8sHC6qwDmNp21KkUyBYJbvRw@mail.gmail.com> References: <CAMpsgwaONSm_BGAB3hjrSde=yJjGEWH2Cw5r-t01vaMnz6i=dQ@mail.gmail.com> <CAMSv6X2hXLwB8NOSgONbVve2D1rMSXjEPmFWJVesrc4uaDxRYw@mail.gmail.com> <20121118124156.1e2651a0@pitrou.net> <CAPZV6o-BFQmA_+O_s+10cbnzk5pocS2agAajf19AEi6d18pQ0A@mail.gmail.com> <20121118161134.2e7de294@pitrou.net> <CAPZV6o9u+EYHMJDYMWw3wF=f5w8sHC6qwDmNp21KkUyBYJbvRw@mail.gmail.com> Message-ID: <20121118183149.7625db59@pitrou.net> On Sun, 18 Nov 2012 11:27:32 -0500 Benjamin Peterson <benjamin at python.org> wrote: > 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: > > On Sun, 18 Nov 2012 09:37:57 -0500 > > Benjamin Peterson <benjamin at python.org> wrote: > > > >> 2012/11/18 Antoine Pitrou <solipsis at pitrou.net>: > >> > Also, I would point out that the reference counting behaviour is an > >> > important feature of *C*Python (to the point that we have test cases > >> > checking against reference cycles), so we can't break it nilly-willy. > >> > >> The tests about reference cycles are just tests that garbage is > >> collected, a Python language feature. Those aren't technically CPython > >> specific. > > > > We do have tests which check reference cycles are not created. > > Oh, those. Those are implementation details. At this point I'm not sure which statement you are arguing against. Regards Antoine. From tjreedy at udel.edu Sun Nov 18 20:12:12 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 18 Nov 2012 14:12:12 -0500 Subject: [Python-Dev] externals? In-Reply-To: <CAD+XWwrf_Qedv+_+-T+Y=Wj9waY5ud0dFX-L8Nu9aZ+QCLvVQg@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> <CAD+XWwrf_Qedv+_+-T+Y=Wj9waY5ud0dFX-L8Nu9aZ+QCLvVQg@mail.gmail.com> Message-ID: <k8bbuf$j4b$1@ger.gmane.org> On 11/18/2012 12:05 PM, Brian Curtin wrote: > On Sun, Nov 18, 2012 at 6:18 AM, <martin at v.loewis.de> wrote: >> >> Zitat von Armin Rigo <arigo at tunes.org>: >>> Or have all versions in the same repo as usual (with branches), but >>> have hg subrepos point to different repos: ones extracted from the >>> main repo by containing only the correct branch. But it might be a >>> bit delicate to pull this off. (hg clone takes a "-r" option and >>> copies only things needed for the given revision or branch, but >>> apparently we can't pass this option automatically to the cloning of >>> subrepos. (Maybe it points out that subrepos are a hack best done >>> without altogether, which is what we did in pypy.)) >> >> >> I'd like to stress that we don't need any versioning here. wget and >> tar would be sufficient, except that it's Windows, so we have neither >> wget nor tar. However, including a PowerShell script may be an option; >> most developers will have PowerShell already on their system. AFAICT, >> PowerShell can do HTTP downloads and extract zip files. > > I would hope we can just write a simple Python script to do this, > rather than require PowerShell. > > I'm 99.99999% certain anyone building Python on Windows will already > have a version of Python installed. Plus, they're going to need it > anyway to build OpenSSL (see PCbuild/build_ssl.py and the references > to it in VS projects). After reading the thread, I realize that I do not actually want externam dependency files moved to hg. I and others are not going to push changes back, so we do not need hg clones. What would be good would to be able to access the files and use them to build python without svn installed. I don't know the best way to do that, but if tarred or zipped releases were made for each version that should be downloaded, our urllib, tarfile/ziplib, and any other modules needed should be sufficient to transfer and unpack. -- Terry Jan Reedy From kristjan at ccpgames.com Mon Nov 19 11:29:38 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 19 Nov 2012 10:29:38 +0000 Subject: [Python-Dev] need reviewers for #16475 and #16487 In-Reply-To: <CAP7+vJ+f+csADFCSpZFvUxhVV=X_pu5jGsHDc2yJaz1wYcuoZw@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DE035B@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJ+f+csADFCSpZFvUxhVV=X_pu5jGsHDc2yJaz1wYcuoZw@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DE1B76@RKV-IT-EXCH103.ccp.ad.local> Thank you! The sensitivity of this issue obviously is born out of our collective bad conscience for this unjust incarceration. K > -----Original Message----- > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf > Of Guido van Rossum > > > > This fixes a regression in marshal between 2.x and 3.x, reinstating > > string reuse and internment support. In addition, it generalizes > > string reuse to > > It's not internment -- that means imprisonment. The term we use is > interning. (The dictionary will tell you that means imprisonment too > -- but it's long been used as the name for this particular technique. > Internment has not.) From kristjan at ccpgames.com Mon Nov 19 11:43:03 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 19 Nov 2012 10:43:03 +0000 Subject: [Python-Dev] externals? In-Reply-To: <k8bbuf$j4b$1@ger.gmane.org> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> <CAD+XWwrf_Qedv+_+-T+Y=Wj9waY5ud0dFX-L8Nu9aZ+QCLvVQg@mail.gmail.com> <k8bbuf$j4b$1@ger.gmane.org> Message-ID: <EFE3877620384242A686D52278B7CCD329DE1BC0@RKV-IT-EXCH103.ccp.ad.local> But that's what hg clone does. You have a lorry for your work at the mine. You don't need a Mini to go to the fishmongers. You can use your lorry even if you are not going to dump a tonne of ore on the pavement. K > -----Original Message----- > > What would be good would to be able to access the files and use them to > build python without svn installed. I don't know the best way to do that, but > if tarred or zipped releases were made for each version that should be > downloaded, our urllib, tarfile/ziplib, and any other modules needed should > be sufficient to transfer and unpack. From trent at snakebite.org Mon Nov 19 12:42:31 2012 From: trent at snakebite.org (Trent Nelson) Date: Mon, 19 Nov 2012 06:42:31 -0500 Subject: [Python-Dev] externals? In-Reply-To: <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> References: <EFE3877620384242A686D52278B7CCD329DD7EF4@RKV-IT-EXCH103.ccp.ad.local> <CAPZV6o8geJWfgDp3L-dOmt_6=pfgVayh6V7EJAEGr0R+=DPoyw@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DDC6E0@RKV-IT-EXCH103.ccp.ad.local> <20121116121251.GA13953@snakebite.org> <EFE3877620384242A686D52278B7CCD329DE039E@RKV-IT-EXCH103.ccp.ad.local> <20121117234757.Horde.9Jf8druWis5QqBQd8O5BnVA@webmail.df.eu> <k8a2at$tg5$1@ger.gmane.org> <CAMSv6X1PvEvV-4zBzfiUKsVwpC+5LU7=Ug36vKsjkcpHj2C=cw@mail.gmail.com> <20121118131843.Horde.8WNlTtjz9kRQqNIjgKDTaVA@webmail.df.eu> Message-ID: <20121119114126.GA31524@snakebite.org> On Sun, Nov 18, 2012 at 04:18:43AM -0800, martin at v.loewis.de wrote: > I'd like to stress that we don't need any versioning here. wget and > tar would be sufficient, except that it's Windows, so we have neither > wget nor tar. However, including a PowerShell script may be an option; > most developers will have PowerShell already on their system. AFAICT, > PowerShell can do HTTP downloads and extract zip files. I'm disturbed that I subconsciously interpreted this as a challenge to do it via a standalone batch/.bat ;-) Trent. From brian at python.org Mon Nov 19 16:01:44 2012 From: brian at python.org (Brian Curtin) Date: Mon, 19 Nov 2012 09:01:44 -0600 Subject: [Python-Dev] New Contributor Experience in Python and other FOSS Communities - A Survey Message-ID: <CAD+XWwoVUSaP1C4qsMbn05ROApkA5fAo2Fr=vXssQy=iAcJSzg@mail.gmail.com> Hi all, Along with a number of other free and open communities, Python is being included in a survey of new contributors since January 2010. The survey is being done by Kevin Carillo, a PhD student at Victoria University of Wellington who is studying various approaches that projects use to gain and work with new contributors. If you have written patches, reviewed issues, or done anything to contribute to the development of Python and you started this after January 2010, I hope you can spare around 20 minutes to complete this survey: https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151&lang=en There's a longer blog post up at http://blog.python.org/2012/11/new-contributor-experience-in-python.html if you would like a bit more information. On behalf of Kevin Carillo, I thank you for your time and consideration of this survey. Brian Curtin From dholth at gmail.com Tue Nov 20 00:53:45 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 18:53:45 -0500 Subject: [Python-Dev] Accept just PEP-0426 Message-ID: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. PEP 426 doesn't have anything to do with the Wheel PEPs 425 and 427, other than that its features are necessary to usefully represent a large number of existing Python packages. How about moving this one along to focus on the other two. I'm not sure what the Post-History should be. We have been talking about it for a while. Thanks, Daniel Holth PEP: 426 Title: Metadata for Python Software Packages 1.3 Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth <dholth at fastmail.fm> Discussions-To: Distutils SIG Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 Aug 2012 Abstract ======== This PEP describes a mechanism for adding metadata to Python distributions. It includes specifics of the field names, and their semantics and usage. This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. Version 1.3 of the metadata format adds fields designed to make third-party packaging of Python Software easier and defines a formal extension mechanism. The fields are "Setup-Requires-Dist" "Provides-Extra", and "Extension". This version also adds the `extra` variable to the `environment markers` specification and allows the description to be placed into a payload section. Metadata Files ============== The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value format with case-insensitive keys and no maximum line length, followed by a blank line and an arbitrary payload. It is parseable by the ``email`` module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. There are two standard locations for these metadata files: * the ``PKG-INFO`` file included in the base directory of Python source distribution archives (as created by the distutils ``sdist`` command) * the ``.dist-info/METADATA`` files in a Python installation database, as described in PEP 376. Other tools involved in Python distribution may also use this format. Encoding ======== Metadata 1.3 files are UTF-8 with the restriction that keys must be ASCII. Parser implementations should be aware that older versions of the Metadata specification do not specify an encoding. Fields ====== This section specifies the names and semantics of each of the supported metadata fields. In a single Metadata 1.3 file, fields marked with "(optional)" may occur 0 or 1 times. Fields marked with "(multiple use)" may be specified 0, 1 or more times. Only "Metadata-Version", "Name", "Version", and "Summary" must appear exactly once. The fields may appear in any order within the file. Metadata-Version :::::::::::::::: Version of the file format; "1.3" is the only legal value. Example:: Metadata-Version: 1.3 Name :::: The name of the distribution. Example:: Name: BeagleVote Version ::::::: A string containing the distribution's version number. This field must be in the format specified in PEP 386. Example:: Version: 1.0a2 Summary ::::::: A one-line summary of what the distribution does. Example:: Summary: A module for collecting votes from beagles. Platform (multiple use) ::::::::::::::::::::::: A Platform specification describing an operating system supported by the distribution which is not listed in the "Operating System" Trove classifiers. See "Classifier" below. Examples:: Platform: ObscureUnix Platform: RareDOS Supported-Platform (multiple use) ::::::::::::::::::::::::::::::::: Binary distributions containing a metadata file will use the Supported-Platform field in their metadata to specify the OS and CPU for which the binary distribution was compiled. The semantics of the Supported-Platform field are not specified in this PEP. Example:: Supported-Platform: RedHat 7.2 Supported-Platform: i386-win32-2791 Description (optional, deprecated) :::::::::::::::::::::::::::::::::: A longer description of the distribution that can run to several paragraphs. Software that deals with metadata should not assume any maximum size for this field. The contents of this field can be written using reStructuredText markup [1]_. For programs that work with the metadata, supporting markup is optional; programs can also display the contents of the field as-is. This means that authors should be conservative in the markup they use. Since a line separator immediately followed by another line separator indicates the end of the headers section, any line separators in the description must be suffixed by whitespace to indicate continuation. Since Metadata 1.3 the recommended place for the description is in the payload section of the document, after the last header. The description does not need to be reformatted when it is included in the payload. Keywords (optional) ::::::::::::::::::: A list of additional keywords to be used to assist searching for the distribution in a larger catalog. Example:: Keywords: dog puppy voting election Home-page (optional) :::::::::::::::::::: A string containing the URL for the distribution's home page. Example:: Home-page: http://www.example.com/~cschultz/bvote/ Download-URL (optional) ::::::::::::::::::::::: A string containing the URL from which this version of the distribution can be downloaded. (This means that the URL can't be something like ".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) Author (optional) ::::::::::::::::: A string containing the author's name at a minimum; additional contact information may be provided. Example:: Author: C. Schultz, Universal Features Syndicate, Los Angeles, CA <cschultz at peanuts.example.com> Author-email (optional) ::::::::::::::::::::::: A string containing the author's e-mail address. It contains a name and e-mail address in the RFC 5322 recommended ``Address Specification`` format. Example:: Author-email: "C. Schultz" <cschultz at example.com> Maintainer (optional) ::::::::::::::::::::: A string containing the maintainer's name at a minimum; additional contact information may be provided. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author``. Example:: Maintainer: C. Schultz, Universal Features Syndicate, Los Angeles, CA <cschultz at peanuts.example.com> Maintainer-email (optional) ::::::::::::::::::::::::::: A string containing the maintainer's e-mail address. It has the same format as ``Author-email``. Note that this field is intended for use when a project is being maintained by someone other than the original author: it should be omitted if it is identical to ``Author-email``. Example:: Maintainer-email: "C. Schultz" <cschultz at example.com> License (optional) :::::::::::::::::: Text indicating the license covering the distribution where the license is not a selection from the "License" Trove classifiers. See "Classifier" below. This field may also be used to specify a particular version of a license which is named via the ``Classifier`` field, or to indicate a variation or exception to such a license. Examples:: License: This software may only be obtained by sending the author a postcard, and then the user promises not to redistribute it. License: GPL version 3, excluding DRM provisions The full text of the license would normally be included in a separate file. Classifier (multiple use) ::::::::::::::::::::::::: Each entry is a string giving a single classification value for the distribution. Classifiers are described in PEP 301 [2]. Examples:: Classifier: Development Status :: 4 - Beta Classifier: Environment :: Console (Text Based) Requires-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming some other distutils project required by this distribution. The format of a requirement string is identical to that of a distutils project name (e.g., as found in the ``Name:`` field. optionally followed by a version declaration within parentheses. The distutils project names should correspond to names as found on the `Python Package Index`_. Version declarations must follow the rules described in `Version Specifiers`_ Examples:: Requires-Dist: pkginfo Requires-Dist: PasteDeploy Requires-Dist: zope.interface (>3.5.0) Setup-Requires-Dist (multiple use) :::::::::::::::::::::::::::::::::: Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. Commonly used to generate a manifest from version control. Examples:: Setup-Requires-Dist: custom_setup_command Dependencies mentioned in `Setup-Requires-Dist` may be installed exclusively for setup and are not guaranteed to be available at run time. Provides-Dist (multiple use) :::::::::::::::::::::::::::: Each entry contains a string naming a Distutils project which is contained within this distribution. This field *must* include the project identified in the ``Name`` field, followed by the version : Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been bundled together. For instance, source distributions of the ``ZODB`` project have historically included the ``transaction`` project, which is now available as a separate distribution. Installing such a source distribution satisfies requirements for both ``ZODB`` and ``transaction``. A distribution may also provide a "virtual" project name, which does not correspond to any separately-distributed project: such a name might be used to indicate an abstract capability which could be supplied by one of multiple projects. E.g., multiple projects might supply RDBMS bindings for use by a given ORM: each project might declare that it provides ``ORM-bindings``, allowing other projects to depend only on having at most one of them installed. A version declaration may be supplied and must follow the rules described in `Version Specifiers`_. The distribution's version number will be implied if none is specified. Examples:: Provides-Dist: OtherProject Provides-Dist: AnotherProject (3.4) Provides-Dist: virtual_package Obsoletes-Dist (multiple use) ::::::::::::::::::::::::::::: Each entry contains a string describing a distutils project's distribution which this distribution renders obsolete, meaning that the two projects should not be installed at the same time. Version declarations can be supplied. Version numbers must be in the format specified in `Version Specifiers`_. The most common use of this field will be in case a project name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon distribution should be removed. Examples:: Obsoletes-Dist: Gorgon Obsoletes-Dist: OtherProject (<3.0) Requires-Python (optional) :::::::::::::::::::::::::: This field specifies the Python version(s) that the distribution is guaranteed to be compatible with. Version numbers must be in the format specified in `Version Specifiers`_. Examples:: Requires-Python: 2.5 Requires-Python: >2.1 Requires-Python: >=2.3.4 Requires-Python: >=2.5,<2.7 Requires-External (multiple use) :::::::::::::::::::::::::::::::: Each entry contains a string describing some dependency in the system that the distribution is to be used. This field is intended to serve as a hint to downstream project maintainers, and has no semantics which are meaningful to the ``distutils`` distribution. The format of a requirement string is a name of an external dependency, optionally followed by a version declaration within parentheses. Because they refer to non-Python software releases, version numbers for this field are **not** required to conform to the format specified in PEP 386: they should correspond to the version scheme used by the external dependency. Notice that there's is no particular rule on the strings to be used. Examples:: Requires-External: C Requires-External: libpng (>=1.5) Project-URL (multiple use) :::::::::::::::::::::::::: A string containing a label and a browsable URL for the project, separated by the last occurrence of comma and space ", ". Example:: Bug, Issue Tracker, http://bitbucket.org/tarek/distribute/issues/ The label is a free text. Provides-Extra (multiple use) ::::::::::::::::::::::::::::: A string containing the name of an optional feature. Must be printable ASCII, not containing whitespace, comma (,), or square brackets []. May be used to make a dependency conditional on whether the optional feature has been requested. Example:: Name: beaglevote Provides-Extra: pdf Requires-Dist: reportlab; extra == 'pdf' Requires-Dist: nose; extra == 'test' Requires-Dist: sphinx; extra == 'doc' A second distribution requires an optional dependency by placing it inside square brackets and can request multiple features by separating them with a comma (,). The full set of requirements is the union of the `Requires-Dist` sets evaluated with `extra` set to `None` and then to the name of each requested feature. Example:: Requires-Dist: beaglevote[pdf] -> requires beaglevote, reportlab Requires-Dist: beaglevote[test, doc] -> requires beaglevote, sphinx, nose Two feature names `test` and `doc` are reserved to mark dependencies that are needed for running automated tests and generating documentation, respectively. It is legal to specify `Provides-Extra` without referencing it in any `Requires-Dist`. It is an error to request a feature name that has not been declared with `Provides-Extra`. Extension (multiple use) :::::::::::::::::::::::: An ASCII string, not containing whitespace or the / character, that indicates the presence of extended metadata. Additional tags defined by an `Extension: Chili` must be of the form `Chili/Name`:: Extension: Chili Chili/Type: Poblano Chili/Heat: Mild An implementation might iterate over all the declared `Extension:` fields to invoke the processors for those extensions. As the order of the fields is not used, the `Extension: Chili` field may appear before or after its declared tags `Chili/Type:` etc. Version Specifiers ================== Version specifiers are a series of conditional operators and version numbers, separated by commas. Conditional operators must be one of "<", ">", "<=", ">=", "==" and "!=". Any number of conditional operators can be specified, e.g. the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. The comma (",") is equivalent to the **and** operator. Each version number must be in the format specified in PEP 386. When a version is provided, it always includes all versions that starts with the same value. For example the "2.5" version of Python will include versions like "2.5.2" or "2.5.3". Pre and post releases in that case are excluded. So in our example, versions like "2.5a1" are not included when "2.5" is used. If the first version of the range is required, it has to be explicitly given. In our example, it will be "2.5.0". Notice that some projects might omit the ".0" prefix for the first release of the "2.5.x" series: - 2.5 - 2.5.1 - 2.5.2 - etc. In that case, "2.5.0" will have to be explicitly used to avoid any confusion between the "2.5" notation that represents the full range. It is a recommended practice to use schemes of the same length for a series to completely avoid this problem. Some Examples: - ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, excluding post or pre-releases. - ``Requires-Dist: zope.interface (==3.1)``: equivalent to ``Requires-Dist: zope.interface (3.1)``. - ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with 3.1.0, excluding post or pre-releases. Since that particular project doesn't use more than 3 digits, it also means "only the 3.1.0 release". - ``Requires-Python: 3``: Any Python 3 version, no matter wich one, excluding post or pre-releases. - ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including post releases of 2.6, pre and post releases of 2.7. It excludes pre releases of Python 3. - ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would have include all versions of the 2.6.2 series. - ``Requires-Python: 2.5.0``: Equivalent to ">=2.5.0,<2.5.1". - ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with 3.1, excluding post or pre-releases of 3.1 *and* excluding any version that starts with "3.1.3". For this particular project, this means: "any version of the 3.1 series but not 3.1.3". This is equivalent to: ">=3.1,!=3.1.3,<3.2". Environment markers =================== An **environment marker** is a marker that can be added at the end of a field after a semi-colon (";"), to add a condition about the execution environment. Here are some example of fields using such markers:: Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform The micro-language behind this is a simple subset of Python: it compares only strings, with the ``==`` and ``in`` operators (and their opposites), and with the ability to combine expressions. Parenthesis are supported for grouping. The pseudo-grammar is :: EXPR [in|==|!=|not in] EXPR [or|and] ... where ``EXPR`` belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - python_full_version = sys.version.split()[0] - os.name = os.name - sys.platform = sys.platform - platform.version = platform.version() - platform.machine = platform.machine() - platform.python_implementation = platform.python_implementation() - a free string, like ``'2.4'``, or ``'win32'`` - extra = (name of requested feature) or None Notice that ``in`` is restricted to strings, meaning that it is not possible to use other sequences like tuples or lists on the right side. The fields that benefit from this marker are: - Requires-Python - Requires-External - Requires-Dist - Setup-Requires-Dist - Provides-Dist - Obsoletes-Dist - Classifier (The `extra` variable is only meaningful for Requires-Dist.) Summary of Differences From PEP 345 =================================== * Metadata-Version is now 1.3. * Values are now expected to be UTF-8. * A payload (containing the description) may appear after the headers. * Added `extra` to environment markers. * Most fields are now optional. * Changed fields: - Description - Project-URL - Requires-Dist * Added fields: - Extension - Provides-Extra - Setup-Requires-Dist References ========== This document specifies version 1.3 of the metadata format. Version 1.0 is specified in PEP 241. Version 1.1 is specified in PEP 314. Version 1.2 is specified in PEP 345. .. [1] reStructuredText markup: http://docutils.sourceforge.net/ .. _`Python Package Index`: http://pypi.python.org/pypi/ .. [2] PEP 301: http://www.python.org/dev/peps/pep-0301/ Appendix ======== Parsing and generating the Metadata 1.3 serialization format using Python 3.3:: # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/bed97c73/attachment-0001.html> From pje at telecommunity.com Tue Nov 20 01:37:52 2012 From: pje at telecommunity.com (PJ Eby) Date: Mon, 19 Nov 2012 19:37:52 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> Message-ID: <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: > I think this PEP is a significant improvement from its predecessor. It > represents features like extras (provides-extra) and build requirements > (setup-requires-dist) that are in use in the Python community but cannot be > represented in older versions of the format, it finally specifies a UTF-8 > encoding, removes RFC 822, provides an extension mechanism, and allows the > description to be placed in the document payload. > Can we maybe kill Provides-Dist and its associated baggage first, though? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/41f4efdf/attachment.html> From donald.stufft at gmail.com Tue Nov 20 01:41:56 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 19:41:56 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> Message-ID: <B8924F82A9A24F04A0B8DC4C28CAC9E6@gmail.com> On Monday, November 19, 2012 at 7:37 PM, PJ Eby wrote: > Can we maybe kill Provides-Dist and its associated baggage first, though? > > I would love to kill Provides-Dist. The biggest question there is how do you handle it's functionality? If someone needs setuptools but they have distribute installed they both shouldn't get installed. The need for it for the "2 packages being distributed together" I'm (personally) less concerned about since with proper dependency data we should be able to just depend on things instead of bundling them. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/8bd03665/attachment.html> From dholth at gmail.com Tue Nov 20 01:43:37 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 19:43:37 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> Message-ID: <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects? Daniel Holth On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com> wrote: > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: >> I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. > > Can we maybe kill Provides-Dist and its associated baggage first, though? > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/61693074/attachment.html> From dholth at gmail.com Tue Nov 20 01:46:41 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 19:46:41 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <B8924F82A9A24F04A0B8DC4C28CAC9E6@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <B8924F82A9A24F04A0B8DC4C28CAC9E6@gmail.com> Message-ID: <D5207188-D525-4049-88B0-85E32BD32973@gmail.com> The "I bundled a renamed copy of six" is a totally different case which would not invoke provides-dist. "I merged sqlalchemy with a previously separate but wildly popular declarative / database support / whatever extension" would invoke provides-dist. Daniel Holth On Nov 19, 2012, at 7:41 PM, Donald Stufft <donald.stufft at gmail.com> wrote: > On Monday, November 19, 2012 at 7:37 PM, PJ Eby wrote: >> Can we maybe kill Provides-Dist and its associated baggage first, though? > I would love to kill Provides-Dist. The biggest question there is how do you handle it's > functionality? If someone needs setuptools but they have distribute installed they > both shouldn't get installed. > > The need for it for the "2 packages being distributed together" I'm (personally) > less concerned about since with proper dependency data we should be > able to just depend on things instead of bundling them. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/82e91941/attachment.html> From donald.stufft at gmail.com Tue Nov 20 01:49:41 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 19:49:41 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> Message-ID: <4AB62E69A53048589648EA9A94CAC02A@gmail.com> Other languages seem to get along fine without it. Even the OS managers which have it don't allow it to be used to masquerade as another project, only to make generic virtual packages (e.g. "email"). On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects? > > Daniel Holth > > On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com (mailto:pje at telecommunity.com)> wrote: > > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com (mailto:dholth at gmail.com)> wrote: > > > I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. > > > > Can we maybe kill Provides-Dist and its associated baggage first, though? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org (mailto:Python-Dev at python.org) > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/03ab0f9b/attachment.html> From dholth at gmail.com Tue Nov 20 02:08:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 20:08:31 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <4AB62E69A53048589648EA9A94CAC02A@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> Message-ID: <9317AC70-4833-475C-924D-768CA0E7A069@gmail.com> The distro repos are centrally managed, too. Try getting setuptools to provide a virtual package just because you want to fork.. and then update the dependent packages? Daniel Holth On Nov 19, 2012, at 7:49 PM, Donald Stufft <donald.stufft at gmail.com> wrote: > Other languages seem to get along fine without it. Even the OS > managers which have it don't allow it to be used to masquerade > as another project, only to make generic virtual packages (e.g. "email"). > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > >> Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects? >> >> Daniel Holth >> >> On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com> wrote: >> >>> On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: >>>> I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. >>> >>> Can we maybe kill Provides-Dist and its associated baggage first, though? >>> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/560ea2e7/attachment.html> From donald.stufft at gmail.com Tue Nov 20 02:14:47 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 20:14:47 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <9317AC70-4833-475C-924D-768CA0E7A069@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <9317AC70-4833-475C-924D-768CA0E7A069@gmail.com> Message-ID: <DE944FC1D4AD4C419E450D2A275B3566@gmail.com> On Monday, November 19, 2012 at 8:08 PM, Daniel Holth wrote: > The distro repos are centrally managed, too. Try getting setuptools to provide a virtual package just because you want to fork.. and then update the dependent packages? > > Daniel Holth Sorry didn't mean to make it sound like I thought we should do it like the OS packagers, just mentioning that very few (any?) other languages do it like that and they seem to be getting along just fine. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/b37c3ce7/attachment.html> From dholth at gmail.com Tue Nov 20 02:31:01 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 20:31:01 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <4AB62E69A53048589648EA9A94CAC02A@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> Message-ID: <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> We are getting along fine too. No tool parses metadata 1.x for package management reasons and provides has existed forever with no implementation. So it is not inconveniencing anyone. I would prefer to leave it alone. Daniel Holth On Nov 19, 2012, at 7:49 PM, Donald Stufft <donald.stufft at gmail.com> wrote: > Other languages seem to get along fine without it. Even the OS > managers which have it don't allow it to be used to masquerade > as another project, only to make generic virtual packages (e.g. "email"). > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > >> Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects? >> >> Daniel Holth >> >> On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com> wrote: >> >>> On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: >>>> I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. >>> >>> Can we maybe kill Provides-Dist and its associated baggage first, though? >>> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/287e5b65/attachment.html> From donald.stufft at gmail.com Tue Nov 20 02:33:19 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 20:33:19 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> Message-ID: <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> So you want to leave metadata in that you think people shouldn't implement? Or you do think people should implement it and the point about it existing forever without an implementation is? At the very least there needs to be some sort of guidelines as to what to do with the field in the various states it could be in. On Monday, November 19, 2012 at 8:31 PM, Daniel Holth wrote: > We are getting along fine too. No tool parses metadata 1.x for package management reasons and provides has existed forever with no implementation. So it is not inconveniencing anyone. I would prefer to leave it alone. > > Daniel Holth > > On Nov 19, 2012, at 7:49 PM, Donald Stufft <donald.stufft at gmail.com (mailto:donald.stufft at gmail.com)> wrote: > > > Other languages seem to get along fine without it. Even the OS > > managers which have it don't allow it to be used to masquerade > > as another project, only to make generic virtual packages (e.g. "email"). > > > > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > > > > > Does it really have baggage? I think it is necessary, although it doesn't do favors to the implementer (and has never been implemented). How else is anyone supposed to fork or merge projects? > > > > > > Daniel Holth > > > > > > On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com (mailto:pje at telecommunity.com)> wrote: > > > > > > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com (mailto:dholth at gmail.com)> wrote: > > > > > I think this PEP is a significant improvement from its predecessor. It represents features like extras (provides-extra) and build requirements (setup-requires-dist) that are in use in the Python community but cannot be represented in older versions of the format, it finally specifies a UTF-8 encoding, removes RFC 822, provides an extension mechanism, and allows the description to be placed in the document payload. > > > > > > > > Can we maybe kill Provides-Dist and its associated baggage first, though? > > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org (mailto:Python-Dev at python.org) > > > http://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > > > > > > > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/5f931f92/attachment-0001.html> From a.badger at gmail.com Tue Nov 20 02:35:50 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Mon, 19 Nov 2012 17:35:50 -0800 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <4AB62E69A53048589648EA9A94CAC02A@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> Message-ID: <20121120013550.GR2133@unaka.lan> On Mon, Nov 19, 2012 at 07:49:41PM -0500, Donald Stufft wrote: > Other languages seem to get along fine without it. Even the OS > managers which have it don't allow it to be used to masquerade > as another project, only to make generic virtual packages (e.g. "email"). > I'm not sure this assertion about OS package managers is correct. I've only just read: http://www.python.org/dev/peps/pep-0426/#provides-dist-multiple-use but the rough rpm analogue seems to be the Provides: tag. Provides is given a string which is parsed into a name or a name and version like this: Provides: python Provides: python = 3.1.0 rpm has no way at package build time to tell that a particular name given in a provides in one package is the actual name of another package. At installtime, rpm keeps package names and provides names separately but in dependency comparisons either one can be used to satisfy a requirement. What that means is that when asking about information on a package with name "python", you'll get information about the python package with that name and not about anything else that Provides: "python". But if you are installing something that has a requirement on "python" either the package with the name python or any package that Provides: python can satisfy the requirement. Package managers with builtin dep solvers can be built on top of rpm. The one that I am familiar with is yum. Since yum is downloading the packages that are being fed into rpm, yum could choose to prefer the package name instead of things in Provides when it downloads. It doesn't, though. Just like the underlying rpm, it treats package names and names specificed through Provides: as equivalent. -Toshio > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > > Does it really have baggage? I think it is necessary, although it doesn't > do favors to the implementer (and has never been implemented). How else is > anyone supposed to fork or merge projects? > > Daniel Holth > > On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com> wrote: > > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: > > I think this PEP is a significant improvement from its predecessor. > It represents features like extras (provides-extra) and build > requirements (setup-requires-dist) that are in use in the Python > community but cannot be represented in older versions of the > format, it finally specifies a UTF-8 encoding, removes RFC 822, > provides an extension mechanism, and allows the description to be > placed in the document payload. > > > Can we maybe kill Provides-Dist and its associated baggage first, > though? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > donald.stufft%40gmail.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/d7bbc53a/attachment.pgp> From donald.stufft at gmail.com Tue Nov 20 02:40:14 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 20:40:14 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <20121120013550.GR2133@unaka.lan> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <20121120013550.GR2133@unaka.lan> Message-ID: <D187BCC00CDA42F7B03228AA208C7B7F@gmail.com> On Monday, November 19, 2012 at 8:35 PM, Toshio Kuratomi wrote: > On Mon, Nov 19, 2012 at 07:49:41PM -0500, Donald Stufft wrote: > > Other languages seem to get along fine without it. Even the OS > > managers which have it don't allow it to be used to masquerade > > as another project, only to make generic virtual packages (e.g. "email"). > > > > I'm not sure this assertion about OS package managers is correct. I've only > just read: > http://www.python.org/dev/peps/pep-0426/#provides-dist-multiple-use > > but the rough rpm analogue seems to be the Provides: tag. > > Provides is given a string which is parsed into a name or a name and version > like this: > > Provides: python > Provides: python = 3.1.0 > > rpm has no way at package build time to tell that a particular name given in > a provides in one package is the actual name of another package. > > At installtime, rpm keeps package names and provides names separately but in > dependency comparisons either one can be used to satisfy a requirement. > What that means is that when asking about information on a package with name > "python", you'll get information about the python package with that name and > not about anything else that Provides: "python". But if you are installing > something that has a requirement on "python" either the package with the > name python or any package that Provides: python can satisfy the requirement. > Are you saying the RPM documentation is wrong? http://www.rpm.org/max-rpm/s1-rpm-inside-tags.html The provides tag is used to specify a *virtual package* that the packaged software makes available when it is installed. Normally, this tag would be used when different packages provide equivalent services. For example, any package that allows a user to read mail might provide the mail-reader virtual package. Another package that depends on a mail reader of some sort, could require the mail-reader virtual package. It would then install without dependency problems, if any one of several mail programs were installed. It pretty clearly states that it is not to be used for masquerading as a different package, which was my point. I wasn't making any claims about wether it was technically possible to do so or not, just what it's intended purpose was. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/03a38ab7/attachment.html> From turnbull at sk.tsukuba.ac.jp Tue Nov 20 02:58:12 2012 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 20 Nov 2012 10:58:12 +0900 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> Message-ID: <871ufp6nnv.fsf@uwakimon.sk.tsukuba.ac.jp> Daniel Holth writes: > Does [Provides-Dist] really have baggage? I think it is necessary, > although it doesn't do favors to the implementer (and has never > been implemented). How else is anyone supposed to fork or merge > projects? It doesn't bother me personally if this traffic is on python-dev, but this looks rather technical. Shouldn't the distutils-sig at least be CC'd? Steve From ncoghlan at gmail.com Tue Nov 20 03:01:58 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Nov 2012 12:01:58 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <D187BCC00CDA42F7B03228AA208C7B7F@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <20121120013550.GR2133@unaka.lan> <D187BCC00CDA42F7B03228AA208C7B7F@gmail.com> Message-ID: <CADiSq7cswEfqRtPX6Q_-mrWFCsdR8ykaociXEiFVjZYbcc=4_g@mail.gmail.com> Look more closely at the docs for "Obsoletes" in RPM, not just those for "Provides". Being able to transparently replace an existing package with a renamed one that installs files with the same names is certainly part of the purpose/capabilities of the RPM dependency machinery (i.e. precisely the distribute vs setuptools situation). We may want to clarify the wording to ensure it is clear that the provision of the dist name (as posted on PyPI) is implied, though. Cheers, Nick. -- Sent from my phone, thus the relative brevity :) On Nov 20, 2012 11:45 AM, "Donald Stufft" <donald.stufft at gmail.com> wrote: > On Monday, November 19, 2012 at 8:35 PM, Toshio Kuratomi wrote: > > On Mon, Nov 19, 2012 at 07:49:41PM -0500, Donald Stufft wrote: > > Other languages seem to get along fine without it. Even the OS > managers which have it don't allow it to be used to masquerade > as another project, only to make generic virtual packages (e.g. "email"). > > I'm not sure this assertion about OS package managers is correct. I've only > just read: > http://www.python.org/dev/peps/pep-0426/#provides-dist-multiple-use > > but the rough rpm analogue seems to be the Provides: tag. > > Provides is given a string which is parsed into a name or a name and > version > like this: > > Provides: python > Provides: python = 3.1.0 > > rpm has no way at package build time to tell that a particular name given > in > a provides in one package is the actual name of another package. > > At installtime, rpm keeps package names and provides names separately but > in > dependency comparisons either one can be used to satisfy a requirement. > What that means is that when asking about information on a package with > name > "python", you'll get information about the python package with that name > and > not about anything else that Provides: "python". But if you are installing > something that has a requirement on "python" either the package with the > name python or any package that Provides: python can satisfy the > requirement. > > Are you saying the RPM documentation is wrong? > > http://www.rpm.org/max-rpm/s1-rpm-inside-tags.html > > The provides tag is used to specify a *virtual package* that the packaged > software makes available when it is installed. Normally, this tag would be > used when different packages provide equivalent services. For example, > any package that allows a user to read mail might provide the mail-reader > virtual package. Another package that depends on a mail reader of some > sort, could require the mail-reader virtual package. It would then install > without dependency problems, if any one of several mail programs were > installed. > > It pretty clearly states that it is not to be used for masquerading as a > different package, which was my point. I wasn't making any claims about > wether it was technically possible to do so or not, just what it's intended > purpose was. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/62834995/attachment.html> From donald.stufft at gmail.com Tue Nov 20 03:04:51 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 21:04:51 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CADiSq7cswEfqRtPX6Q_-mrWFCsdR8ykaociXEiFVjZYbcc=4_g@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <20121120013550.GR2133@unaka.lan> <D187BCC00CDA42F7B03228AA208C7B7F@gmail.com> <CADiSq7cswEfqRtPX6Q_-mrWFCsdR8ykaociXEiFVjZYbcc=4_g@mail.gmail.com> Message-ID: <31346DA2FFF7423BA7F7D21286F1DD65@gmail.com> On Monday, November 19, 2012 at 9:01 PM, Nick Coghlan wrote: > Look more closely at the docs for "Obsoletes" in RPM, not just those for "Provides". Being able to transparently replace an existing package with a renamed one that installs files with the same names is certainly part of the purpose/capabilities of the RPM dependency machinery (i.e. precisely the distribute vs setuptools situation). > We may want to clarify the wording to ensure it is clear that the provision of the dist name (as posted on PyPI) is implied, though. > Cheers, > Nick. I'm not sure if you're responding to me or not, but we also have Obsoletes-Dist in the metadata. (Which I don't like the name of but that's for another argument). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/634c75db/attachment.html> From dholth at gmail.com Tue Nov 20 03:24:31 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 21:24:31 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> Message-ID: <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> Mostly it seems a bit silly to have so much conversations about parts of the pep that remain unchanged from previously accepted versions... On Nov 19, 2012 8:33 PM, "Donald Stufft" <donald.stufft at gmail.com> wrote: > So you want to leave metadata in that you think people shouldn't > implement? > > Or you do think people should implement it and the point about it existing > forever without an implementation is? > > At the very least there needs to be some sort of guidelines as to what > to do with the field in the various states it could be in. > > On Monday, November 19, 2012 at 8:31 PM, Daniel Holth wrote: > > We are getting along fine too. No tool parses metadata 1.x for package > management reasons and provides has existed forever with no implementation. > So it is not inconveniencing anyone. I would prefer to leave it alone. > > Daniel Holth > > On Nov 19, 2012, at 7:49 PM, Donald Stufft <donald.stufft at gmail.com> > wrote: > > Other languages seem to get along fine without it. Even the OS > managers which have it don't allow it to be used to masquerade > as another project, only to make generic virtual packages (e.g. "email"). > > On Monday, November 19, 2012 at 7:43 PM, Daniel Holth wrote: > > Does it really have baggage? I think it is necessary, although it doesn't > do favors to the implementer (and has never been implemented). How else is > anyone supposed to fork or merge projects? > > Daniel Holth > > On Nov 19, 2012, at 7:37 PM, PJ Eby <pje at telecommunity.com> wrote: > > On Mon, Nov 19, 2012 at 6:53 PM, Daniel Holth <dholth at gmail.com> wrote: > > I think this PEP is a significant improvement from its predecessor. It > represents features like extras (provides-extra) and build requirements > (setup-requires-dist) that are in use in the Python community but cannot be > represented in older versions of the format, it finally specifies a UTF-8 > encoding, removes RFC 822, provides an extension mechanism, and allows the > description to be placed in the document payload. > > > Can we maybe kill Provides-Dist and its associated baggage first, though? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/7e223ea1/attachment.html> From donald.stufft at gmail.com Tue Nov 20 03:32:43 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Mon, 19 Nov 2012 21:32:43 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> Message-ID: <E88CB816E0164340BFC8468448FE2974@gmail.com> On Monday, November 19, 2012 at 9:24 PM, Daniel Holth wrote: > Mostly it seems a bit silly to have so much conversations about parts of the pep that remain unchanged from previously accepted versions... Well, I think the PEP should describe what we expect to be implemented *shrug*. Either we should expect it to be implemented and it should be part of the spec, or we shouldn't expect people to implement it and it should be removed. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/e2faabd6/attachment-0001.html> From dholth at gmail.com Tue Nov 20 04:30:59 2012 From: dholth at gmail.com (Daniel Holth) Date: Mon, 19 Nov 2012 22:30:59 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <E88CB816E0164340BFC8468448FE2974@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <E88CB816E0164340BFC8468448FE2974@gmail.com> Message-ID: <CAG8k2+5+xrCn-LTD7unoD-NwVfCk_W17D_1ujjVignBMyrYiRQ@mail.gmail.com> The section could definitely be much clearer. How about: Provides-Dist (multiple use) Each entry contains a string naming a requirement that is satisfied by installing this distribution. This field *must* include the project identified in the ``Name`` field, optionally followed by the version: Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into a single distribution or to indicate that this project is a substitute for another. For instance distribute (a fork of setuptools) could ``Provides-Dist`` setuptools to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121119/914f3f09/attachment.html> From kristjan at ccpgames.com Tue Nov 20 10:06:30 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 20 Nov 2012 09:06:30 +0000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> I'm intrigued. I thought this was merely so that one could do python -m mypackage.mysubpackage Can you refer me to the rationale and discussion about this feature? K From: Nick Coghlan [mailto:ncoghlan at gmail.com] Sent: 18. n?vember 2012 11:25 To: Kristj?n Valur J?nsson Cc: Christian Tismer; python-dev at python.org Subject: Re: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) Easily bundling dependencies is a key principle behind the ability to execute directories and zipfiles that contain a top level __main__.py file that was added back in 2.6 (although the zipfile version doesn't play nicely with extension modules). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com<mailto:ncoghlan at gmail.com> | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/66800b3c/attachment.html> From ncoghlan at gmail.com Tue Nov 20 12:39:22 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 20 Nov 2012 21:39:22 +1000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> On Tue, Nov 20, 2012 at 7:06 PM, Kristj?n Valur J?nsson < kristjan at ccpgames.com> wrote: > I?m intrigued. I thought this was merely so that one could do**** > > python ?m mypackage.mysubpackage**** > > Can you refer me to the rationale and discussion about this feature? > It was part of a fairly long progression in changes to the command line execution support :) Top level package execution with -m came first in 2.4, arbitrary package execution for -m arrived in 2.5 (along with the runpy module), directory and zipfile execution (by supplying a valid sys.path entry as the "script" command line argument) was added in 2.6/3.0, and finally officially supported package execution via -m only arrived in 2.7 and 3.1 (a broken version of the last accidentally existed in 2.5, but that was just a mistake that arose when merging the import emulations in runpy and pkgutil, and had a side effect that violated at least one of the import system invariants). In the specific case of directory and zipfile execution, discussion happened on the tracker: http://bugs.python.org/issue1739468 It was never brought up on python-dev because Guido was participating directly in the tracker discussion. Unfortunately, we then also forgot to mention it in the original version of the 2.6 What's New doc, so the vast majority of people missed the addition :( Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/29969587/attachment.html> From tismer at stackless.com Tue Nov 20 14:45:25 2012 From: tismer at stackless.com (Christian Tismer) Date: Tue, 20 Nov 2012 14:45:25 +0100 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> Message-ID: <50AB8975.3020405@stackless.com> On 20.11.12 12:39, Nick Coghlan wrote: > On Tue, Nov 20, 2012 at 7:06 PM, Kristj?n Valur J?nsson > <kristjan at ccpgames.com <mailto:kristjan at ccpgames.com>> wrote: > > I?m intrigued. I thought this was merely so that one could do > > python ?m mypackage.mysubpackage > > Can you refer me to the rationale and discussion about this feature? > > > It was part of a fairly long progression in changes to the command > line execution support :) > > Top level package execution with -m came first in 2.4, arbitrary > package execution for -m arrived in 2.5 (along with the runpy module), > directory and zipfile execution (by supplying a valid sys.path entry > as the "script" command line argument) was added in 2.6/3.0, and > finally officially supported package execution via -m only arrived in > 2.7 and 3.1 (a broken version of the last accidentally existed in 2.5, > but that was just a mistake that arose when merging the import > emulations in runpy and pkgutil, and had a side effect that violated > at least one of the import system invariants). > > In the specific case of directory and zipfile execution, discussion > happened on the tracker: http://bugs.python.org/issue1739468 > > It was never brought up on python-dev because Guido was participating > directly in the tracker discussion. Unfortunately, we then also forgot > to mention it in the original version of the 2.6 What's New doc, so > the vast majority of people missed the addition :( Hi Nick, thank you very much for this story and the link to the issue tracker! A very good move for Python which is really not mentioned enough to make people more aware of a great feature. I think part of this discussion should get a more prominent place for gems that can be found without knowing what to search. ;-) Is the issue tracker permanent enough to reference it? Maybe there could be some auxiliary info page with proper keywords that collects links to relevant discussions like this. Do we have such a thing already? ciao - chris -- Christian Tismer :^) <mailto:tismer at stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/a2715f50/attachment.html> From vinay_sajip at yahoo.co.uk Tue Nov 20 15:35:07 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 14:35:07 +0000 (UTC) Subject: [Python-Dev] Accept just PEP-0426 References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> Message-ID: <loom.20121120T150418-29@post.gmane.org> Daniel Holth <dholth <at> gmail.com> writes: > Mostly it seems a bit silly to have so much conversations about parts of the > pep that remain unchanged from previously accepted versions... I don't agree with the suggestion that we shouldn't discuss it because it was accepted in a previous version. Perhaps it didn't receive the right scrutiny at that time, but since it hasn't been implemented, it's reasonable to discuss it. ISTM that implementing it as suggested in the PEP can lead to certain problems, since it is a multi-valued field. If it is left in, then something should be said in the PEP about the potential difficulties and if/how they can be resolved. The difficulties I am talking about relate to dependency resolution. Given the current definition of Provides-Dist, it is possible for a package A on PyPI to "Provide" all of e.g. "A (1.0)", "B (1.2)" and "C (1.5)", and it is also possible for packages B and C on PyPI to provide the same (or slightly different) versions of logical packages of A, B, and C. This will likely lead to the need for a sophisticated dependency resolver because the dependency graph can get quite convoluted. (Remember, we might need to do this resolution when removing packages as well as when installing them.) I know there are SAT solvers and such, but I'm not sure we need that level of sophistication, or whether its complexity cost is outweighed by any benefit. Remember, we are managing fine without multi-valued Provides-Dist, and while a case has been made for virtual packages and forks (which just require a single-valued field), no compelling case has been made for bundling packages in general (I understand that such requirements might sometimes arise in certain corporate environments, but they don't seem to be a mainstream use case). Hence, no strong case has been made for a multi-valued "Provides" field. If we have a good index and packaging infrastructure, there is no general need for packages to bundle other packages, unless those bundled packages are changed in some way to suit the bundler's needs. In that case, I don't know how you could be sure that a bundled "A (1.0)" hasn't diverged from the equivalent package on PyPI. The "Provides" seems essentially useless in a metadata index, since if, when asked to install D which has a dependency on A, you would download and install A to resolve it rather than B or C, and I can't see when you would want to query the index to say "who provides A?" and then use some heuristic to pick e.g. B or C, rather than A. distlib currently contains support for the multi-valued "Provides", but I'm not confident that will work as expected given pathological cases like the example I suggested, without getting "complicated" in the Zen of Python sense. I'm not convinced that the maintenance burden of a complicated solution is worth the heretofore unnecessary ability to bundle stuff in arbitrary ways. Regards, Vinay Sajip From ncoghlan at gmail.com Tue Nov 20 15:44:53 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 00:44:53 +1000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <50AB8975.3020405@stackless.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> Message-ID: <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> On Tue, Nov 20, 2012 at 11:45 PM, Christian Tismer <tismer at stackless.com>wrote: > On 20.11.12 12:39, Nick Coghlan wrote: > > On Tue, Nov 20, 2012 at 7:06 PM, Kristj?n Valur J?nsson < > kristjan at ccpgames.com> wrote: > >> I?m intrigued. I thought this was merely so that one could do >> >> python ?m mypackage.mysubpackage >> >> Can you refer me to the rationale and discussion about this feature? >> > > It was part of a fairly long progression in changes to the command line > execution support :) > > Top level package execution with -m came first in 2.4, arbitrary package > execution for -m arrived in 2.5 (along with the runpy module), directory > and zipfile execution (by supplying a valid sys.path entry as the "script" > command line argument) was added in 2.6/3.0, and finally officially > supported package execution via -m only arrived in 2.7 and 3.1 (a broken > version of the last accidentally existed in 2.5, but that was just a > mistake that arose when merging the import emulations in runpy and pkgutil, > and had a side effect that violated at least one of the import system > invariants). > > In the specific case of directory and zipfile execution, discussion > happened on the tracker: http://bugs.python.org/issue1739468 > > It was never brought up on python-dev because Guido was participating > directly in the tracker discussion. Unfortunately, we then also forgot to > mention it in the original version of the 2.6 What's New doc, so the vast > majority of people missed the addition :( > > > Hi Nick, > > thank you very much for this story and the link to the issue tracker! > A very good move for Python which is really not mentioned enough > to make people more aware of a great feature. > > I think part of this discussion should get a more prominent place > for gems that can be found without knowing what to search. ;-) > Technically, that's the "using" guide: http://docs.python.org/2/using/cmdline.html#interface-options (see the explanation of what's executable under the "<script>" tag) I also wrote up a summary last year on my blog: http://www.boredomandlaziness.org/2011/03/what-is-python-script.html (The main change since that post is that the Python launcher brings shebang line support to Windows, although I haven't checked if it works properly with prefixed zip files) Maybe I should plan to sign up to present an updated version of my PyCon AU 2010 "What is a Python script?" lightning talk at PyCon US 2013 :) > Is the issue tracker permanent enough to reference it? > I've been referencing that particular issue for years now, so I certainly think so :) > Maybe there could be some auxiliary info page with proper keywords > that collects links to relevant discussions like this. > Do we have such a thing already? > I've sometimes thought it may be a good idea to split out a separate "What is a Python script?" section in the using guide, separate from the existing descriptions under the interpreter options. I've never tried to figure out the details of how that would actually work, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/642d32f1/attachment.html> From dholth at gmail.com Tue Nov 20 15:56:14 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 09:56:14 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T150418-29@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> Message-ID: <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> On Tue, Nov 20, 2012 at 9:35 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote: > Daniel Holth <dholth <at> gmail.com> writes: > > > Mostly it seems a bit silly to have so much conversations about parts of > the > > pep that remain unchanged from previously accepted versions... > > I don't agree with the suggestion that we shouldn't discuss it because it > was > accepted in a previous version. Perhaps it didn't receive the right > scrutiny at > that time, but since it hasn't been implemented, it's reasonable to > discuss it. > > ISTM that implementing it as suggested in the PEP can lead to certain > problems, > since it is a multi-valued field. If it is left in, then something should > be > said in the PEP about the potential difficulties and if/how they can be > resolved. > > The difficulties I am talking about relate to dependency resolution. Given > the > current definition of Provides-Dist, it is possible for a package A on > PyPI to > "Provide" all of e.g. "A (1.0)", "B (1.2)" and "C (1.5)", and it is also > possible for packages B and C on PyPI to provide the same (or slightly > different) versions of logical packages of A, B, and C. This will likely > lead > to the need for a sophisticated dependency resolver because the dependency > graph can get quite convoluted. (Remember, we might need to do this > resolution > when removing packages as well as when installing them.) I know there are > SAT > solvers and such, but I'm not sure we need that level of sophistication, or > whether its complexity cost is outweighed by any benefit. Remember, we are > managing fine without multi-valued Provides-Dist, and while a case has been > made for virtual packages and forks (which just require a single-valued > field), > no compelling case has been made for bundling packages in general (I > understand > that such requirements might sometimes arise in certain corporate > environments, > but they don't seem to be a mainstream use case). Hence, no strong case has > been made for a multi-valued "Provides" field. > > If we have a good index and packaging infrastructure, there is no general > need > for packages to bundle other packages, unless those bundled packages are > changed > in some way to suit the bundler's needs. In that case, I don't know how you > could be sure that a bundled "A (1.0)" hasn't diverged from the equivalent > package on PyPI. > > The "Provides" seems essentially useless in a metadata index, since if, > when > asked to install D which has a dependency on A, you would download and > install > A to resolve it rather than B or C, and I can't see when you would want to > query the index to say "who provides A?" and then use some heuristic to > pick > e.g. B or C, rather than A. > > distlib currently contains support for the multi-valued "Provides", but I'm > not confident that will work as expected given pathological cases like the > example I suggested, without getting "complicated" in the Zen of Python > sense. > I'm not convinced that the maintenance burden of a complicated solution is > worth the heretofore unnecessary ability to bundle stuff in arbitrary ways. > If you don't have Provides-Dist, then distribute must continue to bundle an extra .egg-info directory to emulate the feature. This is more than enough justification for me. Name: is essentially an alias for Provides-Dist: (or vice-versa) so there is no such thing as a single-valued Provides-Dist. Having two names for a package is just as complicated as having twenty. You should not implement Provides-Dist by searching for every Provides-Dist: name on PyPI. You should only use it when deciding whether to download setuptools when distribute is already installed and a package depends on setuptools. The bundling term was bad wording on the part of the PEP. No one should ever include non-renamed copies of other dists in their dists "import six" vs. "import django.util.six". I've suggested a new wording in this thread. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/85189559/attachment.html> From vinay_sajip at yahoo.co.uk Tue Nov 20 16:16:25 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 15:16:25 +0000 (UTC) Subject: [Python-Dev] Accept just PEP-0426 References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> Message-ID: <loom.20121120T160434-5@post.gmane.org> Daniel Holth <dholth <at> gmail.com> writes: > If you don't have Provides-Dist, then distribute must continue to bundle an > extra .egg-info directory to emulate the feature. This is more than enough > justification for me. Name: is essentially an alias for Provides-Dist: (or > vice-versa) so there is no such thing as a single-valued Provides-Dist. Having > two names for a package is just as complicated as having twenty. I'm not so sure. In the case of two names, it could be assumed that one was a fork of the other (as in the specific cases of distribute/setuptools, or PIL/ Pillow). You cannot reasonably make this assumption if you have twenty entries in your Provides-Dist. > You should not implement Provides-Dist by searching for every Provides-Dist: > name on PyPI. I wasn't seriously suggesting that this approach be taken - merely pointing out that Provides-Dist isn't of much use in a metadata index. > should only use it when deciding whether to download setuptools when distribute > is already installed and a package depends on setuptools.The bundling term was > bad wording on the part of the PEP. No one should ever include non-renamed > copies of other dists in their dists "import six" vs. "import django.util.six". > I've suggested a new wording in this thread. So apart from the setuptools/distribute and PIL/Pillow scenarios, what are the scenarios where you would have 3 or more values in Provides-Dist? If they are e.g. a bundled SQLAlchemy, why would that be preferable to an entry in Requires-Dist? Regards, Vinay Sajip From ncoghlan at gmail.com Tue Nov 20 16:49:28 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 01:49:28 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T160434-5@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> Message-ID: <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> On Wed, Nov 21, 2012 at 1:16 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote: > Daniel Holth <dholth <at> gmail.com> writes: > > > If you don't have Provides-Dist, then distribute must continue to bundle > an > > extra .egg-info directory to emulate the feature. This is more than > enough > > justification for me. Name: is essentially an alias for Provides-Dist: > (or > > vice-versa) so there is no such thing as a single-valued Provides-Dist. > Having > > two names for a package is just as complicated as having twenty. > > I'm not so sure. In the case of two names, it could be assumed that one > was a > fork of the other (as in the specific cases of distribute/setuptools, or > PIL/ > Pillow). You cannot reasonably make this assumption if you have twenty > entries > in your Provides-Dist. > > > You should not implement Provides-Dist by searching for every > Provides-Dist: > > name on PyPI. > > I wasn't seriously suggesting that this approach be taken - merely > pointing out > that Provides-Dist isn't of much use in a metadata index. > > > should only use it when deciding whether to download setuptools when > distribute > > is already installed and a package depends on setuptools.The bundling > term was > > bad wording on the part of the PEP. No one should ever include > non-renamed > > copies of other dists in their dists "import six" vs. "import > django.util.six". > > I've suggested a new wording in this thread. > > So apart from the setuptools/distribute and PIL/Pillow scenarios, what are > the > scenarios where you would have 3 or more values in Provides-Dist? If they > are > e.g. a bundled SQLAlchemy, why would that be preferable to an entry in > Requires-Dist? > Provides/Requires/Obsoletes are *not* for bundling. Publishing bundled packages on the index is bad, and people shouldn't do it. What they're for is tracking name changes over time, so that you can fork and rename and merge projects without breaking the world for people that depend on your projects (one example used in the Fedora RPM docs is the apache package being renamed to httpd: https://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html-single/RPM_Guide/index.html#ch-dependencies ). The fact distribute can provide setuptools and Pillow can provide PIL are examples of the simple fork/rename case - they're designed to be drop in replacements for the projects they forked, so it's appropriate for them to advertise that fact in a way the deployment tools can understand. The multi-value support is then needed if you have multiple name changes over time (e.g. if someone were to create a distribute2 that provided both distribute and setuptools), or if you merge two projects together (e.g. if a popular extension to a project was folded into the main distribution for that project). It's likely fine if an installer doesn't use sophisticated graph analysis to find the "best" way to satisfy a set of requirements - you can just as easily use it in the simple way Daniel describes of only using these fields to check for existing locally installed packages with the necessary capabilities, before going out to get whatever is missing from the package index based purely on the distribution names. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/04b5f179/attachment.html> From vinay_sajip at yahoo.co.uk Tue Nov 20 17:04:06 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 16:04:06 +0000 (UTC) Subject: [Python-Dev] Accept just PEP-0426 References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> Message-ID: <loom.20121120T165425-38@post.gmane.org> Nick Coghlan <ncoghlan <at> gmail.com> writes: > Provides/Requires/Obsoletes are *not* for bundling. Publishing bundled packages > on the index is bad, and people shouldn't do it. [detail snipped] > It's likely fine if an installer doesn't use sophisticated graph > analysis to find the "best" way to satisfy a set of requirements - you can > just as easily use it in the simple way Daniel describes of only using these > fields to check for existing locally installed packages with the necessary > capabilities, before going out to get whatever is missing from the package > index based purely on the distribution names. In which case, it seems sensible to put these constraints into the PEP, so that both PEP implementers and users of those implementations have these guidelines clarified. Regards, Vinay Sajip From dholth at gmail.com Tue Nov 20 17:18:00 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 11:18:00 -0500 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> Message-ID: <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> On Tue, Nov 20, 2012 at 9:44 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Tue, Nov 20, 2012 at 11:45 PM, Christian Tismer <tismer at stackless.com>wrote: > >> On 20.11.12 12:39, Nick Coghlan wrote: >> >> On Tue, Nov 20, 2012 at 7:06 PM, Kristj?n Valur J?nsson < >> kristjan at ccpgames.com> wrote: >> >>> I?m intrigued. I thought this was merely so that one could do >>> >>> python ?m mypackage.mysubpackage >>> >>> Can you refer me to the rationale and discussion about this feature? >>> >> >> It was part of a fairly long progression in changes to the command line >> execution support :) >> >> Top level package execution with -m came first in 2.4, arbitrary package >> execution for -m arrived in 2.5 (along with the runpy module), directory >> and zipfile execution (by supplying a valid sys.path entry as the "script" >> command line argument) was added in 2.6/3.0, and finally officially >> supported package execution via -m only arrived in 2.7 and 3.1 (a broken >> version of the last accidentally existed in 2.5, but that was just a >> mistake that arose when merging the import emulations in runpy and pkgutil, >> and had a side effect that violated at least one of the import system >> invariants). >> >> In the specific case of directory and zipfile execution, discussion >> happened on the tracker: http://bugs.python.org/issue1739468 >> >> It was never brought up on python-dev because Guido was participating >> directly in the tracker discussion. Unfortunately, we then also forgot to >> mention it in the original version of the 2.6 What's New doc, so the vast >> majority of people missed the addition :( >> >> >> Hi Nick, >> >> thank you very much for this story and the link to the issue tracker! >> A very good move for Python which is really not mentioned enough >> to make people more aware of a great feature. >> >> I think part of this discussion should get a more prominent place >> for gems that can be found without knowing what to search. ;-) >> > > Technically, that's the "using" guide: > http://docs.python.org/2/using/cmdline.html#interface-options (see the > explanation of what's executable under the "<script>" tag) > > I also wrote up a summary last year on my blog: > http://www.boredomandlaziness.org/2011/03/what-is-python-script.html > > (The main change since that post is that the Python launcher brings > shebang line support to Windows, although I haven't checked if it works > properly with prefixed zip files) > > Maybe I should plan to sign up to present an updated version of my PyCon > AU 2010 "What is a Python script?" lightning talk at PyCon US 2013 :) > > >> Is the issue tracker permanent enough to reference it? >> > > I've been referencing that particular issue for years now, so I certainly > think so :) > > >> Maybe there could be some auxiliary info page with proper keywords >> that collects links to relevant discussions like this. >> Do we have such a thing already? >> > > I've sometimes thought it may be a good idea to split out a separate "What > is a Python script?" section in the using guide, separate from the existing > descriptions under the interpreter options. I've never tried to figure out > the details of how that would actually work, though. > The wheel projects' own wheel file (a zip file) takes advantage of this feature in its own way: python wheel-0.14.0-py2.py3-none-any.whl/wheel Python looks inside the /wheel directory of the zip file, finds __main__.py, and executes it; the archive can be used to install itself or any other wheel file. (There is no __main__.py at the root because the wheel design would install that __main__.py into the root of site-packages). This underutilized feature of executing directories with __main__.py is very useful for implementing Python applications instead of just libraries. It might be interesting to define a "wheels" format which would be a bunch of unpacked wheel files and a __main__.py like: __main__.py package1-x86.whl/... package1-armel.whl/... package2-noarch.whl/ __main__.py would add the appropriate packages to PYTHONPATH based on the architecture, unpack dll's pkg_resources style, and run the program. There is some activity in the tracker about adding the missing "add this to PYTHONPATH" / "isolate self from the environment" command line arguments to Python. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/d9b97357/attachment.html> From dholth at gmail.com Tue Nov 20 17:19:32 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 11:19:32 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T165425-38@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> Message-ID: <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> Edit the following text: Provides-Dist (multiple use) Each entry contains a string naming a requirement that is satisfied by installing this distribution. This field *must* include the project identified in the ``Name`` field, optionally followed by the version Name (Version). A distribution may provide additional names, e.g. to indicate that multiple projects have been merged into a single distribution or to indicate that this project is a substitute for another. For instance distribute (a fork of setuptools) could ``Provides-Dist`` setuptools to prevent the conflicting package from being downloaded and installed when distribute is already installed. A distribution that has been merged with another might ``Provides-Dist`` the obsolete name(s) to satisfy any projects that require the obsolete distribution's name. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/a61380f2/attachment.html> From ncoghlan at gmail.com Tue Nov 20 17:36:00 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 02:36:00 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T165425-38@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> Message-ID: <CADiSq7eaBHcFcNZC7V3tcS3TZ9RiPm4cj7i2B-sazxotEeprFw@mail.gmail.com> On Wed, Nov 21, 2012 at 2:04 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote: > Nick Coghlan <ncoghlan <at> gmail.com> writes: > > > Provides/Requires/Obsoletes are *not* for bundling. Publishing bundled > packages > > on the index is bad, and people shouldn't do it. > [detail snipped] > > It's likely fine if an installer doesn't use sophisticated graph > > analysis to find the "best" way to satisfy a set of requirements - you > can > > just as easily use it in the simple way Daniel describes of only using > these > > fields to check for existing locally installed packages with the > necessary > > capabilities, before going out to get whatever is missing from the > package > > index based purely on the distribution names. > > In which case, it seems sensible to put these constraints into the PEP, > so that both PEP implementers and users of those implementations have these > guidelines clarified. > Yes, I thought Daniel's rewording looked pretty reasonable on that front. However, the details of how an installer uses this information is really up to the installer developers and what their users expect/demand. It certainly isn't *practical* to do a full dependency analysis when PyPI doesn't provide the same kind of precalculated metadata that a yum repo does, but that's not something that should be spelled out in the distribution metadata PEP, any more than it is spelled out in the RPM format spec. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/e8c1bbd4/attachment.html> From vinay_sajip at yahoo.co.uk Tue Nov 20 17:49:56 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 16:49:56 +0000 (UTC) Subject: [Python-Dev] Accept just PEP-0426 References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> Message-ID: <loom.20121120T172255-830@post.gmane.org> Daniel Holth <dholth <at> gmail.com> writes: > Edit the following text: Okay, here is a possible version: --------------------------------- Provides-Dist (multiple use) Each entry contains a string naming a requirement that is satisfied by installing this distribution. The entry must consist of a name and version. This name of the project identified in the ``Name`` field is implicitly considered as provided, with the version specified in the ``Version`` field. The use of multiple names in this field *must not* be used for bundling distributions together. It is intended for use when projects are forked and merged over time, while providing essentially the same function. Multiple names reflect the evolution of the project over time and not the bringing together of different packages, already distributed elsewhere, in a bundle. Thus, the 'distribute' distribution, a fork of setuptools, could say that it ``Provides-Dist`` a particular version of setuptools, to prevent setuptools from being downloaded and installed when distribute is already installed. If, over time, distribute evolved into a new package called 'distribute2' (for argument's sake), then that could say that it ``Provides-Dist`` a specific version of distribute and a specific version of setuptools. ----------------------------------- Some comments on the above: I'm not entirely comfortable with a Provides-Dist entry which does not specify a version, since it does not allow to you to test that a requirement is actually met. So, I've removed the "optional" qualification from the version. Also: what happens when a requirement is for setuptools (>= X.Y), but the distribute fork hasn't kept pace, and so only supports setuptools at a lower version than X.Y? I take it we're entirely comfortable with installing setuptools X.Y in that case? How would you ensure the right setuptools is always loaded, since presumably both are on sys.path? Regards, Vinay Sajip From pje at telecommunity.com Tue Nov 20 21:46:27 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 20 Nov 2012 15:46:27 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T172255-830@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> Message-ID: <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> On Tue, Nov 20, 2012 at 11:49 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk>wrote: > Also: what happens when a requirement is for setuptools (>= X.Y), but the > distribute fork hasn't kept pace, and so only supports setuptools at a > lower > version than X.Y? I take it we're entirely comfortable with installing > setuptools X.Y in that case? How would you ensure the right setuptools is > always loaded, since presumably both are on sys.path? > Egg-based tools don't have any problem with this, since they set sys.path to include the eggs needed for the running program. Other tools will have to tell the user and let them work it out, e.g. by using a different virtualenv. I personally don't think that forks claiming to "provide" something is really a good thing to encourage; ISTM that saying a package *conflicts* with another is more accurate, e.g. distribute Conflicts-Dist setuptools. I also think distributions should say they are obsoleted, rather than allowing other distributions to obsolete them. That is, centralized packaging systems rely on a central authority to resolve issues of who provides what and obsoletes what; there's an implicit "x obsoletes y [by decree of semi-independent third-party z]". However, in Python package metadata, it's "x obsoletes y [by decree of x]". IMO, this should be reversed to, "Y is obsoleted by x [by decree of y]", and "installing Y will conflict with X [by decree of X]", so that in each case the scope of authority for the statement is clear. That is, in each case (conflict or obsolescence), the project's developers are declaring under what conditions they will not be supporting an installation. In the case of obsolescence, the developer is saying, "this is being phased out, you should use that other thing instead". In the case of forks, the developer is saying, "If you install both versions, something's gonna break." Note that installation conflict is a more conservative claim anyway: a conflict between forked "foobar" packages is permanent, in the sense that it doesn't matter what versions of both packages you're interested in: they both want to install a foobar/__init__.py. (Of course, installers can and should detect that condition automatically, but not until they download the package first.) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/9952bd9c/attachment-0001.html> From jimjjewett at gmail.com Tue Nov 20 21:58:54 2012 From: jimjjewett at gmail.com (Jim J. Jewett) Date: Tue, 20 Nov 2012 12:58:54 -0800 (PST) Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <loom.20121120T172255-830@post.gmane.org> Message-ID: <50abef0e.ab96320a.6675.ffffb439@mx.google.com> Vinay Sajip reworded the 'Provides-Dist' definition to explicitly say: > The use of multiple names in this field *must not* be used for > bundling distributions together. It is intended for use when > projects are forked and merged over time ... (1) Then how *should* the "bundle-of-several-components" case be represented? (2) How is 'Provides-Dist' different from 'Obsoletes-Dist'? The only difference I can see is that it may be a bit more polite to people who do want to install multiple versions of a (possibly abstract) package. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ From v+python at g.nevcal.com Tue Nov 20 22:07:52 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 20 Nov 2012 13:07:52 -0800 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> Message-ID: <50ABF128.4020605@g.nevcal.com> On 11/20/2012 12:46 PM, PJ Eby wrote: > On Tue, Nov 20, 2012 at 11:49 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk > <mailto:vinay_sajip at yahoo.co.uk>> wrote: > > Also: what happens when a requirement is for setuptools (>= X.Y), > but the > distribute fork hasn't kept pace, and so only supports setuptools > at a lower > version than X.Y? I take it we're entirely comfortable with installing > setuptools X.Y in that case? How would you ensure the right > setuptools is > always loaded, since presumably both are on sys.path? > > > Egg-based tools don't have any problem with this, since they set > sys.path to include the eggs needed for the running program. Other > tools will have to tell the user and let them work it out, e.g. by > using a different virtualenv. > > I personally don't think that forks claiming to "provide" something is > really a good thing to encourage; ISTM that saying a package > *conflicts* with another is more accurate, e.g. distribute > Conflicts-Dist setuptools. I also think distributions should say they > are obsoleted, rather than allowing other distributions to obsolete them. Obsolete distributions won't say they are obsoleted, unless they receive further maintenance. However, if the distribution is obsolete because the maintainer has lost interest, they won't receive further maintenance. > That is, centralized packaging systems rely on a central authority to > resolve issues of who provides what and obsoletes what; there's an > implicit "x obsoletes y [by decree of semi-independent third-party z]". > > However, in Python package metadata, it's "x obsoletes y [by decree of > x]". IMO, this should be reversed to, "Y is obsoleted by x [by decree > of y]", and "installing Y will conflict with X [by decree of X]", so > that in each case the scope of authority for the statement is clear. > > That is, in each case (conflict or obsolescence), the project's > developers are declaring under what conditions they will not be > supporting an installation. In the case of obsolescence, the > developer is saying, "this is being phased out, you should use that > other thing instead". In the case of forks, the developer is saying, > "If you install both versions, something's gonna break." > > Note that installation conflict is a more conservative claim anyway: a > conflict between forked "foobar" packages is permanent, in the sense > that it doesn't matter what versions of both packages you're > interested in: they both want to install a foobar/__init__.py. (Of > course, installers can and should detect that condition automatically, > but not until they download the package first.) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/70c7ae5f/attachment.html> From dholth at gmail.com Tue Nov 20 22:31:24 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 16:31:24 -0500 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <50abef0e.ab96320a.6675.ffffb439@mx.google.com> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> Message-ID: <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> On Tue, Nov 20, 2012 at 3:58 PM, Jim J. Jewett <jimjjewett at gmail.com> wrote: > > > Vinay Sajip reworded the 'Provides-Dist' definition to explicitly say: > > > The use of multiple names in this field *must not* be used for > > bundling distributions together. It is intended for use when > > projects are forked and merged over time ... > > (1) Then how *should* the "bundle-of-several-components" case be > represented? > > (2) How is 'Provides-Dist' different from 'Obsoletes-Dist'? > The only difference I can see is that it may be a bit more polite > to people who do want to install multiple versions of a (possibly > abstract) package. > The useful way to bundle a bunch of things would be to just include them all in an executable folder or zipfile with __main__.py. PEP 426 and the package database would not get involved. The bundle would be distributed as an application you can download and use, not as an sdist on PyPI. The intent of Provides and Obsoletes is different. Obsoletes would not satisfy a requirement during dependency resolution. The RPM guide explains a similar system: This brings the total to four types of dependencies that the RPM system tracks: - Requires, which tracks the capabilities a package requires - Provides, which tracks the capabilities a package provides for other packages - Conflicts, which describes the capabilities that if installed, conflict with capabilities in a package - Obsoletes, which describes the capabilities that this package will make obsolete Packages advertise this dependency information. Each dependency holds the type, such as requires, a capability, such as a shared library or a package name, and optionally a version number, such as requiring the python package at a version number greater than or equal to 2.2 (python >= 2.2). http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-dependencies.html#RPM_Guide-Dependencies-obsoletes -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/da409fec/attachment.html> From pje at telecommunity.com Tue Nov 20 22:48:51 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 20 Nov 2012 16:48:51 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <50ABF128.4020605@g.nevcal.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> Message-ID: <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> On Tue, Nov 20, 2012 at 4:07 PM, Glenn Linderman <v+python at g.nevcal.com>wrote: > On 11/20/2012 12:46 PM, PJ Eby wrote: > > I personally don't think that forks claiming to "provide" something is > really a good thing to encourage; ISTM that saying a package *conflicts* > with another is more accurate, e.g. distribute Conflicts-Dist setuptools. > I also think distributions should say they are obsoleted, rather than > allowing other distributions to obsolete them. > > Obsolete distributions won't say they are obsoleted, unless they receive > further maintenance. However, if the distribution is obsolete because the > maintainer has lost interest, they won't receive further maintenance. > (We've been over this before, the last time this discussion came up on the Distutils-SIG for a previous Metadata PEP a year or two back, but here goes....) Obsoleting a package is for handling renames and support transitions. For example, if it actually did anything to do so, I'd mark RuleDispatch as obsoleted-by PEAK, the Pylons folks might mark some version of that as obsoleted-by Pyramid, etc. To put it another way, marking a package obsolete is part of deprecation and replacement, not an unsubstantiated third-party claim about the maintenance status of an unrelated project. If a package is *actually* dead, there's no real point to declaring that something else obsoletes it, and certainly no reason to put it in metadata form. Otherwise, we could have Twisted claiming to obsolete GEvent and vice-versa at the same time. Which one should an installer believe? It makes no sense in a standard where the project's maintainers can say whatever they want about somebody else's project. The scope of authority for automatically-consumed metadata should *only* encompass the project that provided the metadata. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/9b613f22/attachment.html> From donald.stufft at gmail.com Tue Nov 20 22:52:11 2012 From: donald.stufft at gmail.com (Donald Stufft) Date: Tue, 20 Nov 2012 16:52:11 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> Message-ID: <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> On Tuesday, November 20, 2012 at 4:48 PM, PJ Eby wrote: > Words I agree that obsoletes is terrible, it's very specific and not something we particularly require. I'd much rather just have a generic "conflicts". -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/0deed046/attachment.html> From dholth at gmail.com Tue Nov 20 23:01:02 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 17:01:02 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> Message-ID: <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> I think the Metadata 1.1 treatment of these concepts is in some ways better. (Metadata 1.2 added the -Dist suffix to the fields in an attempt to make it clear that dependency names are PyPI names and not "import x" names.) http://www.python.org/dev/peps/pep-0314/ says: Provides (multiple use) Each entry contains a string describing a package or module that will be provided by this package once it is installed. These strings should match the ones used in Requirements fields. A version declaration may be supplied (without a comparison operator); the package's version number will be implied if none is specified. Example: Provides: xml Provides: xml.utils Provides: xml.utils.iso8601 Provides: xml.dom Provides: xmltools (1.3) Obsoletes (multiple use) Each entry contains a string describing a package or module that this package renders obsolete, meaning that the two packages should not be installed at the same time. Version declarations can be supplied. The most common use of this field will be in case a package name changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. When you install Torqued Python, the Gorgon package should be removed. Example: Obsoletes: Gorgon They mean pretty much what the same words mean in RPM and do not need further bikeshedding. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/bbc8a131/attachment.html> From brett at python.org Tue Nov 20 23:12:28 2012 From: brett at python.org (Brett Cannon) Date: Tue, 20 Nov 2012 17:12:28 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None In-Reply-To: <3Y5dwB3ZCMzMlG@mail.python.org> References: <3Y5dwB3ZCMzMlG@mail.python.org> Message-ID: <CAP1=2W4KTyoDYW6jO_1oQd+rYDE_cboKgsOk=05R5FzjPxEyqQ@mail.gmail.com> Should you also insert None into sys.path_importer_cache to signify there is no finder for the path entry? I guess the real problem with that is there is no guarantee the path entry is hashable, so that probably won't work. So nevermind. =) On Tue, Nov 20, 2012 at 3:35 PM, barry.warsaw <python-checkins at python.org>wrote: > http://hg.python.org/cpython/rev/291406748217 > changeset: 80529:291406748217 > branch: 3.3 > parent: 80527:4537dd27b2dc > user: Barry Warsaw <barry at python.org> > date: Tue Nov 20 15:22:51 2012 -0500 > summary: > - Issue #16514: Fix regression causing a traceback when sys.path[0] is > None > (actually, any non-string or non-bytes type). > > files: > Doc/library/sys.rst | 4 +- > Doc/reference/import.rst | 24 +- > Lib/importlib/_bootstrap.py | 2 + > Lib/test/test_importlib/import_/test_path.py | 25 +- > Misc/NEWS | 3 + > Python/importlib.h | 130 +++++---- > 6 files changed, 111 insertions(+), 77 deletions(-) > > > diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst > --- a/Doc/library/sys.rst > +++ b/Doc/library/sys.rst > @@ -783,7 +783,9 @@ > current directory first. Notice that the script directory is inserted > *before* > the entries inserted as a result of :envvar:`PYTHONPATH`. > > - A program is free to modify this list for its own purposes. > + A program is free to modify this list for its own purposes. Only > strings > + and bytes should be added to :data:`sys.path`; all other data types are > + ignored during import. > > > .. seealso:: > diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst > --- a/Doc/reference/import.rst > +++ b/Doc/reference/import.rst > @@ -540,7 +540,10 @@ > implementation-specific defaults. Entries in :data:`sys.path` can name > directories on the file system, zip files, and potentially other > "locations" > (see the :mod:`site` module) that should be searched for modules, such as > -URLs, or database queries. > +URLs, or database queries. Only strings and bytes should be present on > +:data:`sys.path`; all other data types are ignored. The encoding of bytes > +entries is determined by the individual :term:`path entry finders <path > entry > +finder>`. > > The :term:`path based finder` is a :term:`meta path finder`, so the import > machinery begins the :term:`import path` search by calling the path > @@ -563,14 +566,17 @@ > the path based finder to perform the path entry search again [#fnpic]_. > > If the path entry is not present in the cache, the path based finder > iterates > -over every callable in :data:`sys.path_hooks`. Each of the > -:term:`path entry hooks <path entry hook>` in this list is called with a > -single argument, the path entry to be searched. This callable may either > -return a :term:`path entry finder` that can handle the path entry, or it > may > -raise :exc:`ImportError`. > -An :exc:`ImportError` is used by the path based finder to signal that the > hook > -cannot find a :term:`path entry finder` for that :term:`path entry`. The > -exception is ignored and :term:`import path` iteration continues. > +over every callable in :data:`sys.path_hooks`. Each of the :term:`path > entry > +hooks <path entry hook>` in this list is called with a single argument, > the > +path entry to be searched. This callable may either return a :term:`path > +entry finder` that can handle the path entry, or it may raise > +:exc:`ImportError`. An :exc:`ImportError` is used by the path based > finder to > +signal that the hook cannot find a :term:`path entry finder` for that > +:term:`path entry`. The exception is ignored and :term:`import path` > +iteration continues. The hook should expect either a string or bytes > object; > +the encoding of bytes objects is up to the hook (e.g. it may be a file > system > +encoding, UTF-8, or something else), and if the hook cannot decode the > +argument, it should raise :exc:`ImportError`. > > If :data:`sys.path_hooks` iteration ends with no :term:`path entry finder` > being returned, then the path based finder's :meth:`find_module()` method > diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py > --- a/Lib/importlib/_bootstrap.py > +++ b/Lib/importlib/_bootstrap.py > @@ -1281,6 +1281,8 @@ > # the list of paths that will become its __path__ > namespace_path = [] > for entry in path: > + if not isinstance(entry, (str, bytes)): > + continue > finder = cls._path_importer_cache(entry) > if finder is not None: > if hasattr(finder, 'find_loader'): > diff --git a/Lib/test/test_importlib/import_/test_path.py > b/Lib/test/test_importlib/import_/test_path.py > --- a/Lib/test/test_importlib/import_/test_path.py > +++ b/Lib/test/test_importlib/import_/test_path.py > @@ -1,15 +1,14 @@ > from importlib import _bootstrap > from importlib import machinery > +from importlib import import_module > from .. import util > from . import util as import_util > -import imp > import os > import sys > -import tempfile > -from test import support > -from types import MethodType > +from types import ModuleType > import unittest > import warnings > +import zipimport > > > class FinderTests(unittest.TestCase): > @@ -89,6 +88,24 @@ > self.assertIs(loader, importer) > self.assertIn(os.curdir, sys.path_importer_cache) > > + def test_None_on_sys_path(self): > + # Putting None in sys.path[0] caused an import regression from > Python > + # 3.2: http://bugs.python.org/issue16514 > + new_path = sys.path[:] > + new_path.insert(0, None) > + new_path_importer_cache = sys.path_importer_cache.copy() > + new_path_importer_cache.pop(None, None) > + new_path_hooks = [zipimport.zipimporter, > + _bootstrap.FileFinder.path_hook( > + *_bootstrap._get_supported_file_loaders())] > + with util.uncache('email'): > + with util.import_state(meta_path=sys.meta_path[:], > + path=new_path, > + > path_importer_cache=new_path_importer_cache, > + path_hooks=new_path_hooks): > + module = import_module('email') > + self.assertIsInstance(module, ModuleType) > + > > def test_main(): > from test.support import run_unittest > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -12,6 +12,9 @@ > Core and Builtins > ----------------- > > +- Issue #16514: Fix regression causing a traceback when sys.path[0] is > None > + (actually, any non-string or non-bytes type). > + > - Issue #16306: Fix multiple error messages when unknown command line > parameters where passed to the interpreter. Patch by Hieu Nguyen. > > diff --git a/Python/importlib.h b/Python/importlib.h > --- a/Python/importlib.h > +++ b/Python/importlib.h > [stripped] > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/0c7139ce/attachment-0001.html> From vinay_sajip at yahoo.co.uk Tue Nov 20 23:26:01 2012 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 20 Nov 2012 22:26:01 +0000 (UTC) Subject: [Python-Dev] Accept just PEP-0426 References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> Message-ID: <loom.20121120T231553-707@post.gmane.org> Daniel Holth <dholth <at> gmail.com> writes: > They mean pretty much what the same words mean in RPM and do not need further > bikeshedding. But isn't it the case that the scenarios are different because in the case of RPMs, we have a presumed authority which can determine e.g. what obsoletes what, whereas with Python distributions, there's no central authority that has this function? Regards, Vinay Sajip From barry at python.org Tue Nov 20 23:35:49 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Nov 2012 17:35:49 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None In-Reply-To: <CAP1=2W4KTyoDYW6jO_1oQd+rYDE_cboKgsOk=05R5FzjPxEyqQ@mail.gmail.com> References: <3Y5dwB3ZCMzMlG@mail.python.org> <CAP1=2W4KTyoDYW6jO_1oQd+rYDE_cboKgsOk=05R5FzjPxEyqQ@mail.gmail.com> Message-ID: <20121120173549.01e9107e@resist.wooz.org> On Nov 20, 2012, at 05:12 PM, Brett Cannon wrote: >Should you also insert None into sys.path_importer_cache to signify there >is no finder for the path entry? I guess the real problem with that is >there is no guarantee the path entry is hashable, so that probably won't >work. So nevermind. =) I explicitly popped None from path_importer_cache, to try to force it to ignore the cache. Maybe it's not totally correct (and in practice probably doesn't matter), but it WFM. -Barry From barry at python.org Wed Nov 21 00:41:07 2012 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Nov 2012 18:41:07 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None In-Reply-To: <50AC05BD.3080402@udel.edu> References: <3Y5dwB3ZCMzMlG@mail.python.org> <50AC05BD.3080402@udel.edu> Message-ID: <20121120184107.16713a99@limelight.wooz.org> On Nov 20, 2012, at 05:35 PM, Terry Reedy wrote: >On 11/20/2012 3:35 PM, barry.warsaw wrote: > >> for entry in path: >> + if not isinstance(entry, (str, bytes)): >> + continue > >Given that a non-(str,bytes) entry could indicate a programming error, should >a warning be emitted before continuing? That's not what happens in Python 3.2. In fact, this bug report was triggered by someone who was already inserting None to sys.path[0]. It is silently ignored in Python 3.2, but tracebacked in 3.3. Cheers, -Barry From dholth at gmail.com Wed Nov 21 00:43:32 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 18:43:32 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <loom.20121120T231553-707@post.gmane.org> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <loom.20121120T231553-707@post.gmane.org> Message-ID: <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> No. We trust the packages we install, including the way they decide to use the metadata. A bad package could delete all our files or cause dependency resolution to fail. Mostly they won't. Daniel Holth On Nov 20, 2012, at 5:26 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote: > Daniel Holth <dholth <at> gmail.com> writes: > >> They mean pretty much what the same words mean in RPM and do not need further >> bikeshedding. > > But isn't it the case that the scenarios are different because in the case of > RPMs, we have a presumed authority which can determine e.g. what obsoletes what, > whereas with Python distributions, there's no central authority that has this > function? > > Regards, > > Vinay Sajip > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com From a.badger at gmail.com Wed Nov 21 01:12:08 2012 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 20 Nov 2012 16:12:08 -0800 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> References: <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <loom.20121120T231553-707@post.gmane.org> <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> Message-ID: <20121121001208.GW2133@unaka.lan> On Tue, Nov 20, 2012 at 06:43:32PM -0500, Daniel Holth wrote: > No. We trust the packages we install, including the way they decide to use > the metadata. A bad package could delete all our files or cause dependency > resolution to fail. Mostly they won't. > Agreed. And this is closer to the way that distributions' tools have to operate than they'd want to :-( Within the distribution we like to pretend that we only need to care about the packages that we generate. But we also know that whether or not we support it, ordinary users will install pacakges from outside of our walls. That means that the packaging tools that we create will need to deal with things that we might not condone within our "presumed authority". We trust that people are going to do more or less the right thing with the tools we offer. Once in a while they don't but by and large they do. -Toshio > Daniel Holth > > On Nov 20, 2012, at 5:26 PM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote: > > > Daniel Holth <dholth <at> gmail.com> writes: > > > >> They mean pretty much what the same words mean in RPM and do not need further > >> bikeshedding. > > > > But isn't it the case that the scenarios are different because in the case of > > RPMs, we have a presumed authority which can determine e.g. what obsoletes what, > > whereas with Python distributions, there's no central authority that has this > > function? > > > > Regards, > > > > Vinay Sajip > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/a.badger%40gmail.com -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/dee0fea7/attachment.pgp> From v+python at g.nevcal.com Wed Nov 21 01:13:58 2012 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 20 Nov 2012 16:13:58 -0800 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> Message-ID: <50AC1CC6.1010100@g.nevcal.com> On 11/20/2012 1:48 PM, PJ Eby wrote: > (We've been over this before, the last time this discussion came up on > the Distutils-SIG for a previous Metadata PEP a year or two back, but > here goes....) Thanks. I wasn't over there. Makes it clear that clarifying PEPs to reflect discussions is a good idea. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/529d3790/attachment.html> From jimjjewett at gmail.com Wed Nov 21 01:18:24 2012 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 20 Nov 2012 19:18:24 -0500 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> Message-ID: <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> On 11/20/12, Daniel Holth <dholth at gmail.com> wrote: > On Tue, Nov 20, 2012 at 3:58 PM, Jim J. Jewett <jimjjewett at gmail.com> > wrote: >> Vinay Sajip reworded the 'Provides-Dist' definition to explicitly say: >> > The use of multiple names in this field *must not* be used for >> > bundling distributions together. It is intended for use when >> > projects are forked and merged over time ... >> (1) Then how *should* the "bundle-of-several-components" case be >> represented? > The useful way to bundle a bunch of things would be to just include them > all in an executable folder or zipfile with __main__.py. PEP 426 and the > package database would not get involved. The bundle would be distributed as > an application you can download and use, not as an sdist on PyPI. When I look at, for example, twisted, there are some fairly fine distinctions. I can imagine some people wanting to handle each little piece differently, since that is the level at which they would be replaced by a more efficient implementation. That doesn't mean that someone using the default should have to manage 47 separate little packages individually. Also note that ZODB is mentioned as a bundling example in the current (2012-11-14) PEP. What does the PEP recommend that they do? Stop including transaction? Keep including it but stop 'Provides-Dist'-ing it? The current PEP also specifies that "This field must include the project identified in the Name field, followed by the version : Name (Version)." but the examples do not always include version. Why is the MUST there? Is there some way to distinguish between concrete and abstract provisions? For example, if MyMail (2012.11.10) includes 'Provides-Dist: email', does that really get parsed as 'Provides-Dist: email (2012.11.10)'? >> (2) How is 'Provides-Dist' different from 'Obsoletes-Dist'? >> The only difference I can see is that it may be a bit more polite >> to people who do want to install multiple versions of a (possibly >> abstract) package. > The intent of Provides and Obsoletes is different. Obsoletes would not > satisfy a requirement during dependency resolution. > The RPM guide explains a similar system: As best I can understand, Obsoletes means "Go ahead and uninstall that other package." Saying that *without* providing the same functionality seems like a sneaky spelling of "Please break whatever relies on that other package." I'm willing to believe that there is a more useful meaning. I'm also willing to believe that they are logically redundant but express different intentions. The current wording doesn't tell me which is true. (Admittedly, that is arguably an upstream bug with other package systems, but you should still either fix it or explicitly delegate the definitions.) And as long as I'm asking for clarification, can foopkg-3.4 obsolete foopgk3.2? If not, is it a semantics problem, or just not idiomatic? If so, does it have a precise meaning, such as "no longer interoperates with"? And now that I've looked more carefully ... Can a "Key: Value" pair be continued onto another line? The syntax description under "Metadata Files" does not say so, but later text suggests that either leading whitespace or a leading tab specifically (from the example code) will work. (And is description a special case?) Is the payload assumed to be utf8 text? Can it be itself a mime message? Are there any restrictions on 'Name'? e.g., Can the name include spaces? line breaks? Must it be a valid python identifier? A valid python qualname? 'Version' says that it must be in the format specified in PEP 386. Unfortunately, it doesn't say which part of 386. Do you mean that it must be acceptable to verlib.NormalizedVersion without first having to call suggest_normalized_version? 'Summary' specifies that it must be one line. Is there a character limit, or do you just mean "no line breaks"? Do you want to add a "Should be less than 80 characters" or some such, based on typical tool presentation? Would it be worth repeating the advice that longer descriptions should go in the payload, after all headers? (Otherwise, they have to find 'Description' *and* notice that it is deprecated and figure out what to do instead.) Under 'Description', it isn't entirely clear whether what terminates the field. "Multiple paragraphs" suggests that there can be multiple lines, but I'm guessing that -- in practice -- they have to be a single logical line, with all but the first starting with whitespace. Under 'Classifier', is PEP 301 really the current authority for classifiers? I would prefer at least a reference to http://pypi.python.org/pypi?%3Aaction=list_classifiers demonstrating which classifiers are currently meaningful. Under 'Requires-Dist', there is an unclosed parenthesis. Does the 'Setup-Requires-Dist' set implicitly include the 'Requires-Dist' set, or should a package be listed both ways if it is required at both setup and runtime? The Summary of Differences from PEP 345 mentions changes to Requires-Dist, but I don't know what they were -- even the unclosed parentheses seemed the same. The appendix gives code for generating and parsing continuation lines that suggests the continuation whitespace is exactly one tab -- is other whitespace OK too? -jJ From pje at telecommunity.com Wed Nov 21 03:44:54 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 20 Nov 2012 21:44:54 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> Message-ID: <CALeMXf7JzAFqocJbjt_VUYnrn5KT_hKQ6iryOYFkv8ZUsVz-Jg@mail.gmail.com> On Tue, Nov 20, 2012 at 5:01 PM, Daniel Holth <dholth at gmail.com> wrote: > http://www.python.org/dev/peps/pep-0314/ says: > > The most common use of this field will be in case a package name > changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. > When you install Torqued Python, the Gorgon package should be > removed. > > Example: > > Obsoletes: Gorgon > > > They mean pretty much what the same words mean in RPM and do not need > further bikeshedding. > The problem is that the above *makes no sense*. "Torqued Python" and "Gorgon" are veiled pseudonyms for Twisted and Medusa.... and Twisted is not actually a plug-and-play substitute for Medusa, AFAIK. Can anybody suggest an *actual* use case for "Obsoletes", and explain how it is supposed to work in software? The last time this discussion came up, nobody had any use cases that stood up to the "how's that actually going to work and/or help?" test. Here's a post of mine summarizing this and related points in the previous thread: http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/d84d27cf/attachment.html> From dholth at gmail.com Wed Nov 21 04:01:48 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 22:01:48 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf7JzAFqocJbjt_VUYnrn5KT_hKQ6iryOYFkv8ZUsVz-Jg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <CALeMXf7JzAFqocJbjt_VUYnrn5KT_hKQ6iryOYFkv8ZUsVz-Jg@mail.gmail.com> Message-ID: <CAG8k2+5AayaYpWgLAS=qMOnmvJFYUdKm4tHWeTvKn_EGLfW2fQ@mail.gmail.com> On Tue, Nov 20, 2012 at 9:44 PM, PJ Eby <pje at telecommunity.com> wrote: > On Tue, Nov 20, 2012 at 5:01 PM, Daniel Holth <dholth at gmail.com> wrote: > > http://www.python.org/dev/peps/pep-0314/ says: >> >> The most common use of this field will be in case a package name >> changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. >> When you install Torqued Python, the Gorgon package should be >> removed. >> >> Example: >> >> Obsoletes: Gorgon >> >> >> They mean pretty much what the same words mean in RPM and do not need >> further bikeshedding. >> > > The problem is that the above *makes no sense*. "Torqued Python" and > "Gorgon" are veiled pseudonyms for Twisted and Medusa.... and Twisted is > not actually a plug-and-play substitute for Medusa, AFAIK. > > Can anybody suggest an *actual* use case for "Obsoletes", and explain how > it is supposed to work in software? The last time this discussion came up, > nobody had any use cases that stood up to the "how's that actually going to > work and/or help?" test. Here's a post of mine summarizing this and > related points in the previous thread: > > Again I didn't write any of this. Someone mentioned ZODB + transaction. The PEP should have used the word "merged" instead of "bundled". When two packages become one, and the redundant package is no longer being developed, Provides-Dist can be used. I re-named a package once just because I did not like the name. I used "Obsoletes" for that. It is documentation. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/19c6484d/attachment.html> From pje at telecommunity.com Wed Nov 21 04:10:58 2012 From: pje at telecommunity.com (PJ Eby) Date: Tue, 20 Nov 2012 22:10:58 -0500 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <loom.20121120T231553-707@post.gmane.org> <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> Message-ID: <CALeMXf4m6_oFWXTdHffLmFdzkreSoCtrrvDxTTx0OtR3g7QaCg@mail.gmail.com> On Tue, Nov 20, 2012 at 6:43 PM, Daniel Holth <dholth at gmail.com> wrote: > No. We trust the packages we install, including the way they decide to use > the metadata. A bad package could delete all our files or cause dependency > resolution to fail. Mostly they won't. > That's sort of beside the point. The *only* use case which Obsoletes provides over Obsoleted-By is that it allows third parties to unilaterally advertise their forked project as a substitute for the original, and maybe block users from switching back to the un-forked project -- regardless of the status of the original project or the consent of the original project's maintainer. This use case, however, benefits nobody besides the forkers. There are many other legitimate channels by which the forkers can advertise themselves as a replacement for their parent project, and no reason for the installing end user to be bothered with the subject, except in case of a conflict. For somebody obsoleting their own package, on the other hand, it's likely well worth the effort to at least update their PyPI metadata to reflect the change in status -- especially if this can be done through the web interface. It's likely they would wish to update their description as well, to notify human beings of the change. But here's the thing that kills "Obsoletes" dead in the first place as a practical tool: unless installers use a PyPI search before installing *every single project*, there is no way for them to realize that the obsoleting package exists! By contrast, if a package is "Obsoleted-By", then installing that package (or declaring a dependency on it) provides an opportunity to inform the user of the need to make a transition. This can't be done with an "Obsoletes" field. Conversely, if you have already installed a package that says it "Obsoletes" another package, this does *not* tell you that the obsolete package shouldn't still be installed! A replacement project doesn't necessarily share the same API, and may exist in a different package namespace altogether. In short, "Obsoletes" is virtually *useless* as a machine-consumed metadata field, because there is nothing you can actually do with it in a practical installer. I'm against adding more fields to the metadata which do not have a specification for how they should be used in practice; the presence of such fields has been a problem with most of the preceding metadata specs, IMO. > I re-named a package once just because I did not like the name. I used "Obsoletes" for that. It is documentation. Note that "Obsoleted-By" would also serve that use case, and have the additional benefit of being able to notify people who install new copies of the replaced project. (By the way Daniel, I'm sorry I didn't comment on this PEP sooner; I'd forgotten about the previous PEP 345 rehashing in 2010, or rather, I just sort of assumed that the results of that discussion had been incorporated into the newer PEP, and didn't notice the reappearance of the noise fields until your call for approval just now. Sorry!) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/51defd4e/attachment-0001.html> From ncoghlan at gmail.com Wed Nov 21 04:12:18 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 13:12:18 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf7JzAFqocJbjt_VUYnrn5KT_hKQ6iryOYFkv8ZUsVz-Jg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <CALeMXf7JzAFqocJbjt_VUYnrn5KT_hKQ6iryOYFkv8ZUsVz-Jg@mail.gmail.com> Message-ID: <CADiSq7d4SP09O6y72u5MN9_K0VHkEMYTB60XvYK_4P0A6xBGdA@mail.gmail.com> On Wed, Nov 21, 2012 at 12:44 PM, PJ Eby <pje at telecommunity.com> wrote: > > Can anybody suggest an *actual* use case for "Obsoletes", and explain how > it is supposed to work in software? The last time this discussion came up, > nobody had any use cases that stood up to the "how's that actually going to > work and/or help?" test. Here's a post of mine summarizing this and > related points in the previous thread: > > http://mail.python.org/pipermail/catalog-sig/2010-October/003364.html > The problem is that the above *makes no sense*. "Torqued Python" and > "Gorgon" are veiled pseudonyms for Twisted and Medusa.... and Twisted is > not actually a plug-and-play substitute for Medusa, AFAIK. > Sure. This is an RPM example, but exactly the same thing applies at the Python level. One of the dependencies of PulpDist (a directory mirroring tool I wrote), is the Pulp project (originally just an RPM mirroring tool, but now with plugin-based support for mirroring other things). The upstream version of Pulp that I currently use is missing Kerberos login support, so I have patched that in via RPMs patching features. To avoid messing up others sharing the internal yum repo where this is published, I actually use the Provides/Conflict/Obsoletes features of RPM to make sure my patched and renamed copy and the upstream version don't interfere with each other (and certainly can't be installed on the same system, as they would trample all over each other by attempting to install the same files). Mostly though, these labelling tools are especially useful for internal forks and mergers - the ones you *don't* share with the wider internet, except perhaps in the form of upstream patches (For example: https://bugzilla.redhat.com/show_bug.cgi?id=831937). On a public index, drop-in replacements are *always* going to be controversial from a social point of view, which is why there are only two current examples on PyPI I am aware of (i.e. distribute vs setuptools and Pillow vs PIL). The first was essentially a hostile fork, while the latter started as an attempt to provide decent packaging support when the current maintainer didn't show any interest in doing so. In such cases, it is absolutely essential that the *forking* project is able to declare that it is a replacement for the original project. It is then up to the community to decide whether or not the claims of being a suitable replacement are valid, which will be shown most clearly in relative uptake numbers between the original project and the forked one. I do consider it unfortunate that Python has only copied 3 of the 4 RPM dependency management fields (i.e. only Provides, Requires, and Obsoletes, without copying the more value neutral Conflicts) and I also prefer the "capability" terminology in the Fedora RPM guide that makes it clear that these are really arbitrary strings from a tooling point of view that only match the package name by convention. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/5a18c4b9/attachment.html> From dholth at gmail.com Wed Nov 21 04:20:01 2012 From: dholth at gmail.com (Daniel Holth) Date: Tue, 20 Nov 2012 22:20:01 -0500 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> Message-ID: <CAG8k2+60zucMTDVVDL0pW2thZh=rYEpMu_BFPDJG+MMd79xArg@mail.gmail.com> On Tue, Nov 20, 2012 at 7:18 PM, Jim Jewett <jimjjewett at gmail.com> wrote: > On 11/20/12, Daniel Holth <dholth at gmail.com> wrote: > > On Tue, Nov 20, 2012 at 3:58 PM, Jim J. Jewett <jimjjewett at gmail.com> > > wrote: > > >> Vinay Sajip reworded the 'Provides-Dist' definition to explicitly say: > > >> > The use of multiple names in this field *must not* be used for > >> > bundling distributions together. It is intended for use when > >> > projects are forked and merged over time ... > > >> (1) Then how *should* the "bundle-of-several-components" case be > >> represented? > > > The useful way to bundle a bunch of things would be to just include them > > all in an executable folder or zipfile with __main__.py. PEP 426 and the > > package database would not get involved. The bundle would be distributed > as > > an application you can download and use, not as an sdist on PyPI. > > When I look at, for example, twisted, there are some fairly fine > distinctions. > > I can imagine some people wanting to handle each little piece > differently, since that is the level at which they would be replaced > by a more efficient implementation. That doesn't mean that someone > using the default should have to manage 47 separate little packages > individually. > > Also note that ZODB is mentioned as a bundling example in the current > (2012-11-14) PEP. What does the PEP recommend that they do? Stop > including transaction? Keep including it but stop 'Provides-Dist'-ing > it? > > The current PEP also specifies that "This field must include the > project identified in the Name field, followed by the version : Name > (Version)." but the examples do not always include version. Why is > the MUST there? > ZODB is a bad example. The word "bundling" will be struck from the PEP entirely. Two sdists should not be combined into one sdist when both packages are still being developed. If A and B are merged into a single PyPI package C, and A and B will no longer be developed, then C may Provides-Dist A and B. http://www.python.org/dev/peps/pep-0426/#requires-dist-multiple-use No MUST on the Requires-Dist version. If no version is there, it should satisfy any version requirement. Is there some way to distinguish between concrete and abstract > provisions? For example, if MyMail (2012.11.10) includes > 'Provides-Dist: email', does that really get parsed as 'Provides-Dist: > email (2012.11.10)'? > No. >> (2) How is 'Provides-Dist' different from 'Obsoletes-Dist'? >> The only difference I can see is that it may be a bit more polite >> to people who do want to install multiple versions of a (possibly >> abstract) package. > The intent of Provides and Obsoletes is different. Obsoletes would not > > satisfy a requirement during dependency resolution. > > > The RPM guide explains a similar system: > > As best I can understand, Obsoletes means "Go ahead and uninstall that > other package." > Saying that *without* providing the same functionality seems like a > sneaky spelling of "Please break whatever relies on that other > package." > > I'm willing to believe that there is a more useful meaning. I'm also > willing to believe that they are logically redundant but express > different intentions. The current wording doesn't tell me which is > true. (Admittedly, that is arguably an upstream bug with other > package systems, but you should still either fix it or explicitly > delegate the definitions.) > > And as long as I'm asking for clarification, can foopkg-3.4 obsolete > foopgk3.2? If not, is it a semantics problem, or just not idiomatic? > If so, does it have a precise meaning, such as "no longer > interoperates with"? > When I used Obsoletes, it meant "I am no longer developing this other package that is identical to this re-named package". The system of requirements/conflicts (as the RPM system) does not appear to be entirely orthogonal. And now that I've looked more carefully ... > > Can a "Key: Value" pair be continued onto another line? The syntax > description under "Metadata Files" does not say so, but later text > suggests that either leading whitespace or a leading tab specifically > (from the example code) will work. (And is description a special > case?) > Description (now in the payload, please) is the only field that is commonly multi-line. Any field could continue onto the next line as far as the parser is concerned. It probably would not make sense. Is the payload assumed to be utf8 text? Can it be itself a mime message? > The entire file needs to be utf-8. The payload is assumed to be utf-8 text in this version. Wouldn't a mime message also be utf-8 text? (we wouldn't know what to do with it) > Are there any restrictions on 'Name'? e.g., Can the name include > spaces? line breaks? Must it be a valid python identifier? A valid > python qualname? > setuptools constrains it to alphanumeric characters. Metadata 1.3 doesn't say. 'Version' says that it must be in the format specified in PEP 386. > Unfortunately, it doesn't say which part of 386. Do you mean that it > must be acceptable to verlib.NormalizedVersion without first having to > call suggest_normalized_version? > It means it is expected to match: http://www.python.org/dev/peps/pep-0386/#the-new-versioning-algorithm expr = r"""^ (?P<version>\d+\.\d+) # minimum 'N.N' (?P<extraversion>(?:\.\d+)*) # any number of extra '.N' segments (?: (?P<prerel>[abc]|rc) # 'a' = alpha, 'b' = beta # 'c' or 'rc' = release candidate (?P<prerelversion>\d+(?:\.\d+)*) )? (?P<postdev>(\.post(?P<post>\d+))?(\.dev(?P<dev>\d+))?)? $""" Please do not enforce this regexp in your Metadata parser. > 'Summary' specifies that it must be one line. Is there a character > limit, or do you just mean "no line breaks"? Do you want to add a > "Should be less than 80 characters" or some such, based on typical > tool presentation? Would it be worth repeating the advice that longer > descriptions should go in the payload, after all headers? (Otherwise, > they have to find 'Description' *and* notice that it is deprecated and > figure out what to do instead.) > > Under 'Description', it isn't entirely clear whether what terminates > the field. "Multiple paragraphs" suggests that there can be multiple > lines, but I'm guessing that -- in practice -- they have to be a > single logical line, with all but the first starting with whitespace. > Non-blank lines that don't start with whitespace are keys. email.parser.Parser() takes care of this in an e-mail-inspired (but not any literal RFC) way. The distutils documentation has guidelines on the short description / summary. > Under 'Classifier', is PEP 301 really the current authority for > classifiers? I would prefer at least a reference to > http://pypi.python.org/pypi?%3Aaction=list_classifiers demonstrating > which classifiers are currently meaningful. > > Under 'Requires-Dist', there is an unclosed parenthesis. > > Does the 'Setup-Requires-Dist' set implicitly include the > 'Requires-Dist' set, or should a package be listed both ways if it is > required at both setup and runtime? > > The Summary of Differences from PEP 345 mentions changes to > Requires-Dist, but I don't know what they were -- even the unclosed > parentheses seemed the same. > The addition of the allowed "extra" variable in the ; condition is the most significant change. > The appendix gives code for generating and parsing continuation lines > that suggests the continuation whitespace is exactly one tab -- is > other whitespace OK too? > Any whitespace would work. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121120/2726fab8/attachment.html> From ncoghlan at gmail.com Wed Nov 21 04:20:41 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 13:20:41 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CALeMXf4m6_oFWXTdHffLmFdzkreSoCtrrvDxTTx0OtR3g7QaCg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <loom.20121120T231553-707@post.gmane.org> <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> <CALeMXf4m6_oFWXTdHffLmFdzkreSoCtrrvDxTTx0OtR3g7QaCg@mail.gmail.com> Message-ID: <CADiSq7ehwkeu7AzVfR1Gy=m0BQQTkgHXs6yyHoTBCds11i5SZg@mail.gmail.com> On Wed, Nov 21, 2012 at 1:10 PM, PJ Eby <pje at telecommunity.com> wrote: > Conversely, if you have already installed a package that says it > "Obsoletes" another package, this does *not* tell you that the obsolete > package shouldn't still be installed! A replacement project doesn't > necessarily share the same API, and may exist in a different package > namespace altogether. > Then that's a bug in the metadata of the project misusing "Obsoletes", and should be reported as such. If the new package is not a drop-in replacement, then it has no business claiming to obsolete the other package. I think one of the big reasons this kind of use is rare in the Python community is that project name changes are almost always accompanied by *package* name changes, and as soon as you change the package name, you're changing the public API, and thus it is no longer appropriate to use Provides or Obsoletes, as the renamed project is no longer a drop-in replacement for the original. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/5db6a035/attachment.html> From sdl.web at gmail.com Wed Nov 21 04:57:20 2012 From: sdl.web at gmail.com (Leo) Date: Wed, 21 Nov 2012 11:57:20 +0800 Subject: [Python-Dev] operator.attrgetter(attr[, args...]) etc. Message-ID: <m2ehjn4nhb.fsf@gmail.com> Sorry the python issue tracker seems broken (I cannot log in). So I am posting it here. In the doc: operator.attrgetter(attr[, args...]) operator.itemgetter(item[, args...]) operator.methodcaller(name[, args...]) The signatures of these functions seem confusing. ARGS is not documented and does not match the python implementation in the doc. Leo From ncoghlan at gmail.com Wed Nov 21 05:01:24 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Nov 2012 14:01:24 +1000 Subject: [Python-Dev] Accept just PEP-0426 In-Reply-To: <CADiSq7ehwkeu7AzVfR1Gy=m0BQQTkgHXs6yyHoTBCds11i5SZg@mail.gmail.com> References: <CAG8k2+4SyyD+Ht73CDbYf54PEL79R1jpxSqwg+c56sGg_-4pNw@mail.gmail.com> <CALeMXf6ZWPuoQ0AJ7oEmngNVdKbRZch29UOneg+eHuzUWnBA9g@mail.gmail.com> <98BC16BE-3507-4D54-9AFC-8B1A0983B6F0@gmail.com> <4AB62E69A53048589648EA9A94CAC02A@gmail.com> <B7D9A3F9-29DF-4389-ADCF-50E8291EF6F7@gmail.com> <DBA510759A4D450A9A73C7E70EFE42CD@gmail.com> <CAG8k2+6F5G+=M5RUF8qpo-Kxyy2hKL0xumsOy7WXLttHS3LEjA@mail.gmail.com> <loom.20121120T150418-29@post.gmane.org> <CAG8k2+5ALzzCspea2NtjQW5KPqnzUOade33x0s_cPw1KxeUnWQ@mail.gmail.com> <loom.20121120T160434-5@post.gmane.org> <CADiSq7eNeh+AU2yyfE=QD1MVAsQkaOb6iFiitRNpgH1ukmtgew@mail.gmail.com> <loom.20121120T165425-38@post.gmane.org> <CAG8k2+7yB+h-wEK3abKiJq_vLN2uuHGKFwVh_YKCjnTSweD3sQ@mail.gmail.com> <loom.20121120T172255-830@post.gmane.org> <CALeMXf7EtrORHEq8GYQ0nXo52n5b2L0-KQRJAgJSiX101egiCw@mail.gmail.com> <50ABF128.4020605@g.nevcal.com> <CALeMXf6nK+0k091=BjmZuX+7LmPHrA1ibF5dru_P_k85fHwCDg@mail.gmail.com> <C616B797714D4BB397ACAB9EC81B1C51@gmail.com> <CAG8k2+7DOcYxDo1vj_Fh5RE3Jbrmb3iQU_Tr_T59JaB5d_uNwA@mail.gmail.com> <loom.20121120T231553-707@post.gmane.org> <B3C5A7B3-CF5F-44C3-863D-A7B4402F06CE@gmail.com> <CALeMXf4m6_oFWXTdHffLmFdzkreSoCtrrvDxTTx0OtR3g7QaCg@mail.gmail.com> <CADiSq7ehwkeu7AzVfR1Gy=m0BQQTkgHXs6yyHoTBCds11i5SZg@mail.gmail.com> Message-ID: <CADiSq7epB_dkia-HzVjGXNk5p0_6PjE3Y6v1kW18+=SCZrW8Pw@mail.gmail.com> On Wed, Nov 21, 2012 at 1:20 PM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Wed, Nov 21, 2012 at 1:10 PM, PJ Eby <pje at telecommunity.com> wrote: > >> Conversely, if you have already installed a package that says it >> "Obsoletes" another package, this does *not* tell you that the obsolete >> package shouldn't still be installed! A replacement project doesn't >> necessarily share the same API, and may exist in a different package >> namespace altogether. >> > > Then that's a bug in the metadata of the project misusing "Obsoletes", and > should be reported as such. If the new package is not a drop-in > replacement, then it has no business claiming to obsolete the other package. > > I think one of the big reasons this kind of use is rare in the Python > community is that project name changes are almost always accompanied by > *package* name changes, and as soon as you change the package name, you're > changing the public API, and thus it is no longer appropriate to use > Provides or Obsoletes, as the renamed project is no longer a drop-in > replacement for the original. > I realised that my comments above are more about the appropriate use of "Provides", rather than "Obsoletes". For a practically useful "Obsoletes", I think I'm inclined to agree with you, as "Obsoleted-By" provides a way for a maintainer to explicitly declare that a project is no longer receiving updates, and users should migrate to the replacement project if they want to continue to receive fixes and improvements. The current version of "Obsoletes" is, as Daniel describes, really only useful as documentation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/71924996/attachment.html> From stephen at xemacs.org Wed Nov 21 06:00:43 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 21 Nov 2012 14:00:43 +0900 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <CAG8k2+60zucMTDVVDL0pW2thZh=rYEpMu_BFPDJG+MMd79xArg@mail.gmail.com> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> <CAG8k2+60zucMTDVVDL0pW2thZh=rYEpMu_BFPDJG+MMd79xArg@mail.gmail.com> Message-ID: <87pq375z44.fsf@uwakimon.sk.tsukuba.ac.jp> Daniel Holth writes: > When I used Obsoletes, it meant "I am no longer developing this other > package that is identical to this re-named package". But as a user I could care less! The authors may care, but I don't care if Torqued "obsoletes" Gorgon, because in using Torqued I'm DTRT'ing even though I don't know it. What I care about is when I'm using Gorgon, and there's something "better" (or worse, "correct") to use in my application. It might be a good idea to have a just-like-Amazon While-This-Package-Is-Great-You-Might-Also-Consider: field. Tongue-in-cheeki-ly y'rs, From pje at telecommunity.com Wed Nov 21 07:33:19 2012 From: pje at telecommunity.com (PJ Eby) Date: Wed, 21 Nov 2012 01:33:19 -0500 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <87pq375z44.fsf@uwakimon.sk.tsukuba.ac.jp> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> <CAG8k2+60zucMTDVVDL0pW2thZh=rYEpMu_BFPDJG+MMd79xArg@mail.gmail.com> <87pq375z44.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <CALeMXf6cGSycaCwos5=Pk8M9qd22OqrxTcvkBFKLVC8axSHvCw@mail.gmail.com> On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > Daniel Holth writes: > > > When I used Obsoletes, it meant "I am no longer developing this other > > package that is identical to this re-named package". > > But as a user I could care less! The authors may care, but I don't > care if Torqued "obsoletes" Gorgon, because in using Torqued I'm > DTRT'ing even though I don't know it. > > What I care about is when I'm using Gorgon, and there's something > "better" (or worse, "correct") to use in my application. > Hence my suggestion for an Obsoleted-By field, in which Gorgon would be able to suggest alternatives. > It might be a good idea to have a just-like-Amazon > > While-This-Package-Is-Great-You-Might-Also-Consider: > > field. > Yeah, that's basically what Obsoleted-By is for. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/183b4d96/attachment.html> From stephen at xemacs.org Wed Nov 21 08:54:17 2012 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 21 Nov 2012 16:54:17 +0900 Subject: [Python-Dev] Keyword meanings [was: Accept just PEP-0426] In-Reply-To: <CALeMXf6cGSycaCwos5=Pk8M9qd22OqrxTcvkBFKLVC8axSHvCw@mail.gmail.com> References: <loom.20121120T172255-830@post.gmane.org> <50abef0e.ab96320a.6675.ffffb439@mx.google.com> <CAG8k2+6Q1r+i__Ea29yawxBJztY1F7iGGKYj78TEVJatVYFeXg@mail.gmail.com> <CA+OGgf7nh5Zc1eKSE9YwF5Gz0iP7PJBi7swJczcUkbD7gKbA6w@mail.gmail.com> <CAG8k2+60zucMTDVVDL0pW2thZh=rYEpMu_BFPDJG+MMd79xArg@mail.gmail.com> <87pq375z44.fsf@uwakimon.sk.tsukuba.ac.jp> <CALeMXf6cGSycaCwos5=Pk8M9qd22OqrxTcvkBFKLVC8axSHvCw@mail.gmail.com> Message-ID: <87obir5r2u.fsf@uwakimon.sk.tsukuba.ac.jp> PJ Eby writes: > On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull <stephen at xemacs.org>wrote: > > What I care about is when I'm using Gorgon, and there's something > > "better" (or worse, "correct") to use in my application. > > Hence my suggestion for an Obsoleted-By field, in which Gorgon would be > able to suggest alternatives. My bad, my precise intention was to follow up on your idea (which, credit where credit is due, I had *not* hit upon independently). I should have made that clear. (I really shouldn't be answering English email at a Japanese-speaking conference, my brain thinks it knows what it's doing but shirazuni ? ????????....) > > It might be a good idea to have a just-like-Amazon > > > > While-This-Package-Is-Great-You-Might-Also-Consider: > > > > field. > > Yeah, that's basically what Obsoleted-By is for. Well, Obsoleted-By is pretty strong language for suggesting possible alternatives. But I suspect that few projects would really want to be suggesting competitors' products *or* their own oldie-but-still-goodie that they'd really like to obsolete ASAP (put an Obsoleted-By line in every Python 2 distribution, anyone? :-) From oscar.j.benjamin at gmail.com Wed Nov 21 15:49:05 2012 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 21 Nov 2012 14:49:05 +0000 Subject: [Python-Dev] operator.attrgetter(attr[, args...]) etc. In-Reply-To: <m2ehjn4nhb.fsf@gmail.com> References: <m2ehjn4nhb.fsf@gmail.com> Message-ID: <CAHVvXxSO1o_-5tzeQpccP9VbkYOuLExRmyf-_F-NmcPwVq4qSw@mail.gmail.com> On 21 November 2012 03:57, Leo <sdl.web at gmail.com> wrote: > Sorry the python issue tracker seems broken (I cannot log in). So I am > posting it here. > > In the doc: > > operator.attrgetter(attr[, args...]) > operator.itemgetter(item[, args...]) > operator.methodcaller(name[, args...]) > > The signatures of these functions seem confusing. ARGS is not documented > and does not match the python implementation in the doc. What documentation are you reading? $ python Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import operator >>> help(operator.attrgetter) Help on class attrgetter in module operator: class attrgetter(__builtin__.object) | attrgetter(attr, ...) --> attrgetter object | | Return a callable object that fetches the given attribute(s) from its operand. | After, f=attrgetter('name'), the call f(r) returns r.name. | After, g=attrgetter('name', 'date'), the call g(r) returns (r.name, r.date). | After, h=attrgetter('name.first', 'name.last'), the call h(r) returns | (r.name.first, r.name.last). [snip] I think the above explains how attrgetter works with multiple arguments. Here's a demo: >>> class X: ... a = 2 ... b = 3 ... c = 1 ... >>> getter = operator.attrgetter('a', 'b', 'c') >>> getter(X) (2, 3, 1) Oscar From rdmurray at bitdance.com Wed Nov 21 16:16:55 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 21 Nov 2012 10:16:55 -0500 Subject: [Python-Dev] operator.attrgetter(attr[, args...]) etc. In-Reply-To: <CAHVvXxSO1o_-5tzeQpccP9VbkYOuLExRmyf-_F-NmcPwVq4qSw@mail.gmail.com> References: <m2ehjn4nhb.fsf@gmail.com> <CAHVvXxSO1o_-5tzeQpccP9VbkYOuLExRmyf-_F-NmcPwVq4qSw@mail.gmail.com> Message-ID: <20121121151655.BE4F32500E1@webabinitio.net> On Wed, 21 Nov 2012 14:49:05 +0000, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote: > On 21 November 2012 03:57, Leo <sdl.web at gmail.com> wrote: > > Sorry the python issue tracker seems broken (I cannot log in). So I am > > posting it here. > > > > In the doc: > > > > operator.attrgetter(attr[, args...]) > > operator.itemgetter(item[, args...]) > > operator.methodcaller(name[, args...]) > > > > The signatures of these functions seem confusing. ARGS is not documented > > and does not match the python implementation in the doc. > > What documentation are you reading? He's reading the operator module documentation. I just ran into that same thing the other day, but didn't think to file a bug report. It looks like the ``args`` stuff was incorrectly copy and pasted from the methodcaller docs, where it is actually correct. I've opened an issue: http://bugs.python.org/issue16523 --David From brett at python.org Wed Nov 21 17:33:40 2012 From: brett at python.org (Brett Cannon) Date: Wed, 21 Nov 2012 11:33:40 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.3): - Issue #16514: Fix regression causing a traceback when sys.path[0] is None In-Reply-To: <20121120184107.16713a99@limelight.wooz.org> References: <3Y5dwB3ZCMzMlG@mail.python.org> <50AC05BD.3080402@udel.edu> <20121120184107.16713a99@limelight.wooz.org> Message-ID: <CAP1=2W5Gj-q-UJb6Lp08a2ov4=fiHNPYoTkOy0Y_YCQ4e47N_w@mail.gmail.com> On Tue, Nov 20, 2012 at 6:41 PM, Barry Warsaw <barry at python.org> wrote: > On Nov 20, 2012, at 05:35 PM, Terry Reedy wrote: > > >On 11/20/2012 3:35 PM, barry.warsaw wrote: > > > >> for entry in path: > >> + if not isinstance(entry, (str, bytes)): > >> + continue > > > >Given that a non-(str,bytes) entry could indicate a programming error, > should > >a warning be emitted before continuing? > > That's not what happens in Python 3.2. Fine, but warnings are off by default and this is simply wrong behaviour to do. We should not paper over it just because import.c did. > In fact, this bug report was triggered > by someone who was already inserting None to sys.path[0]. It is silently > ignored in Python 3.2, but tracebacked in 3.3. But why were they doing that? What did they hope to get out of it? -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121121/9c3a1dc1/attachment.html> From chris.jerdonek at gmail.com Thu Nov 22 03:12:49 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 21 Nov 2012 18:12:49 -0800 Subject: [Python-Dev] [Python-checkins] devguide: Add instructions for handling merge conflicts during null merges. In-Reply-To: <3Y6PDh5mRgzMvQ@mail.python.org> References: <3Y6PDh5mRgzMvQ@mail.python.org> Message-ID: <CAOTb1wc+JqbaxMh7AoSyiNhKddfVsCrB2Q4DeiX=wsDiWx=rqw@mail.gmail.com> On Wed, Nov 21, 2012 at 6:07 PM, chris.jerdonek <python-checkins at python.org> wrote: > http://hg.python.org/devguide/rev/78a69b929ab7 > changeset: 573:78a69b929ab7 > user: Chris Jerdonek <chris.jerdonek at gmail.com> > date: Wed Nov 21 18:04:35 2012 -0800 > summary: > Add instructions for handling merge conflicts during null merges. This was for issue #16517: http://bugs.python.org/issue16517 --Chris > > files: > committing.rst | 11 +++++++++-- > 1 files changed, 9 insertions(+), 2 deletions(-) > > > diff --git a/committing.rst b/committing.rst > --- a/committing.rst > +++ b/committing.rst > @@ -306,17 +306,24 @@ > # Fix any conflicts; compile; run the test suite > hg commit > > +.. index:: null merging > + > .. note:: > - *If the patch shouldn't be ported* from Python 3.3 to Python 3.4, you must > - also make it explicit: merge the changes but revert them before committing:: > + If the patch should *not* be ported from Python 3.3 to Python 3.4, you must > + also make this explicit by doing a *null merge*: merge the changes but > + revert them before committing:: > > hg update default > hg merge 3.3 > hg revert -ar default > + hg resolve -am # needed only if the merge created conflicts > hg commit > > This is necessary so that the merge gets recorded; otherwise, somebody > else will have to make a decision about your patch when they try to merge. > + (Using a three-way merge tool generally makes the ``hg resolve`` step > + in the above unnecessary; also see `this bug report > + <http://bz.selenic.com/show_bug.cgi?id=2706>`__.) > > When you have finished your porting work (you can port several patches one > after another in your local repository), you can push **all** outstanding > > -- > Repository URL: http://hg.python.org/devguide > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From kristjan at ccpgames.com Thu Nov 22 12:44:26 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Thu, 22 Nov 2012 11:44:26 +0000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> Where in the tracker? I tried searching but didn't find it. I contributed to the pep405 discussions with similar concerns back in march: http://mail.python.org/pipermail/python-dev/2012-March/117894.html From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf Of Daniel Holth There is some activity in the tracker about adding the missing "add this to PYTHONPATH" / "isolate self from the environment" command line arguments to Python. Daniel Holth -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121122/407890da/attachment.html> From ncoghlan at gmail.com Thu Nov 22 13:09:10 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 22 Nov 2012 22:09:10 +1000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CADiSq7e0PhtkozwD0-4EP=hRs4V9M7GK8PoaWqz1moRvEeKJ6w@mail.gmail.com> On Thu, Nov 22, 2012 at 9:44 PM, Kristj?n Valur J?nsson < kristjan at ccpgames.com> wrote: > Where in the tracker? I tried searching but didn?t find it. > This one: http://bugs.python.org/issue13475 This and the issue about being able to configure coverage.py cleanly in subprocesses (http://bugs.python.org/issue14803) are the two issues which have got me thinking about the interpreter startup and configuration process in general lately. Both would add *more* complexity to an already complicated and hard to follow initialisation sequence, so I now think we should be look at opportunities for rationalisation *first*, before we make things even messier. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121122/b00a17de/attachment.html> From status at bugs.python.org Fri Nov 23 18:07:15 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 23 Nov 2012 18:07:15 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20121123170715.8D2531CAB2@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-11-16 - 2012-11-23) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3799 ( -2) closed 24518 (+54) total 28317 (+52) Open issues with patches: 1656 Issues opened (37) ================== #8627: Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c http://bugs.python.org/issue8627 reopened by mark.dickinson #16306: Multiple error line for unknown command line parameter http://bugs.python.org/issue16306 reopened by pitrou #16490: "inspect.getargspec()" and "inspect.getcallargs()" don't work http://bugs.python.org/issue16490 opened by jcea #16491: try-except-raise-bug http://bugs.python.org/issue16491 opened by Jocjo #16492: Add a load_parents argument to importlib.find_loader() http://bugs.python.org/issue16492 opened by brett.cannon #16494: Add a method on importlib.SourceLoader for creating bytecode f http://bugs.python.org/issue16494 opened by brett.cannon #16495: bytes_decode() unnecessarily examines encoding http://bugs.python.org/issue16495 opened by chris.jerdonek #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 opened by serhiy.storchaka #16499: CLI option for isolated mode http://bugs.python.org/issue16499 opened by christian.heimes #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 opened by christian.heimes #16501: deprecate RISCOS "support" http://bugs.python.org/issue16501 opened by pitrou #16504: IDLE - fatal error when opening a file with certain tokenizing http://bugs.python.org/issue16504 opened by serwy #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 opened by trent #16508: include the "object" type in the lists of documented types http://bugs.python.org/issue16508 opened by chris.jerdonek #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 opened by strcat #16510: Using appropriate checks in tests http://bugs.python.org/issue16510 opened by serhiy.storchaka #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 opened by doanviettrung #16512: imghdr doesn't support jpegs with an ICC profile http://bugs.python.org/issue16512 opened by joril #16515: TypeError message incorrect for max() with one keyword arg http://bugs.python.org/issue16515 opened by chris.jerdonek #16516: argparse types (and actions) must be hashable http://bugs.python.org/issue16516 opened by jnothman #16518: add "buffer protocol" to glossary http://bugs.python.org/issue16518 opened by chris.jerdonek #16519: site.venv() should use abspath(executable) http://bugs.python.org/issue16519 opened by christian.heimes #16520: subprocess.Popen() TypeError message incorrect without args ar http://bugs.python.org/issue16520 opened by chris.jerdonek #16521: logging.basicConfig creates empty file when using handlers http://bugs.python.org/issue16521 opened by Jovik #16523: attrgetter and itemgetter signatures in docs need cleanup http://bugs.python.org/issue16523 opened by r.david.murray #16524: File access not always working with Python for Windows 32 bits http://bugs.python.org/issue16524 opened by Antoine.Brodin #16525: wave file module does not support 32bit float format http://bugs.python.org/issue16525 opened by Sebastian.Kraft #16526: Python does not cross compile properly http://bugs.python.org/issue16526 opened by Lothsahn #16529: Compiler error when trying to compile ceval.c on OpenSUSE 11.3 http://bugs.python.org/issue16529 opened by lemburg #16530: documentation of os.wait3 http://bugs.python.org/issue16530 opened by quiver #16531: Allow IPNetwork to take a tuple http://bugs.python.org/issue16531 opened by pitrou #16532: AMD64 Windows 7 build failures http://bugs.python.org/issue16532 opened by pitrou #16533: HPUX: Unable to fork() in thread http://bugs.python.org/issue16533 opened by skrah #16534: test_float failure on IA64 (HPUX) http://bugs.python.org/issue16534 opened by skrah #16535: json encoder unable to handle decimal http://bugs.python.org/issue16535 opened by eric.araujo #16537: setup.py throws a ValueError when self.extensions is empty http://bugs.python.org/issue16537 opened by jhosmer #16540: Make itertools count, cycle, and repeat objects subscriptable http://bugs.python.org/issue16540 opened by neil.g Most recent 15 issues with no replies (15) ========================================== #16533: HPUX: Unable to fork() in thread http://bugs.python.org/issue16533 #16530: documentation of os.wait3 http://bugs.python.org/issue16530 #16523: attrgetter and itemgetter signatures in docs need cleanup http://bugs.python.org/issue16523 #16521: logging.basicConfig creates empty file when using handlers http://bugs.python.org/issue16521 #16519: site.venv() should use abspath(executable) http://bugs.python.org/issue16519 #16518: add "buffer protocol" to glossary http://bugs.python.org/issue16518 #16516: argparse types (and actions) must be hashable http://bugs.python.org/issue16516 #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 #16508: include the "object" type in the lists of documented types http://bugs.python.org/issue16508 #16494: Add a method on importlib.SourceLoader for creating bytecode f http://bugs.python.org/issue16494 #16492: Add a load_parents argument to importlib.find_loader() http://bugs.python.org/issue16492 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16472: Distutils+mingw links agains msvcr90, while python27.dll is li http://bugs.python.org/issue16472 #16463: test_timeout failure on the RHEL buildbot http://bugs.python.org/issue16463 #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 Most recent 15 issues waiting for review (15) ============================================= #16537: setup.py throws a ValueError when self.extensions is empty http://bugs.python.org/issue16537 #16515: TypeError message incorrect for max() with one keyword arg http://bugs.python.org/issue16515 #16512: imghdr doesn't support jpegs with an ICC profile http://bugs.python.org/issue16512 #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 #16510: Using appropriate checks in tests http://bugs.python.org/issue16510 #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 #16504: IDLE - fatal error when opening a file with certain tokenizing http://bugs.python.org/issue16504 #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 #16499: CLI option for isolated mode http://bugs.python.org/issue16499 #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 #16495: bytes_decode() unnecessarily examines encoding http://bugs.python.org/issue16495 #16488: Add context manager support to epoll object http://bugs.python.org/issue16488 #16487: Allow ssl certificates to be speficfied from memory rather tha http://bugs.python.org/issue16487 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16485: FD leaks in aifc module http://bugs.python.org/issue16485 Top 10 most discussed issues (10) ================================= #16475: Support object instancing and recursion in marshal http://bugs.python.org/issue16475 25 msgs #16496: Simplify and optimize random_seed() http://bugs.python.org/issue16496 18 msgs #15031: Split .pyc parsing from module loading http://bugs.python.org/issue15031 13 msgs #16500: Add an 'afterfork' module http://bugs.python.org/issue16500 13 msgs #16306: Multiple error line for unknown command line parameter http://bugs.python.org/issue16306 10 msgs #16499: CLI option for isolated mode http://bugs.python.org/issue16499 10 msgs #16480: pyvenv 3.3 fails to create symlinks for <virtualenv>/local/{bi http://bugs.python.org/issue16480 9 msgs #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 8 msgs #12848: pickle.py treats 32bit lengths as signed, but _pickle.c as uns http://bugs.python.org/issue12848 7 msgs #8627: Unchecked PyErr_WarnPy3k return value in Objects/typeobject.c http://bugs.python.org/issue8627 6 msgs Issues closed (44) ================== #1160: Medium size regexp crashes python http://bugs.python.org/issue1160 closed by pitrou #3580: failures in test_os http://bugs.python.org/issue3580 closed by pitrou #4182: warnings.warn shows the wrong filename and line number for sta http://bugs.python.org/issue4182 closed by brett.cannon #6308: termios fix for QNX breaks HP-UX http://bugs.python.org/issue6308 closed by skrah #6526: importlib.import_module affects permissions of .pyc files subs http://bugs.python.org/issue6526 closed by brett.cannon #7782: new test for test_iter.py http://bugs.python.org/issue7782 closed by ezio.melotti #9761: stale .pyc files aren't cleaned out http://bugs.python.org/issue9761 closed by brett.cannon #10966: eliminate use of ImportError implicitly representing SkipTest http://bugs.python.org/issue10966 closed by brett.cannon #11679: readline interferes with characters beginning with byte \xe9 http://bugs.python.org/issue11679 closed by takluyver #11981: dupe self.fp.tell() in zipfile.ZipFile.writestr http://bugs.python.org/issue11981 closed by ezio.melotti #12005: modulo result of Decimal differs from float/int http://bugs.python.org/issue12005 closed by mark.dickinson #12614: Allow to explicitly set the method of urllib.request.Request http://bugs.python.org/issue12614 closed by orsenthil #13538: Improve doc for str(bytesobject) http://bugs.python.org/issue13538 closed by chris.jerdonek #14313: zipfile should raise an exception for unsupported compression http://bugs.python.org/issue14313 closed by ezio.melotti #14525: ia64-hp-hpux11.31 won't compile 2.7 without -D_TERMIOS_INCLUDE http://bugs.python.org/issue14525 closed by skrah #14631: Instance methods and WeakRefs don't mix. http://bugs.python.org/issue14631 closed by pitrou #15627: Add a method to importlib.abc.SourceLoader for converting sour http://bugs.python.org/issue15627 closed by brett.cannon #16053: "strict" parameter is not documented in csv module http://bugs.python.org/issue16053 closed by ezio.melotti #16157: Irrelevant references to Misc/News http://bugs.python.org/issue16157 closed by ezio.melotti #16213: Expose private functions in marshal used by importlib http://bugs.python.org/issue16213 closed by brett.cannon #16215: Possible double memory free in str.replace http://bugs.python.org/issue16215 closed by pitrou #16240: Document a way to escape metacharacters in glob/fnmatch http://bugs.python.org/issue16240 closed by ezio.melotti #16408: FD leaks in zipfile http://bugs.python.org/issue16408 closed by pitrou #16451: Remove duplication between slice_indices and compute_slice_ind http://bugs.python.org/issue16451 closed by mark.dickinson #16461: wave module: wrong integer format http://bugs.python.org/issue16461 closed by python-dev #16470: Backport set and dictionary comprehensions in tutorial to 2.7 http://bugs.python.org/issue16470 closed by ezio.melotti #16489: importlib find_loader returns a loader for a non existent modu http://bugs.python.org/issue16489 closed by brett.cannon #16493: Document the 'optimize' argument to compile() http://bugs.python.org/issue16493 closed by ezio.melotti #16497: zipimport.zipimporter.find_module() does not work with dotted http://bugs.python.org/issue16497 closed by brett.cannon #16498: Unwanted link between volatile and shelve storage http://bugs.python.org/issue16498 closed by serhiy.storchaka #16502: PEP 305: eaten backslashes http://bugs.python.org/issue16502 closed by ezio.melotti #16503: Unclear documentation regarding apply(), 'extended call syntax http://bugs.python.org/issue16503 closed by ezio.melotti #16505: Drop Py_TPFLAGS_INT_SUBCLASS http://bugs.python.org/issue16505 closed by python-dev #16506: devguide should have table of contents http://bugs.python.org/issue16506 closed by chris.jerdonek #16513: SGMLParser processing <tr> which include two <a> will have pro http://bugs.python.org/issue16513 closed by ezio.melotti #16514: Cryptic traceback when sys.path[0] is None http://bugs.python.org/issue16514 closed by barry #16517: address merge conflicts in devguide null-merge instructions http://bugs.python.org/issue16517 closed by chris.jerdonek #16522: Add 'FAIL_FAST' flag to doctest http://bugs.python.org/issue16522 closed by r.david.murray #16527: (very) long list of elif causes segfault http://bugs.python.org/issue16527 closed by benjamin.peterson #16528: 3.2 docs not updating on docs.python.org http://bugs.python.org/issue16528 closed by r.david.murray #16536: test_cmd_line: failure on Ubuntu Shared http://bugs.python.org/issue16536 closed by ezio.melotti #16538: The docs doesn't describe MAKE_CLOSURE correctly http://bugs.python.org/issue16538 closed by asvetlov #667770: import C API mess http://bugs.python.org/issue667770 closed by brett.cannon #16539: Turn off 'No handlers could be found for logger' message http://bugs.python.org/issue16539 closed by vinay.sajip From MatatTHC at gmx.de Sun Nov 25 13:14:11 2012 From: MatatTHC at gmx.de (Matthias Bernt) Date: Sun, 25 Nov 2012 13:14:11 +0100 Subject: [Python-Dev] logging and rotation Message-ID: <CAFwXb28UYBwWtd5RDJcOSg5pypLyNGU+LtbATkSHp4wz0zniqg@mail.gmail.com> Dear mailing list, I'm using the logging module and write my log messages via the FileHandler. I just realized that using an external log rotation mechanism does not work. That is, new messages are not added to the file after rotation. In my opinion external log rotate mechanisms should work with the standard file handler. websearch pointed me to TimedRotatingFileHandler . But this only seems like a workaround. For instance I would like to get my log file mailed once a week. But it seems difficult to sync the cron job doing this and the logging-rotation mechanism. I think TimedRotatingFileHandler is important, e.g. for windows users. But for *nux users this seems to be unnecessary. Just want to gather some opinions before filing a bug. Regards, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121125/00953511/attachment.html> From phd at phdru.name Sun Nov 25 18:02:12 2012 From: phd at phdru.name (Oleg Broytman) Date: Sun, 25 Nov 2012 21:02:12 +0400 Subject: [Python-Dev] logging and rotation In-Reply-To: <CAFwXb28UYBwWtd5RDJcOSg5pypLyNGU+LtbATkSHp4wz0zniqg@mail.gmail.com> References: <CAFwXb28UYBwWtd5RDJcOSg5pypLyNGU+LtbATkSHp4wz0zniqg@mail.gmail.com> Message-ID: <20121125170212.GA29778@iskra.aviel.ru> On Sun, Nov 25, 2012 at 01:14:11PM +0100, Matthias Bernt <MatatTHC at gmx.de> wrote: > I'm using the logging module and write my log messages via the FileHandler. > I just realized that using an external log rotation mechanism does not > work. That is, new messages are not added to the file after > rotation. An external log rotation mechanism ought to send a signal to the application and the application ought to close and reopen logs. That is, this is an application-level problem, not logging module-level. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From python-dev at masklinn.net Sun Nov 25 18:34:07 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Sun, 25 Nov 2012 18:34:07 +0100 Subject: [Python-Dev] logging and rotation In-Reply-To: <20121125170212.GA29778@iskra.aviel.ru> References: <CAFwXb28UYBwWtd5RDJcOSg5pypLyNGU+LtbATkSHp4wz0zniqg@mail.gmail.com> <20121125170212.GA29778@iskra.aviel.ru> Message-ID: <836590DB-1933-4CB5-825F-B9731EAA45CC@masklinn.net> On 2012-11-25, at 18:02 , Oleg Broytman wrote: > On Sun, Nov 25, 2012 at 01:14:11PM +0100, Matthias Bernt <MatatTHC at gmx.de> wrote: >> I'm using the logging module and write my log messages via the FileHandler. >> I just realized that using an external log rotation mechanism does not >> work. That is, new messages are not added to the file after >> rotation. > > An external log rotation mechanism ought to send a signal to the > application and the application ought to close and reopen logs. That is, > this is an application-level problem, not logging module-level. I don't know that FileHandler officially supports reopening its underlying file. On the other hand, WatchedFileHandler[0] does exactly that and is specifically advertised for use with external log rotators: > WatchedFileHandler [?] watches the file it is logging to. If the file > changes, it is closed and reopened using the file name. > A file change can happen because of usage of programs such as newsyslog > and logrotate which perform log file rotation. [?] If the file has changed, > the old file stream is closed, and the file opened to get a new stream. [0] http://docs.python.org/2/library/logging.handlers.html#watchedfilehandler From guido at python.org Sun Nov 25 18:33:58 2012 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Nov 2012 09:33:58 -0800 Subject: [Python-Dev] logging and rotation In-Reply-To: <20121125170212.GA29778@iskra.aviel.ru> References: <CAFwXb28UYBwWtd5RDJcOSg5pypLyNGU+LtbATkSHp4wz0zniqg@mail.gmail.com> <20121125170212.GA29778@iskra.aviel.ru> Message-ID: <CAP7+vJLE2hFD3P1BhfXLH1iJuGqbLvewFt5cSETiU-aXayWDAg@mail.gmail.com> On Sun, Nov 25, 2012 at 9:02 AM, Oleg Broytman <phd at phdru.name> wrote: > On Sun, Nov 25, 2012 at 01:14:11PM +0100, Matthias Bernt <MatatTHC at gmx.de> wrote: >> I'm using the logging module and write my log messages via the FileHandler. >> I just realized that using an external log rotation mechanism does not >> work. That is, new messages are not added to the file after >> rotation. > > An external log rotation mechanism ought to send a signal to the > application and the application ought to close and reopen logs. That is, > this is an application-level problem, not logging module-level. I'm not so sure. Logging ought to "just work" without the application having to do much about it the details. Log rotation is definitely a useful feature; I've run into this very issue before and never found a satisfactory solution. Surely it should be possible for the file handler to occasionally stat() its output file by name and compare that to fstat() for the file descriptor and if they differ, reopen? -- --Guido van Rossum (python.org/~guido) From chris.jerdonek at gmail.com Mon Nov 26 06:01:18 2012 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sun, 25 Nov 2012 21:01:18 -0800 Subject: [Python-Dev] type vs. class terminology Message-ID: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> I would like to know when we should use "class" in the Python 3 documentation, and when we should use "type." Are these terms synonymous in Python 3, and do we have a preference for which to use and when? I'm sure this has been discussed before. But if this terminology issue has already been resolved, the resolution doesn't seem to be reflected in the docs. For example, the glossary entries for type and class don't reference each other. Thoughts? --Chris From ncoghlan at gmail.com Mon Nov 26 07:54:20 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Nov 2012 16:54:20 +1000 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> Message-ID: <CADiSq7e-gkVrrtegLuzfR5nn33fbhxvZYW+E1V_KMGeOs+7xuA@mail.gmail.com> On Mon, Nov 26, 2012 at 3:01 PM, Chris Jerdonek <chris.jerdonek at gmail.com>wrote: > I would like to know when we should use "class" in the Python 3 > documentation, and when we should use "type." Are these terms > synonymous in Python 3, and do we have a preference for which to use > and when? > > I'm sure this has been discussed before. But if this terminology > issue has already been resolved, the resolution doesn't seem to be > reflected in the docs. For example, the glossary entries for type and > class don't reference each other. > The historical distinction between "builtin types" and "user-defined classes" predates new-style classes (which unified the type system) and Python 3 (which eliminated the "instance" type that was provided to preserve the legacy user-defined class semantics in Python 2). The glossary unfortunately still reflects this distinction, which no longer exists in Python 3. A slightly more useful distinction would be if type was used consistently to refer to type(x), while class was used to refer to x.__class__, since they can and do differ in the case of proxy types (like weakref.proxy). However, it's probably too late for that kind of fine distinction - in reality, the two terms are now used pretty much interchangeably. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121126/7ee8f491/attachment.html> From python-dev at masklinn.net Mon Nov 26 08:50:52 2012 From: python-dev at masklinn.net (Xavier Morel) Date: Mon, 26 Nov 2012 08:50:52 +0100 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <CADiSq7e-gkVrrtegLuzfR5nn33fbhxvZYW+E1V_KMGeOs+7xuA@mail.gmail.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> <CADiSq7e-gkVrrtegLuzfR5nn33fbhxvZYW+E1V_KMGeOs+7xuA@mail.gmail.com> Message-ID: <C917503C-DA5C-4010-B527-4E01853267D7@masklinn.net> On 2012-11-26, at 07:54 , Nick Coghlan wrote: > On Mon, Nov 26, 2012 at 3:01 PM, Chris Jerdonek <chris.jerdonek at gmail.com>wrote: > >> I would like to know when we should use "class" in the Python 3 >> documentation, and when we should use "type." Are these terms >> synonymous in Python 3, and do we have a preference for which to use >> and when? >> >> I'm sure this has been discussed before. But if this terminology >> issue has already been resolved, the resolution doesn't seem to be >> reflected in the docs. For example, the glossary entries for type and >> class don't reference each other. >> > > The historical distinction between "builtin types" and "user-defined > classes" predates new-style classes (which unified the type system) and > Python 3 (which eliminated the "instance" type that was provided to > preserve the legacy user-defined class semantics in Python 2). The glossary > unfortunately still reflects this distinction, which no longer exists in > Python 3. > > A slightly more useful distinction would be if type was used consistently > to refer to type(x), while class was used to refer to x.__class__, since > they can and do differ in the case of proxy types (like weakref.proxy). > However, it's probably too late for that kind of fine distinction - in > reality, the two terms are now used pretty much interchangeably. There's an other possible usage which is between `type` subclasses and `type` instances (`type` essentially becomes a synonym for `metaclass`), I've seen that kind of usage at least once in the docs: http://docs.python.org/2/c-api/object.html#PyObject_IsInstance > If cls is a type object rather than a class object, PyObject_IsInstance() > returns 1 if inst is of type cls. From hrvoje.niksic at avl.com Mon Nov 26 09:04:21 2012 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Mon, 26 Nov 2012 09:04:21 +0100 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> Message-ID: <50B32285.2010806@avl.com> On 11/26/2012 06:01 AM, Chris Jerdonek wrote: > I would like to know when we should use "class" in the Python 3 > documentation, and when we should use "type." Are these terms > synonymous in Python 3, and do we have a preference for which to use > and when? Some people like to use "class" for the subset of types created by Python's "class" statement or its moral equivalent (explicit invocation of the metaclass). It makes sense that "class" is used to create classes. The word "type" then refers to both classes and built-in and extension types, such as "list" or "array.array". From kristjan at ccpgames.com Mon Nov 26 12:49:39 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Mon, 26 Nov 2012 11:49:39 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets Message-ID: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> Regarding the recent discussion on python-ideas about asyncronous IO, I'd like to ask a question about python socket's Timeout feature. Specifically this: Is it a documented or a guaranteed feature that a send/receive operation that times out with a socket.timeout error is re-startable on that socket? The reason I ask is that depending on the implementation, a timeout may leave a socket in an undefined state. As it is implemented in the standard cPython implementation, the timeout feature is done with an internal select() call. Thus, if the select() call times out, socket send/receive is not even called, so a retry is possible without issue. However, other implementations of python sockets, e.g. ones that rely on IO completion, may not have the luxury of using select. For example, on Windows, there is no way to abort an IOCP socket call, so a timeout must be implemented by aborting the wait. Dealing with the resulting race can be an interesting challenge. K -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121126/d2a661ba/attachment.html> From guido at python.org Mon Nov 26 16:52:41 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Nov 2012 07:52:41 -0800 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <50B32285.2010806@avl.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> <50B32285.2010806@avl.com> Message-ID: <CAP7+vJKgOui+xC_fGenAzGNXrpEs5ufqJ8xGtHjAkfatjnqMLA@mail.gmail.com> Hm. None of the distinctions brought up so far really hit true with me (though they all are part of the picture). For example, I think the distinction between type(x) and x.__class__ is rarely significant -- I bet that if anyone were to rely on this they'd first have to change a lot of code that used one or the other without much thinking about the difference. So we might as well consider these equivalent. Part of the distinction is probably just in historical usage -- many idioms hark back to when there *was* a difference. If I had to invent an artificial difference, I'd say that I use "type" when talking about the type as a fairly abstract concept, such as the type of a variable (which may be required to be in a given set, for example), whereas I'll say "class" when I'm interested in the class as an object, or its definition (through a class statement). So, if I'm going to ask for the name, the phrase "the name of the class" rolls easier off my tongue than "the name of the type". OTOH if I'm going to just assert set membership, I might slightly prefer "x's type must be int or str". It's clear that we ought to at least cross-link the two glossary entries and point out that they are the same concept (type(x) vs. x.__class__) notwithstanding. I don't think it's important to try and eradicate the word type from our conversation or our docs -- so no sweeping global changes, please. However, if you come across a use of either one in the docs that seems odd given modern usage, feel free to clean it up, perhaps after checking with someone else to make sure your intuition matches that of others. -- --Guido van Rossum (python.org/~guido) From guido at python.org Mon Nov 26 16:59:08 2012 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Nov 2012 07:59:08 -0800 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> If you're talking about the standard socket module, I'm not aware that it uses IOCP on Windows. Are you asking this just in the abstract, or do you know of a Python implementation that uses IOCP to implement the standard socket type? As to the design of the async I/O library (which I am still working on!), I cannot guarantee anything, and the issue will probably be moot -- the API won't have the same kind of timeout as the current socket object (it will have other ways to set deadlines though). Confusedly yours, --Guido On Mon, Nov 26, 2012 at 3:49 AM, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > Regarding the recent discussion on python-ideas about asyncronous IO, I?d > like to ask a question about python socket?s Timeout feature. > > Specifically this: Is it a documented or a guaranteed feature that a > send/receive operation that times out with a socket.timeout error is > re-startable on that socket? > > > > The reason I ask is that depending on the implementation, a timeout may > leave a socket in an undefined state. > > As it is implemented in the standard cPython implementation, the timeout > feature is done with an internal select() call. Thus, if the select() call > times out, socket send/receive is not even called, so a retry is possible > without issue. > > However, other implementations of python sockets, e.g. ones that rely on IO > completion, may not have the luxury of using select. For example, on > Windows, there is no way to abort an IOCP socket call, so a timeout must be > implemented by aborting the wait. Dealing with the resulting race can be an > interesting challenge. > > > > K > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From shibturn at gmail.com Mon Nov 26 17:05:04 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Mon, 26 Nov 2012 16:05:04 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <k903vo$2v4$1@ger.gmane.org> On 26/11/2012 11:49am, Kristj?n Valur J?nsson wrote: > However, other implementations of python sockets, e.g. ones that rely on > IO completion, may not have the luxury of using select. For example, on > Windows, there is no way to abort an IOCP socket call, so a timeout must > be implemented by aborting the wait. Dealing with the resulting race > can be an interesting challenge. I am not quite sure what you mean by "aborting the wait". But you can abort an overlapped operation using CancelIo()/CancelIoEx(). I have just done some experimenting. Using CancelIo()/CancelIoEx() to abort an operation started with WSARecv() does not seem to cause a problem -- you just call GetOverlappedResult() afterwards to check whether the operation completed before it could be aborted. But aborting an operation started with WSASend() sometimes seems to "break" the connection: a subsequent WSARecv()/WSASend() will fail with WSAECONNABORTED or WSAECONNRESET depending on which end of the connection you are on. So, as you say, if you abort a send then you cannot expect to successfully resend the data later. -- Richard From glyph at twistedmatrix.com Mon Nov 26 20:23:55 2012 From: glyph at twistedmatrix.com (Glyph) Date: Mon, 26 Nov 2012 14:23:55 -0500 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <k903vo$2v4$1@ger.gmane.org> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <k903vo$2v4$1@ger.gmane.org> Message-ID: <50A9C7B9-10FE-4CDE-A348-5C5669B0E569@twistedmatrix.com> On Nov 26, 2012, at 11:05 AM, Richard Oudkerk <shibturn at gmail.com> wrote: > Using CancelIo()/CancelIoEx() to abort an operation started with WSARecv() does not seem to cause a problem (emphasis mine) Little command-line experiments are not the right way to verify the behavior of high-performance I/O APIs. You need to do careful reading of the documentation, significant testing under load and experiments on a huge variety of platforms. Windows runs in _lots_ of horrible, horrible places. I think that the safest option would really be to better document the somewhat mangled state that a timeout may leave a socket in. I don't believe the feature was intended for pipelined protocols; if you receive a timeout by using the stdlib timeout functionality you have generally burned the socket. And, generally, things that care enough about performance or scalability enough to use IOCP operations should never use timeout-sockets anyway; it may do a select() internally (and on Windows, where poll() is never available, it _will_ do a select() internally), which limits the number of file descriptors you might even have in your process before you start encountering spurious errors. The use case it supports is when you have a little tool that just needs to fetch a URL or something really simple, but wants to be able to get on with things if that doesn't work or takes too long. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121126/fd3daa8d/attachment.html> From kristjan at ccpgames.com Tue Nov 27 10:35:15 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 27 Nov 2012 09:35:15 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <k903vo$2v4$1@ger.gmane.org> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <k903vo$2v4$1@ger.gmane.org> Message-ID: <EFE3877620384242A686D52278B7CCD329DEBF5F@RKV-IT-EXCH103.ccp.ad.local> This is getting off-topic, but: CancelIoEx() is a new api that I wasn't aware of (my IOCP solution dates back to 2005). It appears that this can be used, although the documentation is sketchy. This worries me: "If the file handle is associated with a completion port, an I/O completion packet is not queued to the port if a synchronous operation is successfully canceled. For asynchronous operations still pending, the cancel operation will queue an I/O completion packet." This _might_ mean that synchronizing the cancel request with a callback that expects to be called, but then, may not be called, can be difficult. It is possible that MS didn't think their API completely through (nothing new here.) Anyway, that is somewhat beside the point Even if we can cancel an ongoing operation there needs to be synchronization, so that any data that is received(or sent) is correctly communicated ot the app. If there isn't a cancel option in the API (not talking about windows here) then this would mean queueing up data for recv(), and no simple solution for send(). So, basically, what I'm saying is, that enabling re-startable socket timeout semantics for sockets implemented with "completion" semantics, rather than "ready" semantics _can_ be difficult, hence my question. > -----Original Message----- > From: Python-Dev [mailto:python-dev- > bounces+kristjan=ccpgames.com at python.org] On Behalf Of Richard > Oudkerk > Sent: 26. n?vember 2012 16:05 > To: python-dev at python.org > Subject: Re: [Python-Dev] Socket timeout and completion based sockets > > On 26/11/2012 11:49am, Kristj?n Valur J?nsson wrote: > > However, other implementations of python sockets, e.g. ones that rely > > on IO completion, may not have the luxury of using select. For > > example, on Windows, there is no way to abort an IOCP socket call, so > > a timeout must be implemented by aborting the wait. Dealing with the > > resulting race can be an interesting challenge. > > I am not quite sure what you mean by "aborting the wait". But you can abort > an overlapped operation using CancelIo()/CancelIoEx(). > > I have just done some experimenting. > > Using CancelIo()/CancelIoEx() to abort an operation started with > WSARecv() does not seem to cause a problem -- you just call > GetOverlappedResult() afterwards to check whether the operation > completed before it could be aborted. > > But aborting an operation started with WSASend() sometimes seems to > "break" the connection: a subsequent WSARecv()/WSASend() will fail with > WSAECONNABORTED or WSAECONNRESET depending on which end of the > connection you are on. > > So, as you say, if you abort a send then you cannot expect to successfully > resend the data later. > > -- > Richard > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/kristjan%40ccpgames.com From kristjan at ccpgames.com Tue Nov 27 10:23:02 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Tue, 27 Nov 2012 09:23:02 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> Yes, well, as a matter of fact, I do have an IOCP based socket implementation with stackless python. >From the programmer's perspective, operations appear blocking while IOCP is used to switch tasklets in the background. But my socket timeout implementation does not guarantee that the socket is left in a valid state for retrying a recv() operation. I was under the (perhaps mistaken) impression that the current async work _could_ result in a standardized way to create such alternative socket implementations, ones that might do their magic using greenlets, tasklets, or generators. But if that were the case, such loosely defined features of the socket API would need clearer definitions. K > -----Original Message----- > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf > Of Guido van Rossum > Sent: 26. n?vember 2012 15:59 > To: Kristj?n Valur J?nsson > Cc: Python-Dev (python-dev at python.org) > Subject: Re: [Python-Dev] Socket timeout and completion based sockets > > If you're talking about the standard socket module, I'm not aware that it uses > IOCP on Windows. Are you asking this just in the abstract, or do you know of > a Python implementation that uses IOCP to implement the standard socket > type? > > As to the design of the async I/O library (which I am still working on!), I > cannot guarantee anything, and the issue will probably be moot > -- the API won't have the same kind of timeout as the current socket object > (it will have other ways to set deadlines though). From trent at snakebite.org Tue Nov 27 12:30:11 2012 From: trent at snakebite.org (Trent Nelson) Date: Tue, 27 Nov 2012 06:30:11 -0500 Subject: [Python-Dev] Tru64 support In-Reply-To: <CAL3CFcXFVsAF6LomdX7ysuQJ0XfJgQWS8eNNVk=Q+-=8r6NFbg@mail.gmail.com> References: <CAL3CFcXFVsAF6LomdX7ysuQJ0XfJgQWS8eNNVk=Q+-=8r6NFbg@mail.gmail.com> Message-ID: <20121127113011.GF90314@snakebite.org> On Tue, Sep 04, 2012 at 02:24:10AM -0700, Andrew Svetlov wrote: > Is it still up to date? Bug has been created in 2004. > I don't see Tru64 in list of available buildbots. > > Do we need to care about this platform? And how to make sure what > existing code works fine for that? There's a Snakebite Tru64 box now, accessible via t5 from the sb menu. 2.7 compiles fine but hangs on some of the subprocess epipe tests, I'll set up a slave once I've addressed that. One nice thing about Tru64 is that everything is implicitly 64-bit. There's no notion of dual 32-bit/64-bit libs/binaries or a 32-bit memory architecture. The 64-bit `python` on Tru64 is in a much better state than said binaries on AIX and HP-UX (which still need a lot of work). As for VMS/OpenVMS, still working with HP to get access to media. Trent. From shibturn at gmail.com Tue Nov 27 13:33:10 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 27 Nov 2012 12:33:10 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <EFE3877620384242A686D52278B7CCD329DEBF5F@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <k903vo$2v4$1@ger.gmane.org> <EFE3877620384242A686D52278B7CCD329DEBF5F@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <k92buf$r8u$1@ger.gmane.org> On 27/11/2012 9:35am, Kristj?n Valur J?nsson wrote: > This worries me: > "If the file handle is associated with a completion port, an I/O completion > packet is not queued to the port if a synchronous operation is successfully canceled... I think you can only abort a synchronous operation if you use CancelIoEx() from a different thread. That won't be an issue if you do all your IO stuff in one thread. -- Richard From guido at python.org Tue Nov 27 16:54:19 2012 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Nov 2012 07:54:19 -0800 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> On Tue, Nov 27, 2012 at 1:23 AM, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > Yes, well, as a matter of fact, I do have an IOCP based socket implementation with stackless python. It would have been nice if you had given more context and stated your objective upfront instead of asking what appeared to be an obscure question about a technical detail. I have been distracted by other stuff temporarily, but I do plan to continue working on tulip more, and I want to assure you that EVERYTHING IS STILL OPEN. I.e. don't take the current tulip code base as a draft PEP -- it is just my way of learning about the issues. Also note that Richard Oudkerk has developed an alternative, which you can find in the "proactor" branch of the tulip repository. > From the programmer's perspective, operations appear blocking while IOCP is used to switch tasklets in the background. > But my socket timeout implementation does not guarantee that the socket is left in a valid state for retrying a recv() operation. > I was under the (perhaps mistaken) impression that the current async work _could_ result in a standardized way to create such > alternative socket implementations, ones that might do their magic using greenlets, tasklets, or generators. But if that were > the case, such loosely defined features of the socket API would need clearer definitions. If you have a specific requirement, please state it explicitly, rather than just hoping tulip will be your solution. FWIW, it is likely that tulip will hide the actual socket object completely inside a higher-level abstraction, so that the actual semantics of sockets (which are extremely platform-dependent) cannot affect the specification of the API. There will be platform-specific ways to reach inside, but they will be of limited use -- it's possible that on Windows (at least when using IOCP) there won't be a socket object at all. Finally, I am not at all interested in greenlets(*). Tulip will support two API surfaces: a low-level one, suitable for interfacing with e.g. Twisted or Tornado, that uses callbacks to signal ready-ness or completion. And a high-level one, useful for e.g. writing protocol handling libraries and applications, that will somehow use yield and/or yield-from. The details of each layer are very much unspecified at this point. NOW WOULD BE A GOOD TIME TO WRITE DOWN YOUR REQUIREMENTS. (*) Greenlets are a fine mechanism for some application areas, but ultimately not fit for the standard library, and they have some significant downsides. --Guido > K >> -----Original Message----- >> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf >> Of Guido van Rossum >> Sent: 26. n?vember 2012 15:59 >> To: Kristj?n Valur J?nsson >> Cc: Python-Dev (python-dev at python.org) >> Subject: Re: [Python-Dev] Socket timeout and completion based sockets >> >> If you're talking about the standard socket module, I'm not aware that it uses >> IOCP on Windows. Are you asking this just in the abstract, or do you know of >> a Python implementation that uses IOCP to implement the standard socket >> type? >> >> As to the design of the async I/O library (which I am still working on!), I >> cannot guarantee anything, and the issue will probably be moot >> -- the API won't have the same kind of timeout as the current socket object >> (it will have other ways to set deadlines though). > > -- --Guido van Rossum (python.org/~guido) From brian at python.org Tue Nov 27 21:41:13 2012 From: brian at python.org (Brian Curtin) Date: Tue, 27 Nov 2012 14:41:13 -0600 Subject: [Python-Dev] New Contributor Experience in Python and other FOSS Communities - A Survey In-Reply-To: <CAD+XWwoVUSaP1C4qsMbn05ROApkA5fAo2Fr=vXssQy=iAcJSzg@mail.gmail.com> References: <CAD+XWwoVUSaP1C4qsMbn05ROApkA5fAo2Fr=vXssQy=iAcJSzg@mail.gmail.com> Message-ID: <CAD+XWwryF-9=9Y2BdeTEmcg4r8+kmbBFkFu0BAL9H3AZ1V0gMA@mail.gmail.com> On Mon, Nov 19, 2012 at 9:01 AM, Brian Curtin <brian at python.org> wrote: > Hi all, > > Along with a number of other free and open communities, Python is > being included in a survey of new contributors since January 2010. The > survey is being done by Kevin Carillo, a PhD student at Victoria > University of Wellington who is studying various approaches that > projects use to gain and work with new contributors. > > If you have written patches, reviewed issues, or done anything to > contribute to the development of Python and you started this after > January 2010, I hope you can spare around 20 minutes to complete this > survey: https://limesurvey.sim.vuw.ac.nz/index.php?sid=65151&lang=en > > There's a longer blog post up at > http://blog.python.org/2012/11/new-contributor-experience-in-python.html > if you would like a bit more information. > > On behalf of Kevin Carillo, I thank you for your time and > consideration of this survey. > > Brian Curtin Just a quick reminder - Kevin could really use the help if any of you recent newcomers can spare the time. I took the survey myself since I became a committer right after the survey range begins, and I think it took me 15 minutes. From trent at snakebite.org Tue Nov 27 23:49:28 2012 From: trent at snakebite.org (Trent Nelson) Date: Tue, 27 Nov 2012 17:49:28 -0500 Subject: [Python-Dev] Handling support for newer OS features at run time Message-ID: <20121127224928.GK91191@snakebite.org> The hackiest part of my WSAPoll patch is this: --- a/PC/pyconfig.h Sun Nov 18 10:42:42 2012 +0000 +++ b/PC/pyconfig.h Sun Nov 18 17:27:29 2012 -0500 @@ -158,12 +158,12 @@ /* set the version macros for the windows headers */ #ifdef MS_WINX64 /* 64 bit only runs on XP or greater */ -#define Py_WINVER 0x0501 /* _WIN32_WINNT_WINXP */ -#define Py_NTDDI NTDDI_WINXP +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WINXP */ +#define Py_NTDDI NTDDI_WIN6 #else /* Python 2.6+ requires Windows 2000 or greater */ -#define Py_WINVER 0x0500 /* _WIN32_WINNT_WIN2K */ -#define Py_NTDDI NTDDI_WIN2KSP4 +#define Py_WINVER 0x0600 /* _WIN32_WINNT_WIN2K */ +#define Py_NTDDI NTDDI_WIN6 #endif Basically, because of the way our build is currently configured, I had to bump the minimum supported Windows version from XP to Vista just to get access to the WSAPoll headers. (Issue: http://bugs.python.org/issue16507 Patch: http://bugs.python.org/file28038/wsapoll.patch) Ideally, a Windows binary should make WSAPoll/select.poll() available if running on Vista or above, without impacting the ability to run on XP. I don't think we've currently got the ability to do this, right? Is there a precedent set anywhere else? I suspect it's not as much of an issue on *NIX platforms as you'll typically compile from source. Windows, not so much. Thoughts? A single binary that dynamically loads applicable modules seems cleaner but will obviously take more work. Other approach would be to start distributing multiple versions of Python based on the underlying Windows version. Not the nicest approach. Trent. From doko at ubuntu.com Wed Nov 28 00:09:12 2012 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 28 Nov 2012 00:09:12 +0100 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <20121127224928.GK91191@snakebite.org> References: <20121127224928.GK91191@snakebite.org> Message-ID: <50B54818.2090907@ubuntu.com> Am 27.11.2012 23:49, schrieb Trent Nelson: > I don't think we've currently got the ability to do this, right? > Is there a precedent set anywhere else? I suspect it's not as > much of an issue on *NIX platforms as you'll typically compile > from source. Windows, not so much. > > Thoughts? A single binary that dynamically loads applicable > modules seems cleaner but will obviously take more work. Other > approach would be to start distributing multiple versions of > Python based on the underlying Windows version. Not the nicest > approach. Usually I have to build a python package on a build daemon running the kernel of the latest stable (or latest stable long term support) release. Thus if a configure test checks the current kernel, then you may get an unexpected answer and a missing feature. It is somehow different that I already build different binaries (per release), but the hard-coding of kernel features during configure time looks like the same issue. Currently working around it by patching configure to remove the test and hard-code it for a particular build. Another solution maybe would to have something like --enable-kernel=<version> (as found in glibc), and hope that you can deduce the features from the kernel version. From shibturn at gmail.com Wed Nov 28 00:14:00 2012 From: shibturn at gmail.com (Richard Oudkerk) Date: Tue, 27 Nov 2012 23:14:00 +0000 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <20121127224928.GK91191@snakebite.org> References: <20121127224928.GK91191@snakebite.org> Message-ID: <k93hg3$2gh$1@ger.gmane.org> On 27/11/2012 10:49pm, Trent Nelson wrote: > Ideally, a Windows binary should make WSAPoll/select.poll() > available if running on Vista or above, without impacting > the ability to run on XP. I assume you can do something like int WSAAPI (*pWSAPoll)(WSAPOLLFD *, ULONG, INT); HINSTANCE hWinsock2 = GetModuleHandle("WS2_32"); *(FARPROC *)&pWSAPoll = GetProcAddress(hWinsock2, "WSAPoll"); if (pWSAPoll == NULL) ... to get a function pointer for WSAPoll on versions which support it. Modules/_winapi.c does something similar to get CancelIoEx. -- Richard From trent at snakebite.org Wed Nov 28 00:19:19 2012 From: trent at snakebite.org (Trent Nelson) Date: Tue, 27 Nov 2012 18:19:19 -0500 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <50B54818.2090907@ubuntu.com> References: <20121127224928.GK91191@snakebite.org> <50B54818.2090907@ubuntu.com> Message-ID: <20121127231919.GM91191@snakebite.org> On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: > Am 27.11.2012 23:49, schrieb Trent Nelson: > > I don't think we've currently got the ability to do this, right? > > Is there a precedent set anywhere else? I suspect it's not as > > much of an issue on *NIX platforms as you'll typically compile > > from source. Windows, not so much. > > > > Thoughts? A single binary that dynamically loads applicable > > modules seems cleaner but will obviously take more work. Other > > approach would be to start distributing multiple versions of > > Python based on the underlying Windows version. Not the nicest > > approach. > > Usually I have to build a python package on a build daemon running the > kernel of the latest stable (or latest stable long term support) > release. Thus if a configure test checks the current kernel, then you > may get an unexpected answer and a missing feature. It is somehow > different that I already build different binaries (per release), but > the hard-coding of kernel features during configure time looks like > the same issue. Currently working around it by patching configure to > remove the test and hard-code it for a particular build. Another > solution maybe would to have something like --enable-kernel=<version> > (as found in glibc), and hope that you can deduce the features from > the kernel version. Hmmm. How often do Linux kernel versions expose new features that we can make use of in Python? i.e. do you run into this problem often? Can you cite some recent examples? I'm not sure how much could be shared between Windows and Linux with what you've just described. With Windows, specifically with regards to providing dynamic select.poll() support, I was thinking of having multiple modules: Python33\ DLLs\ select.pyd select_win6.pyd select_win7.pyd select_win8.pyd And Python would automatically import the appropriate one. I don't think this would be that useful on Linux -- as you say, the decision is typically made at ./configure time. Trent. From trent at snakebite.org Wed Nov 28 00:30:28 2012 From: trent at snakebite.org (Trent Nelson) Date: Tue, 27 Nov 2012 18:30:28 -0500 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <k93hg3$2gh$1@ger.gmane.org> References: <20121127224928.GK91191@snakebite.org> <k93hg3$2gh$1@ger.gmane.org> Message-ID: <20121127233027.GN91191@snakebite.org> On Tue, Nov 27, 2012 at 03:14:00PM -0800, Richard Oudkerk wrote: > On 27/11/2012 10:49pm, Trent Nelson wrote: > > Ideally, a Windows binary should make WSAPoll/select.poll() > > available if running on Vista or above, without impacting > > the ability to run on XP. > > I assume you can do something like > > int WSAAPI (*pWSAPoll)(WSAPOLLFD *, ULONG, INT); > HINSTANCE hWinsock2 = GetModuleHandle("WS2_32"); > *(FARPROC *)&pWSAPoll = GetProcAddress(hWinsock2, "WSAPoll"); > if (pWSAPoll == NULL) > ... > > to get a function pointer for WSAPoll on versions which support it. Ah! Good point, that's probably a much better approach in this instance, given I only need one or two more symbols. Thanks! Trent. From ericsnowcurrently at gmail.com Wed Nov 28 05:19:28 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 27 Nov 2012 21:19:28 -0700 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CADiSq7e0PhtkozwD0-4EP=hRs4V9M7GK8PoaWqz1moRvEeKJ6w@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7e0PhtkozwD0-4EP=hRs4V9M7GK8PoaWqz1moRvEeKJ6w@mail.gmail.com> Message-ID: <CALFfu7BNqM5scr6WZQXfNToFwqup7E_ZLqfK_F_P4BC2dCO25A@mail.gmail.com> On Thu, Nov 22, 2012 at 5:09 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: > On Thu, Nov 22, 2012 at 9:44 PM, Kristj?n Valur J?nsson > <kristjan at ccpgames.com> wrote: >> >> Where in the tracker? I tried searching but didn?t find it. > > > This one: http://bugs.python.org/issue13475 > > This and the issue about being able to configure coverage.py cleanly in > subprocesses (http://bugs.python.org/issue14803) are the two issues which > have got me thinking about the interpreter startup and configuration process > in general lately. Both would add *more* complexity to an already > complicated and hard to follow initialisation sequence, so I now think we > should be look at opportunities for rationalisation *first*, before we make > things even messier. There's also http://bugs.python.org/issue15716 ("Ability to specify the PYTHONPATH via a command line flag"). -eric From ericsnowcurrently at gmail.com Wed Nov 28 05:46:23 2012 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 27 Nov 2012 21:46:23 -0700 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CALFfu7BNqM5scr6WZQXfNToFwqup7E_ZLqfK_F_P4BC2dCO25A@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7e0PhtkozwD0-4EP=hRs4V9M7GK8PoaWqz1moRvEeKJ6w@mail.gmail.com> <CALFfu7BNqM5scr6WZQXfNToFwqup7E_ZLqfK_F_P4BC2dCO25A@mail.gmail.com> Message-ID: <CALFfu7Az9u0wRXOtjf0C9X5r+tiO-NF0+1=dER=5wp0Q_ADUnQ@mail.gmail.com> On Tue, Nov 27, 2012 at 9:19 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote: > On Thu, Nov 22, 2012 at 5:09 AM, Nick Coghlan <ncoghlan at gmail.com> wrote: >> On Thu, Nov 22, 2012 at 9:44 PM, Kristj?n Valur J?nsson >> <kristjan at ccpgames.com> wrote: >>> >>> Where in the tracker? I tried searching but didn?t find it. >> >> >> This one: http://bugs.python.org/issue13475 >> >> This and the issue about being able to configure coverage.py cleanly in >> subprocesses (http://bugs.python.org/issue14803) are the two issues which >> have got me thinking about the interpreter startup and configuration process >> in general lately. Both would add *more* complexity to an already >> complicated and hard to follow initialisation sequence, so I now think we >> should be look at opportunities for rationalisation *first*, before we make >> things even messier. > > There's also http://bugs.python.org/issue15716 ("Ability to specify > the PYTHONPATH via a command line flag"). > > -eric I knew there was one more: http://bugs.python.org/issue16499 ("CLI option for isolated mode"). -eric From greg at krypto.org Wed Nov 28 06:37:00 2012 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 27 Nov 2012 21:37:00 -0800 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <20121127231919.GM91191@snakebite.org> References: <20121127224928.GK91191@snakebite.org> <50B54818.2090907@ubuntu.com> <20121127231919.GM91191@snakebite.org> Message-ID: <CAGE7PNL8y28Ud2CxkWzgPLEPi+P1dHNGGwKSraj9Ear1Rqg3Mg@mail.gmail.com> On Tue, Nov 27, 2012 at 3:19 PM, Trent Nelson <trent at snakebite.org> wrote: > On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: > > Am 27.11.2012 23:49, schrieb Trent Nelson: > > > I don't think we've currently got the ability to do this, right? > > > Is there a precedent set anywhere else? I suspect it's not as > > > much of an issue on *NIX platforms as you'll typically compile > > > from source. Windows, not so much. > > > > > > Thoughts? A single binary that dynamically loads applicable > > > modules seems cleaner but will obviously take more work. Other > > > approach would be to start distributing multiple versions of > > > Python based on the underlying Windows version. Not the nicest > > > approach. > > > > Usually I have to build a python package on a build daemon running the > > kernel of the latest stable (or latest stable long term support) > > release. Thus if a configure test checks the current kernel, then you > > may get an unexpected answer and a missing feature. It is somehow > > different that I already build different binaries (per release), but > > the hard-coding of kernel features during configure time looks like > > the same issue. Currently working around it by patching configure to > > remove the test and hard-code it for a particular build. Another > > solution maybe would to have something like --enable-kernel=<version> > > (as found in glibc), and hope that you can deduce the features from > > the kernel version. > > Hmmm. How often do Linux kernel versions expose new features that > we can make use of in Python? i.e. do you run into this problem > often? Can you cite some recent examples? > Here's an example of using the pipe2() system call. The code is only compiled in if the C library supports it. If the C library supports it, the system call can still return an error because the running kernel doesn't support it (ENOSYS). In that case it falls back to the legacy code: http://hg.python.org/cpython/file/618ea5612e83/Modules/_posixsubprocess.c#l738 -gps > > I'm not sure how much could be shared between Windows and Linux with > what you've just described. With Windows, specifically with regards > to providing dynamic select.poll() support, I was thinking of having > multiple modules: > > Python33\ > DLLs\ > select.pyd > select_win6.pyd > select_win7.pyd > select_win8.pyd > > And Python would automatically import the appropriate one. I don't > think this would be that useful on Linux -- as you say, the decision > is typically made at ./configure time. > > Trent. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121127/9e642f9f/attachment-0001.html> From doko at ubuntu.com Wed Nov 28 07:09:16 2012 From: doko at ubuntu.com (Matthias Klose) Date: Wed, 28 Nov 2012 07:09:16 +0100 Subject: [Python-Dev] Handling support for newer OS features at run time In-Reply-To: <CAGE7PNL8y28Ud2CxkWzgPLEPi+P1dHNGGwKSraj9Ear1Rqg3Mg@mail.gmail.com> References: <20121127224928.GK91191@snakebite.org> <50B54818.2090907@ubuntu.com> <20121127231919.GM91191@snakebite.org> <CAGE7PNL8y28Ud2CxkWzgPLEPi+P1dHNGGwKSraj9Ear1Rqg3Mg@mail.gmail.com> Message-ID: <50B5AA8C.9070401@ubuntu.com> Am 28.11.2012 06:37, schrieb Gregory P. Smith: > On Tue, Nov 27, 2012 at 3:19 PM, Trent Nelson <trent at snakebite.org> wrote: > >> On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote: >>> Am 27.11.2012 23:49, schrieb Trent Nelson: >>>> I don't think we've currently got the ability to do this, right? >>>> Is there a precedent set anywhere else? I suspect it's not as >>>> much of an issue on *NIX platforms as you'll typically compile >>>> from source. Windows, not so much. >>>> >>>> Thoughts? A single binary that dynamically loads applicable >>>> modules seems cleaner but will obviously take more work. Other >>>> approach would be to start distributing multiple versions of >>>> Python based on the underlying Windows version. Not the nicest >>>> approach. >>> >>> Usually I have to build a python package on a build daemon running the >>> kernel of the latest stable (or latest stable long term support) >>> release. Thus if a configure test checks the current kernel, then you >>> may get an unexpected answer and a missing feature. It is somehow >>> different that I already build different binaries (per release), but >>> the hard-coding of kernel features during configure time looks like >>> the same issue. Currently working around it by patching configure to >>> remove the test and hard-code it for a particular build. Another >>> solution maybe would to have something like --enable-kernel=<version> >>> (as found in glibc), and hope that you can deduce the features from >>> the kernel version. >> >> Hmmm. How often do Linux kernel versions expose new features that >> we can make use of in Python? i.e. do you run into this problem >> often? Can you cite some recent examples? >> > > Here's an example of using the pipe2() system call. The code is only > compiled in if the C library supports it. If the C library supports it, the > system call can still return an error because the running kernel doesn't > support it (ENOSYS). In that case it falls back to the legacy code: > > > http://hg.python.org/cpython/file/618ea5612e83/Modules/_posixsubprocess.c#l738 another one is the check for working semaphores for the multiprocessing module, seen at https://launchpad.net/bugs/630511 From ncoghlan at gmail.com Wed Nov 28 12:20:12 2012 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Nov 2012 21:20:12 +1000 Subject: [Python-Dev] Generally boared by installation (Re: Setting project home path the best way) In-Reply-To: <CALFfu7Az9u0wRXOtjf0C9X5r+tiO-NF0+1=dER=5wp0Q_ADUnQ@mail.gmail.com> References: <27E194BD-5864-46CC-82F5-F77DC7977D03@stackless.com> <EFE3877620384242A686D52278B7CCD329DDC6B1@RKV-IT-EXCH103.ccp.ad.local> <2842C4E9-FEF4-46D1-B09D-C755B33A1925@stackless.com> <EFE3877620384242A686D52278B7CCD329DE0BF0@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7dfP8kjFXWrqan6uzJi7xJp_KL=153VnzsrTyhdaFq3og@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE2A53@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7ccgu9aMCXUqmSfmRCr3_ZJY5-A8hwemwYTFzUMXHdKzw@mail.gmail.com> <50AB8975.3020405@stackless.com> <CADiSq7c=E867L7_1PT4dCPVh1UO+kc3xwoXmK3tXbeXBxBApJQ@mail.gmail.com> <CAG8k2+6+rRwyTndvf87gGrSGH9jh0h=5L8k+m_nHQyA59DxEJg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DE76CF@RKV-IT-EXCH103.ccp.ad.local> <CADiSq7e0PhtkozwD0-4EP=hRs4V9M7GK8PoaWqz1moRvEeKJ6w@mail.gmail.com> <CALFfu7BNqM5scr6WZQXfNToFwqup7E_ZLqfK_F_P4BC2dCO25A@mail.gmail.com> <CALFfu7Az9u0wRXOtjf0C9X5r+tiO-NF0+1=dER=5wp0Q_ADUnQ@mail.gmail.com> Message-ID: <CADiSq7eusbcTEwbJB7Jwxu=fp4eBS_4oCB5j_MnfAZKd9+qpKg@mail.gmail.com> On Wed, Nov 28, 2012 at 2:46 PM, Eric Snow <ericsnowcurrently at gmail.com>wrote: > I knew there was one more: http://bugs.python.org/issue16499 ("CLI > option for isolated mode"). > Along with another PYIOENCODING related one that the Blender folks reported (Christian Heimes pointed it out to me earlier today). Anyway, I created a page on the wiki for the data gathering process: http://wiki.python.org/moin/CPythonInterpreterInitialization It's now a matter of going through and sorting out: 1. What gets set during startup? 2. Where does it get set (or modified)? 3. How is that configured? Once we have a good view of that, *then* we can start looking for ways to simplify the code, make the whole system more embedding friendly (i.e. by giving the embedding app total control via a simple and clean API) and still support the proposals for improvements. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121128/9fc6bafa/attachment.html> From kristjan at ccpgames.com Wed Nov 28 13:13:15 2012 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 28 Nov 2012 12:13:15 +0000 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> Message-ID: <EFE3877620384242A686D52278B7CCD329DED074@RKV-IT-EXCH103.ccp.ad.local> I'm sorry, I thought it was something that people did more often, to create different implementations of of the "socket" api, for which cPython provided a mere reference implementation. I know of at least three different alternative implementations, so I thought that the question were clear enough: Is the timeout mechanism "supposed" to be re-startable for an api that aims to conform to the "socket" module, or is that a mere coincidence falling out from the select/bsd based reference implementation in cPython? The docs don't say either way. (For c-level timeout mechanisms implemented for various c implementations of the bsd socket api, it is not uncommon to see it stated that after a socket operation times out, the socket is in an undefined state and should be discarded, e.g. here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms740476(v=vs.85).aspx " If a send or receive operation times out on a socket, the socket state is indeterminate, and should not be used; TCP sockets in this state have a potential for data loss, since the operation could be canceled at the same moment the operation was to be completed.") Anyway, as for concrete requirements: The issue I have always seen with various asynchronous libraries is their lack of composability. Everyone writes their own application loop and event queue. Merely having a standard spec and reference implementation of an application main loop object, and main event queue object, in the spirit of WSGI, would possibly remedy this. You could then hopefully assemble various different libraries in the same application, including greenlet(*) based ones. (*) Greenlets or stackless can be just another way of hiding asynchronous operations from the programmer. My favourite one, in fact. The main trick here is unwinding and replaying of calling contexts, the specific implementation by stack-slicking is mere technical detail, since it can be achieved in other ways (see soft-switching in stackless python) Cheers, K > -----Original Message----- > From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf > Of Guido van Rossum > Sent: 27. n?vember 2012 15:54 with stackless python. > > It would have been nice if you had given more context and stated your > objective upfront instead of asking what appeared to be an obscure question > about a technical detail > Finally, I am not at all interested in greenlets > ... > very much unspecified at this point. NOW WOULD BE A GOOD TIME TO > WRITE DOWN YOUR REQUIREMENTS. > > (*) Greenlets are a fine mechanism for some application areas, but ultimately > not fit for the standard library, and they have some significant downsides. > From solipsis at pitrou.net Wed Nov 28 17:29:43 2012 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Nov 2012 17:29:43 +0100 Subject: [Python-Dev] Socket timeout and completion based sockets References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DED074@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <20121128172943.521e5de5@pitrou.net> Le Wed, 28 Nov 2012 12:13:15 +0000, Kristj?n Valur J?nsson <kristjan at ccpgames.com> a ?crit : > I'm sorry, I thought it was something that people did more often, to > create different implementations of of the "socket" api, for which > cPython provided a mere reference implementation. I know of at least > three different alternative implementations, so I thought that the > question were clear enough: Is the timeout mechanism "supposed" to > be re-startable for an api that aims to conform to the "socket" > module, or is that a mere coincidence falling out from the select/bsd > based reference implementation in cPython? I think recv() and send() (and other simple ops) should certainly be restartable. sendall() is another matter, I think it should be considered a best effort thing. > Anyway, as for concrete requirements: The issue I have always seen > with various asynchronous libraries is their lack of composability. Note: if you are using an asynchronous library, you probably shouldn't be using any form of socket timeout. Instead, you should be using timer callbacks as provided by the asynchronous library. (and the sockets themselves, of course, should be put in non-blocking mode) Regards Antoine. From guido at python.org Wed Nov 28 18:04:26 2012 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Nov 2012 09:04:26 -0800 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <EFE3877620384242A686D52278B7CCD329DED074@RKV-IT-EXCH103.ccp.ad.local> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DED074@RKV-IT-EXCH103.ccp.ad.local> Message-ID: <CAP7+vJJKJrYbXFF8EjBSYdicVMKZv1J4A_rc4rdw6VkMkEg6Fg@mail.gmail.com> On Wed, Nov 28, 2012 at 4:13 AM, Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote: > I'm sorry, I thought it was something that people did more often, to create different implementations of of the "socket" api, for which cPython provided a mere reference implementation. I know of at least three different alternative implementations, so I thought that the question were clear enough: Is the timeout mechanism "supposed" to be re-startable for an api that aims to conform to the "socket" module, or is that a mere coincidence falling out from the select/bsd based reference implementation in cPython? The docs don't say either way. We're going to have to decide here, since nobody has thought about this enough apparently. I see two possible answers: we can make it implementation-dependent, or we can require conforming implementations to implement properly restartable semantics (if they support timeouts at all). A third option would be to require these semantics *if the timeout option is supported* but leave it up to the implementation to support it at all (ditto for nonblocking, i.e. timeout=0). > (For c-level timeout mechanisms implemented for various c implementations of the bsd socket api, it is not uncommon to see it stated that after a socket operation times out, the socket is in an undefined state and should be discarded, e.g. here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms740476(v=vs.85).aspx " If a send or receive operation times out on a socket, the socket state is indeterminate, and should not be used; TCP sockets in this state have a potential for data loss, since the operation could be canceled at the same moment the operation was to be completed.") Is this relevant to CPython though? Its socket implementation uses select() to implement timeouts, even on Windows, AFAIK. > Anyway, as for concrete requirements: The issue I have always seen with various asynchronous libraries is their lack of composability. Everyone writes their own application loop and event queue. Merely having a standard spec and reference implementation of an application main loop object, and main event queue object, in the spirit of WSGI, would possibly remedy this. You could then hopefully assemble various different libraries in the same application, including greenlet(*) based ones. Hm. I agree with the first part of this -- and indeed I am planning to make it so that tulip's event loop can easily be replaced by another one. I'm less sure about the yield-from-based scheduler, that's the kind of thing for which it doesn't really make sense to have multiple implementations. If greenlets can work with the standard event loop interface, good for them. (Either by providing a conforming implementation that also supports greenlets, or by just using the standard implementation.) > (*) Greenlets or stackless can be just another way of hiding asynchronous operations from the programmer. My favourite one, in fact. The main trick here is unwinding and replaying of calling contexts, the specific implementation by stack-slicking is mere technical detail, since it can be achieved in other ways (see soft-switching in stackless python) Yes. While none of these belong in the stdlib, I certainly want to support their viability as a 3rd party alternative to yield-from. Note however that even Christian Tismer has expressed doubts about hiding async blocks completely -- for the same reasons I'm not keen on them myself: when it's not obvious whether a particular call can cause a context switch (e.g. due to something it uses indirectly doing some kind of I/O that requires a context switch to avoid blocking), you're back in the world of threads and explicit locks and all the nightmares that come with that. Using yield or yield-from to mark switch points means you will always be aware of the possibility that a call switches (unfortunately there are other costs). --Guido > Cheers, > > K > >> -----Original Message----- >> From: gvanrossum at gmail.com [mailto:gvanrossum at gmail.com] On Behalf >> Of Guido van Rossum >> Sent: 27. n?vember 2012 15:54 > with stackless python. >> >> It would have been nice if you had given more context and stated your >> objective upfront instead of asking what appeared to be an obscure question >> about a technical detail > >> Finally, I am not at all interested in greenlets >> ... >> very much unspecified at this point. NOW WOULD BE A GOOD TIME TO >> WRITE DOWN YOUR REQUIREMENTS. >> >> (*) Greenlets are a fine mechanism for some application areas, but ultimately >> not fit for the standard library, and they have some significant downsides. >> > > -- --Guido van Rossum (python.org/~guido) From glyph at twistedmatrix.com Wed Nov 28 20:15:07 2012 From: glyph at twistedmatrix.com (Glyph) Date: Wed, 28 Nov 2012 14:15:07 -0500 Subject: [Python-Dev] Socket timeout and completion based sockets In-Reply-To: <CAP7+vJJKJrYbXFF8EjBSYdicVMKZv1J4A_rc4rdw6VkMkEg6Fg@mail.gmail.com> References: <EFE3877620384242A686D52278B7CCD329DEB553@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJWDEwAvKB1VVrz-RX6b9kO3TpxnwUGnoq57746MV1WFg@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DEBF4B@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJKpeEDV9ZpvJqN2_joOt4cKj9B1BJ5gYeZusnwcnu_VwQ@mail.gmail.com> <EFE3877620384242A686D52278B7CCD329DED074@RKV-IT-EXCH103.ccp.ad.local> <CAP7+vJJKJrYbXFF8EjBSYdicVMKZv1J4A_rc4rdw6VkMkEg6Fg@mail.gmail.com> Message-ID: <1D9BE0CD-5BF4-480D-8D40-5A409E40760D@twistedmatrix.com> On Nov 28, 2012, at 12:04 PM, Guido van Rossum <guido at python.org> wrote: >> Anyway, as for concrete requirements: The issue I have always seen with various asynchronous libraries is their lack of composability. Everyone writes their own application loop and event queue. Merely having a standard spec and reference implementation of an application main loop object, and main event queue object, in the spirit of WSGI, would possibly remedy this. You could then hopefully assemble various different libraries in the same application, including greenlet(*) based ones. > > Hm. I agree with the first part of this -- and indeed I am planning to > make it so that tulip's event loop can easily be replaced by another > one. I'm less sure about the yield-from-based scheduler, that's the > kind of thing for which it doesn't really make sense to have multiple > implementations. If greenlets can work with the standard event loop > interface, good for them. (Either by providing a conforming > implementation that also supports greenlets, or by just using the > standard implementation.) I'm really happy that you are building this in as a core feature of Tulip. It's really important. Very early on, Twisted attempted to avoid this lack of composability by explicitly delegating to other application loops; it's one of my favorite features of Twisted. Granted, no two loops we have attempted to use have themselves been composable, but there's not much we can do about that :). Still, code written on top of Twisted can always be plugged in to any other loop by simply using the appropriate reactor. (There's also a plug-in interface for the reactor and a plug-in discovery mechanism so that third parties can easily provide their own reactors if they have an unusual main loop that isn't supported by twisted itself.) I would also like to bring up <https://github.com/lvh/async-pep> again. If anyone really wants to dig in and enumerate the use-cases for the _lower-level_ event delivery portions of Tulip, something that would be compatible with Twisted and Tornado and so on, that PEP already has a good skeleton and could use some pull requests. -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121128/d75584b0/attachment.html> From eliben at gmail.com Fri Nov 30 05:55:54 2012 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 29 Nov 2012 20:55:54 -0800 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> Message-ID: <CAF-Rda9V88s+51zqf42_uz0seVeTpM0=K-Nhc+vtNk30J_DaeA@mail.gmail.com> On Sun, Nov 25, 2012 at 9:01 PM, Chris Jerdonek <chris.jerdonek at gmail.com>wrote: > I would like to know when we should use "class" in the Python 3 > documentation, and when we should use "type." Are these terms > synonymous in Python 3, and do we have a preference for which to use > and when? > > I'm sure this has been discussed before. But if this terminology > issue has already been resolved, the resolution doesn't seem to be > reflected in the docs. For example, the glossary entries for type and > class don't reference each other. > Good question, [shameless plug follows, I post this because I truly believe it's very relevant to the discussion] I had the same doubts some months ago, which led to writing this article (relevant to Python 3): http://eli.thegreenplace.net/2012/03/30/python-objects-types-classes-and-instances-a-glossary/ It examines the class vs. type issue, as well as object vs. instance And this diagram can also be useful to understand how similar built-in and user-defined types are in Python 3: http://eli.thegreenplace.net/2012/04/03/the-fundamental-types-of-python-a-diagram/ Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121129/bf5d19a4/attachment.html> From status at bugs.python.org Fri Nov 30 18:07:23 2012 From: status at bugs.python.org (Python tracker) Date: Fri, 30 Nov 2012 18:07:23 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20121130170723.D44591C98E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2012-11-23 - 2012-11-30) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3792 ( -7) closed 24566 (+48) total 28358 (+41) Open issues with patches: 1653 Issues opened (27) ================== #8824: Improve documentation of exec http://bugs.python.org/issue8824 reopened by mark.dickinson #9400: multiprocessing.pool.AsyncResult.get() messes up exceptions http://bugs.python.org/issue9400 reopened by sbt #15587: IDLE is pixelated on the Macbook Pro with Retina Display http://bugs.python.org/issue15587 reopened by Tyler.Crompton #16541: tk_setPalette doesn't accept keyword parameters http://bugs.python.org/issue16541 opened by HJarausch #16543: improve TypeError messages for missing arguments (meta issue) http://bugs.python.org/issue16543 opened by ezio.melotti #16544: Add external link to ast docs http://bugs.python.org/issue16544 opened by asvetlov #16547: IDLE raises an exception in tkinter after fresh file's text ha http://bugs.python.org/issue16547 opened by ebfe #16550: pickletools.py treats 32bit lengths as signed, but pickle.py a http://bugs.python.org/issue16550 opened by serhiy.storchaka #16551: Cleanup the pure Python pickle implementation http://bugs.python.org/issue16551 opened by serhiy.storchaka #16554: The description of the argument of MAKE_FUNCTION and MAKE_CLOS http://bugs.python.org/issue16554 opened by daniel.urban #16555: Add es_cu to locale aliases http://bugs.python.org/issue16555 opened by Leiser.Fern??ndez.Gallo #16557: PEP 380 isn't reflected in the Functional Programming HOWTO http://bugs.python.org/issue16557 opened by msmhrt #16561: Windows installer doesn't use UAC, then crashes http://bugs.python.org/issue16561 opened by Redoute #16562: Optimize dict equality test http://bugs.python.org/issue16562 opened by rhettinger #16564: email.generator.BytesGenerator fails with bytes payload http://bugs.python.org/issue16564 opened by Alexander.Kruppa #16565: Increase Py_AddPendingCall array size http://bugs.python.org/issue16565 opened by felipecruz #16566: Structure._anonymous_ should not allow strings http://bugs.python.org/issue16566 opened by techtonik #16568: allow constructors to be documented separately from class http://bugs.python.org/issue16568 opened by chris.jerdonek #16569: Preventing errors of simultaneous access in zipfile http://bugs.python.org/issue16569 opened by serhiy.storchaka #16572: Bad multi-inheritance support in some libs like threading or m http://bugs.python.org/issue16572 opened by thomas.chiroux #16574: clarify policy on updates to final peps http://bugs.python.org/issue16574 opened by chris.jerdonek #16575: ctypes: unions as arguments http://bugs.python.org/issue16575 opened by arigo #16576: ctypes: structure with bitfields as argument http://bugs.python.org/issue16576 opened by arigo #16577: Suspect test.test_codeccallbacks.test_mutatingdecodehandler http://bugs.python.org/issue16577 opened by amaury.forgeotdarc #16579: .pyw disturb multiprocessing behavior http://bugs.python.org/issue16579 opened by Alex.stein #16580: Add examples to int.to_bytres and int.from_bytes http://bugs.python.org/issue16580 opened by paddy3118 #16581: define "PEP editor" in PEP 1 http://bugs.python.org/issue16581 opened by chris.jerdonek Most recent 15 issues with no replies (15) ========================================== #16581: define "PEP editor" in PEP 1 http://bugs.python.org/issue16581 #16580: Add examples to int.to_bytres and int.from_bytes http://bugs.python.org/issue16580 #16575: ctypes: unions as arguments http://bugs.python.org/issue16575 #16561: Windows installer doesn't use UAC, then crashes http://bugs.python.org/issue16561 #16557: PEP 380 isn't reflected in the Functional Programming HOWTO http://bugs.python.org/issue16557 #16551: Cleanup the pure Python pickle implementation http://bugs.python.org/issue16551 #16550: pickletools.py treats 32bit lengths as signed, but pickle.py a http://bugs.python.org/issue16550 #16516: argparse types (and actions) must be hashable http://bugs.python.org/issue16516 #16509: sqlite3 docs do not explain check_same_thread http://bugs.python.org/issue16509 #16494: Add a method on importlib.SourceLoader for creating bytecode f http://bugs.python.org/issue16494 #16492: Add a load_parents argument to importlib.find_loader() http://bugs.python.org/issue16492 #16486: Add context manager support to aifc module http://bugs.python.org/issue16486 #16463: test_timeout failure on the RHEL buildbot http://bugs.python.org/issue16463 #16450: test_missing_localfile masks problems in urlopen http://bugs.python.org/issue16450 #16429: Emit SyntaxWarning for code that risks UnboundLocalError http://bugs.python.org/issue16429 Most recent 15 issues waiting for review (15) ============================================= #16572: Bad multi-inheritance support in some libs like threading or m http://bugs.python.org/issue16572 #16569: Preventing errors of simultaneous access in zipfile http://bugs.python.org/issue16569 #16562: Optimize dict equality test http://bugs.python.org/issue16562 #16554: The description of the argument of MAKE_FUNCTION and MAKE_CLOS http://bugs.python.org/issue16554 #16551: Cleanup the pure Python pickle implementation http://bugs.python.org/issue16551 #16550: pickletools.py treats 32bit lengths as signed, but pickle.py a http://bugs.python.org/issue16550 #16543: improve TypeError messages for missing arguments (meta issue) http://bugs.python.org/issue16543 #16537: Python???s setup.py raises a ValueError when self.extensions i http://bugs.python.org/issue16537 #16525: wave file module does not support 32bit float format http://bugs.python.org/issue16525 #16523: attrgetter and itemgetter signatures in docs need cleanup http://bugs.python.org/issue16523 #16515: TypeError message incorrect for max() with one keyword arg http://bugs.python.org/issue16515 #16512: imghdr doesn't support jpegs with an ICC profile http://bugs.python.org/issue16512 #16511: IDLE configuration file: blank height and width fields trip up http://bugs.python.org/issue16511 #16510: Using appropriate checks in tests http://bugs.python.org/issue16510 #16507: Patch selectmodule.c to support WSAPoll on Windows http://bugs.python.org/issue16507 Top 10 most discussed issues (10) ================================= #16543: improve TypeError messages for missing arguments (meta issue) http://bugs.python.org/issue16543 21 msgs #16518: add "buffer protocol" to glossary http://bugs.python.org/issue16518 15 msgs #16574: clarify policy on updates to final peps http://bugs.python.org/issue16574 15 msgs #4591: 32-bits unsigned user/group identifier http://bugs.python.org/issue4591 11 msgs #4945: json checks True/False by identity, not boolean value http://bugs.python.org/issue4945 11 msgs #16547: IDLE raises an exception in tkinter after fresh file's text ha http://bugs.python.org/issue16547 9 msgs #7976: warnings should provide a public API for accessing its option http://bugs.python.org/issue7976 8 msgs #16565: Increase Py_AddPendingCall array size http://bugs.python.org/issue16565 8 msgs #16566: Structure._anonymous_ should not allow strings http://bugs.python.org/issue16566 8 msgs #11175: allow argparse FileType to accept encoding and errors argument http://bugs.python.org/issue11175 6 msgs Issues closed (45) ================== #1827: svnversion_init() doesn't support svn urls in sandbox/trunk http://bugs.python.org/issue1827 closed by christian.heimes #1977: Python reinitialization test http://bugs.python.org/issue1977 closed by christian.heimes #2039: Pymalloc patch for int and float objects http://bugs.python.org/issue2039 closed by christian.heimes #3410: platform.version() don't work as expected in Vista in portugue http://bugs.python.org/issue3410 closed by ezio.melotti #4473: POP3 missing support for starttls http://bugs.python.org/issue4473 closed by pitrou #9011: ast_for_factor unary minus optimization changes AST http://bugs.python.org/issue9011 closed by mark.dickinson #9176: module termios doesn't build on HP-UX http://bugs.python.org/issue9176 closed by skrah #10259: Entry text not set if all of 'Font', 'Foreground' and 'Justify http://bugs.python.org/issue10259 closed by gpolo #11076: Iterable argparse Namespace http://bugs.python.org/issue11076 closed by asvetlov #12848: pickle.py treats 32bit lengths as signed, but _pickle.c as uns http://bugs.python.org/issue12848 closed by pitrou #14478: Decimal hashing very slow, could be cached http://bugs.python.org/issue14478 closed by mark.dickinson #15990: solidify argument/parameter terminology http://bugs.python.org/issue15990 closed by chris.jerdonek #16205: update :class:`str` references to link to the str type section http://bugs.python.org/issue16205 closed by chris.jerdonek #16209: add a "class str" entry to the docs http://bugs.python.org/issue16209 closed by chris.jerdonek #16323: Wrong C API documentation for locale encoding http://bugs.python.org/issue16323 closed by asvetlov #16333: Trailing whitespace in json dump when using indent http://bugs.python.org/issue16333 closed by ezio.melotti #16339: Document "exec(stmt, global_dict, local_dict)" form in Python http://bugs.python.org/issue16339 closed by mark.dickinson #16423: urllib data URL http://bugs.python.org/issue16423 closed by pitrou #16464: urllib.request: opener not resetting content-length http://bugs.python.org/issue16464 closed by asvetlov #16477: tarfile fails to close file handles in case of exception http://bugs.python.org/issue16477 closed by asvetlov #16483: Make int(float('inf')) raise ValueError rather than OverflowEr http://bugs.python.org/issue16483 closed by mark.dickinson #16519: site.venv() should use abspath(executable) http://bugs.python.org/issue16519 closed by python-dev #16521: logging.basicConfig creates empty file when using handlers http://bugs.python.org/issue16521 closed by python-dev #16524: File access not always working with Python for Windows 32 bits http://bugs.python.org/issue16524 closed by terry.reedy #16530: documentation of os.wait3 http://bugs.python.org/issue16530 closed by ezio.melotti #16532: AMD64 Windows 7 build failures http://bugs.python.org/issue16532 closed by pitrou #16534: -Olimit: unsupported option: warnings and ./configure failures http://bugs.python.org/issue16534 closed by skrah #16540: Make itertools count, cycle, and repeat objects subscriptable http://bugs.python.org/issue16540 closed by rhettinger #16542: http//bugs.python/joko.suwito http://bugs.python.org/issue16542 closed by ezio.melotti #16545: ast.FunctionDef sets a bad value for kw_defaults when keyword- http://bugs.python.org/issue16545 closed by brett.cannon #16546: ast.YieldFrom needlessly has its expr value as optional http://bugs.python.org/issue16546 closed by mark.dickinson #16548: os.system won't run any command and there is no error message http://bugs.python.org/issue16548 closed by r.david.murray #16549: regression: -m json.tool module is broken http://bugs.python.org/issue16549 closed by ezio.melotti #16552: os.path.basename() docs should link to os.path.split() http://bugs.python.org/issue16552 closed by chris.jerdonek #16553: named kwd form of OrderedDict ctor yield random key order http://bugs.python.org/issue16553 closed by benjamin.peterson #16556: Update string.Formatter.vformat documentation to say "**kwargs http://bugs.python.org/issue16556 closed by ezio.melotti #16558: multiprocessing fails to raise exception with parameters http://bugs.python.org/issue16558 closed by sbt #16559: Update JSON??tests http://bugs.python.org/issue16559 closed by serhiy.storchaka #16560: Python sighandlers delayed for no reason http://bugs.python.org/issue16560 closed by zdenek.pavlas #16563: re.match loops forever on simple regexp http://bugs.python.org/issue16563 closed by mark.dickinson #16567: Implementing .= for variable operator http://bugs.python.org/issue16567 closed by amaury.forgeotdarc #16570: Absolute imports fail to take full path into account? http://bugs.python.org/issue16570 closed by brett.cannon #16571: Iterating over inconsistently-indented code block causes execu http://bugs.python.org/issue16571 closed by mark.dickinson #16573: 2to3 should treat enumerate like sorted for zip, map, filter, http://bugs.python.org/issue16573 closed by python-dev #16578: Regular expressions with empty named groups break isname check http://bugs.python.org/issue16578 closed by Gabriel.Rodr??guez.Alberich From brett at python.org Fri Nov 30 20:38:12 2012 From: brett at python.org (Brett Cannon) Date: Fri, 30 Nov 2012 14:38:12 -0500 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20121130170723.D44591C98E@psf.upfronthosting.co.za> References: <20121130170723.D44591C98E@psf.upfronthosting.co.za> Message-ID: <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> Do we have a graph of the historical trend of the number of bugs (or at least the historical details stored somewhere)? I think we have had a net decrease in open bugs the last couple of weeks and it would be neat to see an absolute and relative graph of the overall trend since Python 3.3.0 was released. Also might make a nice motivator to try to close issues faster. =) Otherwise is the code public for this somewhere? I assume it's making an XML-RPC call or something every week to get the results, but if I decide to do a little App Engine app to store historical data and do a graph I would rather not have to figure all of this out from scratch. =) Although I could I guess also parse the email if I wanted to ignore all other emails. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121130/716191d4/attachment.html> From tjreedy at udel.edu Fri Nov 30 21:15:54 2012 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 30 Nov 2012 15:15:54 -0500 Subject: [Python-Dev] type vs. class terminology In-Reply-To: <CAF-Rda9V88s+51zqf42_uz0seVeTpM0=K-Nhc+vtNk30J_DaeA@mail.gmail.com> References: <CAOTb1wdDvta+DFBeGHaOtU4Ftsq2xLZrfrL2i-Sdk22sZoYReQ@mail.gmail.com> <CAF-Rda9V88s+51zqf42_uz0seVeTpM0=K-Nhc+vtNk30J_DaeA@mail.gmail.com> Message-ID: <k9b46e$j6i$1@ger.gmane.org> On 11/29/2012 11:55 PM, Eli Bendersky wrote: > On Sun, Nov 25, 2012 at 9:01 PM, Chris Jerdonek > <chris.jerdonek at gmail.com <mailto:chris.jerdonek at gmail.com>> wrote: > > I would like to know when we should use "class" in the Python 3 > documentation, and when we should use "type." Are these terms > synonymous in Python 3, and do we have a preference for which to use > and when? > > I'm sure this has been discussed before. But if this terminology > issue has already been resolved, the resolution doesn't seem to be > reflected in the docs. For example, the glossary entries for type and > class don't reference each other. > > > Good question, > > [shameless plug follows, I post this because I truly believe it's very > relevant to the discussion] > > I had the same doubts some months ago, which led to writing this article > (relevant to Python 3): > http://eli.thegreenplace.net/2012/03/30/python-objects-types-classes-and-instances-a-glossary/ > It examines the class vs. type issue, as well as object vs. instance You usage seems to me to be stuck in the now mostly useless Python 1 distinction between built-in types and user-defined classes. In Python 3, all instances of type are classes, whether defined with the C or Python API. Indeed, the existence of a C API to make classes is an implementation detail, not a language feature. The second parameter of isinstance or issubclass is a class or set thereof (implemented as a (homogeneous!) tuple for historical reasons), without distinction of definition mode. When using an imported class, it mostly does not matter how the class was defined. I agree with Guido that it is more useful to use 'class' for the concrete run-time object and 'type' (except when referring to the object of that name) for abstract (and static) types. (From this viewpoint, duck-typing rather than duck-classing *is* the proper term). Consider the quote from the manual. "An object?s type determines the operations that the object supports (e.g., ?does it have a length??)." An object potentially supports len(), and one might say should support it, if and only if it is a 'finite collection'. That is a 'type' (duck type) of object, but not a class in the Python sense. Whether an object actually supports len depends on its run-time class. Similarly, an object 'should' support sqrt if it is a non-negative scalar number or a complex number. Square-root-able is also an abstract type, not a concrete class. I might suggest that types are used to reason about programs while classes are used to execute programs. -- Terry Jan Reedy From storchaka at gmail.com Fri Nov 30 21:34:37 2012 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 30 Nov 2012 22:34:37 +0200 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> References: <20121130170723.D44591C98E@psf.upfronthosting.co.za> <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> Message-ID: <k9b58u$rvs$1@ger.gmane.org> On 30.11.12 21:38, Brett Cannon wrote: > Also might make a nice motivator to try to close issues faster. =) May be introduce the title of "Issue closer of the month"? ;) From rdmurray at bitdance.com Fri Nov 30 22:07:47 2012 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 30 Nov 2012 16:07:47 -0500 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> References: <20121130170723.D44591C98E@psf.upfronthosting.co.za> <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> Message-ID: <20121130210748.3679B250117@webabinitio.net> On Fri, 30 Nov 2012 14:38:12 -0500, Brett Cannon <brett at python.org> wrote: > Do we have a graph of the historical trend of the number of bugs (or at > least the historical details stored somewhere)? I think we have had a net Not really. Ezio made one by hand once, but there is nothing automated. The historical details are stored only in the mailing list archives, as far as I know. In theory I think you could re-calculate them from the Roundup DB, but for various reasons the numbers would probably come out slightly different. Still, getting the data from the DB would be better than parsing the emails, since for one reason and another there are missing Friday reports, and reports that were issued on non-Friday dates. > decrease in open bugs the last couple of weeks and it would be neat to see > an absolute and relative graph of the overall trend since Python 3.3.0 was > released. Also might make a nice motivator to try to close issues faster. =) > > Otherwise is the code public for this somewhere? I assume it's making an Yes. It is in the software repository for our roundup instances: http://hg.python.org/tracker/python-dev/file/default/scripts/roundup-summary (Be warned that that isn't the location from which the script is executed, so it is possible for what is actually running to get out of sync with what is checked in at that location.) > XML-RPC call or something every week to get the results, but if I decide to Nope, it talks directly to the DB. And as you will see, it is more than a bit gnarly. > do a little App Engine app to store historical data and do a graph I would > rather not have to figure all of this out from scratch. =) Although I could > I guess also parse the email if I wanted to ignore all other emails. I'm not sure how one would go about integrating the above with an App Engine app. I suspect that not quite enough information is available through the XML-RPC interface to replicate that script, but maybe you could manage just the open-close counting part of it. I haven't looked at what it would take. --David From brett at python.org Fri Nov 30 22:52:43 2012 From: brett at python.org (Brett Cannon) Date: Fri, 30 Nov 2012 16:52:43 -0500 Subject: [Python-Dev] Summary of Python tracker Issues In-Reply-To: <20121130210748.3679B250117@webabinitio.net> References: <20121130170723.D44591C98E@psf.upfronthosting.co.za> <CAP1=2W6U6z0ZwnWYUEdqQKvEuTmsJd0fiuyPeRJN-xv_SR5z9w@mail.gmail.com> <20121130210748.3679B250117@webabinitio.net> Message-ID: <CAP1=2W53qoCp6fy4haHrOwSEH7SReSJdaWNT72HfKfKG6SC4ag@mail.gmail.com> On Fri, Nov 30, 2012 at 4:07 PM, R. David Murray <rdmurray at bitdance.com>wrote: > On Fri, 30 Nov 2012 14:38:12 -0500, Brett Cannon <brett at python.org> wrote: > > Do we have a graph of the historical trend of the number of bugs (or at > > least the historical details stored somewhere)? I think we have had a net > > Not really. Ezio made one by hand once, but there is nothing automated. > > The historical details are stored only in the mailing list archives, as > far as I know. In theory I think you could re-calculate them from the > Roundup DB, but for various reasons the numbers would probably come out > slightly different. Still, getting the data from the DB would be better > than parsing the emails, since for one reason and another there are > missing Friday reports, and reports that were issued on non-Friday > dates. > > > decrease in open bugs the last couple of weeks and it would be neat to > see > > an absolute and relative graph of the overall trend since Python 3.3.0 > was > > released. Also might make a nice motivator to try to close issues > faster. =) > > > > Otherwise is the code public for this somewhere? I assume it's making an > > Yes. It is in the software repository for our roundup instances: > > > http://hg.python.org/tracker/python-dev/file/default/scripts/roundup-summary > > (Be warned that that isn't the location from which the script is > executed, so it is possible for what is actually running to get out of > sync with what is checked in at that location.) > > > XML-RPC call or something every week to get the results, but if I decide > to > > Nope, it talks directly to the DB. And as you will see, it is more > than a bit gnarly. > > I think I could also download the csv file and parse that to get whatever data I wanted. > > do a little App Engine app to store historical data and do a graph I > would > > rather not have to figure all of this out from scratch. =) Although I > could > > I guess also parse the email if I wanted to ignore all other emails. > > I'm not sure how one would go about integrating the above with an App > Engine app. I suspect that not quite enough information is available > through the XML-RPC interface to replicate that script, but maybe you > could manage just the open-close counting part of it. I haven't > looked at what it would take. > It really depends on what statistics I cared about (e.g. there are less than 4000 bugs while there are less than 25,000 closed bugs). If I just did high-level statistics it wouldn't be bad, but if I try to track every issue independently that might be annoying (and actually cost money for me, although I already personally pay for py3ksupport.appspot.com so I can probably piggyback off of that app's quota). We will see if this ever goes anywhere. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20121130/376f598f/attachment.html>