Python-Dev November 2011

python-dev@python.org

73 participants
75 discussions

Long term development external named branches and periodic merges from python

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have a question and I would rather have an answer instead of actually trying and getting myself in a messy situation. Let say we have the following scenario: 1. A programer clones hg.python.org. 2. Programer creates a named branch and start to develop a new feature. 3. She adds her repository&named branch to the bugtracker. 4. From time to time, she posts updates in the tracker using the "Create Patch" button. So far so good. Now, the question: 5. Development of the new feature is taking a long time, and python canonical version keeps moving forward. The clone+branch and the original python version are diverging. Eventually there are changes in python that the programmer would like in her version, so she does a "pull" and then a merge for the original python branch to her named branch. 6. What would be posted in the bug tracker when she does a new "Create Patch"?. Only her changes, her changes SINCE the merge, her changes plus merged changes or something else?. What if the programmer cherrypick changesets from the original python branch?. Thanks! :-). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTs50H5lgi5GaxT1NAQJsTAP6AsUsLo2REdxxyVvPBDQ51GjZermCXD08 jOqKkKY9cre4OHx/+uZHEvO8j7RJ5X3o2/0Yl4OeDSTBDY8/eWINc9cgtuNqrJdW W27fu1+UTIpgl1oLh06P23ufOEWPWh90gsV6eiVnFlj7r+b3HkP7PNdZCmqU2+UW 92Ac9B1JOvU= =goYv -----END PGP SIGNATURE-----

9 years, 4 months

LZMA support has landed

by Nadeem Vawda

Hey folks, I'm pleased to announce that as of changeset 74d182cf0187, the standard library now includes support for the LZMA compression algorithm (as well as the associated .xz and .lzma file formats). The new lzma module has a very similar API to the existing bz2 module; it should serve as a drop-in replacement for most use cases. If anyone has any feedback or any suggestions for improvement, please let me know. I'd like to ask the owners of (non-Windows) buildbots to install the XZ Utils development headers so that they can build the new module. For Debian-derived Linux distros, the relevant package is named "liblzma-dev"; on Fedora I believe the correct package is "xz-devel". Binaries for OS X are available from the project's homepage at <http://tukaani.org/xz/>. Finally, a big thanks to everyone who contributed feedback during this module's development! Cheers, Nadeem

9 years, 4 months

Deprecation policy

by Ezio Melotti

Hi, our current deprecation policy is not so well defined (see e.g. [0]), and it seems to me that it's something like: 1) deprecate something and add a DeprecationWarning; 2) forget about it after a while; 3) wait a few versions until someone notices it; 4) actually remove it; I suggest to follow the following process: 1) deprecate something and add a DeprecationWarning; 2) decide how long the deprecation should last; 3) use the deprecated-remove[1] directive to document it; 4) add a test that fails after the update so that we remember to remove it[2]; Other related issues: PendingDeprecationWarnings: * AFAIK the difference between PDW and DW is that PDW are silenced by default; * now DW are silence by default too, so there are no differences; * I therefore suggest we stop using it, but we can leave it around[3] (other projects might be using it for something different); Deprecation Progression: Before, we more or less used to deprecated in release X and remove in X+1, or add a PDW in X, DW in X+1, and remove it in X+2. I suggest we drop this scheme and just use DW until X+N, where N is >=1 and depends on what is being removed. We can decide to leave the DW for 2-3 versions before removing something widely used, or just deprecate in X and remove in X+1 for things that are less used. Porting from 2.x to 3.x: Some people will update directly from 2.7 to 3.2 or even later versions (3.3, 3.4, ...), without going through earlier 3.x versions. If something is deprecated on 3.2 but not in 2.7 and then is removed in 3.3, people updating from 2.7 to 3.3 won't see any warning, and this will make the porting even more difficult. I suggest that: * nothing that is available and not deprecated in 2.7, will be removed until 3.x (x needs to be defined); * possibly we start backporting warnings to 2.7 so that they are visible while running with -3; Documenting the deprecations: In order to advertise the deprecations, they should be documented: * in their doc, using the deprecated-removed directive (and possibly not the 'deprecated' one); * in the what's new, possibly listing everything that is currently deprecated, and when it will be removed; Django seems to do something similar[4]. (Another thing I would like is a different rending for deprecated functions. Some part of the docs have a deprecation warning on the top of the section and the single functions look normal if you miss that. Also while linking to a deprecated function it would be nice to have it rendered with a different color or something similar.) Testing the deprecations: Tests that fail when a new release is made and the version number is bumped should be added to make sure we don't forget to remove it. The test should have a related issue with a patch to remove the deprecated function and the test. Setting the priority of the issue to release blocker or deferred blocker can be done in addition/instead, but that works well only when N == 1 (the priority could be updated for every release though). The tests could be marked with an expected failure to give some time after the release to remove them. All the deprecation-related tests might be added to the same file, or left in the test file of their module. Where to add this: Once we agree about the process we should write it down somewhere. Possible candidates are: * PEP387: Backwards Compatibility Policy[5] (it has a few lines about this); * a new PEP; * the devguide; I think having it in a PEP would be good, the devguide can then link to it. Best Regards, Ezio Melotti [0]: http://bugs.python.org/issue13248 [1]: deprecated-removed doesn't seem to be documented in the documenting doc, but it was added here: http://hg.python.org/cpython/rev/03296316a892 [2]: see e.g. http://hg.python.org/cpython/file/default/Lib/unittest/test/test_case.py#l1… [3]: we could also introduce a MetaDeprecationWarning and make PendingDeprecationWarning inherit from it so that it can be used to pending-deprecate itself. Once PendingDeprecationWarning is gone, the MetaDeprecationWarning will become useless and can then be used to meta-deprecate itself. [4]: https://docs.djangoproject.com/en/dev/internals/deprecation/ [5]: http://www.python.org/dev/peps/pep-0387/

9 years, 4 months

Re: [Python-Dev] cpython: Issue #6715: Add module for compression using the LZMA algorithm.

by Antoine Pitrou

On Tue, 29 Nov 2011 23:36:58 +0100 nadeem.vawda <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/74d182cf0187 > changeset: 73794:74d182cf0187 > user: Nadeem Vawda <nadeem.vawda(a)gmail.com> > date: Wed Nov 30 00:25:06 2011 +0200 > summary: > Issue #6715: Add module for compression using the LZMA algorithm. Congratulations, Nadeem! Regards Antoine.

9 years, 4 months

PyPy 1.7 - widening the sweet spot

by Maciej Fijalkowski

================================== PyPy 1.7 - widening the sweet spot ================================== We're pleased to announce the 1.7 release of PyPy. As became a habit, this release brings a lot of bugfixes and performance improvements over the 1.6 release. However, unlike the previous releases, the focus has been on widening the "sweet spot" of PyPy. That is, classes of Python code that PyPy can greatly speed up should be vastly improved with this release. You can download the 1.7 release here: http://pypy.org/download.html What is PyPy? ============= PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (`pypy 1.7 and cpython 2.7.1`_ performance comparison) due to its integrated tracing JIT compiler. This release supports x86 machines running Linux 32/64, Mac OS X 32/64 or Windows 32. Windows 64 work is ongoing, but not yet natively supported. The main topic of this release is widening the range of code which PyPy can greatly speed up. On average on our benchmark suite, PyPy 1.7 is around **30%** faster than PyPy 1.6 and up to **20 times** faster on some benchmarks. .. _`pypy 1.7 and cpython 2.7.1`: http://speed.pypy.org Highlights ========== * Numerous performance improvements. There are too many examples which python constructs now should behave faster to list them. * Bugfixes and compatibility fixes with CPython. * Windows fixes. * PyPy now comes with stackless features enabled by default. However, any loop using stackless features will interrupt the JIT for now, so no real performance improvement for stackless-based programs. Contact pypy-dev for info how to help on removing this restriction. * NumPy effort in PyPy was renamed numpypy. In order to try using it, simply write:: import numpypy as numpy at the beginning of your program. There is a huge progress on numpy in PyPy since 1.6, the main feature being implementation of dtypes. * JSON encoder (but not decoder) has been replaced with a new one. This one is written in pure Python, but is known to outperform CPython's C extension up to **2 times** in some cases. It's about **20 times** faster than the one that we had in 1.6. * The memory footprint of some of our RPython modules has been drastically improved. This should impact any applications using for example cryptography, like tornado. * There was some progress in exposing even more CPython C API via cpyext. Things that didn't make it, expect in 1.8 soon ============================================== There is an ongoing work, which while didn't make it to the release, is probably worth mentioning here. This is what you should probably expect in 1.8 some time soon: * Specialized list implementation. There is a branch that implements lists of integers/floats/strings as compactly as array.array. This should drastically improve performance/memory impact of some applications * NumPy effort is progressing forward, with multi-dimensional arrays coming soon. * There are two brand new JIT assembler backends, notably for the PowerPC and ARM processors. Fundraising =========== It's maybe worth mentioning that we're running fundraising campaigns for NumPy effort in PyPy and for Python 3 in PyPy. In case you want to see any of those happen faster, we urge you to donate to `numpy proposal`_ or `py3k proposal`_. In case you want PyPy to progress, but you trust us with the general direction, you can always donate to the `general pot`_. .. _`numpy proposal`: http://pypy.org/numpydonate.html .. _`py3k proposal`: http://pypy.org/py3donate.html .. _`general pot`: http://pypy.org

9 years, 4 months

Merging 3.2 to 3.3 is messy because "Misc/NEWS"

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 When merging from 3.2 to 3.3 "Misc/NEWS" always conflicts (lately). Instead of copy&paste the test manually between versions, has anybody a better workflow?. Since any change applied to 3.2 should be applied to 3.3 too (except very few cases), Mercurial merge machinery should be able to merge both versions except when the changes are very near the version headers. I haven't checked, but I guess that the problem is that the different issues have been added in different positions in the file, so both branches are diverging, instead of only divert in the python versions referenced. If that is the case, could be acceptable to reorganize 3.3 version to ease future merges?. Would that solve it? Ideas?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTrlPl5lgi5GaxT1NAQKVggP/bn6vUhQlHjEYg+pFEInnVXYSudamPafP m6bgX6hKS/MtaixVJGlRnAwJ6UQ/nftjmVn80Yd7CsxnsyPApUZVgzkaLMLOhh++ H08gwxgoh1skciYmtyjsy4Vi4xi/4tehu2IVc73SVXkLVbnkc4z1c2Xmsu4TZ2ai r2ncgxRkHgw= =pCHL -----END PGP SIGNATURE-----

9 years, 4 months

404 in (important) documentation in www.python.org and contributor agreement

by Jesus Cea

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Trying to clear the licensing issues surrounding my DTrace work (http://bugs.python.org/issue13405) I am contacting Sun/Oracle guys. Checking documentation abut the contributor license agreement, I had encounter a wrong HTML link in http://www.python.org/about/help/ : * "Python Patch Guidelines" points to http://www.python.org/dev/patches/, that doesn't exist. Other links in that page seems OK. PS: The devguide doesn't say anything (AFAIK) about the contributor agreement. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTs7ayZlgi5GaxT1NAQLOfwQAoa1GFuQZKhbXD3FnmH3XUiylzTMBmXMh vB++AdDP8fcEBC/NYZ9j0DH+AGspXrPg4YVta09zJJ/1kHa2UxRVmtXM8centl3V Jkad+6lJw6YYjtXXgM4QExlzClsYNn1ByhYaRSiSa8g9dtsFq4YTlKzfeAXLPC50 DUju8DavMyo= =xOEe -----END PGP SIGNATURE-----

9 years, 4 months

PEP 380

by Guido van Rossum

Mea culpa for not keeping track, but what's the status of PEP 380? I really want this in Python 3.3! -- --Guido van Rossum (python.org/~guido)

9 years, 4 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2011-11-18 - 2011-11-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3134 (+19) closed 22128 (+31) total 25262 (+50) Open issues with patches: 1328 Issues opened (41) ================== #2286: Stack overflow exception caused by test_marshal on Windows x64 http://bugs.python.org/issue2286 reopened by brian.curtin #13387: suggest assertIs(type(obj), cls) for exact type checking http://bugs.python.org/issue13387 reopened by eric.araujo #13433: String format documentation contains error regarding %g http://bugs.python.org/issue13433 opened by Christian.Iversen #13434: time.xmlrpc.com dead http://bugs.python.org/issue13434 opened by pitrou #13435: Copybutton does not hide tracebacks http://bugs.python.org/issue13435 opened by lehmannro #13436: compile() doesn't work on ImportFrom with level=None http://bugs.python.org/issue13436 opened by Janosch.Gräf #13437: Provide links to the source code for every module in the docum http://bugs.python.org/issue13437 opened by Julian #13438: "Delete patch set" review action doesn't work http://bugs.python.org/issue13438 opened by Oleg.Plakhotnyuk #13439: turtle: Errors in docstrings of onkey and onkeypress http://bugs.python.org/issue13439 opened by smichr #13440: Explain the "status quo wins a stalemate" principle in the dev http://bugs.python.org/issue13440 opened by ncoghlan #13441: TestEnUSCollation.test_strxfrm() fails on Solaris http://bugs.python.org/issue13441 opened by haypo #13443: wrong links and examples in the functional HOWTO http://bugs.python.org/issue13443 opened by eli.bendersky #13444: closed stdout causes error on stderr when the interpreter unco http://bugs.python.org/issue13444 opened by Ronny.Pfannschmidt #13445: Enable linking the module pysqlite with Berkeley DB SQL instea http://bugs.python.org/issue13445 opened by Lauren.Foutz #13446: imaplib, fetch: improper behaviour on read-only selected mailb http://bugs.python.org/issue13446 opened by char.nikolaou #13447: Add tests for Tools/scripts/reindent.py http://bugs.python.org/issue13447 opened by eric.araujo #13448: PEP 3155 implementation http://bugs.python.org/issue13448 opened by pitrou #13449: sched - provide an "async" argument for run() method http://bugs.python.org/issue13449 opened by giampaolo.rodola #13450: add assertions to implement the intent in ''.format_map test http://bugs.python.org/issue13450 opened by akira #13451: sched.py: speedup cancel() method http://bugs.python.org/issue13451 opened by giampaolo.rodola #13452: PyUnicode_EncodeDecimal: reject error handlers different than http://bugs.python.org/issue13452 opened by haypo #13453: Tests and network timeouts http://bugs.python.org/issue13453 opened by haypo #13454: crash when deleting one pair from tee() http://bugs.python.org/issue13454 opened by PyryP #13455: Reorganize tracker docs in the devguide http://bugs.python.org/issue13455 opened by ezio.melotti #13456: Providing a custom HTTPResponse class to HTTPConnection http://bugs.python.org/issue13456 opened by r.david.murray #13461: Error on test_issue_1395_5 with Python 2.7 and VS2010 http://bugs.python.org/issue13461 opened by sable #13462: Improve code and tests for Mixin2to3 http://bugs.python.org/issue13462 opened by eric.araujo #13463: Fix parsing of package_data http://bugs.python.org/issue13463 opened by eric.araujo #13464: HTTPResponse is missing an implementation of readinto http://bugs.python.org/issue13464 opened by r.david.murray #13465: A Jython section in the dev guide would be great http://bugs.python.org/issue13465 opened by fwierzbicki #13466: new timezones http://bugs.python.org/issue13466 opened by Rioky #13467: Typo in doc for library/sysconfig http://bugs.python.org/issue13467 opened by naoki #13471: setting access time beyond Jan. 2038 on remote share failes on http://bugs.python.org/issue13471 opened by Thorsten.Simons #13472: devguide doesn’t list all build dependencies http://bugs.python.org/issue13472 opened by eric.araujo #13473: Add tests for files byte-compiled by distutils[2] http://bugs.python.org/issue13473 opened by eric.araujo #13474: Mention of "-m" Flag Missing From Doc on Execution Model http://bugs.python.org/issue13474 opened by eric.snow #13475: Add '-p'/'--path0' command line option to override sys.path[0] http://bugs.python.org/issue13475 opened by ncoghlan #13476: Simple exclusion filter for unittest autodiscovery http://bugs.python.org/issue13476 opened by ncoghlan #13477: tarfile module should have a command line http://bugs.python.org/issue13477 opened by brandon-rhodes #13478: No documentation for timeit.default_timer http://bugs.python.org/issue13478 opened by sandro.tosi #13479: pickle too picky on re-defined classes http://bugs.python.org/issue13479 opened by kxroberto Most recent 15 issues with no replies (15) ========================================== #13478: No documentation for timeit.default_timer http://bugs.python.org/issue13478 #13476: Simple exclusion filter for unittest autodiscovery http://bugs.python.org/issue13476 #13474: Mention of "-m" Flag Missing From Doc on Execution Model http://bugs.python.org/issue13474 #13467: Typo in doc for library/sysconfig http://bugs.python.org/issue13467 #13464: HTTPResponse is missing an implementation of readinto http://bugs.python.org/issue13464 #13463: Fix parsing of package_data http://bugs.python.org/issue13463 #13456: Providing a custom HTTPResponse class to HTTPConnection http://bugs.python.org/issue13456 #13438: "Delete patch set" review action doesn't work http://bugs.python.org/issue13438 #13421: PyCFunction_* are not documented anywhere http://bugs.python.org/issue13421 #13413: time.daylight incorrect behavior in linux glibc http://bugs.python.org/issue13413 #13408: Rename packaging.resources back to datafiles http://bugs.python.org/issue13408 #13403: Option for XMLPRC Server to support HTTPS http://bugs.python.org/issue13403 #13397: Option for XMLRPC clients to automatically transform Fault exc http://bugs.python.org/issue13397 #13372: handle_close called twice in poll2 http://bugs.python.org/issue13372 #13369: timeout with exit code 0 while re-running failed tests http://bugs.python.org/issue13369 Most recent 15 issues waiting for review (15) ============================================= #13473: Add tests for files byte-compiled by distutils[2] http://bugs.python.org/issue13473 #13455: Reorganize tracker docs in the devguide http://bugs.python.org/issue13455 #13452: PyUnicode_EncodeDecimal: reject error handlers different than http://bugs.python.org/issue13452 #13451: sched.py: speedup cancel() method http://bugs.python.org/issue13451 #13450: add assertions to implement the intent in ''.format_map test http://bugs.python.org/issue13450 #13449: sched - provide an "async" argument for run() method http://bugs.python.org/issue13449 #13448: PEP 3155 implementation http://bugs.python.org/issue13448 #13446: imaplib, fetch: improper behaviour on read-only selected mailb http://bugs.python.org/issue13446 #13436: compile() doesn't work on ImportFrom with level=None http://bugs.python.org/issue13436 #13429: provide __file__ to extension init function http://bugs.python.org/issue13429 #13420: newer() function in dep_util.py discard changes in the same se http://bugs.python.org/issue13420 #13410: String formatting bug in interactive mode http://bugs.python.org/issue13410 #13405: Add DTrace probes http://bugs.python.org/issue13405 #13402: Document absoluteness of sys.executable http://bugs.python.org/issue13402 #13396: new method random.getrandbytes() http://bugs.python.org/issue13396 Top 10 most discussed issues (10) ================================= #13441: TestEnUSCollation.test_strxfrm() fails on Solaris http://bugs.python.org/issue13441 13 msgs #10318: "make altinstall" installs many files with incorrect shebangs http://bugs.python.org/issue10318 9 msgs #13429: provide __file__ to extension init function http://bugs.python.org/issue13429 9 msgs #13448: PEP 3155 implementation http://bugs.python.org/issue13448 9 msgs #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 8 msgs #13455: Reorganize tracker docs in the devguide http://bugs.python.org/issue13455 8 msgs #9530: integer undefined behaviors http://bugs.python.org/issue9530 7 msgs #12776: argparse: type conversion function should be called only once http://bugs.python.org/issue12776 7 msgs #12890: cgitb displays <p> tags when executed in text mode http://bugs.python.org/issue12890 7 msgs #13466: new timezones http://bugs.python.org/issue13466 7 msgs Issues closed (29) ================== #7049: decimal.py: Three argument power issues http://bugs.python.org/issue7049 closed by mark.dickinson #8270: Should socket.PF_PACKET be removed, in favor of socket.AF_PACK http://bugs.python.org/issue8270 closed by neologix #9614: _pickle is not entirely 64-bit safe http://bugs.python.org/issue9614 closed by pitrou #12156: test_multiprocessing.test_notify_all() timeout (1 hour) on Fre http://bugs.python.org/issue12156 closed by neologix #12245: Document the meaning of FLT_ROUNDS constants for sys.float_inf http://bugs.python.org/issue12245 closed by mark.dickinson #12934: pysetup doesn’t work for the docutils project http://bugs.python.org/issue12934 closed by eric.araujo #13093: Redundant code in PyUnicode_EncodeDecimal() http://bugs.python.org/issue13093 closed by haypo #13156: _PyGILState_Reinit assumes auto thread state will always exist http://bugs.python.org/issue13156 closed by neologix #13215: multiprocessing Manager.connect() aggressively retries refused http://bugs.python.org/issue13215 closed by neologix #13245: sched.py kwargs addition and default time functions http://bugs.python.org/issue13245 closed by giampaolo.rodola #13313: test_time fails: tzset() do not change timezone http://bugs.python.org/issue13313 closed by haypo #13338: Not all enumerations used in _Py_ANNOTATE_MEMORY_ORDER http://bugs.python.org/issue13338 closed by python-dev #13393: Improve BufferedReader.read1() http://bugs.python.org/issue13393 closed by pitrou #13401: test_argparse fails with root permissions http://bugs.python.org/issue13401 closed by python-dev #13411: Hashable memoryviews http://bugs.python.org/issue13411 closed by pitrou #13415: del os.environ[key] ignores errors http://bugs.python.org/issue13415 closed by python-dev #13417: faster utf-8 decoding http://bugs.python.org/issue13417 closed by pitrou #13425: http.client.HTTPMessage.getallmatchingheaders() always returns http://bugs.python.org/issue13425 closed by stachjankowski #13431: Pass context information into the extension module init functi http://bugs.python.org/issue13431 closed by scoder #13432: Encoding alias "unicode" http://bugs.python.org/issue13432 closed by georg.brandl #13442: Better support for pipe I/O encoding in subprocess http://bugs.python.org/issue13442 closed by ncoghlan #13457: Display module name as string in `ImportError` http://bugs.python.org/issue13457 closed by cool-RR #13458: _ssl memory leak in _get_peer_alt_names http://bugs.python.org/issue13458 closed by pitrou #13459: logger.propagate=True behavior clarification http://bugs.python.org/issue13459 closed by python-dev #13460: urllib methods should demand unicode, instead demand str http://bugs.python.org/issue13460 closed by r.david.murray #13468: Python 2.7.1 SegmentationFaults when given high recursion limi http://bugs.python.org/issue13468 closed by benjamin.peterson #13469: TimedRotatingFileHandler fails to handle intervals of several http://bugs.python.org/issue13469 closed by vinay.sajip #13470: A user may need ... when she has ... http://bugs.python.org/issue13470 closed by pitrou #13480: range exits loop without action when start is higher than end http://bugs.python.org/issue13480 closed by r.david.murray

9 years, 4 months

file.readinto performance regression in Python 3.2 vs. 2.7?

by Eli Bendersky

Hi there, I was doing some experiments with the buffer interface of bytearray today, for the purpose of quickly reading a file's contents into a bytearray which I can then modify. I decided to do some benchmarking and ran into surprising results. Here are the functions I was timing: def justread(): # Just read a file's contents into a string/bytes object f = open(FILENAME, 'rb') s = f.read() def readandcopy(): # Read a file's contents and copy them into a bytearray. # An extra copy is done here. f = open(FILENAME, 'rb') b = bytearray(f.read()) def readinto(): # Read a file's contents directly into a bytearray, # hopefully employing its buffer interface f = open(FILENAME, 'rb') b = bytearray(os.path.getsize(FILENAME)) f.readinto(b) FILENAME is the name of a 3.6MB text file. It is read in binary mode, however, for fullest compatibility between 2.x and 3.x Now, running this under Python 2.7.2 I got these results ($1 just reflects the executable name passed to a bash script I wrote to automate these runs): $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()' 1000 loops, best of 3: 461 usec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readandcopy()' 100 loops, best of 3: 2.81 msec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()' 1000 loops, best of 3: 697 usec per loop Which make sense. The readinto() approach is much faster than copying the read buffer into the bytearray. But with Python 3.2.2 (built from the 3.2 branch today): $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()' 1000 loops, best of 3: 336 usec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readandcopy()' 100 loops, best of 3: 2.62 msec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()' 100 loops, best of 3: 2.69 msec per loop Oops, readinto takes the same time as copying. This is a real shame, because readinto in conjunction with the buffer interface was supposed to avoid the redundant copy. Is there a real performance regression here, is this a well-known issue, or am I just missing something obvious? Eli P.S. The machine is quad-core i7-2820QM, running 64-bit Ubuntu 10.04

9 years, 4 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev November 2011