Python-Dev December 2015

python-dev@python.org

84 participants
46 discussions

Re: [Python-Dev] Idea: Dictionary references

by Victor Stinner

2015-12-17 11:54 GMT+01:00 Franklin? Lee <leewangzhong+python(a)gmail.com>: > My suggestion should improve *all* function calls which refer to > outside names. Ok, I now think that you should stop hijacking the FAT Python thread. I start a new thread. IMHO your dictionary reference idea is completly unrelated to FAT Python. FAT Python is about guards and specialized bytecode. > Each function keeps an indirect, automagically updated > reference to the current value of the names they use, and will never > need to look things up again.[*] Indirections, nested dictionaries, creation of new "reference" objects... IMHO you are going to have major implementation issues :-/ The design looks *very* complex. I'm quite sure that you are going to make namespace lookups *slower*. It reminds me Python before the with statement and PyPy garbage collector. Many applications relied on the exact behaviour of CPython garbage collector. For example, they expected that a file is written on disk when the last reference to the file object is destroyed. In PyPy, it wasn't (it isn't) true, the write can be delayed. I guess that with all your complex machinery for dict lookups, you will have similar issues of object lifetime. It's unclear to me when and how "reference" objects are destroyed, nor when dict values are destroyed. What happens if a dict key is removed and a reference object is still alive? Is the dict value immediatly destroyed? Does the reference object contain a strong or a weak reference to the value? Victor

5 years, 6 months

Summary of Python tracker Issues

by Python tracker

ACTIVITY SUMMARY (2015-12-11 - 2015-12-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5324 (+27) closed 32341 (+38) total 37665 (+65) Open issues with patches: 2344 Issues opened (45) ================== #7283: test_site failure when .local/lib/pythonX.Y/site-packages hasn http://bugs.python.org/issue7283 reopened by serhiy.storchaka #25591: refactor imaplib tests http://bugs.python.org/issue25591 reopened by maciej.szulik #25843: lambdas on the same line may incorrectly share code objects http://bugs.python.org/issue25843 opened by Tijs Van Oevelen #25844: Pylauncher, launcher.c: Assigning NULL to a pointer instead of http://bugs.python.org/issue25844 opened by Alexander Riccio #25846: Use of Py_ARRAY_LENGTH on pointer in posixmodule.c, win32_wchd http://bugs.python.org/issue25846 opened by Alexander Riccio #25847: CPython not using Visual Studio code analysis! http://bugs.python.org/issue25847 opened by Alexander Riccio #25848: Tkinter tests failed on Windows buildbots http://bugs.python.org/issue25848 opened by serhiy.storchaka #25849: files, opened in unicode (text): write() returns symbols count http://bugs.python.org/issue25849 opened by mmarkk #25850: Building extensions with MSVC 2015 Express fails http://bugs.python.org/issue25850 opened by Sami Salonen #25852: smtplib's SMTP.connect() should store the server name in ._hos http://bugs.python.org/issue25852 opened by labrat #25853: Compile error with pytime.h - struct timespec declared inside http://bugs.python.org/issue25853 opened by jamespharvey20 #25856: The __module__ attribute of non-heap classes is not interned http://bugs.python.org/issue25856 opened by serhiy.storchaka #25858: Structure field size/ofs __str__ wrong with large size fields http://bugs.python.org/issue25858 opened by Charles Machalow #25859: EOFError in test_nntplib.NetworkedNNTPTests.test_starttls() http://bugs.python.org/issue25859 opened by martin.panter #25860: os.fwalk() silently skips remaining directories when error occ http://bugs.python.org/issue25860 opened by Samson Lee #25862: TextIOWrapper assertion failure after read() and SEEK_CUR http://bugs.python.org/issue25862 opened by martin.panter #25863: ISO-2022 seeking forgets state http://bugs.python.org/issue25863 opened by martin.panter #25864: collections.abc.Mapping should include a __reversed__ that rai http://bugs.python.org/issue25864 opened by abarnert #25865: 7.2 Assignment statements documentation is vague and slightly http://bugs.python.org/issue25865 opened by abarnert #25866: Reference 3. Data Model: miscellaneous minor cleanups on the w http://bugs.python.org/issue25866 opened by abarnert #25867: os.stat raises exception when using unicode and no locale is s http://bugs.python.org/issue25867 opened by sejvlond #25868: test_eintr.test_sigwaitinfo() hangs on "AMD64 FreeBSD CURRENT http://bugs.python.org/issue25868 opened by haypo #25869: Faster ElementTree deepcopying http://bugs.python.org/issue25869 opened by serhiy.storchaka #25872: multithreading traceback KeyError when modifying file http://bugs.python.org/issue25872 opened by Michael Allen #25873: Faster ElementTree iterating http://bugs.python.org/issue25873 opened by serhiy.storchaka #25874: Add notice that XP is not supported on Python 3.5+ http://bugs.python.org/issue25874 opened by crwilcox #25876: test_gdb: use subprocess._args_from_interpreter_flags() to tes http://bugs.python.org/issue25876 opened by haypo #25878: CPython on Windows builds with /W3, not /W4 http://bugs.python.org/issue25878 opened by Alexander Riccio #25880: u'..'.encode('idna') → UnicodeError: label empty or too long http://bugs.python.org/issue25880 opened by spaceone #25881: A little faster ElementTree serializing http://bugs.python.org/issue25881 opened by serhiy.storchaka #25882: argparse help error: arguments created by add_mutually_exclusi http://bugs.python.org/issue25882 opened by balage #25883: python 2.7.11 mod_wsgi regression on windows http://bugs.python.org/issue25883 opened by stephan #25884: inspect.getmro() fails when base class lacks __bases__ attribu http://bugs.python.org/issue25884 opened by billyziege #25887: awaiting on coroutine more than once should be an error http://bugs.python.org/issue25887 opened by yselivanov #25888: awaiting on coroutine that is being awaited should be an error http://bugs.python.org/issue25888 opened by yselivanov #25894: unittest subTest failure causes result to be omitted from list http://bugs.python.org/issue25894 opened by zach.ware #25895: urllib.parse.urljoin does not handle WebSocket URLs http://bugs.python.org/issue25895 opened by imrehg #25896: array.array accepting byte-order codes in format strings http://bugs.python.org/issue25896 opened by Zoinkity.. #25898: Check for subsequence inside a sequence http://bugs.python.org/issue25898 opened by seblin #25900: unittest ignores the first ctrl-c when it shouldn't http://bugs.python.org/issue25900 opened by mgedmin #25901: make test crash http://bugs.python.org/issue25901 opened by fluyy #25902: Fixed various refcount issues in ElementTree iteration http://bugs.python.org/issue25902 opened by serhiy.storchaka #25905: IDLE fails to display the README file http://bugs.python.org/issue25905 opened by serhiy.storchaka #25906: Worker stall in multiprocessing.Pool http://bugs.python.org/issue25906 opened by chroxvi #25907: Documentation i18n: Added trans tags in sphinx templates http://bugs.python.org/issue25907 opened by sizeof Most recent 15 issues with no replies (15) ========================================== #25907: Documentation i18n: Added trans tags in sphinx templates http://bugs.python.org/issue25907 #25906: Worker stall in multiprocessing.Pool http://bugs.python.org/issue25906 #25905: IDLE fails to display the README file http://bugs.python.org/issue25905 #25902: Fixed various refcount issues in ElementTree iteration http://bugs.python.org/issue25902 #25901: make test crash http://bugs.python.org/issue25901 #25900: unittest ignores the first ctrl-c when it shouldn't http://bugs.python.org/issue25900 #25896: array.array accepting byte-order codes in format strings http://bugs.python.org/issue25896 #25876: test_gdb: use subprocess._args_from_interpreter_flags() to tes http://bugs.python.org/issue25876 #25872: multithreading traceback KeyError when modifying file http://bugs.python.org/issue25872 #25866: Reference 3. Data Model: miscellaneous minor cleanups on the w http://bugs.python.org/issue25866 #25863: ISO-2022 seeking forgets state http://bugs.python.org/issue25863 #25862: TextIOWrapper assertion failure after read() and SEEK_CUR http://bugs.python.org/issue25862 #25860: os.fwalk() silently skips remaining directories when error occ http://bugs.python.org/issue25860 #25834: getpass falls back when sys.stdin is changed http://bugs.python.org/issue25834 #25830: _TypeAlias: Discrepancy between docstring and behavior http://bugs.python.org/issue25830 Most recent 15 issues waiting for review (15) ============================================= #25907: Documentation i18n: Added trans tags in sphinx templates http://bugs.python.org/issue25907 #25902: Fixed various refcount issues in ElementTree iteration http://bugs.python.org/issue25902 #25900: unittest ignores the first ctrl-c when it shouldn't http://bugs.python.org/issue25900 #25898: Check for subsequence inside a sequence http://bugs.python.org/issue25898 #25895: urllib.parse.urljoin does not handle WebSocket URLs http://bugs.python.org/issue25895 #25888: awaiting on coroutine that is being awaited should be an error http://bugs.python.org/issue25888 #25887: awaiting on coroutine more than once should be an error http://bugs.python.org/issue25887 #25881: A little faster ElementTree serializing http://bugs.python.org/issue25881 #25878: CPython on Windows builds with /W3, not /W4 http://bugs.python.org/issue25878 #25876: test_gdb: use subprocess._args_from_interpreter_flags() to tes http://bugs.python.org/issue25876 #25874: Add notice that XP is not supported on Python 3.5+ http://bugs.python.org/issue25874 #25873: Faster ElementTree iterating http://bugs.python.org/issue25873 #25869: Faster ElementTree deepcopying http://bugs.python.org/issue25869 #25860: os.fwalk() silently skips remaining directories when error occ http://bugs.python.org/issue25860 #25859: EOFError in test_nntplib.NetworkedNNTPTests.test_starttls() http://bugs.python.org/issue25859 Top 10 most discussed issues (10) ================================= #25843: lambdas on the same line may incorrectly share code objects http://bugs.python.org/issue25843 34 msgs #19475: Add timespec optional flag to datetime isoformat() to choose t http://bugs.python.org/issue19475 18 msgs #25847: CPython not using Visual Studio code analysis! http://bugs.python.org/issue25847 15 msgs #25849: files, opened in unicode (text): write() returns symbols count http://bugs.python.org/issue25849 14 msgs #25864: collections.abc.Mapping should include a __reversed__ that rai http://bugs.python.org/issue25864 13 msgs #1753718: base64 "legacy" functions violate RFC 3548 http://bugs.python.org/issue1753718 11 msgs #25846: Use of Py_ARRAY_LENGTH on pointer in posixmodule.c, win32_wchd http://bugs.python.org/issue25846 8 msgs #25880: u'..'.encode('idna') → UnicodeError: label empty or too long http://bugs.python.org/issue25880 8 msgs #25878: CPython on Windows builds with /W3, not /W4 http://bugs.python.org/issue25878 7 msgs #25823: Speed-up oparg decoding on little-endian machines http://bugs.python.org/issue25823 6 msgs Issues closed (40) ================== #6478: time.tzset does not reset _strptime's locale time cache http://bugs.python.org/issue6478 closed by serhiy.storchaka #19771: runpy should check ImportError.name before wrapping it http://bugs.python.org/issue19771 closed by martin.panter #20837: Ambiguity words in base64 documentation http://bugs.python.org/issue20837 closed by martin.panter #20954: Bug in subprocess._args_from_interpreter_flags causes MemoryEr http://bugs.python.org/issue20954 closed by gregory.p.smith #21436: Consider leaving importlib.abc.Loader.load_module() http://bugs.python.org/issue21436 closed by berker.peksag #23788: test_urllib2_localnet.test_bad_address fails: OSError not rais http://bugs.python.org/issue23788 closed by martin.panter #25272: asyncio tests are getting noisy http://bugs.python.org/issue25272 closed by yselivanov #25495: binascii documentation incorrect http://bugs.python.org/issue25495 closed by r.david.murray #25580: async and await missing from token list http://bugs.python.org/issue25580 closed by yselivanov #25608: ascynio readexactly() should raise ValueError if passed length http://bugs.python.org/issue25608 closed by yselivanov #25610: Add typing.Awaitable http://bugs.python.org/issue25610 closed by gvanrossum #25683: __context__ for yields inside except clause http://bugs.python.org/issue25683 closed by yselivanov #25696: "make -j9 install" fails because bininstall target requires th http://bugs.python.org/issue25696 closed by haypo #25755: Test test_property failed if run twice http://bugs.python.org/issue25755 closed by berker.peksag #25773: Deprecate deleting with PyObject_SetAttr, PyObject_SetAttrStri http://bugs.python.org/issue25773 closed by serhiy.storchaka #25809: "Invalid" tests on locales http://bugs.python.org/issue25809 closed by martin.panter #25838: Lib/httplib.py: Resend http request on server close connection http://bugs.python.org/issue25838 closed by r.david.murray #25842: Installer does not set permissions correctly? http://bugs.python.org/issue25842 closed by zach.ware #25845: _ctypes\cfield.c identical subexpressions in Z_set http://bugs.python.org/issue25845 closed by martin.panter #25851: installing 3.5 on windows server 2003 x86 R2 Standard Edition http://bugs.python.org/issue25851 closed by eryksun #25854: rest in _interpolate_some is a list not str http://bugs.python.org/issue25854 closed by rhettinger #25855: str.title() http://bugs.python.org/issue25855 closed by ezio.melotti #25857: csv: unexpected result http://bugs.python.org/issue25857 closed by r.david.murray #25861: Can't use Pickle. AttributeError: 'module' object has no attri http://bugs.python.org/issue25861 closed by serhiy.storchaka #25870: textwrap is very slow on long words without spaces http://bugs.python.org/issue25870 closed by r.david.murray #25871: textwrap.dedent doesn't find common substring when spaces and http://bugs.python.org/issue25871 closed by Chris Tozer #25875: PYODBC talk to Oracle under Windows 10. http://bugs.python.org/issue25875 closed by r.david.murray #25877: python av docs has broken links http://bugs.python.org/issue25877 closed by r.david.murray #25879: Code objects from same line can compare equal http://bugs.python.org/issue25879 closed by KirkMcDonald #25885: ast Str type does not annotate the string type when it parses http://bugs.python.org/issue25885 closed by brett.cannon #25886: ast module is combining string literals that are concatenated http://bugs.python.org/issue25886 closed by SilentGhost #25889: Find_BOM accepts a char*, but is passed an unsigned char*; and http://bugs.python.org/issue25889 closed by serhiy.storchaka #25890: PyObject *po in _listdir_windows_no_opendir is initialized but http://bugs.python.org/issue25890 closed by serhiy.storchaka #25891: Stray variable meth_idx in enable_symlink http://bugs.python.org/issue25891 closed by serhiy.storchaka #25892: PyObject *exc in encode_code_page_strict is initialized but no http://bugs.python.org/issue25892 closed by serhiy.storchaka #25893: Second variable DWORD reqdSize in getpythonregpath is initiali http://bugs.python.org/issue25893 closed by serhiy.storchaka #25897: Python 3.5.1 and Active Tcl/Tk 8.6.4.1 http://bugs.python.org/issue25897 closed by ned.deily #25899: Unnecessary non-ASCII characters in standard library http://bugs.python.org/issue25899 closed by serhiy.storchaka #25903: SUGGESTION: Optimize code in PYO http://bugs.python.org/issue25903 closed by r.david.murray #25904: SUGGESTION: New Datatypes http://bugs.python.org/issue25904 closed by mark.dickinson

5 years, 6 months

pypi simple index

by Carlos Barera

Hi, I'm using install_requires in setup.py to specify a specific package my project is dependant on. When running python setup.py install, apparently the simple index is used as an older package is taken from pypi. While in https://pypi.python.org/pypi, there's a newer package. When installing directly using pip, the latest package is installed successfully. I noticed that the new package is only available as a wheel and older versions of setup tools won't install wheels for install_requires. However, upgrading setuptools didn't help. Several questions: 1. What's the difference between the pypi simple index and the general pypi index? 2. Why is setup.py defaulting to the simple index? 3. How can I make the setup.py triggered install use the main pypi index instead of simple Thanks!

5 years, 6 months

pypi simple index

by Carlos Barera

Hi, I'm using install_requires in setup.py to specify a specific package my project is dependant on. When running python setup.py install, apparently the simple index is used as an older package is taken from pypi. While in https://pypi.python.org/pypi, there's a newer package. When installing directly using pip, the latest package is installed successfully. Several questions: 1. What's the difference between the pypi simple index and the general pypi index? 2. Why is setup.py defaulting to the simple index? 3. How can I make the setup.py triggered install use the main pypi index instead of simple Thanks! Carlos

5 years, 6 months

Third milestone of FAT Python

by Franklin? Lee

On Sat, Dec 04, 2015 at 7:49 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > Versionned dictionary > ===================== > > In the previous milestone of FAT Python, the versionned dictionary was a > new type inherited from the builtin dict type which added a __version__ > read-only (global "version" of dictionary, incremented at each change), > a getversion(key) method (version of a one key) and it added support for > weak references. I was thinking (as an alternative to versioning dicts) about a dictionary which would be able to return name/value pairs, which would also be internally used by the dictionary. This would be way less sensitive to irrelevant changes in the scope dictionary, but cost an extra pointer to each key. Here's how it would work: pair = scope.item(name) scope[name] = newval assert pair.value is newval assert pair.key is name assert pair is scope.item(name) # Alternatively, to only create pair objects when `item` is called, have `==` compare the underlying pair. del scope[name] assert pair.key is None # name-dicts can't have `None` keys assert pair.value is None # Alternatively, pair.value is scope.NULL This dict will allow one to hold references to its entries (with the caller promising not to change them, enforced by exceptions). You won't have to keep looking up keys (unless the name is deleted), and functions are allowed to change. For inlining, you can detect whether the function has been redefined by testing the saved pair.value against the saved function, and go into the slow path if needed (or recompile the inlining). I am not sure whether deleting from the dict and then readding the same key should reuse the pair container. I think the only potential issue for the Python version is memory use. There aren't going to be THAT many names being deleted, right? So I say that deleted things in the scope dict should not be removed from the inner dict. I predict that this will simplify a lot of other things, especially when deleting and readding the same name: if you save a pair, and it becomes invalid, you don't have to do another lookup to make sure that it's REALLY gone. If memory is a real concern, deleted pairs can be weakrefed (and saved in a second dict?) until they are reused. This way, pairs which aren't saved by something outside will be removed. For implementation, a Python implementation of the idea has probably already been done. Here are some details: - set: Internally store d._d[k] = k,v. - get: Reject k if d._d[k].key is None. (Names must be strings.) - del: Set d._d[k].key = None and .val = d.NULL to invalidate this entry. For the CPython version, CPython's dict already stores its entries as PyDictKeyEntry (hash, *key, *value), but those entries can move around on resizing. Two possible implementations: 1. Fork dict to store {hash, *kv_pair}. 2. Use an inner dict (like in the Python suggestion) where values are kv_pair. Write the indirection code in C, because scope dicts must be fast. For exposing a pair to Python code, here are two possibilities: 1. Make them Python objects in the first place. 2. Keep a second hash table in lockstep with the first (so that you can do a lookup to find the index in the first, and then use that same index with the second). In this table, store pair objects that have been created. (They can be weakrefed, as before. Unless it's impossible to weakref something you're returning?) This will save memory for pairs that aren't ever exposed. If compact dictionaries are implemented, the second hash table will be a second array instead. To use this kind of scopedict, functions would have to store a list of used names, which is memory overhead. But for what you're doing, some overhead will be necessary anyway.

5 years, 6 months

async/await behavior on multiple calls

by Roy Williams

Howdy, I'm experimenting with async/await in Python 3, and one very surprising behavior has been what happens when calling `await` twice on an Awaitable. In C#, Hack/HHVM, and the new async/await spec in Ecmascript 7. In Python, calling `await` multiple times results in all future results getting back `None`. Here's a small example program: async def echo_hi(): result = '' echo_proc = await asyncio.create_subprocess_exec( 'echo', 'hello', 'world', stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL) result = await echo_proc.stdout.read() await echo_proc.wait() return result async def await_twice(awaitable): print('first time is {}'.format(await awaitable)) print('second time is {}'.format(await awaitable)) loop = asyncio.get_event_loop() loop.run_until_complete(await_twice(echo_hi())) This makes writing composable APIs using async/await in Python very difficult since anything that takes an `awaitable` has to know that it wasn't already awaited. Also, since the behavior is radically different than in the other programming languages implementing async/await it makes adopting Python's flavor of async/await difficult for folks coming from a language where it's already implemented. In C#/Hack/JS calls to `await` return a Task/AwaitableHandle/Promise that can be awaited multiple times and either returns the result or throws any thrown exceptions. It doesn't appear that the Awaitable class in Python has a `result` or `exception` field but `asyncio.Future` does. Would it make sense to shift from having `await` functions return a ` *Future-like`* return object to returning a Future? Thanks, Roy

5 years, 6 months

Python semantic: Is it ok to replace not x == y with x != y? (no)

by Victor Stinner

Hi, I implemented more constant folding optimizations in my FAT Python project, but it looks like I made a subtle change in the Python semantic. Replacing "not x == y" with "x != y" changes the behaviour of Python. For example, this optimization breaks test_unittest because unittest.mock._Call implements __eq__() but not __ne__(). Is it expected that "not x.__eq__(y)" can be different than "x.__ne__(y)"? Is it part of the Python semantic? IMHO it's a bug in the unittest.mock module, but it's "acceptable" because "it just works" :-) So FAT Python must not replace "not x == y" with "x != y" to not break the code. Should Python emit a warning when __eq__() is implemented but not __ne__()? Should Python be modified to call "not __eq__()" when __ne__() is not implemented? For me, it can be an annoying and sublte bug, hard to track. Victor

5 years, 6 months

"python.exe is not a valid Win32 app"

by Alexei Belenki

Installed python 3.5 (from https://www.python.org/downloads/) on Windows XPsp3/32 On starting >>python.exe got the text above in the Windows message box. Any suggestions?Thanks.AB

5 years, 6 months

Python for android - successfully cross-compiled without patches

by Vitaly Murashev

A lot of talks and patches around how to cross-compile python for andriod ... Dear python-dev@, I just want to say thanks to all of you for the high quality cross-platform code. Using alternative Android NDK named CrystaX (home page - https://www.crystax.net ) which provides high quality posix support in comparison with google's one we managed to cross-compile python 2.7 and 3.5 completely without any patches applied.

5 years, 6 months

Re: [Python-Dev] Python semantic: Is it ok to replace not x == y with x != y? (no)

by Adam Bartoš

Hello, the comparisons >=, <=, >, < cannot be optimized this way. Not every order is a total order. For example, sets a = {1, 2} and b = {2, 3} are incomparable (in the sense that both a >= b and a <= b is False), and it is no pathology. Regards, Adam Bartoš

5 years, 6 months

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev December 2015