Mailman 3 November 2005 - Python-Dev

urlparse brokenness
by Paul Jimenez Nov. 28, 2005

Nov. 28, 2005

It is my assertion that urlparse is currently broken. Specifically, I think that urlparse breaks an abstraction boundary with ill effect. In writing a mailclient, I wished to allow my users to specify their imap server as a url, such as 'imap://user:password@host:port/'. Which worked fine. I then thought that the natural extension to support configuration of imapssl would be 'imaps://user:password@host:port/'.... which failed - user:passwrod@host:port got parsed as the *path* of the URL … [View More]instead of the network location. It turns out that urlparse keeps a table of url schemes that 'use netloc'... that is to say, that have a 'user:password@host:port' part to their URL. I think this 'special knowledge' about particular schemes 1) breaks an abstraction boundary by having a function whose charter is to pull apart a particularly-formatted string behave differently based on the meaning of the string instead of the structure of it and 2) fails to be extensible or forward compatible due to hardcoded 'magic' strings - if schemes were somehow 'registerable' as 'netloc using' or not, then this objection might be nullified, but the previous objection would still stand. So I propose that urlsplit, the main offender, be replaced with something that looks like: def urlsplit(url, scheme='', allow_fragments=1, default=('','','','','')): """Parse a URL into 5 components: <scheme>://<netloc>/<path>?<query>#<fragment> Return a 5-tuple: (scheme, netloc, path, query, fragment). Note that we don't break the components up in smaller bits (e.g. netloc is a single string) and we don't expand % escapes.""" key = url, scheme, allow_fragments, default cached = _parse_cache.get(key, None) if cached: return cached if len(_parse_cache) >= MAX_CACHE_SIZE: # avoid runaway growth clear_cache() if "://" in url: uscheme, npqf = url.split("://", 1) else: uscheme = scheme if not uscheme: uscheme = default[0] npqf = url pathidx = npqf.find('/') if pathidx == -1: # not found netloc = npqf path, query, fragment = default[1:4] else: netloc = npqf[:pathidx] pqf = npqf[pathidx:] if '?' in pqf: path, qf = pqf.split('?',1) else: path, qf = pqf, ''.join(default[3:5]) if ('#' in qf) and allow_fragments: query, fragment = qf.split('#',1) else: query, fragment = default[3:5] tuple = (uscheme, netloc, path, query, fragment) _parse_cache[key] = tuple return tuple Note that I'm not sold on the _parse_cache, but I'm assuming it was there for a reason so I'm leaving that functionality as-is. If this isn't the right forum for this discussion, or the right place to submit code, please let me know. Also, please cc: me directly on responses as I'm not subscribed to the firehose that is python-dev. --pj [View Less]

6 7

Re: [Python-Dev] For Python 3k, drop default/implicit hash, and comparison
by Phillip J. Eby Nov. 27, 2005

Nov. 27, 2005

At 01:29 PM 11/6/2005 -0800, Guido van Rossum wrote: >On 11/6/05, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > At 12:58 PM 11/6/2005 -0800, Guido van Rossum wrote: > > >The main way this breaks down is when comparing objects of different > > >types. While most comparisons typically are defined in terms of > > >comparisons on simpler or contained objects, two objects of different > > >types that happen to have the same "key" shouldn't … [View More]

8 10

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser Nov. 27, 2005

Nov. 27, 2005

Patch / Bug Summary ___________________ Patches : 372 open ( -7) / 2980 closed (+12) / 3352 total ( +5) Bugs : 908 open ( -2) / 5395 closed (+11) / 6303 total ( +9) RFE : 200 open ( +0) / 191 closed ( +0) / 391 total ( +0) New / Reopened Patches ______________________ CodeContext - Improved text indentation (2005-11-21) http://python.org/sf/1362975 opened by Tal Einat test_cmd_line expecting English error messages (2005-11-23) CLOSED http://python.org/sf/… [View More]

1 0

ast status, memory leaks, etc
by Neal Norwitz Nov. 26, 2005

Nov. 26, 2005

There's still more clean up work to go, but the current AST is hopefully much closer to the behaviour before it was checked in. There are still a few small memory leaks. After running the test suite, the total references were around 380k (down from over 1,000k). I'm not sure exactly what the total refs were just before AST was checked in, but I believe it was over 340k. So there are likely some more ref leaks that should be investigated. It would be good to know the exact number before AST … [View More]was checked in and now, minus any new tests. There is one memory reference error in test_coding: Invalid read of size 1 at 0x41304E: tok_nextc (tokenizer.c:876) by 0x413874: PyTokenizer_Get (tokenizer.c:1099) by 0x411962: parsetok (parsetok.c:124) by 0x498D1F: PyParser_ASTFromFile (pythonrun.c:1292) by 0x48D79A: load_source_module (import.c:777) by 0x48E90F: load_module (import.c:1665) by 0x48ED61: import_submodule (import.c:2259) by 0x48EF60: load_next (import.c:2079) by 0x48F44D: import_module_ex (import.c:1921) by 0x48F715: PyImport_ImportModuleEx (import.c:1955) by 0x46D090: builtin___import__ (bltinmodule.c:44) Address 0x1863E8F6 is 2 bytes before a block of size 8192 free'd at 0x11B1BA8A: free (vg_replace_malloc.c:235) by 0x4127DB: decoding_fgets (tokenizer.c:167) by 0x412F1F: tok_nextc (tokenizer.c:823) by 0x413874: PyTokenizer_Get (tokenizer.c:1099) by 0x411962: parsetok (parsetok.c:124) by 0x498D1F: PyParser_ASTFromFile (pythonrun.c:1292) by 0x48D79A: load_source_module (import.c:777) by 0x48E90F: load_module (import.c:1665) by 0x48ED61: import_submodule (import.c:2259) by 0x48EF60: load_next (import.c:2079) by 0x48F44D: import_module_ex (import.c:1921) by 0x48F715: PyImport_ImportModuleEx (import.c:1955) by 0x46D090: builtin___import__ (bltinmodule.c:44) I had a patch for this somewhere, I'll try to find it. However, I only fixed this exact error, there was another path that could still be problematic. Most of the memory leaks show up when we are forking in: test_fork1 test_pty test_subprocess Here's what I have so far. There are probably some more. It would be great if someone could try to find and fix these leaks. n -- 16 bytes in 1 blocks are definitely lost in loss record 25 of 599 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4CA102: alias (Python-ast.c:1066) by 0x4CD918: alias_for_import_name (ast.c:2199) by 0x4D0C4E: ast_for_stmt (ast.c:2244) by 0x4D15E3: PyAST_FromNode (ast.c:234) by 0x499078: Py_CompileStringFlags (pythonrun.c:1275) by 0x46D6DF: builtin_compile (bltinmodule.c:457) 56 bytes in 1 blocks are definitely lost in loss record 87 of 599 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C9C92: Name (Python-ast.c:860) by 0x4CE4BA: ast_for_expr (ast.c:1222) by 0x4D1021: ast_for_stmt (ast.c:1900) by 0x4D15E3: PyAST_FromNode (ast.c:234) by 0x499078: Py_CompileStringFlags (pythonrun.c:1275) by 0x46D6DF: builtin_compile (bltinmodule.c:457) 112 bytes in 2 blocks are definitely lost in loss record 198 of 674 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C9C92: Name (Python-ast.c:860) by 0x4CE4BA: ast_for_expr (ast.c:1222) by 0x4D1021: ast_for_stmt (ast.c:1900) by 0x4D16D5: PyAST_FromNode (ast.c:275) by 0x499078: Py_CompileStringFlags (pythonrun.c:1275) by 0x46D6DF: builtin_compile (bltinmodule.c:457) 56 bytes in 1 blocks are definitely lost in loss record 89 of 599 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C9C92: Name (Python-ast.c:860) by 0x4CF3AF: ast_for_arguments (ast.c:650) by 0x4D1BFF: ast_for_funcdef (ast.c:830) by 0x4D15E3: PyAST_FromNode (ast.c:234) by 0x499161: PyRun_StringFlags (pythonrun.c:1275) by 0x47B1B2: PyEval_EvalFrameEx (ceval.c:4221) by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739) by 0x47ABCC: PyEval_EvalFrameEx (ceval.c:3657) by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739) by 0x4C27F8: function_call (funcobject.c:550) 112 bytes in 2 blocks are definitely lost in loss record 189 of 651 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C9C92: Name (Python-ast.c:860) by 0x4CE4BA: ast_for_expr (ast.c:1222) by 0x4D02F7: ast_for_stmt (ast.c:2028) by 0x4D16D5: PyAST_FromNode (ast.c:275) by 0x499078: Py_CompileStringFlags (pythonrun.c:1275) by 0x46D6DF: builtin_compile (bltinmodule.c:457) 56 bytes in 1 blocks are definitely lost in loss record 118 of 651 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C9A41: Num (Python-ast.c:751) by 0x4CE578: ast_for_expr (ast.c:1237) by 0x4CF4ED: ast_for_arguments (ast.c:629) by 0x4D1BFF: ast_for_funcdef (ast.c:830) by 0x4D15E3: PyAST_FromNode (ast.c:234) by 0x499161: PyRun_StringFlags (pythonrun.c:1275) by 0x47B1B2: PyEval_EvalFrameEx (ceval.c:4221) by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739) by 0x47ABCC: PyEval_EvalFrameEx (ceval.c:3657) by 0x47CCCC: PyEval_EvalCodeEx (ceval.c:2739) by 0x4C27F8: function_call (funcobject.c:550) 112 (56 direct, 56 indirect) bytes in 1 blocks are definitely lost in loss record 185 of 651 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x4C97CA: GeneratorExp (Python-ast.c:648) by 0x4CEE4F: ast_for_expr (ast.c:1251) by 0x4D1021: ast_for_stmt (ast.c:1900) by 0x4D16D5: PyAST_FromNode (ast.c:275) by 0x499078: Py_CompileStringFlags (pythonrun.c:1275) by 0x46D6DF: builtin_compile (bltinmodule.c:457) 1024 bytes in 1 blocks are definitely lost in loss record 441 of 651 at 0x11B1AF13: malloc (vg_replace_malloc.c:149) by 0x43F8C4: PyObject_Malloc (obmalloc.c:500) by 0x4B808F: PyNode_AddChild (node.c:95) by 0x4B8386: PyParser_AddToken (parser.c:126) by 0x411944: parsetok (parsetok.c:165) by 0x499062: Py_CompileStringFlags (pythonrun.c:1271) by 0x46D6DF: builtin_compile (bltinmodule.c:457) [View Less]

7 14

Python 3
by Reinhold Birkenfeld Nov. 26, 2005

Nov. 26, 2005

Hi, don't know if this is known here, but it seems we have quite a long way to go: http://kuerzer.de/python3 Reinhold <wink>

1 0

Bug bz2.BZ2File(...).seek(0,2) + patch
by Victor STINNER Nov. 25, 2005

Nov. 25, 2005

Hi, I found a bug in bz2 python module. Example: import bz2 bz2.BZ2File("test.bz2","r") bz2.seek(0,2) assert bz2.tell() != 0 Details and *patch* at: http://sourceforge.net/tracker/index.php?func=detail&aid=1366000&group_id=5… Please CC-me for all your answers. Bye, Victor -- Victor Stinner - student at the UTBM (Belfort, France) http://www.haypocalc.com/wiki/Accueil

1 0

Re: [Python-Dev] Problems with the Python Memory Manager
by Travis Oliphant Nov. 25, 2005

Nov. 25, 2005

Jim Jewett wrote: >Do you have the code that caused problems? > > Yes. I was able to reproduce his trouble and was trying to debug it. >The things I would check first are > >(1) Is he allocating (peak usage) a type (such as integers) that >never gets returned to the free pool, in case you need more of that >same type? > > No, I don't think so. >(2) Is he allocating new _types_, which I think don't get properly > > collected. > > Bingo. … [View More]

8 14

(no subject)
by Frank Nov. 25, 2005

Nov. 25, 2005

hi, test mail list :) 　　　　　　　　致礼！　　　　　　　　Frank 　　　　　　　　fanghao(a)corp.netease.com 　　　　　　　　　　2005-11-25

1 0

registering unicode codecs
by Neal Norwitz Nov. 24, 2005

Nov. 24, 2005

While running regrtest with -R to find reference leaks I found a usage issue. When a codec is registered it is stored in the interpreter state and cannot be removed. Since it is stored as a list, if you repeated add the same search function, you will get duplicates in the list and they can't be removed. This shows up as a reference leak (which it really isn't) in test_unicode with this code modified from test_codecs_errors: import codecs def search_function(encoding): def encode1(input, … [View More]

2 3

Re: Regular expressions
by Duncan Grisby Nov. 24, 2005

Nov. 24, 2005

On Thursday 24 November, Donovan Baarda wrote: > I don't know if this will help, but in my experience compiling re's > often takes longer than matching them... are you sure that it's the > match and not a compile that is taking a long time? Are you using > pre-compiled re's or are you dynamically generating strings and using > them? It's definitely matching time. The res are all pre-compiled. [...] > > A quick look at the code in _sre.c suggests that for most of the time,… [View More]

2 1