Mailman 3 October 2009 - Python-ideas

One obvious way to do interning [Was: Retrieve an arbitrary element from a set without removing it]
by Alexander Belopolsky 26 Oct '09

26 Oct '09

Changing the subject to reflect branched discussion and forwarding to python-ideas where it probably belongs. On Mon, Oct 26, 2009 at 12:02 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Alexander Belopolsky wrote: > >> Here is an alternative idea on how storing interned objects in a set >> can be supported. Currently set.add method returns None and has no >> effect when set already has an object equal to the one being added. I >> propose to consider changing that behavior to make set.add return the >> added object or the set member that is equal to the object being >> added. It is unlikely that many programs rely on the return value >> being None (with doctests being a probable exception), so adding this >> feature is unlikely to cause much grief. > > I had exactly the same idea, but did not post because it violates the > general rule that mutators return None. Is there such a rule? What about set/dict pop? > On the other hand, the returned > value here would not be the mutated collection, so no chaining is possible. I assume you refer to chaining as in s.add(1).add(2) which would be enabled if s.add(..) returned s. My proposal would enable a different type of "chaining" which I would find useful, but ready to hear objections: v = compute_value() s.add(v) # use v can, with my proposal, be rewritten as v = s.add(compute_value()) with an added benefit that v that is used is the "canonical" value. > And 'add' is clearly intended to change something. > Is this an argument for or against the proposal? > On the other hand, frozensets do not have an add method. However, PySet_Add "works with frozenset instances (like PyTuple_SetItem() it can be used to fill-in the values of brand new frozensets before they are exposed to other code). " <http://docs.python.org/3.1/c-api/set.html#PySet_Add> I will experiment with changing PySet_Add to see how much it would break and whether it will be helpful in implementing set-based interning of python strings.

1 0

add a global tick register to the interpreter?
by Sturla Molden 23 Oct '09

23 Oct '09

(Please forgive me if this belongs on python-dev.) When measuring the performance of a Python or Python program, it is often required to keep precise count of interpreter tics and periodic checks. There is no way to to this reliably, except modify ceval.c and rebuild Python. I think facilities for performance measures should be available. It should not be neccesary edit ceval.c and rebuild Python just to measure how Python itself or a Python program performs. I am therefore proposing this change to ceval.c: New global declarations: /* tick and check counters */ volatile PY_LONG_LONG _Py_Interpreter_Ticks = 0; volatile PY_LONG_LONG _Py_Interpreter_Checks = 0; The big infinite loop in PyEval_EvalFrameEx becomes for (;;Py_Interpreter_Ticks++) { #ifdef WITH_TSC if (inst1 == 0) { with periodic checks like this: if (--_Py_Ticker < 0) { _Py_Interpreter_Checks++; The reason for this change is that _Py_Ticker cannot be used for performance measures, as the variable can be set to zero during check intervals themselves or asynchronously from callbacks/signals. The variables _Py_Interpreter_Checks and Py_Interpreter_Ticks would be available to extension developers using the declaration: extern const volatile PY_LONG_LONG _Py_Interpreter_Ticks; extern const volatile PY_LONG_LONG _Py_Interpreter_Checks; #define PyEval_GetInterpreterTicks (_Py_Interpreter_Ticks) /* or an inline function */ #define PyEval_GetInterpreterChecks (_Py_Interpreter_Checks) (A 'const volatile' is a C variable that cannot be written to, but can suddenly be changed from the environment.) From the Python side we would have: sys.tickcounter # or sys.getinterpreterticks? sys.checkcounter The counters must be 64 bit because of the speed of Python. On my computer I measure close to 3 million interpreter ticks per second. That would exhaust a (signed) 32 bit counter in 11 minutes. A signed 64 bit counter could go on for 24 thousand years. (This is not covered by Guido's moratorium as it is an implementation and library issue.) Regards, Sturla Molden

2 3

Proposal : Python Trusted Computing API
by Abhiram Kasina 23 Oct '09

23 Oct '09

Hi all Thank you very much for the response. I have tried to answer all the issues as much as I can. > Hm... Given that most infections these days are JavaScript based and > run in the browser, how does this provide any protection? I'm > presuming you're going to say that it doesn't but that there are other > use cases where it *does* provide protection; but most likely those > use cases are only relevant for Windows (since that's what most > attackers attack anyway). Guido, I don't know the exact security model and I am currently going through it. But it has a mechanism in which none of the passwords or anything can be ever given out. So, phishing and pharming could be fought better. I agree that most of the cases of viruses are in Windows, but there are many cases in the past in which they have affected linux machines as well. So, if security is being made better, it should be used. > Since this intefaces with the hardware, doesn't it require some kind > of cooperation from the Linux kernel? And wouldn't it be better if > Python was never allowed access to any of the protected resources in > the first place? There would be a wrapper on top of the drivers running at the kernel level. Lets say there is a Python application which uses some sort of password authentication. If the password is present somehow in the memory, it could be stolen. But if its present inside a completely separate hardware, which doesn't allow it to be retrieved, I think that will be better. > Where/what is Umit? (Google gives several meanings but it's unclear > which you might mean.) By Umit, I meant Umit Project [1]. Its the GUI for nmap and its being built completely in Python. > You'd first have to tell us more about the security model. What is a > "secure application" and what does it protect against? And how? Yes, I will take some time to understand the model and present you the details of the model well when I get some time. > I personally have grave concerns about this technology, independent of > who is advancing it- just reading the links you provide makes it clear > that its primary purpose is to restrict the degree of control a user can > exercise over their own data, programs, and machine. Is it any wonder > that the same people who find DRM abhorrent find the technologies > used to advance it equally distasteful? Geremy, I have read the TPM specifications to some extent and the only thing enabling DRM here is the presence of a system-specific key. And it is heavily impractical to encrypt every music file with that key, when the vendor is selling it. It could be done even now, they want to do it. > If you were suggesting adding a crypto API to python, I'd be all > for it- but you're suggesting adding the ability to have Python > software vendors remotely cripple the code on your machine. > I just can't get behind that, and while you're sure to hear wildly > divergent opinions on this board, I suspect that mine will not > be an uncommon sentiment. I appreciate the opinion and I am not suggesting adding a crypto API. Thanks Abhiram Kasina [1] http://www.umitproject.org/

2 1

a new lambda syntax
by starwing 23 Oct '09

23 Oct '09

In ruby, we can add a lambda function into a function call. but it's a little ugly: [1,2,3,4].each { |x| print x } but, it really convenient. current lambda only support single line syntax. it's not very good. maybe we can add a new syntax: call: primary "(" argument-list ")" [ "lambda" parameter-list ":" suite ] the postfix lambda will translate to a single "function" object, and pass to the &arg parameter of function. so the function call will become func(*arg1, **arg2, &arg3) for example, we can write this: def print_list(*lists, &func = lambda x: true): for item in lists: map(a) lambda x: if func(x): print x call with: print_list([1,2,3], [1,2]) lambda x: if x % 2: return True else return False

12 35

Re: [Python-ideas] Proposal: Moratorium on Python language changes
by Larry Bugbee 23 Oct '09

23 Oct '09

On Wed, Oct 21, 2009 at 12:42 PM, Guido van Rossum wrote: > I propose a moratorium on language changes. This would be a > period of several years during which no changes to Python's > grammar or language semantics will be accepted. I agree with a moratorium. I cannot imagine it doing any harm. I'd even go so far as to say new language syntax and semantics should only come in only at top level version numbers like 2.0, 3.0, 4.0, etc. ...not at 3.2, 3.3, 3.4, etc. And at this later date, I submit language changes should not come in unless there is a compelling reason. That said, two areas where language support would be welcome: - the optional declaration of data types - parallel processing FWIW, I do a lot of crypto and for now have chosen to remain at 2.5.x. For me, byte strings are all important, and a few quick forays into Python 3 and its default Unicode suggested I should proceed very carefully. My experiments with Python 2.6 left me uncomfortable as well. There I ran into a couple of standard libraries that had the same API but their implementation/behavior was different. (No, that was many months ago and I do not remember which. Sorry.) In time I will move to Py3.

6 10

nonlocal functions
by Bruce Leban 22 Oct '09

22 Oct '09

On Tue, Oct 20, 2009 at 4:04 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > The difference lies in the fact that in C, object references are > non-local by default - you have to declare them explicitly in the > current scope to make them local. > > Accordingly, I find the idea of a new function-like construct where all > non-argument variable references are nonlocal by default to be a > potentially interesting one. > Hmm. That's an interesting idea. I don't know enough about internals. It might be possible to do this with a decorator def foo(): a = 1 @allnonlocal def bar(): a = 2 bar() return a foo() => 2 However, even if possible, that would make everything nonlocal with no way to pick which ones. Here's another idea: def foo(): a = 1 b = 3 nonlocal def bar(): local b a = 2 b = 4 return (a,b) foo() => (2, 3) Prepending nonlocal to a function definition is equivalent to applying nonlocal to every variable referenced in that function unless the variable is declared local. For backward compatibility, the local statement would only be only recognized inside nonlocal functions. (Thus if you have a function that uses 'local' as a variable it would not need to be changed unless you decided to stick nonlocal in front of it.) Alternative syntax: def bar() nonlocal: --- Bruce http://www.vroospeak.com

9 15

A standard location for Python configuration files.
by Tarek Ziadé 22 Oct '09

22 Oct '09

Hello, As discussed in issue #7175 and on irc, I would like to propose the addition of a new function called 'getuserconfig' in site.py. This function will return the path to a directory that will contain configuration files for Python. The name I am suggesting for the path of the folder is: ~/.python with an alternative location for win32 : ~/python This win32 alternative is proposed to avoid the fact that it's not possible in some flavors of windows to manually create a folder that starts with a dot when using file graphical browser. That's disturbing for end users. This will be used by Distutils to gather the configuration files it uses right now ([py]distutils.cfg and pypirc) and gently deprecate the old location (~) I am bringing this discussion to python-ideas to make sure those are the best choices, and to propose ~/[.]python as the new standard location for any user configuration file Python currently uses or might use in the future. Regards, Tarek

8 14

Re: [Python-ideas] Proposal: Moratorium on Python language changes
by Raymond Hettinger 22 Oct '09

22 Oct '09

[Guido van Rossum] > Note, the moratorium would only cover the language itself plus > built-in functions, not the standard library. That makes sense. There may be a few areas that still have some rough edges where you may want to allow changes if needed (tweaks to the nested with-statement syntax, bytes/text interaction, star-args unpacking, or string formatting). These areas probably have not been exercised much and there may still be problems that need to be ironed-out. I don't have anything specific in mind. Am just thinking that those features aren't yet mature. Also, do you know if there any plans afoot to do something with function annotations? > I also want to exclude > details of the CPython implementation, including the C API from being > completely frozen -- for example, if someone came up with (otherwise > acceptable) changes to get rid of the GIL I wouldn't object. FWIW, I have pending updates for the set/frozenset implementation (no api change). Also, I'm hoping the recently submitted C implementation for decimal gets accepted (as performance issues seem to be slowing broader use of the module). It looks like substantial work has already been done. > But the moratorium would clearly apply to proposals for anonymous > blocks, "yield from" (PEP 380), changes to decorator syntax, and the > like. (I'm sure it won't stop *discussion* of those proposals, and > that's not the purpose of the moratorium; but at least it will stop > worries elsewhere that such proposals might actually be *accepted* any > time soon.) Are you rejecting PEP 380? Raymond

5 5

Re: [Python-ideas] Proposal: Moratorium on Python language changes
by Larry Bugbee 22 Oct '09

22 Oct '09

On Wed, Oct 21, 2009 at 12:42 PM, Guido van Rossum wrote: > I propose a moratorium on language changes. This would be a > period of several years during which no changes to Python's > grammar or language semantics will be accepted. I think a moratorium would be Goodness. I cannot imagine it doing any harm. I'd even go so far as to say new language syntax and semantics should only come in only at top level version numbers like 2.0, 3.0, 4.0, etc. ...not at 3.2, 3.3, 3.4, etc. And at this later date, I submit language changes should not come in unless there is a compelling reason. Two areas where language support would be welcome: - the optional declaration of data types - parallel processing FWIW, I do a lot of crypto and for now have chosen to remain at 2.5.x. For me, byte strings are all important, and a few quick forays into Python 3 and its default Unicode suggested I should proceed very carefully. My experiments with Python 2.6 left me uncomfortable as well. There I ran into a couple of standard libraries that had the same API but their implementation/behavior was different. (No, that was many months ago and I do not remember which. Sorry.) In time I will move to Py3.

1 0

Re: [Python-ideas] [Python-Dev] GIL behaviour under Windows
by Sturla Molden 22 Oct '09

22 Oct '09

Antoine Pitrou skrev: > Kristján sent me a patch which I applied and is supposed to fix this. > Anyway, thanks for the numbers. The GIL does seem to fare a bit better (zero > latency with the Pi calculation in the background) than under Linux, although it > may be caused by the limited resolution of time.time() under Windows. > > My critisism of the GIL on python-ideas was partly motivated by this: http://blip.tv/file/2232410 However, David Beazley is not talking about Windows. Since the GIL is apparently not a mutex on Windows, it could behave differently. So I wrote a small script that contructs a GIL battle, and record how often a check-interval results in a thread-switch or not. For monitoring check intervals, I used a small C extension to read _Py_Ticker from ceval.c. It is not declared static so I could easily hack into it. With two threads and a check interval og 100, only 61 of 100000 check intervals failed to produce a thread-switch in the interpreter. I'd call that rather fair. :-) And in case someone asks, the nthreads=1 case is just for verification. S.M. D:\>test.py check interval = 1 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=57809, missed=42191 nthreads=3, swiched=91535, missed=8465 nthreads=4, swiched=99751, missed=249 nthreads=5, swiched=95839, missed=4161 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 10 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99858, missed=142 nthreads=3, swiched=99992, missed=8 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 100 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99939, missed=61 nthreads=3, swiched=100000, missed=0 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0 D:\>test.py check interval = 1000 nthreads=1, swiched=0, missed=100000 nthreads=2, swiched=99999, missed=1 nthreads=3, swiched=100000, missed=0 nthreads=4, swiched=100000, missed=0 nthreads=5, swiched=100000, missed=0 nthreads=6, swiched=100000, missed=0

1 1