From zathras at thwackety.com Fri Oct 1 00:36:02 2004 From: zathras at thwackety.com (Michael Sparks) Date: Fri Oct 1 00:08:06 2004 Subject: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration In-Reply-To: <5.1.1.6.0.20040930153803.02bc0140@mail.telecommunity.com> Message-ID: <Pine.LNX.4.44.0409302241070.14797-100000@pingu.thwackety.com> On Thu, 30 Sep 2004, Phillip J. Eby wrote: ... > A mechanism to pass values or exceptions into generators [ Possibly somewhat off topic, and apologies if it is, and I'm positive someone's done something similar before, but I think it's relevant to the discussion in hand -- largely because the above use case *doesn't* require changes to python... ] A minimal(ish) example of a technique for doing this I presented last week at a streaming workshop looked like the following. (amongst other stuff...) Create a decorator that wraps the generator inside a class, derived from a supplied base class. (default to object) import copy def wrapgenerator(bases=object, **attrs): def decorate(func): class statefulgenerator(bases): __doc__ = func.__doc__ def __init__(self,*args): super(statefulgenerator, self).__init__(*args) self.func=func(self,*args) for k in attrs.keys(): self.__dict__[k] = copy.deepcopy(attrs[k]) self.next=self.__iter__().next def __iter__(self): return iter(self.func) return statefulgenerator return decorate Create a class to handle the behaviour you wish to use to communicate with the generator from outside: class component(object): def __init__(self, *args): # Default queues self.queues = {"inbox":[],"control":[],"outbox":[],"signal":[]} def send(self, box, object): self.queues[box].append(object) def dataReady(self,box): return len(self.queues[box])>0 def recv(self, box): # NB. Exceptions aren't caught X=self.queues[box][0] del self.queues[box][0] return X Then just use something like this: @wrapgenerator(component) def forwarderNay(self): "Simple data forwarding generator" while 1: if self.dataReady("inbox"): self.send("outbox",self.recv("inbox")+"Nay") elif self.dataReady("control"): if self.recv("control") == "shutdown": break yield 1 self.send("signal","shutdown") yield 0 In conjuction with a simplistic scheduler, and linkage functions this allows you to have something similar to CSP. I've come to the conclusion recently that the fact you can't* yield across multiple levels is actually beneficial because it encourages you to use many small components. Used standalone you can do this though: f=forwarderNay() for word in ["hello", "world", "test"]: f.send("inbox", word) f.next() print f.recv("outbox"), print Which of course outputs: * helloNay worldNay testNay For small scale things this amount of cruft is a bit of a pain. We're using *essentially* this approach to build network servers and it seems a rather satisfying way of doing so TBH. (not exactly this approach so if this looks odd, that's why - I'm not allowed to release the source for precisely what we're doing :-( but the above I was on the slides I *was* able to talk about... :-) Hope that's of some use... Best Regards, Michael. From pje at telecommunity.com Fri Oct 1 01:17:43 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Oct 1 01:17:44 2004 Subject: [Python-Dev] PEP 334 - Simple Coroutines via SuspendIteration In-Reply-To: <Pine.LNX.4.44.0409302241070.14797-100000@pingu.thwackety.c om> References: <5.1.1.6.0.20040930153803.02bc0140@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040930191017.03a5fc00@mail.telecommunity.com> At 11:36 PM 9/30/04 +0100, Michael Sparks wrote: >On Thu, 30 Sep 2004, Phillip J. Eby wrote: >... > > A mechanism to pass values or exceptions into generators > >[ Possibly somewhat off topic, and apologies if it is, and I'm positive > someone's done something similar before, but I think it's relevant to > the discussion in hand -- largely because the above use case *doesn't* > require changes to python... ] I know it doesn't; peak.events does this now, but in order to have a decent programmer interface, the implementation involves a fair amount of magic. Similarly, PEP 334 doesn't call for anything you can't do with a bit of work and magic. I was suggesting, however, that a true "simple co-routine" (similar to a generator but with bidirectional communication of values and exceptions) would be a valuable addition to the language, in the area of simplifying async programming in e.g. Twisted and peak.events. To put it another way: Slap the current title of PEP 334 onto the body of PEP 288, and change its syntax so you have a way to pass values and exceptions *in* to a suspended "coroutine" (a new and different animal from a generator), and I'm sold. From ncoghlan at email.com Fri Oct 1 03:09:02 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 03:09:08 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <5.1.1.6.0.20040930111348.038beb60@mail.telecommunity.com> References: <20040930135718.GA208@panix.com> <1096539066.415bdbbaaed43@mail.iinet.net.au> <20040930135718.GA208@panix.com> <5.1.1.6.0.20040930111348.038beb60@mail.telecommunity.com> Message-ID: <1096592942.415cae2e6cada@mail.iinet.net.au> Quoting "Phillip J. Eby" <pje@telecommunity.com>: > Using the C equivalent of 'imp.find_module()' should cover all these cases, > and any new forms of PY_SOURCE or PY_COMPILED that come up in future. That's essentially what the patch does. PyRun_SimpleModuleFlags uses _PyImport_FindModule to obtain the absolute file location, then uses PyRun_SimpleFileExFlags to actually run the script. _PyImport_FindModule is a trivial wrapper around import.c's find_module() (which, I believe, is what imp.find_module() invokes) When multiple versions exist, the patch operates on what find_module() returns (which ends up using the preference order '.py', '.pyw', '.py[co]') Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Fri Oct 1 03:31:30 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 03:31:36 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096554430.20270.23.camel@geddy.wooz.org> References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096554430.20270.23.camel@geddy.wooz.org> Message-ID: <1096594290.415cb372f2f41@mail.iinet.net.au> Quoting Barry Warsaw <barry@python.org>: > > At 08:21 PM 9/30/04 +1000, Nick Coghlan wrote: > > >However, another possibility occurred to me: > > > > > >try: > > > # Do stuff > > >except sys.special_exceptions: > > > raise > > >except: > > > # Deal with all the mundane stuff > > +0, except that I'd rather see it put in the exceptions module and given > a name in builtins. Hmm, I forgot about the existence of the exceptions module. I agree that makes a more sensible location than sys. As for it being a builtin, I have no objections to that. I'll come up with two patches, though. One to create the tuple in exceptions, and one to give it a name in builtins (since the latter would presumably be more controversial, like any new builtin). My current list of exceptions for inclusion is KeyboardInterrupt, MemoryError, SystemExit & StopIteration. The inclusion of StopIteration is what makes me prefer 'special_exceptions' as the name of the tuple, rather than 'critical_exceptions'. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From bac at OCF.Berkeley.EDU Fri Oct 1 04:44:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 04:44:33 2004 Subject: [Python-Dev] Re: Dangerous exceptions (was Re: Another test_compilermystery) In-Reply-To: <1f7befae040907201426c33006@mail.gmail.com> References: <002d01c48083$9a89a6c0$5229c797@oemcomputer> <20040816112916.GA19969@vicky.ecs.soton.ac.uk> <1f7befae04090422024afaee58@mail.gmail.com> <e8bf7a5304090518425dc3ebec@mail.gmail.com> <chh4cr$qre$1@sea.gmane.org> <e8bf7a5304090705092ee4daa7@mail.gmail.com> <413DBB19.40602@zope.com> <1094566431.8341.25.camel@geddy.wooz.org> <413DCFBD.7010306@theopalgroup.com> <1094572868.8342.43.camel@geddy.wooz.org> <1f7befae040907201426c33006@mail.gmail.com> Message-ID: <415CC48C.7050209@ocf.berkeley.edu> Tim Peters wrote: > After a bit more thought (and it's hard to measure how little), I'd > like to see "bare except" deprecated. That doesn't mean no way to > catch all exceptions, it means being explicit about intent. Only a > few of the bare excepts I've seen in my Python life did what was > actually intended, and there's something off in the design when the > easiest thing to say usually does a wrong thing. > Giving it same amount of thought as Tim, I like this idea as well. I don't think the burden of having to always specify an exception to catch is that much work and would definitely be more explicit. Plus it is not hard to teach to newbies; just tell them to catch the exception all of the "safe" exceptions inherit from. Is it PEP time? We have now had the idea of reorganizing the exception hierarchy come up in this thread and in early August (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception for the thread on reorganizing for the purpose of using new-style classes and the idea of having all raisable objects inherit from a specific base class). It seems like a serious enough of an idea that it will happen for Python 3000 and thus should be discussed; never too early to start. But maybe it at least deserves to be mentioned in PEP 3000 (Guido?)? > Oh well. We should elect a benevolent dictator for Python! > Here, here! =) -Brett From bac at OCF.Berkeley.EDU Fri Oct 1 05:11:41 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 05:11:45 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-01 through 2004-09-15 [draft] Message-ID: <415CCAED.9050203@ocf.berkeley.edu> ===================== Summary Announcements ===================== Python 2.4a3 has been released. Go to http://www.python.org/2.4/ and give it twirl. Sorry for this summary being so short, but school has started back up again so I am in the middle of suddenly having to switch back into homework mode after spending the summer just having a 9:00-17:00 job. And since it is a new school year I am going to abuse this space and say that anyone in San Luis Obispo, including students, should join the `SLO Meetup`_ coming up on October 14. .. _SLO Meetup: http://python.meetup.com/95/ ========= Summaries ========= -------------------- Movement in PEP Land -------------------- `PEP 334`_ (Simple Coroutines via SuspendIteration) came into existence. `PEP 328`_ (Relative Imports) got some discussion on postponing making imports absolute instead of the relative/absolute semantics they have now. As it stands it looks like the changeover might get pushed off. `PEP 292`_ (Simpler String Substitutions) seems to finally be done and locked down. `PEP 335`_ (Overloadable Boolean Operators) came into existence. .. _PEP 334: http://www.python.org/peps/pep-0334.html .. _PEP 328: http://www.python.org/peps/pep-0328.html .. _PEP 292: http://www.python.org/peps/pep-0292.html .. _PEP 335: http://www.python.org/peps/pep-0335.html Contributing threads: - `PEP 334 - Simple Coroutines via SuspendIteration <>`__ - `PEP 328 - Relative Imports <>`__ - `Re: Alternative Implementation for PEP 292:Simple String Substitutions <>`__ - `ANN: PEP 335: Overloadable Boolean Operators <>`__ ------------------------------------------------------ __str__, __unicode__, and how to have them play nicely ------------------------------------------------------ Did you know that __str__ methods are allowed to return Unicode objects? Well, it turns out they can, but that str() (which calls PyObject_Str()) automatically tries to convert the value returned by __str__ into ASCII. Basically __str__ shouldn't return Unicode if you can help it and you should use __unicode__ instead and reserve __str__ to return str objects only. Contributing threads: - `unicode and __str__ <>`__ ---------------------- Backporting C APIs bad ---------------------- Somebody (*cough* Guido *cough*) asked if the datetime C API could be backported to 2.3 . The argument was that the only person who would probably use it is the person who asked for it, the author of cx_Oracle. Well, pretty much everyone spoke up against this. The argument went that adding an API would just be bad since there would suddenly be a point in the 2.3 releases where backwards compatibility was broken. People brought up the point of 2.2 where in 2.2.1 booleans were added and how that has caused compatibility headaches for some people. In the end the API was not backported. Contributing threads: - `Re: [Python-checkins] python/dist/src/Modules threadmodule.c, 2.56, 2.56.8.1 <>`__ ---------------------------------------------------------------------- Got to love race conditions thanks to the filesystem and external apps ---------------------------------------------------------------------- Tim Peters found a race condition one can have on Windows if you have an app that uses the low-level hooks into the filesystem. If you create a file, delete it, and then try to delete the directory the directory deletion will fail since the file is not deleted yet. What can happen is an indexing program can still be indexing the file before the filesystem is allowed to delete it and thus the directory is not truly empty when the directory deletion is executed. Fun stuff. Contributing threads: - `Coernic Desktop Search versus shutil.rmtree <>`__ ------------------------------- Python 2.4a3 is out the door!!! ------------------------------- Go to http://www.python.org/2.4/ , download it (using the bz2 version if possible so as to save on bandwidth), and run it against your code, run the test suite, put it on your head and sell yourself to an art gallery, etc. Contributing threads: - `RELEASED Python 2.4, alpha 3 <>`__ ---------------------------- Cleaning the Exception House ---------------------------- The idea of reorganizing the exceptions hierarchy came up again (see http://www.python.org/dev/summary/2004-08-01_2004-08-15.html#an-exception-is-an-exception-unless-it-doesn-t-inherit-from-exception for a previous discussion on this). This time, the idea of separating the hierarchy into exceptions one would like to catch with a bare 'except' and those that you wouldn't was brought up. The idea is that some exceptions, such as MemoryError, one does not want to catch in a blanket statement usually. Chances of recovering from that kind of exception is low and should only be caught if you know what you are doing. So tweaking the exception hierarchy so that exceptions that were not near-catastrophic could inherit from an exception class that people could catch so as to allow the proper exceptions to propagate to the top-level without issue. Tim Peters even went as far as to suggest deprecating bare 'except' statements. This would force people to be explicit about what they want to catch, whether it be all "safe" exceptions or *all* exceptions. As it stands now no officially decision has been made for Python 3000 since that is about the only place this could happen. Contributing threads: - `Dangerous exceptions <>`__ --------------------------------------------------------------- Making decorators not look like decorators to the outside world --------------------------------------------------------------- Raymond Hettinger pointed out that a decorator does not, by default, look like the function that it is fiddling with (if that is the intent). Since most decorators will most likely be a wrapper function some things need to be set in the wrapper in order not to mask things in the wrapped function (doc string, argument parameters, etc.). So Raymond pointed out some things one can do. This also led to the suggestion of having a common name used to store a reference back to the wrapped function. There was also the mention that a decorator-oriented module in the stdlib will probably materialize in Python 2.5 . For now, though, stick recipes either in the Python Cookbook or in the Python wiki at http://www.python.org/cgi-bin/moinmoin/PythonDecoratorLibrary . Contributing threads: - `decorator support <>`__ =============== Skipped Threads =============== - random.py still broken wrt. urandom - random.py fixage - Re: [Python-checkins] python/dist/src/Lib/test test_compiler.py, 1.5, 1.6 test_decimal.py, 1.13, 1.14 - assert failure on obmalloc - Re: [Python-checkins] python/dist/src/Modules socketmodule.c, 1.304, 1.305 - Install-on-first-use vs. optional extensions - Console vs. GUI applications - Adding status code constants to httplib - PEP 292: method names - PEP 265 - Sorting dicts by value - httplib is not v6 compatible, is this going to be fixed? - OT: Unicode history - --with-tsc compile fails - tempfile.TemporaryFile on Windows NT - PyExc_UnicodeDecodeError - urllib.urlopen() vs IDNs, percent-encoded hosts, ':' - tabs in httplib.py and test_httplib.py There is now a vimrc file in the Misc directory that sets things up to follow PEPs 7 & 8 - strawman decision: @decorator won't change - unicode inconsistency? From ncoghlan at email.com Fri Oct 1 06:47:15 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 06:47:22 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <1096539066.415bdbbaaed43@mail.iinet.net.au> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> Message-ID: <1096606035.415ce15346b75@mail.iinet.net.au> Quoting Nick Coghlan <ncoghlan@email.com>: > - run the located script as __main__ (note that containing packages are NOT > imported first - it's as if the relevant module was executed directly from > the > command line) I've realised that this means that the '-m' option doesn't understand packages that modify __path__ in their __init__.py scripts. What do people think of semantics which say "python -m some.package.module" means that "some.package" gets imported before "module" gets executed as "__main__"? The advantages are that __path__ will be interpreted correctly and package initialisation code will be invoked before the module is executed. It seems slightly odd to be importing things before the script starts executing, but these semantics seem to be more in line with the behaviour of the rest of Python's import machinery. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From pje at telecommunity.com Fri Oct 1 06:57:45 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Oct 1 06:57:48 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <1096606035.415ce15346b75@mail.iinet.net.au> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> <1096539066.415bdbbaaed43@mail.iinet.net.au> Message-ID: <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> At 02:47 PM 10/1/04 +1000, Nick Coghlan wrote: >Quoting Nick Coghlan <ncoghlan@email.com>: > > - run the located script as __main__ (note that containing packages are NOT > > imported first - it's as if the relevant module was executed directly from > > the > > command line) > >I've realised that this means that the '-m' option doesn't understand packages >that modify __path__ in their __init__.py scripts. > >What do people think of semantics which say "python -m some.package.module" >means that "some.package" gets imported before "module" gets executed as >"__main__"? > >The advantages are that __path__ will be interpreted correctly and package >initialisation code will be invoked before the module is executed. > >It seems slightly odd to be importing things before the script starts >executing, >but these semantics seem to be more in line with the behaviour of the rest of >Python's import machinery. Although it would be nice if the new functionality supported existing scripts, I'd almost rather see the semantics of '-m foo.bar' be the same as '-c "from foo.bar import __main__; __main__()"', since the latter's semantics are much more well-defined. Alternatively, one could use '-m foo.bar.__main__' or '-m timeit.main' or '-m pydoc.cli' or '-m unittest.main', and thus be explicit about exactly what will be run. In either case, though, I think import semantics are easier to explain/understand than __name__=='__main__' semantics, especially in situations where the "script" may be re-imported by other code it imports (e.g. the unittest module). From ncoghlan at email.com Fri Oct 1 08:03:02 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:03:08 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096594290.415cb372f2f41@mail.iinet.net.au> References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096554430.20270.23.camel@geddy.wooz.org> <1096594290.415cb372f2f41@mail.iinet.net.au> Message-ID: <1096610582.415cf316c0dfe@mail.iinet.net.au> A candidate implementation is up at SF. Patch # 1038256. It uses the name 'special_exceptions' and includes SystemExit, MemoryError, KeyboardInterrupt and StopIteration (adding/removing exceptions from the list is easy, as is changing the name). One version of the patch creates 'exceptions.special_exceptions' only, and the other version also creates a builtin. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From python at rcn.com Fri Oct 1 08:28:17 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 1 08:29:47 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <1096610582.415cf316c0dfe@mail.iinet.net.au> Message-ID: <003201c4a77f$da68c140$e841fea9@oemcomputer> > A candidate implementation is up at SF. Patch # 1038256. > > It uses the name 'special_exceptions' and includes SystemExit, > MemoryError, > KeyboardInterrupt and StopIteration (adding/removing exceptions from the > list is > easy, as is changing the name). > > One version of the patch creates 'exceptions.special_exceptions' only, and > the > other version also creates a builtin. Since we're trying to catch anything *not* special, is the intended usage something like this: try: func() except special_exceptions: raise except: altfunc() # handle non-special exceptions Raymond From ncoghlan at email.com Fri Oct 1 08:36:27 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:36:33 2004 Subject: [Python-Dev] Running a module as a script In-Reply-To: <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> References: <1096539066.415bdbbaaed43@mail.iinet.net.au> <1096539066.415bdbbaaed43@mail.iinet.net.au> <5.1.1.6.0.20041001005128.02b54750@mail.telecommunity.com> Message-ID: <1096612587.415cfaeb7ee5d@mail.iinet.net.au> Quoting "Phillip J. Eby" <pje@telecommunity.com>: > Although it would be nice if the new functionality supported existing > scripts, I'd almost rather see the semantics of '-m foo.bar' be the same as > '-c "from foo.bar import __main__; __main__()"', since the latter's > semantics are much more well-defined. > > Alternatively, one could use '-m foo.bar.__main__' or '-m timeit.main' or > '-m pydoc.cli' or '-m unittest.main', and thus be explicit about exactly > what will be run. > > In either case, though, I think import semantics are easier to > explain/understand than __name__=='__main__' semantics, especially in > situations where the "script" may be re-imported by other code it imports > (e.g. the unittest module). Jim suggested something similar. The thing is, such idioms are already quite easy to handle using '-c' (see above for an example ;). What isn't possible is invoking the "if __name__ == '__main__':" idiom without knowing the location of a module in the file system. About the closest we get for Unix'y platforms is to run 'python `python -c "import inspect; import x; print inspect.getsourcefile(x)"`'. (if the quotes aren't clear, those are backticks around the python -c invocation). Anyway, I think -m really only makes sense if it gives us the power to invoke a module as "__main__". Otherwise we might as well stick to using -c. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Fri Oct 1 08:51:07 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Fri Oct 1 08:51:14 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <003201c4a77f$da68c140$e841fea9@oemcomputer> References: <003201c4a77f$da68c140$e841fea9@oemcomputer> Message-ID: <1096613467.415cfe5bcf784@mail.iinet.net.au> Quoting Raymond Hettinger <python@rcn.com>: > Since we're trying to catch anything *not* special, is the intended > usage something like this: > > try: > func() > except special_exceptions: > raise > except: > altfunc() # handle non-special exceptions Yep. It's essentially a workaround for the fact that we can't do anything too drastic to the exception heirarchy without serious backwards compatibility problems, but have some 'critical' exceptions that most code shouldn't be suppressing. At the moment, 'except:' is bad, and in most cases 'except Exception:' isn't any better (which surprised me - I didn't realise this until Tim brought it up recently). Tim suggested 'except StandardError:' which is an improvement but still not quite right (SystemExit and StopIteration make it through then, but MemoryError and KeyboardInterrupt still get eaten). I'm dreaming of the day when I can hit 'Ctrl-C' on any Python script and actually have the darn thing stop without hitting it 10 more times ;) Cheers, Nick. -- Nick Coghlan Brisbane, Australia From mwh at python.net Fri Oct 1 13:43:56 2004 From: mwh at python.net (Michael Hudson) Date: Fri Oct 1 13:43:58 2004 Subject: [Python-Dev] Proposing a sys.special_exceptions tuple In-Reply-To: <5.1.1.6.0.20040930153454.02bc04c0@mail.telecommunity.com> (Phillip J. Eby's message of "Thu, 30 Sep 2004 15:37:17 -0400") References: <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <1096539707.415bde3ba1425@mail.iinet.net.au> <5.1.1.6.0.20040930101453.0244e8f0@mail.telecommunity.com> <5.1.1.6.0.20040930153454.02bc04c0@mail.telecommunity.com> Message-ID: <2mwtyapssj.fsf@starship.python.net> "Phillip J. Eby" <pje@telecommunity.com> writes: > At 12:52 PM 9/30/04 -0300, Lalo Martins wrote: >>On Thu, Sep 30, 2004 at 10:19:22AM -0400, Phillip J. Eby wrote: >> > >> > Also, maybe in 2.5 we could begin warning about bare excepts that aren't >> > preceded by non-bare exceptions. >> >>try: >> foo() >>except: >> print_or_log_exception_in_a_way_that_is_meaningful() >> raise >> >>doesn't seem to be incorrect to me. For example, if the program >>is a daemon, I want the exception logged somewhere so that I can >>see it later, because I won't be watching stderr. > > 1. If the exception raised is a MemoryError, your daemon is in trouble. Not necessarily. Typing 'range(sys.maxint)' into the interactive interpreter gives a fairly harmless MemoryError (the only exception PyRepl doesn't catch is SystemExit). > 2. I said *warn*, and it'd be easy to suppress the warning using > except Exception:', if that's what you really mean Well, apart from the fact that this doesn't catch, uh, exceptions that don't derive from Exception? Until/unless that's enforced (something that gets thrashed around every other month) there's still a place for 'bare except:'. Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods From goodger at python.org Fri Oct 1 16:24:39 2004 From: goodger at python.org (David Goodger) Date: Fri Oct 1 16:26:05 2004 Subject: [Python-Dev] ConfigParser patches Message-ID: <415D68A7.3080702@python.org> I have patches up on SF for two ConfigParser bugs, which I'd like to check in ASAP. Any objections? http://www.python.org/sf/1017864: Case sensitivity bug in ConfigParser ====================================================================== This is a simple bug relating to default values passed in as dictionaries, illustrated here: >>> import ConfigParser >>> cp = ConfigParser.ConfigParser({"foo": "Bar"}) >>> cp.get("DEFAULT", "Foo") 'Bar' >>> cp = ConfigParser.ConfigParser({"FOO": "Bar"}) # <-- note CAPS >>> cp.get("DEFAULT", "FOO") # same for "foo", "Foo", etc. Traceback (most recent call last): ... ConfigParser.NoOptionError: No option 'foo' in section: 'DEFAULT' The patch applies ConfigParser.optionxform to keys of defaults when supplied, to be consistent with the handling of keys of config file entries and runtime-set options. A test patch is also included. http://www.python.org/sf/997050: ConfigParser behavior change ============================================================= This may be more controversial. In Python 2.3 and earlier, ConfigParser implicitly allowed non-string values to be set. http://python.org/sf/810843 asked for clarification, and in rev 1.65 an explicit type check was added that raises TypeError. The problem is that this breaks Docutils code, and I suspect it will break other code as well, which we won't hear about until after Python 2.4 final is released. Setting non-string values worked just fine for application-internal use, as long as interpolation is turned off (RawConfigParser is used or ConfigParser with "raw" parameter set) and config files aren't written. A new SafeConfigParser class was added in Python 2.3: New applications should prefer this version if they don't need to be compatible with older versions of Python. My solution is to add the new string-only restriction to the SafeConfigParser class, leave existing ConfigParser & RawConfigParser behavior alone, and document the conditions under which non-string values work. Code, doc, and test patches included. What say you? -- David Goodger <http://python.net/~goodger> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/d193ff4f/signature.pgp From python at rcn.com Fri Oct 1 22:07:44 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 1 22:09:03 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415D68A7.3080702@python.org> Message-ID: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> [David Goodger] > In Python 2.3 and earlier, > ConfigParser implicitly allowed non-string values to be set. > http://python.org/sf/810843 asked for clarification, and in rev 1.65 > an explicit type check was added that raises TypeError. Instead of an explicit type check, it may be better to wrap the interpolation step in a try/except. For non-string objects, raise a TypeError with an informative error message. This would meet the OP's need for an informative error message while leaving the module backwards compatible for internal uses of ConfigParser. Ideally, the docs should discourage further non-string uses and advise that ConfigParser will be string only for Py3.0. Raymond From fdrake at acm.org Fri Oct 1 22:37:57 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 1 22:38:06 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> Message-ID: <200410011637.57629.fdrake@acm.org> On Friday 01 October 2004 04:07 pm, Raymond Hettinger wrote: > Ideally, the docs should discourage further non-string uses and advise > that ConfigParser will be string only for Py3.0. Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and exposes a really poor model even for .ini-style configuration files. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From bac at OCF.Berkeley.EDU Fri Oct 1 23:45:39 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 23:48:14 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-01 through 2004-09-15 [draft] In-Reply-To: <415CCAED.9050203@ocf.berkeley.edu> References: <415CCAED.9050203@ocf.berkeley.edu> Message-ID: <415DD003.1040000@ocf.berkeley.edu> I forgot to say that I plan to send out this summary some time between Saturday evening and Monday night. -Brett From bac at OCF.Berkeley.EDU Fri Oct 1 23:52:09 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 1 23:53:12 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <200410011637.57629.fdrake@acm.org> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> Message-ID: <415DD189.7000900@ocf.berkeley.edu> Fred L. Drake, Jr. wrote: > On Friday 01 October 2004 04:07 pm, Raymond Hettinger wrote: > > Ideally, the docs should discourage further non-string uses and advise > > that ConfigParser will be string only for Py3.0. > > Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and > exposes a really poor model even for .ini-style configuration files. > Ditto from me. Personally I would want something that used XML (like property lists on OS X), but I realize people still want a config file style that is easy to modify in a text editor, so I don't see the .ini style going away. But we could stand to come up with a uniform interface that both an XML and .ini config file parsers could use for consistency and thus support both styles. Wasn't there talk for a while of doing an shootout of config file packages like we did for optparse? -Brett From gvanrossum at gmail.com Sat Oct 2 00:09:02 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 00:09:05 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DD189.7000900@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> Message-ID: <ca471dc20410011509509e83a8@mail.gmail.com> > > > Ideally, the docs should discourage further non-string uses and advise > > > that ConfigParser will be string only for Py3.0. > > > > Yikes! Can't we just toss it for Py3K? This module just hasn't held up, and > > exposes a really poor model even for .ini-style configuration files. Hear hear. > Ditto from me. Personally I would want something that used XML (like property > lists on OS X), but I realize people still want a config file style that is > easy to modify in a text editor, so I don't see the .ini style going away. But > we could stand to come up with a uniform interface that both an XML and .ini > config file parsers could use for consistency and thus support both styles. Well, for me personally, .ini style config files still win over XML every day. And I now have significant experience with both here at ESI. What sucks (relatively) is the specific way that ConfigParser provides access to .ini files; I always end up writing a wrapping layer around it. > Wasn't there talk for a while of doing an shootout of config file packages like > we did for optparse? There are two different axes here: the config file format and the API for accessing it. I'm not sure that attempting to provide a single API for both XML and .ini files will work; the API should reflect the structure of the file (to some extent) and that's just too different. But I may be wrong; I do see some similarities: What I have come to like, both for dealing with XML and with .ini files, is something that lets you map the configuration values (or whatever it is that you're parsing, really) to Python attributes. This is really some kind of simplified DTD support, and there are different ways to go about it; but the net result is that you end up writing e.g. options.client.max_retries (which is an int with a default value) rather than options.getint("client", "max-retries"). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From goodger at python.org Sat Oct 2 00:27:59 2004 From: goodger at python.org (David Goodger) Date: Sat Oct 2 00:28:11 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> Message-ID: <415DD9EF.7000606@python.org> [Raymond Hettinger] > Instead of an explicit type check, it may be better to wrap the > interpolation step in a try/except. Problem with this approach: the interpolation is done at .get() time, so any exception raised would be far from the culprit, .set(). > Ideally, the docs should discourage further non-string uses My doc patch makes clear that non-string uses don't work with interpolation & file output. > and advise that ConfigParser will be string only for Py3.0. Are there any other Python 3.0 advisories in the docs now? Seeing as how there's no definite schedule or plan for Python 3.0, indeed it's "a hypothetical future release" (PEP 3000), that may not be a precedent we want to set. In any case, I don't know that ConfigParser *should* be declared string-only at all. Docutils uses it only to read config files; the non-string values come from interpreting those values. It's quite convenient to plug the interpreted values back into the ConfigParser dicts and use those. -- David Goodger <http://python.net/~goodger> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/cc8fc5fc/signature.pgp From tim.peters at gmail.com Sat Oct 2 00:29:49 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sat Oct 2 00:29:53 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc20410011509509e83a8@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <1f7befae0410011529587160e2@mail.gmail.com> [Guido] ... > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). LOL. I wrote a Config wrapper for SpamBayes like that. Then I stopped paying attention, and next thing I knew all the option-accessing code had been changed to look like SB's current prob = options["Classifier", "unknown_word_prob"] instead. But even that's better than options.getint("client", "max-retries")! Encoding the return type at every access point is Plain Wrong. From bac at OCF.Berkeley.EDU Sat Oct 2 00:30:44 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 00:30:55 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc20410011509509e83a8@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <415DDA94.4060505@ocf.berkeley.edu> Guido van Rossum wrote: [SNIP] >>Ditto from me. Personally I would want something that used XML (like property >>lists on OS X), but I realize people still want a config file style that is >>easy to modify in a text editor, so I don't see the .ini style going away. But >>we could stand to come up with a uniform interface that both an XML and .ini >>config file parsers could use for consistency and thus support both styles. > > > Well, for me personally, .ini style config files still win over XML > every day. And I now have significant experience with both here at > ESI. > OK. Do realize that plists are basically .ini style just expressed in XML:: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Section</key> <dict> <key>key</key> <string>value</string> <key>key2</key> <string>value2</string> </dict> </dict> </plist> I am not thinking of anything fancy or beyond something like this; .ini files expressed in XML. Just thinking that XML might be nice since all of those poor souls who don't use Python have easy access to an XML parser but not necessarily a .ini file parser. > What sucks (relatively) is the specific way that ConfigParser provides > access to .ini files; I always end up writing a wrapping layer around > it. > > >>Wasn't there talk for a while of doing an shootout of config file packages like >>we did for optparse? > > > There are two different axes here: the config file format and the API > for accessing it. > > I'm not sure that attempting to provide a single API for both XML and > .ini files will work; the API should reflect the structure of the file > (to some extent) and that's just too different. But I may be wrong; I > do see some similarities: > For free-wheeling XML, yes, I would agree, but if we restrict the DTD/schema/spec to be fairly similar to .ini files then I don't think the structure of the files will be that drastic. And as long as people use the API to create/edit/save the files then we can enforce universal rules that makes sure that the difference is really just in the structure of the output. > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). > That is what I am thinking. You have sections and then key/value pairs within the sections in both the XML and the .ini format. We then just agree on how case-sensitivity, naming, etc. works within the file formats and then provide a common-place interface much like the one you lay out above. Is this worth working on now or wait until Py3k? It probably seems like I am playing the PEP drum (not a fan of the harp =) a lot lately in regards to stuff that is going to not be an issue until Py3k, but the longer we have to work on something the better chance we have of having something that everyone likes. Plus for things like modules we can provide it for people now as an external module for experimentation and thus be a little willy-nilly with the API initially until we have it nailed and either integrate into the stdlib beofre Py3k or just wait until then. -Brett From janssen at parc.com Sat Oct 2 00:31:14 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Oct 2 00:31:38 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Your message of "Fri, 01 Oct 2004 15:09:02 PDT." <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <04Oct1.153122pdt."58617"@synergy1.parc.xerox.com> > Well, for me personally, .ini style config files still win over XML > every day. And I now have significant experience with both here at > ESI. > > What sucks (relatively) is the specific way that ConfigParser provides > access to .ini files; I always end up writing a wrapping layer around > it. Me too. I like the .ini file format (mail header formats will always win over XML for config files, I suspect), but had to wrap it for my latest project. Mainly for two reasons: to describe a hierarchy of config files to process when called, and to provide getint, getbool, and getfloat methods. Bill From goodger at python.org Sat Oct 2 00:29:22 2004 From: goodger at python.org (David Goodger) Date: Sat Oct 2 00:32:37 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc20410011509509e83a8@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <415DDA42.3010904@python.org> [Guido van Rossum] > What sucks (relatively) is the specific way that ConfigParser > provides access to .ini files; I always end up writing a wrapping > layer around it. Unfortunately ConfigParser discourages such wrappers by hiding too much of its implementation as _private attributes (not __class_privates, so they *are* accessible, but even single-underscore _privacy screams "don't touch this -- subject to change without notice"). Back on topic though, I'd like to see +/-'s on the two patches. The first one (http://www.python.org/sf/1017864) is IMO a no-brainer. The second (http://www.python.org/sf/997050) is what I'm really interested in: do we maintain the existing API and functionality, or is it OK to break existing code by *removing* functionality? -- David Goodger <http://python.net/~goodger> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/c85cd3b4/signature.pgp From barry at python.org Sat Oct 2 00:38:48 2004 From: barry at python.org (Barry Warsaw) Date: Sat Oct 2 00:38:55 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc20410011509509e83a8@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <1096670328.8358.46.camel@geddy.wooz.org> On Fri, 2004-10-01 at 18:09, Guido van Rossum wrote: > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). An enthusiastic +1. I'll just note that I often find that generally useful when dealing with XML files, not just those that represent configuration values. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041001/4330b42b/attachment.pgp From gvanrossum at gmail.com Sat Oct 2 00:43:35 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 00:43:42 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DDA94.4060505@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <ca471dc2041001154338f9c617@mail.gmail.com> > > Well, for me personally, .ini style config files still win over XML > > every day. And I now have significant experience with both here at > > ESI. > > OK. Do realize that plists are basically .ini style just expressed in XML:: > > <?xml version="1.0" encoding="UTF-8"?> > <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" > "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> > <plist version="1.0"> > <dict> > <key>Section</key> > <dict> > <key>key</key> > <string>value</string> > <key>key2</key> > <string>value2</string> > </dict> > </dict> > </plist> > > I am not thinking of anything fancy or beyond something like this; .ini files > expressed in XML. Just thinking that XML might be nice since all of those poor > souls who don't use Python have easy access to an XML parser but not > necessarily a .ini file parser. This reveals IMO a big mistake in thinking about configuration files. The most important user of a config file is not the programmer who has to get data out of it; the most important user is the user who has to edit the config file. The outrageous verbosity of XML makes the above example a complete usability liability. Now, if you're talking about config files that represent options that the user edits in a convenient application-specific options dialog, that's a different story; I think XML is well-suited for that; but I'm talking about the classic configuration file pattern where you use your favorite flat-file text editor to edit the options file. In that situation, using XML is insane. > Is this worth working on now or wait until Py3k? I see no advantage in waiting until Py3K; this is not a language issue and there is no problem with having several library modules (as long as it's clear which one is deprecated). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 01:18:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 01:18:36 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001154338f9c617@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <415DE5C4.9080107@ocf.berkeley.edu> Guido van Rossum wrote: [SNIP] > > This reveals IMO a big mistake in thinking about configuration files. > The most important user of a config file is not the programmer who has > to get data out of it; the most important user is the user who has to > edit the config file. The outrageous verbosity of XML makes the above > example a complete usability liability. > > Now, if you're talking about config files that represent options that > the user edits in a convenient application-specific options dialog, > that's a different story; I think XML is well-suited for that; but I'm > talking about the classic configuration file pattern where you use > your favorite flat-file text editor to edit the options file. In that > situation, using XML is insane. > I am thinking of the app-specific options dialog situation for XML and I do think that using XML for the classic style of editing the flat file by hand is insane. My thinking on the XML idea is that people do write apps where the whole thing is behind a GUI and thus XML makes good sense. So why not make there lives a little easier by giving them a basic API? But it is not a make-or-break thing for me so I am willing to let it go if it is deemed not worth the effort. > >>Is this worth working on now or wait until Py3k? > > > I see no advantage in waiting until Py3K; this is not a language > issue and there is no problem with having several library modules (as > long as it's clear which one is deprecated). > OK, so how do people want to proceeed with this? PEP? Shootout? -Brett From gvanrossum at gmail.com Sat Oct 2 01:22:57 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 01:23:06 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DE5C4.9080107@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415DE5C4.9080107@ocf.berkeley.edu> Message-ID: <ca471dc2041001162247f825d8@mail.gmail.com> > My thinking on the XML idea is that people do write apps where the whole thing > is behind a GUI and thus XML makes good sense. So why not make there lives a > little easier by giving them a basic API? People who want to use XML already have all the rope they need. I would be in for adding something like my simplified XML handling class to the library, but I first have to lobby my employer to open-source it (not a problem, I just have to time the request right so it won't be in 2.4). Anyway, this needn't and shouldn't be specific to the problem of storing option values. > OK, so how do people want to proceeed with this? PEP? Shootout? I'm all for working code in this case, so let's do a shootout. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 01:31:39 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 01:31:44 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001162247f825d8@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415DE5C4.9080107@ocf.berkeley.edu> <ca471dc2041001162247f825d8@mail.gmail.com> Message-ID: <415DE8DB.4060908@ocf.berkeley.edu> Guido van Rossum wrote: >>My thinking on the XML idea is that people do write apps where the whole thing >>is behind a GUI and thus XML makes good sense. So why not make there lives a >>little easier by giving them a basic API? > > > People who want to use XML already have all the rope they need. I > would be in for adding something like my simplified XML handling class > to the library, but I first have to lobby my employer to open-source > it (not a problem, I just have to time the request right so it won't > be in 2.4). Anyway, this needn't and shouldn't be specific to the > problem of storing option values. > OK. > >>OK, so how do people want to proceeed with this? PEP? Shootout? > > > I'm all for working code in this case, so let's do a shootout. > OK. Should we do an announcement to python-announce and c.l.py saying we are looking for a new API for .ini file editing/accessing? -Brett From fperez528 at yahoo.com Sat Oct 2 01:34:20 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Sat Oct 2 01:34:25 2004 Subject: [Python-Dev] Re: ConfigParser patches References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> Message-ID: <cjkpht$va4$1@sea.gmane.org> Guido van Rossum wrote: > What I have come to like, both for dealing with XML and with .ini > files, is something that lets you map the configuration values (or > whatever it is that you're parsing, really) to Python attributes. This > is really some kind of simplified DTD support, and there are different > ways to go about it; but the net result is that you end up writing > e.g. options.client.max_retries (which is an int with a default value) > rather than options.getint("client", "max-retries"). That's exactly what I wrote for ipython's option handler, with a fairly complex (and brittle) code to allow recursive loading of config files, since ipython has the (very useful, it turns out) concept of configuration 'profiles' which can include generic defaults and override them for specialized tasks. This means that options files can specify include statements to pull in other files, recursively. This has proven to be very useful, the rc format is a plain text key/value pair, and internally the options structure is (almost) a simple class with attributes for the options. Because I wrote this in my second week of learning python, the actual implementation is an atrocious mess, and the options class unfortunately has its own methods, which prevents having options with certain reserved names (basically all the methods of a dict). This last part was obviously a big mistake. But the overall idea, despite the shortcomings of my implementation, I think satisfies multiple important (for me) requirements: - flat text for users to edit without pulling them into the mudpit of XML. I simply refuse to ask a user to hand-edit XML, period. From an aesthetic standpoint, I think it would be borderline criminal to do so. - recursive inclusion for handling complex configuration situations. - exposes the options as a simple object with attributes accessible as rc.opt1, rc.opt2, etc. This makes the code which uses all these options (IPython has a _ton_ of cmd line flags) as readable as possible. Anyway, this is just $.02 from yet another one who reinvented this particular wheel... Best, f. ps. On the topic of @decorators and IPython, after some consultations with the ipython user community, the next release will use % for magic functions. This should be out in a few weeks, far before 2.4 is officially out. From janssen at parc.com Sat Oct 2 03:03:35 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Oct 2 03:04:07 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: Your message of "Fri, 01 Oct 2004 15:30:44 PDT." <415DDA94.4060505@ocf.berkeley.edu> Message-ID: <04Oct1.180337pdt."58617"@synergy1.parc.xerox.com> > Just thinking that XML might be nice since all of those poor > souls who don't use Python have easy access to an XML parser but not > necessarily a .ini file parser. Sigh. And here I just wrote a .ini parser for Java... :-). Bill From pinard at iro.umontreal.ca Sat Oct 2 02:43:54 2004 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Sat Oct 2 03:29:10 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001154338f9c617@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <20041002004354.GA24775@titan.progiciels-bpi.ca> [Guido van Rossum] > The outrageous verbosity of XML makes the above example a complete > usability liability. [...] I'm talking about the classic configuration > file pattern where you use your favorite flat-file text editor to edit > the options file. In that situation, using XML is insane. XML may sometimes be useful _when_ a configuration _has_ to be a tree of thick (contents heavy) nodes. For simpler configurations files, and this covers a flurry of cases, Guido above expresses my feelings exactly. There are XML-fanatics -- the same as there are Unicode-fanatics :-). Not so long ago, I've seen a pushy trend by which `/etc/passwd' and a lot of other simple system files would be reformulated as XML, to be XML all over. (Maybe after an announcement of Microsoft -- real or made-up, I do not know -- that they aim such a thing for the next "Windows".) We should not go overboard with XML. Configuration files with lines and words for structuring of a two-level hierarchy have long proven their value. ConfigParser `.ini' files add a third level to these. and with good sense, we can still go another long way without resorting to XML. Besides, when configuration files are going to have some more complex structure, but explicitly made to drive big applications written in Python -- a common case for most of us presumably -- I found out that Python itself is a very convenient and powerful tool for expressing such parameterisation. Users can easily modify a carefully designed Python-written configuration template without knowing a bit about Python itself. If they know the concepts of the applications they are configuring, in my experience at least, they quickly get what needs to be done, and even grasp by mere mimetism, without much explanations, the bits of syntax to respect and follow. With proper care, this can be made incredibly powerful, while staying _way_ more legible than XML. Also compare speed and simplicity, for the application programmer, for bringing such configuration in memory, type concerns already addressed. Of course, XML better inter-operates with foreign programming languages and systems, and would undoubtedly allow easier communication with alien races from neighbouring galaxies :-). But deep down, I do not think that this need in configuration neutrality is common enough to warrant the added complexity, and loss of legibility, for most practical cases. -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From gvanrossum at gmail.com Sat Oct 2 03:40:25 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 03:40:31 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415DE8DB.4060908@ocf.berkeley.edu> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415DE5C4.9080107@ocf.berkeley.edu> <ca471dc2041001162247f825d8@mail.gmail.com> <415DE8DB.4060908@ocf.berkeley.edu> Message-ID: <ca471dc204100118402443d57a@mail.gmail.com> > OK. Should we do an announcement to python-announce and c.l.py saying we are > looking for a new API for .ini file editing/accessing? I don't see the point -- it's not like this is the most important issue of the year. I would rather just see if any of the readers here happen to have code lying around that they like. It sounds like maybe the IPython stuff could be used after a thorough rewrite. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From just at letterror.com Sat Oct 2 11:16:29 2004 From: just at letterror.com (Just van Rossum) Date: Sat Oct 2 11:16:37 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <r01050400-1026-C14748E0145311D9A81C003065D5E7E4@[10.0.0.23]> Guido van Rossum wrote: > > > Well, for me personally, .ini style config files still win over > > > XML every day. And I now have significant experience with both > > > here at ESI. > > > > OK. Do realize that plists are basically .ini style just expressed > > in XML:: > > > > <?xml version="1.0" encoding="UTF-8"?> > > <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" > > "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> > > <plist version="1.0"> > > <dict> > > <key>Section</key> > > <dict> > > <key>key</key> > > <string>value</string> > > <key>key2</key> > > <string>value2</string> > > </dict> > > </dict> > > </plist> > > > > I am not thinking of anything fancy or beyond something like this; > > .ini files expressed in XML. Just thinking that XML might be nice > > since all of those poor souls who don't use Python have easy access > > to an XML parser but not necessarily a .ini file parser. > > This reveals IMO a big mistake in thinking about configuration files. > The most important user of a config file is not the programmer who has > to get data out of it; the most important user is the user who has to > edit the config file. The outrageous verbosity of XML makes the above > example a complete usability liability. Plist is handy when: - you need a simple storage format for simple data structures - you want to be able to edit a file manually in exceptional situations If you need to *routinely* edit these files by hand than plist is indeed not the right choice. It's not a pretty format, but it's well defined. I find it extremely handy and use it all over the place. I'd be happy to contribute plistlib.py to the std library (ie. move it one level up from Lib/plat-mac/ ;-), but I doubt there's enough interest. Just From arigo at tunes.org Sat Oct 2 11:18:46 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Oct 2 11:24:04 2004 Subject: [Python-Dev] [OT] stats on type-inferencing atomic types in local variables in the stdlib In-Reply-To: <415C665A.4060706@ocf.berkeley.edu> References: <415C665A.4060706@ocf.berkeley.edu> Message-ID: <20041002091846.GA16603@vicky.ecs.soton.ac.uk> Hello Brett, On Thu, Sep 30, 2004 at 01:02:34PM -0700, Brett C. wrote: > (101, ('BINARY_MULTIPLY', (8, 4))), > (106, ('BINARY_SUBSCR', 128)), > (118, ('GET_ITER', 128)), > (124, ('BINARY_MODULO', None)), > (195, ('meth_join', 4)), > (204, ('BINARY_ADD', (8, 8))), > (331, ('BINARY_ADD', (4, 4))), > (513, ('BINARY_LSHIFT', (8, 8))), > (840, ('meth_append', 128)), > (1270, ('PRINT_ITEM', 4)), > (1916, ('BINARY_MODULO', 4)), > (12302, ('STORE_SUBSCR', 64))] I was surprized at first to see so few operations involving integers. There isn't even LOAD_SUBSCR for a list and an integer, though I believe it to be a very common operation. It makes me guess that your type-inferencer does not recognize 'i' to be an integer in constructions like for i in range(...) ? If so, it might be worth considering that some built-ins (most likely range(), len(), enumerate()) should be treated specially. Remember there was also discussion in this list at some point about the compiler producing opcodes like BUILTIN_LEN. This means that it might be acceptable to break the precise semantics (which involve looking up the name 'len' or 'range' in the globals first) and just special-case these common built-ins. Armin From anthony at interlink.com.au Sat Oct 2 14:04:57 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat Oct 2 14:07:43 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001154338f9c617@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <415E9969.1040207@interlink.com.au> All this talk of a replacement for ConfigParser is all well and good (and I agree that a more pythonic interface would be nice)... but. We're under 2 weeks from beta1, and I can't see this new module being designed, implemented, and committed before then. Remember, once b1 is out, we're stuck with the API that we have come up with. I'd much rather see one or more packages developed alongside Python - we can then take the best of them (or maybe a merger of the best ideas for them) for 2.5. Which leaves David's original question about the two patches. While CP isn't perfect, and it would be nice to replace it, for 2.4, I don't think these patches make things any worse, and they add useful functionality (back, in one case) so I'm inclined towards accepting them. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From gvanrossum at gmail.com Sat Oct 2 18:22:42 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sat Oct 2 18:22:54 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <415E9969.1040207@interlink.com.au> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415E9969.1040207@interlink.com.au> Message-ID: <ca471dc2041002092223d55c95@mail.gmail.com> > All this talk of a replacement for ConfigParser is all well > and good (and I agree that a more pythonic interface would > be nice)... but. We're under 2 weeks from beta1, and I can't > see this new module being designed, implemented, and committed > before then. Remember, once b1 is out, we're stuck with the > API that we have come up with. Of course. Other recent proposals for brand new stuff also seem poorly timed. > I'd much rather see one or more packages developed alongside > Python - we can then take the best of them (or maybe a merger > of the best ideas for them) for 2.5. Right. So it went for optparse and logging. > Which leaves David's original question about the two patches. > While CP isn't perfect, and it would be nice to replace it, for > 2.4, I don't think these patches make things any worse, and they > add useful functionality (back, in one case) so I'm inclined > towards accepting them. If you think so, go ahead. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bac at OCF.Berkeley.EDU Sat Oct 2 20:26:17 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Oct 2 20:26:29 2004 Subject: [Python-Dev] [OT] stats on type-inferencing atomic types in local variables in the stdlib In-Reply-To: <20041002091846.GA16603@vicky.ecs.soton.ac.uk> References: <415C665A.4060706@ocf.berkeley.edu> <20041002091846.GA16603@vicky.ecs.soton.ac.uk> Message-ID: <415EF2C9.20400@ocf.berkeley.edu> Armin Rigo wrote: > Hello Brett, > > On Thu, Sep 30, 2004 at 01:02:34PM -0700, Brett C. wrote: > >> (101, ('BINARY_MULTIPLY', (8, 4))), >> (106, ('BINARY_SUBSCR', 128)), >> (118, ('GET_ITER', 128)), >> (124, ('BINARY_MODULO', None)), >> (195, ('meth_join', 4)), >> (204, ('BINARY_ADD', (8, 8))), >> (331, ('BINARY_ADD', (4, 4))), >> (513, ('BINARY_LSHIFT', (8, 8))), >> (840, ('meth_append', 128)), >> (1270, ('PRINT_ITEM', 4)), >> (1916, ('BINARY_MODULO', 4)), >> (12302, ('STORE_SUBSCR', 64))] > > > I was surprized at first to see so few operations involving integers. There > isn't even LOAD_SUBSCR for a list and an integer, though I believe it to be a > very common operation. It's picked up and listed above; 106 times. I already type check that lists are being indexed by integer or an object so I didn't bother to keep separate stats for those two cases. > It makes me guess that your type-inferencer does not > recognize 'i' to be an integer in constructions like > > for i in range(...) > > ? Right, I don't infer those situations. > If so, it might be worth considering that some built-ins (most likely > range(), len(), enumerate()) should be treated specially. Remember there was > also discussion in this list at some point about the compiler producing > opcodes like BUILTIN_LEN. This means that it might be acceptable to break the > precise semantics (which involve looking up the name 'len' or 'range' in the > globals first) and just special-case these common built-ins. > Not going to break semantics for my thesis (the whole was not to), but this is all going to be mentioned in the Future Work section on what can be done to improve the situation for type inferencing. It would definitely help if built-ins were known at compile-time, but since I have limited time I had to limit myself to what could be done without adding features to the compiler beyond just type inferencing. -Brett From ncoghlan at email.com Sat Oct 2 22:55:18 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 2 22:55:25 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041002092223d55c95@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415E9969.1040207@interlink.com.au> <ca471dc2041002092223d55c95@mail.gmail.com> Message-ID: <1096750518.415f15b65512d@mail.iinet.net.au> Quoting Guido van Rossum <gvanrossum@gmail.com>: > Of course. Other recent proposals for brand new stuff also seem poorly > timed. > I may be guilty of that. Would it be worth adding a Python 2.5 group to the SF Patch tracker for things which shouldn't be considered seriously until after 2.4 final is out the door? Cheers, Nick. -- Nick Coghlan Brisbane, Australia From python at rcn.com Sun Oct 3 00:22:48 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 00:24:09 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <1096750518.415f15b65512d@mail.iinet.net.au> Message-ID: <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> [Guido] > > Of course. Other recent proposals for brand new stuff also seem poorly > > timed. [Nick] > I may be guilty of that. Would it be worth adding a Python 2.5 group to > the SF > Patch tracker for things which shouldn't be considered seriously until > after 2.4 > final is out the door? Though ill timed, the -m idea is honking good. If it can be reliably worked out in its simplest form (not trying to be all things to all packages), it's worth going ahead for Py2.4. Command line guys like me need this every day. His patch will address a continual source of irritation and make the tool more pleasant to use. Raymond From python at rcn.com Sun Oct 3 00:41:25 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 00:42:43 2004 Subject: [Python-Dev] Py2.5 tracker group In-Reply-To: <1096750518.415f15b65512d@mail.iinet.net.au> Message-ID: <002601c4a8d0$f30d1000$e841fea9@oemcomputer> > Would it be worth adding a Python 2.5 group to > the SF > Patch tracker for things which shouldn't be considered seriously until > after 2.4 > final is out the door? Done. Raymond From gvanrossum at gmail.com Sun Oct 3 02:45:54 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 02:46:01 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> References: <1096750518.415f15b65512d@mail.iinet.net.au> <002001c4a8ce$5b72dc40$e841fea9@oemcomputer> Message-ID: <ca471dc2041002174576806cc0@mail.gmail.com> > Though ill timed, the -m idea is honking good. If it can be reliably > worked out in its simplest form (not trying to be all things to all > packages), it's worth going ahead for Py2.4. Command line guys like me > need this every day. His patch will address a continual source of > irritation and make the tool more pleasant to use. I was in fact thinking of the -m proposal when I wrote that... It's so easy to define an alias or use a one-line shell script for invoking Python with a full pathname that I'm really not sure I like the -m idea, and I worry that it would be done the wrong way because it's so close to beta. Half the time the problem is getting the path set in the first place, and there -m doesn't help you. Supporting it with packages seems insanity. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Sun Oct 3 03:14:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 03:16:10 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041002174576806cc0@mail.gmail.com> Message-ID: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> > I was in fact thinking of the -m proposal when I wrote that... The timing does suck. > It's so easy to define an alias or use a one-line shell script for > invoking Python with a full pathname that I'm really not sure I like > the -m idea, My understanding was that it wasn't about a full pathname to python, it was about searching sys.path for the darned script so us poor Windows guys don't have to change directories a million times a day (no aliases or shebangs for us either). I have a big pile of batch files just to invoke timeit, texchecker, profile, etc. It bites the big one. If the patch doesn't get worked out for Py2.4, I hope it doesn't get abandoned. > Supporting it with packages seems insanity. No doubt. Raymond From ncoghlan at email.com Sun Oct 3 05:27:10 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sun Oct 3 05:27:16 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> Message-ID: <1096774030.415f718ed9a87@mail.iinet.net.au> Quoting Raymond Hettinger <python@rcn.com>: > > It's so easy to define an alias or use a one-line shell script for > > invoking Python with a full pathname that I'm really not sure I like > > the -m idea, > > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). Correct. Ilya's original example was wanting to invoke pdb's script behaviour in order to debug another script. At the moment, you have to run: python -c "from inspect import getsourcefile; import pdb; print getsourcefile(pdb)" or python -c "from imp import find_module; print find_module("pdb")[1]" to find out where the file is, and then run python normally with the filename that gets printed by one of the above scripts. (With a Unix-style shell, backticks make that easier. Doesn't help on Windows though). The "-m" option aims to automate this process (using the latter 'find_module' approach in C code) > > Supporting it with packages seems insanity. > > No doubt. I'm starting to lean that way myself. The semantics for a straight module are easy, but the package version is horrible (Either we have a package getting imported before "__main__" starts running, or else we don't properly support package __path__ variables. I don't find either option appealing) I think I'll simply drop all package support from the patch and allow modules only. That will still cover the initial use cases (pdb, profile and the like) Adding support for packages later (in a different patch) would still be possible if there was interest. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From barry at python.org Sun Oct 3 05:43:06 2004 From: barry at python.org (Barry Warsaw) Date: Sun Oct 3 05:43:09 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096774030.415f718ed9a87@mail.iinet.net.au> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> Message-ID: <1096774986.7313.72.camel@geddy.wooz.org> On Sat, 2004-10-02 at 23:27, Nick Coghlan wrote: > I'm starting to lean that way myself. The semantics for a straight module are > easy, but the package version is horrible (Either we have a package getting > imported before "__main__" starts running, or else we don't properly support > package __path__ variables. I don't find either option appealing) > > I think I'll simply drop all package support from the patch and allow modules > only. That will still cover the initial use cases (pdb, profile and the like) > Adding support for packages later (in a different patch) would still be possible > if there was interest. +1 and I'd still love to see this in Python 2.4. Could you please assign the patch to someone other than me though? I don't honestly think I'll have time to review it before 2.4b1. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041002/4475e5a7/attachment-0001.pgp From fperez528 at yahoo.com Sun Oct 3 06:28:48 2004 From: fperez528 at yahoo.com (Fernando Perez) Date: Sun Oct 3 06:28:23 2004 Subject: [Python-Dev] RE: ConfigParser patches References: <ca471dc2041002174576806cc0@mail.gmail.com> <000801c4a8e6$6217fc20$e841fea9@oemcomputer> Message-ID: <cjnv50$ei6$1@sea.gmane.org> Raymond Hettinger wrote: > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). > > I have a big pile of batch files just to invoke timeit, texchecker, > profile, etc. It bites the big one. Sorry for the plug, but I hope you'll forgive it if it actually turns out to be useful. You may want to have a look at ipython (http://ipython.scipy.org). IPython has a 'magic' run command, which allows you to run any python script from its command line: In [1]: cat argv.py #!/usr/bin/env python import sys print 'argv:',sys.argv In [2]: run argv argv: ['argv.py'] In its current form, it is just an execfile() on steroids. But it would be trivial to add to it a -m option so that it would search in sys.path if nothing in the current directory is found matching the given name. Ipython also gives you aliases and 'bookmarks', a persistent list of directories you can change to with a single 'cd' command: ############### 'screenshot' In [1]: pdoc alias Define an alias for a system command. '@alias alias_name cmd' defines 'alias_name' as an alias for 'cmd' Then, typing 'alias_name params' will execute the system command 'cmd params' (from your underlying operating system). [... snip. It's a long docstring] In [6]: pdoc bookmark Manage IPython's bookmark system. @bookmark <name> - set bookmark to current dir @bookmark <name> <dir> - set bookmark to <dir> @bookmark -l - list all bookmarks @bookmark -d <name> - remove bookmark @bookmark -r - remove all bookmarks You can later on access a bookmarked folder with: @cd -b <name> or simply '@cd <name>' if there is no directory called <name> AND there is such a bookmark defined. ################ /screenshot Using ipython's profile system (referred to earlier in this thread), you can use it as a quasi-system shell. The provided 'pysh' profile loads all of your $PATH (with proper extensions in Windows) as ipython aliases, and you can do regular system things, but retaining python syntax. It has also a mechanism for capturing shell output into python variables and expanding back to the shell: ############### 'screenshot' fperez@maqroll[~/test]|4> print 'Yes, this is still %s !' % 'Python' Yes, this is still Python ! fperez@maqroll[~/test]|5> $$files=ls fperez@maqroll[~/test]|6> type files ------------------------> type(files) <6> <type 'list'> fperez@maqroll[~/test]|7> len files ------------------------> len(files) <7> 31 fperez@maqroll[~/test]|8> for f in files: |.> if len(f)>10: |.> wc -l $f |.> 4 inspectbug.py 73 ramptest.py fperez@maqroll[~/test]|9> run argv.py argv: ['argv.py'] ################ /screenshot Note that the 'run' command optionally takes a -p switch, to run code under the profiler's control. Ipython can also automatically activate the pdb debugger inside any uncaught exception, thanks to the @pdb magic. I simply point these things in the hope that they are useful to you. From what I've read on this thread, ipython seems to already give most of the motivations for the -m switch, and it works today with python >= 2.2. Try it, you might like it :) (I'm not trying to say the -m switch is a bad idea, simply that ipython may provide a useful alternative today). Ok, end of blatant plug. Sorry if it was out of line. Cheers, f From gvanrossum at gmail.com Sun Oct 3 06:33:09 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 06:33:12 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096774030.415f718ed9a87@mail.iinet.net.au> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> Message-ID: <ca471dc2041002213365222500@mail.gmail.com> > Correct. Ilya's original example was wanting to invoke pdb's script behaviour in > order to debug another script. At the moment, you have to run: > > python -c "from inspect import getsourcefile; import pdb; print getsourcefile(pdb)" > > or > > python -c "from imp import find_module; print find_module("pdb")[1]" > > to find out where the file is, and then run python normally with the filename > that gets printed by one of the above scripts. (With a Unix-style shell, > backticks make that easier. Doesn't help on Windows though). > > The "-m" option aims to automate this process (using the latter 'find_module' > approach in C code) I'm not going to stop you from adding this, but I do think you're overlooking the fact that if you need this a lot, it's trivial to write a tiny Python script that takes the place of "python -m", and putting that python script in your PATH. Something like: #!/usr/bin/env python import sys, imp fn = imp.find_module(sys.argv[1])[1] del sys.argv[0] sys.argv[0] = fn execfile(fn) seems do the trick (though it could use nicer error handling). Wouldn't it be easier to add *this* to the Tools/scripts directory (and the list of scripts that are installed by "make install" or the Windows equivalent) than to burden the Python interpreter with C code to do the same? Why does every handy thing that anyone ever thought of have to be a Python command line option? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Sun Oct 3 10:29:46 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 10:37:17 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <20041003012803.FEA1.JCARLSON@uci.edu> Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah From bob at redivi.com Sun Oct 3 10:58:11 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Oct 3 10:58:45 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> References: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <5A34A47C-151A-11D9-AFF0-000A95686CD8@redivi.com> On Oct 3, 2004, at 4:29 AM, Josiah Carlson wrote: > I have produced a patch against the latest CVS to add support for two > new formatting characters in the struct module. It is currently an > RFE, > which I include a link to at the end of this post. Please read the > email before you respond to it. -- > I had proposed two new formatting characters, 'g' and 'G' (for biGint > or > lonG int). > > There was one primary purpose, to offer users the opportunity to > specify > their own integer lengths (very useful for cryptography, and real-world > applications that involve non-standard sized integers). Current > solutions involve shifting, masking, and multiple passes over data. I'm +1 on this.. I've dealt with 24, 48, and 128 bit integers before and it's always been a pain with the struct module. Another addition I'd like to see is bit length struct fields, but that opens an entirely different can of worms. -bob From python at rcn.com Sun Oct 3 17:56:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Oct 3 17:57:28 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> > If you believe this functionality is useful, or even if you think that I > am full of it, tell us: http://python.org/sf/1023290 The suggested alternative is adding two functions to binascii that parallel the existing hexlify() and unhexlify(). From gvanrossum at gmail.com Sun Oct 3 18:06:47 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 3 18:06:53 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> References: <20041003012803.FEA1.JCARLSON@uci.edu> <001f01c4a961$7f564aa0$b529cb97@oemcomputer> Message-ID: <ca471dc2041003090624c1ef54@mail.gmail.com> > > If you believe this functionality is useful, or even if you think that > I > > am full of it, tell us: http://python.org/sf/1023290 > > The suggested alternative is adding two functions to binascii that > parallel the existing hexlify() and unhexlify(). Those functions ought to exist whether or not this RFE is accepted. Here's the crux I think. Is this used often enough in a context where (a) the length of the number is fixed (not determined by a count in a previous field) and (b) preceded or followed by other fixed-length fields so that it makes sense to use the struct module for parsing or formatting those other fields? I have often found that amongst less-experienced programmers there is a great mystery about the correspondence between the "binary" representation of numbers (in strings of bytes) and the numeric objects that Python makes available (int, lont). Often the struct module is considered the only way to cross this boundary, while in fact there are many other approaches; often using the built-in functions ord() or chr() and shifting and masking works just as well, but you have to think about it the right way. I apologize for not having read the entire post before responding; in case the motivation is already there, that's great, and let it be a response to Raymond. If it is not there, I like Raymond's proposal better. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From goodger at python.org Sun Oct 3 18:09:04 2004 From: goodger at python.org (David Goodger) Date: Sun Oct 3 18:09:07 2004 Subject: [Python-Dev] Re: ConfigParser patches In-Reply-To: <ca471dc2041002092223d55c95@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <415E9969.1040207@interlink.com.au> <ca471dc2041002092223d55c95@mail.gmail.com> Message-ID: <41602420.1020002@python.org> [Anthony Baxter] >> Which leaves David's original question about the two patches. >> While CP isn't perfect, and it would be nice to replace it, for >> 2.4, I don't think these patches make things any worse, and they >> add useful functionality (back, in one case) so I'm inclined >> towards accepting them. [Guido van Rossum] > If you think so, go ahead. Taking this as approval, and since it's easier to ask forgiveness (and probably would've been, in the first place ;-), I've gone ahead and checked in the patches to SF bugs 1017864 & 997050, closed and marked the bug reports "accepted". -- David Goodger <http://python.net/~goodger> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 253 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041003/816d5dec/signature.pgp From tim.peters at gmail.com Sun Oct 3 20:14:35 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 3 20:15:10 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <20041003012803.FEA1.JCARLSON@uci.edu> References: <20041003012803.FEA1.JCARLSON@uci.edu> Message-ID: <1f7befae0410031114724f5f2c@mail.gmail.com> [Josiah Carlson] > ... > The 'p' format specifier is also not a standard C type, and yet it > is included in struct, specifically because it is useful. I never used it <wink>. > Argument: > You can already do the same thing with: > pickle.encode_long(long_int) > pickle.decode_long(packed_long) That isn't an argument I've seen before, although I pointed out these pickle functions in the tracker item. The argument there is that pickle demonstrates actual long<->bytes use cases in the core, implemented painfully in Python, and that the struct patch wouldn't make them easier. The argument was not that you can already use the pickle functions, it was that the pickle functions shouldn't need to exist -- they're isolated hacks that address only one part of the whole problem, and even that part is addressed painfully now. The workalike implementation in cPickle.c was actually easier than the Python implementation in pickle.py, because the former get to use the flexible C API functions I added for "this kind of thing". > and some likely soon-to-be included additions to the binascii module. The pickle functions *may* argue in favor of binascii functions, if those were actually specified. I'm not sure Raymond has ever spelled out what their signatures would be, so I don't know. If suitable functions in binascii will exist, then that (as Guido said) raises the bar for adding similar functionality to struct too, but does not (as Guido also said) preclude adding similar functionality to struct. > My Response: > That is not the same. Nontrivial problems require multiple passes > over your data with multiple calls. A simple: > struct.unpack('H3G3G', st) My problem with that use case is that I've never had it, and have never seen an app that had it. Since I've been around for decades, that triggers a suspicion that it's not a common need. ... > Argument: > The struct module has a steep learning curve already, and this new > format specifier doesn't help it. That's another argument I haven't seen before, but bears an hallucinatory resemblance to one I made. People have wondered how to convert between long and bytestring in Python for years, and prior to this iteration, they have always asked whether there's "a function" to do it. Like all the use cases I ever had, they have one long to convert, or one bytestring, at a time. "Ah, see the functions in binascii" would be a direct answer to their question, and one that wouldn't require that they climb any part of struct's learning curve. IOW, if it *could* be done without struct, I'm sure that would make life easier for most people who ever asked about it. For people who already know struct, the marginal learning burden of adding another format code is clearly minor. > I can't see how a new format specifier would necessarily make the > learning curve any more difficult, Neither can I, for people who already know struct. > if it was even difficult in the first place. It is difficult. There are many format codes, they don't all work the same way, and there are distinctions among: - native, standard, and forced endianness - native and standard (== no) alignment - native and standard sizes for some types Newbie confusion about how to use struct is common on c.l.py, and is especially acute among those who don't know C (just *try* to read the struct docs from the POV of someone who hasn't used C). > ... > If you believe this functionality is useful, or even if you think that I > am full of it, tell us: http://python.org/sf/1023290 I certainly would like to see more people say they'd *use* the g and G codes in struct even if "one shot" functions in binascii were available. I'd also like to see a specific design for binascii functions. I don't think "simple" would be an accurate description of those, if they're flexible enough to handle the common use cases I know about. They'd be more like long2bytes(n, length=None, unsigned=False, msdfirst=False) where, by default (length is None), long2bytes(n) is the same as pickle.encode_long(n), except that long2bytes(0) would produce '\0' instead of an empty string. Specifying length <= 0 is a ValueError. length > 0 would be akin to the C API _PyLong_AsByteArray(n, buf, length, not msdfirst, not unsigned) ValueError if n < 0 and unsigned=True. OverflowError if length > 0 and n's magnitude is too large to fit in length bytes. bytes2long(bytes, unsigned=False, msdfirst=False) would be akin to the C API _PyLong_FromByteArray(bytes, len(bytes), not msdfirst, not unsigned) except that bytes=="" would raise ValueError instead of returning 0. From jcarlson at uci.edu Sun Oct 3 20:22:30 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 20:29:40 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <ca471dc2041003090624c1ef54@mail.gmail.com> References: <001f01c4a961$7f564aa0$b529cb97@oemcomputer> <ca471dc2041003090624c1ef54@mail.gmail.com> Message-ID: <20041003110405.FEB5.JCARLSON@uci.edu> > > The suggested alternative is adding two functions to binascii that > > parallel the existing hexlify() and unhexlify(). > > Those functions ought to exist whether or not this RFE is accepted. No argument here. I believe the binascii and struct additions have orthogonal use-cases. > Here's the crux I think. Is this used often enough in a context where > (a) the length of the number is fixed (not determined by a count in a > previous field) and All use-cases I have for it, yes. Seemingly this is a yes with Bob Ippolito as well (he stated he uses 3, 6 and 16 byte integers, which certainly seem fixed). > (b) preceded or followed by other fixed-length > fields so that it makes sense to use the struct module for parsing or > formatting those other fields? In my case, yes. As provided in the email, two that I use right now are 'H3G3G' and '3G5G'. > I have often found that amongst less-experienced programmers there is > a great mystery about the correspondence between the "binary" > representation of numbers (in strings of bytes) and the numeric > objects that Python makes available (int, lont). Often the struct > module is considered the only way to cross this boundary, while in > fact there are many other approaches; often using the built-in > functions ord() or chr() and shifting and masking works just as well, > but you have to think about it the right way. Indeed, this /can/ be the case, but it is not in my case. Before I knew of struct, I had written my own binary packers and unpackers using ord and chr. While it worked, I nearly wept with joy when I discovered struct, which did 90% of what I needed it to do, in a much cleaner fashion. > I apologize for not having read the entire post before responding; in Yeah, it was a bit long, I believed there was a lot to say. - Josiah From shane.holloway at ieee.org Sun Oct 3 21:28:30 2004 From: shane.holloway at ieee.org (Shane Holloway (IEEE)) Date: Sun Oct 3 21:29:02 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <416052DE.3020809@ieee.org> Since "long" is supposed to be a full-fledged member of the python building blocks, I'm +1 for functions being added in both binascii and struct. One of the greatest things I use struct is for is packing (and unpacking) the python building blocks for "external use" -- network, database, and (usually C) libraries. I think it would be best if all the building blocks could be packed and unpacked from one module. The additions to binascii would be more convenient to use of the two additions. But truth to tell, I rarely use binascii. I tend to prefer struct.pack with str.encode. What do you think about adding long.tobytes()/long.frombytes() to go with the new bytes() type? <wink> -Shane Holloway From jcarlson at uci.edu Sun Oct 3 21:48:43 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun Oct 3 21:56:42 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <20041003112337.FEB8.JCARLSON@uci.edu> > > The 'p' format specifier is also not a standard C type, and yet it > > is included in struct, specifically because it is useful. > > I never used it <wink>. If you have used PostgreSQL, all strings are a variant of pascal strings. It may be the case in other databases, but I have little experience with them. > > My Response: > > That is not the same. Nontrivial problems require multiple passes > > over your data with multiple calls. A simple: > > struct.unpack('H3G3G', st) > > My problem with that use case is that I've never had it, and have > never seen an app that had it. Since I've been around for decades, > that triggers a suspicion that it's not a common need. It may be domain specific. I've only been using Python for 4 1/2 years, yet I have used such structs to build socket protocols, databases and search engines, for both class and contract. Heck, I find it useful enough that I have considered donating to the PSF just to see this feature included. > > Argument: > > The struct module has a steep learning curve already, and this new > > format specifier doesn't help it. > > That's another argument I haven't seen before, but bears an > hallucinatory resemblance to one I made. People have wondered how to This was one argument that Raymond has offered a few times. In the case of native alignment issues that seem to be the cause of much frustration among new struct learners, this particular format specifier doesn't much apply; it is not native. > binascii" would be a direct answer to their question, and one that ... So there is no confusion, I agree with Raymond, Guido, and you, that the binascii function additions should occur. > Newbie confusion about how to use struct is common on c.l.py, and is > especially acute among those who don't know C (just *try* to read the > struct docs from the POV of someone who hasn't used C). Good point about the docs regarding not using C. Does it make sense to include documentation regarding C structs (perhaps in an appendix) to help those who have otherwise not experienced C? > I certainly would like to see more people say they'd *use* the g and G > codes in struct even if "one shot" functions in binascii were > available. <raise hand> <wink> > I'd also like to see a specific design for binascii functions. I > don't think "simple" would be an accurate description of those, if > they're flexible enough to handle the common use cases I know about. > They'd be more like [snip conversion operations] Those would work great for packing and unpacking of single longs. - Josiah From lars.blockken at telenet.be Sun Oct 3 22:27:12 2004 From: lars.blockken at telenet.be (Lars Blockken) Date: Sun Oct 3 22:27:13 2004 Subject: [Python-Dev] introducing Message-ID: <002101c4a987$5d5d9630$e701a8c0@lars> Hello, I'm Lars and i'm knew for python so i hope you can answer my questions in the future. Greetz Lars -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041003/837912ce/attachment-0001.htm From carribeiro at gmail.com Sun Oct 3 22:52:12 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Oct 3 22:52:14 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <416052DE.3020809@ieee.org> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> Message-ID: <864d370904100313524c83ef7@mail.gmail.com> On Sun, 03 Oct 2004 13:28:30 -0600, Shane Holloway (IEEE) <shane.holloway@ieee.org> wrote: > One of the greatest things I use struct is for is packing (and > unpacking) the python building blocks for "external use" -- network, > database, and (usually C) libraries. I think it would be best if all > the building blocks could be packed and unpacked from one module. > > The additions to binascii would be more convenient to use of the two > additions. But truth to tell, I rarely use binascii. I tend to prefer > struct.pack with str.encode. > > What do you think about adding long.tobytes()/long.frombytes() to go > with the new bytes() type? <wink> Sorry for introducing my not-very-qualified words on this topic, but... I've read the thread up to this point wondering why the bytes() type were not being thought of as a clean and definitive solution to this problem. It would allow to greatly simplify everything regarding struct, binascii and arbitrary low level data manipulation for networking and similar stuff. I also agree with Tim Peters comments regarding struct's C heritage -- I never really liked C even when I *had* to use it daily, and the struct syntax still reads alien to me. I know this is another timeframe entirely, but *if* my vote counted, I would be +1 for a future struct implementation tightly integrated with the bytes() type. But that's me anyway. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From martin at v.loewis.de Sun Oct 3 23:18:20 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 3 23:18:22 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <864d370904100313524c83ef7@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> <864d370904100313524c83ef7@mail.gmail.com> Message-ID: <41606C9C.5060908@v.loewis.de> Carlos Ribeiro wrote: > Sorry for introducing my not-very-qualified words on this topic, > but... I've read the thread up to this point wondering why the bytes() > type were not being thought of as a clean and definitive solution to > this problem. It would allow to greatly simplify everything regarding > struct, binascii and arbitrary low level data manipulation for > networking and similar stuff. No, it wouldn't. If you have a 'long' value, and you want to convert it to 'bytes', how exactly would you do that? Two's complement, I suppose - but that would close out people who want unsigned numbers. Also, do you want big-endian or little-endian? What about a minimum width, what about overflows? Tim has proposed a signature for binascii that covers all these scenarios, and I doubt it could get simpler then that and still useful. > I also agree with Tim Peters comments regarding struct's C heritage -- > I never really liked C even when I *had* to use it daily, and the > struct syntax still reads alien to me. I know this is another > timeframe entirely, but *if* my vote counted, I would be +1 for a > future struct implementation tightly integrated with the bytes() type. I think you will find that the struct module *already* supports the bytes type. The bytes type will be just a synonym for the current string type, except that people will stop associating characters with the individual bytes; plus the bytes type will be possibly mutable. As the struct module creates (byte) strings today, it will trivially support the bytes type. Regards, Martin From martin at v.loewis.de Sun Oct 3 23:20:47 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 3 23:20:51 2004 Subject: [Python-Dev] introducing In-Reply-To: <002101c4a987$5d5d9630$e701a8c0@lars> References: <002101c4a987$5d5d9630$e701a8c0@lars> Message-ID: <41606D2F.6070004@v.loewis.de> Lars Blockken wrote: > Hello, > I'm Lars and i'm knew for python so i hope you can answer my questions > in the future. Hello Lars, If you are new to Python, python-dev is not the list you should use; it is meant for the development *of* Python, not for the development *with* Python. You can use the python-tutor list if you want to actively learn Python, or python-list (aka news:comp.lang.python) for general discussions and questions. Regards, Martin From carribeiro at gmail.com Sun Oct 3 23:35:41 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Sun Oct 3 23:35:44 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <41606C9C.5060908@v.loewis.de> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <416052DE.3020809@ieee.org> <864d370904100313524c83ef7@mail.gmail.com> <41606C9C.5060908@v.loewis.de> Message-ID: <864d3709041003143579759fcd@mail.gmail.com> On Sun, 03 Oct 2004 23:18:20 +0200, Martin v. L?wis <martin@v.loewis.de> wrote: > Carlos Ribeiro wrote: > > Sorry for introducing my not-very-qualified words on this topic, > > but... I've read the thread up to this point wondering why the bytes() > > type were not being thought of as a clean and definitive solution to > > this problem. It would allow to greatly simplify everything regarding > > struct, binascii and arbitrary low level data manipulation for > > networking and similar stuff. > > No, it wouldn't. If you have a 'long' value, and you want to convert > it to 'bytes', how exactly would you do that? Two's complement, I > suppose - but that would close out people who want unsigned numbers. > Also, do you want big-endian or little-endian? What about a minimum > width, what about overflows? Well, I don't intend to get way too off topic. But in my mind, it makes sense to have a few methods to allow any struct-type hack to work *directly* with the bytes() type. For example: the bytes() type could have a constructor that would take a struct-type string, as in: bytes(data, 'format-string-in-struct-format') or bytes.fromstruct(data, 'format-string-in-struct-format') Alternatively, an append method could take two parameters, one being the data to be appended, and the other the 'transformation rule' -- big endian, little endian, etc: bytes.append(data, 'format-string-in-struct-format') The interface for the data specification could also be a little bit cleaner; I don't see great value at specifying everything with single-character codes. It may look obvious to old C coders, but it doesn't mean that it's the only, or the better, way to do it. Besides concatenation, a few other transformations are useful for bytes -- shifting and rotation in both directions (working at bit level, perhaps?). That's how I thought it should work. (... and, as far as binascii is concerned, I see it more as a way to encode/decode binary data in true string formats than anything else.) > Tim has proposed a signature for binascii that covers all these > scenarios, and I doubt it could get simpler then that and still useful. > > > I also agree with Tim Peters comments regarding struct's C heritage -- > > I never really liked C even when I *had* to use it daily, and the > > struct syntax still reads alien to me. I know this is another > > timeframe entirely, but *if* my vote counted, I would be +1 for a > > future struct implementation tightly integrated with the bytes() type. > > I think you will find that the struct module *already* supports > the bytes type. The bytes type will be just a synonym for the current > string type, except that people will stop associating characters > with the individual bytes; plus the bytes type will be possibly mutable. > As the struct module creates (byte) strings today, it will trivially > support the bytes type. As I've explained above, it's a good first step, but a true bytes() type could have a little bit more functionality than char strings have. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From adurdin at gmail.com Mon Oct 4 00:53:10 2004 From: adurdin at gmail.com (Andrew Durdin) Date: Mon Oct 4 00:53:13 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <1f7befae0410031114724f5f2c@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> Message-ID: <59e9fd3a041003155370ea8761@mail.gmail.com> On Sun, 3 Oct 2004 14:14:35 -0400, Tim Peters <tim.peters@gmail.com> wrote: > > I certainly would like to see more people say they'd *use* the g and G > codes in struct even if "one shot" functions in binascii were > available. I have an application where I have to read and write a series of 24-bit integers in a binary file. The g and G codes would make this more convenient, as well as making all the reading and writing code more consistent (as the rest of it uses struct). From skip at pobox.com Sat Oct 2 19:03:14 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Oct 4 04:00:35 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <ca471dc2041001154338f9c617@mail.gmail.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <16734.57170.116719.419227@montanaro.dyndns.org> Guido> This reveals IMO a big mistake in thinking about configuration Guido> files. The most important user of a config file is not the Guido> programmer who has to get data out of it; the most important user Guido> is the user who has to edit the config file. The outrageous Guido> verbosity of XML makes the above example a complete usability Guido> liability. Agreed. What about YaML? It's human readable/editable and uses indentation to denote structure instead of <tags>. I'd Google for a reference but I'm off-net at the moment. Skip From pje at telecommunity.com Mon Oct 4 04:44:31 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Oct 4 04:44:23 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <16734.57170.116719.419227@montanaro.dyndns.org> References: <ca471dc2041001154338f9c617@mail.gmail.com> <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> Message-ID: <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> At 12:03 PM 10/2/04 -0500, Skip Montanaro wrote: > Guido> This reveals IMO a big mistake in thinking about configuration > Guido> files. The most important user of a config file is not the > Guido> programmer who has to get data out of it; the most important user > Guido> is the user who has to edit the config file. The outrageous > Guido> verbosity of XML makes the above example a complete usability > Guido> liability. > >Agreed. > >What about YaML? It's human readable/editable and uses indentation to >denote structure instead of <tags>. I'd Google for a reference but I'm >off-net at the moment. YaML isn't very friendly to non-techies, IMO. I consider it a very good human *readable* format, but not a human *writable* format. I honestly have an easier time writing XML than YaML, because there are fewer symbols to remember, the format is more regular, and I don't have to think as hard about what the processor is looking for. Of course, I *definitely* prefer .ini files to XML, if they are sufficient for the use case. From carribeiro at gmail.com Mon Oct 4 04:53:44 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 04:53:50 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> References: <000c01c4a7f2$50c92420$e841fea9@oemcomputer> <200410011637.57629.fdrake@acm.org> <415DD189.7000900@ocf.berkeley.edu> <ca471dc20410011509509e83a8@mail.gmail.com> <415DDA94.4060505@ocf.berkeley.edu> <ca471dc2041001154338f9c617@mail.gmail.com> <16734.57170.116719.419227@montanaro.dyndns.org> <5.1.1.6.0.20041003224007.036bf4b0@mail.telecommunity.com> Message-ID: <864d370904100319535d14ce67@mail.gmail.com> On Sun, 03 Oct 2004 22:44:31 -0400, Phillip J. Eby <pje@telecommunity.com> wrote: > Of course, I *definitely* prefer .ini files to XML, if they are sufficient > for the use case. Which means almost always, as far as text configuration files are concerned. (if the configuration is complex enough as to *require* XML, then it's probably better to provide a full fledged user interface for the customization anyway). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ncoghlan at email.com Mon Oct 4 05:28:06 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 05:28:16 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <ca471dc2041002213365222500@mail.gmail.com> References: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> <1096774030.415f718ed9a87@mail.iinet.net.au> <ca471dc2041002213365222500@mail.gmail.com> Message-ID: <1096860486.4160c346380a5@mail.iinet.net.au> Quoting Guido van Rossum <gvanrossum@gmail.com>: > #!/usr/bin/env python > import sys, imp > fn = imp.find_module(sys.argv[1])[1] > del sys.argv[0] > sys.argv[0] = fn > execfile(fn) > > seems do the trick (though it could use nicer error handling). And is basically identical to the C version of this code (which uses _PyImp_FindModule and PyRun_SimpleFileExFlags). I like this idea, as it should automatically work with other versions of the interpreter like Jython and IronPython, whereas the command line option would required reimplementation in each interpreter. The hardest part would be to come up with a name for the script (pymod?, pymodule?, pyrun?, runpy?). It will probably be slightly more verbose to invoke than '-m' would be (particularly on an uninstalled development build), but also easier to enhance (e.g supporting modules inside packages). Since I'm on Linux at the moment, and can't recall how the Windows installer handles things like pydoc, I'll wait and see what Raymond and others think about how well this would work on Windows. Unless they think it won't work for other platforms, I'm inclined to reject my own patch and look at a script-based approach. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From martin at v.loewis.de Mon Oct 4 08:16:33 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Oct 4 08:16:36 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <59e9fd3a041003155370ea8761@mail.gmail.com> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <59e9fd3a041003155370ea8761@mail.gmail.com> Message-ID: <4160EAC1.4010904@v.loewis.de> Andrew Durdin wrote: > I have an application where I have to read and write a series of > 24-bit integers in a binary file. The g and G codes would make this > more convenient, as well as making all the reading and writing code > more consistent (as the rest of it uses struct). Out of curiosity: is this a fixed number of of integers in that series? If so, how many? If not, it seems you might be better served if this was an extension to the array module. Regards, Martin From python at rcn.com Mon Oct 4 09:03:52 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Oct 4 09:05:14 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <ca471dc2041002213365222500@mail.gmail.com> Message-ID: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> [Guido] > #!/usr/bin/env python > import sys, imp > fn = imp.find_module(sys.argv[1])[1] > del sys.argv[0] > sys.argv[0] = fn > execfile(fn) > > seems do the trick (though it could use nicer error handling). > > Wouldn't it be easier to add *this* to the Tools/scripts directory Though not as clean as -m, this script helps. For my own insufficiently advanced windows box, I added a launcher batch file, pythonm.bat: @python \py24\tools\scripts\runpy.py %1 %2 %3 %4 %5 %6 %7 %8 %9 It runs pretty well: C:\tmp> pythonm timeit -s "a=range(20)" "a.append(1)" "a.pop()" 100000 loops, best of 3: 1.75 usec per loop C:\tmp> pythonm regrtest test_string test_string 1 test OK. The combination of batch file and run.py isn't as good as the -m option, but it works and mostly meets my needs. The batch file is rather dumb, doesn't work with other python command line options, and won't work pipes. > Why does every handy thing that anyone ever thought of > have to be a Python command line option? Could Nick's idea be done without an -m option? If a user types, "python abc.py" and "abc.py" is not found, before aborting, try looking for it on sys.path. Raymond From adurdin at gmail.com Mon Oct 4 13:21:48 2004 From: adurdin at gmail.com (Andrew Durdin) Date: Mon Oct 4 13:21:50 2004 Subject: [Python-Dev] proposed struct module format code addition In-Reply-To: <4160EAC1.4010904@v.loewis.de> References: <20041003012803.FEA1.JCARLSON@uci.edu> <1f7befae0410031114724f5f2c@mail.gmail.com> <59e9fd3a041003155370ea8761@mail.gmail.com> <4160EAC1.4010904@v.loewis.de> Message-ID: <59e9fd3a04100404213f20ffb6@mail.gmail.com> On Mon, 04 Oct 2004 08:16:33 +0200, Martin v. L?wis <martin@v.loewis.de> wrote: > Andrew Durdin wrote: > > I have an application where I have to read and write a series of > > 24-bit integers in a binary file. The g and G codes would make this > > more convenient, as well as making all the reading and writing code > > more consistent (as the rest of it uses struct). > > Out of curiosity: is this a fixed number of of integers in that series? > If so, how many? > > If not, it seems you might be better served if this was an extension > to the array module. There are a fixed number of them -- though it's somewhere in the thousands range... The array module would handle them quite nicely if it supported 3-byte integers; but in general, a more generic struct module will be handier than a more generic array module (I've never dealt with a tuple with thousands of entries before -- is it likely to be a problem? Anyway, wrapping it in a function to read all the ints in blocks and put them in a list is very little trouble). From ncoghlan at email.com Mon Oct 4 13:43:30 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 13:43:36 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> Message-ID: <1096890210.4161376227c2d@mail.iinet.net.au> Quoting Raymond Hettinger <python@rcn.com>: > The combination of batch file and run.py isn't as good as the -m option, > but it works and mostly meets my needs. The batch file is rather dumb, > doesn't work with other python command line options, and won't work > pipes. Hmm, that point about command line options is a good one. I know I'd find it handy to be able to throw in a '-i' when I wanted one. Then again for commands with minor variations, I tend to rely on command line history rather than using a batch file (that is, I'll type out a long invocation sequence once a session, then edit it as needed) > Could Nick's idea be done without an -m option? To be fair to Ilya - his idea, my implementation :) > If a user types, > "python abc.py" and "abc.py" is not found, before aborting, try looking > for it on sys.path. That's do-able, but the command line option seems cleaner if the interpreter is going to provide C-level support for this. I interpreted Guido's last message as saying he's prepared to tolerate the command line option if there's sufficient support for it. To summarise the differences between the two approaches: Pros (-m over runpy.py, in order of significance as I see it): - easy to combine with other Python command line options - OS & environment independent - easier to use with a non-installed interpreter - no work for Python packaging folks - more concise to invoke - no namespace issues with naming a script - C API for those embedding python Cons: - YACLO (Yet Another Command Line Option) - CPython specific (other interpreters could choose to reimplement) With YACLO seeming to be Guido's main reason for being -0 on the idea. It looks like the script option would require input from those with better knowledge than me of the packaging set up for Windows, *nix and Mac as to how it could be made to work 'nicely' in each environment. And the issue of combining it with other command line options would remain. It's all very well to have "runpy.py" as an executable script, but providing extra options to the interpreter would require: 1. Locate runpy.py (e.g. /usr/bin/local/runpy.py, /usr/bin/runpy.py) 2. Invoke python <options> <dir>/runpy.py <module> Which means we're back to having to find out where at least part of the Python installation lives on each machine (only once per machine, which is an improvement over the status quo, but still not as tidy as a single command line option). I find that persuasive enough that I think it's worthwhile to finish the '-m' patch, and add a documentation patch. Regards, Nick. -- Nick Coghlan Brisbane, Australia From bhvijaykumar at lucent.com Mon Oct 4 14:03:28 2004 From: bhvijaykumar at lucent.com (B H, Vijaya Kumar (Vijaya)) Date: Mon Oct 4 14:03:35 2004 Subject: [Python-Dev] new member Message-ID: <6733C768256DEC42A72BAFEFA9CF06D20D04C980@ii0015exch002u.iprc.lucent.com> Hi all My name is Vijay I am new member of this group I have been working on python from last 1 year , i love this language for its simpicity , i also worked on jython which i used in few of my automation tasks Regards Vijaya Kumar B.H. From aahz at pythoncraft.com Mon Oct 4 16:01:10 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Oct 4 16:01:47 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> References: <ca471dc2041002213365222500@mail.gmail.com> <000301c4a9e0$4eb575e0$6633c797@oemcomputer> Message-ID: <20041004140109.GA22637@panix.com> On Mon, Oct 04, 2004, Raymond Hettinger wrote: > > Could Nick's idea be done without an -m option? If a user types, > "python abc.py" and "abc.py" is not found, before aborting, try looking > for it on sys.path. -1 -- too much magic "Explicit is better than implicit" -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines." --Ralph Waldo Emerson From carribeiro at gmail.com Mon Oct 4 16:29:41 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 16:29:44 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <20041004140109.GA22637@panix.com> References: <ca471dc2041002213365222500@mail.gmail.com> <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <20041004140109.GA22637@panix.com> Message-ID: <864d3709041004072960b5a213@mail.gmail.com> On Mon, 4 Oct 2004 10:01:10 -0400, Aahz <aahz@pythoncraft.com> wrote: > On Mon, Oct 04, 2004, Raymond Hettinger wrote: > > > > Could Nick's idea be done without an -m option? If a user types, > > "python abc.py" and "abc.py" is not found, before aborting, try looking > > for it on sys.path. > > -1 -- too much magic It's exactly what virtually every shell does. I sincerely don't think this would come up as surprise. On the other hand, I've often became frustrated to have to type long (I mean looooong) paths by hand, without autocompletion to help (welcome to the Windows DOS box!), just to discover that the script wasn't exactly at *that* point, but still it could be found somewhere down the path. (Just to mention, and for the sake of completeness: instead of sys.path, it could search on the shell path... but that's not a good solution either, it seems even more arbitrary) OTOH, I'm also a little bit concerned about YACLO, because there's always more cruft to add. And finally, why '-m'? Just because this letter was available ;-) ? I think that for some stuff, long names are better, even if I have to type them sometimes, and specially if I can have the default stored somewhere. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From gvanrossum at gmail.com Mon Oct 4 17:05:01 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 4 17:05:07 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096890210.4161376227c2d@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: <ca471dc2041004080589d870@mail.gmail.com> > Pros (-m over runpy.py, in order of significance as I see it): > - easy to combine with other Python command line options The script approach can do this too > - OS & environment independent So is the script -- probably more so, since the script can use Python's OS independence layer. > - easier to use with a non-installed interpreter > - no work for Python packaging folks No contest. > - more concise to invoke Depends on the length of the name of the script. :-) > - no namespace issues with naming a script Actually, one-letter options are a much scarcer resource than script names. > - C API for those embedding python And who needs that? > Cons: > - YACLO (Yet Another Command Line Option) > - CPython specific (other interpreters could choose to reimplement) Additional Pros for using a script: - less code to maintain - can work with older Python versions - shows users how to do a similar thing themselves with additional features (e.g. a common addition I expect will be to hardcode a personal sys.path addition) PS a big honking -1000 on Carlos Ribeiro's suggestion of doing this automatically if the filename path isn't found. Too many chances for accidentally invoking the wrong thing -- including mysyerious silences when the named module happens to exist but doesn't do anything when run as a script (i.e. the majority of modules). (As an aside, I wonder why every single of Carlos' interactions starts of with an idea that is completely misguided and then ends with him vehemently defending it against all opposition. Sounds like a recipe for flame wars to me.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Mon Oct 4 17:30:35 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Mon Oct 4 17:29:24 2004 Subject: [Python-Dev] Re: '-m' option (was RE: ConfigParser patches) In-Reply-To: <864d3709041004072960b5a213@mail.gmail.com> References: <ca471dc2041002213365222500@mail.gmail.com> <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <20041004140109.GA22637@panix.com> <864d3709041004072960b5a213@mail.gmail.com> Message-ID: <cjrq8e$kkf$1@sea.gmane.org> Carlos Ribeiro wrote: > ... I've often became frustrated to have to type long (I mean looooong) > paths by hand, without autocompletion to help (Windows DOS box!)... Carlos, if you are using CMD.EXE (that is, on Win2K or XP), try running CMD /F:ON (I have a desktop link to cmd.exe that does this). This gives Ctrl-F to complete filenames and Ctrl-D for directories. You can change the keys used by tweaking the registry, but I've never felt the need. You'll get a description from "help cmd". -- -- Scott David Daniels Scott.Daniels@Acm.Org From carribeiro at gmail.com Mon Oct 4 18:30:31 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 4 18:30:36 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <ca471dc2041004080589d870@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> Message-ID: <864d370904100409307d342465@mail.gmail.com> On Mon, 4 Oct 2004 08:05:01 -0700, Guido van Rossum <gvanrossum@gmail.com> wrote: > PS a big honking -1000 on Carlos Ribeiro's suggestion of doing this > automatically if the filename path isn't found. Too many chances for > accidentally invoking the wrong thing -- including mysyerious silences > when the named module happens to exist but doesn't do anything when > run as a script (i.e. the majority of modules). (As an aside, I wonder > why every single of Carlos' interactions starts of with an idea that > is completely misguided and then ends with him vehemently defending it > against all opposition. Sounds like a recipe for flame wars to me.) Sorry. Really. I may have expressed myself VERY badly on this topic. Please, let me defend myself as I don't like to leave this impression as I'm just starting here: -- The original idea was not mine -- I was just commenting because it seemed good to me, as I'm limited to a DOS box most of the time, and searching modules by hand tend to become an issue in this enviroment; -- I really meant to say this (regarding YACLO): I *don't* think that adding command line options for everything is a good idea for any software. Also, I think that single line options are confusing, and longer option names are better/more readable. It seems that the message was read with the opposite meaning. -- As far as the rest of the comments are concerned, I apologize if I tend to sound so 'flamish'. Actually, I have a tendency to think aloud sometimes, and to explore different ideas. Sometimes it makes for amusing comments, sometimes it's weird, and sometimes it just sounds dumb. Sometimes I get it right. But overall it's not the right thing to do, specially as I'm yet to make some useful contribution. In the end, I think I should learn to keep my mouth/fingers shut. Lesson taken, and really, really, sorry. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From ilya at bluefir.net Mon Oct 4 19:33:26 2004 From: ilya at bluefir.net (Ilya Sandler) Date: Mon Oct 4 19:31:55 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096890210.4161376227c2d@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> Message-ID: <Pine.LNX.4.58.0410041017040.12744@bagira> > To summarise the differences between the two approaches: > > Pros (-m over runpy.py, in order of significance as I see it): > - easy to combine with other Python command line options > - OS & environment independent > - easier to use with a non-installed interpreter > - no work for Python packaging folks > - more concise to invoke > - no namespace issues with naming a script > - C API for those embedding python There is one more small pro: easier to version: e.g if there are multiple installed versions of python on your machine, you could could invoke a module from a particular python version: python2.4 -m pydoc with a wrapper script you will need a wrapper per version...(Or some way to find/invoke the right version of python from the wrapper) Ilya > > Cons: > - YACLO (Yet Another Command Line Option) > - CPython specific (other interpreters could choose to reimplement) > > With YACLO seeming to be Guido's main reason for being -0 on the idea. > > It looks like the script option would require input from those with better > knowledge than me of the packaging set up for Windows, *nix and Mac as to how it > could be made to work 'nicely' in each environment. > > And the issue of combining it with other command line options would remain. It's > all very well to have "runpy.py" as an executable script, but providing extra > options to the interpreter would require: > > 1. Locate runpy.py (e.g. /usr/bin/local/runpy.py, /usr/bin/runpy.py) > 2. Invoke python <options> <dir>/runpy.py <module> > > Which means we're back to having to find out where at least part of the Python > installation lives on each machine (only once per machine, which is an > improvement over the status quo, but still not as tidy as a single command line > option). > > I find that persuasive enough that I think it's worthwhile to finish the '-m' > patch, and add a documentation patch. > > Regards, > Nick. > > -- > Nick Coghlan > Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From p.f.moore at gmail.com Mon Oct 4 20:15:52 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Mon Oct 4 20:16:01 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: <ca471dc2041004080589d870@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> Message-ID: <79990c6b04100411157d14cf43@mail.gmail.com> The problem here is really one of packaging. How should utility scripts be distributed, either with Python, or with extensions? There are many different approaches: 1. a library module with useful behaviour in the __main__ block. 2. a script in sys.prefix + '/scripts' with no extension and a #! line 3. a script as above, but with a .py extension 4. a script as (2) plus a .bat wrapper for Windows 5. a script as (3) plus a .bat wrapper for Windows However, *none* of these give the best behaviour, uniformly across all Windows platforms (the Unix story is far better, as there is a consistent and useful behaviour across all shells). While case (1) is a bit of an anomaly, Guido's script would probably cover it, *if* a really acceptable solution for scripts could be found. Accepting that Unix is generally not an issue, let's look at Windows. On cmd.exe, at least on Windows 2000 and up, the following makes .py files "executable" by python: assoc .py=Python.File ftype Python.File=C:\Python23\python.exe "%1" %* set PATHEXT=%PATHEXT%;.py Can anyone test this on COMMAND.COM on Win9x? It works using the COMMAND.COM supplied with Windows 2000, whatever that proves :-) Of these, the first two are set by the standard Windows install. The final one could be. With that change, simply putting a .py script into C:\Python23\Scripts would be enough (assuming C:\Python23\Scripts was on the user's PATH, which it isn't by default, but the user can add it). Proposal: - Change the Windows Python binary to add .py (and .pyw, maybe) to PATHEXT - Add a variation of Guido's script to the standard install, as something like runmod.py - Document that the way for extensions to include scripts is as .py files in the sys.prefix+'/scripts' directory, with a #! line for Unix - For Unix users who don't like extensions, suggest (again in the docs) that a hardlink can be added without the extension. Maybe include a suitable setup.py snippet to show how. - Possibly add sys.path+'/scripts' to the PATH - I'm not sure about this, as on Windows the installer seems to try hard to *not* modify PATH, so maybe this is a policy issue. I'm punting on the issue of python.exe flags. If someone wants to run a supplied script as python -E -S -O script.py, I don't have a solution (other than doing so "longhand"). But I don't have a use case, either, so I'll leave that one for others... Paul From tim.peters at gmail.com Mon Oct 4 20:49:43 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 4 20:49:49 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: <79990c6b04100411157d14cf43@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> Message-ID: <1f7befae04100411492fc43961@mail.gmail.com> [Paul Moore] > ... > Accepting that Unix is generally not an issue, let's look at Windows. > On cmd.exe, at least on Windows 2000 and up, the following makes .py > files "executable" by python: > > assoc .py=Python.File > ftype Python.File=C:\Python23\python.exe "%1" %* > set PATHEXT=%PATHEXT%;.py > > Can anyone test this on COMMAND.COM on Win9x? PATHEXT has no special meaning on 9x; you can set PATHEXT on 9x, but it has no effect. ... > - Change the Windows Python binary to add .py (and .pyw, maybe) to > PATHEXT I don't like that. The Python Windows installer doesn't add, or modify, any environment variables now. It's anti-social to muck with them. Anyone using "a DOS box" should know how to do that themself -- if that's what they want. I'll note that I spend much of my life in a WinXP DOS box running Python programs, but I haven't set PATHEXT. Since cmd.exe's path completion fills in the trailing .py on a .py file all by itself, setting PATHEXT wouldn't save me any typing. From gvanrossum at gmail.com Mon Oct 4 21:24:43 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 4 21:24:46 2004 Subject: [Python-Dev] Utility scripts (Was: '-m' option) In-Reply-To: <1f7befae04100411492fc43961@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: <ca471dc204100412242056acec@mail.gmail.com> > > - Change the Windows Python binary to add .py (and .pyw, maybe) to > > PATHEXT > > I don't like that. The Python Windows installer doesn't add, or > modify, any environment variables now. It's anti-social to muck with > them. Anyone using "a DOS box" should know how to do that themself -- > if that's what they want. I'll note that I spend much of my life in a > WinXP DOS box running Python programs, but I haven't set PATHEXT. > Since cmd.exe's path completion fills in the trailing .py on a .py > file all by itself, setting PATHEXT wouldn't save me any typing. Right. Typing the .py is usually the right thing to do. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From p.f.moore at gmail.com Mon Oct 4 22:03:24 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Mon Oct 4 22:11:07 2004 Subject: [Python-Dev] Re: Utility scripts References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> Message-ID: <uis9qcktv.fsf@yahoo.co.uk> Tim Peters <tim.peters@gmail.com> writes: > [Paul Moore] >> ... >> Can anyone test this on COMMAND.COM on Win9x? > > PATHEXT has no special meaning on 9x; you can set PATHEXT on 9x, but > it has no effect. Right. That means that this isn't a general Windows solution. On the other hand, further experimentation seems to imply that all that PATHEXT actually does is allow the ".py" to be omitted - I hadn't realised that. So as things stand, putting a ".py" script, with a #! line, into sys.prefix+'/scripts', and setting the executable bit on Unix, is already a general way of installing a command - with two provisos: 1. The user must add the directory to his PATH 2. The command must be typed with the .py extension specified So why do existing extensions not do this? Things like wxPython and Twisted seem to jump through a lot of hoops for little benefit beyond allowing users to omit the .py extension... Maybe it would be worth adding a paragraph to section 3.4 of "Distributing Python Modules" (the "Installing Scripts" section) saying something like: Scripts should be given names with a .py extension. This allows them to be executed on both Unix and Windows systems (Windows uses the extension to locate the correct interpreter, in this case Python, in much the same way that Unix uses the #! line). Of course, this will enrage Unix users who dislike file extensions :-) Maybe having a user-specifiable option, something like --strip-ext, to strip the .py extension from scripts when installing, would be useful. (I recall something like this being discussed on the distutils-sig a while back. Maybe this discussion should move there.) >> - Change the Windows Python binary to add .py (and .pyw, maybe) to >> PATHEXT > > I don't like that. The Python Windows installer doesn't add, or > modify, any environment variables now. It's anti-social to muck with > them. OK, that's the policy I was thinking of. Paul. -- Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind. -- Dr. Seuss From fdrake at acm.org Mon Oct 4 22:45:48 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Oct 4 22:45:57 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <uis9qcktv.fsf@yahoo.co.uk> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1f7befae04100411492fc43961@mail.gmail.com> <uis9qcktv.fsf@yahoo.co.uk> Message-ID: <200410041645.48976.fdrake@acm.org> On Monday 04 October 2004 04:03 pm, Paul Moore wrote: > Of course, this will enrage Unix users who dislike file extensions :-) Yes, indeed it will, as well it should. > Maybe having a user-specifiable option, something like --strip-ext, to > strip the .py extension from scripts when installing, would be useful. > (I recall something like this being discussed on the distutils-sig a > while back. Maybe this discussion should move there.) We discussed something very similar to this, but it wasn't so much a per-user distinction. As a package author, the name of the executables that get installed as part of my package are part of the user interface; I should be abe to control them on each platform I support. There are enough twists to this that we didn't come up with a general solution at the time, and I've not had time to revisit the topic since then, though I'd like to get something more workable over the next few months. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From ncoghlan at email.com Mon Oct 4 23:07:22 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:07:28 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <ca471dc2041004080589d870@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> Message-ID: <1096924042.4161bb8a0a35a@mail.iinet.net.au> Aside for Carlos: 'm' is short for 'module', and beefing up Python's option parsing to handle things like '--module' is a whole 'nother story. 'python -h' gives a decent description of all the options, though. Guido van Rossum wrote: >>Pros (-m over runpy.py, in order of significance as I see it): >> - easy to combine with other Python command line options > > The script approach can do this too Only by spelling out the invocation if we want something non-standard. Using the laptop I'm typing on as an example. . . Python 2.3 is the installed version, so "#!/usr/bin/env python" in a script will invoke Py 2.3 without any extra options If I (or a Python installer) were to create "/usr/local/bin/runpy.py" with that shebang line (i.e. the same as the one you posted), we have the following for profiling a sample script in my current directory with different versions of the interpreter: Installed: runpy.py profile demo.py Prompt After: python -i /usr/local/bin/runpy.py profile demo.py Alt install: python2.4 /usr/local/bin/runpy.py profile demo.py Build dir: ./python /usr/local/bin/runpy.py profile demo.py If we wanted to use the version of the script that came with the relevant version of python, those last two would become: Alt install*: python2.4 /usr/local/lib/python2.4/runpy.py profile demo.py Build dir: ./python Tools/scripts/runpy.py profile demo.py (* This is based on what happens to pydoc under 'make altinstall'. The shebang line is version agnostic, so it tries to use the Py2.3 interpreter with the Py2.4 library modules and it all falls apart. So to run a library module of the altinstall, this is how I have to do it. And this is assuming the script would get installed at all, which it may not, as it isn't actually a library module, unlike pydoc) Using -m, those become: Installed: python -m profile demo.py Prompt After: python -i -m profile demo.py Alt install: python24 -m profile demo.py Build dir: ./python -m profile demo.py >> - OS & environment independent > > So is the script -- probably more so, since the script can use > Python's OS independence layer. Paul's message goes into detail on what I meant here. The script itself is highly portable, the mechanism for invoking it really isn't. The C code in the patch is platform independent - _PyImport_FindModule (a trivial wrapper around the existing find_module in import.c) and PyRun_SimpleFileExFlags do all the heavy lifting. In fact, sans error-checking, the guts of PyRun_SimpleModuleFlags looks remarkably similar to the Python code in your script. (I initially thought the Python version might handle zip imports, while the C version didn't. However, a quick experiment shows that *neither* of them can handle zip imports. And the source code confirms it - imp.find_module and imp.load_module don't allow zip imports, and PyRun_Module in the patch doesn't allow them either. Amusingly, the current limitations of the imp module make it easier to support zip imports with the *C* version. Allowing imp.find_module to return a 4th value for the module loader would require adding an optional boolean argument to avoid breaking existing code, whereas the patch's new private C API for _PyImport_FindModule already exposes the loader argument) >> - more concise to invoke > Depends on the length of the name of the script. :-) See the examples above for what I meant with this one. For the vanilla case you're right, but as soon as we do anything slightly different, the story changes. >> - no namespace issues with naming a script > Actually, one-letter options are a much scarcer resource than script names. Well, with '-m' in place, we'd be using 17 out of the available 62 (upper & lower alpha, plus digits). The difference is that we're only competing with ourselves and the other Python interpreter authors for characters to use, whereas we're competing with all and sundry for unique executable names. (Windows isn't immune, either, given enough applications with directories on PATH. Although retaining the '.py' helps a lot there) (I have checked that Jython at least doesn't use '-m' for anything. I don't know about other interpreters) >> - C API for those embedding python > And who needs that? There's a reason this one was last on my list :) > Additional Pros for using a script: > - less code to maintain Once we factor in the additional packaging requirements to make the script as easy to use on all target platforms as -m would be, I think this one is at least arguable (script + packaging vs option-parsing and C function). > - can work with older Python versions > - shows users how to do a similar thing themselves with additional > features Certainly, dropping a version of this script into Tools/scripts in CVS couldn't hurt, regardless of whether or not '-m' gets adopted. The same would go for an entry in the Python cookbook. > (e.g. a common addition I expect will be to hardcode a > personal sys.path addition) Except that this feature isn't so much for your *own* scripts, as it is for installed modules that are also usable as scripts (like profile and pdb). In the former case, you *know* where those scripts are (for me ~/script_name usually does the trick on *nix). In the latter case, though, it'd be nice to be able to use these things easily 'out of the box'. For those who want to tweak the search behaviour, all the usual environment variables apply (PYTHONPATH in particular). Heck, there's nothing to stop someone from doing something like the following if they really want to: python -m my_run_module_script some_other_module The command line interface is one of the major holdouts in Python where we really need to care where the source file for a module lives. It'd be nice to change that. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From ncoghlan at email.com Mon Oct 4 23:21:41 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:25:32 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096924042.4161bb8a0a35a@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <1096924042.4161bb8a0a35a@mail.iinet.net.au> Message-ID: <1096924901.4161bee5b2cec@mail.iinet.net.au> Quoting Nick Coghlan <ncoghlan@email.com>: > Using -m, those become: > > Installed: python -m profile demo.py OK, of course Python 2.3 doesn't have a '-m' switch. Consider the example to start with the first version to ship with '-m' installed. For actual Python 2.3, this would obviously have to use the absolute path, or some flavour of Guido's script (probably ~/runpy, since that is handiest). Cheers, Nick. -- Nick Coghlan Brisbane, Australia From tim.peters at gmail.com Mon Oct 4 23:29:28 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 4 23:29:36 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <uis9qcktv.fsf@yahoo.co.uk> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> <uis9qcktv.fsf@yahoo.co.uk> Message-ID: <1f7befae0410041429dfb729b@mail.gmail.com> [Paul Moore] ... > So why do existing extensions not do this? Things like wxPython > and Twisted seem to jump through a lot of hoops for little benefit > beyond allowing users to omit the .py extension... ... > Of course, this will enrage Unix users who dislike file extensions :-) Bingo -- that's why. There's no other reason. ... > (I recall something like this being discussed on the distutils-sig a > while back. Yes. > Maybe this discussion should move there.) You're batting 1000 <wink>. From dubnerm-news at mail.ru Mon Oct 4 23:44:58 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Mon Oct 4 23:44:33 2004 Subject: [Python-Dev] Guidelines for Logging Usage Message-ID: <4161C45A.4050205@mail.ru> Hello, I beg your pardon in advance for my English, which isn't my native language. I'm posting PEProposal for discussion. Last part is reasoning for posting it in this hot time. I'm ready to implement it all by myself... :-) -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English -------------- next part -------------- PEP: XXX Title: Guidelines for Logging Usage Version: $Revision: $ Last-Modified: $Date: $ Author: Michael P. Dubner <dubnerm@mindless.com> Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Oct-2004 Python-Version: 2.5 Post-History: Abstract This PEP defines guidelines for using logging system (PEP 282 [1]) in standard library. Implementing this PEP will simplify development of daemon applications. As a downside this PEP requires to slightly modify (however in backportable way) large number of standard modules. After implementing this PEP one can use following filtering scheme:: logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) Rationale There are couple of situations when output to stdout or stderr is very incomfortable. - Daemon application where framework doesn't allows to redirect standard output to some file, but assumes use of some other way of logging. Examples are syslog under *nix'es and EventLog under WinNT+. - GUI application which want to output every new log entry in separate popup window (i.e. fading OSD). Also sometimes application want to filter output enties based on it's source or severity. This requirement can't be implemented using simple redirection. At last sometimes output need to be marked with time of event, which can be acqured with logging system with ease. Proposal Every module usable for daemon and GUI applications should be rewritten to use logging system instead of 'print' or 'sys.stdout.write'. There should be code like this included in the beginning of every modified module:: import logging _log = logging.getLogger('stdlib.<module-name>') Prefix of 'stdlib.' must be used by all modules included in standard library distributed along with Python, and only by such modules (unverifiable). Using of "_log" is intentional as we don't want to auto-export it. For modules that using log only in one class logger can be created inside class definition as following:: class XXX: __log = logging.getLogger('stdlib.<module-name>') Then this class can create access methods to log to this private logger. So print's and "sys.std{out|err}.write" should be replaced with "_log.{debug|info}" and "traceback.print_exception" with "_log.exception" or sometimes "_log.debug('...',exc_info=1)". Module List Here is (possibly incomplete) list of modules to be reworked: - asyncore (dispatcher.log, dispatcher.log_info) - BaseHTTPServer (BaseHTTPRequestHandler.log_request, BaseHTTPRequestHandler.log_error, BaseHTTPRequestHandler.log_message) - cgi (possibly - is cgi.log used by somebody?) - ftplib (if FTP.debugging) - gopherlib (get_directory) - httplib (HTTPResponse, HTTPConnection) - ihooks (_Verbose) - imaplib (IMAP4._mesg) - mhlib (MH.error) - nntplib (NNTP) - pipes (Template.makepipeline) - pkgutil (extend_path) - platform (_syscmd_ver) - poplib (if POP3._debugging) - profile (if Profile.verbose) - robotparser (_debug) - smtplib (if SGMLParser.verbose) - shlex (if shlex.debug) - smtpd (SMTPChannel/PureProxy where print >> DEBUGSTREAM) - smtplib (if SMTP.debuglevel) - SocketServer (BaseServer.handle_error) - telnetlib (if Telnet.debuglevel) - threading? (_Verbose._note, Thread.__bootstrap) - timeit (Timer.print_exc) - trace - uu (decode) Additionaly there are couple of modules with commented debug output or modules where debug output should be added. For example: - urllib At last possibly some modules should be extended to provide more debug information. Doubtful Modules Here should be placed modules that community will propose for addition to module list and modules that community say should be removed from module list. - tabnanny (check) Guidelines for Logging Usage Also we can provide some recommendation to authors of library modules so they all follow same format of naming loggers. I propose that non-standard library modules should use logger named after their full names, so module "spam" in sub-package "junk" of package "dummy" will be named "dummy.junk.spam" and, of cause, "__init__" module of same package will have logger "dummy.junk". Implementation Schedule Proposal As one can see whole bunch of changes required to fullfil this proposal is rather large. I propose to delay these changes until after 2.4 release, and change only modules critical for server applications: - BaseHTTPServer - ftplib - httplib - poplib - smtpd - smtplib - SocketServer This can be done rather fast and reliable. References [1] PEP 282, A Logging System, Vinay Sajip, Trent Mick http://www.python.org/peps/pep-0282.html Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From ncoghlan at email.com Mon Oct 4 23:51:08 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Mon Oct 4 23:51:14 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <e443ad0e0410041418165fd900@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <1096924042.4161bb8a0a35a@mail.iinet.net.au> <e443ad0e0410041418165fd900@mail.gmail.com> Message-ID: <1096926668.4161c5cc9c49e@mail.iinet.net.au> Quoting Paramjit Oberoi <psoberoi@gmail.com>: > With a slightly longer script, and a little work in the installer, > these could be written as: > > Installed: runpy.py profile demo.py > Prompt After: runpy.py -i profile demo.py > Alt install: runpy2.4.py profile demo.py > Build dir: ./runpy.py profile demo.py > - or - ./scripts/runpy.py profile demo.py Indeed, it may be possible to get those to *run*, but the problem is that the semantics of the following two commands are likely to differ: python <interpreter-options> runpy.py module <script-args> python runpy.py <interpreter-options> module <script-args And getting the semantics close enough that the remaining differences don't matter is likely to be a pain. Certainly, you won't be able to use execfile() any more, because it doesn't support passing compiler flags. And the script doesn't have to get much more complicated before it becomes _harder_ to understand and maintain than the C code required to implement '-m' (which is really pretty straightforward when it is limited to top-level modules). Cheers, Nick. -- Nick Coghlan Brisbane, Australia From gvanrossum at gmail.com Tue Oct 5 00:23:33 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 5 00:23:42 2004 Subject: '-m' option (was RE: [Python-Dev] ConfigParser patches) In-Reply-To: <1096926668.4161c5cc9c49e@mail.iinet.net.au> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <1096924042.4161bb8a0a35a@mail.iinet.net.au> <e443ad0e0410041418165fd900@mail.gmail.com> <1096926668.4161c5cc9c49e@mail.iinet.net.au> Message-ID: <ca471dc204100415232acf8b06@mail.gmail.com> I'm going to drop out of this discussion; I'm still thinking this is just code bloat, but I won't stop you from adding it anyway. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From exarkun at divmod.com Tue Oct 5 00:39:24 2004 From: exarkun at divmod.com (exarkun@divmod.com) Date: Tue Oct 5 00:39:29 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <uis9qcktv.fsf@yahoo.co.uk> Message-ID: <20041004223924.29723.995823090.divmod.quotient.9779@ohm> On Mon, 04 Oct 2004 21:03:24 +0100, Paul Moore <p.f.moore@gmail.com> wrote: > [snip] > So why do existing extensions not do this? Things like wxPython > and Twisted seem to jump through a lot of hoops for little benefit > beyond allowing users to omit the .py extension... Twisted jumps through some hoops to let developers run uninstalled versions. The lack of .py extensions is incidental and mostly unimportant. It also does things differently on UNIX than on Windows (it sets things up on Windows so uses can just type "mktap", "twistd", etc (as opposed to path\to\mktap, not as opposed to mktap.py)). Jp From garthk at gmail.com Tue Oct 5 03:48:01 2004 From: garthk at gmail.com (Garth T Kidd) Date: Tue Oct 5 03:48:04 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <uis9qcktv.fsf@yahoo.co.uk> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> <uis9qcktv.fsf@yahoo.co.uk> Message-ID: <8aeed5d0041004184865e697ff@mail.gmail.com> >>> - Change the Windows Python binary to add .py (and .pyw, maybe) to >>> PATHEXT >> >> I don't like that. The Python Windows installer doesn't add, or >> modify, any environment variables now. It's anti-social to muck with >> them. Half the software on my workstation has changed environment variables, social or not. To me, it's less social to leave me to manually change PATHEXT. I'd much rather have Python add its extensions to PATHEXT -- and also have distutils add .py to the extension of Python scripts dropped into the Scripts directory. I used to be a control freak, but now I'm a control freak with kids, and the utter lack of spare time really changes one's perspective on such things. :) Regards, Garth. From paul at pfdubois.com Tue Oct 5 04:02:19 2004 From: paul at pfdubois.com (Paul F. Dubois) Date: Tue Oct 5 04:02:24 2004 Subject: [Python-Dev] grant applications Message-ID: <416200AB.6000006@pfdubois.com> Dear devs, You'll be pleased to learn that the PSF grants process brought in sixty-four proposals. I think that is a stunning success, and the wide variety of these proposals is a testimony to the breadth and strength of the community. We should seek some branding sponsorship in the future so we can accept more of these. I'm thinking along the likes of the "Kraft Cottage Cheese Grant". (:-> Picture of smiling Guido enjoying the product at his terminal. From tim.peters at gmail.com Tue Oct 5 04:08:12 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 5 04:08:21 2004 Subject: [Python-Dev] Re: Utility scripts In-Reply-To: <8aeed5d0041004184865e697ff@mail.gmail.com> References: <000301c4a9e0$4eb575e0$6633c797@oemcomputer> <1096890210.4161376227c2d@mail.iinet.net.au> <ca471dc2041004080589d870@mail.gmail.com> <79990c6b04100411157d14cf43@mail.gmail.com> <1f7befae04100411492fc43961@mail.gmail.com> <uis9qcktv.fsf@yahoo.co.uk> <8aeed5d0041004184865e697ff@mail.gmail.com> Message-ID: <1f7befae041004190839b5921f@mail.gmail.com> [Garth T Kidd] > Half the software on my workstation has changed environment variables, > social or not. To me, it's less social to leave me to manually change > PATHEXT. I'd much rather have Python add its extensions to PATHEXT -- > and also have distutils add .py to the extension of Python scripts > dropped into the Scripts directory. So contribute installer code to do so -- and don't forget to make the uninstaller smart enough to remove them again, without damaging what other installers may have added in the meantime. Ah, you don't want it enough to do anything to get it <wink>. That's OK. What good would it do you if it were there? As was made plain in the rest of this thread, PATHEXT does nothing except on cmd.exe systems. On cmd.exe systems, it only allows to leave the trailing ".py" off, and since cmd.exe's path completion fills in the ".py" for you, you then have to manually delete the ".py" in order to get a benefit from PATHEXT. Unless you add directories containing Python scripts to your PATH envar too -- in which case you can't credibly claim that adding .py to PATHEXT is beyond your abilities or time. Adding ".py" to scripts is a different issue, but that belongs on the distutils list. From dubnerm-news at mail.ru Tue Oct 5 08:18:18 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Tue Oct 5 08:17:24 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <8aeed5d0041004191553da8d0@mail.gmail.com> References: <4161C45A.4050205@mail.ru> <8aeed5d0041004191553da8d0@mail.gmail.com> Message-ID: <41623CAA.9070207@mail.ru> Garth T Kidd wrote: >Strikes me that not much work could let someone keep all of their >existing print statements more or less intact and use the logging >module properly, too: > > import logging > > class LoggingHandle: > def __init__(self, wrappedmethod): > self.wrappedmethod = wrappedmethod > def write(self, msg): > msg = msg.rstrip() > if msg: > self.wrappedmethod(msg.rstrip()) > > logging.basicConfig() > log = logging.getLogger('whatever') > stderr = LoggingHandle(log.error) > stdout = LoggingHandle(log.info) > > print >> stdout, "This is informational." # logged as INFO > print >> stderr, "This is an error." # guess. > >Add some basic infrastructure to logging, and you're done. The >Extremely Brave could even alter sys.stderr and sys.stdout. > > Main purpose of logging module is not redirection, but filtering capability. -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From vinay_sajip at red-dove.com Tue Oct 5 15:03:07 2004 From: vinay_sajip at red-dove.com (Vinay Sajip at Red Dove) Date: Tue Oct 5 15:03:26 2004 Subject: [Python-Dev] Guidelines for Logging Usage Message-ID: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> [Michael Dubner] > Main purpose of logging module is not redirection, but filtering capability. I don't agree. In general, both the redirection capability (targeting events at a particular audience) and the filtering capability (verbosity control) are important. Regards, Vinay Sajip From dubnerm-news at mail.ru Tue Oct 5 23:48:37 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Tue Oct 5 23:46:46 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> References: <000501c4aadb$ae9f1f00$0400a8c0@DELTA> Message-ID: <416316B5.3080905@mail.ru> Vinay Sajip at Red Dove wrote: >[Michael Dubner] > > >>Main purpose of logging module is not redirection, but filtering >> >> >capability. > >I don't agree. In general, both the redirection capability (targeting events >at a particular audience) and the filtering capability (verbosity control) >are important. > > I mean "purpose of logging module" only in content of PEP. -- Best regards, Michael Dubner (dubnerm@mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From garthk at gmail.com Wed Oct 6 00:28:13 2004 From: garthk at gmail.com (Garth T Kidd) Date: Wed Oct 6 00:28:24 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <41623CAA.9070207@mail.ru> References: <4161C45A.4050205@mail.ru> <8aeed5d0041004191553da8d0@mail.gmail.com> <41623CAA.9070207@mail.ru> Message-ID: <8aeed5d004100515283cf64a93@mail.gmail.com> It since occurred to me that people might want the standard library to be the canonical example of Good Python rather than some messy kludge. :) So, replacing print with appropriate direct calls to logging or warnings would be more appropriate than hooking print into those frameworks by futzing around with handles. Speaking of which: any thoughts on having warnings use logging rather than print? I'm showing 224 files in Lib\ without 'test' in the name and with print statements, though some of those are sensible (like zipfile.ZipFile.printdir) and some look like internal tests. Only cookielib, _LWPCookieJar.py and _MozillaCookieJar seem to import the logging module. So, you've got some real work on your hands updating everything to use logging. Cookie, cookielib, distutils, email, filecmp, idlelib, pickle, plat-mac, posixfile, pydoc, random, reconvert, regsub, shelve, statcache, tempfile, tzparse, whrandom and xmllib all import warnings. I'm not sure if it's appropriate to timestamp the root logger going to console by default; WARNING:module:message is messy enough without a timestamp. Regards, Garth. On Tue, 05 Oct 2004 10:18:18 +0400, Michael P. Dubner <dubnerm-news@mail.ru> wrote: > > > Garth T Kidd wrote: > > >Strikes me that not much work could let someone keep all of their > >existing print statements more or less intact and use the logging > >module properly, too: > > > > import logging > > > > class LoggingHandle: > > def __init__(self, wrappedmethod): > > self.wrappedmethod = wrappedmethod > > def write(self, msg): > > msg = msg.rstrip() > > if msg: > > self.wrappedmethod(msg.rstrip()) > > > > logging.basicConfig() > > log = logging.getLogger('whatever') > > stderr = LoggingHandle(log.error) > > stdout = LoggingHandle(log.info) > > > > print >> stdout, "This is informational." # logged as INFO > > print >> stderr, "This is an error." # guess. > > > >Add some basic infrastructure to logging, and you're done. The > >Extremely Brave could even alter sys.stderr and sys.stdout. > > Main purpose of logging module is not redirection, but filtering capability. From jcarlson at uci.edu Sun Oct 3 03:08:57 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 6 03:48:56 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <415F5129.3040006@uci.edu> Good day everyone, I have produced a patch against the latest CVS to add support for two new formatting characters in the struct module. It is currently an RFE, which I include a link to at the end of this post. Please read the email before you respond to it. Generally, the struct module is for packing and unpacking of binary data. It includes support to pack and unpack the c types: byte, char, short, long, long long, char[], *, and certain variants of those (signed/unsigned, big/little endian, etc.) Purpose ------- I had proposed two new formatting characters, 'g' and 'G' (for biGint or lonG int). There was one primary purpose, to offer users the opportunity to specify their own integer lengths (very useful for cryptography, and real-world applications that involve non-standard sized integers). Current solutions involve shifting, masking, and multiple passes over data. There is a secondary purpose, and that is that future n-byte integers (like 16-byte/128-bit integers as supported by SSE2) are already taken care of. It also places packing and unpacking of these larger integers in the same module as packing and packing of other integers, floats, etc. This makes documentation easy. Functionality-wise, it merely uses the two C functions _PyLong_FromByteArray() and _PyLong_ToByteArray(), with a few lines to handle interfacing with the pack and unpack functions in the struct module. An example of use is as follows: >>> struct.pack('>3g', -1) '\xff\xff\xff' >>> struct.pack('>3g', 2**23-1) '\x7f\xff\xff' >>> struct.pack('>3g', 2**23) Traceback (most recent call last): File "<stdin>", line 1, in ? OverflowError: long too big to convert >>> struct.pack('>3G', 2**23) '\x80\x00\x00' It follows the struct module standard 'lowercase for signed, uppercase for unsigned'. Arguments --------- There seem to be a few arguments against its inclusion into structmodule.c... Argument: The size specifier is variable, so you must know the size/magnitude of the thing you are (un)packing before you (un)pack it. My Response: All use cases I have for this particular mechanism involve not using 'variable' sized structs, but fixed structs with integers of non-standard byte-widths. Specifically, I have a project in which I use some 3 and 5 byte unsigned integers. One of my (un)pack format specifiers is '>H3G3G', and another is '>3G5G' (I have others, but these are the most concise). Certainly this does not fit the pickle/cPickle long (un)packing use-case, but that problem relies on truely variable long integer lengths, of which this specifier does not seek to solve. Really, the proposed 'g' and 'G' format specifiers are only as variable as the previously existing 's' format specifier. Argument: The new specifiers are not standard C types. My Response: Certainly they are not standard C types, but they are flexible enough to subsume all current integer C type specifiers. The point was to allow a user to have the option of specifying their own integer lengths. This supports use cases involving certain kinds of large dataset processing (my use case, which I may discuss after we release) and cryptography, specifically in the case of PKC... while 1: blk = get_block() iblk = struct.unpack('>128G', blk)[0] uiblk = pow(iblk, power, modulous) write_block(struct.pack('>128G', uiblk)) The 'p' format specifier is also not a standard C type, and yet it is included in struct, specifically because it is useful. Argument: You can already do the same thing with: pickle.encode_long(long_int) pickle.decode_long(packed_long) and some likely soon-to-be included additions to the binascii module. My Response: That is not the same. Nontrivial problems require multiple passes over your data with multiple calls. A simple: struct.unpack('H3G3G', st) becomes: pickle.decode_long(st[:2]) #or an equivalent struct call pickle.decode_long(st[2:5]) pickle.decode_long(st[5:8]) And has no endian or sign options, or requires the status quo using of masks and shifts to get the job done. As previously stated, one point of the module is to reduce the amount of bit shifting and masking required. Argument: We could just document a method for packing/unpacking these kinds of things in the struct module, if this really is where people would look for such a thing. My Response: I am not disputing that there are other methods of doing this, I am saying that the struct module includes a framework and documentation location that can include this particular modification with little issue, which is far better than any other proposed location for equivalent functionality. Note that functionality equivalent to pickle.encode/decode_long is NOT what this proposed enhancement is for. Argument: The struct module has a steep learning curve already, and this new format specifier doesn't help it. My Response: I can't see how a new format specifier would necessarily make the learning curve any more difficult, if it was even difficult in the first place. Why am I even posting --------------------- Raymond has threatened to close this RFE due to the fact that only I have been posting to state that I would find such an addition useful. If you believe this functionality is useful, or even if you think that I am full of it, tell us: http://python.org/sf/1023290 - Josiah From jcarlson at uci.edu Wed Oct 6 04:09:47 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 6 04:17:27 2004 Subject: [Python-Dev] proposed struct module format code addition Message-ID: <20041005190838.ECF7.JCARLSON@uci.edu> If the proposed struct module post seemed to have been a dupe, it is because it sat in a moderator queue for a few days. Sorry about that. - Josiah From kbk at shore.net Wed Oct 6 06:02:47 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Oct 6 06:02:53 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410060402.i9642lO6022228@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 241 open ( +6) / 2640 closed ( +3) / 2881 total ( +9) Bugs : 766 open ( -2) / 4492 closed (+12) / 5258 total (+10) RFE : 154 open ( +2) / 131 closed ( +0) / 285 total ( +2) New / Reopened Patches ______________________ ftplib PASV error bug (2004-09-30) http://python.org/sf/1037516 opened by Tim Nelson urllib2 HTTP digest authentication fix (2004-09-30) http://python.org/sf/1037974 opened by Mathieu Fenniak Provide tuple of "special" exceptions (2004-10-01) http://python.org/sf/1038256 opened by Nick Coghlan __main__ for whichdb.py (2004-10-01) http://python.org/sf/1038388 opened by Oleg Broytmann Fix struct.pack on 64-bit archs (broken on 2.*) (2004-10-02) http://python.org/sf/1038854 opened by Stefan Grundmann pydoc method documentation lookup enhancement (2004-10-02) http://python.org/sf/1038909 opened by Alexander Schmolck OS X: Can't use #!/usr/bin/pythonw (2004-10-02) http://python.org/sf/1038911 opened by Nathan Gray repair typos (2004-10-02) CLOSED http://python.org/sf/1038917 opened by George Yoshida SimpleXMLRPCServer optional allow_none / encoding arguments (2004-10-02) http://python.org/sf/1039083 opened by Guillermo M. Narvaja Patches Closed ______________ repair typos (2004-10-02) http://python.org/sf/1038917 closed by rhettinger doctest: allow custom matchers for testing if got=want (2004-04-10) http://python.org/sf/932935 closed by edloper doctest: Add Tester params to DocTestSuite (2004-04-10) http://python.org/sf/932932 closed by edloper New / Reopened Bugs ___________________ len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) CLOSED http://python.org/sf/1037373 opened by Nick Coghlan textReader: reading after close is a seg fault (2004-09-30) http://python.org/sf/1038013 opened by Jyrki Alakuijala silent installation on windows: no drive in registry values (2004-10-01) CLOSED http://python.org/sf/1038410 opened by Matteo Gallivanoni Python 2.3+ socket._fileobject handles EAGAIN with data loss (2004-10-01) http://python.org/sf/1038591 opened by Jon Nelson bad link in Tkinter docs (2004-10-01) http://python.org/sf/1038693 opened by Brett Cannon Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) CLOSED http://python.org/sf/1038935 opened by Malte Helmert apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) CLOSED http://python.org/sf/1039092 opened by Jp Calderone time zone tests fail on Windows (2004-10-03) CLOSED http://python.org/sf/1039270 opened by George Yoshida os.times() is bogus (2004-10-04) http://python.org/sf/1040026 opened by Guido van Rossum Missing documentation on How To Link With libpython (2004-10-05) http://python.org/sf/1040439 opened by Sir Raorn Bugs Closed ___________ Odd behavior with unicode.translate on OSX. (2004-09-22) http://python.org/sf/1032615 closed by doerwalter Can't inherit slots from new-style classes implemented in C (2004-09-24) http://python.org/sf/1034178 closed by philthompson10 len(str(x)) in decimal.py is not Python 2.3 compatible (2004-09-30) http://python.org/sf/1037373 closed by ncoghlan printf() in dynload_shlib.c should be PySys_WriteStderr (2004-09-29) http://python.org/sf/1036752 closed by loewis hex() and oct() documentation is incorrect (2004-09-27) http://python.org/sf/1035279 closed by rhettinger silent installation on windows: no drive in registry values (2004-10-01) http://python.org/sf/1038410 closed by theller Typo "signiture" in HTML generated by DocXMLRPCServer (2004-10-02) http://python.org/sf/1038935 closed by bcannon apply.__doc__ says "Deprecated since release 2.3" (2004-10-02) http://python.org/sf/1039092 closed by rhettinger Message.get_filename() problem (2003-12-04) http://python.org/sf/854102 closed by bwarsaw time zone tests fail on Windows (2004-10-02) http://python.org/sf/1039270 closed by bcannon Case sensitivity bug in ConfigParser (2004-08-27) http://python.org/sf/1017864 closed by goodger ConfigParser behavior change (2004-07-24) http://python.org/sf/997050 closed by goodger New / Reopened RFE __________________ win32 silent installation: allow MAINDIR on command line (2004-10-04) http://python.org/sf/1039857 opened by Matteo Gallivanoni os.run function for convinience and robustness (2004-10-04) http://python.org/sf/1040267 opened by Alexander Schmolck From amk at amk.ca Wed Oct 6 21:57:05 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Oct 6 21:58:51 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <4161C45A.4050205@mail.ru> References: <4161C45A.4050205@mail.ru> Message-ID: <20041006195705.GB17032@rogue.amk.ca> On Tue, Oct 05, 2004 at 01:44:58AM +0400, Michael P. Dubner wrote: > I'm posting PEProposal for discussion. Last part is reasoning for > posting it in this hot time. I like this proposal; it begins to impose some degree of sanity on the standard library. (If you want editing help with the grammar and text of the PEP, let me know; I'll happily polish the wording.) > logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) There's been discussion in the past of having a name for the Python standard library, so that 'from <name> import httplib' would always get the standard httplib module. This name should match whatever's using in the logging, so the logging should use a name everyone is happy with using in import statements. Whether that's stdlib or std or __std__ or Lib isn't important, as long as the logging matches. PEP 8 should also gain some text encouraging the use of the logging module in code and describing the dummy.junk convention. --amk From barry at python.org Wed Oct 6 22:16:22 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 6 22:16:30 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <1097093782.20140.39.camel@geddy.wooz.org> On Wed, 2004-10-06 at 15:57, A.M. Kuchling wrote: > There's been discussion in the past of having a name for the Python > standard library, so that 'from <name> import httplib' would always > get the standard httplib module. This name should match whatever's > using in the logging, so the logging should use a name everyone is > happy with using in import statements. Whether that's stdlib or std > or __std__ or Lib isn't important, as long as the logging matches. +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041006/84bdc3bb/attachment.pgp From ncoghlan at email.com Thu Oct 7 00:18:39 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Thu Oct 7 00:18:43 2004 Subject: [Python-Dev] Generator expression in _strptime.py TimeRE.__init__ Message-ID: <1097101119.41646f3f1177a@mail.iinet.net.au> This is currently breaking the tests for me, as [:] doesn't work for generator expressions. Either TimeRE.__init__ needs to go back to using a list comp, or it needs to use list() in __seqToRE instead of [:]. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From bac at OCF.Berkeley.EDU Thu Oct 7 00:51:08 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Oct 7 00:51:20 2004 Subject: [Python-Dev] Generator expression in _strptime.py TimeRE.__init__ In-Reply-To: <1097101119.41646f3f1177a@mail.iinet.net.au> References: <1097101119.41646f3f1177a@mail.iinet.net.au> Message-ID: <416476DC.90808@ocf.berkeley.edu> Nick Coghlan wrote: > This is currently breaking the tests for me, as [:] doesn't work for generator > expressions. > > Either TimeRE.__init__ needs to go back to using a list comp, or it needs to use > list() in __seqToRE instead of [:]. > Damn, sorry everyone. Fixed now. I would have sworn that I ran the regression tests before I did the commit, but I guess I didn't between the last commit and the previous one. But it is fixed now. -Brett From pythondev-dang at lazytwinacres.net Thu Oct 7 01:48:59 2004 From: pythondev-dang at lazytwinacres.net (Daniel 'Dang' Griffith) Date: Thu Oct 7 01:49:04 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <20041003034331.46FB91E400A@bag.python.org> References: <20041003034331.46FB91E400A@bag.python.org> Message-ID: <4164846B.3040205@lazytwinacres.net> At 11:43 PM 10/2/2004, python-dev-request@python.org wrote: > Date: Sat, 2 Oct 2004 21:14:07 -0400 > From: "Raymond Hettinger" <python@rcn.com> > Subject: RE: [Python-Dev] ConfigParser patches > To: <guido@python.org> > Cc: python-dev@python.org > Message-ID: <000801c4a8e6$6217fc20$e841fea9@oemcomputer> > Content-Type: text/plain; charset="us-ascii" > > > I was in fact thinking of the -m proposal when I wrote that... > > The timing does suck. > > > It's so easy to define an alias or use a one-line shell script for > > invoking Python with a full pathname that I'm really not sure I like > > the -m idea, > > My understanding was that it wasn't about a full pathname to python, it > was about searching sys.path for the darned script so us poor Windows > guys don't have to change directories a million times a day (no aliases > or shebangs for us either). > > I have a big pile of batch files just to invoke timeit, texchecker, > profile, etc. It bites the big one. Am I missing something? I run python scripts all the time from the command line in Windows. Modify your PATHEXT environment variable: PATHEXT=.PY;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH Then put your script directory into your PATH: PATH=C:\Python23\lib;%PATH% The first time you run timeit (or whatever.py), Windows will prompt you to associate the .py extension with a program. Pick your favorite python.exe. You can also do this manually before running your script. From then on, you're set: C:\>timeit -s "a=range(20)" "a.append(1)" "a.pop()" 1000000 loops, best of 3: 1.13 usec per loop --dang p.s. You can put .pyc in your path too, but then you'll get the compiled version even if the uncompiled version is newer. But you can always compile them. From pje at telecommunity.com Thu Oct 7 01:51:21 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Oct 7 01:51:07 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? Message-ID: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> I was looking up something in the distutils source today, and I noticed that there are some new, undocumented, and not quite complete features that were added earlier this year. Specifically, the 'requires' and 'provides' arguments to 'setup()', and the 'checkdep' command. Should these be removed for 2.4? It doesn't seem likely they'd be able to be fleshed out before the beta, especially since as far as I can recall they haven't been discussed on the Distutils-SIG yet, and some of the features don't seem to be in alignment with the relevant PEPs (e.g. PEP 262's definition of "requires" and "provides"). And, the actual current implementation of the dependency checking is (per the author's comments) "kinda hacky". From fdrake at acm.org Thu Oct 7 03:40:34 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Oct 7 03:40:47 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? In-Reply-To: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> References: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> Message-ID: <200410062140.34725.fdrake@acm.org> On Wednesday 06 October 2004 07:51 pm, Phillip J. Eby wrote: > I was looking up something in the distutils source today, and I noticed > that there are some new, undocumented, and not quite complete features > that were added earlier this year. Specifically, the 'requires' and > 'provides' arguments to 'setup()', and the 'checkdep' command. Anthony and I threw those together pretty quickly at PyCon; it's not clear that they're useful. For Zope Corp., they won't be. We've decided that we will not be adding __version__ assignments into the codebase, because there lies a stupid maintenance hassle. I have no intention of documenting that stuff myself. > Should these be removed for 2.4? Yes. > It doesn't seem likely they'd be able to > be fleshed out before the beta, especially since as far as I can recall > they haven't been discussed on the Distutils-SIG yet, and some of the Frankly, I have no confidence that we'll ever end up with an agreement on something that's sufficient and still easy to use. Versioning simply isn't any easy aspect of packaging. > features don't seem to be in alignment with the relevant PEPs (e.g. PEP > 262's definition of "requires" and "provides"). And, the actual current > implementation of the dependency checking is (per the author's comments) > "kinda hacky". You're being generous. ;-) It's unfortunate that PEP 262 was deferred, but that's what happens when no one has time to work on the beast. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From garthk at gmail.com Thu Oct 7 04:08:43 2004 From: garthk at gmail.com (Garth T Kidd) Date: Thu Oct 7 04:08:46 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <8aeed5d004100619083a1b8bcf@mail.gmail.com> > PEP 8 should also gain some text encouraging the use of the logging > module in code and describing the dummy.junk convention. Index: pep-0008.txt =================================================================== RCS file: /cvsroot/python/python/nondist/peps/pep-0008.txt,v retrieving revision 1.25 diff -c -r1.25 pep-0008.txt *** pep-0008.txt 6 Aug 2004 18:47:26 -0000 1.25 --- pep-0008.txt 7 Oct 2004 02:07:02 -0000 *************** *** 600,605 **** --- 600,608 ---- No: if greeting == True: Yes: if greeting: + - Use the print statement only where the user's objective is + to print. Use the logging method to deliver status information + to users on the fly. For more information, see PEP xxxx [6]. References *************** *** 614,619 **** --- 617,624 ---- [5] Barry's GNU Mailman style guide http://barry.warsaw.us/software/STYLEGUIDE.txt + [6] PEP xxxx, Guidelines for Logging Usage + Copyright From anthony at interlink.com.au Thu Oct 7 05:11:24 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Oct 7 05:11:33 2004 Subject: [Python-Dev] Status of undocumented distutils features in 2.4? In-Reply-To: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> References: <5.1.1.6.0.20041006192638.02de26d0@mail.telecommunity.com> Message-ID: <4164B3DC.1040804@interlink.com.au> Phillip J. Eby wrote: > I was looking up something in the distutils source today, and I noticed > that there are some new, undocumented, and not quite complete features > that were added earlier this year. Specifically, the 'requires' and > 'provides' arguments to 'setup()', and the 'checkdep' command. They were something we worked on at PyCon. In hindsight, they're probably not the right way to go about it, and should be removed. If you want to remove them, great!, otherwise I'll do it in the next day or so. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From mwh at python.net Thu Oct 7 12:16:24 2004 From: mwh at python.net (Michael Hudson) Date: Thu Oct 7 12:16:25 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Doc/tut tut.tex, 1.253, 1.254 In-Reply-To: <E1CFS3D-00078a-Qb@sc8-pr-cvs1.sourceforge.net> (rhettinger@users.sourceforge.net's message of "Wed, 06 Oct 2004 23:46:55 -0700") References: <E1CFS3D-00078a-Qb@sc8-pr-cvs1.sourceforge.net> Message-ID: <2mlleikf47.fsf@starship.python.net> rhettinger@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Doc/tut > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv27294/Doc/tut > > Modified Files: > tut.tex > Log Message: > SF patch #1035498: -m option to run a module as a script > (Contributed by Nick Coghlan.) Does anyone know how to edit Misc/python.man to reflect this change? Heck, does anyone know enough Nroff or whatever to check that python.man is even vaguely up to date? I see it mentions -E, so it can't be *that* stale... Cheers, mwh -- <xyld> what proportion of Swedes speak english? <bea> xyld: all those with teeth From amk at amk.ca Thu Oct 7 14:40:32 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 7 14:42:19 2004 Subject: [Python-Dev] Man pages In-Reply-To: <2mlleikf47.fsf@starship.python.net> References: <E1CFS3D-00078a-Qb@sc8-pr-cvs1.sourceforge.net> <2mlleikf47.fsf@starship.python.net> Message-ID: <20041007124032.GA20713@rogue.amk.ca> On Thu, Oct 07, 2004 at 11:16:24AM +0100, Michael Hudson wrote: > Does anyone know how to edit Misc/python.man to reflect this change? Done. The man page seems up to date; it even lists a few environment variables that 'python -h' doesn't. This reminds me: the Debian packaging of Python includes a man page for pydoc, written by Moshe Zadka; I think there's another added man page or two. It would be nice to include these pages with the main Python release, but I can't figure out what license Debian's additions are under. Is anyone here from the debian-python mailing list who wants to pursue this? --amk From amk at amk.ca Thu Oct 7 18:31:56 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 7 18:34:11 2004 Subject: [Python-Dev] Version # for PSF license? Message-ID: <20041007163156.GB21908@rogue.amk.ca> DOAP (http://usefulinc.com/doap) is yet another schema for recording data about software projects. I'd like to add support for it to PyPI; however, before doing that a URL needs to be allocated for the PSF license in the listing at http://usefulinc.com/doap/licenses . Question: does the PSF license have a version number of its own or do we have to say 'PSF license for Python 2.2' (2.3, 2.4, etc)? --amk From vegeta.z at gmail.com Thu Oct 7 19:25:20 2004 From: vegeta.z at gmail.com (daniel narf) Date: Thu Oct 7 19:31:28 2004 Subject: [Python-Dev] Toward Python's future article Message-ID: <loom.20041007T192434-234@post.gmane.org> Hi i am sure most of you have read the article of Andrew Kuchling about focusing more in the standart library than language newFeatures/tweaking and probably i as many others would like to know what the python hackers/developers think about this proposal.Maybe this post is out of place but oh well.. i am personaly very interested in improving the stdlib which is very messy in my opinion right now. the article(several comments): http://www.amk.ca/diary/archives/cat_python.html#003382 From python at rcn.com Fri Oct 8 01:55:03 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 8 01:56:30 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <loom.20041007T192434-234@post.gmane.org> Message-ID: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> > Hi i am sure most of you have read the article of Andrew Kuchling about > focusing > more in the standart library than language newFeatures/tweaking and > probably i > as many others would like to know what the python hackers/developers think > about > this proposal. There is a great deal of merit to efforts for cataloging resources and thinking about ways to improve the standard library. Whether successful or not likely depends on who wants to volunteer their time for what. Where there are multiple, competing third party solutions, Guido has historically resisted ending exploration by locking in a single solution. Instead, he said he prefers "category killers" such as the unittest and logging modules. Martin has also been clear that community acceptance and a commitment to on-going maintenance are also important considerations. For the core, python-dev discussions indicate that several things are still in the works and will probably happen depending on who has the time, interest, and ability: * Someday, decimal may become a built-in type. * The AST version of the compiler may get finished. * A mutable bytes type may emerge on the road to all strings being Unicode. * Someday, C99 may rule the roost and cmath will be updated. * One of three proposals may be accepted for optimized attribute lookup. * A bytecode verifier seems to have a chance. * reST support may be added when it becomes mature enough. * The project to transition to unittests and increase coverage is continuing. * If the class/instance semantics get worked out, exceptions may become new style classes along the road to Py3.0. FWIW, there is a trend toward providing pure python equivalents to CPython implementations (such as that for compiler, sets, bisect, heapq, deques, itertools, decimal, etc). The thought is that these may outlive their C counterparts. > i am personaly very interested in improving the stdlib which is very messy > in my > opinion right now. It's not clear whether you're volunteering or just making a vague blanket request/criticism. If it is the former, then the backlog of bug reports and patch reviews is a good place to start. Raymond From jepler at unpythonic.net Fri Oct 8 02:15:54 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Fri Oct 8 02:15:54 2004 Subject: [Python-Dev] Python Bug Day this month? Message-ID: <20041008001554.GC6737@unpythonic.net> I thought the Python Bug Day was going to return this month, and I was hoping to spare some of my time to help out (on one previous bug day I reviewed some doc patches), but I haven't seen anything about it. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041007/0f14311b/attachment.pgp From carribeiro at gmail.com Fri Oct 8 02:24:01 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Fri Oct 8 02:24:10 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> References: <loom.20041007T192434-234@post.gmane.org> <002301c4acc9$10a26a60$5bae2c81@oemcomputer> Message-ID: <864d370904100717246f51e6fa@mail.gmail.com> On Thu, 7 Oct 2004 19:55:03 -0400, Raymond Hettinger <python@rcn.com> wrote: > Where there are multiple, competing third party solutions, Guido has > historically resisted ending exploration by locking in a single > solution. Instead, he said he prefers "category killers" such as the > unittest and logging modules. Martin has also been clear that community > acceptance and a commitment to on-going maintenance are also important > considerations. I see the reasoning behind this position. But I fear it can lead to lack of standardization in some areas. There is at least one area where a firm position would greatly clarify things: standard interfaces for modules where multiple providers are always the norm, such as the DB API and WSGI. WSGI is working is way out, but it's still too early to tell if they will be successful. But the DB API could really take advantage of a BDFL-style intervention to settle things. (it does not *need* to be Guido, but it has to be someone with power enough to take a firm stand and enforce it; but I don't know anyone who could assume this role as the undisputed leader of the DB-API effort). > For the core, python-dev discussions indicate that several things are > still in the works and will probably happen depending on who has the > time, interest, and ability: > > * Someday, decimal may become a built-in type. > * The AST version of the compiler may get finished. > * A mutable bytes type may emerge on the road to all strings being > Unicode. > * Someday, C99 may rule the roost and cmath will be updated. > * One of three proposals may be accepted for optimized attribute lookup. > * A bytecode verifier seems to have a chance. > * reST support may be added when it becomes mature enough. > * The project to transition to unittests and increase coverage is > continuing. > * If the class/instance semantics get worked out, exceptions may become > new style classes along the road to Py3.0. Nice summary. > FWIW, there is a trend toward providing pure python equivalents to > CPython implementations (such as that for compiler, sets, bisect, heapq, > deques, itertools, decimal, etc). The thought is that these may outlive > their C counterparts. That's great -- I mean, taken to its maximum expression, Python would be able to bootstrap itself this way (a Python VM written in Python?). It's a sign of maturity, don't you think? > > i am personaly very interested in improving the stdlib which is very > messy > > in my > > opinion right now. > > It's not clear whether you're volunteering or just making a vague > blanket request/criticism. If it is the former, then the backlog of bug > reports and patch reviews is a good place to start. I still need to find my way there (docs/sourceforge stuff). Depending on the way my current projects go, I may still be able to dedicate some time to it. I'm really looking forward to it... -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From mal at egenix.com Fri Oct 8 10:54:34 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Oct 8 10:55:18 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <864d370904100717246f51e6fa@mail.gmail.com> References: <loom.20041007T192434-234@post.gmane.org> <002301c4acc9$10a26a60$5bae2c81@oemcomputer> <864d370904100717246f51e6fa@mail.gmail.com> Message-ID: <416655CA.1020006@egenix.com> Carlos Ribeiro wrote: > On Thu, 7 Oct 2004 19:55:03 -0400, Raymond Hettinger <python@rcn.com> wrote: > >>Where there are multiple, competing third party solutions, Guido has >>historically resisted ending exploration by locking in a single >>solution. Instead, he said he prefers "category killers" such as the >>unittest and logging modules. Martin has also been clear that community >>acceptance and a commitment to on-going maintenance are also important >>considerations. > > > I see the reasoning behind this position. But I fear it can lead to > lack of standardization in some areas. There is at least one area > where a firm position would greatly clarify things: standard > interfaces for modules where multiple providers are always the norm, > such as the DB API and WSGI. WSGI is working is way out, but it's > still too early to tell if they will be successful. But the DB API > could really take advantage of a BDFL-style intervention to settle > things. (it does not *need* to be Guido, but it has to be someone with > power enough to take a firm stand and enforce it; but I don't know > anyone who could assume this role as the undisputed leader of the > DB-API effort). I don't know anything about the way the WSGI specification is developed. The DB-API specification was developed outside the Python development process and has so far been very successful at that. The DB-SIG which manages the specification is in charge of the process and will continue to work together with module authors and users to accommodate for new features that may become generally available in databases or their interfaces. Note that there is no need to "settle things": the DB-API spec has been around for many years, is very stable, widely used and still flexible enough to meet requirements of various different database backends. If you feel that a certain feature is missing, I'd suggest you direct your constructive criticism to the db-sig@python.org. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From astrand at lysator.liu.se Fri Oct 8 13:07:02 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Fri Oct 8 13:07:13 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> Message-ID: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> [From Mail to python-announce list] > I'd like to give you an updated status report of the popen5 project. Since > my last post at 2004-06-04, this has happened: ... > With these changes, the subprocess module now feels quite complete. In the > near future, I will focus on getting the module included in the standard > library. I've recieved very positive feedback on my module. Many users are also asking me if this module will be included in the standard library. That is, of course, my wish as well. So, can the subprocess module be accepted? If not, what needs to be done? /Peter ?strand <astrand@lysator.liu.se> From walter at livinglogic.de Fri Oct 8 13:35:41 2004 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri Oct 8 13:35:44 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> References: <002301c4acc9$10a26a60$5bae2c81@oemcomputer> Message-ID: <41667B8D.8090406@livinglogic.de> Raymond Hettinger wrote: > [...] > * If the class/instance semantics get worked out, exceptions may become > new style classes along the road to Py3.0. So should we switch from a PendingDeprecationWarning to a DeprecationWaring for string exceptions in 2.4? Bye, Walter D?rwald From amk at amk.ca Fri Oct 8 14:19:47 2004 From: amk at amk.ca (A.M. Kuchling) Date: Fri Oct 8 14:21:38 2004 Subject: [Python-Dev] Python Bug Day this month? In-Reply-To: <20041008001554.GC6737@unpythonic.net> References: <20041008001554.GC6737@unpythonic.net> Message-ID: <20041008121947.GA27702@rogue.amk.ca> On Thu, Oct 07, 2004 at 07:15:54PM -0500, Jeff Epler wrote: > I thought the Python Bug Day was going to return this month, and I was > hoping to spare some of my time to help out (on one previous bug day I > reviewed some doc patches), but I haven't seen anything about it. I don't have time to do the organization for it or to spare an entire day for it. If someone else wants to make arrangements, feel free! --amk From theller at python.net Fri Oct 8 16:30:49 2004 From: theller at python.net (Thomas Heller) Date: Fri Oct 8 16:30:38 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> (Peter Astrand's message of "Fri, 8 Oct 2004 13:07:02 +0200 (MEST)") References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> Message-ID: <ekk91duu.fsf@python.net> Peter Astrand <astrand@lysator.liu.se> writes: > [From Mail to python-announce list] > >> I'd like to give you an updated status report of the popen5 project. Since >> my last post at 2004-06-04, this has happened: > ... >> With these changes, the subprocess module now feels quite complete. In the >> near future, I will focus on getting the module included in the standard >> library. > > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? I would suggest to replace the _subprocess extension module by a Python implementation based on ctypes (and include ctypes in 2.4, at least for Windows). How many _xxxx.pyd windows specific modules do we need? Thomas From carribeiro at gmail.com Fri Oct 8 17:46:57 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Fri Oct 8 17:47:00 2004 Subject: [Python-Dev] Toward Python's future article In-Reply-To: <416655CA.1020006@egenix.com> References: <loom.20041007T192434-234@post.gmane.org> <002301c4acc9$10a26a60$5bae2c81@oemcomputer> <864d370904100717246f51e6fa@mail.gmail.com> <416655CA.1020006@egenix.com> Message-ID: <864d3709041008084665e0956f@mail.gmail.com> Hello all, There must be something about the way I write my comments that may be making them sound worst than I want. I have _no_ intention to start a flame war... and I really would like to make it sound constructive. I apologize if it sounded otherwise. (I have a few comments that still belong here, if only to clarify things) On Fri, 08 Oct 2004 10:54:34 +0200, M.-A. Lemburg <mal@egenix.com> wrote: > Note that there is no need to "settle things": the DB-API > spec has been around for many years, is very stable, > widely used and still flexible enough to meet requirements > of various different database backends. There are several packages around that don't implement the DB API correctly, or implement a completely different API. This is clearly not a fault of the DB API itself, or of its developers, but it's an issue for its users. It's a difficult situation, because nobody is really in a position to enforce compliance. In this sense, bringing the issue to the standard library _could_ be helpful. One suggestion is to have a reference implementation -- for example, a simple DB API module, or a library that makes use of DB-API compliant modules. Of course, it's open to discussion whether this is feasible or not; for now it's only a personal opinion. > If you feel that a certain feature is missing, I'd suggest > you direct your constructive criticism to the db-sig@python.org. I think the DB API itself is fine. What is lacking is some way to make sure that everyone _comply_ to it. If that's can be done within the DB-SIG group alone, fine. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From theller at python.net Fri Oct 8 20:09:30 2004 From: theller at python.net (Thomas Heller) Date: Fri Oct 8 20:09:19 2004 Subject: [Python-Dev] HAVE_USABLE_WCHAR_T Message-ID: <mzyxytd1.fsf@python.net> The Include/unicodeobject.h file says (line 103): /* If the compiler provides a wchar_t type we try to support it through the interface functions PyUnicode_FromWideChar() and PyUnicode_AsWideChar(). */ This isn't true - grepping the CVS sources for this symbol shows that it is used in these ways: - When defined together with the WANT_WCTYPE_FUNCTIONS symbol, the compiler's wctype.h functions are used instead of the ones supplied with Python. Include/unicodeobject.h, line 294. - When defined together with MS_WINDOWS, it makes available mbcs_enocde and mbcs_decode functions (in Modules/_codecsmodule.c), plus the PyUnicode_DecodeMBCS and PyUnicode_AsMBCSString functions in Objects/unicodeobject.c. - Contrary to the comment at the top of this message, the PyUnicode_FromWideChar and PyUnicode_AsWideChar functions are compiled when HAVE_WCHAR_H is defined. The HAVE_USABLE_WCHAR_T symbol is only used to determine whether memcpy is used, or the unicode characters are copied one by one. - Finally, again when defined together with MS_WINDOWS, it sets the filesystem encoding to mbcs. So, it seems that the HAVE_USABLE_WCHAR_T symbol doesn't play any role for the extension programmer *at all*. The preprocessor flag that plays a role for extensions seem to be HAVE_WCHAR_H since they mark whether the PyUnicode_FromWideChar and PyUnicode_AsWideChar are available or not. This has caused me quite some confusion, and so I suggest the comment above in the Include/unicodeobject.h file should be fixed. Finally, the docs also seem to get it wrong (although I haven't followed that in detail). Can't reach python.org at the moment, but Python C/api manual, section 7.3.2, unicode objects says: Py_UNICODE This type represents a 16-bit unsigned storage type which is used by Python internally as basis for holding Unicode ordinals. On platforms where wchar_t is available and also has 16-bits, Py_UNICODE is a typedef alias for wchar_t to enhance native platform compatibility. On all other platforms, Py_UNICODE is a typedef alias for unsigned short. Isn't the size 32 bits for wide unicode builds? Please, please fix this - unicode is already complicated enough even without this confusion! Thomas From lunz at falooley.org Fri Oct 8 20:16:20 2004 From: lunz at falooley.org (Jason Lunz) Date: Fri Oct 8 20:16:24 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> Message-ID: <ck6lhj$k72$1@sea.gmane.org> astrand@lysator.liu.se said: > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. > That is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be > done? I've been watching the progression of subprocess with some interest. It looks encouraging, and is exactly the sort of thing I need for my work. One small nit I've noticed: aren't the names of subprocess.call() and subprocess.callv() reversed? If you look at unix execl() and execv(), execl() takes a variable-length argument list and execv() takes a list (vector?) of arguments. But it's the opposite for subprocess -- callv() takes a variable-length arg list and call() takes a list of args. Am I missing something? Can these be renamed now before it gets standardized? Jason From mal at egenix.com Fri Oct 8 21:30:11 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Oct 8 21:30:15 2004 Subject: [Python-Dev] HAVE_USABLE_WCHAR_T In-Reply-To: <mzyxytd1.fsf@python.net> References: <mzyxytd1.fsf@python.net> Message-ID: <4166EAC3.7080708@egenix.com> Thomas Heller wrote: > The Include/unicodeobject.h file says (line 103): > > /* If the compiler provides a wchar_t type we try to support it > through the interface functions PyUnicode_FromWideChar() and > PyUnicode_AsWideChar(). */ > > This isn't true - grepping the CVS sources for this symbol shows that it > is used in these ways: > > - When defined together with the WANT_WCTYPE_FUNCTIONS symbol, the > compiler's wctype.h functions are used instead of the ones supplied with > Python. Include/unicodeobject.h, line 294. > > - When defined together with MS_WINDOWS, it makes available mbcs_enocde > and mbcs_decode functions (in Modules/_codecsmodule.c), plus the > PyUnicode_DecodeMBCS and PyUnicode_AsMBCSString functions in > Objects/unicodeobject.c. > > - Contrary to the comment at the top of this message, the > PyUnicode_FromWideChar and PyUnicode_AsWideChar functions are compiled > when HAVE_WCHAR_H is defined. The HAVE_USABLE_WCHAR_T symbol is only > used to determine whether memcpy is used, or the unicode characters are > copied one by one. > > - Finally, again when defined together with MS_WINDOWS, it sets the > filesystem encoding to mbcs. > > > So, it seems that the HAVE_USABLE_WCHAR_T symbol doesn't play any role > for the extension programmer *at all*. That symbol is defined by the configure script for use in the interpreter - why did you think it is usable for extensions ? The HAVE_USABLE_WCHAR_T symbol only means that we can use wchar_t as synonym for Py_UNICODE and thus makes some APIs more efficient, e.g. on Windows - nothing more. > The preprocessor flag that plays > a role for extensions seem to be HAVE_WCHAR_H since they mark whether > the PyUnicode_FromWideChar and PyUnicode_AsWideChar are available or > not. Right, since wchar.h is the include file that defines the wchar_t type. > This has caused me quite some confusion, and so I suggest the comment > above in the Include/unicodeobject.h file should be fixed. > > Finally, the docs also seem to get it wrong (although I haven't followed > that in detail). Can't reach python.org at the moment, but Python C/api > manual, section 7.3.2, unicode objects says: > > Py_UNICODE > > This type represents a 16-bit unsigned storage type which is used by > Python internally as basis for holding Unicode ordinals. On platforms > where wchar_t is available and also has 16-bits, Py_UNICODE is a > typedef alias for wchar_t to enhance native platform compatibility. On > all other platforms, Py_UNICODE is a typedef alias for unsigned short. > > Isn't the size 32 bits for wide unicode builds? Yes. > Please, please fix this - unicode is already complicated enough even > without this confusion! Please add a bug report to SF for this. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 08 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From vegeta.z at gmail.com Fri Oct 8 21:32:03 2004 From: vegeta.z at gmail.com (daniel narf) Date: Fri Oct 8 21:32:04 2004 Subject: [Python-Dev] Re: Toward Python's future article References: <loom.20041007T192434-234@post.gmane.org> Message-ID: <ck6pvd$1ir$1@sea.gmane.org> i know there are a lot of volunters,fans,developers who are very actively developing/improving the library but the main point of the argument(i think),is to realy focus on the stdlib improvements,not implying that the interpreter improvements are less important,just that the stdlib is currently lacking a bit in certain aspects already mentioned,to try to balance the ecosystem. A more formal,focused,organized aproach to improve the stdlib is the point of the proposal,some sort of plan,roadmap is what is needed(in my opinion). Some things that comes to mind and others based on feedback about the AFK non formal proposal are: to refactor(someway) stdlib parts that are a bit inconsistent,as duplicated functionality of some modules,modules with just one function,obscure ways of working of some of them,to stablish realy strong naming,style coding conventions ie: modules names with get_type,getType,gettype. To realy start a stdlib documentation effort,aspiring it to be the best,clearest documented stdlib out there. To develop the greatest,more advanced library packaging system out there,taking the strengh and weaknesses of others(as CPAM) as a guide. To start a planed and formal draft for py 3.0,some similar system as the iniciative from zope,the zope X3 project and maybe to #adelantar a litle python 3.0 realease(to jump from python 2.7 to python 3000 ^_^). And many other great ideas to start making python the best language ever. From anthony at interlink.com.au Fri Oct 8 22:13:06 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 8 22:18:38 2004 Subject: [Python-Dev] Re: Toward Python's future article In-Reply-To: <ck6pvd$1ir$1@sea.gmane.org> References: <loom.20041007T192434-234@post.gmane.org> <ck6pvd$1ir$1@sea.gmane.org> Message-ID: <4166F4D2.4070107@interlink.com.au> daniel narf wrote: > i know there are a lot of volunters,fans,developers who are very actively > developing/improving the library but the main point of the argument(i > think),is to realy focus on the stdlib improvements,not implying that the > interpreter improvements are less important,just that the stdlib is > currently lacking a bit in certain aspects already mentioned,to try to > balance the ecosystem. As I said in a comment to Andrew's piece - the _best_ way is to pick up a piece of the stdlib that annoys you, and fix it. Attempting to setup a roadmap, or a plan, or whatever, will only mean that nothing ever gets done. For instance, look at the long and sorry history before PyPI. Many, many attempts to do some big picture thing, and no result. What we have now isn't perfect, but it's a start. And it was accomplished by one person (Richard) getting annoyed enough to just roll up the sleeves and do it. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From fredrik at pythonware.com Fri Oct 8 22:24:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Oct 8 22:22:19 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> Message-ID: <ck6stn$9ln$1@sea.gmane.org> Peter Astrand wrote: > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? just check it in, already. (can os.popen and popen2 be refactored to use subprocess, btw?) </F> From dubnerm-news at mail.ru Sat Oct 9 00:18:44 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Sat Oct 9 00:16:56 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <20041006195705.GB17032@rogue.amk.ca> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> Message-ID: <41671244.5070003@mail.ru> A.M. Kuchling wrote: >I like this proposal; it begins to impose some degree of sanity on the >standard library. > > I was the reason behind this PEP. :-) >(If you want editing help with the grammar and text of the PEP, let me >know; I'll happily polish the wording.) > > I will be happy to get such help. I haven't grammar checker handy, so as I recall I've only checked spelling once (and possibly not last edition). Sorry to native English speakers. >> logging.getLogger('stdlib.BaseHTTPServer').setLevel(logging.FATAL) >> >> >There's been discussion in the past of having a name for the Python >standard library, so that 'from <name> import httplib' would always >get the standard httplib module. This name should match whatever's >using in the logging, so the logging should use a name everyone is >happy with using in import statements. Whether that's stdlib or std >or __std__ or Lib isn't important, as long as the logging matches. > > I'm not really sure that these names MUST be equal. Name to use for import statement can be __stdlib__ or ever __python_library__ to ensure uniqueness and specific name MUST draw attention to this place. But for logger name this prefix should be as short as possible because it's to be displayed, and sometime logger plus time, level and message text must fit 80 character line... -- Best regards, Michael Dubner (dubnerm.at.mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From ncoghlan at email.com Sat Oct 9 02:04:01 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 9 02:04:07 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <41671244.5070003@mail.ru> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> Message-ID: <1097280241.41672af18e6be@mail.iinet.net.au> Quoting "Michael P. Dubner" <dubnerm-news@mail.ru>: > >There's been discussion in the past of having a name for the Python > >standard library, so that 'from <name> import httplib' would always > >get the standard httplib module. This name should match whatever's > >using in the logging, so the logging should use a name everyone is > >happy with using in import statements. Whether that's stdlib or std > >or __std__ or Lib isn't important, as long as the logging matches. > > > > > I'm not really sure that these names MUST be equal. Name to use for > import statement can be __stdlib__ > or ever __python_library__ to ensure uniqueness and specific name MUST > draw attention to this place. But > for logger name this prefix should be as short as possible because it's > to be displayed, and sometime logger > plus time, level and message text must fit 80 character line... I agree with Andrew that the names should be equal - consistency is good, and brevity is a benefit for 'from x import' as well. I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. Short, to the point, and I'd be surprised if anyone was using a bare "py" as a package or module name. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From fredrik at pythonware.com Sat Oct 9 08:07:20 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 9 08:05:16 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se><Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ekk91duu.fsf@python.net> Message-ID: <ck7v2o$267$1@sea.gmane.org> Thomas Heller wrote: > How many _xxxx.pyd windows specific modules do we need? as many as it takes to make Python an excellent tool. it's not like one more entry in the makefile, and 20k in the binary distribution will cause problem for anyone. if you want to stop contributions to Python's standard library unless they use your tools, you have a serious problem. </F> From astrand at lysator.liu.se Sat Oct 9 12:08:20 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 12:30:33 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ck6lhj$k72$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> > I've been watching the progression of subprocess with some interest. It > looks encouraging, and is exactly the sort of thing I need for my work. > > One small nit I've noticed: aren't the names of subprocess.call() and > subprocess.callv() reversed? If you look at unix execl() and execv(), > execl() takes a variable-length argument list and execv() takes a list > (vector?) of arguments. But it's the opposite for subprocess -- callv() > takes a variable-length arg list and call() takes a list of args. Oh. Yes, you are right. > Am I missing something? Can these be renamed now before it gets > standardized? I'd prefer not to rename the call() function. The name is short and simple, and the function is very much used. I'm positive to renaming the callv() function, though. One obvious name would be "calll", but that's quite ugly. How about "lcall"? Then we can keep the "callv" name for backwards compatibility. Or, we could just keep the "callv" name, and pretend that "v" stands for "variable number of arguments". /Peter ?strand <astrand@lysator.liu.se> From fredrik at pythonware.com Sat Oct 9 13:06:36 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 9 13:04:33 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se><Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> Message-ID: <ck8gjs$1qe$1@sea.gmane.org> Peter Astrand wrote: > > Am I missing something? Can these be renamed now before it gets > > standardized? > > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. I'm positive to renaming the > callv() function, though. One obvious name would be "calll", but that's > quite ugly. How about "lcall"? Then we can keep the "callv" name for > backwards compatibility. do we need both? you could rename "callv" to "call", and let people type an extra "*" if they want to pass in a list of arguments: subprocess.call("stty", "sane", "-F", device) subprocess.call(*["stty", "sane", "-F", device]) or, more likely: args = ["somecommand"] # several lines of code to add options and parameters # to the args list subprocess.call(*args) > Or, we could just keep the "callv" name, and pretend that "v" stands for > "variable number of arguments". I have no problem with that... </F> From astrand at lysator.liu.se Sat Oct 9 14:04:33 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 14:29:47 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: <ck8gjs$1qe$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se><Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> On Sat, 9 Oct 2004, Fredrik Lundh wrote: > > I'd prefer not to rename the call() function. The name is short and > > simple, and the function is very much used. I'm positive to renaming the > > callv() function, though. One obvious name would be "calll", but that's > > quite ugly. How about "lcall"? Then we can keep the "callv" name for > > backwards compatibility. > > do we need both? you could rename "callv" to "call", and let people type > an extra "*" if they want to pass in a list of arguments: > > subprocess.call("stty", "sane", "-F", device) > subprocess.call(*["stty", "sane", "-F", device]) I'd like "call" to stay as it is. It's interface is pretty clean, and maps directly to the Popen constructor. "callv" is only syntactic sugar. It's for people that thinks that: subprocess.call(["ls", "-l", "/foo/bar"]) ...is to ugly compared to: os.system("ls -l /foo/bar") With callv, it is: subprocess.callv("ls", "-l", "/foo/bar") ...which is slighly nicer. The drawback with callv is that it does not allow specifying the program and it's arguments as a whitespace-separated string: The entire (first) string would be intepreted as the executable. So, you cannot do: subprocess.callv("somewindowsprog.exe some strange command line") because then the system would try to execute a program called "somewindowsprog.exe some strange command line", which doesn't exist. You cannot do this either: subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. To summarize: call() must stay as it is for completeness. callv() is just syntactic sugar, but probably deserves to stay as well. /Peter ?strand <astrand@lysator.liu.se> From lunz at falooley.org Sat Oct 9 15:28:29 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 15:28:33 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> Message-ID: <ck8p1s$eq4$1@sea.gmane.org> astrand@lysator.liu.se said: > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. That's understandable. Though if people are going to go through the pain of changing it, it's better that it happen before it becomes a standard part of python. > I'm positive to renaming the callv() function, though. One obvious > name would be "calll", but that's quite ugly. How about "lcall"? Then > we can keep the "callv" name for backwards compatibility. How recently was callv added? I'd prefer not to have a callv at all than to have a call/callv pair that don't map naturally to execl/execv. > Or, we could just keep the "callv" name, and pretend that "v" stands for > "variable number of arguments". I really don't want to do this. I can tell already I'll be forever forgetting which one I need, and probably anyone else with C/unix experience will be in the same boat. It's the kind of irritant I'd like to wipe out now while there's still the opportunity. Jason From lunz at falooley.org Sat Oct 9 15:54:22 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 15:54:26 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> Message-ID: <ck8qid$ln6$1@sea.gmane.org> astrand@lysator.liu.se said: > ...which is slighly nicer. The drawback with callv is that it does not > allow specifying the program and it's arguments as a whitespace-separated > string: The entire (first) string would be intepreted as the executable. > So, you cannot do: > > subprocess.callv("somewindowsprog.exe some strange command line") > > because then the system would try to execute a program called > "somewindowsprog.exe some strange command line", which doesn't exist. You > cannot do this either: > > subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") > > ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. I'm not sure I understand what the MSC runtime has to do with the naming of call/callv. Your examples don't work with call either, right? Their call() equivalents: subprocess.call(["somewindowsprog.exe some strange command line"]) subprocess.call(["somewindowsprog.exe", "some", "strange", "command", "line"]) are just as broken, no? Overall, I agree that callv() is superfluous. In my programming, I always end up using the "v" variants of exec functions, because there's always _something_ you do to the command line first, and it's easier to handle arguments as a list. [The above paragraph makes my point: "I always use execv(), so we should drop subprocess.callv()?" The naming hurts my poor brain.] Jason From astrand at lysator.liu.se Sat Oct 9 15:57:50 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 16:26:16 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ck8p1s$eq4$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8p1s$eq4$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410091542460.7564@koeberg.lysator.liu.se> On Sat, 9 Oct 2004, Jason Lunz wrote: > > I'm positive to renaming the callv() function, though. One obvious > > name would be "calll", but that's quite ugly. How about "lcall"? Then > > we can keep the "callv" name for backwards compatibility. > > How recently was callv added? I'd prefer not to have a callv at all than > to have a call/callv pair that don't map naturally to execl/execv. callv has been around even longer than call actually, although callv was earlier called "run". > > Or, we could just keep the "callv" name, and pretend that "v" stands for > > "variable number of arguments". > > I really don't want to do this. I can tell already I'll be forever > forgetting which one I need, and probably anyone else with C/unix > experience will be in the same boat. It's the kind of irritant I'd like > to wipe out now while there's still the opportunity. I don't have a very strong opinion about callv, so if the general opinion wants to remove it, that's OK with me. /Peter ?strand <astrand@lysator.liu.se> From astrand at lysator.liu.se Sat Oct 9 16:06:23 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sat Oct 9 16:47:55 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ck8qid$ln6$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> > > subprocess.callv("somewindowsprog.exe", "some", "strange", "command", "line") > > > > ...if somewindowsprog.exe doesn't use the MS C runtime argument rules. > > I'm not sure I understand what the MSC runtime has to do with the naming > of call/callv. In that case, my explanation wasn't good enough :) It's somewhat complicated. Most people will never have any problems with these issues, but I've taken care so that the API should support all cornercases. >Your examples don't work with call either, right? They work with call if you use a string argument. That's the core of the problem: The callv function doesn't support passing a string-type args argument to the Popen constructor. >Their > call() equivalents: > > subprocess.call(["somewindowsprog.exe some strange command line"]) > subprocess.call(["somewindowsprog.exe", "some", "strange", "command", "line"]) > > are just as broken, no? Yes. You'll need to do: subprocess.call("somewindowsprog.exe some strange command line") /Peter ?strand <astrand@lysator.liu.se> From ncoghlan at email.com Sat Oct 9 18:24:00 2004 From: ncoghlan at email.com (Nick Coghlan) Date: Sat Oct 9 18:50:38 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> Message-ID: <1097339040.416810a0f3a95@mail.iinet.net.au> Quoting Peter Astrand <astrand@lysator.liu.se>: > >Your examples don't work with call either, right? > > They work with call if you use a string argument. That's the core of the > problem: The callv function doesn't support passing a string-type args > argument to the Popen constructor. > > > >Their > > call() equivalents: > > > > subprocess.call(["somewindowsprog.exe some strange command line"]) > > subprocess.call(["somewindowsprog.exe", "some", "strange", "command", > "line"]) > > > > are just as broken, no? > > Yes. You'll need to do: > > subprocess.call("somewindowsprog.exe some strange command line") Given that call is only a shortcut function, wouldn't the following suffice?: def call(*args, **kwds): if len(args) <= 1: return Popen(*args, **kwds) else: return Popen(args, **kwds) With that implementation, a single string, a list, a sequence of strings and Popen keywords only would all work as arguments to call. That is: call("ls -l") -> Popen("ls -l") call("ls", "-l") -> Popen(["ls", "-l"]) call(["ls", "-l"]) -> Popen(["ls", "-l"]) call(args="ls -l") -> Popen(args="ls -l") All it would mean is that if you want to use the optional arguments to Popen, you either don't use call, or you use keywords. Cheers, Nick. -- Nick Coghlan Brisbane, Australia From erik at heneryd.com Sat Oct 9 19:46:38 2004 From: erik at heneryd.com (Erik Heneryd) Date: Sat Oct 9 19:46:43 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> Message-ID: <416823FE.6080206@heneryd.com> Haven't really read the PEP/tried the module, so I can't comment on it specifically, but +1 on a proper subprocess module. Implemented something like this myself some time ago. Peter Astrand wrote: >>Am I missing something? Can these be renamed now before it gets >>standardized? > > > I'd prefer not to rename the call() function. The name is short and > simple, and the function is very much used. I'm positive to renaming the > callv() function, though. One obvious name would be "calll", but that's > quite ugly. How about "lcall"? Then we can keep the "callv" name for > backwards compatibility. Don't think backwards compatibility is that much of an issue. Since you're renaming it subprocess (+1 on the name) old code will have to be updated anyway. -1 on function names conflicting with the exec/spawn way of naming things. Erik From lunz at falooley.org Sat Oct 9 20:01:13 2004 From: lunz at falooley.org (Jason Lunz) Date: Sat Oct 9 20:01:23 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> Message-ID: <ck9918$kop$1@sea.gmane.org> astrand@lysator.liu.se said: > They work with call if you use a string argument. That's the core of the > problem: The callv function doesn't support passing a string-type args > argument to the Popen constructor. OK, I read the code again and I see what you mean. So yes, this argues even more against the existence of callv(). I would have expected that it would always be possible to translate a call() invocation to its callv() equivalent and vice-versa. That turns out not to be true in the case of call() users who want MSC-style command-line parsing done on a bare string (whether by CreateProcess() on windows or cmdline2list() on unix), because the _execute_child() implementations need to know whether args was originally a string or a list, and this information is hidden by callv()'s list encapsulation of the args. This all makes me think there could be a better approach to cross-platform handling of command-line arguments. When is anyone ever going to want the braindamaged MS cmdline expansion done while they're trying to invoke a subprocess on unix? I don't see that getting used a lot. I think I need to understand better the division of labor between parent and child on Windows when it comes to passing the command line during process creation. I'd like to think that unix execv() is a superset of the interface offered by CreateProcess(), in that you can initialize your child's argv however you like without regard to whitespace or quoting. So it would be best if it were possible to offer the execv() interface on both platforms if that's possible. Jason From pf_moore at yahoo.co.uk Sat Oct 9 21:18:37 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sat Oct 9 21:18:41 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> <ck9918$kop$1@sea.gmane.org> Message-ID: <ubrfb8zua.fsf@yahoo.co.uk> Jason Lunz <lunz@falooley.org> writes: > I think I need to understand better the division of labor between parent > and child on Windows when it comes to passing the command line during > process creation. It's simple. The parent passes a *command line*, and the child retrieves it using the GetCommandLine API. There is no concept of argv at the Windows level - it is implemented by the C runtime parsing the return value of GetCommandLine() and passing the resulting arguments to main(). Hence on Windows, a command line is the fundamental unit, whereas on Unix an argument list is fundamental. The biggest problem on Windows is that not all executables use the Microsoft C runtime. Some use other C runtimes, others parse the command line directly and don't use argv at all. > I'd like to think that unix execv() is a superset of the interface > offered by CreateProcess(), in that you can initialize your child's > argv however you like without regard to whitespace or quoting. It would be nice to think that, but it ain't true <wink>. > So it would be best if it were possible to offer the execv() > interface on both platforms if that's possible. The unix execv is just *different*. Both the Windows and the Unix interfaces have capabilities the other doesn't offer. And as a result, it's not possible to offer an execv interface on Windows, at least not without running the risk of garbling some commands. Luckily, the oddball cases are rare. So 99% of the time, either interface will do OK (os.system on Unix, os.spawnv on Windows). But the 1% can be crucial - shell-exploit style security holes on Unix, garbled commands on Windows. I think Peter's approach of supporting both forms - a single string as a command line, and list of strings as an argv list, and converting both to the more natural OS-native form as needed, is sensible (I would, I argued for it when he was developing it!) Paul -- Ooh, how Gothic. Barring the milk. From nhodgson at bigpond.net.au Sun Oct 10 01:06:48 2004 From: nhodgson at bigpond.net.au (Neil Hodgson) Date: Sun Oct 10 01:06:53 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module Message-ID: <015001c4ae54$a7b32d40$a44a8890@neil> Fredrik Lundh: > Thomas Heller wrote: > > > How many _xxxx.pyd windows specific modules do we need? > > as many as it takes to make Python an excellent tool. it's not like one > more entry in the makefile, and 20k in the binary distribution will cause > problem for anyone. Requiring windows interface modules to be written in C rather than Python reduces the number of Python users that can add, modify and fix these modules. It also increases the amount of effort involved in working on these modules. Neil From gvanrossum at gmail.com Sun Oct 10 01:18:29 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 10 01:18:36 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: <015001c4ae54$a7b32d40$a44a8890@neil> References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: <ca471dc20410091618487e6d01@mail.gmail.com> > Requiring windows interface modules to be written in C rather than Python > reduces the number of Python users that can add, modify and fix these > modules. It also increases the amount of effort involved in working on these > modules. I don't know. Writing Windows interface modules is a highly specialized form of torture, requiring arcane expertise. Most Python users don't have that expertise, and could do more damage than good. I'd expect those folks that *do* have the expertise to have learned what they know by coding in C/C++, so I don't see that requiring the use of C is any particular burden. Rather, having to be familiar with the intricate details of the mapping between C and Python is likely to trip up folks occasionally. And what about all the stuff that's defined in header files? In the past, in the Unix world, I experimented with translating .h files to Python modules to save me from the burden of having to add large numbers of symbol definitions to C extensions. (Look for files named Lib/plat-XXX/regen in the source tree.) One by one, that approach has proven to be problematic, and nearly all of those have eventually been turned into systematic lists of symbol definitions in C code. See for example the posix, socket, fcntl, and signal modules. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pgimpelj at sympatico.ca Sun Oct 10 02:49:18 2004 From: pgimpelj at sympatico.ca (Paul Gimpelj) Date: Sun Oct 10 02:48:16 2004 Subject: [Python-Dev] scropion References: <015001c4ae54$a7b32d40$a44a8890@neil> Message-ID: <03c701c4ae62$facd40c0$c901010a@zoom> hello, I am new to python, and I have found a robot vision system that uses python. It looks pretty good. Its called scorpion. It runs on xp platform. This is bringing me back to python. I was worried you would change the architecture. My question is for performance: isn't binary code faster than p code? So If I can vote, I would vote for c or c++ in the numerics functions and data transfer areas. even assembler. The quicker the better. Thanks Paul From nhodgson at bigpond.net.au Sun Oct 10 04:02:59 2004 From: nhodgson at bigpond.net.au (Neil Hodgson) Date: Sun Oct 10 04:03:09 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> <ca471dc20410091618487e6d01@mail.gmail.com> Message-ID: <027b01c4ae6d$446deea0$a44a8890@neil> Guido van Rossum: > I don't know. Writing Windows interface modules is a highly > specialized form of torture, requiring arcane expertise. Most Python > users don't have that expertise, and could do more damage than good. > I'd expect those folks that *do* have the expertise to have learned > what they know by coding in C/C++, so I don't see that requiring the > use of C is any particular burden. Visual Basic programmers often write to the Win32 API without knowing enough C to write an extension module. They can declare a function and then call it: Declare Function GetComputerName Lib "kernel32" Alias "GetComputerNameA" (ByVal sBuffer As String, lSize As Long) As Long strString = String(255, Chr$(0)) GetComputerName strString, 255 The incantations are a little more arcane than for C but there is quite a lot of documentation available about how to do this so VB programmers often don't need to read the C header files. One source is win32api.txt which lists the structs, constants, and functions in the Win32 API as VB declarations. > And what about all the stuff that's defined in header files? ctypes doesn't yet have a good solution to header files. http://starship.python.net/crew/theller/moin.cgi/WrappingWin32Api > In the > past, in the Unix world, I experimented with translating .h files to > Python modules to save me from the burden of having to add large > numbers of symbol definitions to C extensions. (Look for files named > Lib/plat-XXX/regen in the source tree.) One by one, that approach has > proven to be problematic, and nearly all of those have eventually been > turned into systematic lists of symbol definitions in C code. > See for example the posix, socket, fcntl, and signal modules. Where the problems are caused by multiple vendor or version variations then Windows is much more stable with additions occurring often but amendments, removals and world changes (Win16->Win32) being very rare. Looking at the all_ins function in posix, it seems to have been made by hand. This wouldn't be a great approach to a large API (there are thousands of values #defined in Win32) where finding the names of the constants (and not, for example, function names which are often #defined to either narrow or wide functions) is as difficult as finding their values. When coding an interface module in Python, it would be fastest to copy just the relevant declarations from the headers or win32api.txt and convert to Python syntax by hand. Neil From dubnerm-news at mail.ru Sun Oct 10 04:27:29 2004 From: dubnerm-news at mail.ru (Michael P. Dubner) Date: Sun Oct 10 04:25:12 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: <41689E11.10106@mail.ru> Nick Coghlan wrote: >I agree with Andrew that the names should be equal - consistency is good, and >brevity is a benefit for 'from x import' as well. > >I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. >Short, to the point, and I'd be surprised if anyone was using a bare "py" as a >package or module name. > > +1 -- Best regards, Michael Dubner (dubnerm.at.mindless.com) Brainbench MVP/HTML+JavaScript (http://www.brainbench.com) PS: Sorry for my English From fdrake at acm.org Sun Oct 10 05:14:55 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Oct 10 05:15:09 2004 Subject: [Python-Dev] Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: <200410092314.55154.fdrake@acm.org> On Friday 08 October 2004 08:04 pm, Nick Coghlan wrote: > I agree with Andrew that the names should be equal - consistency is good, > and brevity is a benefit for 'from x import' as well. Yes, and yes. > I'd propose taking a tip from Java and using "py." as the prefix for the > stdlib. Short, to the point, and I'd be surprised if anyone was using a > bare "py" as a package or module name. Works for me! +1. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From rkern at ucsd.edu Sun Oct 10 05:32:22 2004 From: rkern at ucsd.edu (Robert Kern) Date: Sun Oct 10 05:40:56 2004 Subject: [Python-Dev] Re: Guidelines for Logging Usage In-Reply-To: <1097280241.41672af18e6be@mail.iinet.net.au> References: <4161C45A.4050205@mail.ru> <20041006195705.GB17032@rogue.amk.ca> <41671244.5070003@mail.ru> <1097280241.41672af18e6be@mail.iinet.net.au> Message-ID: <ckaag8$5lr$1@sea.gmane.org> Nick Coghlan wrote: > I'd propose taking a tip from Java and using "py." as the prefix for the stdlib. > Short, to the point, and I'd be surprised if anyone was using a bare "py" as a > package or module name. Well, there's Py, n?e PyCrust, but it's now buried under the main wxPython package. >>> from wx import py http://www.wxpython.org/py.php -- Robert Kern rkern@ucsd.edu "In the fields of hell where the grass grows high Are the graves of dreams allowed to die." -- Richard Harter From tim.peters at gmail.com Sun Oct 10 07:19:19 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 10 07:19:24 2004 Subject: [Python-Dev] assert failure on obmalloc In-Reply-To: <e8bf7a5304090708123c619f4@mail.gmail.com> References: <e8bf7a5304090708123c619f4@mail.gmail.com> Message-ID: <1f7befae04100922196d299c3e@mail.gmail.com> [Jeremy Hylton, on Tue, 7 Sep 2004] > Failure running the test suite today with -u compiler enabled on Windows XP. > > test_logging > Assertion failed: bp != NULL, file > \code\python\dist\src\Objects\obmalloc.c, line 604 > > The debugger says the error is here: > msvcr71d.dll!_assert(const char * expr=0x1e22bcc0, const char * > filename=0x1e22bc94, unsigned int lineno=604) Line 306 C > python24_d.dll!PyObject_Malloc(unsigned int nbytes=100) Line 604 + 0x1b C > python24_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=84) Line > 1014 + 0x9 C > python24_d.dll!PyThreadState_New(_is * interp=0x00951028) Line 136 + 0x7 C > python24_d.dll!PyGILState_Ensure() Line 430 + 0xc C > python24_d.dll!t_bootstrap(void * boot_raw=0x02801d48) Line 431 + 0x5 C > python24_d.dll!bootstrap(void * call=0x04f0d264) Line 166 + 0x7 C > msvcr71d.dll!_threadstart(void * ptd=0x026a2320) Line 196 + 0xd C > > I've been seeing this sort of error on-and-off for at least a year > with my Python 2.3 install. It's the usual reason my spambayes > popproxy dies. I can't recell seeing it before on Windows or while > running the test suite. I expect bug 1041645 is relevant. That was suffering the debug-build problem identified later in this thread by Michael Hudson, and also from that the internal TLS function find_key() was thread-insane (regardless of build type). Those should all be fixed now, so gripe if you see this again after updating. BTW, those are all critical bugfixes, but I don't have time to backport them. If anyone does, there was one checkin in the chain that changed when PyGILState_Release() deleted its TLS key, but I'm pretty sure there was no actual need for that change. From fredrik at pythonware.com Sun Oct 10 08:42:01 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun Oct 10 08:40:02 2004 Subject: [Python-Dev] Re: Re: Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> <ca471dc20410091618487e6d01@mail.gmail.com> Message-ID: <ckalfo$l7h$1@sea.gmane.org> Guido wrote: > I don't know. Writing Windows interface modules is a highly > specialized form of torture, requiring arcane expertise. Despite that, some of us can do this in our sleep (including writing the bindings). I find it somewhat sad that some people don't want us to contribute to Python. While we have you on line, can you pronounce on the subprocess module? The code is written by highly experienced programmers, it has been widely tested, it's entirely self-contained, builds and runs under 2.2 and later, has an extensive test suite and a PEP (which can be quickly turned into a library reference section), and replaces several fragile API:s in the current library with a single, well-defined construct. If this isn't good enough for the standard library, I don't see what is. </F> From gjc at inescporto.pt Sun Oct 10 13:41:08 2004 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Sun Oct 10 13:41:20 2004 Subject: [Python-Dev] Patch to make GIL API calls noops in single threaded applications Message-ID: <1097408468.2328.6.camel@emperor> Can someone review my patch[1]? It makes GIL API calls noops in single threaded applications, i.e. if PyEval_InitThreads hasn't been called. It's a very simple patch, but it makes a lot of difference if you want to make a Python extension module thread-aware without incurring an extra (unnecessary) performance penalty in single-threaded programs. Without this in Python, each extension library has to keep track of thread state (enabled or not), and one extension library doesn't work well with another without some explicit synchronization between the two. Please, we should fix this before Python 2.4 final. Regards. [1] https://sourceforge.net/tracker/? func=detail&aid=1011380&group_id=5470&atid=105470 -- Gustavo J. A. M. Carneiro <gjc@inescporto.pt> <gustavo@users.sourceforge.net> The universe is always one step beyond logic From lunz at falooley.org Sun Oct 10 14:49:55 2004 From: lunz at falooley.org (Jason Lunz) Date: Sun Oct 10 14:49:59 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <015001c4ae54$a7b32d40$a44a8890@neil> <ca471dc20410091618487e6d01@mail.gmail.com> <ckalfo$l7h$1@sea.gmane.org> Message-ID: <ckbb5j$ttd$1@sea.gmane.org> fredrik@pythonware.com said: > While we have you on line, can you pronounce on the subprocess module? > The code is written by highly experienced programmers, it has been > widely tested, it's entirely self-contained, builds and runs under 2.2 > and later, has an extensive test suite and a PEP (which can be quickly > turned into a library reference section), and replaces several fragile > API:s in the current library with a single, well-defined construct. > If this isn't good enough for the standard library, I don't see what > is. hear, hear! Jason From lunz at falooley.org Sun Oct 10 15:17:41 2004 From: lunz at falooley.org (Jason Lunz) Date: Sun Oct 10 15:17:46 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> <ck9918$kop$1@sea.gmane.org> <ubrfb8zua.fsf@yahoo.co.uk> Message-ID: <ckbcpl$46a$1@sea.gmane.org> pf_moore@yahoo.co.uk said: > Hence on Windows, a command line is the fundamental unit, whereas on > Unix an argument list is fundamental. Yes, you're right. I read up on CreateProcess(), GetCommandLine(), and CommandLineToArgvW() after posting. Interestingly, MS's sample code for CommandLineToArgvW is buggy because of confusion between the two interfaces. http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/commandlinetoargvw.asp Also, it can fail. > The biggest problem on Windows is that not all executables use the > Microsoft C runtime. Some use other C runtimes, others parse the > command line directly and don't use argv at all. So why does subprocess use cmdline2list() in the parent on unix to emulate the way a child subprocess might parse the string on windows? (But only if it's written in C, uses the MSC runtime, and uses the argv/argc handed to main() rather than calling GetCommandLine() itself). Why not emulate CommandLineToArgvW()? or something else entirely? I think it would be cleaner not to emulate anything at all. > The unix execv is just *different*. Both the Windows and the Unix > interfaces have capabilities the other doesn't offer. Well, the windows interface is a subset of the unix one. The length of argv on windows is limited to 1. > I think Peter's approach of supporting both forms - a single string as > a command line, and list of strings as an argv list, and converting > both to the more natural OS-native form as needed, is sensible (I > would, I argued for it when he was developing it!) I can see that it's trying to be symmetric and orthogonal, but I don't think that the result is worth it in this case. In what scenario is the use of cmdline2list() really useful? Jason From jasone at canonware.com Sun Oct 10 21:39:58 2004 From: jasone at canonware.com (Jason Evans) Date: Sun Oct 10 21:39:40 2004 Subject: [Python-Dev] Cyclic GC issues Message-ID: <20041010193958.GF341@canonware.com> Since the spring of 2003, I have been developing Crux, which is a computer program for phylogenetic inferencing (bioinformatics) research. In March of 2004, I switched Crux to using Python from having used a different embeddable interpreter. For the most part, I have been very happy with Python, but Python's garbage collector has been a major source of frustration. Below, I describe my trials and tribulations with Python's GC. I also offer some suggestions for changes to Python; if any of the proposed changes receive favorable feedback, I am willing to help develop patches to Python. Naturally, if I am somehow abusing Python, and there are better ways to do things, I'd be happy to hear how to improve Crux. The important aspect of Crux is that it deals with trees. These trees are unrooted (there is no up or down), and multifurcating (nodes can have an arbitrary number of neighboring nodes). Thus, the trees are self-referential, and without the cyclic GC capabilities of Python, there would be little hope of making these trees integrate well with Python. Following is a diagram that illustrates the linkage between various objects for a simple tree. Crux exposes all of the components of the trees as Python objects. All lines in the diagram represent bi-directional references (except for the T-->N line). Every object refers to the tree object; those lines are left out in order to reduce clutter. T: Tree N N N: Node \ / E: Edge R R R: Ring \ / E E \ / R---------R / \ / \ / \ / \ | \ / | | \ / | | T--->N | | | | \ | / \ | / \----R----/ | E | R | N At the C (not Python object) level, the R-E-R construct is actually a set of structures that are allocated/deallocated as a single unit. Edges are *always* connected to two rings, so there's no point in allocating these separately. Also, lone ring objects actually are rings with one member; they refer to themselves (prev and next pointers). That should be enough information to understand the problems I encountered. 1) I originally had lone rings refer to themselves (ob_refcnt started out at 3; 2 self-references and one reference held by the associated edge). This didn't work. It appears that the cyclic GC does not actually calculate the number of live references to objects (references that are reachable by traversing all objects accessible from the root set); instead it assumes that if tp_clear() doesn't drop the reference count to a number that equals the number of references from live objects, there must still be references from live objects. Unfortunately, visit()ing self does not work, so there is no way to convince Python that all references are from unreachable objects. Working around this in Crux requires a lot of extra reference counting complexity, because there are three different cases for reference counts, depending on how many members there are in a ring (1, 2, or 3+ members). 2) This issue is really just a different manifestation of issue (1). At the C (not Python object) level, each node only actually stores a pointer to a single member of the associated ring. Given a single ring member, it is possible to traverse the ring and reach all other ring members. As mentioned in issue (1), the cyclic GC expects tp_traverse() to call visit() once for each reference held. It is not enough for a node to visit() one ring member; it must visit() all ring members, in order for the GC to count how many references are from unreachable objects, versus reachable from the root set. In summary, issues (1) and (2) are due to how the cyclic GC does the "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is that it should be sufficient to assure that all objects reachable from the root set are visit()ed at least once; it should not be important how many times each unreachable object is visit()ed. I don't have a deep enough understanding of the Python interpreter to give a detailed suggestion for improvement. I suspect that determining the root set is not an easy operation; if this is the case, then I think we're stuck with the current design. If determining the root set *is* easy (or even possible), then I would suggest using a standard mark/sweep collector, where unreachable objects are scheduled for destruction. tp_traverse(), tp_clear(), and tp_dealloc() would retain the same structure; the only difference would be the logic that determines which objects can be destroyed. 3) A strange thing can happen when tp_clear() is called. Suppose that an edge object is being cleared, and it drops its references to the associated rings. If ob_refcnt of one of the rings drops to 0 as a result, Python will tp_dealloc() the ring *right* *now*, without ever calling tp_clear() on the ring. That means that I have to keep track of whether tp_clear() has been called on objects, if it is at all important that tp_clear() be called, so that I can manually do so in tp_dealloc(), if necessary. It is in my opinion reasonable to have cleanup code in tp_clear(), with the assumption that it will be called precisely once, but Python makes me do extra work to make sure that this happens. This should be pretty easy to change. A single bit per object is needed to keep track of whether tp_clear() has been called. I think this only needs to be done for object types that support cyclic GC. 4) There is no way to control the order in which objects are tp_dealloc()ed. This is a problem for the R-E-R construct, since at a low level, these objects are always linked together. What I would like to do is defer tp_dealloc() on the edge until after both rings have been deleted. Instead, I am forced to use a reference-counted deletion function. Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until later is not a reasonable option, because this defers deletion of the edge until a later round of garbage collection. This could be addressed in the Python interpreter by paying heed to the return value of tp_dealloc(). If the return value is non-zero, move the object to the end of the list of objects to be destroyed, so that destruction is tried later. This allows the module to impose its own destruction ordering. I look forward to feedback. Thank you, Jason Evans From martin at v.loewis.de Sun Oct 10 23:36:11 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun Oct 10 23:36:14 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041010193958.GF341@canonware.com> References: <20041010193958.GF341@canonware.com> Message-ID: <4169AB4B.9030605@v.loewis.de> > 1) I originally had lone rings refer to themselves (ob_refcnt started out > at 3; 2 self-references and one reference held by the associated edge). I would have thought the refcount should have been 4: two self references, plus one from E, plus one from N (maybe I don't understand what make a ring "lone", though). > This didn't work. It appears that the cyclic GC does not actually > calculate the number of live references to objects (references that are > reachable by traversing all objects accessible from the root set); That's correct. It doesn't need to. > instead it assumes that if tp_clear() doesn't drop the reference count > to a number that equals the number of references from live objects, > there must still be references from live objects. That's also correct. tp_clear is called for objects that are not reachable from the root set, anymore, so it *should* break all non-reachable cycles. > Unfortunately, > visit()ing self does not work, so there is no way to convince Python > that all references are from unreachable objects. Wrong. If tp_clear is called, you already know for certain that "self" is garbage. So you need to clear all references - regardless of whether you think you should: trust us, we know what we do. > Working around this in Crux requires a lot of extra reference counting > complexity, because there are three different cases for reference > counts, depending on how many members there are in a ring (1, 2, or 3+ > members). Why is that? Just DECREF every PyObject* in "self", then set the PyObject* to NULL. No need for any additional reference counts. > 2) This issue is really just a different manifestation of issue (1). > > At the C (not Python object) level, each node only actually stores a > pointer to a single member of the associated ring. Given a single ring > member, it is possible to traverse the ring and reach all other ring > members. As mentioned in issue (1), the cyclic GC expects tp_traverse() > to call visit() once for each reference held. It is not enough for a > node to visit() one ring member; it must visit() all ring members, in > order for the GC to count how many references are from unreachable > objects, versus reachable from the root set. Hmmm. Even if a ring member holds only a single reference to another ring member, would not calling visit() on this member in turn invoke visit() on the other one, which would, in turn, invoke visit() on the next one, and so on? > In summary, issues (1) and (2) are due to how the cyclic GC does the > "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is > that it should be sufficient to assure that all objects reachable from > the root set are visit()ed at least once; it should not be important how > many times each unreachable object is visit()ed. Correct. Indeed, cyclic GC makes sure visit is not called more often than needed. > I don't have a deep enough understanding of the Python interpreter to > give a detailed suggestion for improvement. I suspect that determining > the root set is not an easy operation; if this is the case, then I think > we're stuck with the current design. If determining the root set *is* > easy (or even possible), then I would suggest using a standard > mark/sweep collector, where unreachable objects are scheduled for > destruction. That is already the case. tp_clear is the procedure that performs the destruction. > tp_traverse(), tp_clear(), and tp_dealloc() would retain > the same structure; the only difference would be the logic that > determines which objects can be destroyed. I don't see a need for change. The logic that determines the objects to be destroyed is already "precise", determining only unreachable objects. > 3) A strange thing can happen when tp_clear() is called. Suppose that an > edge object is being cleared, and it drops its references to the > associated rings. If ob_refcnt of one of the rings drops to 0 as a > result, Python will tp_dealloc() the ring *right* *now*, without ever > calling tp_clear() on the ring. That means that I have to keep track of > whether tp_clear() has been called on objects, if it is at all important > that tp_clear() be called, so that I can manually do so in tp_dealloc(), > if necessary. It is in my opinion reasonable to have cleanup code in > tp_clear(), with the assumption that it will be called precisely once, > but Python makes me do extra work to make sure that this happens. Your expectation is wrong. tp_clear is not the place to perform cleanup. It is the place to break cycles. tp_dealloc is the place to perform cleanup. > 4) There is no way to control the order in which objects are > tp_dealloc()ed. Correct. Unfortunately, there is no way to change that. For cyclic structures, there is no natural "starting point". Python picks an arbitrary starting point, assuming that the order does not matter. If class objects have an __del__, it assumes that the order does matter, and gives up - putting the objects into gc.garbage. > This could be addressed in the Python interpreter by paying heed to the > return value of tp_dealloc(). If the return value is non-zero, move the > object to the end of the list of objects to be destroyed, so that > destruction is tried later. This allows the module to impose its own > destruction ordering. I don't understand. tp_dealloc does not have a return value, so what does "pay heed" mean? Regards, Martin From tim.peters at gmail.com Mon Oct 11 00:45:00 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 00:45:10 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041010193958.GF341@canonware.com> References: <20041010193958.GF341@canonware.com> Message-ID: <1f7befae04101015454de20b5e@mail.gmail.com> [Jason Evans] > Since the spring of 2003, I have been developing Crux, which is a computer > program for phylogenetic inferencing (bioinformatics) research. In March > of 2004, I switched Crux to using Python from having used a different > embeddable interpreter. For the most part, I have been very happy with > Python, but Python's garbage collector has been a major source of > frustration. Then you're not using it correctly -- it's easy if you are. You didn't show actual code, though, and I don't have time to guess what you might have done. > ... > The important aspect of Crux is that it deals with trees. The details shouldn't matter; Python's cyclic gc handles arbitrary graph structures, including multiple self-loops if you want them. The Python core creates graph structures of boggling complexity, so it might help you to study how core objects implement their tp_traverse and tp_clear methods. They're mostly very simple. Where they're not very simple, it's usually because someone had a silly idea of "optimization" in mind. ... > 1) I originally had lone rings refer to themselves (ob_refcnt started out > at 3; 2 self-references and one reference held by the associated edge). > This didn't work. "Didn't work" leaves the reader guessing about everything. Showing code, and being specific about both what you expected and what you observed, are necessary. python-dev isn't really the right place for that, though. > It appears that the cyclic GC does not actually calculate the number of live > references to objects (references that are reachable by traversing all objects > accessible from the root set); Python's gc has no "root set" concept, so thinking in those terms won't help. It won't grow a root-set concept, either: "foreign" extension modules (like Tcl/Tk, or library packages written in Fortran) can hold what would normally be considered root-set objects for Python, and that's fine. They're not required to cooperate with Python's memory management beyond respecting Python's refcount rules. Forcing modules (via some new API) to identify root-set objects they own would be backward-incompatible (not to mention a major new burden on extension authors and users), so won't happen. > instead it assumes that if tp_clear() doesn't drop the reference count > to a number that equals the number of references from live objects, > there must still be references from live objects. Unfortunately, > visit()ing self does not work, so there is no way to convince Python > that all references are from unreachable objects. Again I can't guess what "does not work" means. But I can assure you it "would work" if I wrote it <wink>. It's easy: an object's tp_traverse must call the visit() callback exactly once for each non-NULL PyObject* *directly* (in one step) reachable from the object, and that's all. If self is reachable from self via a refcounted pointer held by self, then self's tp_traverse() implementation must call visit(self, arg). But I think you're getting off track even *caring* what your pointers point at. If some object type T has three Py_Object* pointers pa, pb, pc, then using Python 2.4's Py_VISIT macro, T's tp_traverse should be coded static int T_traverse(T *self, visitproc visit, void *arg) { Py_VISIT(self->pa); Py_VISIT(self->pb); Py_VISIT(self->pc); return 0; } It doesn't matter what pa (etc) points at. If pa points to self, fine; if it doesn't, also fine. > Working around this in Crux requires a lot of extra reference counting > complexity, because there are three different cases for reference > counts, depending on how many members there are in a ring (1, 2, or 3+ > members). Please show code. If, e.g., you have a vector v of length n holding the PyObject* "members", then int i; for (i = 0; i < self->n; ++i) Py_VISIT(self->v[i]); return 0; is an appropriate tp_traverse(). For example, that's exactly what the tp_traverse() for Python's builtin list objects does, and a Python list L can certainly be an element of itself: >>> L = [] >>> L.append(L) >>> L[0] is L True >>> There's no special-casing needed for the number of containees, or for the nature of what they do or don't point at. It's also irrelevant what can be reached *from* the direct containees. > 2) This issue is really just a different manifestation of issue (1). > > At the C (not Python object) level, each node only actually stores a > pointer to a single member of the associated ring. Please show code. If the C object contains exactly one pointer, then its tp_traverse should call visit() exactly once; etc. The correct C-level code has entirely to do with what the C struct contains, and nothing to do with the view presented at the Python level. > Given a single ring member, it is possible to traverse the ring and reach all > other ring members. That's irrelevant to gc. tp_traverse() is only responsible for revealing the objects *directly* reachable from self, or, more precisely, for revealing the PyObject* pointers on which self owns a reference. gc computes the transitive closure itself; tp_traverse() isn't responsible for that. > As mentioned in issue (1), the cyclic GC expects tp_traverse() > to call visit() once for each reference held. Yes. > It is not enough for a node to visit() one ring member; Don't know what it means for "a node to visit()". If the C struct has exactly one pointer, then it is enough for visit() to be invoked exactly once with that pointer. It would be incorrect not to visit() that pointer, and it would be incorrect to visit() more pointers than just that one. > it must visit() all ring members, If and only if for each ring member M, self contains a pointer directly to M. Showing code is much better than English. View your C-level objects as a directed graph. tp_traverse at a node `self` is reponsible for revealing self's outgoing edges, no less and no more than that. > in order for the GC to count how many references are from unreachable > objects, versus reachable from the root set. > > In summary, issues (1) and (2) are due to how the cyclic GC does the > "marking" phase of mark/sweep GC. It's not a mark/sweep collector. > My expectation of mark/sweep GC is You expectation is irrelevant <wink>, since it's not a mark/sweep collector. > that it should be sufficient to assure that all objects reachable from > the root set are visit()ed at least once; it should not be important how > many times each unreachable object is visit()ed. Forget roots sets and mark/sweep collectors. If you want to know how Python's gc actually works, read the comments in gcmodule.c's collect() function, and drill down from there as deep as your curiosity takes you. > I don't have a deep enough understanding of the Python interpreter to > give a detailed suggestion for improvement. I suspect that determining > the root set is not an easy operation; if this is the case, then I think > we're stuck with the current design. We are, but I don't feel "stuck" with it -- it's a very good design. > If determining the root set *is* easy (or even possible), then I would suggest > using a standard mark/sweep collector, where unreachable objects are > scheduled for destruction. The root set is impossible to determine without active cooperation from extension modules. Python's gc can infer the existence of roots, but not their identies; it can infer the transitive closure of what's reachable from the root set intersected with the set of objects that have registered with gc, but knows nothing about the roots themselves. > tp_traverse(), tp_clear(), and tp_dealloc() would retain > the same structure; the only difference would be the logic that > determines which objects can be destroyed. > > 3) A strange thing can happen when tp_clear() is called. Suppose that an > edge object is being cleared, and it drops its references to the > associated rings. If ob_refcnt of one of the rings drops to 0 as a > result, Python will tp_dealloc() the ring *right* *now*, without ever > calling tp_clear() on the ring. That depends on how the tp_dealloc() for the ring is coded, so is up to you. The only *necessary* thing for tp_clear() to do is to clear enough pointers to break cycles. Some people do only that much. For example, Python tuples don't even implement tp_clear, because a cycle involving tuples necessarily involves non-tuple objects too. Other people write tp_clear() so that it can be called from their tp_dealloc() function too. That's up to them. If you want to do the latter, it's good to write tp_clear() in such a way that calling it more than once is harmless; e.g., if (self->pa) { Py_DECREF(self->pa); self->pa = NULL; } etc or, using 2.4's Py_CLEAR macro, Py_CLEAR(self->pa); etc. > That means that I have to keep track of whether tp_clear() has been called > on objects, if it is at all important that tp_clear() be called, so that I can > manually do so in tp_dealloc(), As above, make your tp_clear() idempotent and then you don't have to keep track of anything. It's usually good practice for a tp_clear() to be robust against NULL pointers anyway. > if necessary. It is in my opinion reasonable to have cleanup code in > tp_clear(), with the assumption that it will be called precisely once, > but Python makes me do extra work to make sure that this happens. You won't get a guarantee that tp_clear() will be called exactly once. You have a guarantee that tp_dealloc() will be called no more than once. It's actually unusual to find a tp_dealloc() that calls its type's tp_clear(), but I think it's a nice pattern when you can get away with it. In such cases, tp_clear() should be coded so that it can be called multiple times, and that's generally very easy to do. > This should be pretty easy to change. A single bit per object is needed > to keep track of whether tp_clear() has been called. I think this only > needs to be done for object types that support cyclic GC. There's no such thing as "one bit" in reality: because of malloc memory padding, "one more bit" would require adding 4 bytes to every gc'able object. That's an extraordinary expense. Luckily, it's not needed. > 4) There is no way to control the order in which objects are > tp_dealloc()ed. It's true that there's no direct way to control the order. > This is a problem for the R-E-R construct, since at a low level, these objects > are always linked together. Many things are always linked together in the core too. Why that's "a problem" for you isn't clear. It's not a problem in the core, for example. > What I would like to do is defer tp_dealloc() on the edge until after both > rings have been deleted. Instead, I am forced to use a reference-counted > deletion function. Sorry, don't know what that means, and (as above) don't know why you care. > Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until > later is not a reasonable option, because this defers deletion of the > edge until a later round of garbage collection. > > This could be addressed in the Python interpreter by paying heed to the > return value of tp_dealloc(). If the return value is non-zero, move the > object to the end of the list of objects to be destroyed, Except there is no list of objects to be destroyed. > so that destruction is tried later. This allows the module to impose its own > destruction ordering. That part would need clearer motivation, preferably in the form of a PEP. From tim.peters at gmail.com Mon Oct 11 01:18:51 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 01:19:00 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <4169AB4B.9030605@v.loewis.de> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> Message-ID: <1f7befae04101016183eeb535d@mail.gmail.com> [Martin v. L?wis] > Hmmm. Even if a ring member holds only a single reference to another > ring member, would not calling visit() on this member in turn invoke > visit() on the other one, which would, in turn, invoke visit() on > the next one, and so on? Yes, but not recursively (which that description appears to imply), and not because of what's passed *to* visit(). More precisely, visit(some_pointer) does different things during different phases of gc (different phases pass different gcmodule.c callback functions to tp_traverse() implementations). None of them lead to recursive calls to the callback. Maybe a bit ironically, refcount-based deletion can require unbounded stack depth (hence the "trashcan" mechanism to avoid blowing the C stack), but Python's cyclic gc isn't recursive at all. I'm guessing that Jason is confusing the update_refs() and subtract_refs() phases of Python's gc with the "mark" phase of a mark/sweep collector. If so, the visit() callbacks used by those are actually irrelevant to the set of objects whose tp_traverse() is called during those phases. The tp_traverse() of *every* object in the generation being collected is invoked by that pair of phases, and the set of objects passed *to* the visit callbacks during those phases has no effect on the set of objects whose tp_traverse() gets called during those phases. So one of Jason's objects will get its tp_traverse() called by those phases if and only if it's in the generation currently being collected, and it's currently tracked by gc. It doesn't matter whether, e.g., anything points at it, or whether it points at anything (although if nothing anywhere points at it, it would already have been destroyed via refcounting). From theller at python.net Mon Oct 11 08:51:16 2004 From: theller at python.net (Thomas Heller) Date: Mon Oct 11 08:51:00 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ekk91duu.fsf@python.net> (Thomas Heller's message of "Fri, 08 Oct 2004 16:30:49 +0200") References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ekk91duu.fsf@python.net> Message-ID: <d5zpd9y3.fsf@python.net> "Fredrik Lundh" <fredrik@pythonware.com> writes: > Thomas Heller wrote: > >> How many _xxxx.pyd windows specific modules do we need? > > as many as it takes to make Python an excellent tool. it's not like one > more entry in the makefile, and 20k in the binary distribution will cause > problem for anyone. > > if you want to stop contributions to Python's standard library unless they > use your tools, you have a serious problem. Sorry if you misunderstood me, I didn't want to stop this contribution. Here is what I wrote, again: > I would suggest to replace the _subprocess extension module by a Python > implementation based on ctypes (and include ctypes in 2.4, at least for > Windows). > Thomas From jasone at canonware.com Mon Oct 11 08:59:52 2004 From: jasone at canonware.com (Jason Evans) Date: Mon Oct 11 08:59:30 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <4169AB4B.9030605@v.loewis.de> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> Message-ID: <20041011065952.GH341@canonware.com> On Sun, Oct 10, 2004 at 11:36:11PM +0200, "Martin v. L?wis" wrote: > >1) I originally had lone rings refer to themselves (ob_refcnt started out > > at 3; 2 self-references and one reference held by the associated edge). > > I would have thought the refcount should have been 4: two self > references, plus one from E, plus one from N (maybe I don't understand > what make a ring "lone", though). You understand correctly, except that the R-E-R construct starts out not being attached to any nodes. The "lone" ring term was my attempt to differentiate this: R from these: R-R R---R R-R \ / | | R R-R > > Working around this in Crux requires a lot of extra reference counting > > complexity, because there are three different cases for reference > > counts, depending on how many members there are in a ring (1, 2, or 3+ > > members). > > Why is that? Just DECREF every PyObject* in "self", then set the > PyObject* to NULL. No need for any additional reference counts. The low level tree code is implemented completely separately from the Python object wrapper code. This means that, for example, the Python "Node" object does not actually store PyObject* pointers to Ring objects; instead it uses the low level tree code to find out what Ring objects are attached to it. Crux was designed this way in order to be able to implement various low level algorithms such that they never have to call interpreter-related code. As such, reference breaking code must actually tear down the low level tree in such a way that it is always "consistent" (simply setting pointers to NULL doesn't fit well with this way of doing things). > Hmmm. Even if a ring member holds only a single reference to another > ring member, would not calling visit() on this member in turn invoke > visit() on the other one, which would, in turn, invoke visit() on > the next one, and so on? Yes, this is true. The problem I ran into had to do with the node logically holding references to all the ring objects, but only reporting one of them. That is, the node's tp_traverse function was not reporting all references, though it was reporting enough of them to guarantee that all reachable objects were visited. At this point, it is clear to me that Python's cyclic GC does not do mark/sweep GC, and that this won't change, so there's probably not much point in discussing this issue further. > >3) A strange thing can happen when tp_clear() is called. Suppose that an > > edge object is being cleared, and it drops its references to the > > associated rings. If ob_refcnt of one of the rings drops to 0 as a > > result, Python will tp_dealloc() the ring *right* *now*, without ever > > calling tp_clear() on the ring. That means that I have to keep track of > > whether tp_clear() has been called on objects, if it is at all important > > that tp_clear() be called, so that I can manually do so in tp_dealloc(), > > if necessary. It is in my opinion reasonable to have cleanup code in > > tp_clear(), with the assumption that it will be called precisely once, > > but Python makes me do extra work to make sure that this happens. > > Your expectation is wrong. tp_clear is not the place to perform cleanup. > It is the place to break cycles. tp_dealloc is the place to perform > cleanup. What if breaking cycles and cleanup are the same thing? In the case of Crux, they are one and the same, since the low level tree must be torn down one piece at a time. Apparently I was mistaken in viewing tp_clear and tp_dealloc as two stages of destruction. > >4) There is no way to control the order in which objects are > > tp_dealloc()ed. > > Correct. Unfortunately, there is no way to change that. For cyclic > structures, there is no natural "starting point". Python picks an > arbitrary starting point, assuming that the order does not matter. > If class objects have an __del__, it assumes that the order does > matter, and gives up - putting the objects into gc.garbage. > > > This could be addressed in the Python interpreter by paying heed to the > > return value of tp_dealloc(). If the return value is non-zero, move the > > object to the end of the list of objects to be destroyed, so that > > destruction is tried later. This allows the module to impose its own > > destruction ordering. > > I don't understand. tp_dealloc does not have a return value, so what > does "pay heed" mean? I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), the clear() call does not pay attention to the return value. It would be possible to check the return value, and defer deallocation if the return value were non-zero, and ob_refcnt were 1. (However, it would probably have to be a separate list of objects, contrary to what I claimed before.) Thanks for your comments, Martin. I'm starting to understand the Python GC world view a bit better. Jason From fredrik at pythonware.com Mon Oct 11 09:13:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Mon Oct 11 09:11:22 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se><Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ekk91duu.fsf@python.net> <d5zpd9y3.fsf@python.net> Message-ID: <ckdbmi$gdt$1@sea.gmane.org> Thomas Heller wrote: >> if you want to stop contributions to Python's standard library unless they >> use your tools, you have a serious problem. > > Sorry if you misunderstood me, I didn't want to stop this contribution. > Here is what I wrote, again: > >> I would suggest to replace the _subprocess extension module by a Python >> implementation based on ctypes (and include ctypes in 2.4, at least for >> Windows). if you're suggesting a rewrite for political reasons, you're effectively blocking this contribution. don't do that; python-dev have enough problems getting non- core contributions as it is. </F> From theller at python.net Mon Oct 11 09:26:35 2004 From: theller at python.net (Thomas Heller) Date: Mon Oct 11 09:26:20 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module In-Reply-To: <ca471dc20410091618487e6d01@mail.gmail.com> (Guido van Rossum's message of "Sat, 9 Oct 2004 16:18:29 -0700") References: <015001c4ae54$a7b32d40$a44a8890@neil> <ca471dc20410091618487e6d01@mail.gmail.com> Message-ID: <8yadd8b8.fsf@python.net> Guido van Rossum <gvanrossum@gmail.com> writes: > And what about all the stuff that's defined in header files? In the > past, in the Unix world, I experimented with translating .h files to > Python modules to save me from the burden of having to add large > numbers of symbol definitions to C extensions. (Look for files named > Lib/plat-XXX/regen in the source tree.) One by one, that approach has > proven to be problematic, and nearly all of those have eventually been > turned into systematic lists of symbol definitions in C code. See for > example the posix, socket, fcntl, and signal modules. I have two solutions here, both based on gccxml (see http://www.gccxml.org/). This is the C++ parser part of gcc, with a backend that turns the declaration in C header files into xml. gccxml runs on windows too, and since the installer even does some small patches to the MS header files to make them compliant, it parses the windows H files without a problem. You can even request it to simulate MSVC 6, MSVC 7.0, or MSVC 7.1. Now I have written an XML parser/Python code generator combo which generates a valid Python module from header files. This includes enums, structures, unions, and exported functions, but not #define symbol definitions, of course. You could even specify the symbols you want to be generated, it handles dependencies correctly and will include all that is needed. For the #define symbols, I let gccxml dump the preprocessor definitions after running it on the header files. This is to find the defined names. The actual code generation into Python code is done by a set of C++ function templates. This works better than Tools/scripts/h2py.py. Both things are still work in progress, but they do work. Thomas From astrand at lysator.liu.se Mon Oct 11 09:50:04 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Mon Oct 11 10:14:59 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ckbcpl$46a$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> <ck9918$kop$1@sea.gmane.org> <ubrfb8zua.fsf@yahoo.co.uk> <ckbcpl$46a$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410110905520.21470@koeberg.lysator.liu.se> On Sun, 10 Oct 2004, Jason Lunz wrote: > > The biggest problem on Windows is that not all executables use the > > Microsoft C runtime. Some use other C runtimes, others parse the > > command line directly and don't use argv at all. > > So why does subprocess use cmdline2list() in the parent on unix to > emulate the way a child subprocess might parse the string on windows? > (But only if it's written in C, uses the MSC runtime, and uses the > argv/argc handed to main() rather than calling GetCommandLine() itself). > Why not emulate CommandLineToArgvW()? or something else entirely? I > think it would be cleaner not to emulate anything at all. One goal with subprocess is being able to write cross-platform applications. For example, it should be possible to open up www.python.org in Mozilla. The best way to write this is: subprocess.call(["mozilla", "http://www.python.org"]) In this case, the list form is translated to the string form when running on Windows. Why allow the string form on UNIX? Answer: Symmetri, plus that some users that has been using os.system() for a long time thinks it's nice to be able to do: subprocess.call("ls -l /foo/bar") There's a risk that UNIX users might expect UNIX shell-like quoting support rather than the MSC one, though. > > The unix execv is just *different*. Both the Windows and the Unix > > interfaces have capabilities the other doesn't offer. > > Well, the windows interface is a subset of the unix one. The length of > argv on windows is limited to 1. True, if we are talking about the UNIX exec functions. When executing through a UNIX shell, the native interface is a string. > > I think Peter's approach of supporting both forms - a single string as > > a command line, and list of strings as an argv list, and converting > > both to the more natural OS-native form as needed, is sensible (I > > would, I argued for it when he was developing it!) > > I can see that it's trying to be symmetric and orthogonal, but I don't > think that the result is worth it in this case. In what scenario is the > use of cmdline2list() really useful? I don't really have a good example. If we should remove the cmdline2list stuff, what should happen if the users passes a string on UNIX? Do you prefer: 1) Run through the shell (automatic shell=True). or 2) A ValueError raised. I guess alternative 1 is most intuitive. That would line up with popen2 as well. Anyone objecting to this change? /Peter ?strand <astrand@lysator.liu.se> From anthony at interlink.com.au Mon Oct 11 15:09:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Oct 11 15:10:18 2004 Subject: [Python-Dev] compiler bug #1042238 is blocking the 2.4b1 release Message-ID: <416A861D.5020804@interlink.com.au> Anyone with knowledge of Lib/compiler is urged to examine www.python.org/sf/1042238. It's breaking on some genexprs, in particular in Lib/_strptime.py The failing code, distilled down, is: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'<string>', 'exec') It's the combination of the genexps and the function call with another argument. This bug is bad enough (IMHO) to block 2.4b1. Please, if you know about this, have a look at the failure. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From lunz at falooley.org Mon Oct 11 16:57:07 2004 From: lunz at falooley.org (Jason Lunz) Date: Mon Oct 11 16:57:16 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> <ck9918$kop$1@sea.gmane.org> <ubrfb8zua.fsf@yahoo.co.uk> <ckbcpl$46a$1@sea.gmane.org> <Pine.GSO.4.51L2.0410110905520.21470@koeberg.lysator.liu.se> Message-ID: <cke703$rlp$1@sea.gmane.org> astrand@lysator.liu.se said: > One goal with subprocess is being able to write cross-platform > applications. For example, it should be possible to open up www.python.org > in Mozilla. The best way to write this is: > > subprocess.call(["mozilla", "http://www.python.org"]) > > In this case, the list form is translated to the string form when running > on Windows. understood. I personally have a definite need for this in my programs, so I know what's involved. > There's a risk that UNIX users might expect UNIX shell-like quoting > support rather than the MSC one, though. exactly. I just see it confusing people. If someone wants simple string handling on unix, then shell=True. problem solved. If they need the type of portability referred to with your mozilla example, then they should use the list form and let windows do list2cmdline() on it. The converse case of using a string for windows and using cmdline2list() on it for unix is a good try, but there's no guarantee that the actual windows subprocess will even do the MSVCRT-compatible-cmdline2list() conversion. Which leaves you in a very confusing situation indeed. >> I can see that it's trying to be symmetric and orthogonal, but I don't >> think that the result is worth it in this case. In what scenario is the >> use of cmdline2list() really useful? > > I don't really have a good example. > > If we should remove the cmdline2list stuff, what should happen if the > users passes a string on UNIX? Do you prefer: > > 1) Run through the shell (automatic shell=True). > or > 2) A ValueError raised. > > I guess alternative 1 is most intuitive. That would line up with > popen2 as well. Use of the shell should be explicit, not automatic, because of the usual shell metacharacter security concerns. unix programmers used to doing os.system('ls -l') will quickly learn that the subprocess way of doing the same is subprocess.call('ls -l', shell=True). This has the added benifit of making it obvious exactly what's happening. I don't think that the only alternative to number 1) is to raise a ValueError. What do you think of the below patch? Just listify bare strings on unix. This does exactly what it should when the string actually references a binary, and gives a meaningful error when it doesn't. Even if the filename has strange characters of some kind. $ echo '#! /bin/sh' > 'ls -l' $ echo 'echo foo' >> 'ls -l' $ cat 'ls -l' #! /bin/sh echo foo $ python Python 2.3.4 (#2, Sep 24 2004, 08:39:09) >>> from subprocess import call >>> call('./ls -l') Traceback (most recent call last): File "<stdin>", line 1, in ? File "subprocess.py", line 441, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 524, in __init__ errread, errwrite) File "subprocess.py", line 942, in _execute_child raise child_exception OSError: [Errno 13] Permission denied >>> $ chmod +x 'ls -l' $ python Python 2.3.4 (#2, Sep 24 2004, 08:39:09) >>> from subprocess import call >>> call('./ls -l') foo 0 >>> call('./ls -blah') Traceback (most recent call last): File "<stdin>", line 1, in ? File "subprocess.py", line 441, in call return Popen(*args, **kwargs).wait() File "subprocess.py", line 524, in __init__ errread, errwrite) File "subprocess.py", line 942, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Nice, straightforward, easy to understand. And look how much code is removed -- I haven't even cleaned up the corresponding docs yet. Index: subprocess.py =================================================================== RCS file: /cvsroot/python-popen5/popen5/subprocess.py,v retrieving revision 1.15 diff -u -p -r1.15 subprocess.py --- subprocess.py 9 Oct 2004 10:11:06 -0000 1.15 +++ subprocess.py 11 Oct 2004 14:56:26 -0000 @@ -854,11 +854,11 @@ class Popen: errread, errwrite): """Execute program (POSIX version)""" + if isinstance(args, types.StringTypes): + args = [args] + if shell: args = ["/bin/sh", "-c"] + args - else: - if isinstance(args, types.StringTypes): - args = self.cmdline2list(args) if executable == None: executable = args[0] @@ -1051,93 +1051,6 @@ class Popen: self.wait() return (stdout, stderr) - - - def cmdline2list(cmdline): - """ - Translate a command line string into a list of arguments, using - using the same rules as the MS C runtime: - - 1) Arguments are delimited by white space, which is either a - space or a tab. - - 2) A string surrounded by double quotation marks is - interpreted as a single argument, regardless of white space - contained within. A quoted string can be embedded in an - argument. - - 3) A double quotation mark preceded by a backslash is - interpreted as a literal double quotation mark. - - 4) Backslashes are interpreted literally, unless they - immediately precede a double quotation mark. - - 5) If backslashes immediately precede a double quotation mark, - every pair of backslashes is interpreted as a literal - backslash. If the number of backslashes is odd, the last - backslash escapes the next double quotation mark as - described in rule 3. - """ - - # See - # http://msdn.microsoft.com/library/en-us/vccelng/htm/progs_12.asp - - # Step 1: Translate all literal quotes into QUOTE. Justify number - # of backspaces before quotes. - tokens = [] - bs_buf = "" - QUOTE = 1 # \", literal quote - for c in cmdline: - if c == '\\': - bs_buf += c - elif c == '"' and bs_buf: - # A quote preceded by some number of backslashes. - num_bs = len(bs_buf) - tokens.extend(["\\"] * (num_bs//2)) - bs_buf = "" - if num_bs % 2: - # Odd. Quote should be placed literally in array - tokens.append(QUOTE) - else: - # Even. This quote serves as a string delimiter - tokens.append('"') - - else: - # Normal character (or quote without any preceding - # backslashes) - if bs_buf: - # We have backspaces in buffer. Output these. - tokens.extend(list(bs_buf)) - bs_buf = "" - - tokens.append(c) - - # Step 2: split into arguments - result = [] # Array of strings - quoted = False - arg = [] # Current argument - tokens.append(" ") - for c in tokens: - if c == '"': - # Toggle quote status - quoted = not quoted - elif c == QUOTE: - arg.append('"') - elif c in (' ', '\t'): - if quoted: - arg.append(c) - else: - # End of argument. Output, if anything. - if arg: - result.append(''.join(arg)) - arg = [] - else: - # Normal character - arg.append(c) - - return result - - cmdline2list = staticmethod(cmdline2list) def list2cmdline(seq): From seojiwon at gmail.com Mon Oct 11 17:01:24 2004 From: seojiwon at gmail.com (Jiwon Seo) Date: Mon Oct 11 17:02:05 2004 Subject: [Python-Dev] compiler bug #1042238 is blocking the 2.4b1 release In-Reply-To: <416A861D.5020804@interlink.com.au> References: <416A861D.5020804@interlink.com.au> Message-ID: <b008462b04101108012c8da372@mail.gmail.com> Uploaded patch for the bug. The bug was due to a wrong condition check for allowing f(x for x in y) but rejecting f(x for x in y, 1). The patch fixes the bug alright, but you might want to test more. =) Jiwon. On Mon, 11 Oct 2004 23:09:49 +1000, Anthony Baxter <anthony@interlink.com.au> wrote: > Anyone with knowledge of Lib/compiler is urged to examine > www.python.org/sf/1042238. It's breaking on some genexprs, > in particular in Lib/_strptime.py > > The failing code, distilled down, is: > > import compiler > a="""def foo(*x): return x > foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" > compiler.compile(a,'<string>', 'exec') > > It's the combination of the genexps and the function call > with another argument. > > This bug is bad enough (IMHO) to block 2.4b1. Please, > if you know about this, have a look at the failure. > > Anthony > > -- > Anthony Baxter <anthony@interlink.com.au> > It's never too late to have a happy childhood. > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/seojiwon%40gmail.com > From mwh at python.net Mon Oct 11 17:12:53 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 11 17:12:55 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS Message-ID: <2mfz4lgufe.fsf@starship.python.net> As subject. Is this deliberate? It breaks test_compiler, though that's possibly a bug because test_compiler should be using universal newlines modes when opening the file, but still... Cheers, mwh -- The word "Fascism" has now no meaning except in so far as it signifies 'something not desirable'. -- George Orwell in "Politics and the English Language" From gvanrossum at gmail.com Mon Oct 11 17:17:51 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 11 17:17:56 2004 Subject: [Python-Dev] Re: Re: Re: subprocess - updated popen5 module In-Reply-To: <ckalfo$l7h$1@sea.gmane.org> References: <015001c4ae54$a7b32d40$a44a8890@neil> <ca471dc20410091618487e6d01@mail.gmail.com> <ckalfo$l7h$1@sea.gmane.org> Message-ID: <ca471dc20410110817127e6f3a@mail.gmail.com> [Guido] > > I don't know. Writing Windows interface modules is a highly > > specialized form of torture, requiring arcane expertise. [Fredrik] > Despite that, some of us can do this in our sleep (including writing the bindings). > I find it somewhat sad that some people don't want us to contribute to Python. Hey, I wrote that in *support* of you! > While we have you on line, can you pronounce on the subprocess module? The > code is written by highly experienced programmers, it has been widely tested, > it's entirely self-contained, builds and runs under 2.2 and later, has an extensive > test suite and a PEP (which can be quickly turned into a library reference section), > and replaces several fragile API:s in the current library with a single, well-defined > construct. If this isn't good enough for the standard library, I don't see what is. I expect it is, and I value your judgement. I have recently experimented with a cross-platform process management abstraction myself, and I'd like to check out the subprocess module to make sure that it can do everything that I put in my own code (not very much but some specific requirements about I/O redirection, current directory, and setting the priority. I hope to get back to this later today. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Mon Oct 11 17:21:10 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 17:21:18 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <2mfz4lgufe.fsf@starship.python.net> References: <2mfz4lgufe.fsf@starship.python.net> Message-ID: <1f7befae04101108212f0a528@mail.gmail.com> [Michael Hudson] > As subject. Is this deliberate? Don't know, but guess so: it contains bytes outside the set ANSI C says can be used portably in text files. > It breaks test_compiler, That must be platform-specific damage, but you haven't identified your platform. > though that's possibly a bug because test_compiler should be using universal > newlines modes when opening the file, Probably, yes. > but still... So stop whining and fix it <wink>. From mwh at python.net Mon Oct 11 17:25:21 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 11 17:25:23 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <1f7befae04101108212f0a528@mail.gmail.com> (Tim Peters's message of "Mon, 11 Oct 2004 11:21:10 -0400") References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> Message-ID: <2m8yadgtum.fsf@starship.python.net> Tim Peters <tim.peters@gmail.com> writes: > [Michael Hudson] >> As subject. Is this deliberate? > > Don't know, but guess so: it contains bytes outside the set ANSI C > says can be used portably in text files. > >> It breaks test_compiler, > > That must be platform-specific damage, but you haven't identified your platform. Well, yes, but this information isn't especially relavent. It's obviously a different platform from whoever edited the file last... I'm on linux, though. >> though that's possibly a bug because test_compiler should be using universal >> newlines modes when opening the file, > > Probably, yes. > >> but still... > > So stop whining and fix it <wink>. I have done and am waiting for test_compiler to finish ... that allows for a lot of whining :) Cheers, mwh -- There are two kinds of large software systems: those that evolved from small systems and those that don't work. -- Seen on slashdot.org, then quoted by amk From sjoerd at acm.org Mon Oct 11 18:00:03 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Mon Oct 11 18:00:39 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <1f7befae04101108212f0a528@mail.gmail.com> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> Message-ID: <416AAE03.9020209@acm.org> Tim Peters wrote: > [Michael Hudson] > >>As subject. Is this deliberate? > > > Don't know, but guess so: it contains bytes outside the set ANSI C > says can be used portably in text files. However, this is not necessarily enough reason to use -kb. The only things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for updates, and not expanding $ keywords. None of those matter in this file. > >>It breaks test_compiler, > > > That must be platform-specific damage, but you haven't identified your platform. > > >>though that's possibly a bug because test_compiler should be using universal >>newlines modes when opening the file, > > > Probably, yes. > > >>but still... > > > So stop whining and fix it <wink>. > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sjoerd%40acm.org -- Sjoerd Mullender <sjoerd@acm.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041011/cb1df212/signature-0001.pgp From tim.peters at gmail.com Mon Oct 11 19:32:34 2004 From: tim.peters at gmail.com (Tim Peters) Date: Mon Oct 11 19:32:55 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AAE03.9020209@acm.org> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> Message-ID: <1f7befae04101110325f8f0189@mail.gmail.com> [Sjoerd Mullender] > However, this is not necessarily enough reason to use -kb. The only > things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for > updates, and not expanding $ keywords. None of those matter in this file. So fix it. From nbastin at opnet.com Mon Oct 11 21:14:52 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Oct 11 21:15:14 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041011065952.GH341@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> Message-ID: <D3FF724A-1BB9-11D9-AF10-000D932927FE@opnet.com> On Oct 11, 2004, at 2:59 AM, Jason Evans wrote: >>> Working around this in Crux requires a lot of extra reference >>> counting >>> complexity, because there are three different cases for reference >>> counts, depending on how many members there are in a ring (1, 2, >>> or 3+ >>> members). >> >> Why is that? Just DECREF every PyObject* in "self", then set the >> PyObject* to NULL. No need for any additional reference counts. > > The low level tree code is implemented completely separately from the > Python object wrapper code. This means that, for example, the Python > "Node" object does not actually store PyObject* pointers to Ring > objects; > instead it uses the low level tree code to find out what Ring objects > are > attached to it. Crux was designed this way in order to be able to > implement various low level algorithms such that they never have to > call > interpreter-related code. As such, reference breaking code must > actually > tear down the low level tree in such a way that it is always > "consistent" > (simply setting pointers to NULL doesn't fit well with this way of > doing > things). You've now reached the point where the Python-GC-ease-of-implementation breaks down. We have encountered the same problem - the GC is fine as long as you're solely extending Python, but if you're embedding it, chances are you'll encounter some interesting issues like this along the way. It's not that you can't make it work - you can, but need to do a lot more work yourself. Basically, if I understand you correctly (and I may not be), there are times when you do not want tp_dealloc to actually destroy the low-level data structure, because somebody else (not living in the Python reference-counted world) isn't done with it yet. The problem is in knowing when those times are, and when they aren't. In this case, you may need to implement basic reference counting in your own low-level data structures, so you know when you're really done with a ring and ready to destroy it. The other option is to attempt to make sure that you're not creating cycles (in python) and avoid interfacing with GC entirely, but that may not be possible in your case. -- Nick From martin at v.loewis.de Mon Oct 11 22:50:21 2004 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon Oct 11 22:50:22 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AAE03.9020209@acm.org> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> Message-ID: <416AF20D.6010706@v.loewis.de> Sjoerd Mullender wrote: >> Don't know, but guess so: it contains bytes outside the set ANSI C >> says can be used portably in text files. > > > However, this is not necessarily enough reason to use -kb. The only > things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for > updates, and not expanding $ keywords. That is not true. On Apple computers, it also avoids conversion from Latin-1 to Mac-Roman, which Mac CVS does by default for text files. Making the files binary is the only way to avoid this conversion, and that is precisely the reason why the file is binary. You may argue that this is a bug in Mac CVS, and I would agree. However, that specific bug has -kb as a known work-around, and the issue reported here points to a bug in the compiler packages which should be fixed rather than worked-around. Regards, Martin From astrand at lysator.liu.se Mon Oct 11 22:41:38 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Mon Oct 11 23:04:01 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <cke703$rlp$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ck6lhj$k72$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091157420.6228@koeberg.lysator.liu.se> <ck8gjs$1qe$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091350260.6824@koeberg.lysator.liu.se> <ck8qid$ln6$1@sea.gmane.org> <Pine.GSO.4.51L2.0410091558190.7564@koeberg.lysator.liu.se> <ck9918$kop$1@sea.gmane.org> <ubrfb8zua.fsf@yahoo.co.uk> <ckbcpl$46a$1@sea.gmane.org> <Pine.GSO.4.51L2.0410110905520.21470@koeberg.lysator.liu.se> <cke703$rlp$1@sea.gmane.org> Message-ID: <Pine.GSO.4.51L2.0410111923570.24904@koeberg.lysator.liu.se> On Mon, 11 Oct 2004, Jason Lunz wrote: > > If we should remove the cmdline2list stuff, what should happen if the > > users passes a string on UNIX? Do you prefer: > > > > 1) Run through the shell (automatic shell=True). > > or > > 2) A ValueError raised. > > > > I guess alternative 1 is most intuitive. That would line up with > > popen2 as well. > > Use of the shell should be explicit, not automatic, because of the usual > shell metacharacter security concerns. unix programmers used to doing > os.system('ls -l') will quickly learn that the subprocess way of doing > the same is subprocess.call('ls -l', shell=True). This has the added > benifit of making it obvious exactly what's happening. Good points. > I don't think that the only alternative to number 1) is to raise a ValueError. > > What do you think of the below patch? Just listify bare strings on unix. This > does exactly what it should when the string actually references a binary, and > gives a meaningful error when it doesn't. Even if the filename has strange > characters of some kind. I like it. I've submitted your patch (but with documentation updates as well). /Peter ?strand <astrand@lysator.liu.se> From gvanrossum at gmail.com Mon Oct 11 23:18:49 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 11 23:18:52 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> Message-ID: <ca471dc20410111418614a0694@mail.gmail.com> > I've recieved very positive feedback on my module. Many users are also > asking me if this module will be included in the standard library. That > is, of course, my wish as well. > > So, can the subprocess module be accepted? If not, what needs to be done? FWIW, I've given Peter some feedback, but in general I think this code is ready for inclusion into 2.4. I hope it's not too late. (My recommendation on the call/callv issue is to get rid of callv(), and instead let call() interpret multiple string args as such; a single string arg will be interpreted as a whole command line, per Popen(); if you want a single file arg you have to enclose it in [...].) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Tue Oct 12 00:34:58 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Oct 12 00:48:00 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <ca471dc20410111418614a0694@mail.gmail.com> References: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> Message-ID: <20041011152210.08C1.JCARLSON@uci.edu> [snip Guido's mention of subprocess being included in Python 2.4, if at all possible] Speaking of inclusion, we seemed to have come upon a stalemate with the 'gG' format code additions. Some users (without commit access to Python CVS) have stated they would use it (and currently write their own packing and unpacking functions), but some developers (with commit access to Python CVS) have stated they would never use it. At that point, the discussion pretty much ended. I know that the request is coming in a bit late in the 2.4 development cycle, but waiting until Python 2.5 seems like a bit much, considering that the functionality is so simple it can be proven correct quite easily (if it is to be included). A pronouncement would be nice, but I understand that I am in no position to ask/beg for a (yes, we'll add it) response. <.5 wink> Thank you, - Josiah P.S. Note that everyone involved believes that binascii.from/to_long should be added (I think it was even pronounced by Guido). From gvanrossum at gmail.com Tue Oct 12 01:12:38 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 12 01:12:40 2004 Subject: [Python-Dev] Re: Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011152210.08C1.JCARLSON@uci.edu> References: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <20041011152210.08C1.JCARLSON@uci.edu> Message-ID: <ca471dc2041011161243fe34d8@mail.gmail.com> > [snip Guido's mention of subprocess being included in Python 2.4, if at > all possible] > > Speaking of inclusion, we seemed to have come upon a stalemate with the > 'gG' format code additions. Some users (without commit access to Python > CVS) have stated they would use it (and currently write their own > packing and unpacking functions), but some developers (with commit > access to Python CVS) have stated they would never use it. > > At that point, the discussion pretty much ended. > > I know that the request is coming in a bit late in the 2.4 development > cycle, but waiting until Python 2.5 seems like a bit much, considering > that the functionality is so simple it can be proven correct quite > easily (if it is to be included). > > A pronouncement would be nice, but I understand that I am in no position > to ask/beg for a (yes, we'll add it) response. <.5 wink> > > Thank you, > - Josiah > > P.S. > Note that everyone involved believes that binascii.from/to_long should > be added (I think it was even pronounced by Guido). I'm in favor of adding the latter only. Is there a patch? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Tue Oct 12 01:36:32 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 01:37:34 2004 Subject: [Python-Dev] Re: Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <ca471dc2041011161243fe34d8@mail.gmail.com> References: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <20041011152210.08C1.JCARLSON@uci.edu> <ca471dc2041011161243fe34d8@mail.gmail.com> Message-ID: <1f7befae04101116361b3a5185@mail.gmail.com> [Josiah Carlson] >> Note that everyone involved believes that binascii.from/to_long should >> be added (I think it was even pronounced by Guido). [Guido] > I'm in favor of adding the latter only. Is there a patch? No. It would be easy enough, but nobody had time for it. AFAICT, not even the signatures for such functions have been fully worked out yet. Since it would be a small and self-contained addition, I wouldn't worry about adding it after the beta. I wouldn't worry much more about adding g/G after either, for that matter. From tim.peters at gmail.com Tue Oct 12 03:21:57 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 03:22:07 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041011065952.GH341@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> Message-ID: <1f7befae041011182151f7b219@mail.gmail.com> [Jason Evans] ... > The low level tree code is implemented completely separately from the > Python object wrapper code. This means that, for example, the Python > "Node" object does not actually store PyObject* pointers to Ring objects; > instead it uses the low level tree code to find out what Ring objects are > attached to it. I think you're in for a rude awakening if you expect any language's gc support to help you manage arbitrary blobs of memory obtained from the system malloc(). If all the nodes in your object graph derived from PyObject, Python's gc would do it all for you. If you want to plant pointers in and among arbitrary C structs, then you'll have to implement your own gc, or hope that you can bash a general gc thingie (like BDW -- and I'm not sure there is anything else of that sort usefully available, so "like" is probably optimistic) into guessing the layout of your structs. > Crux was designed this way in order to be able to implement various low level > algorithms such that they never have to call interpreter-related code. I don't know why that was thought to be desirable, or even exactly what it means. One plausible meaning is that you don't want to tie the "various low level algorithms" to Python, because you hope to reuse them in other, non-Python contexts. And, in addition to that, you don't want, or don't know how, to abstract the necessary operations into macros or functions that are pluggable for the different contexts you have in mind. As a very simple example, using Py_INCREF in a function doesn't tie that function to Python -- if you want to use it in a non-Python context, #define Py_INCREF(x) gets rid of such calls. But I'm guessing too much about what you might mean, so I'll stop there. > As such, reference breaking code must actually tear down the low level tree in > such a way that it is always "consistent" (simply setting pointers to NULL > doesn't fit well with this way of doing things). Martin had in mind that the pointers were to properly refcounted PyObject*, in which case his advice was sound. [Martin] >> Hmmm. Even if a ring member holds only a single reference to another >> ring member, would not calling visit() on this member in turn invoke >> visit() on the other one, which would, in turn, invoke visit() on >> the next one, and so on? [Jason] > Yes, this is true. It's false, but I covered that one yesterday. What's passed back to visit() has nothing to do with which objects' tp_traverse()s get called. Registering ("tracking") PyObject*s of gc'able types is what determines which objects' tp_traverse()s get called. Python's gc doesn't try to determine what's reachable, it determines what isn't reachable from pointers it *hasn't* been told about. It infers "reachable from outside" from the refcounts: if all the refcounts on an object are not accounted for by the objects gc *has* been told about, then that object is directly reachable from something gc hasn't been told about, so gc dare not touch it. The transitive closure of what's reachable from such objects is the set of all objects gc knows about that are reachable from outside. gc leaves those alone. Everything else gc knows about is necessarily cyclic trash, or reachable only from cyclic trash. It can't know anything about objects that aren't tracked by gc, so in particular can't know anything about pointers that don't point at a PyObject. > The problem I ran into had to do with the node logically holding references to all > the ring objects, but only reporting one of them. If a ring object doesn't derive from PyObject, then visit() must not be passed a pointer to a ring object. Doing so would account for many catastrophic meanings of "doesn't work" <wink>. > That is, the node's tp_traverse function was not reporting all references, though > it was reporting enough of them to guarantee that all reachable objects were > visited. No, the visit() callback has nothing to do with which objects are traversed. gc determines the set of objects to traverse, before any object's tp_traverse() has been called, as a "generational" subset of all objects that have registered with gc. The pointers passed back to visit() callbacks have no effect on that set: they can't add anything to, or remove anything from, that set. The set of objects to traverse is predetermined. > At this point, it is clear to me that Python's cyclic GC does not do > mark/sweep GC, RIght, although we're only talking about CPython here (a particular implementation of Python), > and that this won't change, In CPython that's a safe bet. > so there's probably not much point in discussing this issue further. We can help you use Python's gc, but not if you're unwilling or unable to change anything about what you do. > ... > What if breaking cycles and cleanup are the same thing? In the case of > Crux, they are one and the same, since the low level tree must be torn down > one piece at a time. Again Martin had in mind that all your objects derived from PyObject. If that were so, then Python's gc would guarantee not to tear down anything until every cycle it was part of became unreachable, in which case internal invariants no longer matter (since they're no longer reachable form anything except other unreachable trash). Python's gc is very nice to work with, for Python objects. If you can't make your objects PyObjects, then you get all the pain of implementing a form of gc that does work with your objects. You really should look at how core objects implement tp_traverse() and tp_clear(), to see how easy life *can* be, even in the presence of much hairier object graphs than you're working with. > I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), > the clear() call does not pay attention to the return value. Heh. So true! See http://mail.python.org/pipermail/python-dev/2003-April/034435.html > It would be possible to check the return value, and defer deallocation if the > return value were non-zero, and ob_refcnt were 1. (However, it would probably > have to be a separate list of objects, contrary to what I claimed before.) I'm not sure, but I think your desire for this was still based on misunderstanding of what Python's gc does. From jasone at canonware.com Tue Oct 12 04:25:47 2004 From: jasone at canonware.com (Jason Evans) Date: Tue Oct 12 04:25:23 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <1f7befae041011182151f7b219@mail.gmail.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> Message-ID: <20041012022547.GC27613@canonware.com> On Mon, Oct 11, 2004 at 09:21:57PM -0400, Tim Peters wrote: > [Jason Evans] > > I think you're in for a rude awakening if you expect any language's gc > support to help you manage arbitrary blobs of memory obtained from the > system malloc(). I don't expect that. What I do expect is for there to be clean and efficient mechanisms for integrating custom Python object types into Python. Python more or less meets this expectation. > If all the nodes in your object graph derived from PyObject, Python's gc > would do it all for you. If you want to plant pointers in and among > arbitrary C structs, then you'll have to implement your own gc, or hope > that you can bash a general gc thingie (like BDW -- and I'm not sure > there is anything else of that sort usefully available, so "like" is > probably optimistic) into guessing the layout of your structs. No, this is not necessary. What is necessary is for my code to accurately maintain and report all references to Python objects during traversal. The Crux code does so, and works as designed. > > Crux was designed this way in order to be able to implement various low level > > algorithms such that they never have to call interpreter-related code. > > I don't know why that was thought to be desirable, or even exactly > what it means. One plausible meaning is that you don't want to tie > the "various low level algorithms" to Python, because you hope to > reuse them in other, non-Python contexts. And, in addition to that, > you don't want, or don't know how, to abstract the necessary > operations into macros or functions that are pluggable for the > different contexts you have in mind. First, "hoping" to be able use my code in other contexts is what allowed me to transition from using a different embeddable interpreter to using Python inside of two days. Had I melded all of the scripting glue code with the low level tree code, it would not have been nearly as easy to transition to using Python. Second, I have the scripting language glue and low level tree code separated for two reasons: 1) performance, and 2) modularization. There's no question of whether I want to use cpp magic to be able to munge two large piles of code into one huge pile of code, nor whether I'm capable of doing so. To do so would simply take the code in a bad direction. > [Martin] > >> Hmmm. Even if a ring member holds only a single reference to another > >> ring member, would not calling visit() on this member in turn invoke > >> visit() on the other one, which would, in turn, invoke visit() on > >> the next one, and so on? > > [Jason] > > Yes, this is true. > > It's false, but I covered that one yesterday. It is true, given my interpretation of the original context of the dialogue. We were discussing whether it would work for a node object to hold a reference to a single ring object, and rely on the ring objects to report their neighbors. Indeed this would work. > > The problem I ran into had to do with the node logically holding references to all > > the ring objects, but only reporting one of them. > > If a ring object doesn't derive from PyObject, then visit() must not > be passed a pointer to a ring object. Doing so would account for many > catastrophic meanings of "doesn't work" <wink>. If you want to see the Crux source code, I'll be happy to provide you with a source archive. In the meanwhile, please rest assured that the code doesn't suck nearly as badly as you are insinuating here. > We can help you use Python's gc, but not if you're unwilling or unable > to change anything about what you do. Prior to switching to using Python, Crux integrated with a language that implemented atomic mark and sweep garbage collection. I have had to change a tremendous amount about Crux in order to use a reference counted interpreter. In fact, I made some changes to Crux this morning that were a direct result of your original response to my email (ring objects now report references to themselves, which makes reference counting of ring objects much simpler). Many of the issues I ran into had to do with the impedence mismatch between my previous experience and the way Python works. I appreciate you taking the time to respond to me, especially since you've helped me gain a deeper understanding of how Python's GC works. However, please get it out of your mind that I am incompetent and/or set in my ways. > > I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), > > the clear() call does not pay attention to the return value. > > Heh. So true! See > > http://mail.python.org/pipermail/python-dev/2003-April/034435.html > > > It would be possible to check the return value, and defer deallocation if the > > return value were non-zero, and ob_refcnt were 1. (However, it would probably > > have to be a separate list of objects, contrary to what I claimed before.) > > I'm not sure, but I think your desire for this was still based on > misunderstanding of what Python's gc does. Of the original issues I brought up, I think this one potentially still has merit, though I'm not sure that it's of sufficiently general utility. The idea is to allow extension modules some control over the order in which objects are deallocated, so that it is possible to tear down interconnections in a particular order. However, it occurs to me that since tp_clear can leave references in place, it might be possible to control the object destruction order by leaving a DAG of references in place, and letting Python deallocate objects in dependency order. Does this work? Thanks, Jason From tim.peters at gmail.com Tue Oct 12 05:34:34 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 12 05:34:43 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041012022547.GC27613@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> Message-ID: <1f7befae041011203471688554@mail.gmail.com> [Tim Peters] >> I think you're in for a rude awakening if you expect any language's gc >> support to help you manage arbitrary blobs of memory obtained from the >> system malloc(). [Jason Evans] > I don't expect that. What I do expect is for there to be clean and > efficient mechanisms for integrating custom Python object types into > Python. Python more or less meets this expectation. OK ... but you must want more than just that, else this thread wouldn't have started <wink>. >> If all the nodes in your object graph derived from PyObject, Python's gc >> would do it all for you. If you want to plant pointers in and among >> arbitrary C structs, then you'll have to implement your own gc, or hope >> that you can bash a general gc thingie (like BDW -- and I'm not sure >> there is anything else of that sort usefully available, so "like" is >> probably optimistic) into guessing the layout of your structs. > No, this is not necessary. What is necessary is for my code to accurately > maintain and report all references to Python objects during traversal. The > Crux code does so, and works as designed. Isn't it also necessary for your code to clean up object graphs where the nodes are *not* Python objects? That's what I thought you were saying to Martin, If I misunderstood that, I apologize. Almost everything else I said was based on that belief. >>> Crux was designed this way in order to be able to implement various low level >>> algorithms such that they never have to call interpreter-related code. >> I don't know why that was thought to be desirable, or even exactly >> what it means. One plausible meaning is that you don't want to tie >> the "various low level algorithms" to Python, because you hope to >> reuse them in other, non-Python contexts. And, in addition to that, >> you don't want, or don't know how, to abstract the necessary >> operations into macros or functions that are pluggable for the > > different contexts you have in mind. > First, "hoping" to be able use my code in other contexts is what allowed me > to transition from using a different embeddable interpreter to using Python > inside of two days. Had I melded all of the scripting glue code with the > low level tree code, it would not have been nearly as easy to transition to > using Python. So the "you don't want to tie ... to Python" guess was right (please realize that you haven't *said* so before), and the "don't want to abstract ... into macros or functions" part of the other guess was on target (ditto). I think you're reading judgments into my attempts to extract answers. It's fine by me if you don't want to make everything a PyObject, but then I'm also trying to explain consequences of that choice wrt what Python's gc will and won't do for you. > Second, I have the scripting language glue and low level tree code > separated for two reasons: 1) performance, and 2) modularization. There's > no question of whether I want to use cpp magic to be able to munge two > large piles of code into one huge pile of code, nor whether I'm capable of > doing so. To do so would simply take the code in a bad direction. Maybe. I resist some here because adding just enough PyObject hair to enable a C struct to work smoothly with Python's native gc doesn't usually count as a "large pile" of code by any metric I use, and really does make gc life easy. [Martin] >>>> Hmmm. Even if a ring member holds only a single reference to another >>>> ring member, would not calling visit() on this member in turn invoke >>>> visit() on the other one, which would, in turn, invoke visit() on >>>> the next one, and so on? [Jason] >>> Yes, this is true. >> It's false, but I covered that one yesterday. > It is true, given my interpretation of the original context of the > dialogue. We were discussing whether it would work for a node object to > hold a reference to a single ring object, and rely on the ring objects to > report their neighbors. Indeed this would work. Regardless of context, in Python's gc there is no sense in which calling a visit() callback "in turn" invokes visit() on anything. If there is a sense here in which that is true, the context must be something other than Python's gc. If the ring objects are not PyObject themselves (I thought it was clear that they weren't, but maybe that was wrong), then Python's gc gives them no way to "report their neighbors". >>> The problem I ran into had to do with the node logically holding references to >>> all the ring objects, but only reporting one of them. >> If a ring object doesn't derive from PyObject, then visit() must not >> be passed a pointer to a ring object. Doing so would account for many >> catastrophic meanings of "doesn't work" <wink>. > If you want to see the Crux source code, I'll be happy to provide you with > a source archive. I'd like to say "yes", but I'll never make time to look at it. I would have been happier here if you had confirmed or denied that ring objects don't derive from PyObject. I pressed before on that "doesn't work" isn't helpful, and that never got clarified. Possible meanings include segfaults, error messages, Python exceptions. and memory leaks. > In the meanwhile, please rest assured that the code doesn't suck nearly as > badly as you are insinuating here. I'm not insinuating anything about the code, I'm trying to guess what you meant in your original post by "doesn't work" this-and-that. Passing a non-PyObject* to visit() remains a possible cause for "doesn't work" in my head, because you haven't said anything to rule it out. If you in fact *had* been passing a non-PyObject* to visit(), then the paragraph above could have been helpful ("ah, that's why it didn't work!"). >> We can help you use Python's gc, but not if you're unwilling or unable >> to change anything about what you do. > Prior to switching to using Python, Crux integrated with a language that > implemented atomic mark and sweep garbage collection. I have had to change > a tremendous amount about Crux in order to use a reference counted > interpreter. In fact, I made some changes to Crux this morning that were a > direct result of your original response to my email (ring objects now > report references to themselves, which makes reference counting of ring > objects much simpler). So Ring objects do derive from PyObject*. Forget everything I said then <wink>. > Many of the issues I ran into had to do with the impedence mismatch between > my previous experience and the way Python works. I sure believe that. Python's cyclic gc is unusual in several respects. > I appreciate you taking the time to respond to me, especially since you've > helped me gain a deeper understanding of how Python's GC works. However, > please get it out of your mind that I am incompetent and/or set in my ways. Those aren't in my head. I do think you're so steeped in the details of your code that you're not nearly as successful at communicating the *salient* points as you think you are, though. I'm so steeped in the details of Python's gc that a similar claim could easily be made of me. I don't expect it to be clear the first time, though, so play 20-questions trying to tease out what you really need to know about it. If it's any consolation, this process has been painful on both sides <wink>. >>> I should have said tp_clear here. In Modules/gcmodule.c:delete_garbage(), >>> the clear() call does not pay attention to the return value. >> Heh. So true! See >> >> http://mail.python.org/pipermail/python-dev/2003-April/034435.html >>> It would be possible to check the return value, and defer deallocatin if the >>> return value were non-zero, and ob_refcnt were 1. (However, it would probably >>> have to be a separate list of objects, contrary to what I claimed before.) >> I'm not sure, but I think your desire for this was still based on >> misunderstanding of what Python's gc does. > Of the original issues I brought up, I think this one potentially still has > merit, though I'm not sure that it's of sufficiently general utility. The > idea is to allow extension modules some control over the order in which > objects are deallocated, so that it is possible to tear down > interconnections in a particular order. Well, think about it some more then. "A separate list of objects" is very easy to create efficiently in Python's gc internals -- each gc'able object is in *some* doubly-linked list, so the cost of a new list is just a new header node to identify it, and moving a gc'able object from one list to another is 6 pointer stores. > However, it occurs to me that since tp_clear can leave references in place, > it might be possible to control the object destruction order by leaving a > DAG of references in place, and letting Python deallocate objects in > dependency order. Does this work? It could. Here it should help to know that Python's cyclic gc never calls an object's tp_dealloc directly. The only way tp_dealloc ever triggers in Python is as a side effect of a refcount falling to zero -- no exceptions. So, indeed, "all" cyclic gc does is break cycles, and the only places garbage actually gets recycled in gcmodule.c are as side effects of the Py_DECREF(op); lines in delete_garbage() and release_weakrefs(). Weakrefs are probably irrelevant to you at this point, so there's only one Py_DECREF() in the codebase you care about. So, e.g, if you have a simple cycle like A -> B -> C -> ... -> Z -> A and the tp_clear() calls break only the link from Z to A, that's all they *need* to do, and then A's tp_dealloc() will be called first, then B's tp_dealloc, ..., and finally Z's tp_dealloc. You should be aware that long chains of that sort can end up overflowing the C stack, depeding on how tp_dealloc is coded. That's in CPython, of course. Jython uses Java's native gc, and I suppose (but don't know that) IronPython uses .NET's. From martin at v.loewis.de Tue Oct 12 06:13:35 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 12 06:13:37 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011152210.08C1.JCARLSON@uci.edu> References: <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <20041011152210.08C1.JCARLSON@uci.edu> Message-ID: <416B59EF.8010403@v.loewis.de> Josiah Carlson wrote: > A pronouncement would be nice, but I understand that I am in no position > to ask/beg for a (yes, we'll add it) response. <.5 wink> Well, I can declare, in public :-), that I personally won't add that feature to Python 2.4. There are so many other things to do, and I don't consider this feature (in whatever form) important, as this can be easily done in pure Python for those who need it. Regards, Martin From jcarlson at uci.edu Tue Oct 12 09:00:48 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Oct 12 09:08:16 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416B59EF.8010403@v.loewis.de> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> Message-ID: <20041011232519.08C8.JCARLSON@uci.edu> Martin v. L?wis wrote: > Josiah Carlson wrote: > > A pronouncement would be nice, but I understand that I am in no position > > to ask/beg for a (yes, we'll add it) response. <.5 wink> > > Well, I can declare, in public :-), that I personally won't add that > feature to Python 2.4. Scratch chin. Then I suppose it is a good thing that you aren't the only person with commit access then <full wink and a nose wiggle>. > There are so many other things to do, and I > don't consider this feature (in whatever form) important, as this > can be easily done in pure Python for those who need it. The same thing can be said for some syntactic changes to Python over the years; yet that didn't stop list comprehensions, rich comparisons, iterators (generators being the language change that made them easier), etc., from changing the language. I'm not even asking to change the language, I'm offering a patch that implements a functionality desireable to a group of users by making a module more flexible. Since Tim has said that the change is so minor as to be all right to wait until after the beta, I'll keep out of everyone's hair until then. - Josiah From astrand at lysator.liu.se Tue Oct 12 09:25:14 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Tue Oct 12 09:25:23 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ca471dc20410111418614a0694@mail.gmail.com> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> Message-ID: <Pine.GSO.4.51L2.0410120911550.28834@koeberg.lysator.liu.se> On Mon, 11 Oct 2004, Guido van Rossum wrote: > > So, can the subprocess module be accepted? If not, what needs to be done? > > FWIW, I've given Peter some feedback, but in general I think this code > is ready for inclusion into 2.4. I hope it's not too late. Sounds great. >(My > recommendation on the call/callv issue is to get rid of callv(), callv() has now been removed. >and > instead let call() interpret multiple string args as such; a single > string arg will be interpreted as a whole command line, per Popen(); if > you want a single file arg you have to enclose it in [...].) I haven't done any changes to call(): it passes the arguments directly to Popen(), just like before. I would prefer if we could keep it that way; it's so nice to have a short and simple description of the function: Run command with arguments. Wait for command to complete, then return the returncode attribute. If we should do any more changes to handling of multiple args/strings/sequences, we should probably do it to the Popen class itself. But I think we have found a good API now. Here's the documentation: " args should be a string, or a sequence of program arguments. The program to execute is normally the first item in the args sequence or string, but can be explicitly set by using the executable argument. On UNIX, with shell=False (default): In this case, the Popen class uses os.execvp() to execute the child program. args should normally be a sequence. A string will be treated as a sequence with the string as the only item (the program to execute). On UNIX, with shell=True: If args is a string, it specifies the command string to execute through the shell. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional shell arguments. On Windows: the Popen class uses CreateProcess() to execute the child program, which operates on strings. If args is a sequence, it will be converted to a string using the list2cmdline method. Please note that not all MS Windows applications interpret the command line the same way: The list2cmdline is designed for applications using the same rules as the MS C runtime. " /Peter ?strand <astrand@lysator.liu.se> From sjoerd at acm.org Tue Oct 12 09:39:52 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Tue Oct 12 09:40:01 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416AF20D.6010706@v.loewis.de> References: <2mfz4lgufe.fsf@starship.python.net> <1f7befae04101108212f0a528@mail.gmail.com> <416AAE03.9020209@acm.org> <416AF20D.6010706@v.loewis.de> Message-ID: <416B8A48.6010500@acm.org> Martin v. L?wis wrote: > Sjoerd Mullender wrote: > >>> Don't know, but guess so: it contains bytes outside the set ANSI C >>> says can be used portably in text files. >> >> >> >> However, this is not necessarily enough reason to use -kb. The only >> things -kb does are LF -> CRLF / LF -> CR mapping, not using diff for >> updates, and not expanding $ keywords. > > > That is not true. On Apple computers, it also avoids conversion from > Latin-1 to Mac-Roman, which Mac CVS does by default for text files. > Making the files binary is the only way to avoid this conversion, and > that is precisely the reason why the file is binary. I didn't know this. > You may argue that this is a bug in Mac CVS, and I would agree. However, > that specific bug has -kb as a known work-around, and the issue reported > here points to a bug in the compiler packages which should be fixed > rather than worked-around. I agree that this sounds very much like a MacCVS bug, but it also sounds like an excellent reason to leave this file alone. And the compiler issue should be (and has been, I saw) fixed. -- Sjoerd Mullender <sjoerd@acm.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041012/577896c2/signature.pgp From anthony at interlink.com.au Tue Oct 12 09:57:39 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Oct 12 09:57:57 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011232519.08C8.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> <20041011232519.08C8.JCARLSON@uci.edu> Message-ID: <416B8E73.2090304@interlink.com.au> Josiah Carlson wrote: > Since Tim has said that the change is so minor as to be all right to > wait until after the beta, I'll keep out of everyone's hair until then. I guess this _is_ pretty minor, but please don't assume that this means I'll be letting all sorts of new things in past b1 (at least, not without a lot of complaining on my part ;) Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From just at letterror.com Tue Oct 12 10:18:14 2004 From: just at letterror.com (Just van Rossum) Date: Tue Oct 12 10:18:20 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <416B8A48.6010500@acm.org> Message-ID: <r01050400-1026-456296921C2711D996E7003065D5E7E4@[10.0.0.23]> Sjoerd Mullender wrote: > >> However, this is not necessarily enough reason to use -kb. The > >> only things -kb does are LF -> CRLF / LF -> CR mapping, not using > >> diff for updates, and not expanding $ keywords. > > > > That is not true. On Apple computers, it also avoids conversion > > from Latin-1 to Mac-Roman, which Mac CVS does by default for text > > files. Making the files binary is the only way to avoid this > > conversion, and that is precisely the reason why the file is binary. > > I didn't know this. > > > You may argue that this is a bug in Mac CVS, and I would agree. > > However, that specific bug has -kb as a known work-around, and the > > issue reported here points to a bug in the compiler packages which > > should be fixed rather than worked-around. > > I agree that this sounds very much like a MacCVS bug, but it also > sounds like an excellent reason to leave this file alone. And the > compiler issue should be (and has been, I saw) fixed. On the other hand, I hardly know anyone who uses MacCVS anymore. Before OSX you _had_ to use some Mac port of CVS (there were/are many), but on OSX most people simply use cvs on the command line, which is the standard unix version. Still, you can configure MacCVS to not convert the encoding. In other words: I wouldn't worry about MacCVS anymore, at all. Just From mcherm at mcherm.com Tue Oct 12 16:27:03 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Oct 12 16:26:03 2004 Subject: [Python-Dev] scropion Message-ID: <1097591223.416be9b73a9a4@mcherm.com> > [Paul introduces himself] > My question is for performance: isn't binary code faster than p code? > So If I can vote, I would vote for c or c++ in the numerics functions and > data transfer areas. > even assembler. The quicker the better. Hello, and welcome. Thanks for your input on what kinds of things you value in Python. (The Python developers, who use the python-dev list) DO pay attention to such opinion (although don't expect sudden changes... you are one in a sea of users). You are not alone in caring about performance -- it's been a popular topic lately. I'll comment briefly on your specific suggestions. Binary code is faster than "p code" usually, but in some cases the "p code" is more ammenable to automated optimization. That is why certain kinds of code can run FASTER in java than in C. Python code too will often run faster than the equivalent C code... not because of a powerful just-in-time optimizer, but because the Python code is written using better algorithms and data structures than one would bother with in C. But you specifically mention numerics functions and data transfer... two areas where well-written C code is likely to be much better performing. We certainly share the goal of better performance (so long as it doesn't come at the cost of other things like usability and maintainability). If you know of some specific areas where we could improve, please submit a patch (through SourceForge's patch manager) -- we'd love to have the help! Finally, you mention the possibility of dropping into assembler. Unfortunately, this is sure to be rejected because it conflicts with an important design principle -- Python is designed to be VERY portable, and should compile on just about any system with a standards-compliant C compiler. -- Michael Chermside From anthony at interlink.com.au Tue Oct 12 19:15:55 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue Oct 12 16:31:51 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <ca471dc20410111418614a0694@mail.gmail.com> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> Message-ID: <416C114B.8000906@interlink.com.au> Guido van Rossum wrote: > FWIW, I've given Peter some feedback, but in general I think this code > is ready for inclusion into 2.4. I hope it's not too late. (My > recommendation on the call/callv issue is to get rid of callv(), and > instead let call() interpret multiple string args as such; a single > string arg will be interpreted as a whole command line, per Popen(); > if you want a single file arg you have to enclose it in [...].) 2.4b1 is planned for this Thursday. I'm taking Wednesday off work to commit a backlog of patches I want to see in, then plan to cut the release Thursday. I can probably move this to Friday, if it would help this get in. I _really_ don't want this landing after b1, tho - that kinda makes the entire release process pointless if we're landing stuff that late. I'm off to sleep now - could Peter or whoever else is likely to do the work let me know? If it's just a matter of doing the checkin, and making sure all the fiddly bits are done, you could also point me at a patch and I'll do it. From fredrik at pythonware.com Tue Oct 12 16:50:23 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 16:48:19 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ca471dc20410111418614a0694@mail.gmail.com> <416C114B.8000906@interlink.com.au> Message-ID: <ckgqre$85l$1@sea.gmane.org> Anthony Baxter wrote > I'm off to sleep now - could Peter or whoever else is likely to do > the work let me know? If it's just a matter of doing the checkin, > and making sure all the fiddly bits are done, you could also point > me at a patch and I'll do it. I can check in the source files; someone with a proper Windows setup will have to help us out with the makefile and installer issues. Stay tuned. </F> From fredrik at pythonware.com Tue Oct 12 17:07:17 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 17:05:12 2004 Subject: [Python-Dev] Re: Re: subprocess - updated popen5 module References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ca471dc20410111418614a0694@mail.gmail.com> <416C114B.8000906@interlink.com.au> Message-ID: <ckgrr3$bhg$1@sea.gmane.org> Anthony Baxter wrote: > I'm off to sleep now - could Peter or whoever else is likely to do > the work let me know? If it's just a matter of doing the checkin, > and making sure all the fiddly bits are done, you could also point > me at a patch and I'll do it. I'll check in the source files shortly. Someone with a proper Windows setup will have to help out with makefile and installer tweaks for the PC/_subprocess.c module; drop me a line if you need info. <F> From fredrik at pythonware.com Tue Oct 12 18:16:14 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 12 18:14:10 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 References: <E1CHP2L-000059-FA@sc8-pr-cvs1.sourceforge.net> Message-ID: <ckgvsd$q41$1@sea.gmane.org> > +\begin{seealso} > +\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with > assistance from Fredrik Lundh and others.} > +\end{seealso} I don't know how to add ISO-8859-1 characters to a Latex document, but in HTML notation, Peter's last name should be Åstrand. cheers /F From aparente at adobe.com Tue Oct 12 18:17:49 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Tue Oct 12 18:18:29 2004 Subject: [Python-Dev] Python on Windows 64bits? Message-ID: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Sorry if I'm asking something that has been answered before, but I could not find a clear answer anywhere. Is there a project page for Windows 64bits? Is it already on CVS? My company tries to assess if we can plan on using python on Windows 64 bits. We want to do it natively (i.e. no 32b emulation), because we have some python extensions we want to use in the process. Thanks! alex. From gvanrossum at gmail.com Tue Oct 12 19:50:56 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 12 19:51:17 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: <r01050400-1026-456296921C2711D996E7003065D5E7E4@10.0.0.23> References: <416B8A48.6010500@acm.org> <r01050400-1026-456296921C2711D996E7003065D5E7E4@10.0.0.23> Message-ID: <ca471dc204101210506d4fd0fa@mail.gmail.com> In any case, the necessity of -kb for this file (if it is still necessary) should be documented in a comment inside the file, to prevent a repeat of this discussion three years from now. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Tue Oct 12 20:15:00 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Tue Oct 12 20:13:47 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: <ckgvsd$q41$1@sea.gmane.org> References: <E1CHP2L-000059-FA@sc8-pr-cvs1.sourceforge.net> <ckgvsd$q41$1@sea.gmane.org> Message-ID: <ckh6sm$ham$1@sea.gmane.org> Fredrik Lundh wrote: >>+\seepep{324}{subprocess - ...}{Written ... Peter Astrand, ... > > I don't know how to add ISO-8859-1 characters to a Latex document, > but in HTML notation, Peter's last name should be Åstrand. > > cheers /F I think that should be: +\seepep{324}{subprocess - ...}{Written ... Peter \AA strand, ... -- Scott David Daniels Scott.Daniels@Acm.Org From abingham at alumni.sfu.ca Tue Oct 12 20:39:30 2004 From: abingham at alumni.sfu.ca (Aaron Bingham) Date: Tue Oct 12 20:48:56 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: <ckgvsd$q41$1@sea.gmane.org> References: <E1CHP2L-000059-FA@sc8-pr-cvs1.sourceforge.net> <ckgvsd$q41$1@sea.gmane.org> Message-ID: <416C24E2.4020304@alumni.sfu.ca> Fredrik Lundh wrote: >>+\begin{seealso} >>+\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with >>assistance from Fredrik Lundh and others.} >>+\end{seealso} >> >> > >I don't know how to add ISO-8859-1 characters to a Latex document, >but in HTML notation, Peter's last name should be Åstrand. > > I don't know if it's allowed for for Python docs, but the easiest way I know of is to put \usepackage{isolatin1} in the prelude and just use regular ISO-8859-1 characters in the file. Aaron From barry at python.org Tue Oct 12 21:31:55 2004 From: barry at python.org (Barry Warsaw) Date: Tue Oct 12 21:32:05 2004 Subject: [Python-Dev] subprocess module test failure when --with-pydebug Message-ID: <1097609515.8573.50.camel@geddy.wooz.org> The new subprocess module test is not compatible with a Python built --with-pydebug because the [xxxx refs] output breaks the comparison. I've recently been thinking that it would be nice if that [xxxx refs] output could be suppressed with an envar or a command line switch. FWIW, I generally build our embedding product with a --with-pydebug built Python and that output breaks our unit tests too. I work around this by suppressing the stderr comparison in our testing framework. Once we decide to do it, and what to wrap that call in, it's likely a one or two line change to pythonrun.c. Does one of the existing envars make sense to use? Barring any change here, we need to at least make sure the subprocess test doesn't fail for a debug build. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041012/f96bafc2/attachment.pgp From janssen at parc.com Tue Oct 12 21:31:58 2004 From: janssen at parc.com (Bill Janssen) Date: Tue Oct 12 21:33:32 2004 Subject: [Python-Dev] test_pep263.py is "-kb" in CVS In-Reply-To: Your message of "Tue, 12 Oct 2004 01:18:14 PDT." <r01050400-1026-456296921C2711D996E7003065D5E7E4@[10.0.0.23]> Message-ID: <04Oct12.123203pdt."58617"@synergy1.parc.xerox.com> Just van Rossum writes: > On the other hand, I hardly know anyone who uses MacCVS anymore. Before > OSX you _had_ to use some Mac port of CVS (there were/are many), but on > OSX most people simply use cvs on the command line, which is the > standard unix version. Still, you can configure MacCVS to not convert > the encoding. In other words: I wouldn't worry about MacCVS anymore, at > all. Agreed. Bill From aahz at pythoncraft.com Tue Oct 12 21:36:59 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Oct 12 21:37:08 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Message-ID: <20041012193659.GA26457@panix.com> On Tue, Oct 12, 2004, Alexandre Parenteau wrote: > > Sorry if I'm asking something that has been answered before, but I could > not find a clear answer anywhere. > > Is there a project page for Windows 64bits? Is it already on CVS? > > My company tries to assess if we can plan on using python on Windows 64 > bits. > > We want to do it natively (i.e. no 32b emulation), because we have some > python extensions we want to use in the process. In theory, it should "just work". Python is already 64-bit compliant; any issues are likely to be problems with Microsoft compilers. You should probably ask on comp.lang.python unless you uncover specific problems that you believe are caused by Python itself. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From abkhd at hotmail.com Wed Oct 13 04:47:59 2004 From: abkhd at hotmail.com (Khalid A. B.) Date: Wed Oct 13 04:48:05 2004 Subject: [Python-Dev] test_cwd of test_subprocess.py fails on Win98 Message-ID: <BAY18-F156EyhOwlVZm00069fa3@hotmail.com> Hello. The test fails because it seems to be comparing "C:\WINDOWS\TEMP" with "C:\WINDOWS\Temp" and even though they are the same thing in Win98, they are of cource not equal. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Although I don't know if this will work in other versions of Windows, here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From erik at heneryd.com Wed Oct 13 07:09:22 2004 From: erik at heneryd.com (Erik Heneryd) Date: Wed Oct 13 07:15:01 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <Pine.GSO.4.51L2.0410120911550.28834@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <Pine.GSO.4.51L2.0410120911550.28834@koeberg.lysator.liu.se> Message-ID: <416CB882.3050308@heneryd.com> Quick thought: The subprocess class is still called Popen. Any reason not to drop this last reference to the unix clib function and rename it Subprocess? Erik From martin at v.loewis.de Wed Oct 13 07:22:56 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 07:22:57 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041011232519.08C8.JCARLSON@uci.edu> References: <20041011152210.08C1.JCARLSON@uci.edu> <416B59EF.8010403@v.loewis.de> <20041011232519.08C8.JCARLSON@uci.edu> Message-ID: <416CBBB0.9040809@v.loewis.de> Josiah Carlson wrote: >>There are so many other things to do, and I >>don't consider this feature (in whatever form) important, as this >>can be easily done in pure Python for those who need it. > > > The same thing can be said for some syntactic changes to Python over the > years; yet that didn't stop list comprehensions, rich comparisons, > iterators (generators being the language change that made them easier), > etc., from changing the language. Indeed. I wish some of the changes would not have been made (although the changes on your list are all fine). For those features, the potential user community is *much* larger than for the feature under discussion, and I feel I would waste my time for adding a feature that only few users ever need. If you would like to debate the size of the user community: any significanly-large user community should have come up with a standard, pure-Python solution to the problem by now which would just wait for integration. This is IMO the process that should be followed for all library extensions: the library should be developed elsewhere, and wait some time for API and implementation stabilization. Only *then* it become candidate for inclusion into Python. Regards, Martin From martin at v.loewis.de Wed Oct 13 07:36:36 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 07:36:36 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> Message-ID: <416CBEE4.5040908@v.loewis.de> Alexandre Parenteau wrote: > Sorry if I'm asking something that has been answered before, but I could > not find a clear answer anywhere. > > Is there a project page for Windows 64bits? Is it already on CVS? > > My company tries to assess if we can plan on using python on Windows 64 > bits. > > We want to do it natively (i.e. no 32b emulation), because we have some > python extensions we want to use in the process. Itanium or AMD64? For Itanium, we have installers (of Python 2.4) available: http://www.python.org/2.4/ For AMD64, you need to compile it yourself. Regards, Martin From astrand at lysator.liu.se Wed Oct 13 07:38:31 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 07:38:39 2004 Subject: [Python-Dev] Re: subprocess - updated popen5 module In-Reply-To: <416CB882.3050308@heneryd.com> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <Pine.GSO.4.51L2.0410120911550.28834@koeberg.lysator.liu.se> <416CB882.3050308@heneryd.com> Message-ID: <Pine.GSO.4.51L2.0410130733500.16329@koeberg.lysator.liu.se> On Wed, 13 Oct 2004, Erik Heneryd wrote: > Quick thought: The subprocess class is still called Popen. Any reason > not to drop this last reference to the unix clib function and rename it > Subprocess? It's shorter... :) Also, I think it's good that the name hints about "process open"; that you can read and write to the process. /Peter ?strand <astrand@lysator.liu.se> From fredrik at pythonware.com Wed Oct 13 08:56:24 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 08:54:18 2004 Subject: [Python-Dev] Re: test_cwd of test_subprocess.py fails on Win98 References: <BAY18-F156EyhOwlVZm00069fa3@hotmail.com> Message-ID: <ckijel$jba$1@sea.gmane.org> "Khalid A. B." wrote: > The test fails because it seems to be comparing "C:\WINDOWS\TEMP" with > "C:\WINDOWS\Temp" and even though they are the same thing in Win98, > they are of cource not equal. I just applied the patch below, which I think does the same thing in a slightly more convenient way. </F> --- Lib/test/test_subprocess.py 13 Oct 2004 04:07:12 -0000 1.9 +++ Lib/test/test_subprocess.py 13 Oct 2004 06:52:56 -0000 @@ -216,7 +216,8 @@ 'sys.stdout.write(os.getcwd())'], stdout=subprocess.PIPE, cwd=tmpdir) - self.assertEqual(p.stdout.read(), tmpdir) + normcase = os.path.normcase + self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() From fredrik at pythonware.com Wed Oct 13 09:35:36 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:33:30 2004 Subject: [Python-Dev] latex help wanted! (Re: subprocess - updated popen5 module) References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ca471dc20410111418614a0694@mail.gmail.com><416C114B.8000906@interlink.com.au> <ckgrr3$bhg$1@sea.gmane.org> Message-ID: <ckilo6$org$1@sea.gmane.org> Fredrik Lundh wrote: > I'll check in the source files shortly. Someone with a proper Windows setup > will have to help out with makefile and installer tweaks for the PC/_subprocess.c > module; drop me a line if you need info. Tim fixed the windows issues (and got rid of the banana), but I just realized that we could need some help with converting the documentation from PEP/docstring format to latex... Any volunteers? (the PEP in nondist/peps/pep-0324.txt contains extensive documentation, but it can also be good to check the Lib/subprocess.py docstring for additional examples) (Now, if only someone could convert the docs to "html body + semantic classes" so people with ordinary skills can work on it, without having to learn any new syntaxes or install any new tools, besides python and a web browser...) </F> From jcarlson at uci.edu Wed Oct 13 09:26:38 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 09:34:06 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416CBBB0.9040809@v.loewis.de> References: <20041011232519.08C8.JCARLSON@uci.edu> <416CBBB0.9040809@v.loewis.de> Message-ID: <20041012232318.091E.JCARLSON@uci.edu> > For those features, the > potential user community is *much* larger than for the feature under > discussion, and I feel I would waste my time for adding a feature that > only few users ever need. And yet you continue the discussion? I talk about it because /I/ find the functionality useful, I believe /others/ would find it useful if they were put in similar situations as I, and I believe it adds flexibility to a module (which I think is positive). > If you would like to debate the size of the user community: any > significanly-large user community should have come up with a standard, > pure-Python solution to the problem by now which would just wait for > integration. This is IMO the process that should be followed for all > library extensions: the library should be developed elsewhere, and > wait some time for API and implementation stabilization. Only *then* > it become candidate for inclusion into Python. If every modification to Python required a community of people, who all joined together to advocate something, then nothing would ever get done. People would be spending all their time trying to gather groups of people to agree with them that functionality K is necessary. Thankfully there has been cases in the past where someone has said, "hey, this thing could really use X", some people agree, some people disagree, a sample implementation is done, and sometimes it is accepted. I'm not even saying that we should add a new module. I'm not even saying we should add a new function to a module. Heck, I'm not even asking for a new argument to a function that previously existed. I am, quite literally, asking for the equivalent of less than one bit in a flag (there are currently 22 format/endian characters, which are all orthogonal, which would require 5 bits, if they were flags). The API already exists. The framework already exists. I'm not asking for Python to interpret something that was valid before as something different now. I'm asking for the equivalent of a previously invalid flag, to become valid, in order to expose previously existing translation mechanisms, whose use can be found in databases, network protocols, encryption, etc. Try to read the above paragraph outside of the context of struct and the RFE. Does it make sense to include the change now? If every request to interpret a new flag required significant community involvement, goodness, would it take an act of Guido to get a commit done? Have a good day Martin, - Josiah From theller at python.net Wed Oct 13 09:41:45 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 09:41:29 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib subprocess.py, NONE, 1.1 In-Reply-To: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> (effbot@users.sourceforge.net's message of "Tue, 12 Oct 2004 08:26:29 -0700") References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> Message-ID: <mzyr6p52.fsf@python.net> effbot@users.sourceforge.net writes: > Update of /cvsroot/python/python/dist/src/Lib > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv24287/Lib > > Added Files: > subprocess.py > Log Message: > Added Peter Astrand's subprocess module. I suggest to remove the imports of the pywin32 stuff, it's not needed and confuses tools like freeze or py2exe. if mswindows: import threading import msvcrt try: from _subprocess import * class STARTUPINFO: dwFlags = 0 hStdInput = None hStdOutput = None hStdError = None class pywintypes: error = IOError except ImportError: import pywintypes from win32api import GetStdHandle, STD_INPUT_HANDLE, \ STD_OUTPUT_HANDLE, STD_ERROR_HANDLE from win32api import GetCurrentProcess, DuplicateHandle, \ GetModuleFileName, GetVersion from win32con import DUPLICATE_SAME_ACCESS from win32pipe import CreatePipe from win32process import CreateProcess, STARTUPINFO, \ GetExitCodeProcess, STARTF_USESTDHANDLES, \ CREATE_NEW_CONSOLE from win32event import WaitForSingleObject, INFINITE, WAIT_OBJECT_0 From fredrik at pythonware.com Wed Oct 13 09:48:48 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:46:48 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se><ca471dc20410111418614a0694@mail.gmail.com><416C114B.8000906@interlink.com.au><ckgrr3$bhg$1@sea.gmane.org> <ckilo6$org$1@sea.gmane.org> Message-ID: <ckimgu$qor$1@sea.gmane.org> can bug categories be associated with developers? if so, would it perhaps make sense to add Peter as a developer, and add a category for the subprocess system? </F> From fredrik at pythonware.com Wed Oct 13 09:54:15 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 09:52:10 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> Message-ID: <ckimr5$rkb$1@sea.gmane.org> Thomas Heller wrote: > I suggest to remove the imports of the pywin32 stuff, it's not needed > and confuses tools like freeze or py2exe. That's really a problem with the tools, not with the module. Demanding that users of a dynamic language use only static linking is really silly. Wouldn't it be better to invent a (pragma-level) syntax so that a module can inform the tool about its intentions? (removing the pywin stuff from the module would hamper development of new features, and also make it harder to verify that the _subprocess module works as it should). </F> From theller at python.net Wed Oct 13 10:04:19 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 10:04:04 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <ckimr5$rkb$1@sea.gmane.org> (Fredrik Lundh's message of "Wed, 13 Oct 2004 09:54:15 +0200") References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> Message-ID: <fz4j6o3g.fsf@python.net> "Fredrik Lundh" <fredrik@pythonware.com> writes: > Thomas Heller wrote: > >> I suggest to remove the imports of the pywin32 stuff, it's not needed >> and confuses tools like freeze or py2exe. > > That's really a problem with the tools, not with the module. Yep. > Demanding that users of a dynamic language use only static linking is > really silly. > > Wouldn't it be better to invent a (pragma-level) syntax so that a module can > inform the tool about its intentions? Yes. But I don't think I have the energy to discuss that here, and don't feel it would be easy to get this into the core. > (removing the pywin stuff from the module would hamper development of new > features, and also make it harder to verify that the _subprocess module works > as it should). Hm, that what the tests are for, and not code that is never executed. A working solution would be to comment out this stuff, or put it into an 'if 0:' block, in both cases one could easily activate it again for experimentation. Thomas From fredrik at pythonware.com Wed Oct 13 10:19:26 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 10:17:21 2004 Subject: [Python-Dev] Re: Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net><mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> Message-ID: <ckioac$v84$1@sea.gmane.org> Thomas Heller wrote: >> (removing the pywin stuff from the module would hamper development of new >> features, and also make it harder to verify that the _subprocess module works >> as it should). > > Hm, that what the tests are for, and not code that is never executed. if you remove the _subprocess driver, it is executed. _subprocess is just a win32all emulator... > A working solution would be to comment out this stuff, or put it into an > 'if 0:' block, in both cases one could easily activate it again for > experimentation. does py2exe support "if 0" blocks? </F> From theller at python.net Wed Oct 13 10:33:36 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 10:33:20 2004 Subject: [Python-Dev] Re: Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <ckioac$v84$1@sea.gmane.org> (Fredrik Lundh's message of "Wed, 13 Oct 2004 10:19:26 +0200") References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> Message-ID: <acur6mqn.fsf@python.net> "Fredrik Lundh" <fredrik@pythonware.com> writes: > Thomas Heller wrote: > >>> (removing the pywin stuff from the module would hamper development of new >>> features, and also make it harder to verify that the _subprocess module works >>> as it should). >> >> Hm, that what the tests are for, and not code that is never executed. > > if you remove the _subprocess driver, it is executed. _subprocess is just > a win32all emulator... To be honest, I'm against code in the core that depends on whether pywin32 is installed or not. >> A working solution would be to comment out this stuff, or put it into an >> 'if 0:' block, in both cases one could easily activate it again for >> experimentation. > > does py2exe support "if 0" blocks? py2exe uses modulefinder to scan the byte code of compiled modules, and Python doesn't generate code for 'if 0' blocks, so, yes. Thomas From python at rcn.com Wed Oct 13 10:56:53 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 13 10:58:31 2004 Subject: [Python-Dev] Struct module format code addition In-Reply-To: <20041012232318.091E.JCARLSON@uci.edu> Message-ID: <001501c4b102$96de0480$e841fea9@oemcomputer> [Josiah] > I'm asking for the equivalent of a previously invalid > flag, to become valid, ... IMO, there was an early transition from asking to demanding. Negative comments by some of the top developers did not dissuade you in the least. > If every request to interpret a new flag required significant community > involvement, goodness, would it take an act of Guido to get a commit > done? Yes, if a proposal doesn't win developer support, then having Guido as a champion is the best bet. If you are opinion shopping, then the best so far is Tim's comment which seems to equate to a +0. That would be sufficient only if he cares enough to review the patch, write the docs, test it, maintain it, etc. Perhaps he will, perhaps he won't. FWIW, last year I had dropped one of my own proposals (relating to a non-unicode use for encode/decode) simply based on respect for Martin's -1. For me, that was more important than the +0 to +1 comments from others, more important than my own use cases, and more important than feeling like I was right. In the end, there was an easy pure python equivalent and life went on. Looking at the struct feature request, I think it would be harmless to add it. OTOH, it is easily handled in python and will be easier still when the binascii functions get put in. Arguably, the feature is somewhat trivial and won't change anyone's life for good or ill. So, you have to ask yourself whether it is worth jamming down everyone's throats to get it in after the beta goes out. For my money, the feature request has already consumed more developer resources than it could ever save. Raymond From fredrik at pythonware.com Wed Oct 13 11:57:12 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 13 11:57:18 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net><mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org><fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> Message-ID: <ckiu5o$h27$1@sea.gmane.org> Thomas Heller wrote: > To be honest, I'm against code in the core that depends on whether > pywin32 is installed or not. >> does py2exe support "if 0" blocks? > > py2exe uses modulefinder to scan the byte code of compiled modules, and > Python doesn't generate code for 'if 0' blocks, so, yes. just one more question: is there a specific problem with win32all? if this is a general problem, how do you deal with other optional modules in the standard library (e.g. _xmlrpclib) ? </F> From theller at python.net Wed Oct 13 14:01:49 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 14:01:33 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <ckiu5o$h27$1@sea.gmane.org> (Fredrik Lundh's message of "Wed, 13 Oct 2004 11:57:12 +0200") References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> Message-ID: <wtxu6d3m.fsf@python.net> "Fredrik Lundh" <fredrik@pythonware.com> writes: > Thomas Heller wrote: > >> To be honest, I'm against code in the core that depends on whether >> pywin32 is installed or not. > >>> does py2exe support "if 0" blocks? >> >> py2exe uses modulefinder to scan the byte code of compiled modules, and >> Python doesn't generate code for 'if 0' blocks, so, yes. > > just one more question: is there a specific problem with win32all? if this > is a general problem, how do you deal with other optional modules in the > standard library (e.g. _xmlrpclib) ? If _xmlrpclib is installed, py2exe will find it if the script uses xmlrpclib. I guess that is what the user expects. If _xmlrpclib is not installed, py2exe will warn that it is missing (this warning is probably not very useful, and it seems Bob has supressed this warning in py2app). The point with win32all (now pywin32) is that most windows users will have it installed, for good reasons. py2exe will find it, and include it if the subprocess module is pulled in - whether _subprocess is available or not. But subprocess, the Python 2.4 version, will *never* use it, because _subprocess is always available (isn't this now even a builtin module, in python24.dll?). Hope that answers your question, Thomas From fdrake at acm.org Wed Oct 13 15:26:18 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Oct 13 15:26:26 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Lib/distutils/command __init__.py, 1.19, 1.20 install.py, 1.70, 1.71 checkdep.py, 1.1, NONE In-Reply-To: <E1CHiLq-0007bB-RF@sc8-pr-cvs1.sourceforge.net> References: <E1CHiLq-0007bB-RF@sc8-pr-cvs1.sourceforge.net> Message-ID: <200410130926.18709.fdrake@acm.org> On Wednesday 13 October 2004 08:35 am, Anthony Baxter wrote: > Backing out the basic dependency checking (from pycon sprint). > This support was only a first cut, and doesn't deserve to be in > a released version (where we have to support it in an ongoing > manner) Thanks, Anthony! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From anthony at interlink.com.au Wed Oct 13 16:24:46 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 16:25:02 2004 Subject: [Python-Dev] test_subprocess fails on OSX In-Reply-To: <wtxu6d3m.fsf@python.net> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> Message-ID: <416D3AAE.6090907@interlink.com.au> On OSX 10.3.4, test_subprocess is failing intermittently when run from regrtest: test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/Users/anthonybaxter/pyhead/src/Lib/test/test_subprocess.py", line 434, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '4' != '3' Running just the test by itself works. I'm guessing it could be part of the test suite leaking a FD or something? Anyone know of a tool on OSX that I can run to examine open FDs for a process? -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From bob at redivi.com Wed Oct 13 16:45:41 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed Oct 13 16:46:19 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <wtxu6d3m.fsf@python.net> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> Message-ID: <8DF3F5EC-1D26-11D9-AAC5-000A95686CD8@redivi.com> On Oct 13, 2004, at 8:01 AM, Thomas Heller wrote: > "Fredrik Lundh" <fredrik@pythonware.com> writes: > >> Thomas Heller wrote: >> >>> To be honest, I'm against code in the core that depends on whether >>> pywin32 is installed or not. >> >>>> does py2exe support "if 0" blocks? >>> >>> py2exe uses modulefinder to scan the byte code of compiled modules, >>> and >>> Python doesn't generate code for 'if 0' blocks, so, yes. >> >> just one more question: is there a specific problem with win32all? >> if this >> is a general problem, how do you deal with other optional modules in >> the >> standard library (e.g. _xmlrpclib) ? > > If _xmlrpclib is installed, py2exe will find it if the script uses > xmlrpclib. I guess that is what the user expects. If _xmlrpclib is > not > installed, py2exe will warn that it is missing (this warning is > probably > not very useful, and it seems Bob has supressed this warning in > py2app). I have disabled all missing module warnings, because I've found that are more of a nuisance than anything else. However, I have made it possible to generate a GraphViz DOT graph of everything that did get included, which is useful when you are having problems with too many or too few dependencies being included. Unlike modulefinder, py2app.modulegraph keeps track of why modules have been included, so it's easy enough to say that "pydoc should not depend on Tkinter", and Tkinter won't be included unless something else needs it. For cases like xmlrpclib->_xmlrpclib or xml -> _xmlplus, with a semi-standalone build (depends on an existing Python installation), a dotted edge will be drawn between the module that uses xmlrpclib and _xmlrpclib. The graph output only includes modules that ended up in the application bundle, so the edge is dotted to represent that the dependency exists, but was indirect (since xmlrpclib would not have been shown). That said, if you really wanted to display missing modules, you could find them pretty easily from the ModuleGraph instance, just walk it and keep instances of MissingModule. -bob From jasone at canonware.com Wed Oct 13 16:54:04 2004 From: jasone at canonware.com (Jason Evans) Date: Wed Oct 13 16:53:32 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <1f7befae041011203471688554@mail.gmail.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> Message-ID: <20041013145404.GB33145@canonware.com> On Mon, Oct 11, 2004 at 11:34:34PM -0400, Tim Peters wrote: > [Tim Peters] > >> I think you're in for a rude awakening if you expect any language's gc > >> support to help you manage arbitrary blobs of memory obtained from the > >> system malloc(). > > [Jason Evans] > > I don't expect that. What I do expect is for there to be clean and > > efficient mechanisms for integrating custom Python object types into > > Python. Python more or less meets this expectation. > > OK ... but you must want more than just that, else this thread > wouldn't have started <wink>. I initially started this thread thinking that I had some useful suggestions about how Python's GC might be modified. Had I realized in advance that my suggestions were based on my own misconceptions about Python's GC, I would not have bothered the people in this forum. I apologize for reducing the signal to noise ratio of python-dev. > > Many of the issues I ran into had to do with the impedence mismatch between > > my previous experience and the way Python works. > > I sure believe that. Python's cyclic gc is unusual in several respects. Indeed. This brings up a question about documentation: as far as I know, I've read all of the available documentation regarding cyclic GC (the "Supporting cyclic garbage collection" section of "Extending and Embedding the Python Interpreter"), but those docs came nowhere near giving me an adequate understanding of the design rationale behind the cyclic GC. Did I overlook some documentation? Of course, it may be that I am actually at a cognitive disadvantage to someone who has only experienced Python's memory management strategy... > > I appreciate you taking the time to respond to me, especially since you've > > helped me gain a deeper understanding of how Python's GC works. However, > > please get it out of your mind that I am incompetent and/or set in my ways. > > Those aren't in my head. I do think you're so steeped in the details > of your code that you're not nearly as successful at communicating the > *salient* points as you think you are, though. I'm so steeped in the > details of Python's gc that a similar claim could easily be made of > me. I don't expect it to be clear the first time, though, so play > 20-questions trying to tease out what you really need to know about > it. If it's any consolation, this process has been painful on both > sides <wink>. My critical error in trying to communicate the salient points of the Crux code was that I didn't realize the importance of comminicating the fact that the trees are implemented as a combination of separated low level tree data structures and Python wrappers to the low level implementation. Thanks for your help, Tim. Jason From aparente at adobe.com Wed Oct 13 17:15:50 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 17:16:26 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416CBEE4.5040908@v.loewis.de> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> Message-ID: <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> Thanks all for your responses, I am not interested in Itanium, only in x86_64. Thanks again, alex. At 10:36 PM 10/12/2004, Martin v. L?wis wrote: >Alexandre Parenteau wrote: >>Sorry if I'm asking something that has been answered before, but I could >>not find a clear answer anywhere. >>Is there a project page for Windows 64bits? Is it already on CVS? >>My company tries to assess if we can plan on using python on Windows 64 bits. >>We want to do it natively (i.e. no 32b emulation), because we have some >>python extensions we want to use in the process. > >Itanium or AMD64? For Itanium, we have installers (of Python 2.4) available: > >http://www.python.org/2.4/ > >For AMD64, you need to compile it yourself. > >Regards, >Martin From aahz at pythoncraft.com Wed Oct 13 17:19:56 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Oct 13 17:19:59 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013145404.GB33145@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> Message-ID: <20041013151956.GA9469@panix.com> On Wed, Oct 13, 2004, Jason Evans wrote: > > Indeed. This brings up a question about documentation: as far as I > know, I've read all of the available documentation regarding cyclic GC > (the "Supporting cyclic garbage collection" section of "Extending and > Embedding the Python Interpreter"), but those docs came nowhere near > giving me an adequate understanding of the design rationale behind the > cyclic GC. Did I overlook some documentation? > > Of course, it may be that I am actually at a cognitive disadvantage > to someone who has only experienced Python's memory management > strategy... As Tim said earlier, though less pungently, "Read the source, Luke." -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From nas at arctrix.com Wed Oct 13 17:25:26 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Wed Oct 13 17:25:31 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013145404.GB33145@canonware.com> References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> Message-ID: <20041013152526.GB595@mems-exchange.org> On Wed, Oct 13, 2004 at 07:54:04AM -0700, Jason Evans wrote: > This brings up a question about documentation: as far as I know, > I've read all of the available documentation regarding cyclic GC > (the "Supporting cyclic garbage collection" section of "Extending > and Embedding the Python Interpreter"), but those docs came > nowhere near giving me an adequate understanding of the design > rationale behind the cyclic GC. Did I overlook some > documentation? It's a pretty old document and perhaps a little out of date but here's something I wrote while working on the Python GC: http://arctrix.com/nas/python/gc/ Looking back, I realize that the constraints were pretty extreme which probably accounts for the unique implementation. A few off the top of my head: * binary compatible with previous version of Python * optional * small overhead for programs that don't create reference cycles * __del__ methods should still work Cheers, Neil From tim.peters at gmail.com Wed Oct 13 17:31:17 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 17:31:23 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <wtxu6d3m.fsf@python.net> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> Message-ID: <1f7befae041013083132866606@mail.gmail.com> [Thomas Heller] ... > But subprocess, the Python 2.4 version, will *never* use it, because > _subprocess is always available (isn't this now even a builtin module, in > python24.dll?). Yes, it is. I haven't had time to read the PEP, but I assumed some "backward compatibility" constraint is at work here too, since, e.g., subprocess.py has try: False except NameError: False = 0 True = 1 and that doesn't make sense for 2.4 either. From barry at python.org Wed Oct 13 17:35:54 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 13 17:36:08 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1f7befae041013083132866606@mail.gmail.com> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> <1f7befae041013083132866606@mail.gmail.com> Message-ID: <1097681754.8607.26.camel@geddy.wooz.org> On Wed, 2004-10-13 at 11:31, Tim Peters wrote: > Yes, it is. I haven't had time to read the PEP, but I assumed some > "backward compatibility" constraint is at work here too, since, e.g., > subprocess.py has > > try: > False > except NameError: > False = 0 > True = 1 > > and that doesn't make sense for 2.4 either. If so, then it should be documented in PEP 291, otherwise backward compatibility stuff is fair game for ripping out. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041013/f49b5d7e/attachment-0001.pgp From anthony at interlink.com.au Wed Oct 13 17:39:49 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 17:40:07 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> Message-ID: <416D4C45.3040802@interlink.com.au> Alexandre Parenteau wrote: > Thanks all for your responses, > > I am not interested in Itanium, only in x86_64. Python's worked on the Alphas for approximately ever, and works on Itanium, so I can't imagine you'll find any 32-bit-isms left in the code. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From jcarlson at uci.edu Wed Oct 13 17:37:38 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 17:45:21 2004 Subject: [Python-Dev] Struct module format code addition In-Reply-To: <001501c4b102$96de0480$e841fea9@oemcomputer> References: <20041012232318.091E.JCARLSON@uci.edu> <001501c4b102$96de0480$e841fea9@oemcomputer> Message-ID: <20041013081413.0927.JCARLSON@uci.edu> > Looking at the struct feature request, I think it would be harmless to > add it. OTOH, it is easily handled in python and will be easier still > when the binascii functions get put in. Arguably, the feature is > somewhat trivial and won't change anyone's life for good or ill. It makes processing one's data a 1-pass affair rather than a 2-pass affair. Not a big deal for most, but it gets me about a 20% speedup on a few formats, and saves me from writing custom translations every time I want some nonstandard sized integer. > So, you have to ask yourself whether it is worth jamming down everyone's > throats to get it in after the beta goes out. For my money, the feature > request has already consumed more developer resources than it could ever > save. Good points Raymond. If someone decides they want to add it, great, I'll even write some documentation and field sf.net bug reports and such in regards to the struct module. If not, I'll just use binascii and bite my tongue. My apologies for raising a stink. - Josiah From tim.peters at gmail.com Wed Oct 13 18:15:23 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 18:15:27 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: <ckimgu$qor$1@sea.gmane.org> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <416C114B.8000906@interlink.com.au> <ckgrr3$bhg$1@sea.gmane.org> <ckilo6$org$1@sea.gmane.org> <ckimgu$qor$1@sea.gmane.org> Message-ID: <1f7befae041013091525b1835@mail.gmail.com> [Fredrik Lundh] > can bug categories be associated with developers? Yes. For example, I see that the "Regular Expressions" category is auto-assigned to effbot. > if so, would it perhaps make sense to add Peter as a developer, That would be fine by me, provided he asks for it (it's a commitment, despite that developers routinely vanish for years at a time <wink>). > and add a category for the subprocess system? That one's harder to call. Once a category is added, it can never be removed. So I'd wait on that to see whether enough reports came in against subprocess to justify it. Since it's unlikely that an original reporter will assign "the right" category to begin with, and the original submission is the only time at which auto-assignment has an effect, the major value in all this would be achieved simply by having a subprocess expert with commit privileges. From aparente at adobe.com Wed Oct 13 18:27:16 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 18:27:52 2004 Subject: [Python-Dev] About the Wise installer on Windows Message-ID: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Hi, Again forgive me if the answer is somewhere, but I could not find it. I am looking for a way to build and make an installer out of Python on a Win x86_64 platform which is not supported yet by the Python team. My plan is to compile Python for this platform (several developers on the list told me it should be straightforward), and make an installer out of the Python binaries in order to be able to distribute the Python binaries internally. I downloaded the evaluation copy of Wise Installer System 9, but it complains that the installer file (which I assume is PCBuild\Python20.wse) is corrupted. I need to establish exactly the version of the installer to use, if we need to acquire a license for the Wise installer. - Could someone give me some pointer about the exact version of the Wise Installer to use in order to be able to replicate the same installer that the Python team is distributing on Windows? - The ISS files (Inno) do not seem up to date. Is anybody having a ISS version of the installer which works for 2.3.4? Thanks, alex. From anthony at interlink.com.au Wed Oct 13 19:10:48 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 13 19:10:59 2004 Subject: [Python-Dev] 2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING Message-ID: <416D6198.5000606@interlink.com.au> Ok, I'm going to be cutting the Python release Friday (Australian time). This means that I'm declaring the trunk frozen for the release in about 24 hours. Lets say at 18:00 hours UTC on Thursday 14th (that's 14:00 on the US east coast, if my memory is correct, and about 04:00 where here in Australia). No-one except for the usual team (Fred, Martin, myself) should check into the trunk after that. As per usual, I'll probably keep the trunk frozen for a time after the release, in case of brown-paper-bag issues. Note that this is the last chance you have to get those behaviour- changing bugs^Wfixes introduced, so get to it! After b1, it will become _much_ harder to get new fixes introduced. And if there's a pending bug or patch that needs some lovin', add a comment to it or mail me - I think I've responded to, or fixed, all the ones I've been told about, but if I missed one, tell me now. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From astrand at lysator.liu.se Wed Oct 13 19:50:54 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 19:51:07 2004 Subject: [Python-Dev] test_subprocess fails on OSX In-Reply-To: <416D3AAE.6090907@interlink.com.au> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> <416D3AAE.6090907@interlink.com.au> Message-ID: <Pine.GSO.4.51L2.0410131943250.16762@adriana.lysator.liu.se> On Thu, 14 Oct 2004, Anthony Baxter wrote: > On OSX 10.3.4, test_subprocess is failing intermittently when run > from regrtest: > > test_subprocess > this bit of output is from a test of stdout in a different process ... > test test_subprocess failed -- Traceback (most recent call last): > File "/Users/anthonybaxter/pyhead/src/Lib/test/test_subprocess.py", > line 434, in test_close_fds > self.assertEqual(p.stdout.read(), "3") > AssertionError: '4' != '3' > > Running just the test by itself works. I'm guessing it could be part of > the test suite leaking a FD or something? That shouldn't matter, since when you are using close_fds=True, the subprocess module will close all filedescriptors. For example, you can insert additional lines with os.pipe() at the beginning of test_close_fds, and the test (should) still succeed(s). /Peter ?strand <astrand@lysator.liu.se> From astrand at lysator.liu.se Wed Oct 13 19:56:06 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 19:56:12 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1f7befae041013083132866606@mail.gmail.com> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> <1f7befae041013083132866606@mail.gmail.com> Message-ID: <Pine.GSO.4.51L2.0410131952560.16762@adriana.lysator.liu.se> On Wed, 13 Oct 2004, Tim Peters wrote: > > But subprocess, the Python 2.4 version, will *never* use it, because > > _subprocess is always available (isn't this now even a builtin module, in > > python24.dll?). > > Yes, it is. I haven't had time to read the PEP, but I assumed some > "backward compatibility" constraint is at work here too, since, e.g., > subprocess.py has > > try: > False The PEP doesn't say anything about this, but my intention is that the module should work on Python 2.2 and newer. My employer really needs this. If the subprocess module in the Python CVS won't work with 2.2, I'm forced to keep a separate "fork". I would be happy if this could be avoided. /Peter ?strand <astrand@lysator.liu.se> From astrand at lysator.liu.se Wed Oct 13 20:03:18 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 20:03:31 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <1097681754.8607.26.camel@geddy.wooz.org> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> <1f7befae041013083132866606@mail.gmail.com> <1097681754.8607.26.camel@geddy.wooz.org> Message-ID: <Pine.GSO.4.51L2.0410132000260.16762@adriana.lysator.liu.se> >> Yes, it is. I haven't had time to read the PEP, but I assumed some >> "backward compatibility" constraint is at work here too, since, e.g., >> subprocess.py has >If so, then it should be documented in PEP 291, otherwise backward >compatibility stuff is fair game for ripping out. Below is a trivial patch to PEP 291. I'm stepping forward as a maintainer, if noone else wants to do the job. diff -u -r1.11 pep-0291.txt --- pep-0291.txt 20 Aug 2004 07:32:06 -0000 1.11 +++ pep-0291.txt 13 Oct 2004 18:00:14 -0000 @@ -90,6 +90,7 @@ modulefinder Thomas Heller 2.2 Just van Rossum platform Marc-Andre Lemburg 1.5.2 + subprocess Peter Astrand 2.2 Tool Maintainer(s) Python Version /Peter ?strand <astrand@lysator.liu.se> From barry at python.org Wed Oct 13 20:07:17 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 13 20:07:22 2004 Subject: [Python-Dev] Re: python/dist/src/Lib subprocess.py,NONE,1.1 In-Reply-To: <Pine.GSO.4.51L2.0410131952560.16762@adriana.lysator.liu.se> References: <E1CHOXl-0006Ph-Vn@sc8-pr-cvs1.sourceforge.net> <mzyr6p52.fsf@python.net> <ckimr5$rkb$1@sea.gmane.org> <fz4j6o3g.fsf@python.net> <ckioac$v84$1@sea.gmane.org> <acur6mqn.fsf@python.net> <ckiu5o$h27$1@sea.gmane.org> <wtxu6d3m.fsf@python.net> <1f7befae041013083132866606@mail.gmail.com> <Pine.GSO.4.51L2.0410131952560.16762@adriana.lysator.liu.se> Message-ID: <1097690837.8609.39.camel@geddy.wooz.org> On Wed, 2004-10-13 at 13:56, Peter Astrand wrote: > The PEP doesn't say anything about this, but my intention is that the > module should work on Python 2.2 and newer. My employer really needs this. > If the subprocess module in the Python CVS won't work with 2.2, I'm > forced to keep a separate "fork". I would be happy if this could be > avoided. There's no reason to fork it. (Reasonable) backward compatibility in the 2.4 library can be supported as long as it's documented, which is now is. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041013/dc2f70c5/attachment-0001.pgp From theller at python.net Wed Oct 13 20:12:33 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 13 20:12:19 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> (Alexandre Parenteau's message of "Wed, 13 Oct 2004 09:27:16 -0700") References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: <ekk25vxq.fsf@python.net> Alexandre Parenteau <aparente@adobe.com> writes: > Hi, > > Again forgive me if the answer is somewhere, but I could not find it. > > I am looking for a way to build and make an installer out of Python on > a Win x86_64 platform which is not supported yet by the Python team. > > My plan is to compile Python for this platform (several developers on > the list told me it should be straightforward), and make an installer > out of the Python binaries in order to be able to distribute the > Python binaries internally. > > I downloaded the evaluation copy of Wise Installer System 9, but it > complains that the installer file (which I assume is > PCBuild\Python20.wse) is corrupted. > > I need to establish exactly the version of the installer to use, if we > need to acquire a license for the Wise installer. Wise Installation System 9.0 - standard edition is what is used to build the Python 2.3 installers. The problem that you see has the usual cause - the PCBuild\Python20.wse file in the Python-2.3.4.tgz file, downloaded from Python.org, has unix line endings instead of dos line endings. You should convert them (loading the file into write, and saving it again) and it should load fine. Thomas From astrand at lysator.liu.se Wed Oct 13 20:15:18 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 13 20:15:26 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: <1f7befae041013091525b1835@mail.gmail.com> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <416C114B.8000906@interlink.com.au> <ckgrr3$bhg$1@sea.gmane.org> <ckilo6$org$1@sea.gmane.org> <ckimgu$qor$1@sea.gmane.org> <1f7befae041013091525b1835@mail.gmail.com> Message-ID: <Pine.GSO.4.51L2.0410132010170.16762@adriana.lysator.liu.se> On Wed, 13 Oct 2004, Tim Peters wrote: > > if so, would it perhaps make sense to add Peter as a developer, > > That would be fine by me, provided he asks for it (it's a commitment, > despite that developers routinely vanish for years at a time <wink>). I'm planning to be around for some time :) Developer access would be good, if I should maintain the subprocess module. (My SF account is "astrand".) > > and add a category for the subprocess system? > > That one's harder to call. Once a category is added, it can never be > removed. So I'd wait on that to see whether enough reports came in > against subprocess to justify it. Since it's unlikely that an > original reporter will assign "the right" category to begin with, and > the original submission is the only time at which auto-assignment has > an effect, the major value in all this would be achieved simply by > having a subprocess expert with commit privileges. Sounds good to me. /Peter ?strand <astrand@lysator.liu.se> From martin at v.loewis.de Wed Oct 13 20:51:46 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 20:51:46 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <20041012232318.091E.JCARLSON@uci.edu> References: <20041011232519.08C8.JCARLSON@uci.edu> <416CBBB0.9040809@v.loewis.de> <20041012232318.091E.JCARLSON@uci.edu> Message-ID: <416D7942.2000005@v.loewis.de> Josiah Carlson wrote: >>For those features, the >>potential user community is *much* larger than for the feature under >>discussion, and I feel I would waste my time for adding a feature that >>only few users ever need. > > > And yet you continue the discussion? This tends to get off-topic, but I typically respond when I'm asked a question. As you keep asking questions, I keep responding :-) > If every modification to Python required a community of people, who all > joined together to advocate something, then nothing would ever get done. No. Modifications to fix bugs don't need debating. Only new features do. I do wish that people would focus more on fixing bugs than on introducing new features. > People would be spending all their time trying to gather groups of > people to agree with them that functionality K is necessary. And that would be a good thing, atleast in the specific case. Even if it is undebated that the feature (convert numbers into odd-sized byte strings) is desirable, the specific API needs careful consideration, since it cannot be changed easily after it has been added. So all these new features are a serious legacy if they later turn out to be ill-designed. > I'm not even saying that we should add a new module. I'm not even > saying we should add a new function to a module. Heck, I'm not even > asking for a new argument to a function that previously existed. I am, > quite literally, asking for the equivalent of less than one bit in a > flag (there are currently 22 format/endian characters, which are all > orthogonal, which would require 5 bits, if they were flags). Correct. Yet I would feel more comfortable if you had proposed a new module, or a new function to an existing module. The former would allow distribution independent of Python, and the latter would not put a burden on users of the existing functions. > The API already exists. The framework already exists. I'm not asking > for Python to interpret something that was valid before as something > different now. Yes. The extension you propose probably does not cause backward compatibility problems. Strictly speaking, there are programs that break under this extension, but this is the case for any extension, and such programs likely don't occur in real life. > Try to read the above paragraph outside of the context of struct and the > RFE. Does it make sense to include the change now? No. For such a change, we need to study whether there alternative APIs which achieve the same effect (which there are), and whether the new flag puts an additional learning burden on users of the existing API (which it does). > If every request to interpret a new flag required significant community > involvement, goodness, would it take an act of Guido to get a commit > done? No, I have committed new features myself in the past. See the CVS log for details. It is this specific change I'm opposed to, at this point in time. Regards, Martin From martin at v.loewis.de Wed Oct 13 21:06:04 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 21:06:03 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416D4C45.3040802@interlink.com.au> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> <416D4C45.3040802@interlink.com.au> Message-ID: <416D7C9C.5080600@v.loewis.de> Anthony Baxter wrote: > Python's worked on the Alphas for approximately ever, and > works on Itanium, so I can't imagine you'll find any 32-bit-isms > left in the code. Actually, there are plenty of them, however it is unlikely that anybody will encounter them for a few more years. The biggest problem is that the number of elements in a sequence (string, list, tuple) is represented as "int", whereas it should be a signed variant of size_t. I have a patch that corrects many of these issues, but I plan to publish that only after 2.4. The problem becomes only apparent if you have more than 2G items. For a string, this might be a significant limitation. For a list, it requires 16GiB of memory to represent the list alone, not accounting for any items in the list, so very few people have sufficient hardware to even run a test case that demonstrates the problem. As for Alphas: their Windows port was a 32-bit port; Microsoft did not have Win64 at that time. Regards, Martin From martin at v.loewis.de Wed Oct 13 21:15:08 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 21:15:14 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: <416D7EBC.5040906@v.loewis.de> Alexandre Parenteau wrote: > I am looking for a way to build and make an installer out of Python on a > Win x86_64 platform which is not supported yet by the Python team. Out of curiosity: where did you obtain the operating system, and where will you obtain a C compiler for the platform? To my knowledge, Microsoft has released neither, yet. > My plan is to compile Python for this platform (several developers on > the list told me it should be straightforward), and make an installer > out of the Python binaries in order to be able to distribute the Python > binaries internally. > > I downloaded the evaluation copy of Wise Installer System 9, but it > complains that the installer file (which I assume is > PCBuild\Python20.wse) is corrupted. Alternatively to that approach, you can use the Python 2.4 code base, and build an MSI file (using Tools/msi); no need for any external tool except for a platform SDK installation, and PythonWin. Regards, Martin From nbastin at opnet.com Wed Oct 13 21:31:38 2004 From: nbastin at opnet.com (Nick Bastin) Date: Wed Oct 13 21:32:00 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D7EBC.5040906@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> Message-ID: <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> On Oct 13, 2004, at 3:15 PM, Martin v. L?wis wrote: > Alexandre Parenteau wrote: >> I am looking for a way to build and make an installer out of Python >> on a Win x86_64 platform which is not supported yet by the Python >> team. > > Out of curiosity: where did you obtain the operating system, and where > will you obtain a C compiler for the platform? To my knowledge, > Microsoft has released neither, yet. Microsoft made available an OS to the general public (2003 Pro, I think) for x86_64 at *least* 6 months ago, maybe more, and MSDN members have access to other OS builds as well (at least 2003 server). You have at least 2 compiler options, namely MSVC++ and GCC. ICL does not (yet) support x64_64. -- Nick From jack at performancedrivers.com Wed Oct 13 22:05:37 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Wed Oct 13 22:05:42 2004 Subject: [Python-Dev] 2.4b1 release, this Friday (AU time) -- imminent TRUNK FREEZING In-Reply-To: <416D6198.5000606@interlink.com.au> References: <416D6198.5000606@interlink.com.au> Message-ID: <20041013200537.GJ10507@performancedrivers.com> On Thu, Oct 14, 2004 at 03:10:48AM +1000, Anthony Baxter wrote: > Ok, I'm going to be cutting the Python release Friday (Australian > time). This means that I'm declaring the trunk frozen for the release > in about 24 hours. Lets say at 18:00 hours UTC on Thursday 14th > (that's 14:00 on the US east coast, if my memory is correct, and > about 04:00 where here in Australia). > US East Coast is currently GMT -400, West Coast is -700 I never pretend to know how daylight savings time works but I don't think it will change on a Friday. -Jack ps, flying from Philly to Syndey (layover in LAX) has the fun effect of taking no "time" going one way, and taking two "days" going the other. From martin at v.loewis.de Wed Oct 13 22:11:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 22:11:11 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> Message-ID: <416D8BE0.3000305@v.loewis.de> Nick Bastin wrote: > Microsoft made available an OS to the general public (2003 Pro, I think) > for x86_64 at *least* 6 months ago, maybe more, and MSDN members have > access to other OS builds as well (at least 2003 server). That was no release (in the sense of a product release) of the system, though, was it? My understanding that the system is still considered in beta by Microsoft. > You have at > least 2 compiler options, namely MSVC++ and GCC. ICL does not (yet) > support x64_64. How can you get a copy of MSVC++? VC.NET 2003 does not come with an AMD64 compiler to my knowledge, nor does the February 2003 (?) platform SDK incorporate one (which does incorporate an Itanium compiler). Regards, Martin From tim.peters at gmail.com Wed Oct 13 22:21:04 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 13 22:21:11 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> Message-ID: <1f7befae04101313214c5e4904@mail.gmail.com> [Alexandre Parenteau] > ... > - The ISS files (Inno) do not seem up to date. FYI, they were never up to date, and were never used to build a released installer. > Is anybody having a ISS version of the installer which works for 2.3.4? Sorry, I don't know of an Inno installer for any version of Python. Because there was a minor (but endless) stream of complaints that building the Python Windows installer required Wise, I checked in the "most of the way there, at the time" .iss file so that someone else could volunteer to finish it. Since nobody cared enough to do so, I ignored such complaints ever after <wink>. From nbastin at opnet.com Wed Oct 13 22:30:31 2004 From: nbastin at opnet.com (Nick Bastin) Date: Wed Oct 13 22:30:50 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D8BE0.3000305@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> Message-ID: <BA623DBD-1D56-11D9-8B3F-000D932927FE@opnet.com> On Oct 13, 2004, at 4:11 PM, Martin v. L?wis wrote: > Nick Bastin wrote: >> Microsoft made available an OS to the general public (2003 Pro, I >> think) for x86_64 at *least* 6 months ago, maybe more, and MSDN >> members have access to other OS builds as well (at least 2003 >> server). > > That was no release (in the sense of a product release) of the system, > though, was it? My understanding that the system is still considered > in beta by Microsoft. Yeah, that's why I didn't say 'released'. It's still a beta, but it certainly works and people are doing software development on it. My point was that people definitely have windows for x86_64, so it doesn't seem that unreasonable that they'd be trying to use Python on it - this isn't an isolated case. >> You have at least 2 compiler options, namely MSVC++ and GCC. ICL >> does not (yet) support x64_64. > > How can you get a copy of MSVC++? VC.NET 2003 does not come with an > AMD64 compiler to my knowledge, nor does the February 2003 (?) platform > SDK incorporate one (which does incorporate an Itanium compiler). There is a version of MSVC on the beta DDK CD. -- Nick From martin at v.loewis.de Wed Oct 13 22:59:45 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 13 22:59:44 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <BA623DBD-1D56-11D9-8B3F-000D932927FE@opnet.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> <BA623DBD-1D56-11D9-8B3F-000D932927FE@opnet.com> Message-ID: <416D9741.4080004@v.loewis.de> Nick Bastin wrote: > Yeah, that's why I didn't say 'released'. It's still a beta, but it > certainly works and people are doing software development on it. My > point was that people definitely have windows for x86_64, so it doesn't > seem that unreasonable that they'd be trying to use Python on it - this > isn't an isolated case. I didn't mean to suggest it was unreasonable. I was just wondering what precisely you need (in addition to the actual hardware, of course). Thanks for your elaboration, Martin From jcarlson at uci.edu Wed Oct 13 23:01:12 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed Oct 13 23:08:33 2004 Subject: [Python-Dev] Struct module format code addition (previously was in the subprocess thread) In-Reply-To: <416D7942.2000005@v.loewis.de> References: <20041012232318.091E.JCARLSON@uci.edu> <416D7942.2000005@v.loewis.de> Message-ID: <20041013130013.0944.JCARLSON@uci.edu> > > People would be spending all their time trying to gather groups of > > people to agree with them that functionality K is necessary. > > And that would be a good thing, atleast in the specific case. Even > if it is undebated that the feature (convert numbers into odd-sized > byte strings) is desirable, the specific API needs careful > consideration, since it cannot be changed easily after it has been > added. So all these new features are a serious legacy if they later > turn out to be ill-designed. If it turns out that the feature is ill-designed, then there is a standard method of dealing with it: 1. Declare that it will be Deprecated in the future. 2. Start raising a DeprecationWarning for at least one major version of Python. 3. Remove the functionality In the case of struct and this addition, if it is the case that this modification is catastrophic to Python (very unlikely), I would hope that someone would have said, "hey, this breaks applications A, B, C, D, ..., and it inclusion was the biggest mistake Python has ever made, even moreso than anyone ever responding to Josiah* in the first place." * Me trying to be funny. > > I'm not even saying that we should add a new module. I'm not even > > saying we should add a new function to a module. Heck, I'm not even > > asking for a new argument to a function that previously existed. I am, > > quite literally, asking for the equivalent of less than one bit in a > > flag (there are currently 22 format/endian characters, which are all > > orthogonal, which would require 5 bits, if they were flags). > > Correct. Yet I would feel more comfortable if you had proposed a new > module, or a new function to an existing module. The former would allow > distribution independent of Python, and the latter would not put a > burden on users of the existing functions. For those who use struct as it is now, the proposed additional format codes raise a struct.error exception in older versions of Python. If people are relying on pack/unpack with characters 'g' or 'G' raising a struct.error exception, then something is definitely wrong with the way they have been using struct. If people want to use the modifications with a previous version of Python, the changes are easily backported (I think one can even take structmodule.c from CVS and compile it with an older source tree). > Yes. The extension you propose probably does not cause backward > compatibility problems. Strictly speaking, there are programs that > break under this extension, but this is the case for any extension, > and such programs likely don't occur in real life. Agreed. > No. For such a change, we need to study whether there alternative APIs > which achieve the same effect (which there are), and whether the new > flag puts an additional learning burden on users of the existing API > (which it does). Tim Peters' alternative long2bytes and bytes2long looked quite usable, and indeed produces strikingly similar results, for single item packing and unpacking scenarios. The learning curve of struct was claimed to be steep in the case of native byte alignments, or for those not familliar with C types or structs. Since these codes are not native to any platform, it doesn't make the native byte alignment stuff any more difficult. If people are having trouble understanding the concept of C structs, I don't believe that two new format characters are going to be significantly more difficult to swallow than the 17 others that already exist. Especially when the standard C type codes 'cbhilqspBHILQ' are likely sufficient for their needs. In the case of them not being sufficient, well hey, they would have another two options that may do it for them. > > If every request to interpret a new flag required significant community > > involvement, goodness, would it take an act of Guido to get a commit > > done? > > No, I have committed new features myself in the past. See the CVS log > for details. It is this specific change I'm opposed to, at this point > in time. I was trying to be funny, to perhaps lighten the mood. Hence "act of Guido" rather than "act of God" (where the latter is a cliche). - Josiah From aparente at adobe.com Wed Oct 13 23:29:12 2004 From: aparente at adobe.com (Alexandre Parenteau) Date: Wed Oct 13 23:29:51 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <416D8BE0.3000305@v.loewis.de> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> Message-ID: <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> Thanks for all your responses, Unfortunately I don't know the specifics of how to get the compilers and OS. I am only in charge to find out if Python-Dev has made already the work. I suspect you get the OS and compilers from the MSDN Subscriber program. I will certainly let you know of our experience if/when we decide to compile Python on x86_64. It might not be before a while though, so probably someone else will be able to report before us. Thanks again, alex. At 01:11 PM 10/13/2004, Martin v. L?wis wrote: >Nick Bastin wrote: >>Microsoft made available an OS to the general public (2003 Pro, I think) >>for x86_64 at *least* 6 months ago, maybe more, and MSDN members have >>access to other OS builds as well (at least 2003 server). > >That was no release (in the sense of a product release) of the system, >though, was it? My understanding that the system is still considered >in beta by Microsoft. > >>You have at least 2 compiler options, namely MSVC++ and GCC. ICL does >>not (yet) support x64_64. > >How can you get a copy of MSVC++? VC.NET 2003 does not come with an >AMD64 compiler to my knowledge, nor does the February 2003 (?) platform >SDK incorporate one (which does incorporate an Itanium compiler). > >Regards, >Martin From abkhd at hotmail.com Thu Oct 14 00:05:11 2004 From: abkhd at hotmail.com (A.B., Khalid) Date: Thu Oct 14 00:06:06 2004 Subject: [Python-Dev] Inno-Setup for Python 2.3.4 Message-ID: <BAY18-F6HmdGOApCc1a00029e3f@hotmail.com> [Alexandre Parenteau] >... >- The ISS files (Inno) do not seem up to date.Is anybody having a ISS >version of the installer which works for 2.3.4? Hello Alex, I have been working on and off on an Inno-Setup installer for the MinGW compiled Python I've been working on. The reason being I have Win98 so the Msi.py approach does not work. The script is still not finished, nor is it identical to setup script found in the Python 2.3.4 sources. Moreover, it does not make any registry entries yet, which I am thinking and hoping of doing either through a Python script (which runs after install is finished and before uninstall is started) or an external DLL so as to be able to revert changes. But even in its current unfinished form, the script works for me in that it gathers all of the needed Python 2.3.4 files into one package and offers to install Python and Python's extensions, and uninstalls the files as needed. All the regrtest tests pass when Python 2.3.4 is installed from that script. One can run Python's scripts from the DOS Shell and if file association is urgently needed then hints from the Python.iss found in the original Python source distribution can be of immediate help. Get it here (3.49 KB): http://jove.prohosting.com/iwave/ipython/Inno-Python234.zip Regards Khalid PS: I have pymingw.py which is enclosed in the zip file where I have python23.dll and friends. It will be picked up by the script from there and placed in site-packages for use when uninstalling Python. _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From anthony at interlink.com.au Thu Oct 14 05:04:42 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu Oct 14 05:05:02 2004 Subject: [Python-Dev] Python on Windows 64bits? In-Reply-To: <416D7C9C.5080600@v.loewis.de> References: <6.1.1.1.2.20041012091430.043e0360@mailsj.corp.adobe.com> <416CBEE4.5040908@v.loewis.de> <6.1.1.1.2.20041013081218.02e9d7f0@mailsj.corp.adobe.com> <416D4C45.3040802@interlink.com.au> <416D7C9C.5080600@v.loewis.de> Message-ID: <416DECCA.6030408@interlink.com.au> Martin v. L?wis wrote: > As for Alphas: their Windows port was a 32-bit port; Microsoft > did not have Win64 at that time. However, the Unix port was 64 bits. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From tim.peters at gmail.com Thu Oct 14 05:16:42 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Oct 14 05:16:45 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess - updated popen5module) In-Reply-To: <Pine.GSO.4.51L2.0410132010170.16762@adriana.lysator.liu.se> References: <Pine.GSO.4.51L2.0410081137150.23826@koeberg.lysator.liu.se> <Pine.GSO.4.51L2.0410081256570.24457@koeberg.lysator.liu.se> <ca471dc20410111418614a0694@mail.gmail.com> <416C114B.8000906@interlink.com.au> <ckgrr3$bhg$1@sea.gmane.org> <ckilo6$org$1@sea.gmane.org> <ckimgu$qor$1@sea.gmane.org> <1f7befae041013091525b1835@mail.gmail.com> <Pine.GSO.4.51L2.0410132010170.16762@adriana.lysator.liu.se> Message-ID: <1f7befae041013201677f808ed@mail.gmail.com> [/F] >>> if so, would it perhaps make sense to add Peter as a developer, [Uncle Timmy] >> That would be fine by me, provided he asks for it (it's a commitment, >> despite that developers routinely vanish for years at a time <wink>). [Peter Astrand] > I'm planning to be around for some time :) Developer access would be good, > if I should maintain the subprocess module. (My SF account is "astrand".) If there are no objections before I wake up again, and nobody else has done it, I'll add Peter as a Python developer. From python at rcn.com Thu Oct 14 05:35:01 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Oct 14 05:36:32 2004 Subject: [Python-Dev] and a question to the SF admins (Re: subprocess -updated popen5module) In-Reply-To: <1f7befae041013201677f808ed@mail.gmail.com> Message-ID: <000001c4b19e$c9f5e120$e841fea9@oemcomputer> > [/F] > >>> if so, would it perhaps make sense to add Peter as a developer, > > [Uncle Timmy] > >> That would be fine by me, provided he asks for it (it's a commitment, > >> despite that developers routinely vanish for years at a time <wink>). > > [Peter Astrand] > > I'm planning to be around for some time :) Developer access would be > good, > > if I should maintain the subprocess module. (My SF account is > "astrand".) > > If there are no objections before I wake up again, and nobody else has > done it, I'll add Peter as a Python developer. Peter, your permissions are enabled. Welcome. Raymond From nbastin at opnet.com Wed Oct 13 23:40:45 2004 From: nbastin at opnet.com (Nick Bastin) Date: Thu Oct 14 06:26:18 2004 Subject: [Python-Dev] About the Wise installer on Windows In-Reply-To: <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <416D7EBC.5040906@v.loewis.de> <80AC2834-1D4E-11D9-8B3F-000D932927FE@opnet.com> <416D8BE0.3000305@v.loewis.de> <6.1.1.1.2.20041013135327.03c96e60@mailsj.corp.adobe.com> Message-ID: <8A2D172C-1D60-11D9-8B3F-000D932927FE@opnet.com> On Oct 13, 2004, at 5:29 PM, Alexandre Parenteau wrote: > Thanks for all your responses, > > Unfortunately I don't know the specifics of how to get the compilers > and OS. I am only in charge to find out if Python-Dev has made already > the work. > > I suspect you get the OS and compilers from the MSDN Subscriber > program. > > I will certainly let you know of our experience if/when we decide to > compile Python on x86_64. It might not be before a while though, so > probably someone else will be able to report before us. Oh, well, if you just wanted to know if it worked... :-) It does work. IIRC, you need to play with pyconfig.h a bit for a few 64-bit things, but it pretty much works out of the box. It might be easier at this point to check out the pyconfig.h that's used for the Itanium build - I don't think that was available when I did the first AMD64 build here. -- Nick From bac at OCF.Berkeley.EDU Thu Oct 14 07:20:28 2004 From: bac at OCF.Berkeley.EDU (Brett C) Date: Thu Oct 14 07:20:36 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] Message-ID: <416E0C9C.1020005@ocf.berkeley.edu> So we were all rather quiet in the last half of September. The whole summary fits on two sheets of 8.5x11 (normally it is over 10 and I have hit over 20 when I was summarizing *everything*). Going to send this out no earlier than Friday night so send in corrections by then. ---------------------------------- ===================== Summary Announcements ===================== Wow. This must have been the easiest summary I have ever done. Why can't they all be like this? I didn't even skip that much! ========= Summaries ========= ------------------------------------------ Assume nothing when mutability is possible ------------------------------------------ Tim Peters discovered a new way to create an infinite list thanks to generator expressions. But what really came out of this whole discussion came about when someone else came up with an example that exposed a bug in list.extend(). The first thing was that "you can't assume anything about a mutable object after potentially calling back into Python." Basically you can't assume the state of any mutable object was not changed if you execute Python code from C. While it might seem handy to store state while in a loop for instance, you can't count on things not change by the time you get control back so you just have to do it the hard way and get state all over again. Second was that you need to be careful when dealing with iterators. If you mutate an iterator while iterating you don't have a guarantee it won't explode in your face. Unless you explicitly support it, document it, and take care to protect against it then just don't assume you can mutate an iterator while using it. Contributing threads: - `A cute new way to get an infinite loop <>`__ - `More data points <>`__ ---------------------------- The less licenses the better ---------------------------- The idea of copying some code from OpenSSH_ for better pty handling was proposed. This was frowned upon since that becomes one more legal issue to keep track of. Minimizing the licenses that Python must keep track of and make sure to comply with, no matter how friendly, is a good thing. .. _OpenSSH: http://www.openssh.com/ Contributing threads: - `using openssh's pty code <>`__ ------------------------------------------------------------------------ Trying to deal with the exception hierarchy and a backwards-friendly way ------------------------------------------------------------------------- Nick Coghlan came up with the idea of having a tuple that contained all of the exceptions you normally would not want to catch in a blanket 'except' statement; KeyboardInterrupt, MemoryError, SystemExit, etc.). This tuple was proposed to live in sys.special_exceptions with the intended usage of:: try: pass # stuff... except sys.special_exceptions: raise # exceptions that you would not want to catch should keep propogating up the call chain except: pass # if you reach here the exception should not be a *huge* deal Obviously the best solution is to just clean up the exception hierarchy, but that breaks backwards-compatibility. But this idea seemed to lose steam. Contributing threads: - `Proposing a sys.special_exceptions tuple <>`__ =============== Skipped Threads =============== - Decimal, copyright and license - Planning to drop gzip compression for future releases. - built on beer? - Noam's open regex requests - Socket/Asyncore bug needs attention - open('/dev/null').read() -> MemoryError - Finding the module from PyTypeObject? - Odd compile errors for bad genexps - Running a module as a script From python-dev-list at cstork.org Thu Oct 14 07:34:09 2004 From: python-dev-list at cstork.org (Christian Stork) Date: Thu Oct 14 07:34:13 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109 In-Reply-To: <416C24E2.4020304@alumni.sfu.ca> References: <E1CHP2L-000059-FA@sc8-pr-cvs1.sourceforge.net> <ckgvsd$q41$1@sea.gmane.org> <416C24E2.4020304@alumni.sfu.ca> Message-ID: <20041014053409.GA24001@anthony.ics.uci.edu> On Tue, Oct 12, 2004 at 08:39:30PM +0200, Aaron Bingham wrote: > Fredrik Lundh wrote: ... > >I don't know how to add ISO-8859-1 characters to a Latex document, > >but in HTML notation, Peter's last name should be Åstrand. > > > > > I don't know if it's allowed for for Python docs, but the easiest way I > know of is to put > \usepackage{isolatin1} > in the prelude and just use regular ISO-8859-1 characters in the file. isolatin1.sty is obsolete. Use inputenc.sty instead. See page 11 of http://www.tug.org/tex-archive/info/l2tabu/english/l2tabuen.pdf for further details. -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From ncoghlan at iinet.net.au Thu Oct 14 14:01:37 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 14:00:35 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E0C9C.1020005@ocf.berkeley.edu> References: <416E0C9C.1020005@ocf.berkeley.edu> Message-ID: <416E6AA1.2030107@iinet.net.au> Brett C wrote: > - Running a module as a script That reminds me - the version of this that got checked in is restricted to top-level modules in order to keep things simple. I put a recipe up on the Python cookbook for those that wanted to be able to easily run scripts that live inside packages. The recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 Cheers, Nick. From barry at python.org Thu Oct 14 15:00:14 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 15:00:18 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E6AA1.2030107@iinet.net.au> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> Message-ID: <1097758814.7686.31.camel@geddy.wooz.org> On Thu, 2004-10-14 at 08:01, Nick Coghlan wrote: > That reminds me - the version of this that got checked in is restricted > to top-level modules in order to keep things simple. I put a recipe up > on the Python cookbook for those that wanted to be able to easily run > scripts that live inside packages. > > The recipe: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 Why not add this to the stdlib? I'd call it pyrun.py. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/9795713e/attachment.pgp From theller at python.net Thu Oct 14 15:06:07 2004 From: theller at python.net (Thomas Heller) Date: Thu Oct 14 15:05:50 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <1097758814.7686.31.camel@geddy.wooz.org> (Barry Warsaw's message of "Thu, 14 Oct 2004 09:00:14 -0400") References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> Message-ID: <hdoxla9s.fsf@python.net> Barry Warsaw <barry@python.org> writes: > On Thu, 2004-10-14 at 08:01, Nick Coghlan wrote: > >> That reminds me - the version of this that got checked in is restricted >> to top-level modules in order to keep things simple. I put a recipe up >> on the Python cookbook for those that wanted to be able to easily run >> scripts that live inside packages. >> >> The recipe: >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 > > Why not add this to the stdlib? I'd call it pyrun.py. ... and call it automatically when the '-m' implementation detects a dotted name? Thomas From mwh at python.net Thu Oct 14 15:35:23 2004 From: mwh at python.net (Michael Hudson) Date: Thu Oct 14 15:35:25 2004 Subject: [Python-Dev] Cyclic GC issues In-Reply-To: <20041013151956.GA9469@panix.com> (aahz@pythoncraft.com's message of "Wed, 13 Oct 2004 11:19:56 -0400") References: <20041010193958.GF341@canonware.com> <4169AB4B.9030605@v.loewis.de> <20041011065952.GH341@canonware.com> <1f7befae041011182151f7b219@mail.gmail.com> <20041012022547.GC27613@canonware.com> <1f7befae041011203471688554@mail.gmail.com> <20041013145404.GB33145@canonware.com> <20041013151956.GA9469@panix.com> Message-ID: <2mhdoxfmn8.fsf@starship.python.net> Aahz <aahz@pythoncraft.com> writes: > On Wed, Oct 13, 2004, Jason Evans wrote: >> >> Indeed. This brings up a question about documentation: as far as I >> know, I've read all of the available documentation regarding cyclic GC >> (the "Supporting cyclic garbage collection" section of "Extending and >> Embedding the Python Interpreter"), but those docs came nowhere near >> giving me an adequate understanding of the design rationale behind the >> cyclic GC. Did I overlook some documentation? >> >> Of course, it may be that I am actually at a cognitive disadvantage >> to someone who has only experienced Python's memory management >> strategy... > > As Tim said earlier, though less pungently, "Read the source, Luke." Indeed, but also: don't be scared! Modules/gcmodule.c isn't that long, and much of what it is is comments. Mind you, it was probably easier to understand before it worked properly with weak references... Cheers, mwh -- Unfortunately, nigh the whole world is now duped into thinking that silly fill-in forms on web pages is the way to do user interfaces. -- Erik Naggum, comp.lang.lisp From just at letterror.com Thu Oct 14 15:49:44 2004 From: just at letterror.com (Just van Rossum) Date: Thu Oct 14 15:49:40 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <416E6AA1.2030107@iinet.net.au> Message-ID: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> Nick Coghlan wrote: > Brett C wrote: > > - Running a module as a script > > That reminds me - the version of this that got checked in is > restricted to top-level modules in order to keep things simple. I put > a recipe up on the Python cookbook for those that wanted to be able > to easily run scripts that live inside packages. > > The recipe: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/307772 (Hm, I was going to reply to the discussion there, but I think it's more appropriate here.) I understand the difficulty of implementing it, but as a user I find it a really really stupid restriction. I routively run (doc)tests of individual modules, which usually are submodules of a package. Using -m to do this would help me tremendously. As it stands, -m doesn't help me at all. I'd even go so far and -1 the entire feature if it doesn't support submodules. I guess it's too late for that :) Just From barry at python.org Thu Oct 14 15:58:06 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 15:58:13 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <hdoxla9s.fsf@python.net> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> Message-ID: <1097762286.8756.12.camel@geddy.wooz.org> On Thu, 2004-10-14 at 09:06, Thomas Heller wrote: > ... and call it automatically when the '-m' implementation detects a > dotted name? +1. Also, if a non-dotted module name can't be found, pyrun should be called to see if the module is actually a package (with an __main__ that lives in the package's __init__.py). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/cded5503/attachment.pgp From bob at redivi.com Thu Oct 14 16:04:42 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Oct 14 16:05:20 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <1097762286.8756.12.camel@geddy.wooz.org> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> Message-ID: <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> On Oct 14, 2004, at 9:58 AM, Barry Warsaw wrote: > On Thu, 2004-10-14 at 09:06, Thomas Heller wrote: > >> ... and call it automatically when the '-m' implementation detects a >> dotted name? > > +1. Also, if a non-dotted module name can't be found, pyrun should be > called to see if the module is actually a package (with an __main__ > that > lives in the package's __init__.py). Wouldn't it make more sense to look for main rather than __main__? It seems that __main__ (as a function) occurs EXACTLY ZERO times in the standard library ;) -bob From barry at python.org Thu Oct 14 16:14:12 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 16:14:16 2004 Subject: [Python-Dev] python-dev Summary for 2004-09-16 through 2004-09-30 [draft] In-Reply-To: <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> Message-ID: <1097763252.8756.29.camel@geddy.wooz.org> On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: > > +1. Also, if a non-dotted module name can't be found, pyrun should be > > called to see if the module is actually a package (with an __main__ > > that > > lives in the package's __init__.py). > > Wouldn't it make more sense to look for main rather than __main__? It > seems that __main__ (as a function) occurs EXACTLY ZERO times in the > standard library ;) I think that's a fine convention, so +1. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/2ad7d4e6/attachment.pgp From hpk at trillke.net Thu Oct 14 17:10:10 2004 From: hpk at trillke.net (holger krekel) Date: Thu Oct 14 17:10:13 2004 Subject: [Python-Dev] PyPy Vilnius Sprint 15-23 nov 2004 Message-ID: <20041014151010.GQ15456@solar.trillke.net> Hi Pythonistas and interested developers, PyPy, the python-in-python implementation, is steadily moving on. The next coding sprint will take place in Vilnius, Lithunia, from 15th to 23rd of November, 2004 and is organized by the nice Programmers of Vilnius (POV) company. See http://codespeak.net/pypy/index.cgi?doc for more in-depth information about PyPy. Again, we will be heading towards a first generated C version of our already pretty compliant Python interpreter and types implementation. Last time, before the EuroPython 2004 conference, we actually had a similar goal (a PyPy/C-version) but discovered we had to largely refactor the basic model for attribute accesses. We are now closely mirroring the marvelous "descriptor"-mechanism of CPython. If you are interested to participate in our fun and somewhat mind-altering python sprint event then please subscribe at http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintAttendants and look around for more information. You'll find that most of the core PyPy developers are already determined to come. There are also many areas that need attention so that we should have tasks suited for different levels of expertise. At http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintTasks you'll find our sprint planning task list which will probably grow in the next weeks. Note that our EU funding efforts are at the final stage now. In the next weeks, quite likely before the Vilnius sprint, we _hope_ to get a "go!" from the european commission. One side effect would be that coders - probably restricted to european citizens - may generally apply for getting travel and accomodation costs refunded for PyPy sprints. This would reduce the barrier of entry to the question if you like to spend your time with a pypy sprint. However, we probably need some time to work out the details after when we get more information from the EU. If you have any questions don't hesitate to contact pypy-sprint@codespeak.net or one of us personally. cheers & a bientot, Holger Krekel, Armin Rigo From abkhd at hotmail.com Wed Oct 13 04:32:35 2004 From: abkhd at hotmail.com (Khalid A. B.) Date: Thu Oct 14 21:23:18 2004 Subject: [Python-Dev] test_cwd of test_subprocess.py fails on Win98 Message-ID: <BAY18-F35tg1zSarY6F0003935e@hotmail.com> Hello On Win98 test_cwd of test_subprocess.py tries to assert that "C:\WINDOWS\TEMP" is equal to "C:\WINDOWS\Temp" but since "TEMP" != "Temp" it fails, even when they are the same directory. Details follow: $ python -i Python 2.4a3 (#56, Oct 13 2004, 02:29:05) [GCC 3.4.1 (mingw special)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>>import os, sys, subprocess >>>tmpdir = os.getenv("TEMP", "/tmp") >>>tmpdir = os.path.realpath(tmpdir) >>>tmpdir 'C:\\WINDOWS\\TEMP' >>>p = subprocess.Popen([sys.executable, "-c", ... 'import sys,os;' \ ... 'sys.stdout.write(os.getcwd())'], ... stdout=subprocess.PIPE, ... cwd=tmpdir) >>>subProcessTmpDir = p.stdout.read() >>>subProcessTmpDir 'C:\\WINDOWS\\Temp' >>>tmpdir == subProcessTmpDir False >>> Here is what fixed it for me: Index: python/dist/src/Lib/test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.4 diff -u -d -r1.4 test_subprocess.py --- python/dist/src/Lib/test/test_subprocess.py 12 Oct 2004 22:29:54 -0000 1.4 +++ python/dist/src/Lib/test/test_subprocess.py 13 Oct 2004 02:27:13 -0000 @@ -192,9 +192,16 @@ def test_cwd(self): tmpdir = os.getenv("TEMP", "/tmp") tmpdir = os.path.realpath(tmpdir) - p = subprocess.Popen([sys.executable, "-c", - 'import sys,os;' \ - 'sys.stdout.write(os.getcwd())'], + commandStr = [sys.executable, "-c"] + if mswindows: + tmpdir = tmpdir.upper() + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd().upper())') + else: + commandStr.append('import sys,os;' \ + 'sys.stdout.write(os.getcwd())') + + p = subprocess.Popen(commandStr, stdout=subprocess.PIPE, cwd=tmpdir) self.assertEqual(p.stdout.read(), tmpdir) _________________________________________________________________ Express yourself instantly with MSN Messenger! Download today it's FREE! http://messenger.msn.com/ From ncoghlan at iinet.net.au Thu Oct 14 22:38:36 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 22:37:33 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> Message-ID: <416EE3CC.1030003@iinet.net.au> Just van Rossum wrote: > I understand the difficulty of implementing it, but as a user I find it > a really really stupid restriction. I routively run (doc)tests of > individual modules, which usually are submodules of a package. Using -m > to do this would help me tremendously. As it stands, -m doesn't help me > at all. I'd even go so far and -1 the entire feature if it doesn't > support submodules. I guess it's too late for that :) This may be more a case of my misjudging py-dev's likely collective reaction than anything else. Support for '-m' was lukewarm enough when it last came up, that I didn't expect to get a good reaction if I suggested adding a stdlib module in order to enhance it to support packages. While I wrote a patch to enable it (#1043356 - it uses the simple C-level strategy of 'try to locate at the top level, if that doesn't work, hand it over to the Python version'), we seemed to be too close to the beta to push for inclusion this time around. Add in the fact that I was about to be moving back to Brisbane after being Oregon for three months. . . (I'm back in Brisbane now, though) At the moment, '-m's behaviour is pretty easy to explain: "Look for a top-level module with the specified name. If it is found, run it as if its name had been fully specified on the command line. If it is not found, report an error" The behaviour currently implemented in the enhancement patch is: "Look for a top-level module with the specified name. If it is found, run it as if its name had been fully specified on the command line. If it is not found, attempt to import any containing package, then look for the module within that package. Run the located module as for a top-level module. If it is still not found, report an error. Note: For modules within packages, this differs slightly from running them directly from the command line by filename. Using this switch, the script's containing package is fully imported prior to execution of the script. This does not happen when the script's filename is used directly." As an implementation detail, the top-level scan is done in C, the scan that understands packages is done in Python. The main reasons for that are that the top-level scan gets used to *find* the Python version if it's needed, and even a simple scan looking for dots is a pain in C (although that *would* likely be slightly quicker than the current 'failed lookup' approach for scripts inside modules, it would also be slightly slower for top-level modules, as well as adding more code to main.c). Selling the additional complexity was the main reason I didn't expect to get a good reaction to this idea with the 2.4 beta almost out the door. I'm happy to make whatever changes to that patch are needed for inclusion (e.g. changing the module name, adding it to the docs underwhatever name is chosen) - I guess it's mainly Anthony's call whether he's prepared to accept such a change after the 2.4b1 release. Cheers, Nick. P.S. I'd also like some feedback on a quirk of the current version of that patch - as noted on SF, at the moment it potentially tramples on argv[0] at the C-level, which seems questionable given the existence of Py_GetArgcArgv(). The simplest way around that is to *not* set sys.argv[0] correctly when running pyrun/execmodule implicitly (i.e. sys.argv[0] may contain the name of the interpreter, or a command line switch, rather than the name of pyrun/execmodule - document this possibility with a comment in the __main__ block of pyrun/execmodule). From ncoghlan at iinet.net.au Thu Oct 14 23:06:55 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 14 23:05:52 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <1097763252.8756.29.camel@geddy.wooz.org> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> <1097763252.8756.29.camel@geddy.wooz.org> Message-ID: <416EEA6F.60708@iinet.net.au> Barry Warsaw wrote: > On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: > > >>>+1. Also, if a non-dotted module name can't be found, pyrun should be >>>called to see if the module is actually a package (with an __main__ >>>that >>>lives in the package's __init__.py). Actually, if we were going to do something for packages, I'd be more inclined to look for a script called __main__.py in the package directory, rather than looking for a __main__ function inside __init__.py. Or else simply run __init__.py itself as __main__ (i.e. allow the use of the existing 'Python main' idiom inside a package's __init__.py) (Interestingly, that's at least the second time it has been suggested to turn this idea into 'C-like main functions for Python'. '-m' is about another way to invoke the current 'if __name__ == "__main__":' idiom. It is most definitely *not* about creating a new idiom for main functions - an activity which would seem to be PEP-worthy) >>Wouldn't it make more sense to look for main rather than __main__? It >>seems that __main__ (as a function) occurs EXACTLY ZERO times in the >>standard library ;) > > > I think that's a fine convention, so +1. Except we'd be violating Python's tradition that magic methods start and end with double underscores, as well as potentially breaking existing scripts. At the moment, the following will print 'Hello World', but with 'def main' being special, it would do nothing: def main(*args, **kwds): pass if __name__ == "__main__: print "Hello world" main() I know that I often do any sys.argv manipulation in the 'if __name__' block rather than inside my main() function (which is often actually called 'main', and almost always takes a list of arguments rather than looking at sys.argv for itself). So while I'd be quite happy for code that included a "def __main__" to break (since the Python docs advise against using names in that format), allowing "def main" to suddenly acquire a magic meaning may not break the stdlib, but I bet it would break a lot of single-purpose scripts that are out there. Cheers, Nick. From pje at telecommunity.com Thu Oct 14 23:16:39 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Oct 14 23:16:13 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <416EEA6F.60708@iinet.net.au> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> <1097763252.8756.29.camel@geddy.wooz.org> Message-ID: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> At 07:06 AM 10/15/04 +1000, Nick Coghlan wrote: >Barry Warsaw wrote: >>On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: >> >>>>+1. Also, if a non-dotted module name can't be found, pyrun should be >>>>called to see if the module is actually a package (with an __main__ that >>>>lives in the package's __init__.py). > >Actually, if we were going to do something for packages, I'd be more >inclined to look for a script called __main__.py in the package directory, >rather than looking for a __main__ function inside __init__.py. Or else >simply run __init__.py itself as __main__ (i.e. allow the use of the >existing 'Python main' idiom inside a package's __init__.py) > >(Interestingly, that's at least the second time it has been suggested to >turn this idea into 'C-like main functions for Python'. '-m' is about >another way to invoke the current 'if __name__ == "__main__":' idiom. It >is most definitely *not* about creating a new idiom for main functions - >an activity which would seem to be PEP-worthy) Perhaps this means that -m is premature? I personally would rather wait for 2.5 if it means we get a nice, future-proof "main" convention out of the deal. While -m would not then be "backward compatible" with existing scripts, people could start changing scripts to match the convention as soon as there was an accepted PEP. From barry at python.org Thu Oct 14 23:16:33 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 14 23:16:40 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416EE3CC.1030003@iinet.net.au> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> Message-ID: <1097788593.8756.85.camel@geddy.wooz.org> On Thu, 2004-10-14 at 16:38, Nick Coghlan wrote: > While I wrote a patch to enable it (#1043356 - it uses the simple > C-level strategy of 'try to locate at the top level, if that doesn't > work, hand it over to the Python version'), we seemed to be too close to > the beta to push for inclusion this time around. Add in the fact that I > was about to be moving back to Brisbane after being Oregon for three > months. . . (I'm back in Brisbane now, though) Well, you're practically next door to Anthony (at least, compared to some of us), so I suggest you hop on down there on Friday, buy Anthony a beer or ten as a friendly convincer for slipping this into beta 1. To seal the deal, you can even offer to help (or threaten to harass) him while he makes the release. A win all around! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041014/395dbf4c/attachment.pgp From Scott.Daniels at Acm.Org Thu Oct 14 23:56:55 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu Oct 14 23:55:45 2004 Subject: [Python-Dev] Patch wanted Message-ID: <ckmskp$vu9$1@sea.gmane.org> Ray Hettinger's fix 1.29 to PC/pyconfig.h: Revision 1.29 - (view) (download) (annotate) - [select for diffs] CVS Tags: r24a2, r24a3 Changes since 1.28: +4 -2 lines Restore compilation on MSVC++ 6.0 =================================== /* Atleast VC 7.1 has them. If some compiler does not provide them, #ifdef appropriately .*/ #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 =================================== /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ #if _MSC_VER != 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 #endif =================================== Tests the wrong thing. His test presumes every non VC 7.1 PC compiler has those types defined. For non-MSC compilers, _MSC_VER is undefined, and the expression #if _MSC_VER != 1200 becomes: #if 0 != 1200 (Not what was intended, I presume). Even: #if _MSC_VER >= 1200 would be preferable (better failure, will presumably work with VC 7.2), but I believe the correct test should be: #if HAVE_STD_INT as I inexpertly explained in patch 1020042. This affects any C compilers on Windows that don't have the uintptr_t and intptr_t types (those without the newer include file stdint.h) except VC 6.0. Such compilers cannot compile any file that contains #include "Python.h" I think this needs to be fixed before 2.4 goes out. -- -- Scott David Daniels Scott.Daniels@Acm.Org From ncoghlan at iinet.net.au Fri Oct 15 01:27:05 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 15 01:26:24 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> <1097763252.8756.29.camel@geddy.wooz.org> <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> Message-ID: <416F0B49.1080008@iinet.net.au> Phillip J. Eby wrote: > Perhaps this means that -m is premature? I personally would rather wait > for 2.5 if it means we get a nice, future-proof "main" convention out of > the deal. While -m would not then be "backward compatible" with > existing scripts, people could start changing scripts to match the > convention as soon as there was an accepted PEP. I see the two issues as entirely orthogonal. A new main function idiom should not have to rely on the use of a particular command line switch (particularly since there is no way to 'switch off' the current idiom without significant changes to the interpreter). If we were actually developing a new idiom, I would be thinking more along the following lines: "After initial execution of the __main__ module in non-interactive mode (e.g. running a script supplied by filename or using -m), the interpreter looks for a function also called __main__ in the __main__ module's namespace. If such a function is found, it is executed by the interpreter as if the call __main__() had been appended to the end of the supplied script." Existing scripts would continue to work as is, or else could be converted to use the new idiom by replacing the line "if __name__ == '__main__':" with the line "def __main__():" Scripts that need to be backwards compatible can use the old idiom and include the following to mark themselves as definitely being usable as scripts: def __main__(): pass Or else, they can be made compatible with older versions by adding the following snippet to the end: if __name__ == '__main__': import sys if sys.version_info[0:2] < [2, 5]: __main__() About the only real practical (as opposed to aesthetic) advantage I see to such an idiom is that the question "Is this module useful as a script?" can easily be answered as "Yes" by looking for a __main__ function. (We can't really answer 'No' definitively, since a script may be using the existing idiom without using the 'def __main__(): pass' trick). Although "def __main__():" could be easier to explain to beginners than the current approach. That might also be a slight benefit. *stops to think for a bit* Actually, while proofreading this, one other potential advantage occured to me - this might allow C extensions and the like to define __main__ functions, and so be usable as scripts via -m despite their non-script packaging. Although that seems like an awful lot of work for not much gain - it would presumably be easier to just distribute a package that included both the extension module and a 'launcher' Python script. Anyway, regardless of whether a new main idiom is ever selected or not, I just don't see the link between that and being able to search for scripts using Python's module namespace instead of the OS filesystem's namespace. Cheers, Nick. From python at rcn.com Fri Oct 15 04:55:46 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 15 04:57:31 2004 Subject: [Python-Dev] Patch wanted In-Reply-To: <ckmskp$vu9$1@sea.gmane.org> Message-ID: <001701c4b262$78984240$e841fea9@oemcomputer> > Revision 1.29 . . . > Tests the wrong thing . . . > This affects any C compilers on Windows that don't have the uintptr_t > and intptr_t types (those without the newer include file stdint.h) > except VC 6.0. Here's a proposed patch: --- pyconfig.h 23 Sep 2004 19:11:32 -0000 1.30 +++ pyconfig.h 15 Oct 2004 02:44:27 -0000 @@ -273,10 +273,12 @@ /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ -#if _MSC_VER != 1200 +#if defined(_MSC_VER) +#if _MSC_VER > 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 -#endif +#endif /* _MSC_VER > 1200 */ +#endif /* _MSC_VER */ #endif > Ray Hettinger's fix 1.29 to PC/pyconfig.h . . . > His test presumes . . . Please watch the personalisms. I'm not the one who broke the build in the first place. Ideally, someone who knows all about the various compilers can propose a generic fix so that a lot of the ifdeffery in this file can be taken out. For someone trying to repair a broken build, they can often only test in one environment. Raymond Side note: It looks like I may be the only one testing/maintaining the MSVC++6.0 build. It got broken again this week, so I have to fix it up tonight (looks like a minor repair though). From kbk at shore.net Fri Oct 15 05:48:43 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Fri Oct 15 05:48:50 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410150348.i9F3mhO6013761@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 240 open ( -1) / 2655 closed (+15) / 2895 total (+14) Bugs : 766 open ( +0) / 4514 closed (+22) / 5280 total (+22) RFE : 155 open ( +1) / 131 closed ( +0) / 286 total ( +1) New / Reopened Patches ______________________ True/False instead of 1/0 in libstdtypes.tex (2004-10-06) CLOSED http://python.org/sf/1041364 opened by Gerrit Holl bbetter document popenX 'cmd' argument (2004-10-07) CLOSED http://python.org/sf/1042705 opened by Jeff Epler broken TeX markup (2004-10-08) CLOSED http://python.org/sf/1042969 opened by George Yoshida Decimal.py enhancements (2004-10-08) CLOSED http://python.org/sf/1043218 opened by Facundo Batista Enhancing '-m' to support packages (2004-10-09) http://python.org/sf/1043356 opened by Nick Coghlan External storage protocol for large email messages (2004-10-09) http://python.org/sf/1043706 opened by Barry A. Warsaw tarfile: add extractall() method (2004-10-10) http://python.org/sf/1043890 opened by Lars Gust?bel tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 opened by Lars Gust?bel Bug #1011380 alternate fix - locking is noop without threads (2004-10-11) CLOSED http://python.org/sf/1044089 opened by Nick Coghlan Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 opened by James William Pye improved configure.in output (2004-10-12) http://python.org/sf/1045620 opened by Wummel Remove unncessary NLST from ftp transfers (2004-10-12) http://python.org/sf/1045783 opened by Chris Cogdon Consistent retrieval of Python version. (2004-10-14) http://python.org/sf/1046831 opened by uiltje Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 opened by Thomas Heller Patches Closed ______________ -m option to run a module as a script (2004-09-27) http://python.org/sf/1035498 closed by rhettinger True/False instead of 1/0 in libstdtypes.tex (2004-10-06) http://python.org/sf/1041364 closed by rhettinger pdb enhancements and bug fixes (2004-02-12) http://python.org/sf/896011 closed by jlgijsbers bbetter document popenX 'cmd' argument (2004-10-08) http://python.org/sf/1042705 closed by jlgijsbers popen2.popen2() 'cmd' list syntax docstring (2004-02-01) http://python.org/sf/888380 closed by jlgijsbers broken TeX markup (2004-10-08) http://python.org/sf/1042969 closed by jlgijsbers Decimal.py enhancements (2004-10-08) http://python.org/sf/1043218 closed by rhettinger replacing rfc822.formatdate (2004-08-09) http://python.org/sf/1005857 closed by anthonybaxter email/__init__.py: Parse headersonly (2004-03-20) http://python.org/sf/920037 closed by bwarsaw Bad behavior of email.Charset.Charset when locale is tr_TR (2003-12-29) http://python.org/sf/866982 closed by bwarsaw mailbox / fromline matching (2002-02-22) http://python.org/sf/521478 closed by bwarsaw Bug #1011380 alternate fix - locking is noop without threads (2004-10-10) http://python.org/sf/1044089 closed by tim_one Logger file objects (2004-08-02) http://python.org/sf/1001864 closed by vsajip Signal fixes for BSD systems (2004-06-18) http://python.org/sf/975056 closed by anthonybaxter Allow ldshared to be overridden by environment var (2004-07-01) http://python.org/sf/983206 closed by anthonybaxter New / Reopened Bugs ___________________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) CLOSED http://python.org/sf/1041325 opened by Arthur Lutz SimpleXMLRPCServer example is broken (2004-10-06) CLOSED http://python.org/sf/1041501 opened by Kent Johnson Thread management corrupts heap (2004-10-06) CLOSED http://python.org/sf/1041645 opened by benson margulies Patch (2004-10-07) CLOSED http://python.org/sf/1042179 opened by Michiel de Hoon Lib/compiler chokes on certain genexps (2004-10-07) CLOSED http://python.org/sf/1042238 opened by Michael Hudson mimetypes.guess_extension('text/plain') = '.ksh' ??? (2004-10-08) http://python.org/sf/1043134 opened by Robert Kiendl Uninterruptable loop (2004-10-09) http://python.org/sf/1043446 opened by Jp Calderone compile error with stlport (2004-10-10) http://python.org/sf/1044299 opened by Debug Can't raise "C API version mismatch" warning (2004-10-11) http://python.org/sf/1044382 opened by Frank Sonnenburg docs for Py_UNICODE are wrong (2004-10-11) http://python.org/sf/1044479 opened by Thomas Heller Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 opened by Colin J. Williams strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 opened by Sebastien JUST Problems with os.system and ulimit -f (2004-10-12) http://python.org/sf/1045509 opened by Tomasz Kowaltowski test_subprocess vs Windows (2004-10-12) CLOSED http://python.org/sf/1045748 opened by Tim Peters warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 opened by Eugene Sizikov urllib2: better import ftplib and gopherlib etc late (2004-10-13) http://python.org/sf/1046077 opened by Robert Kiendl HTMLParser fix to accept mailformed tag attributes (2004-10-13) http://python.org/sf/1046092 opened by Vsevolod Novikov improving distutils swig support (2004-10-14) CLOSED http://python.org/sf/1046404 opened by Anthony Baxter improving distutils swig support (2) (2004-10-14) CLOSED http://python.org/sf/1046644 opened by Lars Immisch difflib.HtmlDiff doc errors (EASY) (2004-10-13) CLOSED http://python.org/sf/1046690 opened by Dan Gass Error in the given example (2004-10-14) http://python.org/sf/1046800 opened by Vishnu httplib index out of range (2004-10-14) CLOSED http://python.org/sf/1046855 opened by Grzegorz Makarewicz improving distutils swig support - documentation (2004-10-14) http://python.org/sf/1046945 opened by Lars Immisch cgitb failures (2004-10-14) http://python.org/sf/1047397 opened by Robin Becker Bugs Closed ___________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) http://python.org/sf/1041325 closed by arthur_lutz SimpleXMLRPCServer example is broken (2004-10-06) http://python.org/sf/1041501 closed by akuchling Thread management corrupts heap (2004-10-06) http://python.org/sf/1041645 closed by tim_one PyOS_InputHook broken (2004-09-19) http://python.org/sf/1030629 closed by mwh Patch (2004-10-07) http://python.org/sf/1042179 closed by mwh Lib/compiler chokes on certain genexps (2004-10-07) http://python.org/sf/1042238 closed by mwh Enhancements to pdb.py when invoked as script (2003-06-09) http://python.org/sf/751124 closed by jlgijsbers shutils.rmtree() uses excessive amounts of memory (2004-09-09) http://python.org/sf/1025127 closed by jlgijsbers Conflicting descriptions of application order of decorators (2004-09-21) http://python.org/sf/1031897 closed by akuchling x, y in curses window object documentation (2004-09-04) http://python.org/sf/1022311 closed by akuchling Confusing description of strict option for email.Parser (2004-09-22) http://python.org/sf/1032960 closed by bwarsaw email package uses \n to rebuild content of a message (2004-06-01) http://python.org/sf/964433 closed by bwarsaw smtpd.py docstring says wrong default class (2004-08-16) http://python.org/sf/1010102 closed by bwarsaw email: mishandles Content-Transfer-Enc. for text/* messages (2004-05-09) http://python.org/sf/950747 closed by bwarsaw Email message croaks the new email pkg parser (2004-09-19) http://python.org/sf/1030941 closed by bwarsaw Make GIL acquire/release functions noops if PyEval_Init (2004-08-18) http://python.org/sf/1011380 closed by tim_one test_subprocess vs Windows (2004-10-12) http://python.org/sf/1045748 closed by tim_one BSD restartable signals not correctly disabled (2004-06-09) http://python.org/sf/969574 closed by anthonybaxter Solaris likes sys/loadavg.h (2004-06-22) http://python.org/sf/977343 closed by anthonybaxter improving distutils swig support (2004-10-14) http://python.org/sf/1046404 closed by anthonybaxter improving distutils swig support (2) (2004-10-14) http://python.org/sf/1046644 closed by anthonybaxter difflib.HtmlDiff doc errors (EASY) (2004-10-13) http://python.org/sf/1046690 closed by tim_one httplib index out of range (2004-10-14) http://python.org/sf/1046855 closed by rhettinger From ilya at bluefir.net Fri Oct 15 06:25:14 2004 From: ilya at bluefir.net (Ilya Sandler) Date: Fri Oct 15 06:23:42 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> References: <1097763252.8756.29.camel@geddy.wooz.org> <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> <1097763252.8756.29.camel@geddy.wooz.org> <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> Message-ID: <Pine.LNX.4.58.0410142115280.648@bagira> On Thu, 14 Oct 2004, Phillip J. Eby wrote: > Perhaps this means that -m is premature? I personally would rather wait > for 2.5 if it means we get a nice, future-proof "main" convention out of > the deal. While -m would not then be "backward compatible" with existing > scripts, people could start changing scripts to match the convention as > soon as there was an accepted PEP. But to me -m option and __main__() conventions seem like orthogonal features... Even if current __name__=="__main__" blocks get replaced by a magic __main__() function, you would still benefit from -m cmd line option Or is there some hidden dependency? Ilya > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ilya%40bluefir.net > From anthony at interlink.com.au Fri Oct 15 06:36:37 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 15 06:37:01 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <1097788593.8756.85.camel@geddy.wooz.org> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> Message-ID: <416F53D5.7090606@interlink.com.au> Barry Warsaw wrote: > Well, you're practically next door to Anthony (at least, compared to > some of us), so I suggest you hop on down there on Friday, buy Anthony a > beer or ten as a friendly convincer for slipping this into beta 1. To > seal the deal, you can even offer to help (or threaten to harass) him > while he makes the release. A win all around! I'm extremely unconvinced that the semantics of "-m package" or "-m package.module" are suitably well thought out to see it in b1. If a single compelling way of making it work can be seen in the next week, _maybe_ we could sneak it into b2, but I'm really not hopeful. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From fdrake at acm.org Fri Oct 15 07:40:13 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 15 07:40:26 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416F53D5.7090606@interlink.com.au> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> Message-ID: <200410150140.13330.fdrake@acm.org> On Friday 15 October 2004 12:36 am, Anthony Baxter wrote: > I'm extremely unconvinced that the semantics of "-m package" or > "-m package.module" are suitably well thought out to see it in b1. I don't see this as a useful feature at all, so I guess I'm biased, but if it's hard to decide just what the right semantics are, this certainly isn't the time for it. We won't be able to change it substantially later due to backward compatibility constraints. > If a single compelling way of making it work can be seen in the > next week, _maybe_ we could sneak it into b2, but I'm really not > hopeful. Let's keep it out of 2.4 and see what proposals show up for 2.5. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From ncoghlan at iinet.net.au Fri Oct 15 09:57:19 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 15 09:56:15 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416F53D5.7090606@interlink.com.au> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> Message-ID: <416F82DF.7010005@iinet.net.au> Anthony Baxter wrote: > I'm extremely unconvinced that the semantics of "-m package" or > "-m package.module" are suitably well thought out to see it in b1. This was one of the reasons I wrote execmodule - to figure out semantics which made sense, after my initial attempt at supporting "-m package.module" failed to handle __path__ correctly. Executing anything other than PY_SOURCE or PY_COMPILED modules simply doesn't make sense to me (both execmodule and the current '-m' implementation report errors if you try to do so) > If a single compelling way of making it work can be seen in the > next week, _maybe_ we could sneak it into b2, but I'm really not > hopeful. The approach of 'import the containing package' seems to work fairly well. I can't see any other way to get at the package's __path__ variable in order to locate the module inside it. I haven't seen any good reasons to lift the PY_SOURCE/PY_COMPILED restriction, though. FWIW, the example which convinced me that running modules inside packages was valuable was "python -m pychecker.checker <script>". However, given that execmodule can provide this functionality fairly easily, I voluntarily bumped the relevant patch to Python 2.5 and posted the Cookbook recipe instead. If we do postpone this, I'd suggest putting something in the relevant Python 2.4 "module not found" error message in main.c that notes that packages aren't supported, though (otherwise I expect we'll get at least a few bug reports about not finding modules inside packages). Cheers, Nick. From adamsz at gmail.com Fri Oct 15 10:00:34 2004 From: adamsz at gmail.com (Adam Souzis) Date: Fri Oct 15 10:00:36 2004 Subject: [Python-Dev] Magic main functions In-Reply-To: <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> References: <416E0C9C.1020005@ocf.berkeley.edu> <416E6AA1.2030107@iinet.net.au> <1097758814.7686.31.camel@geddy.wooz.org> <hdoxla9s.fsf@python.net> <1097762286.8756.12.camel@geddy.wooz.org> <FF090A11-1DE9-11D9-9A2E-000A95686CD8@redivi.com> <1097763252.8756.29.camel@geddy.wooz.org> <416EEA6F.60708@iinet.net.au> <5.1.1.6.0.20041014171342.02d01600@mail.telecommunity.com> Message-ID: <d7cee4a704101501004524edcd@mail.gmail.com> Here's another good reason to change the way "main()" works today: A consequence of having the invoked python script's module always be named "__main__" is that when another module imports it by its real name the module will be loaded twice with 2 separate class and function definitions, etc. This can lead to unobvious bugs; for example, an except clause will think your trying to catch another type of exception. to illustrate: anothermodule.py: import mymodule def raiseFoo(): raise mymodule.foo() mymodule.py: import anothermodule class foo(exception): pass if __name__ == '__main__': try: anothermodule.raiseFoo() except foo: #nope, this isn't going to work 'cuz #its trying to catch __main__.foo not mymodule.foo print 'caught foo!' One way to prevent bugs like this (which can be tricky to track down) is to use an idiom like this for modules that can be invoked from the command line: if __name__ == '__main__': import mymodule mymodule.main() else: #define the module contents in this conditional block, including a main() BTW, the way this problem also exists when doing a relative import of module inside a package. e.g.: package1/example.py: import package1.module1 import module1 #module1 is loaded twice, treated as different modules -- adam On Thu, 14 Oct 2004 17:16:39 -0400, Phillip J. Eby <pje@telecommunity.com> wrote: > At 07:06 AM 10/15/04 +1000, Nick Coghlan wrote: > >Barry Warsaw wrote: > >>On Thu, 2004-10-14 at 10:04, Bob Ippolito wrote: > >> > >>>>+1. Also, if a non-dotted module name can't be found, pyrun should be > >>>>called to see if the module is actually a package (with an __main__ that > >>>>lives in the package's __init__.py). > > > >Actually, if we were going to do something for packages, I'd be more > >inclined to look for a script called __main__.py in the package directory, > >rather than looking for a __main__ function inside __init__.py. Or else > >simply run __init__.py itself as __main__ (i.e. allow the use of the > >existing 'Python main' idiom inside a package's __init__.py) > > > >(Interestingly, that's at least the second time it has been suggested to > >turn this idea into 'C-like main functions for Python'. '-m' is about > >another way to invoke the current 'if __name__ == "__main__":' idiom. It > >is most definitely *not* about creating a new idiom for main functions - > >an activity which would seem to be PEP-worthy) > > Perhaps this means that -m is premature? I personally would rather wait > for 2.5 if it means we get a nice, future-proof "main" convention out of > the deal. While -m would not then be "backward compatible" with existing > scripts, people could start changing scripts to match the convention as > soon as there was an accepted PEP. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/adamsz%40gmail.com > From fredrik at pythonware.com Fri Oct 15 10:04:20 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Oct 15 10:02:14 2004 Subject: [Python-Dev] Re: python/dist/src/Objects unicodeobject.c, 2.228, 2.229 References: <E1CIMlw-0002Hs-Jp@sc8-pr-cvs1.sourceforge.net> Message-ID: <cko062$71c$1@sea.gmane.org> > Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar. > > Python 2.3.x candidate. why bother? the unicode object you're copying to holds size+1 characters, so all the code does is copying an extra NULL character... completely harmless. </F> From mal at egenix.com Fri Oct 15 10:05:30 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Fri Oct 15 10:05:33 2004 Subject: [Python-Dev] Re: python/dist/src/Objects unicodeobject.c, 2.228, 2.229 In-Reply-To: <cko062$71c$1@sea.gmane.org> References: <E1CIMlw-0002Hs-Jp@sc8-pr-cvs1.sourceforge.net> <cko062$71c$1@sea.gmane.org> Message-ID: <416F84CA.3090001@egenix.com> Fredrik Lundh wrote: >>Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar. >> >>Python 2.3.x candidate. > > > why bother? > > the unicode object you're copying to holds size+1 characters, so all the > code does is copying an extra NULL character... completely harmless. That's true for Py_UNICODE, but not necessarily for the wchar_t* buffer your dealing with. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 15 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From jjl at pobox.com Fri Oct 15 12:49:04 2004 From: jjl at pobox.com (John J Lee) Date: Fri Oct 15 12:48:27 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 Message-ID: <Pine.LNX.4.58.0409160153330.6042@alice> Would anyone care to check this cookielib.py patch in before 2.4b1? http://python.org/sf/1028908 Quote from above: The patch contains uncontroversial changes to cookielib and associated modules, documentation and tests. Would be unfortunate not to have these in 2.4.0. [...SNIP...] John From barry at python.org Fri Oct 15 13:58:05 2004 From: barry at python.org (Barry Warsaw) Date: Fri Oct 15 13:58:11 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <416F53D5.7090606@interlink.com.au> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> Message-ID: <1097841485.7804.31.camel@geddy.wooz.org> On Fri, 2004-10-15 at 00:36, Anthony Baxter wrote: > I'm extremely unconvinced that the semantics of "-m package" or > "-m package.module" are suitably well thought out to see it in b1. > > If a single compelling way of making it work can be seen in the > next week, _maybe_ we could sneak it into b2, but I'm really not > hopeful. Okay, based on the discussions so far, I'm going to have to agree. While I do think it's a very useful feature, it's more important to do it right than to do it right now. Which means it should probably go through the full PEP treatment. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041015/a98f283a/attachment.pgp From anthony at interlink.com.au Fri Oct 15 14:27:00 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 15 14:27:26 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 In-Reply-To: <Pine.LNX.4.58.0409160153330.6042@alice> References: <Pine.LNX.4.58.0409160153330.6042@alice> Message-ID: <416FC214.7010505@interlink.com.au> John J Lee wrote: > Would anyone care to check this cookielib.py patch in before 2.4b1? Erm - 2.4b1 has been tagged, and I've cut the tarballs. Waiting for the msi installers before I send out the email. Sorry. Anthony From nas at arctrix.com Fri Oct 15 16:18:53 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Fri Oct 15 16:18:56 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <1097841485.7804.31.camel@geddy.wooz.org> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> <1097841485.7804.31.camel@geddy.wooz.org> Message-ID: <20041015141853.GC9544@mems-exchange.org> On Fri, Oct 15, 2004 at 07:58:05AM -0400, Barry Warsaw wrote: > While I do think it's a very useful feature, it's more important to do > it right than to do it right now. Which means it should probably go > through the full PEP treatment. Yes, it should. Neil From python at rcn.com Fri Oct 15 17:24:50 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 15 17:27:06 2004 Subject: [Python-Dev] Patch wanted In-Reply-To: <416FC8D7.80801@Acm.Org> Message-ID: <001b01c4b2cb$25205e40$e841fea9@oemcomputer> > I > did my best to determine what the core condition was and propose a > change there. Thanks for raising the issue in time to get it fixed. It went in last night and the build should be fine now. Raymond From fredrik at pythonware.com Fri Oct 15 17:32:39 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri Oct 15 17:30:40 2004 Subject: [Python-Dev] Re: Re: python/dist/src/Objects unicodeobject.c, 2.228, 2.229 References: <E1CIMlw-0002Hs-Jp@sc8-pr-cvs1.sourceforge.net><cko062$71c$1@sea.gmane.org> <416F84CA.3090001@egenix.com> Message-ID: <ckoqek$htn$1@sea.gmane.org> >>>Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar. >>> >>>Python 2.3.x candidate. >> >> why bother? >> >> the unicode object you're copying to holds size+1 characters, so all the >> code does is copying an extra NULL character... completely harmless. > > That's true for Py_UNICODE, but not necessarily for the > wchar_t* buffer your dealing with. oh, I missed that the patch modified two functions... (and overwriting the null char in FromWideChar is only a good idea if you're overwriting it with another null character...) </F> From martin at v.loewis.de Fri Oct 15 19:17:18 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Oct 15 19:17:14 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 In-Reply-To: <Pine.LNX.4.58.0409160153330.6042@alice> References: <Pine.LNX.4.58.0409160153330.6042@alice> Message-ID: <4170061E.8030302@v.loewis.de> John J Lee wrote: > The patch contains uncontroversial changes to cookielib and associated > modules, documentation and tests. Would be unfortunate not to have these > in 2.4.0. What precisely makes these changes uncontroversial? The fact that no controversion has been developed? At this point, only bug fixes are likely not to cause any controversion. For feature changes (such as the changes to the domain processing), I would need to see examples of what precisely the change is, e.g. by presenting an input to which the output will be different, preferably with an identification of the old and the new output and a reasoning why the new output is better. Regards, Martin From python at rcn.com Fri Oct 15 19:44:18 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Oct 15 19:46:28 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 In-Reply-To: <4170061E.8030302@v.loewis.de> Message-ID: <002601c4b2de$98ed7840$e841fea9@oemcomputer> > John J Lee wrote: > > The patch contains uncontroversial changes to cookielib and associated > > modules, documentation and tests. Would be unfortunate not to have these > > in 2.4.0. [Martin] > What precisely makes these changes uncontroversial? The fact that no > controversion has been developed? I've looked over the patch and think most of it should wait until Py2.5. Refactorings of this magnitude should be done early in the development cycle to allow time to discover and fix issues if something gets mucked up. A couple of the doc changes should probably go in for Py2.4b2 (a note on thread safety and an example of how to display debugging information). If no one objects, I'll put those in after the beta. Also there is a minor readability fix (using escape sequences) for the test suite that is reasonable for Py2.4. Raymond From martin at v.loewis.de Fri Oct 15 19:58:04 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Oct 15 19:58:00 2004 Subject: [Python-Dev] Minor cookielib changes for 2.4b1 In-Reply-To: <002601c4b2de$98ed7840$e841fea9@oemcomputer> References: <002601c4b2de$98ed7840$e841fea9@oemcomputer> Message-ID: <41700FAC.3010205@v.loewis.de> Raymond Hettinger wrote: > A couple of the doc changes should probably go in for Py2.4b2 (a note on > thread safety and an example of how to display debugging information). > If no one objects, I'll put those in after the beta. Also there is a > minor readability fix (using escape sequences) for the test suite that > is reasonable for Py2.4. Applying these doc changes is fine with me. I'm not always sure they are improvements (they don't make things worse, either) - but then, I'm not a native speaker, so not really qualified to judge them. Regards, Martin From db3l at fitlinxx.com Fri Oct 15 20:24:07 2004 From: db3l at fitlinxx.com (David Bolen) Date: Fri Oct 15 20:24:22 2004 Subject: [Python-Dev] Re: About the Wise installer on Windows References: <6.1.1.1.2.20041013091355.03f9a570@mailsj.corp.adobe.com> <1f7befae04101313214c5e4904@mail.gmail.com> Message-ID: <uu0svn8l4.fsf@fitlinxx.com> Tim Peters <tim.peters@gmail.com> writes: > Sorry, I don't know of an Inno installer for any version of Python. > Because there was a minor (but endless) stream of complaints that > building the Python Windows installer required Wise, I checked in the > "most of the way there, at the time" .iss file so that someone else > could volunteer to finish it. Since nobody cared enough to do so, I > ignored such complaints ever after <wink>. I think that occurred right about the same time that Wise donated the more up to date version of their package. Thus the existing Python installer no longer generated (ignoreable) errors under NT, for example, and was brought up to date presentation wise, so that pretty much resolved the complaints. At least that definitely did it for me, and I had been looking at finishing the file at the time. -- David From bac at OCF.Berkeley.EDU Fri Oct 15 20:40:22 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 15 20:40:35 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <1097841485.7804.31.camel@geddy.wooz.org> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> <1097841485.7804.31.camel@geddy.wooz.org> Message-ID: <41701996.7080905@ocf.berkeley.edu> Barry Warsaw wrote: > On Fri, 2004-10-15 at 00:36, Anthony Baxter wrote: > > >>I'm extremely unconvinced that the semantics of "-m package" or >>"-m package.module" are suitably well thought out to see it in b1. >> >>If a single compelling way of making it work can be seen in the >>next week, _maybe_ we could sneak it into b2, but I'm really not >>hopeful. > > > Okay, based on the discussions so far, I'm going to have to agree. > While I do think it's a very useful feature, it's more important to do > it right than to do it right now. Which means it should probably go > through the full PEP treatment. > +1 from me. -Brett From raymond.hettinger at verizon.net Sat Oct 16 00:25:47 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat Oct 16 00:28:18 2004 Subject: [Python-Dev] Wither docs for the subprocess module? Message-ID: <000001c4b305$ebb60700$e841fea9@oemcomputer> Is anyone working on documenting the module for the library reference? Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041015/76420b25/attachment.htm From fredrik at pythonware.com Sat Oct 16 00:44:50 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 16 00:42:44 2004 Subject: [Python-Dev] Re: Wither docs for the subprocess module? References: <000001c4b305$ebb60700$e841fea9@oemcomputer> Message-ID: <ckpjp0$kfb$1@sea.gmane.org> Raymond Hettinger wrote: > Is anyone working on documenting the module for the library reference? it's extensively documented in the PEP, but nobody replied to http://article.gmane.org/gmane.comp.python.devel/64327 so I guess I have to learn latex this weekend... </F> From ncoghlan at iinet.net.au Sat Oct 16 00:49:34 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Oct 16 00:48:28 2004 Subject: [Python-Dev] Supporting packages on the command line In-Reply-To: <41701996.7080905@ocf.berkeley.edu> References: <r01050400-1026-E8A4A8281DE711D988BE003065D5E7E4@[10.0.0.23]> <416EE3CC.1030003@iinet.net.au> <1097788593.8756.85.camel@geddy.wooz.org> <416F53D5.7090606@interlink.com.au> <1097841485.7804.31.camel@geddy.wooz.org> <41701996.7080905@ocf.berkeley.edu> Message-ID: <417053FE.2060404@iinet.net.au> Brett C. wrote: > Barry Warsaw wrote: >> Okay, based on the discussions so far, I'm going to have to agree. >> While I do think it's a very useful feature, it's more important to do >> it right than to do it right now. Which means it should probably go >> through the full PEP treatment. >> > > +1 from me. Works for me, too. I don't think the PEP has to be particularly *complicated* - it just needs to nail down specific semantics so everyone is talking about the same thing. And it should point out that this has nothing to do with changing the idiom for making modules that also work as scripts. If people want to do that, they can write their own PEP ;) I'll see if I can come up with a first draft this weekend. Cheers, Nick. From Scott.Daniels at Acm.Org Fri Oct 15 14:55:51 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat Oct 16 05:15:04 2004 Subject: [Python-Dev] Patch wanted In-Reply-To: <001701c4b262$78984240$e841fea9@oemcomputer> References: <001701c4b262$78984240$e841fea9@oemcomputer> Message-ID: <416FC8D7.80801@Acm.Org> Raymond Hettinger wrote: >Here's a proposed patch: > >--- pyconfig.h 23 Sep 2004 19:11:32 -0000 1.30 >+++ pyconfig.h 15 Oct 2004 02:44:27 -0000 >@@ -273,10 +273,12 @@ > > /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of >1200. > If some compiler does not provide them, modify the #if >appropriately. */ >-#if _MSC_VER != 1200 >+#if defined(_MSC_VER) >+#if _MSC_VER > 1200 > #define HAVE_UINTPTR_T 1 > #define HAVE_INTPTR_T 1 >-#endif >+#endif /* _MSC_VER > 1200 */ >+#endif /* _MSC_VER */ > > #endif > >>Ray Hettinger's fix 1.29 to PC/pyconfig.h >> > . . . > >>His test presumes >> > . . . > >Please watch the personalisms. I'm not the one who broke the build in >the first place. Ideally, someone who knows all about the various >compilers can propose a generic fix so that a lot of the ifdeffery in >this file can be taken out. For someone trying to repair a broken >build, they can often only test in one environment. > I'm sorry. I had no intention to make it sound personal, but I do see (now) how it does. I was trying to describe why this patch, that I had submitted quite some time ago, was important, and how I was not simply inventing a problem that nobody else did or will encounter. I understand you fixed something that was broken (probably on a cramped schedule), and I was trying to explain that the fix, while successful for VC 6.0 and 7.1, was likely to be the wrong test. Please accept my apologies, as I hold your efforts in great esteem. I was hoping that by hunting down in the CVS log to where the change went in, I could figure out where the last change in that area happened. I hoped that that person, at least, might be able to review the patch. I tried to say everything I knew about why this patch might work, not to attack the previous patch, but to explain why more change was needed. I did my best to determine what the core condition was and propose a change there. I don't, however, have the wherewithal (in particular the compilers) to test whether my idea of what should work will in fact work. I can only test on GCC 2.95 and 3.2.2 and VC 6.0. I don't have other GCC versions or the Intel compilers (which, in a more perfect world, should be tested). >Raymond > >Side note: It looks like I may be the only one testing/maintaining the >MSVC++6.0 build. It got broken again this week, so I have to fix it up >tonight (looks like a minor repair though). > -Scott David Daniels Scott.Daniels@Acm.Org From bob at redivi.com Sat Oct 16 05:37:43 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Oct 16 05:38:17 2004 Subject: [Python-Dev] Reducing core dependencies for Python on OS X / distutils nit Message-ID: <BD32B936-1F24-11D9-B97B-000A95686CD8@redivi.com> Remove CoreServices / CoreFoundation dependencies in core http://python.org/sf/1035255 It came to my attention a couple weeks ago that Python unnecessarily links to a few frameworks on OS X, so I generated a patch that removed these dependencies. I would hope that this is uncontroversial. Presumably Jack has been really busy lately so I understand that this patch didn't make it into b1, but I would like to get it rolled into CVS by *somebody* after CVS is reopened so the final of Python 2.4 doesn't link to a bunch of frameworks for no reason, which potentially (not benchmarked) increases memory usage and startup time because the resultant Python.framework and python interpreter no longer have any explicit dependencies at all (just libSystem, which is linked in by default). -- distutils.util.get_platform() should include sys.version[:3] http://python.org/sf/1035703 On a side note, when I was playing with Python 2.4, I noticed that distutils doesn't create build directories that have the major Python version encoded in them, only the operating system information. This doesn't really make much sense, because extensions and bytecode aren't typically compatible between major versions! Nobody commented on this, but I'm willing to write a patch to create build directories with sys.version[:3] in them if it will be accepted... -bob From fredrik at pythonware.com Sat Oct 16 17:11:00 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 16 17:08:53 2004 Subject: [Python-Dev] Re: Wither docs for the subprocess module? References: <000001c4b305$ebb60700$e841fea9@oemcomputer> <ckpjp0$kfb$1@sea.gmane.org> Message-ID: <ckrdi1$cvc$1@sea.gmane.org> >> Is anyone working on documenting the module for the library reference? > > it's extensively documented in the PEP, but nobody replied to > > http://article.gmane.org/gmane.comp.python.devel/64327 > > so I guess I have to learn latex this weekend... here's a first cut: http://www.python.org/sf/1048341 if someone wants to fix this up a little and check it in, be my guest. other- wise, I'll check it in as soon as Anthony unfreezes the trunk (is it still Friday in Australia, btw?) </F> From fredrik at pythonware.com Sat Oct 16 17:23:05 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat Oct 16 17:21:02 2004 Subject: [Python-Dev] Re: Reducing core dependencies for Python on OS X /distutils nit References: <BD32B936-1F24-11D9-B97B-000A95686CD8@redivi.com> Message-ID: <ckre8m$egd$1@sea.gmane.org> Bob Ippolito wrote: > distutils.util.get_platform() should include sys.version[:3] > http://python.org/sf/1035703 > > On a side note, when I was playing with Python 2.4, I noticed that distutils doesn't create build > directories that have the major Python version encoded in them, only the operating system > information. This doesn't really make much sense, because extensions and bytecode aren't > typically compatible between major versions! Nobody commented on this, but I'm willing to write a > patch to create build directories with sys.version[:3] in them if it will be accepted... are you sure about this? the build directories sure have version numbers in them on Unix and Windows: $ python setup.py build building 'sgmlop' extension gcc -g -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include/python2.1 -c sgmlop.c -o build/temp.linux-i686-2.1/sgmlop.o creating build/lib.linux-i686-2.1 gcc -shared build/temp.linux-i686-2.1/sgmlop.o -o build/lib.linux-i686-2.1/sgmlop.so $ python2.3 setup.py build building 'sgmlop' extension creating build/temp.linux-i686-2.3 gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -I/usr/local/include/python2.3 -c sgmlop.c -o build/temp.linux-i686-2.3/sgmlop.o creating build/lib.linux-i686-2.3 gcc -shared build/temp.linux-i686-2.3/sgmlop.o -o build/lib.linux-i686-2.3/sgmlop.so $ python1.5 setup.py build building 'sgmlop' extension creating build/temp.linux-i686-1.5 gcc -g -O2 -fPIC -I/usr/include/python1.5 -c sgmlop.c -o build/temp.linux-i686-1.5/sgmlop.o creating build/lib.linux-i686-1.5 gcc -shared build/temp.linux-i686-1.5/sgmlop.o -o build/lib.linux-i686-1.5/sgmlop.so > python setup.py build building 'sgmlop' extension creating build\temp.win32-2.3 creating build\temp.win32-2.3\Release C:\VisualStudio\VC98\BIN\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -IC:\python2 3\include -IC:\python23\PC /Tcsgmlop.c /Fobuild\temp.win32-2.3\Release\sgmlop.obj sgmlop.c creating build\lib.win32-2.3 C:\VisualStudio\VC98\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\pytho n23\libs /LIBPATH:C:\python23\PCBuild /EXPORT:initsgmlop build\temp.win32-2.3\Re lease\sgmlop.obj /OUT:build\lib.win32-2.3\sgmlop.pyd /IMPLIB:build\temp.win32-2. 3\Release\sgmlop.lib (and so on) </F> From bob at redivi.com Sat Oct 16 17:58:36 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Oct 16 17:59:16 2004 Subject: [Python-Dev] Re: Reducing core dependencies for Python on OS X /distutils nit In-Reply-To: <ckre8m$egd$1@sea.gmane.org> References: <BD32B936-1F24-11D9-B97B-000A95686CD8@redivi.com> <ckre8m$egd$1@sea.gmane.org> Message-ID: <3D255423-1F8C-11D9-B97B-000A95686CD8@redivi.com> On Oct 16, 2004, at 11:23 AM, Fredrik Lundh wrote: > Bob Ippolito wrote: > >> distutils.util.get_platform() should include sys.version[:3] >> http://python.org/sf/1035703 >> >> On a side note, when I was playing with Python 2.4, I noticed that >> distutils doesn't create build >> directories that have the major Python version encoded in them, only >> the operating system >> information. This doesn't really make much sense, because extensions >> and bytecode aren't >> typically compatible between major versions! Nobody commented on >> this, but I'm willing to write a >> patch to create build directories with sys.version[:3] in them if it >> will be accepted... > > are you sure about this? the build directories sure have version > numbers in > them on Unix and Windows: I think you're right, I must've been mistaken by the fact that pure python source files end up in a lib dir not a build dir (I thought it was reusing pyc files, but those are generated at the installation location). My bad. -bob From anthony at python.org Sun Oct 17 16:26:24 2004 From: anthony at python.org (Anthony Baxter) Date: Sun Oct 17 16:27:12 2004 Subject: [Python-Dev] RELEASED Python 2.4, beta 1 Message-ID: <41728110.9020605@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the first beta of Python 2.4. Python 2.4b1 is an beta release. We'd greatly appreciate it if you could download it, kick the tires and let us know of any problems you find, but it is probably not suitable for production usage. ~ http://www.python.org/2.4 Notable changes in beta1 include a new subprocess module to provide a more convenient interface to spawning processes and a new command line argument -m modulename to run a module from the library. See either the highlights, the What's New in Python 2.4, or the detailed NEWS file -- all available from the Python 2.4 webpage. This is the first beta release of Python 2.4. Python 2.4 is now in the beta part of the release cycle - there should be few or no new features, merely bug fixes. There will be at least one more beta in a couple of weeks, followed by a release candidate and then the final release. Please log any problems you have with this release in the SourceForge bug tracker (noting that you're using 2.4b1): ~ http://sourceforge.net/bugs/?group_id=5470 Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFBcoEMDt3F8mpFyBYRAhJhAKCCclb5ytuljQIGvIz9dIV2LmL0KQCgl5DX 357fPses//NSjyK+7B10YPg= =lzWC -----END PGP SIGNATURE----- From anthony at interlink.com.au Sun Oct 17 16:40:39 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun Oct 17 17:16:43 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) Message-ID: <41728467.2090504@interlink.com.au> The first beta is out, so the trunk is unfrozen, and available for checkins. Now that we're in beta, we shouldn't see any new features or behaviour changing fixes going into the trunk, unless it's been seen and agreed to on python-dev. I currently plan for a second beta in either 2 or 3 weeks, and then, assuming all goes well, a release candidate a couple of weeks after that. Anthony (apologies in the pause in getting the release email out - an unexpectedly hectic weekend popped up without notice) From mcherm at mcherm.com Mon Oct 18 00:10:44 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Oct 18 00:10:22 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry Message-ID: <1098051044.4172ede4eab83@mcherm.com> A few weeks ago, the suggestion was made on Python-Dev that it might be time to consider replacing the ConfigParser module and that we should hold a "shootout" (ie ask for implementations and see what we get). Since then I've been playing around with this... not the parsing part (which so far I have completely ignored) but the programmer interface. There needs to be a well-thought-out data model for the information stored, and the user interface needs to be very easy to use, yet not so "magical" that it becomes difficult to understand. I have put together what I think is probably my best proposal. It is based on a superset of ini config files and java .property files. There is a convenient access mechanism ("config.my_app.some_value") as well as more general approaches ("config.values['my_app.serviceByPort.80']"). I have tried to consider issues like unicode (I permit fairly lenient mixing of unicode and str), and unit testing ("... call config.clear_all() in the tearDown() method of any unittests that use the config module..."). I have even considered carefully what to leave OUT (converting to non-string data types, interpolating values, things like that). I think that I am now at the point where I could really use some input from others. So I'd like to invite people to review my design and send me your suggestions. I'm not expecting this as a *useful* module yet (it doesn't yet parse files!), but it seemed like a good stage at which to ask for feedback. I'm attaching two files, config.py and configTest.py, and they are also available from these urls: http://www.mcherm.com/publish/2004-10-17/config.py http://www.mcherm.com/publish/2004-10-17/configTest.py Thanks in advance for reviewing this. -- Michael Chermside From mcherm at mcherm.com Mon Oct 18 00:38:11 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Oct 18 00:37:48 2004 Subject: [Python-Dev] RE: ConfigParser shootout, preliminary entry Message-ID: <1098052691.4172f453e0a4f@mcherm.com> My apologies, my previous email was intended to go to python-list, not python-dev. I didn't intend to bother this list with it until it was at a point where I thought it was a contender for inclusion. Sometimes I really wish there was an "Unsend" option for email. -- Michael Chermside From bac at OCF.Berkeley.EDU Mon Oct 18 00:53:29 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Oct 18 00:53:38 2004 Subject: [Python-Dev] Adding support for %U and %W for strptime in b2 Message-ID: <4172F7E9.1090208@ocf.berkeley.edu> http://www.python.org/sf/1045381 is a bug report about how strptime does nothing with %U and %W directives (week of the year with differences in whether Sunday or Monday starts the week). I just finished getting the algorithm to calculate the date given the year, either of %U or %W and the day of the week. Now the question is whether this is a bug fix or a feature implementation. If it is a bug fix I can check it right in. The only reason it wasn't in since the beginning is I didn't think of the day of the week being all you needed to figure out the date with %W or %U. If it is considered a feature, though, I will just add a note that %U and %W consume values but do nothing with them. So, what do you all think? -Brett From trentm at ActiveState.com Mon Oct 18 01:35:19 2004 From: trentm at ActiveState.com (Trent Mick) Date: Mon Oct 18 01:35:33 2004 Subject: [Python-Dev] ConfigParser patches In-Reply-To: <4164846B.3040205@lazytwinacres.net> References: <20041003034331.46FB91E400A@bag.python.org> <4164846B.3040205@lazytwinacres.net> Message-ID: <417301B7.7000204@activestate.com> > Modify your PATHEXT environment variable: > > PATHEXT=.PY;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH > > ... > You can put .pyc in your path too, but then you'll get the compiled version > even if the uncompiled version is newer. But you can always compile them. Or you can put .pyc _after_ .py on your PATHEXT and then the .py will always get picked up first for Python to decide whether the .pyc can be used. If there is only a .pyc around then it will get picked up. PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.tcl;.py;.pyw;.pyc;.pyo;.pys Trent -- Trent Mick trentm@activestate.com From gvanrossum at gmail.com Mon Oct 18 02:20:43 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 18 02:34:56 2004 Subject: [Python-Dev] Adding support for %U and %W for strptime in b2 In-Reply-To: <4172F7E9.1090208@ocf.berkeley.edu> References: <4172F7E9.1090208@ocf.berkeley.edu> Message-ID: <ca471dc20410171720278dae2e@mail.gmail.com> Bugfix. On Sun, 17 Oct 2004 15:53:29 -0700, Brett C. <bac@ocf.berkeley.edu> wrote: > http://www.python.org/sf/1045381 is a bug report about how strptime does > nothing with %U and %W directives (week of the year with differences in whether > Sunday or Monday starts the week). I just finished getting the algorithm to > calculate the date given the year, either of %U or %W and the day of the week. > > Now the question is whether this is a bug fix or a feature implementation. If > it is a bug fix I can check it right in. The only reason it wasn't in since > the beginning is I didn't think of the day of the week being all you needed to > figure out the date with %W or %U. If it is considered a feature, though, I > will just add a note that %U and %W consume values but do nothing with them. > > So, what do you all think? > > -Brett > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From raynorj at mn.rr.com Mon Oct 18 03:26:55 2004 From: raynorj at mn.rr.com (J Raynor) Date: Mon Oct 18 03:09:34 2004 Subject: [Python-Dev] alternate pty module available Message-ID: <41731BDF.5040607@mn.rr.com> See the following threads for some background: http://mail.python.org/pipermail/python-dev/2004-September/049085.html http://mail.python.org/pipermail/python-dev/2004-September/049092.html I've created a python module called altpty that uses code from Openssh to implement openpty() and forkpty() functions. They should work across more platforms than the functions from the standard pty module. The module can be downloaded from: http://home.mn.rr.com/raynorj/altpty.tar.gz Installation instructions are in the INSTALL file. I believe you'll need python 2.2 or better to build the module. There's still some cleanup work that could be done, but the module works, and I wanted to make it available and see if there was any real interest in it before I put too much effort into getting everything just so. Send comments or bug reports to raynorj AT mn.rr.com. From anthony at interlink.com.au Mon Oct 18 03:37:32 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Oct 18 03:38:16 2004 Subject: [Python-Dev] Adding support for %U and %W for strptime in b2 In-Reply-To: <4172F7E9.1090208@ocf.berkeley.edu> References: <4172F7E9.1090208@ocf.berkeley.edu> Message-ID: <41731E5C.4060206@interlink.com.au> Brett C. wrote: > Now the question is whether this is a bug fix or a feature > implementation. If it is a bug fix I can check it right in. The only > reason it wasn't in since the beginning is I didn't think of the day of > the week being all you needed to figure out the date with %W or %U. If > it is considered a feature, though, I will just add a note that %U and > %W consume values but do nothing with them. This is a bugfix - %U and %W were documented as working, and there's absolutely no conceivable way that anyone could be _relying_ on the current behaviour. -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From t-meyer at ihug.co.nz Mon Oct 18 05:48:23 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Mon Oct 18 05:48:38 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib _strptime.py, 1.35, 1.36 In-Reply-To: <ECBA357DDED63B4995F5C1F5CBE5B1E80126E51E@its-xchg4.massey.ac.nz> Message-ID: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> > Index: _strptime.py [...] > + # For some reason when Dec 31 falls on a Monday the > week of the year is > + # off by a week; verified on both OS X and Solaris. > + elif weekday == 0 and week_of_year_start == 6 and > week_of_year >= 52: > + week_of_year += 1 Is this right? At the moment, if I ask for Monday in the 52nd week of 2008, I get the 5th of January 2009: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) For reference, Monday in the previous week is the 22nd of December: >>> time.strptime("2008 51 1", "%Y %U %w") (2008, 12, 22, 0, 0, 0, 0, 357, -1) Wouldn't December 29 2008 be the right answer? This is the result without that if: >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) This is with WinXP SP1. =Tony Meyer From carribeiro at gmail.com Mon Oct 18 15:28:39 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Mon Oct 18 15:28:44 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <1098051044.4172ede4eab83@mcherm.com> References: <1098051044.4172ede4eab83@mcherm.com> Message-ID: <864d370904101806282c55ef39@mail.gmail.com> On Sun, 17 Oct 2004 15:10:44 -0700, Michael Chermside <mcherm@mcherm.com> wrote: > A few weeks ago, the suggestion was made on Python-Dev that it might be time > to consider replacing the ConfigParser module and that we should hold a > "shootout" (ie ask for implementations and see what we get). Now that we are in 'shootout mode', let me plug my own solution :-) I've written a generic data strucuture templating package, and one of the samples is a ini-file reader. The syntax is elegant and simple. It allows one to define the expected INI file structure, using a class definition. The class is then able to read itself from the INI file. I've not yet implemented the write code, but it should be a snap. Check this sample: class WebServerIni(IniFile): class server(IniSection): socketPort = TypedAttribute(8080) threadPool = TypedAttribute(10) class staticContent(IniSection): bitmaps = TypedAttribute('c:/work/bitmaps') class session(IniSection): storageType = TypedAttribute('ram') Attributes may be 'generic' (untyped) or 'typed', and are stored in the order they are declared in the class statement. The value provided in the definition above is the 'default value'. Simple attributes may also be provided, but won't be type checked or kept in the original order. To read the ini file, just do this: inifile = WebServerIni() inifile.load(<optional-file-name>) If not provided, it will read the file named '<classname>.ini'. It's a pretty simple and clean interface; it's quite easy to provide default arguments; it's more readable than a dict-based configuration; and it works fine, at least for me :-). -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From jcarlson at uci.edu Mon Oct 18 19:46:47 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Oct 18 19:54:46 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <864d370904101806282c55ef39@mail.gmail.com> References: <1098051044.4172ede4eab83@mcherm.com> <864d370904101806282c55ef39@mail.gmail.com> Message-ID: <20041018104036.0A2B.JCARLSON@uci.edu> > Attributes may be 'generic' (untyped) or 'typed', and are stored in > the order they are declared in the class statement. The value provided > in the definition above is the 'default value'. Simple attributes may > also be provided, but won't be type checked or kept in the original > order. To read the ini file, just do this: I think the syntax looks good, but as per a thread in python-list, you cannot discover the order of class variables by any solution (metaclass or otherwise), due to the fact that they become part of the class dictionary; which is arbitrarily unordered. If ordering is important to a user, one could have an optional __order__ attribute that gives the list of items in-order. > class WebServerIni(IniFile): > class server(IniSection): > socketPort = TypedAttribute(8080) > threadPool = TypedAttribute(10) > class staticContent(IniSection): > bitmaps = TypedAttribute('c:/work/bitmaps') > class session(IniSection): > storageType = TypedAttribute('ram') One nice thing about your solution is that one could pull out docstrings to provide per-section documentation in the INI file, though per-item docstrings (like WebServerIni.session.storageType) would be a bit more difficult. - Josiah From bob at redivi.com Mon Oct 18 19:57:31 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon Oct 18 19:58:06 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041018104036.0A2B.JCARLSON@uci.edu> References: <1098051044.4172ede4eab83@mcherm.com> <864d370904101806282c55ef39@mail.gmail.com> <20041018104036.0A2B.JCARLSON@uci.edu> Message-ID: <2EE12B0A-212F-11D9-9236-000A95BA5446@redivi.com> On Oct 18, 2004, at 13:46, Josiah Carlson wrote: >> Attributes may be 'generic' (untyped) or 'typed', and are stored in >> the order they are declared in the class statement. The value provided >> in the definition above is the 'default value'. Simple attributes may >> also be provided, but won't be type checked or kept in the original >> order. To read the ini file, just do this: > > I think the syntax looks good, but as per a thread in python-list, you > cannot discover the order of class variables by any solution (metaclass > or otherwise), due to the fact that they become part of the class > dictionary; which is arbitrarily unordered. > > If ordering is important to a user, one could have an optional > __order__ > attribute that gives the list of items in-order. That's not quite true. TypedAttribute instances and iniSection's __new__ could have serial numbers. -bob From bac at OCF.Berkeley.EDU Mon Oct 18 20:25:27 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Oct 18 20:25:55 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib _strptime.py, 1.35, 1.36 In-Reply-To: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> References: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> Message-ID: <41740A97.6090201@ocf.berkeley.edu> Tony Meyer wrote: >>Index: _strptime.py > > [...] > >>+ # For some reason when Dec 31 falls on a Monday the >>week of the year is >>+ # off by a week; verified on both OS X and Solaris. >>+ elif weekday == 0 and week_of_year_start == 6 and >>week_of_year >= 52: >>+ week_of_year += 1 > > > Is this right? At the moment, if I ask for Monday in the 52nd week of 2008, > I get the 5th of January 2009: > > >>>>time.strptime("2008 52 1", "%Y %U %w") > > (2009, 1, 5, 0, 0, 0, 0, 371, -1) > > For reference, Monday in the previous week is the 22nd of December: > > >>>>time.strptime("2008 51 1", "%Y %U %w") > > (2008, 12, 22, 0, 0, 0, 0, 357, -1) > > Wouldn't December 29 2008 be the right answer? This is the result without > that if: > > >>>>time.strptime("2008 52 1", "%Y %U %w") > > (2008, 12, 29, 0, 0, 0, 0, 364, -1) > Can you file a bug report about this? And the test case I was using that triggered the need for that is 1906-12-31 which is a Monday but changes what week it is based on whether you use U or W. which makes no sense since both U and W should consider it the same week. Had the same result for 1917-12-31. -Brett From ndbecker2 at verizon.net Mon Oct 18 21:22:59 2004 From: ndbecker2 at verizon.net (Neal D. Becker) Date: Mon Oct 18 21:23:05 2004 Subject: [Python-Dev] import w/options Message-ID: <cl156k$1n1$1@sea.gmane.org> I have an application where it would be useful to set RTLD_GLOBAL on dlopen when import is used. Currently, python doesn't provide this option. Why not allow an optional argument to import that would allow this? From jcarlson at uci.edu Mon Oct 18 21:20:46 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Oct 18 21:28:49 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <2EE12B0A-212F-11D9-9236-000A95BA5446@redivi.com> References: <20041018104036.0A2B.JCARLSON@uci.edu> <2EE12B0A-212F-11D9-9236-000A95BA5446@redivi.com> Message-ID: <20041018120633.0A3D.JCARLSON@uci.edu> > > I think the syntax looks good, but as per a thread in python-list, you > > cannot discover the order of class variables by any solution (metaclass > > or otherwise), due to the fact that they become part of the class > > dictionary; which is arbitrarily unordered. > > > > If ordering is important to a user, one could have an optional > > __order__ > > attribute that gives the list of items in-order. > > That's not quite true. TypedAttribute instances and iniSection's > __new__ could have serial numbers. I'm not saying that they can't be numbered, I'm saying that one cannot discover the ordering of assignment of attr1 and attr2 in the following: class foo: attr1 = value1 attr2 = value2 If there is a mechanism for discovering the original ordering of those assignments, there are a group of users in c.l.py who would like to know, and Carlos' seemingly non-existant implementation could also use it. Please advise, - Josiah From bob at redivi.com Mon Oct 18 21:52:03 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon Oct 18 21:52:38 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041018120633.0A3D.JCARLSON@uci.edu> References: <20041018104036.0A2B.JCARLSON@uci.edu> <2EE12B0A-212F-11D9-9236-000A95BA5446@redivi.com> <20041018120633.0A3D.JCARLSON@uci.edu> Message-ID: <2ED206F0-213F-11D9-9236-000A95BA5446@redivi.com> On Oct 18, 2004, at 15:20, Josiah Carlson wrote: >>> I think the syntax looks good, but as per a thread in python-list, >>> you >>> cannot discover the order of class variables by any solution >>> (metaclass >>> or otherwise), due to the fact that they become part of the class >>> dictionary; which is arbitrarily unordered. >>> >>> If ordering is important to a user, one could have an optional >>> __order__ >>> attribute that gives the list of items in-order. >> >> That's not quite true. TypedAttribute instances and iniSection's >> __new__ could have serial numbers. > > > I'm not saying that they can't be numbered, I'm saying that one cannot > discover the ordering of assignment of attr1 and attr2 in the > following: > > class foo: > attr1 = value1 > attr2 = value2 > > If there is a mechanism for discovering the original ordering of those > assignments, there are a group of users in c.l.py who would like to > know, > and Carlos' seemingly non-existant implementation could also use it. That is true, but you do know that the expression value1 is evaluated before the expression value2, so it is possible to sort later for clever enough choices of value1 and value2. Since his proposed syntax invokes something for each attribute, then this trick can certainly be used.. here's a small demonstration: from itertools import count class Value(object): def __init__(self, value, serial=count()): self.serial = serial.next() self.value = value class Container(object): class __metaclass__(type): def __new__(cls, name, bases, dct): sorted = filter(lambda (k,v): isinstance(v, Value), dct.iteritems()) sorted.sort(lambda (ka,va),(kb, vb): cmp(va.serial, vb.serial)) dct['sorted'] = sorted return type.__new__(cls, name, bases, dct) class MyContainer(Container): p = Value(2) y = Value(0) t = Value(-1) h = Value(20) o = Value('x') n = Value('z') print ''.join([k for k,v in MyContainer.sorted]) python -bob From python-erik at rixmail.se Mon Oct 18 22:09:35 2004 From: python-erik at rixmail.se (=?ISO-8859-1?Q?Erik_Anders=E9n?=) Date: Mon Oct 18 22:09:43 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib _strptime.py, 1.35, 1.36 In-Reply-To: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> References: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> Message-ID: <417422FF.4010107@rixmail.se> Tony Meyer wrote: >>Index: _strptime.py >> >> >[...] > > >>+ # For some reason when Dec 31 falls on a Monday the >>week of the year is >>+ # off by a week; verified on both OS X and Solaris. >>+ elif weekday == 0 and week_of_year_start == 6 and >>week_of_year >= 52: >>+ week_of_year += 1 >> >> > >Is this right? At the moment, if I ask for Monday in the 52nd week of 2008, >I get the 5th of January 2009: > > > >>>>time.strptime("2008 52 1", "%Y %U %w") >>>> >>>> >(2009, 1, 5, 0, 0, 0, 0, 371, -1) > >For reference, Monday in the previous week is the 22nd of December: > > > >>>>time.strptime("2008 51 1", "%Y %U %w") >>>> >>>> >(2008, 12, 22, 0, 0, 0, 0, 357, -1) > >Wouldn't December 29 2008 be the right answer? This is the result without >that if: > > > Yes, according to the Python documentation. You should however be aware that this is different from the standard (ISO 8601:2000). According to that standard, Monday, week 52, 2008 is December 22. Implementing a function that gives result different from what people actually use is not very good. It is better to leave it unimplemented. = Erik Andersen >>>>time.strptime("2008 52 1", "%Y %U %w") >>>> >>>> >(2008, 12, 29, 0, 0, 0, 0, 364, -1) > >This is with WinXP SP1. > >=Tony Meyer > >_______________________________________________ >Python-Dev mailing list >Python-Dev@python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-erik%40rixmail.se > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041018/109479e6/attachment-0001.htm From jcarlson at uci.edu Mon Oct 18 22:20:51 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Oct 18 22:28:51 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <2ED206F0-213F-11D9-9236-000A95BA5446@redivi.com> References: <20041018120633.0A3D.JCARLSON@uci.edu> <2ED206F0-213F-11D9-9236-000A95BA5446@redivi.com> Message-ID: <20041018131744.0A43.JCARLSON@uci.edu> > class MyContainer(Container): > p = Value(2) > y = Value(0) > t = Value(-1) > h = Value(20) > o = Value('x') > n = Value('z') I missed that. I am sure that Carlos has even made TypedAttr into factories that produce properties. Very nice; I'm sorry I didn't see it earlier. - Josiah From mwh at python.net Mon Oct 18 22:45:40 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 18 22:45:42 2004 Subject: [Python-Dev] import w/options In-Reply-To: <cl156k$1n1$1@sea.gmane.org> (Neal D. Becker's message of "Mon, 18 Oct 2004 15:22:59 -0400") References: <cl156k$1n1$1@sea.gmane.org> Message-ID: <2macujdabv.fsf@starship.python.net> "Neal D. Becker" <ndbecker2@verizon.net> writes: > I have an application where it would be useful to set RTLD_GLOBAL on dlopen > when import is used. Currently, python doesn't provide this option. Why > not allow an optional argument to import that would allow this? I take it you'd be interested to learn about sys.setdlopenflags? Cheers, mwh -- I'm about to search Google for contract assassins to go to Iomega and HP's programming groups and kill everyone there with some kind of electrically charged rusty barbed thing. -- http://bofhcam.org/journal/journal.html, 2002-01-08 From bac at OCF.Berkeley.EDU Mon Oct 18 23:17:18 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Oct 18 23:17:32 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib _strptime.py, 1.35, 1.36 In-Reply-To: <417422FF.4010107@rixmail.se> References: <ECBA357DDED63B4995F5C1F5CBE5B1E86C3783@its-xchg4.massey.ac.nz> <417422FF.4010107@rixmail.se> Message-ID: <417432DE.2050306@ocf.berkeley.edu> Erik Anders?n wrote: > Tony Meyer wrote: >>>>>time.strptime("2008 51 1", "%Y %U %w") >>>>> >>>>> >>(2008, 12, 22, 0, 0, 0, 0, 357, -1) >> >>Wouldn't December 29 2008 be the right answer? This is the result without >>that if: >> >> >> > Yes, according to the Python documentation. You should however be aware > that this is different from the standard (ISO 8601:2000). > According to that standard, Monday, week 52, 2008 is December 22. > Implementing a function that gives result different from what people > actually use is not very good. It is better to leave it unimplemented. The function has a bug and is not trying to implement something other than what people are expecting. Having a Julian date that is out of range is my fault in the algorithm and in no way would happen if you passed it in directly. And any more discussion on this bug or this algorithm should go on the bug report (http://www.python.org/sf/1045381) and not on python-dev. -Brett From exarkun at divmod.com Mon Oct 18 23:39:47 2004 From: exarkun at divmod.com (exarkun@divmod.com) Date: Mon Oct 18 23:39:51 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <2ED206F0-213F-11D9-9236-000A95BA5446@redivi.com> Message-ID: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> On Mon, 18 Oct 2004 15:52:03 -0400, Bob Ippolito <bob@redivi.com> wrote: >On Oct 18, 2004, at 15:20, Josiah Carlson wrote: > > [snip] > > > > I'm not saying that they can't be numbered, I'm saying that one cannot > > discover the ordering of assignment of attr1 and attr2 in the > > following: > > > > class foo: > > attr1 = value1 > > attr2 = value2 > > > > If there is a mechanism for discovering the original ordering of those > > assignments, there are a group of users in c.l.py who would like to > > know, > > and Carlos' seemingly non-existant implementation could also use it. > > That is true, but you do know that the expression value1 is evaluated > before the expression value2, so it is possible to sort later for > clever enough choices of value1 and value2. Since his proposed syntax > invokes something for each attribute, then this trick can certainly be > used.. here's a small demonstration: > > from itertools import count > > class Value(object): > def __init__(self, value, serial=count()): > self.serial = serial.next() > self.value = value > > class Container(object): > class __metaclass__(type): > def __new__(cls, name, bases, dct): > sorted = filter(lambda (k,v): isinstance(v, Value), > dct.iteritems()) > sorted.sort(lambda (ka,va),(kb, vb): cmp(va.serial, > vb.serial)) > dct['sorted'] = sorted > return type.__new__(cls, name, bases, dct) > > class MyContainer(Container): > p = Value(2) > y = Value(0) > t = Value(-1) > h = Value(20) > o = Value('x') > n = Value('z') This breaks down for the perfectly reasonable case of: hostname = Value("foo") class First(Container): port = Value(64) hostname = hostname username = Value("zoop") class Second(Container): username = Value("pooz") hostname = hostname port = Value(24) ie, it breaks down as soon as you try to re-use anything, which is quite surprising to the unsuspecting user, and pretty unfortunate even once you do understand why. Jp From martin at v.loewis.de Mon Oct 18 23:51:13 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Oct 18 23:51:14 2004 Subject: [Python-Dev] alternate pty module available In-Reply-To: <41731BDF.5040607@mn.rr.com> References: <41731BDF.5040607@mn.rr.com> Message-ID: <41743AD1.1040906@v.loewis.de> J Raynor wrote: > I've created a python module called altpty that uses code from Openssh > to implement openpty() and forkpty() functions. They should work across > more platforms than the functions from the standard pty module. Well done! You might want to announce that to c.l.p.a, and add it to PyPI, for wider distribution. Regards, Martin From bob at redivi.com Tue Oct 19 00:06:17 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Oct 19 00:06:52 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> Message-ID: <EF96B657-2151-11D9-9236-000A95BA5446@redivi.com> On Oct 18, 2004, at 17:39, <exarkun@divmod.com> wrote: > On Mon, 18 Oct 2004 15:52:03 -0400, Bob Ippolito <bob@redivi.com> > wrote: >> On Oct 18, 2004, at 15:20, Josiah Carlson wrote: >> >> [snip] >>> >>> I'm not saying that they can't be numbered, I'm saying that one >>> cannot >>> discover the ordering of assignment of attr1 and attr2 in the >>> following: >>> >>> class foo: >>> attr1 = value1 >>> attr2 = value2 >>> >>> If there is a mechanism for discovering the original ordering of >>> those >>> assignments, there are a group of users in c.l.py who would like to >>> know, >>> and Carlos' seemingly non-existant implementation could also use it. >> >> That is true, but you do know that the expression value1 is evaluated >> before the expression value2, so it is possible to sort later for >> clever enough choices of value1 and value2. Since his proposed syntax >> invokes something for each attribute, then this trick can certainly be >> used.. here's a small demonstration: >> >> from itertools import count >> >> class Value(object): >> def __init__(self, value, serial=count()): >> self.serial = serial.next() >> self.value = value >> >> class Container(object): >> class __metaclass__(type): >> def __new__(cls, name, bases, dct): >> sorted = filter(lambda (k,v): isinstance(v, Value), >> dct.iteritems()) >> sorted.sort(lambda (ka,va),(kb, vb): cmp(va.serial, >> vb.serial)) >> dct['sorted'] = sorted >> return type.__new__(cls, name, bases, dct) >> >> class MyContainer(Container): >> p = Value(2) >> y = Value(0) >> t = Value(-1) >> h = Value(20) >> o = Value('x') >> n = Value('z') > > This breaks down for the perfectly reasonable case of: > > hostname = Value("foo") > > class First(Container): > port = Value(64) > hostname = hostname > username = Value("zoop") > > class Second(Container): > username = Value("pooz") > hostname = hostname > port = Value(24) > > ie, it breaks down as soon as you try to re-use anything, which is > quite surprising to the unsuspecting user, and pretty unfortunate even > once you do understand why. Yes, it breaks down in the general case if you do things like that. The answer is just to not do things like that. You can make Value instances raise an exception the second time they're attached to a class and tell the user to subclass Value instead if they want to provide some default values. -bob From jcarlson at uci.edu Tue Oct 19 00:01:23 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Oct 19 00:08:55 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> References: <2ED206F0-213F-11D9-9236-000A95BA5446@redivi.com> <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> Message-ID: <20041018145039.0A4E.JCARLSON@uci.edu> > This breaks down for the perfectly reasonable case of: > > hostname = Value("foo") > > class First(Container): > port = Value(64) > hostname = hostname > username = Value("zoop") > > class Second(Container): > username = Value("pooz") > hostname = hostname > port = Value(24) > > ie, it breaks down as soon as you try to re-use anything, which is > quite surprising to the unsuspecting user, and pretty unfortunate even > once you do understand why. So you use... hostname = "foo" class First(Container): port = Value(36) hostname = Value(hostname) username = Value("zoop") ... As long as the documentation describes the proper semantics if one cares about orderings (create new TypedAttrribute instance for every value), when anyone asks, a quick "read the documentation page" would be sufficient. One could also explain why the hoops are to be jumped through in the first place (dict non-ordering during class instantiation). - Josiah From ndbecker2 at verizon.net Tue Oct 19 00:43:14 2004 From: ndbecker2 at verizon.net (Neal Becker) Date: Tue Oct 19 00:43:20 2004 Subject: [Python-Dev] Re: import w/options References: <cl156k$1n1$1@sea.gmane.org> <2macujdabv.fsf@starship.python.net> Message-ID: <cl1gu2$245$1@sea.gmane.org> Michael Hudson wrote: > "Neal D. Becker" <ndbecker2@verizon.net> writes: > >> I have an application where it would be useful to set RTLD_GLOBAL on >> dlopen >> when import is used. Currently, python doesn't provide this option. Why >> not allow an optional argument to import that would allow this? > > I take it you'd be interested to learn about sys.setdlopenflags? > Could be. But I checked the current cvs python source, and what I see is dlopen is called with flags set to 0. What am I missing? From fdrake at acm.org Tue Oct 19 04:21:43 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue Oct 19 04:21:53 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> Message-ID: <200410182221.43632.fdrake@acm.org> On Monday 18 October 2004 05:39 pm, exarkun@divmod.com wrote: > This breaks down for the perfectly reasonable case of: ... > ie, it breaks down as soon as you try to re-use anything, which is quite > surprising to the unsuspecting user, and pretty unfortunate even once you > do understand why. This approach is used to advantage for Zope 3 schema. If you don't want to confuse the ordering, you don't re-use field objects (the things that go in schema). If you don't care about the ordering (because you're going to control ordering some other way), you don't need to worry about it. This is still Python; use what makes sense for your application, but know what you're doing. The "consenting adults" playground expects consenting adults to educate themselves. Good documentation comes in handy with frameworks such as these. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From jepler at unpythonic.net Tue Oct 19 04:40:38 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Oct 19 04:40:41 2004 Subject: [Python-Dev] Re: import w/options In-Reply-To: <cl1gu2$245$1@sea.gmane.org> References: <cl156k$1n1$1@sea.gmane.org> <2macujdabv.fsf@starship.python.net> <cl1gu2$245$1@sea.gmane.org> Message-ID: <20041019024038.GD26950@unpythonic.net> It works here: (python 2.3.3 on fedora core 2) $ ltrace -s 74 -e dlopen \ python -c 'import sys,dl; sys.setdlopenflags(dl.RTLD_LAZY); import _tkinter' dlopen("/usr/lib/python2.3/lib-dynload/dlmodule.so", 2) = 0x9ea7cf0 dlopen("/usr/lib/python2.3/lib-dynload/_tkinter.so", 1) = 0x9ea9c88 It's possible that some implementation of shared library loading don't obey sys.setdlopenflags(). In particular, it looks like dlopenflags will only be changed from 0 in dynload_shlib.c when PYCC_GCC is set. (line 111 in my recentish CVS version of that file) Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041018/23003b81/attachment-0001.pgp From ejones at uwaterloo.ca Tue Oct 19 06:00:46 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 06:01:08 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes Message-ID: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> I know that this has been discussed a bit in the past, but I was hoping that some Python gurus could shed some light on this issue, and maybe let me know if there are any plans for solving this problem. I know a hack that might work, but there must be a better way to solve this problem. The short version of the problem is that obmalloc.c never frees memory. This is a great strategy if the application runs for a short time then quits, or if it has fairly constant memory usage. However, applications with very dynamic memory needs and that run for a long time do not perform well because Python hangs on to the peak amount of memory required, even if that memory is only required for a tiny fraction of the run time. With my application, I have a python process which occupy 1 GB of RAM for ~20 hours, even though it only uses that 1 GB for about 5 minutes. This is a problem that needs to be addressed, as it negatively impacts the performance of Python when manipulating very large data sets. In fact, I found a mailing list post where the poster was looking for a workaround for this issue, but I can't find it now. Some posts to various lists [1] have stated that this is not a real problem because virtual memory takes care of it. This is fair if you are talking about a couple megabytes. In my case, I'm talking about ~700 MB of wasted RAM, which is a problem. First, this is wasting space which could be used for disk cache, which would improve the performance of my system. Second, when the system decides to swap out the pages that haven't been used for a while, they are dirty and must be written to swap. If Python ever wants to use them again, they will be brought it from swap. This is much worse than informing the system that the pages can be discarded, and allocating them again later. In fact, the other native object types (ints, lists) seem to realize that holding on to a huge amount of memory indefinitely is a bad strategy, because they explicitly limit the size of their free lists. So why is this not a good idea for other types? Does anyone else see this as a problem? Does anyone think this is not a problem? Proposal: - Python's memory allocator should occasionally free memory if the memory usage has been relatively constant, and has been well below the amount of memory allocated. This will incur additional overhead to free the memory, and additional overhead to reallocate it if the memory is needed again quickly. However, it will make Python co-operate nicely with other processes, and a clever implementation should be able to reduce the overhead. Problem: - I do not completely understand Python's memory allocator, but from what I see, it will not easily support this. Gross Hack: I've been playing with the fact that the "collect" function in the gc module already gets called occasionally. Whenever it gets called for a level 2 collection, I've hacked it to call a cleanup function in obmalloc.c. This function goes through the free pool list, reorganizes it to decrease memory fragmentation and decides based on metrics collected from the last run if it should free some memory. It currently works fine, except that it will permit the arena vector to grow indefinitely, which is also bad for a long running process. It is also bad because these cleanups are relatively slow as they touch every free page that is currently allocated, so I'm trying to figure out a way to integrate them more cleanly into the allocator itself. This also requires that nothing call the allocation functions while this is happening. I believe that this is reasonable, considering that it is getting called from the cyclical garbage collector, but I don't know enough about Python internals to figure that out. Eventually, I hope to do some benchmarks and figure out if this is actually a reasonable strategy. However, I was hoping to get some feedback before I waste too much time on this. Evan Jones [1] http://groups.google.com/groups?selm=mailman.1053801468.4243.python- list%40python.org -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From mwh at python.net Tue Oct 19 11:09:26 2004 From: mwh at python.net (Michael Hudson) Date: Tue Oct 19 11:09:29 2004 Subject: [Python-Dev] Re: import w/options In-Reply-To: <cl1gu2$245$1@sea.gmane.org> (Neal Becker's message of "Mon, 18 Oct 2004 18:43:14 -0400") References: <cl156k$1n1$1@sea.gmane.org> <2macujdabv.fsf@starship.python.net> <cl1gu2$245$1@sea.gmane.org> Message-ID: <2mr7nvgjll.fsf@starship.python.net> Neal Becker <ndbecker2@verizon.net> writes: > Michael Hudson wrote: > >> "Neal D. Becker" <ndbecker2@verizon.net> writes: >> >>> I have an application where it would be useful to set RTLD_GLOBAL on >>> dlopen >>> when import is used. Currently, python doesn't provide this option. Why >>> not allow an optional argument to import that would allow this? >> >> I take it you'd be interested to learn about sys.setdlopenflags? >> > > Could be. But I checked the current cvs python source, and what I see is > dlopen is called with flags set to 0. What am I missing? I don't know. That's not what I see in current CVS... the dlopen call I'm on about is line 130 of Python/dynload_shlib.c. (I notice that sys.setdlopenflags is present on Darwin, where it isn't going to make much difference...). Cheers, mwh -- 39. Re graphics: A picture is worth 10K words - but only those to describe the picture. Hardly any sets of 10K words can be adequately described with pictures. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From lcaamano at gmail.com Tue Oct 19 14:32:05 2004 From: lcaamano at gmail.com (Luis P Caamano) Date: Tue Oct 19 14:32:08 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 15, Issue 46 In-Reply-To: <20041019100214.297BB1E4013@bag.python.org> References: <20041019100214.297BB1E4013@bag.python.org> Message-ID: <c56e219d0410190532555e3381@mail.gmail.com> On Tue, 19 Oct 2004 12:02:14 +0200 (CEST), Evan Jones <ejones@uwaterloo.ca> wrote: > Subject: [Python-Dev] Changing pymalloc behaviour for long running > processes > [ snip ] > > The short version of the problem is that obmalloc.c never frees memory. > This is a great strategy if the application runs for a short time then > quits, or if it has fairly constant memory usage. However, applications > with very dynamic memory needs and that run for a long time do not > perform well because Python hangs on to the peak amount of memory > required, even if that memory is only required for a tiny fraction of > the run time. With my application, I have a python process which occupy > 1 GB of RAM for ~20 hours, even though it only uses that 1 GB for about > 5 minutes. This is a problem that needs to be addressed, as it > negatively impacts the performance of Python when manipulating very > large data sets. In fact, I found a mailing list post where the poster > was looking for a workaround for this issue, but I can't find it now. > > Some posts to various lists [1] have stated that this is not a real > problem because virtual memory takes care of it. This is fair if you > are talking about a couple megabytes. In my case, I'm talking about > ~700 MB of wasted RAM, which is a problem. First, this is wasting space > which could be used for disk cache, which would improve the performance > of my system. Second, when the system decides to swap out the pages > that haven't been used for a while, they are dirty and must be written > to swap. If Python ever wants to use them again, they will be brought > it from swap. This is much worse than informing the system that the > pages can be discarded, and allocating them again later. In fact, the > other native object types (ints, lists) seem to realize that holding on > to a huge amount of memory indefinitely is a bad strategy, because they > explicitly limit the size of their free lists. So why is this not a > good idea for other types? > > Does anyone else see this as a problem? > This is such a big problem for us that we had to rewrite some of our daemons to fork request handlers so that the memory would be freed. That's the only way we've found to deal with it, and it seems, that's the preferred python way of doing things, using processes, IPC, fork, etc. instead of threads. In order to be able to release memory, the interpreter has to allocate memory in chunks bigger than the minimum that can be returned to the OS, e.g., in Linux that'd be 256bytes (iirc), so that libc's malloc would use mmap to allocate that chunk. Otherwise, if the memory was obtained with brk, then in most virtually all OSes and malloc implementations, it won't be returned to the OS even if the interpreter frees the memory. For example, consider the following code in the interactive interpreter: for i in range(10000000): pass That run will create a lot of little integer objects and the virtual memory size of the interpreter will quickly grow to 155MB and then drop to 117MB. The 117MB left are all those little integer objects that are not in use any more that the interpreter would reuse as needed. When the system needs memory, it will page out the pages where these objects have been allocated to swap. In our application, paging to swap is extremely bad because sometimes we're running the OS booted from the net without swap. The daemon has to loop over list of 20 to 40 thousand items at a time and it quickly grows to 60mb on the first run and then continues to grow from there. When something else needs memory, it tries to swap and then crashes. In the example above, the difference between 155MB and 117MB is 37MB, which I assume is the size of the list object returned by 'range()' which contains the references to the integers. The list goes away when the interpreter finishes running the loop and because it was already known how big it was going to be, it was allocated as a big chunk using mmap (my speculation). As a result, that memory was given back to the OS and the virtual memory size of the interpreter went down from 155MB to 117MB. Regards, -- Luis P Caamano Atlanta, GA USA PS I rarely post to python-dev, this is probably the first time, so please let me take this opportunity to thank all the python developers for all your efforts, such a great language, and great tool. My respect and admiration to all of you. From ejones at uwaterloo.ca Tue Oct 19 15:18:56 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 15:19:07 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 15, Issue 46 In-Reply-To: <c56e219d0410190532555e3381@mail.gmail.com> References: <20041019100214.297BB1E4013@bag.python.org> <c56e219d0410190532555e3381@mail.gmail.com> Message-ID: <6E2F71BA-21D1-11D9-B920-0003938016AE@uwaterloo.ca> On Oct 19, 2004, at 8:32, Luis P Caamano wrote: > This is such a big problem for us that we had to rewrite some of our > daemons > to fork request handlers so that the memory would be freed. That's > the only > way we've found to deal with it, and it seems, that's the preferred > python > way of doing things, using processes, IPC, fork, etc. instead of > threads. Phew! I'm glad I'm not the only one who is frustrated by this. I am willing to put the time in to fix this issue, as long as the Python community doesn't think this is a dumb idea. > In order to be able to release memory, the interpreter has to allocate > memory in chunks bigger than the minimum that can be returned to the > OS, e.g., in Linux that'd be 256bytes (iirc), so that libc's malloc > would > use mmap to allocate that chunk. Otherwise, if the memory was > obtained with brk, then in most virtually all OSes and malloc > implementations, > it won't be returned to the OS even if the interpreter frees the > memory. Absolutely correct. Luckily, this is not a problem for Python, as its current behaviour is this: a) If the allocation is > 256 bytes, call the system malloc. b) If the allocation is < 256, use its own malloc implementation, which allocates memory in 256 kB chunks and never releases it. I want to change the behaviour of (b). If I call free() on these 256 kB chunks, they *do* in fact get released to the operating system. > That run will create a lot of little integer objects and the virtual > memory > size of the interpreter will quickly grow to 155MB and then drop to > 117MB. > The 117MB left are all those little integer objects that are not in > use any > more that the interpreter would reuse as needed. Oops, you are right: I was incorrect in my previous post. Python does not limit the number of free integers, etc. that it keeps around. This is part of the problem: Each of Python's own native types maintains its own freelist. If you go create a huge list of ints, deallocate it, then create a huge list of floats, it will allocate more free memory, even though it doesn't need to. If the pymalloc allocator was used instead, it would recycle the memory that was used by the integers. At the moment, even if I change the behaviour of pymalloc, it still won't solve this problem, as each type keeps its own freelist. This is also on my TODO list: benchmark the performance of using the pymalloc allocator for these elements. > In our application, paging to swap is extremely bad because sometimes > we're running the OS booted from the net without swap. The daemon has > to > loop over list of 20 to 40 thousand items at a time and it quickly > grows to > 60mb on the first run and then continues to grow from there. When > something > else needs memory, it tries to swap and then crashes. This is another good point: Some systems do not have swap. The changes I am proposing would not make Python less memory hungry, but they would mean that > In the example above, the difference between 155MB and 117MB is 37MB, > which I > assume is the size of the list object returned by 'range()' which > contains the > references to the integers. The list goes away when the interpreter > finishes > running the loop and because it was already known how big it was going > to be, > it was allocated as a big chunk using mmap (my speculation). As a > result, that > memory was given back to the OS and the virtual memory size of the > interpreter > went down from 155MB to 117MB. That is correct. If you look at the implementation for lists, it keeps a maximum of 80 free lists around, and immediately frees the memory for the containing array. Again, this seems like it is sub-optimal to me: In some cases, if a program uses a lot of lists, 80 lists may not be enough. For others, 80 may be too much. It seems to me that a more dynamic allocation policy could be more efficient. Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From ncoghlan at iinet.net.au Tue Oct 19 15:44:42 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue Oct 19 15:43:29 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 15, Issue 46 In-Reply-To: <6E2F71BA-21D1-11D9-B920-0003938016AE@uwaterloo.ca> References: <20041019100214.297BB1E4013@bag.python.org> <c56e219d0410190532555e3381@mail.gmail.com> <6E2F71BA-21D1-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <41751A4A.5070003@iinet.net.au> Evan Jones wrote: > That is correct. If you look at the implementation for lists, it keeps a > maximum of 80 free lists around, and immediately frees the memory for > the containing array. Again, this seems like it is sub-optimal to me: In > some cases, if a program uses a lot of lists, 80 lists may not be > enough. For others, 80 may be too much. It seems to me that a more > dynamic allocation policy could be more efficient. I knew this discussion sounded familiar. . . http://mail.python.org/pipermail/python-dev/2004-June/045403.html (and assorted replies) I'm not saying I *like* the unbounded lists. . . but there's a reason they're still like that (i.e. getting the memory usage down tends to take some of Python's speed with it - and there isn't exactly a lot of that to be spared!). Still, fresh eyes on the problem may see something new :) Cheers, Nick. From ejones at uwaterloo.ca Tue Oct 19 16:25:27 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 16:25:40 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 15, Issue 46 In-Reply-To: <41751A4A.5070003@iinet.net.au> References: <20041019100214.297BB1E4013@bag.python.org> <c56e219d0410190532555e3381@mail.gmail.com> <6E2F71BA-21D1-11D9-B920-0003938016AE@uwaterloo.ca> <41751A4A.5070003@iinet.net.au> Message-ID: <B8D0DCCE-21DA-11D9-B920-0003938016AE@uwaterloo.ca> On Oct 19, 2004, at 9:44, Nick Coghlan wrote: > I knew this discussion sounded familiar. . . > > http://mail.python.org/pipermail/python-dev/2004-June/045403.html > (and assorted replies) Great! Thank you for the reference, this discussion is useful. Why is it that it is so hard to find stuff in mailing list archives? I spent about 5 hours poking around Google and manually browsing the archives, and didn't see that thread. > I'm not saying I *like* the unbounded lists. . . but there's a reason > they're still like that (i.e. getting the memory usage down tends to > take some of Python's speed with it - and there isn't exactly a lot of > that to be spared!). Of course, and this has to be carefully considered and carefully benchmarked as well. I don't think that a bounded free list approach is the answer, as it can be shown that any arbitrary, fixed limit is bad is some situations. Basically, I'm talking about including some cache management into Python. My idea is to add some instrumentation that monitors the size of the free lists over time. If the free lists are much larger than what is being used, go and free some of them. This will allow Python's memory usage to dynamically adjust to the workload, and should limit the overhead. There are two challenges to this idea: 1. How can we run such an occasional task? It should be called regularly, if Python is actively executing or not, just like a garbage collector thread would be. I don't think making this collection part of the normal call to allocate/free objects is a good idea, because that would add some expensive overhead in what needs to be a fast path for Python. 2. Can this be done with minimal overhead? The instrumentation needs to have near zero overhead, and the collection task needs to have small overhead. I haven't thought this out completely, but it seems to me that something better should be possible. Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From gvanrossum at gmail.com Tue Oct 19 17:00:23 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Tue Oct 19 17:00:26 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <200410182221.43632.fdrake@acm.org> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> Message-ID: <ca471dc20410190800766db981@mail.gmail.com> Let me suggest two variations that I have used successfully in my day job (which is also my night job :). 1. For parsing .ini files, I wrote a wrapper around ConfigParser. The Python-level API looks like this (anonymized and idealized): from XXX import MyConfigWrapper, optstr, optint, optbool, optfloat class Config(MyConfigWrapper): poll_time = optfloat("network-parameters", "poll-time") use_ssl = optbool("network-parameters", "use-ssl") window_title = optstr("ui-parameters", "window-title") # etc. This allows the Python names for variables to differ from the names used in the .ini file, and abstracts away the section names completely from the Python API. This makes it possible to rename variables and sections in the config file without having to touch the Python code that uses them. This will save my butt when our marketing team comes up with the *real* name for our product, since currently the section names all have the provisional product name in it, and the config file is considered "user-facing" so references to the old product name have to be expunged. Also, I don't see much point in having to use longer references in my Python code -- the total number of config parameters and their uses are such that I can easily come up with a single unique name for every option, and yet in the .ini file I'd like to have more than one section. Note that optstr etc. construct full properties that allow me to set and delete the parameter values as well, and then ConfigParser can be told to write back the modified .ini file. This loses the ordering and comment, but I don't care (although I wish ConfigParser would order things alphabetically rather than per dictionary hash). I don't think I have to explain that optint returns a Python int, etc. 2. We're handling modest amounts of XML, all using home-grown DTDs and with no specific requirements to interface to other apps or XML tools. I wrote a metaclass which lets me specify the DTD using Python syntax. Again, my approach is slightly lower-level than previous proposals here but has the advantage of letting you be explicit about the mapping between Python and XML names, both for attributes and for subelements. The metaclass handles reading and writing. It supports elements containing text (is that CDATA? I never know) or sub-elements, but not both. For sub-elements, it supports cases where one element has any number of sub-elements of a certain type, which are then collected in a list, so you can refer to them using Python sequence indexing/slicing notation. It also supports elements that have zero or one sub-element of a certain type; absence is indicated by setting the corresponding attribute to None. I don't support namespaces, although I expect it would be easy enough to add them. I don't support unrecognized elements or attributes: while everything can be omitted (and defaults to None), unrecognized attributes or elements are always rejected. (I suppose that could be fixed too if desired.) Here's an example: from XXX import ElementMetaClass, String, Integer, Float, Boolean, Date class Inner(ElementMetaClass): "Definition for <inner>" __element__ = "inner" __attributes__ = [("count", Integer), ("name", String), ("expiration-date", Date), ("date-created", Date), ("special", Boolean)] __characters__ = "text" # CDATA (?) is stored as self.text class Outer(ElementMetaClass): "Definition for <outer>" __element__ = "outer" __attributes__ = [("name", String)] __children__ = [("innerElements[]", Inner)] Note that for attributes, the name given is used both as the Python name and as the XML name, except that hyphens in XML are translated into underscores in Python, and vice versa. For sub-elements, the __element__ attribute of the class determines the element name, and the name given in the list of __children__ determines the Python name; if this ends in "[]" it is a repeatable element. I'm undecided on whether I like the approach with lists of (name, type) tuples better than the approach with property factories like in the first example; the list approach allows me to order the attributes and sub-elements consistently upon rendering, but I'm not particularly keen on typing string quotes around Python identifiers. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From alex.nanou at gmail.com Tue Oct 19 17:32:08 2004 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Tue Oct 19 17:32:10 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? Message-ID: <36f8892204101908321d8d0e9d@mail.gmail.com> here it is.. actually quite simple, the option not to register extensions simply unregisters them.... thanks. -- Alex. From tim.peters at gmail.com Tue Oct 19 18:14:45 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 19 18:14:49 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <1f7befae04101909146b53edd4@mail.gmail.com> [Evan Jones] > I know that this has been discussed a bit in the past, but I was hoping > that some Python gurus could shed some light on this issue, and maybe > let me know if there are any plans for solving this problem. I know a > hack that might work, but there must be a better way to solve this > problem. I agree there are several issues here that are important for significant classes of apps, but have no plans to do anything about them (I simply don't have time for it). I'm not aware of anyone else intending to work on these areas either, so it's all yours <wink>. > The short version of the problem is that obmalloc.c never frees memory. True. That's one major problem for some apps. Another major problem for some apps is due to unbounded internal free lists outside of obmalloc. Another is that the platform OS+libc may not shrink VM at times even when memory is returned to the system free(). ... > In fact, the other native object types (ints, lists) seem to realize that holding on > to a huge amount of memory indefinitely is a bad strategy, because they > explicitly limit the size of their free lists. Most native object types don't have free lists (there are *many* native object types); they use pymalloc or the system malloc; type-specific free lists are generally found attached only to "high use" native types, where speed and/or memory-per-object was thought important enough to bother with a custom free list. Not all custom free lists are implemented in the same basic way. The most important oddballs are the free lists for ints and floats, which are unbounded and immortal. ... > Proposal: > - Python's memory allocator should occasionally free memory That's a worthy goal. > if the memory usage has been relatively constant, and has been well > below the amount of memory allocated. That's a possible implementation strategy. I think you'll find it helpful to distinguish goals from implementations. > This will incur additional overhead to free the memory, and additional overhead > to reallocate it if the memory is needed again quickly. However, it will make > Python co-operate nicely with other processes, This is so complicated in real life -- depends on the OS, depends on details of the system malloc's implementation, "what works" on one platform may not work on another, etc. > and a clever implementation should be able to reduce the overhead. > > Problem: > - I do not completely understand Python's memory allocator, but from > what I see, it will not easily support this. Of course if it were easy for obmalloc to release unused arenas, it would already do so <0.3 wink>. > Gross Hack: > > I've been playing with the fact that the "collect" function in the gc > module already gets called occasionally. Whenever it gets called for a > level 2 collection, I've hacked it to call a cleanup function in > obmalloc.c. This function goes through the free pool list, reorganizes > it to decrease memory fragmentation Unsure what this means, because an object in CPython can never be relocated. If I view an obmalloc arena as an alternating sequence of blocks (a contiguous region of allocated objects) and gaps (a contiguous region of available bytes), then if I can't rearrange the blocks (and I can't), I can't rearrange the gaps either -- the set of gaps is the complement of the set of blocks. Maybe you just mean that you collapse adjacent free pools into a free pool of a larger size class, when possible? If so, that's a possible step on the way toward identifying unused arenas, but I wouldn't call it an instance of decreasing memory fragmentation. In apps with steady states, between steady-state transitions it's not a good idea to "artificially" collapse free pools into free pools of larger size, because the app is going to want to reuse pools of the specific sizes it frees, and obmalloc optimizes for that case. If the real point of this (whatever it is <wink>) is to identify free arenas, I expect that could be done a lot easier by keeping a count of allocated pools in each arena; e.g., maybe at the start of the arena, or by augmenting the vector of arena base addresses. > and decides based on metric collected from the last run if it should free > some memory. It currently works fine, except that it will permit the arena > vector to grow indefinitely, which is also bad for a long running process. It > is also bad because these cleanups are relatively slow as they touch every > free page that is currently allocated, so I'm trying to figure out a way to > integrate them more cleanly into the allocator itself. > > This also requires that nothing call the allocation functions while > this is happening. I believe that this is reasonable, considering that > it is getting called from the cyclical garbage collector, but I don't > know enough about Python internals to figure that out. In theory, the calling thread holds the GIL (global interpreter lock) whenever an obmalloc function is called. That's why the lock macros inside obmalloc expand to nothing (and not locking inside obmalloc is a significant speed win). But in some versions of reality, that isn't true. The best available explanation is in new_arena()'s long internal comment block: because of historical confusions about what Python's memory API *is*, it's possible that extension modules outside the core are incorrectly calling the obmalloc free() when they should be calling the system free(), and doing so without holding the GIL. At the time obmalloc last got a rework, we did find some extensions that were in fact mixing PyObject_{New, NEW} with PyMem_{Del, DEL, Free, FREE}. obmalloc endures extreme pain now to try to ensure that still works, despite the lack of proper thread locking. As the end of that comment block says, * Read the above 50 times before changing anything in this * block. Now all such insane uses have been officially deprecated, so you could be bold and just assume obmalloc is always entered by a thread holding the GIL now. I don't know whether it's possible to get away with that, though -- if some "important" extension module is still careless here, it will break in ways that are all of catastrophic, rare, and difficult to reproduce or analyze, If I could make time for this, I'd risk it (but for 2.5, not for 2.4.x), and proactively search for-- and repair --external extension modules that may still be insane in this respect. > Eventually, I hope to do some benchmarks and figure out if this is > actually a reasonable strategy. However, I was hoping to get some > feedback before I waste too much time on this. It's only a waste if it ultimately fails <wink>. From dan at sidhe.org Tue Oct 19 18:29:15 2004 From: dan at sidhe.org (Dan Sugalski) Date: Tue Oct 19 18:29:33 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <1f7befae04101909146b53edd4@mail.gmail.com> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> Message-ID: <a06200406bd9aefc90f72@[10.0.1.5]> At 12:14 PM -0400 10/19/04, Tim Peters wrote: >[Evan Jones] > > The short version of the problem is that obmalloc.c never frees memory. > >True. That's one major problem for some apps. Another major problem >for some apps is due to unbounded internal free lists outside of >obmalloc. Another is that the platform OS+libc may not shrink VM at >times even when memory is returned to the system free(). FWIW, at this point nearly all OSes have a means of allocating memory from the system that can then later be returned to the system. (malloc and free tend *not* to do this) Even on Unix platforms you can play the "mmap a file with no filename" game to get returnable chunks. Unfortunately there's often a limit to the number of these chunks you can get from the OS so it's not safe to unconditionally replace malloc and free. (The performance impact isn't worth it either) If someone's going to do this, I'd suggest the place to start is adding separate allocate and free API entry points for returnable chunks and put in some criteria for getting memory from them (allocation size, particular spots that allocate, or whatever) and see where you go from there. I'll point out that, from experience, this can be a non-trivial thing, and with a non-moving GC system you'll probably find that there are relatively few places where there's a win from it. It does, though, tend to flush out dangling pointers. (Whether this is good or bad is a separate issue, of course ;) -- Dan --------------------------------------it's like this------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From phd at mail2.phd.pp.ru Tue Oct 19 19:01:26 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Tue Oct 19 19:01:30 2004 Subject: [Python-Dev] Patches Message-ID: <20041019170126.GA21167@phd.pp.ru> Hello! I submitted a number of patches to the SF tracker. Few of them were accepted, but most of them still are unreviewd by core team. What can I do to help? (A polite way to say "how can I push them?" :) I have updated them to Python 2.4b1. Actually, only one of them required minor changes (different line numbers). But I must admit that the patches are not a simple bug fix, so it is probably too late to apply them for 2.4. They can wait for 2.5... but I am afraid they will be just forgotten. The patches are: http://python.org/sf/754022 This is the biggest and the oldest. It hangs in the tracker since Python 2.2. In short, it makes the webbrowser.py runs through _tryorder list of browsers and trie to run every browser until one started successfully. Currently webbrowser.py tries to run a browser and if it fails - stops trying. Assigned to Fred Drake. http://python.org/sf/784089 A program to scan python files and list those require -*- coding -*- directive. Reviewed by Marc-Andre Lemburg and Andrew Kuchling. http://python.org/sf/821862 Makes ftplib.py a bit more RFC959-compliant. The RFC requires the FTP protocol to be run not over TCP but over telnet. For most people there is no difference but there is a subtle different for those who use non-ASCII encodings: chr(255) - a special character in telnet - requires a special handling. My patch adds a toggle that allows to turn telnet on (actually, it only doubles chr(255) in command stream, but it is one of two things that are required for FTP-over-telnet). By default it is off to preserve backward compatibility. http://python.org/sf/1038388 The smallest and the simplest of all. 3 lines patch that adds __main__ to whichdb.py, so one can run it on the command line: $ whichdb.py *.db Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ejones at uwaterloo.ca Tue Oct 19 19:25:28 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 19:25:39 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <1f7befae04101909146b53edd4@mail.gmail.com> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> Message-ID: <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> On Oct 19, 2004, at 12:14, Tim Peters wrote: > True. That's one major problem for some apps. Another major problem > for some apps is due to unbounded internal free lists outside of > obmalloc. Another is that the platform OS+libc may not shrink VM at > times even when memory is returned to the system free(). There is absolutely nothing I can do about that, however. On platforms that matter to me (Mac OS X, Linux) some number of large malloc() allocations are done via mmap(), and can be immediately released when free() is called. Hence, large blocks are reclaimable. I have no knowledge about the implementation of malloc() on Windows. Anyone care to enlighten me? Another approach is to not free the memory, but instead to inform the operating system that the pages are unused (on Unix, madvise(2) with MADV_DONTNEED or MADV_FREE). When this happens, the operating system *may* discard the pages, but the address range remains valid: If it is touched again in the future, the OS will allocate the new page. This would require some dramatic changes to Python's internals. >> if the memory usage has been relatively constant, and has been well >> below the amount of memory allocated. > That's a possible implementation strategy. I think you'll find it > helpful to distinguish goals from implementations. You are correct: This is an implementation detail. However, it is a relatively important one, as I do not want to change Python's aggressive memory recycling behaviour. > Maybe you just mean that you collapse adjacent free pools into a free > pool of a larger size class, when possible? If so, that's a possible > step on the way toward identifying unused arenas, but I wouldn't call > it an instance of decreasing memory fragmentation. I am not moving around Python objects, I'm just dealing with free pools and arenas in obmalloc.c at the moment. There two separate things I am doing: 1. Scan through the free pool list, and count the number of free pools in each arena. If an arena is completely unused, I free it. If there is even one pool in use, the arena cannot be freed. 2. Sorting the free pool list so that "nearly full" arenas are used before "nearly empty" arenas. Right now, when a pool is free, it is pushed on the list. When one is needed, it is popped off. This leads to an LRU allocation of memory. What I am doing is removing all the free pools from the list, and putting them back on so that areas that have more free pools are used later, while arenas with less free pools are used first. In my crude tests, the second detail increases the number of completely free arenas. However, I suspect that differentiating between free arenas and used arenas, like is already done for pools, would be a good idea. > In apps with steady states, between steady-state transitions it's not > a good idea to "artificially" collapse free pools into free pools of > larger size, because the app is going to want to reuse pools of the > specific sizes it frees, and obmalloc optimizes for that case. Absolutely: I am not touching that. I'm working from the assumption that pymalloc has been well tested and well tuned and is appropriate for Python workloads. I'm just trying to make it free memory occasionally. > If the real point of this (whatever it is <wink>) is to identify free > arenas, I expect that could be done a lot easier by keeping a count of > allocated pools in each arena; e.g., maybe at the start of the arena, > or by augmenting the vector of arena base addresses. You are correct, and this is something I would like to play with. This is, of course, a tradeoff between overhead on each allocation and deallocation, and one big occasionally overhead caused by the "cleanup" process. I'm going to try and take a look at this tonight, if I get some real work done this afternoon. > But in some versions of reality, that isn't true. The best available > explanation is in new_arena()'s long internal comment block: because > of historical confusions about what Python's memory API *is*, it's > possible that extension modules outside the core are incorrectly > calling the obmalloc free() when they should be calling the system > free(), and doing so without holding the GIL. Let me just make sure I am clear on this: Some extensions use native threads, is that why this is a problem? Because as far as I am aware, the Python interpreter itself is not threaded. So how does the cyclical garbage collector work? Doesn't it require that there is no execution going on? > Now all such insane uses have been officially deprecated, so you could > be bold and just assume obmalloc is always entered by a thread holding > the GIL now. I would rather not break this property of obmalloc. However, this leads to a big problem: I'm not sure it is possible to have an occasional cleanup task be lockless and co-operate nicely with other threads, since by definition it needs to go and mess with all the arenas. One of the reason that obmalloc *doesn't* have this problem is because it never releases memory. > It's only a waste if it ultimately fails <wink>. It is also a waste if the core Python developers decide it is a bad idea, and don't want to accept patches! :) Thanks for your feedback, Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From martin at v.loewis.de Tue Oct 19 20:00:56 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 20:00:56 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <41755658.8080806@v.loewis.de> Evan Jones wrote: > Some posts to various lists [1] have stated that this is not a real > problem because virtual memory takes care of it. This is fair if you > are talking about a couple megabytes. In my case, I'm talking about > ~700 MB of wasted RAM, which is a problem. This is not true. The RAM is not wasted. As you explain later, the pages will be swapped out to swap space, making the RAM available again for other tasks. > First, this is wasting space > which could be used for disk cache, which would improve the performance > of my system. And indeed, this is what the operating system does for you: free the memory (by swapping it out), then using the memory for disk cache, thus improving performance of your system. > Second, when the system decides to swap out the pages > that haven't been used for a while, they are dirty and must be written > to swap. That is true. > If Python ever wants to use them again, they will be brought > it from swap. Yes. However, your assumption is that Python never wants to use them again, because the peek memory consumption is only local. On the design memory management systems, there is the notion of a working set. Python will have a relatively constant working set over short periods of time, and current operating systems will manage to keep the working set in memory if the system has sufficient memory in the first place. As the working set grows or shrinks, pages get swapped in and out. As Tim explains, this is really hard to avoid. > This is much worse than informing the system that the > pages can be discarded, and allocating them again later. Unfortunately, as Tim explains, there is no way to reliably "inform" the system. free(3) may or may not be taken as such information. Regards, Martin From martin at v.loewis.de Tue Oct 19 20:09:40 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 20:09:40 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <41755864.3000408@v.loewis.de> Evan Jones wrote: >> Another is that the platform OS+libc may not shrink VM at >> times even when memory is returned to the system free(). > > > There is absolutely nothing I can do about that, however. You could if you wanted. Don't use malloc/free, but use mmap/munmap, VirtualAlloc/VirtualFree, etc. > Anyone care > to enlighten me? Microsoft ships the source of its malloc(3) implementation together with VC; you need to install the CRT source to see it. > Let me just make sure I am clear on this: Some extensions use native > threads, is that why this is a problem? Because as far as I am aware, > the Python interpreter itself is not threaded. So how does the cyclical > garbage collector work? Doesn't it require that there is no execution > going on? The garbage collector holds the GIL. So while there could be other threads running, they must not manipulate any PyObject*. If they try to, they need to obtain the GIL first, which will make them block until the garbage collector is complete. > It is also a waste if the core Python developers decide it is a bad > idea, and don't want to accept patches! :) That will ultimately depend on the patches. The feature itself would be fine, as Tim explains. However, patches might be rejected because: - they are incorrect, - their correctness cannot easily be established, - they change unrelated aspects of the interpreter, - they have undesirable performance properties, or - they have other problems I can't think of right now :-) Regards, Martin From martin at v.loewis.de Tue Oct 19 20:10:30 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 20:10:31 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <36f8892204101908321d8d0e9d@mail.gmail.com> References: <36f8892204101908321d8d0e9d@mail.gmail.com> Message-ID: <41755896.7030508@v.loewis.de> Alex A. Naanou wrote: > here it is.. > actually quite simple, the option not to register extensions simply > unregisters them.... Arghh, still :-( Regards, Martin From martin at v.loewis.de Tue Oct 19 20:15:51 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 20:15:52 2004 Subject: [Python-Dev] Patches In-Reply-To: <20041019170126.GA21167@phd.pp.ru> References: <20041019170126.GA21167@phd.pp.ru> Message-ID: <417559D7.5030809@v.loewis.de> Oleg Broytmann wrote: > I submitted a number of patches to the SF tracker. Few of them were > accepted, but most of them still are unreviewd by core team. > > What can I do to help? (A polite way to say "how can I push them?" :) I have suggested a procedure in the past, which never has been executed so far. I promise that I will stick to it if somebody actually executes it. Here it goes: - For every patch you want to get reviewed, review 10 (ten) other patches. A review should explain what kind of analysis was performed (code inspection, testing on platform X, etc), and it should include a recommendation for acceptance, rejection, or modification. If a patch already has a suggestion of modification, which was unanswered for some time, either post the modification yourself, or suggest rejection of the patch. - Post your reviews to the relevant patches. - Post a summary of the patches reviewed, including the patch number and review recommendation, to python-dev, along with the patch you want reviewed in turn. Notice that a review at this point may conclude with "acceptable for 2.5", which means that we need to go back to the patch after 2.4 has been forked. Regards, Martin From martin at v.loewis.de Tue Oct 19 21:20:06 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 21:20:14 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <36f8892204101908321d8d0e9d@mail.gmail.com> References: <36f8892204101908321d8d0e9d@mail.gmail.com> Message-ID: <417568E6.9040009@v.loewis.de> Alex A. Naanou wrote: > actually quite simple, the option not to register extensions simply > unregisters them.... I can't reproduce this, on XPSP2. If I install 2.3.4, then install 2.4b1, not selecting "register extensions", then run the program print sys.version it still prints 2.3.4. If I then invoke the MSI file again, chose "Change Python", install the extensions, the same program will print 2.4b1. If I then Change Python again, uninstalling the extensions, nothing will be associated anymore; this is (somewhat) intentional. What precisely did you do? Regards, Martin From fredrik at pythonware.com Tue Oct 19 21:30:03 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue Oct 19 21:27:56 2004 Subject: [Python-Dev] Re: python/dist/src/Doc/lib libzlib.tex,1.28,1.29 References: <E1CJz6J-000669-Um@sc8-pr-cvs1.sourceforge.net> Message-ID: <cl3prn$b31$1@sea.gmane.org> akuchling@users.sourceforge.net wrote: > Index: libzlib.tex > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Doc/lib/libzlib.tex,v > retrieving revision 1.28 > retrieving revision 1.29 > diff -u -d -r1.28 -r1.29 > --- libzlib.tex 21 Jun 2003 14:15:25 -0000 1.28 > +++ libzlib.tex 19 Oct 2004 18:52:49 -0000 1.29 > @@ -9,8 +9,8 @@ > For applications that require data compression, the functions in this > module allow compression and decompression, using the zlib library. > The zlib library has its own home page at > -\url{http://www.gzip.org/zlib/}. Version 1.1.3 is the > -most recent version as of September 2000; use a later version if one > +\url{http://www.gzip.org/zlib/}. Version 1.2.1 is the > +most recent version as of October 2004; use a later version if one > is available. There are known incompatibilities between the Python > module and earlier versions of the zlib library. that's somewhat misleading, isn't it? version 1.2.1 is of course the most recent version right now, but zlib works just fine with 1.1.3 or later. I suggest rephrasing the "known incompatibilities" part to recommend 1.1.4 or later (there's a known security problem with 1.1.3; see http://www.gzip.org/zlib/advisory-2002-03-11.txt for details). </F> From theller at python.net Tue Oct 19 21:49:56 2004 From: theller at python.net (Thomas Heller) Date: Tue Oct 19 21:49:36 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <417568E6.9040009@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Tue, 19 Oct 2004 21:20:06 +0200") References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> Message-ID: <ekjuqyhn.fsf@python.net> "Martin v. L?wis" <martin@v.loewis.de> writes: > Alex A. Naanou wrote: >> actually quite simple, the option not to register extensions simply >> unregisters them.... > > I can't reproduce this, on XPSP2. If I install 2.3.4, then install > 2.4b1, not selecting "register extensions", then run the program > > print sys.version > > it still prints 2.3.4. I could not reproduce it, but also was confused by the behaviour. Can it be that deinstallation of Python-2.4a3.msi (which I did before 2.4b1 was installed) removed the association of Python files with Python-2.3? Thomas From ejones at uwaterloo.ca Tue Oct 19 21:55:37 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 21:55:53 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <41755658.8080806@v.loewis.de> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <41755658.8080806@v.loewis.de> Message-ID: <D8972CD9-2208-11D9-B920-0003938016AE@uwaterloo.ca> On Oct 19, 2004, at 14:00, Martin v. L?wis wrote: >> Some posts to various lists [1] have stated that this is not a real >> problem because virtual memory takes care of it. This is fair if you >> are talking about a couple megabytes. In my case, I'm talking about >> ~700 MB of wasted RAM, which is a problem. > This is not true. The RAM is not wasted. As you explain later, the > pages will be swapped out to swap space, making the RAM available > again for other tasks. Well, it isn't "wasted," but it is not optimal. If the pages were freed, the OS would use them for disk cache (or for other programs). However, because the operating system believes that these pages contain data it must either do one of the following two things: a) Live with less disk cache (lower performance for disk I/O). b) Pre-emptively swap the pages to disk, which is super slow. (On Linux, you can control how pre-emptive the kernel is by adjusting the "swapiness" sysctl). If it chooses to swap them out, the next time Python touches those pages, it will pause as the OS reads them back from disk. It can only help the system's performance if we give it hints about which pages are no longer in use. >> If Python ever wants to use them again, they will be brought it from >> swap. > Yes. However, your assumption is that Python never wants to use them > again, because the peek memory consumption is only local. I am trying to correct the situation where Python is not going to use the pages for a long time. For most applications, Python's memory allocation policies are fine, but if you have a long running process that does nothing most of the time (say a low usage server) or does some huge pre-processing (my application), it keeps a ton of memory around for no reason. Right now, Python has very poor performance for my application because I have this massive memory peak, and very low average memory usage. Were I using Java, its usage would grow and shrink accordingly, thanks to the garbage collector releasing memory to the OS. Yes, with Python, we can't compact memory, but I think we can still do better than nothing. > As the working set grows or shrinks, pages > get swapped in and out. As Tim explains, this is really hard to > avoid. If you actually tell the operating system that the pages are unused, it won't swap unless it actually needs to. Right now, a lot of pages are being swapped in and out that are actually *garbage*. > Unfortunately, as Tim explains, there is no way to reliably > "inform" the system. free(3) may or may not be taken as such > information. As noted before, free() may not be sufficient, but mmap or madvise are. > The garbage collector holds the GIL. So while there could be other > threads running, they must not manipulate any PyObject*. If they try > to, they need to obtain the GIL first, which will make them block > until the garbage collector is complete. But as noted in a previous message, some extensions may not do this correctly, and try to do PyObject_Free anyway. Is that the problem that obmalloc tries to avoid? If the problem is only the possibility of PyObject_Free being called while another thread has the GIL, then I can probably avoid that issue. > That will ultimately depend on the patches. The feature itself would > be fine, as Tim explains. Great! That's basically what I am looking for. > However, patches might be rejected because: [snip] Of course, I certainly hope that Python wouldn't accept garbage patches! :) Thank you for your comments, Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From martin at v.loewis.de Tue Oct 19 22:36:59 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 19 22:36:59 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <ekjuqyhn.fsf@python.net> References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> <ekjuqyhn.fsf@python.net> Message-ID: <41757AEB.8090807@v.loewis.de> Thomas Heller wrote: > Can it be that deinstallation of Python-2.4a3.msi (which I did before > 2.4b1 was installed) removed the association of Python files with > Python-2.3? I don't call exactly, but this could very well be possible. When I tried to support advertising extensions, a side effect was that not installing the extensions would just remove the previous extensions; this is meant to be fixed in 2.4b1. Also, upgrading from 2.4a3 (with extensions installed) to 2.4b1 (with no extensions) would have the same effect - there is simply no support for restoring older versions of the extensions when uninstalling (and upgrading from 2.4a3 to 2.4b1 is actually the same as uninstalling 2.4a3 first). Regards, Martin From lcaamano at gmail.com Tue Oct 19 22:42:23 2004 From: lcaamano at gmail.com (Luis P Caamano) Date: Tue Oct 19 22:42:25 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running In-Reply-To: <20041019192049.4AB991E4008@bag.python.org> References: <20041019192049.4AB991E4008@bag.python.org> Message-ID: <c56e219d04101913425bd388c4@mail.gmail.com> > "Martin v. L?wis" <martin@v.loewis.de> wrote: > > Evan Jones wrote: > > Some posts to various lists [1] have stated that this is not a real > > problem because virtual memory takes care of it. This is fair if you > > are talking about a couple megabytes. In my case, I'm talking about > > ~700 MB of wasted RAM, which is a problem. > > This is not true. The RAM is not wasted. As you explain later, the > pages will be swapped out to swap space, making the RAM available > again for other tasks. > > > First, this is wasting space > > which could be used for disk cache, which would improve the performance > > of my system. > > And indeed, this is what the operating system does for you: free the > memory (by swapping it out), then using the memory for disk cache, thus > improving performance of your system. Your statements above assume that swapping is OK but that's not necessarily true. Some of our daemons run on systems with no swap or extremely slow swap. Either way it's bad if the system has to swap out 50-100 MBs of "unused" memory. Now, that's a waste. -- Luis P Caamano Atlanta, GA USA From tim.peters at gmail.com Tue Oct 19 22:53:07 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 19 22:53:10 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <1f7befae04101913531a388c39@mail.gmail.com> [Evan Jones] ... > There is absolutely nothing I can do about that, however. On platforms > that matter to me (Mac OS X, Linux) some number of large malloc() > allocations are done via mmap(), and can be immediately released when > free() is called. Hence, large blocks are reclaimable. I have no > knowledge about the implementation of malloc() on Windows. Anyone care > to enlighten me? Not me, I'm too short on time. Memory pragmatics on Windows varies both across Windows flavors and MS C runtime releases, so it's not a simple topic. In practice, at least the NT+ flavors of Windows, under MS VC 6.0 and 7.1 + service packs, appear to do a reasonable job of releasing VM reservations when free() gives a large block back. I wouldn't worry about older Windows flavors anymore. The native Win32 API has many functions that could be used for fine control. > ... > I am not moving around Python objects, I'm just dealing with free pools > and arenas in obmalloc.c at the moment. Good. > There two separate things I am doing: > > 1. Scan through the free pool list, and count the number of free pools > in each arena. If an arena is completely unused, I free it. If there is > even one pool in use, the arena cannot be freed. Yup. > 2. Sorting the free pool list so that "nearly full" arenas are used > before "nearly empty" arenas. Right now, when a pool is free, it is > pushed on the list. When one is needed, it is popped off. This leads to > an LRU allocation of memory. It's stack-like: it reuses the pool most recently emptied, because the expectation is that the most recently emptied pool is the most likely of all empty pools to be highest in the memory hierarchy. I really don't know what LRU (or MRU) might mean in this context (it's not like we've evicting something from a cache). > What I am doing is removing all the free pools from the list, and putting them > back on so that areas that have more free pools are used later, while arenas > with less free pools are used first. That sounds reasonable. > In my crude tests, the second detail increases the number of completely > free arenas. However, I suspect that differentiating between free > arenas and used arenas, like is already done for pools, would be a good > idea. Right. ... > Absolutely: I am not touching that. I'm working from the assumption > that pymalloc has been well tested and well tuned and is appropriate > for Python workloads. I'm just trying to make it free memory > occasionally. Harder than it looked, eh <wink>? >> If the real point of this (whatever it is <wink>) is to identify free >> arenas, I expect that could be done a lot easier by keeping a count of >> allocated pools in each arena ... > You are correct, and this is something I would like to play with. This > is, of course, a tradeoff between overhead on each allocation and > deallocation, It shouldn't be. Pool transitions among the "used", "full" and "empty" states don't occur on each alloc and dealloc. Note that PyObject_Free and PyObject_Malloc are both coded with the most frequent paths earliest in the function, and pool transitions don't occur until after a few return statements have passed. It's unusual not to get out via one of the "early returns"; the *bulk* of the code in each function (including pool transitions) isn't executed on most calls; in most calls, the affected pool both enters and leaves in the "used" state. > and one big occasionally overhead caused by the "cleanup" process. Or it may be small overhead, if all it's trying to do is free() empty arenas. Indeed, if arenas "grow states" too, *arena* transitions should be so rare that perhaps they could afford to do extra processing right then to decide whether to free() an arena that just transitioned to its notion of an empty state. ... > Let me just make sure I am clear on this: Some extensions use native > threads, By extension module I mean a module coded in C; and yes, any extension module that uses threads is probably using native threads. > is that why this is a problem? No, threads aren't the problem, in the sense that an alcoholic's problem isn't really alcohol, it's drinking <0.7 wink>. The problem is incorrect usage of the Python C API, and the most dangerous problem there is that old code may be calling PyMem_{Free, FREE, Del, DEL} while not holding the GIL. "Everyone always knew" that PyMem_{Free, FREE, Del, DEL} was just an irritating way to spell "free()", so some old code didn't worry about the GIL when calling it. Such code is fatally broken, but we're still trying to support it (or rather we *were*, when obmalloc was new; now it's still "supported" just in the sense that the excruciating support code still exists). The other twist is that we couldn''t map PyMem_{Free, FREE, Del, DEL} to the system free() directly (which would have solved the problem just above), because *other* broken old code called PyMem_{Free, FREE, Del, DEL} to release an object obtained via PyObject_New(). We're still supporting that too, but again just in the sense that the convolutions *to* support it still exist. If we changed PyMem_{Free, FREE, Del, DEL} to map to the system free(), all would be golden (except for broken old code mixing PyObject_ with PyMem_ calls). If any such broken code still exists, that remapping would lead to dramatic failures, easy to reproduce; and old code broken in the other, infinitely more subtle way (calling PyMem_{Free, FREE, Del, DEL} when not holding the GIL) would continue to work fine. > Because as far as I am aware, the Python interpreter itself is not threaded. Unsure what that means to you. Any number of threads can be running Python code in a single process, although the GIL serializes their execution *while* they're executing Python code. When a thread ends up in C code, it's up to the C code to decide whether to release the GIL and so allow other threads to run at the same time. If it does, that thread must reacquire the GIL before making another Python C API call (with very few exceptions, related to Python C API thread initialization and teardown functions). > So how does the cyclical garbage collector work? The same as every other part of Python's C implementation, *except* for this crazy exception in obmalloc: it assumes the GIL is held, and that no other thread can make a Python C API call until the GIL is released. Note that this doesn't necessarily mean that cyclic gc can assume that no other thread can run Python code until cyclic gc is done. Because gc may trigger destructors that in turn execute Python code (__del__ methods or weakref callbacks), it's all but certain other threads *can* run at such times (invoking Python code ends up in the interpreter main loop, which releases the GIL periodically to allow other threads to run). obmalloc doesn't have *that* problem, though -- nothing obmalloc does can cause Python code to get executed, so obmalloc can assume that the thread calling into it holds the GIL for as long as obmalloc wants. Except, again, for the crazy PyMem_{Free, FREE, Del, DEL} exception. > Doesn't it require that there is no execution going on? As above. >> Now all such insane uses have been officially deprecated, so you could >> be bold and just assume obmalloc is always entered by a thread holding >> the GIL now. > I would rather not break this property of obmalloc. I would -- it's backward compatibility hacks for insane code, which may not even exist anymore, and you'll find that it puts severe contraints on what you can do. > However, this leads to a big problem: I'm not sure it is possible to have an > occasional cleanup task be lockless and co-operate nicely with other threads, > since by definition it needs to go and mess with all the arenas. One of > the reason that obmalloc *doesn't* have this problem is because it > never releases memory. Yes, but that's backwards: obmalloc never releases memory in part *because* of this thread problem. Indeed, when new_arena() has to grow the vector of arena base addresses, it doesn't realloc(), it makes a copy into a new memory area, and deliberately lets the old vector *leak*. That's solely because some broken PyMem_{Free, FREE, Del, DEL} call may be simultaneously trying to access the vector, and without locking it's plain impossible to know whether or when that occurs. You'll have an equally impossible time trying to change the content of the arena base vector in virtually any way -- heck, we've got 40 lines of comments now just trying to explain what it took to support appending new values safely (and that's the only kind of mutation done on that vector now). Change PyMem_{Free, FREE, Del, DEL} to stop resolving to PyObject_ functions, and all that pain can go away -- obmalloc could then do anything it wanted to do without any thread worries. >> It's only a waste if it ultimately fails <wink>. > It is also a waste if the core Python developers decide it is a bad > idea, and don't want to accept patches! :) Sad to say, it's more likely that making time to review patches will be the bottleneck, and in this area careful review is essential. It's great that you can make some time for this now -- be optimistic! From mwh at python.net Tue Oct 19 22:54:55 2004 From: mwh at python.net (Michael Hudson) Date: Tue Oct 19 22:54:57 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <1f7befae04101909146b53edd4@mail.gmail.com> (Tim Peters's message of "Tue, 19 Oct 2004 12:14:45 -0400") References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> Message-ID: <2m1xfuh1i8.fsf@starship.python.net> Tim Peters <tim.peters@gmail.com> writes: > In theory, the calling thread holds the GIL (global interpreter lock) > whenever an obmalloc function is called. That's why the lock macros > inside obmalloc expand to nothing (and not locking inside obmalloc is > a significant speed win). > > But in some versions of reality, that isn't true. The best available > explanation is in new_arena()'s long internal comment block: because > of historical confusions about what Python's memory API *is*, it's > possible that extension modules outside the core are incorrectly > calling the obmalloc free() when they should be calling the system > free(), and doing so without holding the GIL. At the time obmalloc > last got a rework, we did find some extensions that were in fact > mixing PyObject_{New, NEW} with PyMem_{Del, DEL, Free, FREE}. > obmalloc endures extreme pain now to try to ensure that still works, > despite the lack of proper thread locking. As the end of that comment > block says, > > * Read the above 50 times before changing anything in this > * block. > > Now all such insane uses have been officially deprecated, so you could > be bold and just assume obmalloc is always entered by a thread holding > the GIL now. I don't know whether it's possible to get away with > that, though -- if some "important" extension module is still careless > here, it will break in ways that are all of catastrophic, rare, and > difficult to reproduce or analyze, If I could make time for this, I'd > risk it (but for 2.5, not for 2.4.x), and proactively search for-- and > repair --external extension modules that may still be insane in this > respect. Would it be possible to (in a debug build, presumably) do assert(I have the GIL); in PyObject_Free? Cheers, mwh -- If you don't have friends with whom to share links and conversation, you have social problems and you should confront them instead of joining a cultlike pseudo-community. -- "Quit Slashdot.org Today!" (http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq) From tim.peters at gmail.com Tue Oct 19 23:36:03 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 19 23:36:07 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <2m1xfuh1i8.fsf@starship.python.net> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> <2m1xfuh1i8.fsf@starship.python.net> Message-ID: <1f7befae041019143642ebc61a@mail.gmail.com> [Michael Hudson] > Would it be possible to (in a debug build, presumably) do > > assert(I have the GIL); > > in PyObject_Free? There's no canned way to do this. I suppose it could call PyGILState_Ensure(), then assert that the return value is PyGILState_LOCKED. PyGILState_Ensure() has to do a lot of work to figure out whether its calling thread has the GIL, and needs access to pystate.c internals to get the right answer; I expect simpler approaches are doomed to fail in some cases. If we changed the PyMem_Free spellings now (for 2.5, not *now* now <wink>) to call the system free() instead in a release build, the thread craziness obmalloc is trying to protect against would just go away by magic. It would be good to do the assert() you suggest anyway, but for a different reason then (catching unsafe calls in debug builds, where all of the PyMem family ends up in obmalloc). From aahz at pythoncraft.com Tue Oct 19 23:41:43 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Oct 19 23:41:50 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <D8972CD9-2208-11D9-B920-0003938016AE@uwaterloo.ca> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <41755658.8080806@v.loewis.de> <D8972CD9-2208-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <20041019214143.GA11458@panix.com> Here's an idea that may help implementation slightly, and will almost certainly increase the likelihood of any patch getting accepted: do the pool scanning and freeing only on specific call of e.g. gc.free_mem(). IIUC, you'll still need to change the allocation strategy slightly, but that's lower risk. Once your strategy has proven itself, we can start talking about ways to perform the freeing automatically. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From ejones at uwaterloo.ca Tue Oct 19 23:47:21 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 23:47:33 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <1f7befae04101913531a388c39@mail.gmail.com> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101909146b53edd4@mail.gmail.com> <DF275A7A-21F3-11D9-B920-0003938016AE@uwaterloo.ca> <1f7befae04101913531a388c39@mail.gmail.com> Message-ID: <74E064AE-2218-11D9-B920-0003938016AE@uwaterloo.ca> First, let me thank you for this very detailed reply. It really helped me understand a lot more about what is going on inside the Python interpreter. On Oct 19, 2004, at 16:53, Tim Peters wrote: > It's stack-like: it reuses the pool most recently emptied, because > the expectation is that the most recently emptied pool is the most > likely of all empty pools to be highest in the memory hierarchy. I > really don't know what LRU (or MRU) might mean in this context (it's > not like we've evicting something from a cache). Err... Right: MRU. It uses the most recently used free block. This is totally a cache: It's a cache of free memory pages. > Harder than it looked, eh <wink>? Actually, much. I spent about 6 hours figuring out what was going on. At this point, I think I have enough of a handle on the situation that I might as well go about trying to improve it. > Or it may be small overhead, if all it's trying to do is free() empty > arenas. Indeed, if arenas "grow states" too, *arena* transitions > should be so rare that perhaps they could afford to do extra > processing right then to decide whether to free() an arena that just > transitioned to its notion of an empty state. That is true. However, I don't think freeing arenas immediately is the best plan, as we don't really want to do that if the application is cyclical in its memory consumption (ie. it creates a ton of objects, then releases them, then does it again). I still think that some sort of periodic collection is best, as it will help Python adjust to applications with a wide variety of memory profiles. > If we changed PyMem_{Free, FREE, Del, DEL} to map to the system > free(), all would be golden (except for broken old code mixing > PyObject_ with PyMem_ calls). If any such broken code still exists, > that remapping would lead to dramatic failures, easy to reproduce; and > old code broken in the other, infinitely more subtle way (calling > PyMem_{Free, FREE, Del, DEL} when not holding the GIL) would continue > to work fine. Hmm... This seems like a logical approach to me. It certainly gives me a lot more freedom in reworking the memory allocator. Are there any objections to this idea? > Any number of threads can be running > Python code in a single process, although the GIL serializes their > execution *while* they're executing Python code. When a thread ends > up in C code, it's up to the C code to decide whether to release the > GIL and so allow other threads to run at the same time. If it does, > that thread must reacquire the GIL before making another Python C API > call (with very few exceptions, related to Python C API thread > initialization and teardown functions). Ah, now I understand! Creating a Python thread actually creates a native thread then, it's just that because of the GIL they run sequentially when executing Python code. This is an interesting approach! For some reason I was under the impression that the Python interpreter used user level threads to implement Python threads. > obmalloc doesn't have *that* problem, though -- nothing obmalloc does > can cause Python code to get executed, so obmalloc can assume that the > thread calling into it holds the GIL for as long as obmalloc wants. > Except, again, for the crazy PyMem_{Free, FREE, Del, DEL} exception. Terrific. This makes life much, much easier. > I would -- it's backward compatibility hacks for insane code, which > may not even exist anymore, and you'll find that it puts severe > contraints on what you can do. Again, does anyone object to this point of view before I begin working from this assumption? This means that I can assume that only one thread will call code in obmalloc at a time. I can do the same thing that the current obmalloc implementation does: Add the macros for the locks, but have them resolve to nothing. Thanks for the tutorial in the Python interpreter internals, Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From ejones at uwaterloo.ca Tue Oct 19 23:51:36 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Tue Oct 19 23:51:49 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <20041019214143.GA11458@panix.com> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <41755658.8080806@v.loewis.de> <D8972CD9-2208-11D9-B920-0003938016AE@uwaterloo.ca> <20041019214143.GA11458@panix.com> Message-ID: <0CA6B923-2219-11D9-B920-0003938016AE@uwaterloo.ca> On Oct 19, 2004, at 17:41, Aahz wrote: > Here's an idea that may help implementation slightly, and will almost > certainly increase the likelihood of any patch getting accepted: do the > pool scanning and freeing only on specific call of e.g. gc.free_mem(). > IIUC, you'll still need to change the allocation strategy slightly, but > that's lower risk. This is a very reasonable suggestion, I'll definitely do this. It would also be easy enough to support both exposing this call to Python and doing it automatically based on some debugging macro. Evan Jones -- Evan Jones: http://evanjones.ca/ "Computers are useless. They can only give answers" - Pablo Picasso From alex.nanou at gmail.com Wed Oct 20 00:05:21 2004 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Wed Oct 20 00:05:24 2004 Subject: Fwd: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <36f889220410191502716d027f@mail.gmail.com> References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> <ekjuqyhn.fsf@python.net> <41757AEB.8090807@v.loewis.de> <36f889220410191502716d027f@mail.gmail.com> Message-ID: <36f889220410191505491a490e@mail.gmail.com> Martin wrote: > Also, upgrading from 2.4a3 (with extensions installed) to 2.4b1 > (with no extensions) would have the same effect - there is simply > no support for restoring older versions of the extensions when > uninstalling (and upgrading from 2.4a3 to 2.4b1 is actually the > same as uninstalling 2.4a3 first). this appears to be exactly the case with me, I run both the 2.3 and 2.4 branches on the same box... and I upgraded py2.4a3 to py2.4b1. the funny thing is that when I installed Py2.4a3 some time back all went well. this time the installer restored the last setup options (except the setup dir... I do not use %systemdrive%\python24\) and removed ALL file associations, including for Py2.3.4, and messed up the overall python registry configuration! (which is quite odd....). -- Alex. From alex.nanou at gmail.com Wed Oct 20 00:11:54 2004 From: alex.nanou at gmail.com (Alex A. Naanou) Date: Wed Oct 20 00:11:57 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <36f889220410191505491a490e@mail.gmail.com> References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> <ekjuqyhn.fsf@python.net> <41757AEB.8090807@v.loewis.de> <36f889220410191502716d027f@mail.gmail.com> <36f889220410191505491a490e@mail.gmail.com> Message-ID: <36f889220410191511738aea8e@mail.gmail.com> On Wed, 20 Oct 2004 02:05:21 +0400, Alex A. Naanou <alex.nanou@gmail.com> wrote: > Martin wrote: > > > > Also, upgrading from 2.4a3 (with extensions installed) to 2.4b1 > > (with no extensions) would have the same effect - there is simply > > no support for restoring older versions of the extensions when > > uninstalling (and upgrading from 2.4a3 to 2.4b1 is actually the > > same as uninstalling 2.4a3 first). > this appears to be exactly the case with me, I run both the 2.3 and > 2.4 branches on the same box... and I upgraded py2.4a3 to py2.4b1. > the funny thing is that when I installed Py2.4a3 some time back all > went well. this time the installer restored the last setup options > (except the setup dir... I do not use %systemdrive%\python24\) and > removed ALL file associations, including for Py2.3.4, and messed up > the overall python registry configuration! (which is quite odd....). BTW no changes in the setup configuration were made between py2.4a3 and py2.4b1 (the path, which extensions are to be installed all stayed the same)... -- Alex. From stephen at xemacs.org Wed Oct 20 05:56:39 2004 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed Oct 20 05:56:47 2004 Subject: [Python-Dev] Patches In-Reply-To: <417559D7.5030809@v.loewis.de> (Martin v. =?iso-8859-1?q?L=F6wis's?= message of "Tue, 19 Oct 2004 20:15:51 +0200") References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> Message-ID: <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> >>>>> "Martin" == Martin v L?wis <martin@v.loewis.de> writes: Martin> I have suggested a procedure in the past, which never has Martin> been executed so far. I promise that I will stick to it if Martin> somebody actually executes it. Here it goes: Martin> - For every patch you want to get reviewed, review 10 Martin> (ten) other patches. I think 10 is way too high; I would call it prohibitive for anyone who isn't already primarily a Python hacker. I agree it should be more than 1 for 1 for several reasons, but I'd be willing to bet you get excellent results from 3:1 or even 2:1. Even if you think that risks being a burden on your time, since you've had no takers yet (and it's been a while since the last time I saw your proposal), I'd say that lowering the "price" to 5:1 is a reasonable idea, and not very risky. You can always raise the price again if you get 50 useful reviews by next Tuesday.<wink> -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. From bob at redivi.com Wed Oct 20 06:27:52 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed Oct 20 06:28:27 2004 Subject: [Python-Dev] Patches In-Reply-To: <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> Message-ID: <67EA79E4-2250-11D9-A2F8-000A95BA5446@redivi.com> On Oct 19, 2004, at 23:56, Stephen J. Turnbull wrote: >>>>>> "Martin" == Martin v L?wis <martin@v.loewis.de> writes: > > Martin> I have suggested a procedure in the past, which never has > Martin> been executed so far. I promise that I will stick to it if > Martin> somebody actually executes it. Here it goes: > > Martin> - For every patch you want to get reviewed, review 10 > Martin> (ten) other patches. > > I think 10 is way too high; I would call it prohibitive for anyone who > isn't already primarily a Python hacker. > > I agree it should be more than 1 for 1 for several reasons, but I'd be > willing to bet you get excellent results from 3:1 or even 2:1. Even > if you think that risks being a burden on your time, since you've had > no takers yet (and it's been a while since the last time I saw your > proposal), I'd say that lowering the "price" to 5:1 is a reasonable > idea, and not very risky. You can always raise the price again if you > get 50 useful reviews by next Tuesday.<wink> Speaking of "price", perhaps PSF donations should be factored into getting a patch reviewed? I'm probably only saying this because I've donated, have some patches in the queue that aren't yet reviewed, and don't really have the spare time/energy to review a stockpile of Other People's Patches ;) I think my open-source-python-time is better spent working on the Mac distribution problems (py2app, bdist_pkg, etc.) than reviewing other people's patches... -bob From kbk at shore.net Wed Oct 20 06:36:28 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Oct 20 06:36:36 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410200436.i9K4aSO6026546@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 240 open ( +0) / 2661 closed ( +6) / 2901 total ( +6) Bugs : 772 open ( +6) / 4528 closed (+14) / 5300 total (+20) RFE : 155 open ( +0) / 131 closed ( +0) / 286 total ( +0) New / Reopened Patches ______________________ typo in liblocale.tex (2004-10-15) CLOSED http://python.org/sf/1047577 opened by George Yoshida fileinput.py fixed clobbering behavior and added encodings (2004-10-15) http://python.org/sf/1048075 opened by Chris Connett subprocess documentation (2004-10-16) CLOSED http://python.org/sf/1048341 opened by Fredrik Lundh Bug fixes and cleanup for decimal.py (2004-10-17) http://python.org/sf/1048728 opened by Neal Norwitz adding bool support to xdrlib.py (2004-10-18) http://python.org/sf/1049151 opened by Lars Gust?bel PyOS_InputHook inconsistency on Windows (2004-10-19) http://python.org/sf/1049855 opened by Michiel de Hoon Patches Closed ______________ Bad test for HAVE_UINTPTR_T in PC/pyconfig.h (2004-09-01) http://python.org/sf/1020042 closed by loewis typo in liblocale.tex (2004-10-15) http://python.org/sf/1047577 closed by nnorwitz Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 closed by lemburg subprocess documentation (2004-10-16) http://python.org/sf/1048341 closed by effbot Implementation for PEP 318 (Guido's version) (2004-03-31) http://python.org/sf/926860 closed by nnorwitz rfc822.parsedate returns a tuple (2003-04-01) http://python.org/sf/713599 closed by nnorwitz New / Reopened Bugs ___________________ Turtle.py hangs Idle (2004-10-15) http://python.org/sf/1047540 opened by Taro Ogawa Turtle.py hangs Idle (2004-10-15) CLOSED http://python.org/sf/1047549 opened by Taro Ogawa stat documentation is incorrect (2004-10-16) CLOSED http://python.org/sf/1048234 opened by libber ator Memory leaks? (2004-10-17) http://python.org/sf/1048495 opened by Roman Mamedov SystemError with deque (2004-10-17) CLOSED http://python.org/sf/1048498 opened by George Yoshida Add pythonX.dll to KnownDlls (2004-10-17) CLOSED http://python.org/sf/1048681 opened by Miki Tebeka Doc typo (2004-10-17) CLOSED http://python.org/sf/1048756 opened by Scott David Daniels test_subprocess 2.4b1 fails on FreeBSD 5.2 (2004-10-17) http://python.org/sf/1048808 opened by Shalabh Chaturvedi C-k does not work for curses.textpad.Textbox (2004-10-17) CLOSED http://python.org/sf/1048816 opened by Stefan Heimann Only "Overwrite" mode possible with curses.textpad.Textbox (2004-10-17) http://python.org/sf/1048820 opened by Stefan Heimann unexpected floating point exponent behaviour (2004-10-17) CLOSED http://python.org/sf/1048864 opened by Ruwen Hahn codecs.StreamReader.readlines() broken (2004-10-17) CLOSED http://python.org/sf/1048865 opened by Thomas Waldmann call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 reopened by kquick call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 opened by Kevin Quick shutil.copytree copies stat of files, but not of dirs (2004-10-18) http://python.org/sf/1048878 opened by Thomas Waldmann Registry not updated by 2.4b1 installer (2004-10-17) CLOSED http://python.org/sf/1048900 opened by B Sizer strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 reopened by bcannon os.rmtree error handling was always broken (2004-10-17) http://python.org/sf/1048941 opened by Guido van Rossum Install-time module compilation fails with 2.4b1 (2004-10-18) http://python.org/sf/1049003 opened by Tony Meyer Solaris: EINTR exception in select/socket calls in telnetlib (2004-10-18) http://python.org/sf/1049450 opened by Mike Brauwerman smeared title when installing (2004-10-18) http://python.org/sf/1049615 opened by Jim Jewett test_socket PORT conflict with boa-constructor (2004-10-19) http://python.org/sf/1049816 opened by Simon Dahlbacka zlib module documentation wrong version (2004-10-19) CLOSED http://python.org/sf/1049826 opened by Wolfgang Langner rfc822.parseaddr is broken, breaks sendmail call in smtplib (2004-10-19) http://python.org/sf/1050268 opened by Scott Dossey Bugs Closed ___________ Turtle.py hangs Idle (2004-10-15) http://python.org/sf/1047549 closed by tso Error in the given example (2004-10-14) http://python.org/sf/1046800 closed by facundobatista stat documentation is incorrect (2004-10-16) http://python.org/sf/1048234 closed by jlgijsbers Docs for os, popen2 omit list argument to popen2() et al (2004-03-01) http://python.org/sf/907457 closed by jlgijsbers Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 closed by jlgijsbers distutils.util.get_platform() should include sys.version[:3] (2004-09-27) http://python.org/sf/1035703 closed by etrepum SystemError with deque (2004-10-16) http://python.org/sf/1048498 closed by rhettinger textReader: reading after close is a seg fault (2004-09-30) http://python.org/sf/1038013 closed by nnorwitz Add pythonX.dll to KnownDlls (2004-10-17) http://python.org/sf/1048681 closed by loewis Doc typo (2004-10-17) http://python.org/sf/1048756 closed by nnorwitz C-k does not work for curses.textpad.Textbox (2004-10-17) http://python.org/sf/1048816 closed by akuchling unexpected floating point exponent behaviour (2004-10-17) http://python.org/sf/1048864 closed by tim_one codecs.StreamReader.readlines() broken (2004-10-18) http://python.org/sf/1048865 closed by perky call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 closed by rhettinger Registry not updated by 2.4b1 installer (2004-10-18) http://python.org/sf/1048900 closed by loewis strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 closed by bcannon zlib module documentation wrong version (2004-10-19) http://python.org/sf/1049826 closed by akuchling warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 closed by loewis From python at rcn.com Wed Oct 20 08:20:12 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 20 08:21:50 2004 Subject: [Python-Dev] Patches In-Reply-To: <20041019170126.GA21167@phd.pp.ru> Message-ID: <000401c4b66c$dbb657a0$c302a044@oemcomputer> [Oleg Broytmann] > http://python.org/sf/754022 > This is the biggest and the oldest. It hangs in the tracker since > Python 2.2. In short, it makes the webbrowser.py runs through _tryorder > list of browsers and trie to run every browser until one started > successfully. Currently webbrowser.py tries to run a browser and if it > fails - stops trying. Assigned to Fred Drake. This should probably wait until Py2.5 The functionality, command line launching and successive browser tries, appear useful. Feel free to bug Fred to at least comment on the desirability of the patch. If he gives it a thumbs up, almost anyone else can review and apply this one. > http://python.org/sf/784089 > A program to scan python files and list those require -*- coding -*- > directive. Reviewed by Marc-Andre Lemburg and Andrew Kuchling. I will clean this one up a bit and add it Tools/scripts. > http://python.org/sf/821862 > Makes ftplib.py a bit more RFC959-compliant. The RFC requires the FTP > protocol to be run not over TCP but over telnet. For most people there > is no difference but there is a subtle different for those who use > non-ASCII encodings: chr(255) - a special character in telnet - requires > a special handling. My patch adds a toggle that allows to turn telnet on > (actually, it only doubles chr(255) in command stream, but it is one of > two things that are required for FTP-over-telnet). By default it is off > to preserve backward compatibility. Is there a real need for this? As you mention in the patch, FTP is usually not run over telnet. This module has been around for years and there has been zero user demand for the feature. I suspect this is a waste of bits. > http://python.org/sf/1038388 > The smallest and the simplest of all. 3 lines patch that adds > __main__ to whichdb.py, so one can run it on the command line: > > $ whichdb.py *.db This is harmless. I will apply it for you. No charge, Raymond > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python- > dev/python%40rcn.com From martin at v.loewis.de Wed Oct 20 08:47:58 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 20 08:47:57 2004 Subject: [Python-Dev] Patches In-Reply-To: <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> Message-ID: <41760A1E.80800@v.loewis.de> Stephen J. Turnbull wrote: > I think 10 is way too high; I would call it prohibitive for anyone who > isn't already primarily a Python hacker. Some kind of prohibition is necessary for the process of "pushing" patches. If too many people actually perform the procedure, I may not be able to keep my promise of giving their patches more attention. Notice that this is my own, private procedure - the condition under which *I* will make my volunteer time available. Other core developers may have different procedures, such as giving such requests moderate priority, trying to review the patches right away, or completely ignoring such requests entirely (I believe the majority of people with commit access falls in that category, as only a small fraction of committers commits SF patches regularly). People who don't want to use that procedure don't need to at all. There is apparently a constant backlog of 200..300 patches. Patches that look more important to regular reviewers will get attention, patches that somehow seem doubtful will get skipped over - over and over again. If you want to push one of these patches, and you don't want to use the procedure of guaranteed review that I promise, you can still use the begging procedure that most people come up with. > I agree it should be more than 1 for 1 for several reasons, but I'd be > willing to bet you get excellent results from 3:1 or even 2:1. Even > if you think that risks being a burden on your time, since you've had > no takers yet (and it's been a while since the last time I saw your > proposal), I'd say that lowering the "price" to 5:1 is a reasonable > idea, and not very risky. You can always raise the price again if you > get 50 useful reviews by next Tuesday.<wink> That might be an option indeed. So, Oleg, as a special offer, I will do this at a 5:1 ratio :-) Regards, Martin From python at rcn.com Wed Oct 20 08:47:29 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 20 08:49:07 2004 Subject: [Python-Dev] Patches In-Reply-To: <67EA79E4-2250-11D9-A2F8-000A95BA5446@redivi.com> Message-ID: <000701c4b670$ab45fea0$c302a044@oemcomputer> [Bob] > Speaking of "price", perhaps PSF donations should be factored into > getting a patch reviewed? Since we're talking about volunteered time, any reviewer can use any basis they want for selecting what to address. Martin's price of review is a way of trying to leverage his time. Because he has broad expertise, often he is the only to spend time with otherwise neglected bugs or patches. If others followed his example, we would have less of a backlog. > I'm probably only saying this because I've > donated, have some patches in the queue that aren't yet reviewed Please list them out in an email and I'll look them over. >, and > don't really have the spare time/energy to review a stockpile of Other > People's Patches ;) I think my open-source-python-time is better spent > working on the Mac distribution problems (py2app, bdist_pkg, etc.) than > reviewing other people's patches... Ah, there's the rub. Raymond From martin at v.loewis.de Wed Oct 20 09:02:13 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 20 09:02:12 2004 Subject: [Python-Dev] Patches In-Reply-To: <67EA79E4-2250-11D9-A2F8-000A95BA5446@redivi.com> References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> <67EA79E4-2250-11D9-A2F8-000A95BA5446@redivi.com> Message-ID: <41760D75.3060102@v.loewis.de> Bob Ippolito wrote: > Speaking of "price", perhaps PSF donations should be factored into > getting a patch reviewed? I'm probably only saying this because I've > donated, have some patches in the queue that aren't yet reviewed, and > don't really have the spare time/energy to review a stockpile of Other > People's Patches ;) I think my open-source-python-time is better spent > working on the Mac distribution problems (py2app, bdist_pkg, etc.) than > reviewing other people's patches... That is an entirely valid remark, and using PSF funds for that would be an appropriate spending of the funds. Unfortunately, I don't think it will work: we can't hire anybody off the street to do patch reviews and commits for us, since we need to trust reviewers to do a high-quality job. The people who already review and commit patches regularly probably can't be pursued with money to work more on Python (I know that I wouldn't be pursued, atleast not with the amount of money that the PSF could offer). As for spending your volunteer time differently: most certainly, yes. Any volunteer can spend volunteer time anyway they please. This goes both ways, of course: for submitters and reviewers. If you want to push your changes, an alternative procedure might be to push only selected changes, by contacting somebody you know is primary maintainer of that part of Python - and trust that your own reputation on python-dev will not make that person ignore you. Regards, Martin From phd at mail2.phd.pp.ru Wed Oct 20 09:14:11 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Wed Oct 20 09:14:14 2004 Subject: [Python-Dev] Patches In-Reply-To: <000401c4b66c$dbb657a0$c302a044@oemcomputer> References: <20041019170126.GA21167@phd.pp.ru> <000401c4b66c$dbb657a0$c302a044@oemcomputer> Message-ID: <20041020071411.GB3554@phd.pp.ru> On Wed, Oct 20, 2004 at 02:20:12AM -0400, Raymond Hettinger wrote: > [Oleg Broytmann] > > http://python.org/sf/754022 > > This is the biggest and the oldest. It hangs in the tracker since > > Python 2.2. In short, it makes the webbrowser.py runs through > _tryorder > > list of browsers and trie to run every browser until one started > > successfully. Currently webbrowser.py tries to run a browser and if it > > fails - stops trying. Assigned to Fred Drake. > > This should probably wait until Py2.5 Of course. > > http://python.org/sf/784089 > > A program to scan python files and list those require -*- coding > -*- > > directive. Reviewed by Marc-Andre Lemburg and Andrew Kuchling. > > I will clean this one up a bit and add it Tools/scripts. Thank you! > > http://python.org/sf/821862 > > Is there a real need for this? As you mention in the patch, FTP is > usually not run over telnet. It often runs over telnet - I use ProFTPd which implements telnet without an option to turn it off. The problem is that I cannot use ftplib.py to grab files from my own server :( Most Unix servers implement telnet, but 7-bit ASCII-only filenames hide the fact. Making ftplib.py more RFC-compliant with backward compatibility is a good thing, in my opinion. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From theller at python.net Wed Oct 20 09:22:41 2004 From: theller at python.net (Thomas Heller) Date: Wed Oct 20 09:22:21 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <41757AEB.8090807@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Tue, 19 Oct 2004 22:36:59 +0200") References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> <ekjuqyhn.fsf@python.net> <41757AEB.8090807@v.loewis.de> Message-ID: <6555rgzi.fsf@python.net> Another thing I noticed is that there is no icon shown in the start menu for the 'Python Manuals' entry, unless you have clicked the shortcut at least one time. Thomas From anthony at interlink.com.au Wed Oct 20 10:18:05 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 20 10:19:08 2004 Subject: [Python-Dev] Patches In-Reply-To: <41760D75.3060102@v.loewis.de> References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> <67EA79E4-2250-11D9-A2F8-000A95BA5446@redivi.com> <41760D75.3060102@v.loewis.de> Message-ID: <41761F3D.2070203@interlink.com.au> I'm also willing to look at any outstanding bugs or patches if you think it's particularly urgent or important (to you). Obviously, I'm more likely to be able to act on a patch than a bug with no patch, my time is limited - but if there's a patch you think needs some TLC, email me with the details. This is of course only for me - other committers have their own preferences on how they'd prefer to act. Anthony From stephen at xemacs.org Wed Oct 20 12:49:39 2004 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed Oct 20 12:49:47 2004 Subject: [Python-Dev] Patches In-Reply-To: <41760A1E.80800@v.loewis.de> (Martin v. =?iso-8859-1?q?L=F6wis's?= message of "Wed, 20 Oct 2004 08:47:58 +0200") References: <20041019170126.GA21167@phd.pp.ru> <417559D7.5030809@v.loewis.de> <87is966o08.fsf@tleepslib.sk.tsukuba.ac.jp> <41760A1E.80800@v.loewis.de> Message-ID: <871xft7jgc.fsf@tleepslib.sk.tsukuba.ac.jp> >>>>> "Martin" == Martin v L?wis <martin@v.loewis.de> writes: Martin> If you want to push one of these patches, and you don't Martin> want to use the procedure of guaranteed review that I Martin> promise, you can still use the begging procedure that most Martin> people come up with. Sure. I don't suggest reducing price in order to get you to review more patches. What I like about your procedure is that it encourages people to get involved in helping each other in developing Python. While python-dev is really excellent for mentoring design and implementation, this looks like a good way to get people involved in reviewing. I think that activity is normally under-staffed in most projects. I'd-rather-be-coding-too-ly y'rs, -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software. From fredrik at pythonware.com Wed Oct 20 13:17:18 2004 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed Oct 20 13:15:27 2004 Subject: [Python-Dev] Re: and a question to the SF admins (Re: subprocess-updated popen5module) References: <1f7befae041013201677f808ed@mail.gmail.com> <000001c4b19e$c9f5e120$e841fea9@oemcomputer> Message-ID: <cl5hbr$ama$1@sea.gmane.org> Raymond Hettinger wrote: >> If there are no objections before I wake up again, and nobody else has >> done it, I'll add Peter as a Python developer. > > Peter, your permissions are enabled. Welcome. peter's user (astrand) isn't included in the "Assigned to" list in the bug tracker. can this perhaps be fixed? </F> From fdrake at acm.org Wed Oct 20 13:29:49 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Oct 20 13:29:55 2004 Subject: [Python-Dev] Re: and a question to the SF admins (Re: subprocess-updated popen5module) In-Reply-To: <cl5hbr$ama$1@sea.gmane.org> References: <1f7befae041013201677f808ed@mail.gmail.com> <000001c4b19e$c9f5e120$e841fea9@oemcomputer> <cl5hbr$ama$1@sea.gmane.org> Message-ID: <200410200729.49122.fdrake@acm.org> On Wednesday 20 October 2004 07:17 am, Fredrik Lundh wrote: > peter's user (astrand) isn't included in the "Assigned to" list in the > bug tracker. can this perhaps be fixed? This should be fine now. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From amk at amk.ca Wed Oct 20 13:52:39 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Oct 20 14:05:16 2004 Subject: [Python-Dev] Adding a developer Message-ID: <20041020115239.GA27615@rogue.amk.ca> Could someone please add Lars Gustaebel (SF ID "gustaebel") as a developer? He's responsible for writing and maintaining tarfile.py, but his bugfixes usually sit around in the tracker until someone gets around to looking at them, which is inefficient. Thanks! --amk From carribeiro at gmail.com Wed Oct 20 15:04:40 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Wed Oct 20 15:04:44 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc20410190800766db981@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> Message-ID: <864d370904102006044f0e1f36@mail.gmail.com> On Tue, 19 Oct 2004 08:00:23 -0700, Guido van Rossum <gvanrossum@gmail.com> wrote: > Let me suggest two variations that I have used successfully in my day > job (which is also my night job :). > > 1. For parsing .ini files, I wrote a wrapper around ConfigParser. The > Python-level API looks like this (anonymized and idealized): > > from XXX import MyConfigWrapper, optstr, optint, optbool, optfloat > > class Config(MyConfigWrapper): > poll_time = optfloat("network-parameters", "poll-time") > use_ssl = optbool("network-parameters", "use-ssl") > window_title = optstr("ui-parameters", "window-title") > # etc. The opt* property builders could be substituted by a single opt which took the default value as an argument: opt(section_name, key_name, default_value) The default_value type would be used for conversion on reading & writing: class Config(MyConfigWrapper): poll_time = opt("network-parameters", "poll-time", 1.0) use_ssl = opt("network-parameters", "use-ssl", False) window_title = opt("ui-parameters", "window-title", '') It solves the type cheking and the default value requirements with a single parameter. It's also extensible -- any other type that knows how to construct itself out of a string could be used as an option type. > This allows the Python names for variables to differ from the names > used in the .ini file, and abstracts away the section names completely > from the Python API. This makes it possible to rename variables and > sections in the config file without having to touch the Python code > that uses them. This will save my butt when our marketing team comes > up with the *real* name for our product, since currently the section > names all have the provisional product name in it, and the config file > is considered "user-facing" so references to the old product name have > to be expunged. Also, I don't see much point in having to use longer > references in my Python code -- the total number of config parameters > and their uses are such that I can easily come up with a single unique > name for every option, and yet in the .ini file I'd like to have more > than one section. It works well as far as the option names are unique. In some cases, there are sections with a similar internal structure, and in this case the use of nested structures to represent sections is better. I had one such situation in a communications program of mine a few years ago: each interface (com1:, com2:, etc) had its own section in the config file, with the same parameters being stated for each section. > Note that optstr etc. construct full properties that allow me to set > and delete the parameter values as well, and then ConfigParser can be > told to write back the modified .ini file. This loses the ordering and > comment, but I don't care (although I wish ConfigParser would order > things alphabetically rather than per dictionary hash). I don't think > I have to explain that optint returns a Python int, etc. One of the discussion we've had was about what ordering was desired: arbitrary (hash ordering), alphabethical, or declaration ordering. Alphabethical is better than arbitrary, and current implementations can solve this by sorting the key names before writing the data. I personally like to have my config files ordered as per the original declaration ordering. For example: a webserver most relevant parameter for a quick-and-dirty setup is the port number; if stored in alphabethical ordering, it may be hidden down the list. The use of the original source ordering allows for the developer to write the parameters in the order that makes more sense (from a usability standpoint) for the users that may need to read or edit them, without the need to provide a explicity ordering list. The problem with the list is that it's easy to forget to update it as one adds new parameters, specially if this is done using inheritance. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From astrand at lysator.liu.se Wed Oct 20 15:10:28 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 20 15:10:45 2004 Subject: [Python-Dev] Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 In-Reply-To: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> References: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> Message-ID: <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> Comments? Is it safe or not to assume that by the time the Python has started, only fds 0,1,2 are open? /Peter > I think this is the core of the problem. The test_close_fds > test works like this: > > All file descriptors in the forked child (except 0,1,2) are > closed. Then the Python binary is executed via execvp(). A > small test program is passed to the Python binary via the -c > command line option. If the OS and subprocess module works > correctly, we can be sure of that by the time of the > execve() system call, only file descriptors (0,1,2) are open > (well, the errpipe as well, but let's leave that out for > now). But, by the time the Python binary starts executing > the small program, all sorts of things may have happened. > I'm not really sure we can trust Python not to open files > during startup. For example, if we have a PYTHONSTARTUP > file, that open file will have a file descriptor, perhaps 3. > > On one hand, this bug could indicate a bug in the Python > interpreter itself: perhaps a file descriptor leak. On the > other hand, this test might be a bit too unsafe. > > So probably, this test should be removed. From carribeiro at gmail.com Wed Oct 20 15:21:36 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Wed Oct 20 15:21:38 2004 Subject: [Python-Dev] Patches In-Reply-To: <000401c4b66c$dbb657a0$c302a044@oemcomputer> References: <20041019170126.GA21167@phd.pp.ru> <000401c4b66c$dbb657a0$c302a044@oemcomputer> Message-ID: <864d3709041020062130b4faec@mail.gmail.com> On Wed, 20 Oct 2004 02:20:12 -0400, Raymond Hettinger <python@rcn.com> wrote: > [Oleg Broytmann] > > http://python.org/sf/821862 > > Makes ftplib.py a bit more RFC959-compliant. The RFC requires the > FTP > > protocol to be run not over TCP but over telnet. For most people there > > is no difference but there is a subtle different for those who use > > non-ASCII encodings: chr(255) - a special character in telnet - > requires > > a special handling. My patch adds a toggle that allows to turn telnet > on > > (actually, it only doubles chr(255) in command stream, but it is one > of > > two things that are required for FTP-over-telnet). By default it is > off > > to preserve backward compatibility. > > Is there a real need for this? As you mention in the patch, FTP is > usually not run over telnet. This module has been around for years and > there has been zero user demand for the feature. I suspect this is a > waste of bits. It reminds Postel's Law: "Be conservative in what you do, be liberal in what you accept from others." [Section 2.10 - RFC 793] So... if the RFC states that it should run over telnet, then standard implementations should do it. But, according to Martin's proposed rules -- which I entirely agree with -- my vote still doesn't count, as I haven't reviewed any patch yet; so please take it only as a general opinion on the subject. low demand nonetheless. Priority may be low, but its still a relevant patch, IMHO -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From dw-python.org at botanicus.net Wed Oct 20 16:18:13 2004 From: dw-python.org at botanicus.net (David Wilson) Date: Wed Oct 20 16:18:18 2004 Subject: [Python-Dev] What are the goals for ConfigParser 3000? Message-ID: <20041020141813.GA4872@thailand.botanicus.net> On Sun, Oct 17, 2004 at 03:38:11PM -0700, Michael Chermside wrote: > My apologies, my previous email was intended to go to python-list, not > python-dev. I didn't intend to bother this list with it until it was > at a point where I thought it was a contender for inclusion. Sometimes > I really wish there was an "Unsend" option for email. I think it raised a point though - there were a few wildly different ideas about what ConfigParser's replacement should be. Would it be wise to discuss first what ConfigParser is *supposed* to be, and where it fits in peoples' applications? If this hits a chord with you, pump your reply button and discuss! If you look at the current ConfigParser, you might find the following: - It is a simple storage method for text data that categorises sets of keys with sets of values. - Its backend format is simple and fast for other tools to parse (even for on-request applications, like web scripts). - For almost any complex data, it becomes immediately less useful, but with a small amount of external wrapping, can also be used for more complex configuration structures. - It is *not* a means for declaring objects or doing sums: that is what the Python syntax itself is for. My problems with the current ConfigParser are as follows: - The interface is unfriendly and error-prone. Examples: - read(): raises no exception if a file is missing. This is against Python culture, and the unsurprising name causes IMHO surprising behaviour, especially for the standard library. - readfp(): 'more than one way to do it'. File-like objects are such a fundamental part of Python that I think read() should go away entirely. This would mean that a programmer is explicitly reminded of the errors that can occur when opening his configuration file as he explicitly types the file() factory expression himself. - Again, for getting values there are a few methods which essentially repeat themselves - getint(), getfloat(), etc. What is wrong with int(obj.get(...))? - get() shares the same method name as the infinitely popular dict.get() method, yet does not support a <default> argument. - String interpolation! There is a friendly string syntax as a standard library feature these days, and less friendly interpolation has always existed in the language itself. This should go, and be made more explicit for the programmer. In my opinion we should differenciate at an early stage between complex processing tasks (Python itself) and storing random snippets of configuration data. In short, removing the functions that really shouldn't be there and modernizing the code, would be a good starting place for a replacement. Only then we can start talking about ConfigParser-NG. Thanks, David. -- Now that my house has burned down I have a much better view of the moon. From python at rcn.com Wed Oct 20 17:14:50 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 20 17:16:28 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <20041020115239.GA27615@rogue.amk.ca> Message-ID: <001f01c4b6b7$8b709c60$19ac2c81@oemcomputer> > Could someone please add Lars Gustaebel (SF ID "gustaebel") as a > developer? He's responsible for writing and maintaining tarfile.py, > but his bugfixes usually sit around in the tracker until someone gets > around to looking at them, which is inefficient. Have him assign the patches to me. I'm not sure it is a good idea to keep adding developers who do only a handful of checkins and disappear. This is doubly true if they are not active on python-dev, have not asked for access, and have not made some commitment to stay around for maintenance. Raymond From gvanrossum at gmail.com Wed Oct 20 17:21:06 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed Oct 20 17:21:11 2004 Subject: [Python-Dev] Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 In-Reply-To: <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> References: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> Message-ID: <ca471dc2041020082174f55fd8@mail.gmail.com> I'm missing some context here, but the opening of STARTUP and other examples is irrelevant, as long as that code properly closes its fds. Of course, you can't trust shells not to have other fds open for internal purposes; that's a feature of fd inheritance. On Wed, 20 Oct 2004 15:10:28 +0200 (MEST), Peter Astrand <astrand@lysator.liu.se> wrote: > > Comments? Is it safe or not to assume that by the time the Python has > started, only fds 0,1,2 are open? > > /Peter > > > I think this is the core of the problem. The test_close_fds > > test works like this: > > > > All file descriptors in the forked child (except 0,1,2) are > > closed. Then the Python binary is executed via execvp(). A > > small test program is passed to the Python binary via the -c > > command line option. If the OS and subprocess module works > > correctly, we can be sure of that by the time of the > > execve() system call, only file descriptors (0,1,2) are open > > (well, the errpipe as well, but let's leave that out for > > now). But, by the time the Python binary starts executing > > the small program, all sorts of things may have happened. > > I'm not really sure we can trust Python not to open files > > during startup. For example, if we have a PYTHONSTARTUP > > file, that open file will have a file descriptor, perhaps 3. > > > > On one hand, this bug could indicate a bug in the Python > > interpreter itself: perhaps a file descriptor leak. On the > > other hand, this test might be a bit too unsafe. > > > > So probably, this test should be removed. > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Wed Oct 20 17:25:34 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed Oct 20 17:25:37 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <864d370904102006044f0e1f36@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> Message-ID: <ca471dc204102008251a91e30e@mail.gmail.com> > The opt* property builders could be substituted by a single opt which > took the default value as an argument: > > opt(section_name, key_name, default_value) What if I want the default to be None? Sometimes absence of a value is useful info. If you want a single factory function (not sure why) then maybe the type should be passed in (perhaps optional, only if no default is given). > It works well as far as the option names are unique. In some cases, > there are sections with a similar internal structure, and in this case > the use of nested structures to represent sections is better. "Better" is subjective. With my approach, you can give them different prefixes in the Python API independent from their names as seen by the user, and you only have to do this when there are actual conflicts (which are rare except when certain styles are used). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From amk at amk.ca Wed Oct 20 17:42:19 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Oct 20 17:44:46 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <001f01c4b6b7$8b709c60$19ac2c81@oemcomputer> References: <20041020115239.GA27615@rogue.amk.ca> <001f01c4b6b7$8b709c60$19ac2c81@oemcomputer> Message-ID: <20041020154219.GA28582@rogue.amk.ca> On Wed, Oct 20, 2004 at 11:14:50AM -0400, Raymond Hettinger wrote: > I'm not sure it is a good idea to keep adding developers who do only a > handful of checkins and disappear. This is doubly true if they are not Agreed, but based on his track record it seems unlikely that Lars is going to vanish; he contributed the module on 2003/01/05, still maintains the code separately at http://www.gustaebel.de/lars/tarfile/, and has contributed 9 bugfixes since then (compared to 7 bugfixes from others). If we're adding Peter, who'll be maintaining a single module, then adding Lars seems equally reasonable. I like the approach of adding people who will maintain a particular module or package. Long-term, I think it's more sustainable to have a larger number of developers, each person responsible for a small number of modules, than a small number of developers struggling to keep up with everything. --amk From python at rcn.com Wed Oct 20 17:48:30 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 20 17:50:24 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <20041020154219.GA28582@rogue.amk.ca> Message-ID: <002701c4b6bc$3fcfb3e0$19ac2c81@oemcomputer> > Long-term, I think it's more sustainable to have a > larger number of developers, each person responsible for a small > number of modules, than a small number of developers struggling to > keep up with everything. I'll be happy to apply the patches of someone maintaining their own module. Raymond From tim.peters at gmail.com Wed Oct 20 17:58:03 2004 From: tim.peters at gmail.com (Tim Peters) Date: Wed Oct 20 17:58:04 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <20041020154219.GA28582@rogue.amk.ca> References: <20041020115239.GA27615@rogue.amk.ca> <001f01c4b6b7$8b709c60$19ac2c81@oemcomputer> <20041020154219.GA28582@rogue.amk.ca> Message-ID: <1f7befae041020085838934c74@mail.gmail.com> [A.M. Kuchling] ... > If we're adding Peter, who'll be maintaining a single module, then adding Lars > seems equally reasonable. The key difference (to my mind) is that Peter asked for it, but so far Lars has not. If Lars asks, that proves a number of things to me: he knows that we intend to assign tarfile bug reports to him; he agrees that it's appropriate to do so; he confirms that he's comfortable driving CVS; and, last but not least, he knows how to reach python-dev <0.1 wink>. From lists at ghaering.de Wed Oct 20 18:29:05 2004 From: lists at ghaering.de (Gerhard Haering) Date: Wed Oct 20 18:30:31 2004 Subject: [Python-Dev] SQLite module for Python 2.5 Message-ID: <20041020162905.GA9843@mylene.ghaering.de> Hi python-dev-elopers, Last December, we had a short thread discussing the integration of PySQLite into Python 2.4. At the time, I was against inclusion, because I thought PySQLite was not ripe for it, mostly because I thought the API was not stable. Now, I have started writing a new PySQLite module, which has the following key features: - Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step() etc. This way, it is possible to use prepared statements, and for large resultsets, it requires less memory, because the whole resultset isn't fetched into memory at once any longer. - Completely incompatible with the SQLite 0.x/1.x API: I'm free to create a much better API now. - "In the face of ambiguity, refuse the temptation to guess." - PySQLite 1.x tries to "guess" which Python type to convert to. It's pretty good at it, because it queries the column type information. This works for, I'd say 90 % of all cases at least. But as soon as you use anything fancy like functions, aggregates or expressions in SQL, the _typeless_ nature of SQLite breaks through and it will tell us nothing about the declared column type (of course, because the data is not coming from a database column). So I decided to change the default behaviour and make PySQLite typeless by default, too. Everything will be returned as a Unicode string (the default might be user-configurable per connection). Unless, unless of course the user explicitly activates the "guess-mode" ;-) But to do so, she must read the docs then she will be aware of the fact that it only works in 90 % of all cases. So why am I bothering you about this? I think that a simple embedded relational database would be a good thing to have in Python by default. And as Python 2.5 won't happen anytime soon, there's plenty of time for developing it, getting it stable, and integrating it. Especially those of you that have used PySQLite in the past, do you have any suggestions that would make the rewrite a better candidate for inclusion into Python? One problem I see is that even the new PySQLite will grow and try to wrap much of the SQLite API that are not directly related to the DB-API. If such a thing is too complicated/big for the standard library, then maybe it would be better to produce a much simpler PySQLite, especially for the Python standard library that leaves all the fancy stuff out. My codename would be "embsql". So, what would you like to see? "import sqlite", "import embsql", or "pypi.install('pysqlite')" ? -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041020/69edba0d/attachment.pgp From carribeiro at gmail.com Wed Oct 20 18:40:35 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Wed Oct 20 18:40:38 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc204102008251a91e30e@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> Message-ID: <864d3709041020094016d2affa@mail.gmail.com> On Wed, 20 Oct 2004 08:25:34 -0700, Guido van Rossum <gvanrossum@gmail.com> wrote: > > The opt* property builders could be substituted by a single opt which > > took the default value as an argument: > > > > opt(section_name, key_name, default_value) > > What if I want the default to be None? Sometimes absence of a value > is useful info. If you want a single factory function (not sure why) > then maybe the type should be passed in (perhaps optional, only if no > default is given). I like the optional type idea: opt(section_name, key_name, default_value, type) So it can be written this way: opt('section', 'key', None, StringType) ...but -- to answer your question -- the point here isn't really the 'singleness' of the factory function, but the fact that it is type-independent, which (in principle) would allow it to be extended to handle arbitrary types by delegating some functionality to the types themselves. IMHO, it's just a different way to design it: instead of writing a specialized property handler for each supported type, use a standard interface provided by types themselves to handle the conversion to & from the string representation that is used in the config file. Unfortunately, this generalization is not as practical as I thought it would be. In my implementation, I used standard Python type constructors to convert string values to the desired object type. It works for str, int and float [see note #1], which solved nearly all cases in my own experience. However, this trick doesn't work for bool(), which is an important type. To solve it, I had to write my own adapt() method, which just handles str, int and float using the builtin types, and treats bool() as a particular case. User defined types have to expose special conversion methods (as they should anyway). As it is, it's a general solution, but it ended up being more confusing and convoluted that I would like it to be. > > It works well as far as the option names are unique. In some cases, > > there are sections with a similar internal structure, and in this case > > the use of nested structures to represent sections is better. > > "Better" is subjective. With my approach, you can give them different > prefixes in the Python API independent from their names as seen by the > user, and you only have to do this when there are actual conflicts > (which are rare except when certain styles are used). Agreed. It depends a lot on the particular application, and to some extent, on personal taste. My own bias come from some of the applications that I have written, where I feared that the use of prefixes would end up cluttering the namespace. Besides the above mentioned communications software example (where I had several communication ports, each one stored into a section), another real scenario is the configuration of user interface parameters for several UI elements -- for example, font selection and background colors for specific forms. The same parameters ('font', 'font-color', 'background-color', etc.) are repeated for several sections. But surely, my own experience is limited, and I can't speak for other people on this point. -------- [1]As an apart: I was *really* surprised when I first saw that this code worked: >>> IntType('12') 12 >>> FloatType('-1e+12') -1000000000000.0 The following test also worked, but I expected it to convert the value using an octal representation, but it didn't: >>> IntType('012') 12 And finally, the bool test: >>> BooleanType('False') True Which obviously don't work because the 'False' string converts to True, as any other non-empty string do :-P. But it seemed a little bit weird when a I first saw it, specially when compared to FloatType and IntType. curiousness-killed-the-cat'ly-yours, -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From mal at egenix.com Wed Oct 20 19:05:05 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Oct 20 19:05:08 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041020162905.GA9843@mylene.ghaering.de> References: <20041020162905.GA9843@mylene.ghaering.de> Message-ID: <41769AC1.4020603@egenix.com> Gerhard Haering wrote: > Hi python-dev-elopers, > > Last December, we had a short thread discussing the integration of > PySQLite into Python 2.4. At the time, I was against inclusion, > because I thought PySQLite was not ripe for it, mostly because I > thought the API was not stable. > > [...] > > I think that a simple embedded relational database would be a good > thing to have in Python by default. And as Python 2.5 won't happen > anytime soon, there's plenty of time for developing it, getting it > stable, and integrating it. SQLite is a gem and PySQLite works great, but I don't see why we should start adding third-party tools of this size (>38k LOC C code) to the standard Python distribution. Perhaps you we should consider adding only the Python interface and then ship a DLL with the Windows installer like we do for expat and the Sleepycat DBM ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 20 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From carribeiro at gmail.com Wed Oct 20 19:08:32 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Wed Oct 20 19:08:35 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041020162905.GA9843@mylene.ghaering.de> References: <20041020162905.GA9843@mylene.ghaering.de> Message-ID: <864d370904102010083dba2a40@mail.gmail.com> On Wed, 20 Oct 2004 18:29:05 +0200, Gerhard Haering <lists@ghaering.de> wrote: > So, what would you like to see? "import sqlite", "import embsql", or > "pypi.install('pysqlite')" ? Warning: I'm still pretty inexperienced in the python-dev issues. I'm still figuring out how to behave, and how to become a good member of the community. So what follows is a personal opinion, and I hope not to be breaking any community laws by doing so. I'm a enthusiastic user of pysqlite, which I found to solve almost all my development needs without the administrative burden of 'real' rdbms. No only that, but it also has proven more than good enough for actual deployment in several situations. Quite a surprise, in fact. I sincerely believe that a standard rdbms in the standard Python library would greatly improve Python's acceptance and usability in several common scenarios. I also believe that pysqlite has what it takes to be included as such standard module. I also believe that this discussion should be held at the main Python list. There are many active Python developers that don't follow the python-dev lists that may have some interest on this topic. > One problem I see is that even the new PySQLite will grow and try to > wrap much of the SQLite API that are not directly related to the > DB-API. If such a thing is too complicated/big for the standard > library, then maybe it would be better to produce a much simpler > PySQLite, especially for the Python standard library that leaves all > the fancy stuff out. My codename would be "embsql". This is a big issue. If the sqlite library supports a bigger and richer API _as part of the standard Python library_, then everyone else (Python's end users and developers) will naturally expect that all other rdbms will support the same API. Have you posed this question to the DB-SIG people? -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From barry at python.org Wed Oct 20 19:13:52 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 20 19:14:00 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041020162905.GA9843@mylene.ghaering.de> References: <20041020162905.GA9843@mylene.ghaering.de> Message-ID: <1098292432.17343.41.camel@geddy.wooz.org> Background: I've had a lot of occasions to use SQLite and PySQLite recently and I've been so happy with the results that it is a very serious contender for the default storage in Mailman 3. This would replace BSDDB which the current code base uses. In fact I think the bsddb module support in Python's stdlib makes for a good comparison with Gerhard's proposal. On Wed, 2004-10-20 at 12:29, Gerhard Haering wrote: > - Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step() > etc. This way, it is possible to use prepared statements, and for > large resultsets, it requires less memory, because the whole > resultset isn't fetched into memory at once any longer. Cool. BTW, this makes me wonder whether it might be time to work on a DBAPI 3 that takes advantages of some of the more recent developments in Python. Apologies if such an effort is already underway and I haven't seen it. > - Completely incompatible with the SQLite 0.x/1.x API: I'm free to > create a much better API now. It's both fun and scary to be able to "do it right this time" :) > - "In the face of ambiguity, refuse the temptation to guess." - > PySQLite 1.x tries to "guess" which Python type to convert to. It's > pretty good at it, because it queries the column type information. > This works for, I'd say 90 % of all cases at least. But as soon as > you use anything fancy like functions, aggregates or expressions in > SQL, the _typeless_ nature of SQLite breaks through and it will tell > us nothing about the declared column type (of course, because the > data is not coming from a database column). I'd also like to see something in the middle. For example, I know what the intended types of my columns are, so I'd like to be able to set up a mapping of converters and pass that to PySQLite. I'd get the best of both worlds then -- explicit type conversion (because it always uses my mappings) and automatic type conversion (because I'll get the target type back from PySQLite directly without having to apply conversions myself everywhere). > I think that a simple embedded relational database would be a good > thing to have in Python by default. And as Python 2.5 won't happen > anytime soon, there's plenty of time for developing it, getting it > stable, and integrating it. I'm for it. Again, because we have batteries-included support for BSDDB, I see no reason why we can't also have batteries-included support for SQLite. Both are embedded databases, so if you've got the libraries and headers laying around, it should be a snap to configure and build the module. > Especially those of you that have used PySQLite in the past, do you > have any suggestions that would make the rewrite a better candidate > for inclusion into Python? A few. Having used the Python bindings for MySQL also, there are a few things in its interface that I'd like to see in PySQLite (but please correct me if they're there but I missed them!). The converter idea comes from mysql-python. I also liked their use of a 'how' argument to the fetch methods, which allowed me to retrieve the row as a dictionary. Very handy! I'm sure there are other nice features that I've forgotten about. > One problem I see is that even the new PySQLite will grow and try to > wrap much of the SQLite API that are not directly related to the > DB-API. That's a /good/ thing, not a problem! :) > If such a thing is too complicated/big for the standard > library, then maybe it would be better to produce a much simpler > PySQLite, especially for the Python standard library that leaves all > the fancy stuff out. My codename would be "embsql". > > So, what would you like to see? "import sqlite", "import embsql", or > "pypi.install('pysqlite')" ? Personally, I'd like to see both a DB-API interface and a full-blown wrapping of the SQLite API. From what I can tell, it would be much smaller than the BSDDB wrapper, so it's potential size or complication doesn't bother me. i-might-even-be-a-guinea-pig-for-ya-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041020/a83299fe/attachment.pgp From barry at python.org Wed Oct 20 19:16:25 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 20 19:16:29 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <41769AC1.4020603@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> Message-ID: <1098292585.17350.44.camel@geddy.wooz.org> On Wed, 2004-10-20 at 13:05, M.-A. Lemburg wrote: > SQLite is a gem and PySQLite works great, but I don't see why we > should start adding third-party tools of this size (>38k LOC C code) > to the standard Python distribution. > > Perhaps you we should consider adding only the Python interface > and then ship a DLL with the Windows installer like we do for > expat and the Sleepycat DBM ?! Oh, maybe I misread Gerhard's post, but I definitely didn't expect him to do anything other than this! I'd be -1 on adding the SQLite code to Python, but +1 on shipping the wrapper module with the source code, and the DLL on Windows. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041020/1c7975da/attachment.pgp From bob at redivi.com Wed Oct 20 19:18:01 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed Oct 20 19:18:44 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <41769AC1.4020603@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> Message-ID: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> On Oct 20, 2004, at 13:05, M.-A. Lemburg wrote: > Gerhard Haering wrote: >> Hi python-dev-elopers, >> Last December, we had a short thread discussing the integration of >> PySQLite into Python 2.4. At the time, I was against inclusion, >> because I thought PySQLite was not ripe for it, mostly because I >> thought the API was not stable. >> [...] > > >> I think that a simple embedded relational database would be a good >> thing to have in Python by default. And as Python 2.5 won't happen >> anytime soon, there's plenty of time for developing it, getting it >> stable, and integrating it. > > SQLite is a gem and PySQLite works great, but I don't see why we > should start adding third-party tools of this size (>38k LOC C code) > to the standard Python distribution. I don't think he ever said that the SQLite source tree should go into Python. By default can mean that Python builds a SQLite wrapper if SQLite is available, just like it does for bsddb, readline, etc. Binary builds for Win32 and Mac should of course ship with a copy of SQLite for use by the PySQLite extension (w/ a dll or just statically linked in). Heck, Mac OS X 10.4 will be shipping with SQLite anyway <http://www.apple.com/macosx/tiger/unix.html>! > Perhaps you we should consider adding only the Python interface > and then ship a DLL with the Windows installer like we do for > expat and the Sleepycat DBM ?! Python includes expat, doesn't it? -bob From barry at python.org Wed Oct 20 19:20:25 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 20 19:20:30 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <864d370904102010083dba2a40@mail.gmail.com> References: <20041020162905.GA9843@mylene.ghaering.de> <864d370904102010083dba2a40@mail.gmail.com> Message-ID: <1098292825.17345.49.camel@geddy.wooz.org> On Wed, 2004-10-20 at 13:08, Carlos Ribeiro wrote: > This is a big issue. If the sqlite library supports a bigger and > richer API _as part of the standard Python library_, then everyone > else (Python's end users and developers) will naturally expect that > all other rdbms will support the same API. I don't agree. Every one of the FOSS RDBMs has a different C API so I wouldn't expect the Python module wrapping the native APIs to be anything but a more-or-less straight exposure of that -- with some liberties taken to Pythonify the APIs where appropriate, e.g. PyBSDDB has a done a very good job there. The common API is the DB-API, although it would be nice if there were more commonality amongst the various implementations. I do agree that further discussion probably belongs on other lists. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041020/9fdbbf3f/attachment.pgp From niemeyer at conectiva.com Wed Oct 20 19:22:24 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed Oct 20 19:22:42 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041020162905.GA9843@mylene.ghaering.de> References: <20041020162905.GA9843@mylene.ghaering.de> Message-ID: <20041020172223.GA6073@burma.localdomain> Hello Gerhard, > - Uses iterator-style SQLite 3.x API: sqlite3_compile, sqlite3_step() > etc. This way, it is possible to use prepared statements, and for > large resultsets, it requires less memory, because the whole > resultset isn't fetched into memory at once any longer. I'm anxiously waiting for the 3.x-based version! [...] > - "In the face of ambiguity, refuse the temptation to guess." - [...] > So I decided to change the default behaviour and make PySQLite > typeless by default, too. Everything will be returned as a Unicode > string (the default might be user-configurable per connection). I'm wondering if it would be possible to introduce a mechanism allowing one to *explicitly* set column conversion functions at query time. This would avoid having to manually convert rows on every access. [...] > So, what would you like to see? "import sqlite", "import embsql", or > "pypi.install('pysqlite')" ? Even though I'm a big fan of sqlite and pysqlite, my personal feeling is that SQL databases in general are better delivered as add-on modules. -- Gustavo Niemeyer http://niemeyer.net From mal at egenix.com Wed Oct 20 19:48:49 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Oct 20 19:48:52 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> Message-ID: <4176A501.1090601@egenix.com> Bob Ippolito wrote: > > On Oct 20, 2004, at 13:05, M.-A. Lemburg wrote: > >> Gerhard Haering wrote: >> >>> Hi python-dev-elopers, >>> Last December, we had a short thread discussing the integration of >>> PySQLite into Python 2.4. At the time, I was against inclusion, >>> because I thought PySQLite was not ripe for it, mostly because I >>> thought the API was not stable. >>> [...] >> >> > >> >>> I think that a simple embedded relational database would be a good >>> thing to have in Python by default. And as Python 2.5 won't happen >>> anytime soon, there's plenty of time for developing it, getting it >>> stable, and integrating it. >> >> >> SQLite is a gem and PySQLite works great, but I don't see why we >> should start adding third-party tools of this size (>38k LOC C code) >> to the standard Python distribution. > > I don't think he ever said that the SQLite source tree should go into > Python. By default can mean that Python builds a SQLite wrapper if > SQLite is available, just like it does for bsddb, readline, etc. Binary > builds for Win32 and Mac should of course ship with a copy of SQLite for > use by the PySQLite extension (w/ a dll or just statically linked in). > Heck, Mac OS X 10.4 will be shipping with SQLite anyway > <http://www.apple.com/macosx/tiger/unix.html>! If that's what Gerhard meant, no objections. >> Perhaps you we should consider adding only the Python interface >> and then ship a DLL with the Windows installer like we do for >> expat and the Sleepycat DBM ?! > > Python includes expat, doesn't it? True, but it didn't use to be included. The fact that our Fred Drake maintains it made the difference, I guess. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 20 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Wed Oct 20 19:55:01 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 20 19:55:02 2004 Subject: [Python-Dev] Python 2.4b1 on win32 installer bug? In-Reply-To: <6555rgzi.fsf@python.net> References: <36f8892204101908321d8d0e9d@mail.gmail.com> <417568E6.9040009@v.loewis.de> <ekjuqyhn.fsf@python.net> <41757AEB.8090807@v.loewis.de> <6555rgzi.fsf@python.net> Message-ID: <4176A675.3030305@v.loewis.de> Thomas Heller wrote: > Another thing I noticed is that there is no icon shown in the start menu > for the 'Python Manuals' entry, unless you have clicked the shortcut at > least one time. I know, and I don't know what to do about that. Including the icon for .chm files within the MSI package might help, but I'm unsure whether this is legally possible. I'm still looking for a good solution, but likely, this problem will also show up in 2.4. Regards, Martin From martin at v.loewis.de Wed Oct 20 20:13:28 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 20 20:13:27 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <20041020115239.GA27615@rogue.amk.ca> References: <20041020115239.GA27615@rogue.amk.ca> Message-ID: <4176AAC8.6030606@v.loewis.de> A.M. Kuchling wrote: > Could someone please add Lars Gustaebel (SF ID "gustaebel") as a > developer? He's responsible for writing and maintaining tarfile.py, > but his bugfixes usually sit around in the tracker until someone gets > around to looking at them, which is inefficient. In addition to what Tim says (that Lars also should ask for it), I see no real problem with the status quo. I have applied a number of tarfile patches in the past, and keep an eye on any new patches. Of the pending patches, I see no real problem if they are not applied - even if they don't get into 2.4, I think adding them to 2.4.1 might still be appropriate. tarfile is inherently a tricky issue (with the tar file format being so difficult), and I believe Lars appreciates that somebody reviews his changes before applying them. Regards, Martin From carribeiro at gmail.com Wed Oct 20 20:14:51 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Wed Oct 20 20:14:53 2004 Subject: [Python-Dev] What are the goals for ConfigParser 3000? In-Reply-To: <20041020141813.GA4872@thailand.botanicus.net> References: <20041020141813.GA4872@thailand.botanicus.net> Message-ID: <864d37090410201114389b447@mail.gmail.com> On Wed, 20 Oct 2004 15:18:13 +0100, David Wilson <dw-python.org@botanicus.net> wrote: > I think it raised a point though - there were a few wildly different > ideas about what ConfigParser's replacement should be. Would it be wise > to discuss first what ConfigParser is *supposed* to be, and where it > fits in peoples' applications? > > If this hits a chord with you, pump your reply button and discuss! Skip Montanaro has proposed the creation of a ConfigParserShootout page on the Wiki. Shouldn't this discussion be moved there? BTW I have my own ideas on the topic... and I'm willing to put them there on the Wiki, if that's the way to go. And as far as the rest of David's comments -- I agree, let's keep it simple, and move all data processing stuff out of the library. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From astrand at lysator.liu.se Wed Oct 20 20:19:58 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Wed Oct 20 20:20:13 2004 Subject: [Python-Dev] Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 In-Reply-To: <20041020150759.GD968@logilab.fr> References: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> <20041020150759.GD968@logilab.fr> Message-ID: <Pine.GSO.4.51L2.0410202017580.28328@adriana.lysator.liu.se> On Wed, 20 Oct 2004, Ludovic Aubry wrote: > I would say it is not safe: > > f=file("hello") > from os import fork, execl > > if fork()!=0: > execl("python", "python" ) > > This leaves fd=3 open on linux Yes, but with the subprocess module and close_fds=True, subprocess explicitly closes all fds except 0,1,2, between fork and exec. I'm talking about the problem with the Python interpreter opening *new* files during startup. /Peter ?strand <astrand@lysator.liu.se> From fdrake at acm.org Wed Oct 20 21:00:17 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Oct 20 21:00:30 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <4176A501.1090601@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <4176A501.1090601@egenix.com> Message-ID: <200410201500.17944.fdrake@acm.org> On Wednesday 20 October 2004 01:48 pm, M.-A. Lemburg wrote: > True, but it didn't use to be included. The fact that our Fred Drake > maintains it made the difference, I guess. That might have something to do with it, but that's certainly not the only thing, and not reason enough. The biggest reason to include at least basic XML support in the standard library is that new XML file formats are being used for supplemental data by a variety of applications, and it's reasonable for many of them to be handled behind the scenes by libraries that don't expose an XML-related API. If the application using the library itself doesn't require XML support, it really shouldn't need to worry about the fact that one of the libraries does. Making an XML parser and some basic APIs available in the Python standard library (SAX and DOM) works out to make life easier for people putting together applications that may end up touching XML indirectly (via some other library that hides it). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From ianb at colorstudy.com Wed Oct 20 21:04:07 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Wed Oct 20 21:04:29 2004 Subject: [Python-Dev] What are the goals for ConfigParser 3000? In-Reply-To: <864d37090410201114389b447@mail.gmail.com> References: <20041020141813.GA4872@thailand.botanicus.net> <864d37090410201114389b447@mail.gmail.com> Message-ID: <4176B6A7.9060704@colorstudy.com> Carlos Ribeiro wrote: > Skip Montanaro has proposed the creation of a ConfigParserShootout > page on the Wiki. Shouldn't this discussion be moved there? I've created a ConfigParserShootout page: http://www.python.org/moin/ConfigParserShootout I seeded it with some feature ideas, and links to these discussions. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From skip at pobox.com Thu Oct 21 02:14:16 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 21 02:14:28 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> Message-ID: <16758.65368.937378.577791@montanaro.dyndns.org> Bob> By default can mean that Python builds a SQLite wrapper if SQLite Bob> is available, just like it does for bsddb, readline, etc. Then why not MySQLdb, psycopg and sybase-python also? No slight intended against PySQLite, but those other wrapper modules have been around quite a bit longer I think. Skip From janssen at parc.com Thu Oct 21 02:25:31 2004 From: janssen at parc.com (Bill Janssen) Date: Thu Oct 21 02:25:56 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: Your message of "Wed, 20 Oct 2004 12:00:17 PDT." <200410201500.17944.fdrake@acm.org> Message-ID: <04Oct20.172535pdt."58617"@synergy1.parc.xerox.com> > Making an XML parser and some basic APIs available in the Python standard > library (SAX and DOM) works out to make life easier for people putting > together applications that may end up touching XML indirectly (via some other > library that hides it). Yes, yes, yes! And it should support XML 1.1 -- apparently the currently available Python tools don't (I'm told). Bill From allison at sumeru.stanford.EDU Thu Oct 21 02:35:02 2004 From: allison at sumeru.stanford.EDU (Dennis Allison) Date: Thu Oct 21 02:35:17 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16758.65368.937378.577791@montanaro.dyndns.org> Message-ID: <Pine.LNX.4.10.10410201734270.1750-100000@sumeru.stanford.EDU> +1 from my point-of-view. The autobuild for wraopers is one of the very nice features of Python. On Wed, 20 Oct 2004, Skip Montanaro wrote: > > Bob> By default can mean that Python builds a SQLite wrapper if SQLite > Bob> is available, just like it does for bsddb, readline, etc. > > Then why not MySQLdb, psycopg and sybase-python also? No slight intended > against PySQLite, but those other wrapper modules have been around quite a > bit longer I think. > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/allison%40sumeru.stanford.edu > From fdrake at acm.org Thu Oct 21 02:38:53 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Oct 21 02:39:05 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <04Oct20.172535pdt."58617"@synergy1.parc.xerox.com> References: <04Oct20.172535pdt."58617"@synergy1.parc.xerox.com> Message-ID: <200410202038.53369.fdrake@acm.org> On Wednesday 20 October 2004 08:25 pm, Bill Janssen wrote: > Yes, yes, yes! And it should support XML 1.1 -- apparently the > currently available Python tools don't (I'm told). That's correct; no one has had time to update Expat to support the new specification. Patches welcome. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From pje at telecommunity.com Thu Oct 21 02:52:35 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Oct 21 02:52:16 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16758.65368.937378.577791@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> Message-ID: <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> At 07:14 PM 10/20/04 -0500, Skip Montanaro wrote: > Bob> By default can mean that Python builds a SQLite wrapper if SQLite > Bob> is available, just like it does for bsddb, readline, etc. > >Then why not MySQLdb, psycopg and sybase-python also? No slight intended >against PySQLite, but those other wrapper modules have been around quite a >bit longer I think. Well, one difference is that none of the databases you just listed are embeddable. There has to be a separate database server process. SQLite, like other "database" modules in the stdlib, just stores data in a disk file. From skip at pobox.com Thu Oct 21 05:53:30 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 21 05:53:49 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> Message-ID: <16759.12986.759165.964094@montanaro.dyndns.org> Bob> By default can mean that Python builds a SQLite wrapper if SQLite Bob> is available, just like it does for bsddb, readline, etc. >> Then why not MySQLdb, psycopg and sybase-python also? No slight >> intended against PySQLite, but those other wrapper modules have been >> around quite a bit longer I think. Phillip> Well, one difference is that none of the databases you just Phillip> listed are embeddable. There has to be a separate database Phillip> server process. SQLite, like other "database" modules in the Phillip> stdlib, just stores data in a disk file. It seems people misunderstood my comment. I should have been more clear. I see no reason PySQLite should be accorded better status than any of the other relational database wrappers. If MySQLdb, etc aren't included with the distribution I don't think PySQLite should be either. I realize it's easier to administer a PySQLite database than a PostgreSQL database, but from a pure client standpoint there's nothing really easier about it. By including PySQLite we'd somehow be blessing it as a better SQL solution than the other options. That means it will almost certainly be stretched beyond its limits and used in situations where it isn't appropriate (multiple writers, writers that hold the database for a long time, etc). That will reflect badly on both SQLite and Python. Skip From carribeiro at gmail.com Thu Oct 21 05:57:01 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Oct 21 05:57:04 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> Message-ID: <864d370904102020573045c364@mail.gmail.com> On Wed, 20 Oct 2004 20:52:35 -0400, Phillip J. Eby <pje@telecommunity.com> wrote: > At 07:14 PM 10/20/04 -0500, Skip Montanaro wrote: > > > Bob> By default can mean that Python builds a SQLite wrapper if SQLite > > Bob> is available, just like it does for bsddb, readline, etc. > > > >Then why not MySQLdb, psycopg and sybase-python also? No slight intended > >against PySQLite, but those other wrapper modules have been around quite a > >bit longer I think. > > Well, one difference is that none of the databases you just listed are > embeddable. There has to be a separate database server process. SQLite, > like other "database" modules in the stdlib, just stores data in a disk file. Not to mention that it's a snap to install & manage -- no need for administrative accounts and complex daemon setup. Although this is really part of the 'embeddable' concept, it's still something worth noting on its own. Also its worth to note that its license is *much* Python-friendlier than almost every one of the other options, as far as the db engine itself is concerned. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From carribeiro at gmail.com Thu Oct 21 06:06:08 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Oct 21 06:06:12 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16759.12986.759165.964094@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> Message-ID: <864d370904102021062bb470a5@mail.gmail.com> On Wed, 20 Oct 2004 22:53:30 -0500, Skip Montanaro <skip@pobox.com> wrote: > Bob> By default can mean that Python builds a SQLite wrapper if SQLite > Bob> is available, just like it does for bsddb, readline, etc. > > >> Then why not MySQLdb, psycopg and sybase-python also? No slight > >> intended against PySQLite, but those other wrapper modules have been > >> around quite a bit longer I think. > > Phillip> Well, one difference is that none of the databases you just > Phillip> listed are embeddable. There has to be a separate database > Phillip> server process. SQLite, like other "database" modules in the > Phillip> stdlib, just stores data in a disk file. > > It seems people misunderstood my comment. I should have been more clear. I > see no reason PySQLite should be accorded better status than any of the > other relational database wrappers. If MySQLdb, etc aren't included with > the distribution I don't think PySQLite should be either. I realize it's > easier to administer a PySQLite database than a PostgreSQL database, but > from a pure client standpoint there's nothing really easier about it. By > including PySQLite we'd somehow be blessing it as a better SQL solution than > the other options. That means it will almost certainly be stretched beyond > its limits and used in situations where it isn't appropriate (multiple > writers, writers that hold the database for a long time, etc). That will > reflect badly on both SQLite and Python. I think that I understand your argument -- in fact that was my first impression when the thread started. It sounds perfectly reasonable, but it really doesn't hold upon closer inspection. In a very similar situation, the presence of the SimpleHTTPServer on the library hasn't stopped anyone from using Apache, or from writing their own web server engines -- some as extensions of the standard module, some as replacements written from the scratch. Of course, webservers and database engines are different beasts, and Apache is what it is, a true benchmark -- but can't similar the same thing be said about MySQL or PostgreSQL (not to mention Oracle and other commercial offerings)? -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From bob at redivi.com Thu Oct 21 06:15:07 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Oct 21 06:15:47 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16759.12986.759165.964094@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> Message-ID: <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> On Oct 20, 2004, at 23:53, Skip Montanaro wrote: > Bob> By default can mean that Python builds a SQLite wrapper if > SQLite > Bob> is available, just like it does for bsddb, readline, etc. > >>> Then why not MySQLdb, psycopg and sybase-python also? No slight >>> intended against PySQLite, but those other wrapper modules have been >>> around quite a bit longer I think. > > Phillip> Well, one difference is that none of the databases you > just > Phillip> listed are embeddable. There has to be a separate > database > Phillip> server process. SQLite, like other "database" modules in > the > Phillip> stdlib, just stores data in a disk file. > > It seems people misunderstood my comment. I should have been more > clear. I > see no reason PySQLite should be accorded better status than any of the > other relational database wrappers. If MySQLdb, etc aren't included > with > the distribution I don't think PySQLite should be either. I realize > it's > easier to administer a PySQLite database than a PostgreSQL database, > but > from a pure client standpoint there's nothing really easier about it. > By > including PySQLite we'd somehow be blessing it as a better SQL > solution than > the other options. That means it will almost certainly be stretched > beyond > its limits and used in situations where it isn't appropriate (multiple > writers, writers that hold the database for a long time, etc). That > will > reflect badly on both SQLite and Python. By including expat are we blessing it as somehow a better solution than libxml2? PySQLite *is* a better choice for inclusion than the others: because the license permits, it's standalone, easy to use. and can be reasonably included with binary distributions of Python (it can even be linked statically into the extension). More or less any database module that's not embedded (except for ODBC, perhaps) is on shakier ground because the protocol can change between database versions, though I suppose that's not expected to happen very often for something like PostgreSQL. Also, MySQLdb is especially tricky because of the license. I can't imagine how that rather contrived scenario could reflect badly on Python or SQLite.. it certainly wouldn't be any worse than Python's standard library support for networking or XML, or the interpreter's inability to scale with threads. -bob From gvanrossum at gmail.com Thu Oct 21 06:41:17 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu Oct 21 06:41:21 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <864d3709041020094016d2affa@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> <864d3709041020094016d2affa@mail.gmail.com> Message-ID: <ca471dc20410202141263225fc@mail.gmail.com> > I like the optional type idea: > > opt(section_name, key_name, default_value, type) > > So it can be written this way: > > opt('section', 'key', None, StringType) > > ...but -- to answer your question -- the point here isn't really the > 'singleness' of the factory function, but the fact that it is > type-independent, which (in principle) would allow it to be extended > to handle arbitrary types by delegating some functionality to the > types themselves. This is all a nice generalization, but I don't see that much use for it. There's only a handful of types that are worth supporting here. So the cost of the generalization isn't worth the benefits. > IMHO, it's just a different way to design it: > instead of writing a specialized property handler for each supported > type, use a standard interface provided by types themselves to handle > the conversion to & from the string representation that is used in the > config file. Unfortunately, this generalization is not as practical as > I thought it would be. Right, it pretty much only works for int, str, float, and for custome types that were designed with this usage in mind. So again I'm not sure that the generality is worth having. > Agreed. It depends a lot on the particular application, and to some > extent, on personal taste. My own bias come from some of the > applications that I have written, where I feared that the use of > prefixes would end up cluttering the namespace. But your "solution" is to introduce a mandatory extra namespace, which adds just as much typing/clutter to the code for your example (x.foo.bar instead of x.foo_bar) while forcing others who don't need it to use the extra namespace. > Besides the above > mentioned communications software example (where I had several > communication ports, each one stored into a section), another real > scenario is the configuration of user interface parameters for several > UI elements -- for example, font selection and background colors for > specific forms. The same parameters ('font', 'font-color', > 'background-color', etc.) are repeated for several sections. But > surely, my own experience is limited, and I can't speak for other > people on this point. Most likely, you're hinting at a different use case, where there is an *arbitrary* set of sections, and you can't fix their number or names beforehand. In that case, using namespaces of the x.foo.bar form typically doesn't work well, because in your Python code, you can't hardcode foo -- rather, you have a variable whose contents loops over the different values of foo, and with this approach you'd end up doing getattr(x, sectionname).bar. In that case, I'd much rather have an alternative API where I can write x[sectionnname].bar, x.foo.bar now being equivalent to x["foo"].bar (note string quotes). As long as your section names aren't too weird (__getitem__ would be a bad section name :-) this should work well. > -------- > [1]As an apart: I was *really* surprised when I first saw that this code worked: > > >>> IntType('12') > 12 > >>> FloatType('-1e+12') > -1000000000000.0 Normally we write that as int('12') and float('-1e+12'). It was a new feature in Python 2.2; before that int wasn't the same thing as IntType. > The following test also worked, but I expected it to convert the value > using an octal representation, but it didn't: > > >>> IntType('012') > 12 That's intentional; the int() constructor is often used to parse input from non-programming humans, who might cut and paste a number with leading zeros and not expect the zeros to turn the whole thing into octal. If you want this behavior, use int('012', 0). > And finally, the bool test: > > >>> BooleanType('False') > True > > Which obviously don't work because the 'False' string converts to > True, as any other non-empty string do :-P. But it seemed a little bit > weird when a I first saw it, specially when compared to FloatType and > IntType. The saying "a foolish consistency is the hobgoblin of little minds" applies to this kind of hypergeneralization. > curiousness-killed-the-cat'ly-yours, Fortunately cats have nine lives. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Thu Oct 21 09:34:00 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu Oct 21 09:34:06 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc20410202141263225fc@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> <864d3709041020094016d2affa@mail.gmail.com> <ca471dc20410202141263225fc@mail.gmail.com> Message-ID: <41776668.7080507@colorstudy.com> Guido van Rossum wrote: >>I like the optional type idea: >> >> opt(section_name, key_name, default_value, type) >> >>So it can be written this way: >> >> opt('section', 'key', None, StringType) >> >>...but -- to answer your question -- the point here isn't really the >>'singleness' of the factory function, but the fact that it is >>type-independent, which (in principle) would allow it to be extended >>to handle arbitrary types by delegating some functionality to the >>types themselves. > > > This is all a nice generalization, but I don't see that much use for > it. There's only a handful of types that are worth supporting here. So > the cost of the generalization isn't worth the benefits. I definitely disagree. A common case is a constrained type, where only a limited number of strings are allowed. Or an IP address, or domain name, or an internationalized boolean converter ("si" -> True), or a color specification, or a valid CSS class name, or... well, the list goes on forever. The advantage of putting this in the parser is that you could have better error messages when the values were malformed. If the parser doesn't do the conversion, you are likely to have lost the location information by the time you try to do the conversion. Good error messages are one of the primary visible features for people who use the configuration files. An additional complication, though; if you plan to make the configuration file writable, these types shouldn't just support converting from a string to a Python type, but the other direction -- so that ambiguous Python types (like a boolean; easily confused as an integer) can be converted in specific ways to a configuration string. I don't think __repr__ or __str__ of the value to be converted are necessarily appropriate. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Oct 21 09:42:19 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu Oct 21 09:42:24 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16759.12986.759165.964094@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> Message-ID: <4177685B.4060905@colorstudy.com> Skip Montanaro wrote: > It seems people misunderstood my comment. I should have been more clear. I > see no reason PySQLite should be accorded better status than any of the > other relational database wrappers. If MySQLdb, etc aren't included with > the distribution I don't think PySQLite should be either. I realize it's > easier to administer a PySQLite database than a PostgreSQL database, but > from a pure client standpoint there's nothing really easier about it. By > including PySQLite we'd somehow be blessing it as a better SQL solution than > the other options. That means it will almost certainly be stretched beyond > its limits and used in situations where it isn't appropriate (multiple > writers, writers that hold the database for a long time, etc). That will > reflect badly on both SQLite and Python. While I like the idea of SQLite wrappers in the standard library, I think this is a good point -- and indeed, a lot of people run up against the limits of SQLite at some point (e.g. PyPI). I think SQLite is a good transitional database, and as such it will often be a sufficient long-term choice, but for a lot of applications it will ultimately be too limiting. I am particularly concerned if the SQLite bindings become less like the other DB-API bindings, so that it is hard to port applications away from SQLite. Specifically, while the type coercion isn't perfect, it makes SQLite *much* more like other RDBMS's; I'd be bothered if by default SQLite acted significantly different than other databases. While the DB-API doesn't address this issue of return types, it's only an issue for SQLite, since all the other databases are typed. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From gh at ghaering.de Thu Oct 21 10:03:23 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 10:04:51 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <41769AC1.4020603@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> Message-ID: <20041021080323.GA12371@mylene.ghaering.de> On Wed, Oct 20, 2004 at 07:05:05PM +0200, M.-A. Lemburg wrote: > Gerhard Haering wrote: > >[...] > >I think that a simple embedded relational database would be a good > >thing to have in Python by default. And as Python 2.5 won't happen > >anytime soon, there's plenty of time for developing it, getting it > >stable, and integrating it. > > SQLite is a gem and PySQLite works great, but I don't see why we > should start adding third-party tools of this size (>38k LOC C code) > to the standard Python distribution. [...] I never had the faintest thought of merging the SQLite source tree into the Python one. > Perhaps you we should consider adding only the Python interface > and then ship a DLL with the Windows installer like we do for > expat and the Sleepycat DBM ?! That's what I was proposing. -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/7db1dfab/attachment-0001.pgp From gh at ghaering.de Thu Oct 21 10:05:50 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 10:07:18 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <1098292825.17345.49.camel@geddy.wooz.org> References: <20041020162905.GA9843@mylene.ghaering.de> <864d370904102010083dba2a40@mail.gmail.com> <1098292825.17345.49.camel@geddy.wooz.org> Message-ID: <20041021080550.GB12371@mylene.ghaering.de> On Wed, Oct 20, 2004 at 01:20:25PM -0400, Barry Warsaw wrote: > I do agree that further discussion probably belongs on other lists. Good idea. I'll repost my message on python-list. -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/62b3af18/attachment.pgp From mal at egenix.com Thu Oct 21 10:16:01 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu Oct 21 10:16:03 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16758.65368.937378.577791@montanaro.dyndns.org> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> Message-ID: <41777041.8040702@egenix.com> Skip Montanaro wrote: > Bob> By default can mean that Python builds a SQLite wrapper if SQLite > Bob> is available, just like it does for bsddb, readline, etc. > > Then why not MySQLdb, psycopg and sybase-python also? No slight intended > against PySQLite, but those other wrapper modules have been around quite a > bit longer I think. SQLite plays in a different league: it is much more like Sleepycat DBM than a full-blown multi-user database engine where you'd use one of the many other database modules. But I understand what you're saying: by placing one of the many possible solutions into the distribution we would be playing Microsoft, in a sense, by branding this one solution as "better" simply because it's easier to use. However, I don't see this happening in the Python world. E.g. take a look at PyXML vs. the vast and healthy set of tools available through third-parties. Placing PyXML into the core hasn't killed off these external projects. Maybe we should have this discussion on more general grounds: do we really want a fat Python distribution or should we focus more on making installation of third-party tools easier ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 21 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From gh at ghaering.de Thu Oct 21 10:46:30 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 10:47:57 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <4177685B.4060905@colorstudy.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> Message-ID: <20041021084630.GB12496@mylene.ghaering.de> On Thu, Oct 21, 2004 at 02:42:19AM -0500, Ian Bicking wrote: > Skip Montanaro wrote: > >[Putting PySQLite into stdlib] That means it will almost certainly > >be stretched beyond its limits and used in situations where it > >isn't appropriate (multiple writers, writers that hold the database > >for a long time, etc). That will reflect badly on both SQLite and > >Python. > > While I like the idea of SQLite wrappers in the standard library, I > think this is a good point -- and indeed, a lot of people run up against > the limits of SQLite at some point (e.g. PyPI). Off-topic here, but that must have been PySQLite < 0.5, because since then, concurrent readers is no problem any longer. With SQLite3 btw., SQLite has much better concurrency support. And with "using it right", it can scale up a lot better now. But that's irrelevant here, IMO. My point is to include a usable DB-API 2.0 implementation that people can use as a starting point when developing applications that need a relational database. Other languages do the same btw. Java (win32?) includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. > [...] I am particularly concerned if the SQLite bindings become less > like the other DB-API bindings, so that it is hard to port > applications away from SQLite. Specifically, while the type coercion > isn't perfect, it makes SQLite *much* more like other RDBMS's; I'd > be bothered if by default SQLite acted significantly different than > other databases. While the DB-API doesn't address this issue of > return types, it's only an issue for SQLite, since all the other > databases are typed. That's an important issue for me. And because I believe you guys here are good at creating good API designs I'd like to hear suggestions. (*) - Worse is better - stay with the old scheme that works in 90 % of all cases, but in 10 % lets the users be surprised and complain? - Stupid by default, which works 100%. If people want the "smart mode", then they need to read the docs and thus know its limitations. OTOH, the "stupid" behaviour is probably surprising too, but at least coherent. For those not so used to (Py)SQLite: All in all, SQLite *is* still typeless. (*) PySQLite builds all the type guessing on top of SQLite, but because of the limitations in the engine, it can't always guess right. WAIT! I *can* implement something that is smarter than always converting to unicode/string, and that is, I can ask the SQLite engine which type a column has, but the limitation is it will only return its internal types: #define SQLITE_INTEGER 1 #define SQLITE_FLOAT 2 #define SQLITE_TEXT 3 #define SQLITE_BLOB 4 #define SQLITE_NULL 5 As soon as you want anything more fancy, like DATE or TIMESTAMP, or BOOLEAN, or whatever, you need PySQLite support again. Would it be a good default for the standard library if the module only knew about these SQLite internal types? -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/3ff94b41/attachment.pgp From phd at mail2.phd.pp.ru Thu Oct 21 10:49:19 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Oct 21 10:49:25 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <41777041.8040702@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> Message-ID: <20041021084919.GB31377@phd.pp.ru> On Thu, Oct 21, 2004 at 10:16:01AM +0200, M.-A. Lemburg wrote: > Maybe we should have this discussion on more general grounds: > do we really want a fat Python distribution -1 > or should we focus more on making installation of third-party tools > easier ? +1 BTW, just installing is not enough, even when it is come with Python distribution. Installing a newer version of BerkeleyDB breaks older databases due to incompatible file formats. IMO, we should focus on installing AND upgrading. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From gh at ghaering.de Thu Oct 21 10:50:29 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 10:51:56 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <864d370904102010083dba2a40@mail.gmail.com> References: <20041020162905.GA9843@mylene.ghaering.de> <864d370904102010083dba2a40@mail.gmail.com> Message-ID: <20041021085029.GC12496@mylene.ghaering.de> On Wed, Oct 20, 2004 at 02:08:32PM -0300, Carlos Ribeiro wrote: > [...] If the sqlite library supports a bigger and > richer API _as part of the standard Python library_, then everyone > else (Python's end users and developers) will naturally expect that > all other rdbms will support the same API. I don't believe people are so stupid. We should then clearly mark DB-API methods in the documentation, and the rest as nonstandard extensions. If you think it is a real problem, the nonstandard methods can even get a leading underscore. -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/8ff27eec/attachment.pgp From gh at ghaering.de Thu Oct 21 10:57:00 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 10:58:28 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041020172223.GA6073@burma.localdomain> References: <20041020162905.GA9843@mylene.ghaering.de> <20041020172223.GA6073@burma.localdomain> Message-ID: <20041021085700.GD12496@mylene.ghaering.de> On Wed, Oct 20, 2004 at 02:22:24PM -0300, Gustavo Niemeyer wrote: > > [Me:] > > So I decided to change the default behaviour and make PySQLite > > typeless by default, too. Everything will be returned as a Unicode > > string (the default might be user-configurable per connection). > > I'm wondering if it would be possible to introduce a mechanism > allowing one to *explicitly* set column conversion functions at > query time. This would avoid having to manually convert rows on > every access. [...] In fact, this feature will be in the new API. If the "guess mode" is active, then you will only have to set converters for columns that the "guesser" doesn't get right. For example: cu.typecasts = {"mycolumn": int} cu.execute("select a, b, c, func(a) as mycolumn from t1") This will replace the "-- types" hack, which will only return if there's real demand for it, for example if a ZOPE DA needs it. -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/20f72ece/attachment.pgp From mal at egenix.com Thu Oct 21 11:26:12 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu Oct 21 11:26:14 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041021084630.GB12496@mylene.ghaering.de> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> Message-ID: <417780B4.3070107@egenix.com> Gerhard Haering wrote: > My point is to include a usable DB-API 2.0 implementation that people > can use as a starting point when developing applications that need a > relational database. Other languages do the same btw. Java (win32?) > includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. Note that JDBC and ODBC are database driver interfaces much like the Python DB API, not database drivers. You still need to add a JDBC or ODBC driver in order to talk to the database backend of your choice (just like you have to do with the DB API). Adding an SQLite interface goes beyond that since it is a database driver for a specific database backend. If you are just after a "usable database driver", then I have to agree with Skip: any of the other available drivers would fit in just as well. Please clarify this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 21 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From gh at ghaering.de Thu Oct 21 11:41:30 2004 From: gh at ghaering.de (Gerhard Haering) Date: Thu Oct 21 11:42:57 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <417780B4.3070107@egenix.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> Message-ID: <20041021094130.GA12799@mylene.ghaering.de> On Thu, Oct 21, 2004 at 11:26:12AM +0200, M.-A. Lemburg wrote: > Gerhard Haering wrote: > >My point is to include a usable DB-API 2.0 implementation that people > >can use as a starting point when developing applications that need a > >relational database. Other languages do the same btw. Java (win32?) > >includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. > [...] > If you are just after a "usable database driver", then I have to > agree with Skip: any of the other available drivers would fit in > just as well. Please clarify this. I'm aiming at a usable DB-API implementation in the stdlib that does not need a server. I want Python to have an RDBMS interface that works OOTB, no administration required. SQLite seems the obvious choice to me, haven't looked at Gadfly in a while, and MySQLdb/MySQL embedded (GPL) has licensing issues (and adds megabytes to the Python binary download, instead of ca. 270 kB uncompressed as for SQLite). -- Gerhard -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/b93d9f00/attachment.pgp From ncoghlan at iinet.net.au Thu Oct 21 12:58:57 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu Oct 21 12:57:39 2004 Subject: [Python-Dev] Patches In-Reply-To: <864d3709041020062130b4faec@mail.gmail.com> References: <20041019170126.GA21167@phd.pp.ru> <000401c4b66c$dbb657a0$c302a044@oemcomputer> <864d3709041020062130b4faec@mail.gmail.com> Message-ID: <41779671.5040100@iinet.net.au> Carlos Ribeiro wrote: > > So... if the RFC states that it should run over telnet, then standard > implementations should do it. But, according to Martin's proposed > rules -- which I entirely agree with -- my vote still doesn't count, > as I haven't reviewed any patch yet; so please take it only as a > general opinion on the subject. Martin's rules are a bargain where reviewing a certain number of existing patches will get him to have a look at a particular patch you'd like him to review (I'd guess it doesn't even have to be one you wrote - just one you'd like someone with commit access to consider). Your opinion on patches definitely counts even if you haven't made it to Martin's magic number - otherwise his bargain wouldn't make much sense! Cheers, Nick. From jim at zope.com Thu Oct 21 13:35:26 2004 From: jim at zope.com (Jim Fulton) Date: Thu Oct 21 13:35:37 2004 Subject: Need better packaging (was Re: [Python-Dev] SQLite module for Python 2.5) In-Reply-To: <41777041.8040702@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> Message-ID: <41779EFE.30904@zope.com> M.-A. Lemburg wrote: ... > Maybe we should have this discussion on more general grounds: > do we really want a fat Python distribution or should we focus > more on making installation of third-party tools easier ? I think you hit the nail on the head. For myself, I think we want to make it *much* easier to install 3rd-party tools. I also think we need to make it *much* easier to *update* the modules that come with Python without having to wait for a new Python release (much as it is easy to update packages in a linux distribution). While I sympathise with the "batteries included" philosophy, it has a number of drawbacks: - It makes technical decisions very risky. After all, once something is added to the Python distribution, it's had to take it out and it's hard to change it (see below). - It can favor some technologies unfairly - It can cause things to be included that shouldn't be. My favorite examples of this are asyncore and the Berkeley DB extensions. The former is no-longer supported by it's original author and causes an undue burden on the Python developers. The later is inadequately supported and causes instability. (For example, I can't use "make test" in a CVS checkout or the beta release, as Python seg-faults when it gets to the bdb tests.) - It actually stifles development of the library. It's hard to be motivated to improve library modules when the time between releases is sooooo long. Either: - You develop features that you won't be able to use for a year or more. (Large systems like Zope and Twisted can't rely on current versions of Python.) or - You have to use copies of future library modules. We are doing this now with doctest. We've had to do this in the past, with considerable difficulty, for CPickle and ayncore. Part of the problem is that library modules evolve at a different rate than the language. Often this is because library modules are new. Newer systems typically evolve faster than mature systems. Sometimes, mature modules, like doctest, suddenly experience a growth spurt. Finally, I think that the *real* needs that drive "batteries included" would be better served by a packaging system. A packaging system (ala cpan or rpm) would make it much easier for people to get the featureful Python's they need than the current system. (It could be argues that we should just use native packaging systems, like RPM. This approach has 2 serious problems. First, it doen't work on Windows. Second, it raises the packaging bar much higher, as packagers have to create separate packages for each target system. A Python-based packaging system would allow people to create packages that are usable on any platform that Python runs on.) IMO, one of the (if not *the*) most important Python development project is the development of a packaging system. I think there are a number of good starts toward this, such as PEP 262 and various efforts such as the mac-based packaging system and some work Fred Drake has done here at Zope Corp. It would be great to follow through with this and get to the point where its us far less important what happens to be included in a distribution. I'm committed to making this happen. ZC will eventually build something if no one else does. But it can happen *much* sooner if we all work together. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From amk at amk.ca Thu Oct 21 14:34:28 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 21 14:36:57 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <41777041.8040702@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> Message-ID: <20041021123427.GA2991@rogue.amk.ca> On Thu, Oct 21, 2004 at 10:16:01AM +0200, M.-A. Lemburg wrote: > Maybe we should have this discussion on more general grounds: > do we really want a fat Python distribution or should we focus > more on making installation of third-party tools easier ? I keep casting glances at PEP 206 (http://www.python.org/peps/pep-0206.html), Moshe's "Batteries Included" PEP, and thinking it should be updated. PEP 206 suggests something quite simple, a script that downloads a bunch of listed packages and added them to the Python core to make a "sumo distribution". The core once included an Extensions/ subdirectory for things that were automatically built as part of Python's build process. Perhaps we could add some extensibility -- e.g. Python's setup.py will automatically check for subdirectories in Extensions/ and run their setup.py scripts. --amk From theller at python.net Thu Oct 21 15:18:08 2004 From: theller at python.net (Thomas Heller) Date: Thu Oct 21 15:17:51 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041021123427.GA2991@rogue.amk.ca> (A. M. Kuchling's message of "Thu, 21 Oct 2004 08:34:28 -0400") References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> <20041021123427.GA2991@rogue.amk.ca> Message-ID: <3c08nran.fsf@python.net> "A.M. Kuchling" <amk@amk.ca> writes: > On Thu, Oct 21, 2004 at 10:16:01AM +0200, M.-A. Lemburg wrote: >> Maybe we should have this discussion on more general grounds: >> do we really want a fat Python distribution or should we focus >> more on making installation of third-party tools easier ? > > I keep casting glances at PEP 206 > (http://www.python.org/peps/pep-0206.html), Moshe's "Batteries > Included" PEP, and thinking it should be updated. PEP 206 suggests > something quite simple, a script that downloads a bunch of listed > packages and added them to the Python core to make a "sumo > distribution". > > The core once included an Extensions/ subdirectory for things that > were automatically built as part of Python's build process. Perhaps > we could add some extensibility -- e.g. Python's setup.py will > automatically check for subdirectories in Extensions/ and run their > setup.py scripts. It may be a crazy idea, but the more I switch platforms the more I have the impression that getting software via cvs (or svn, maybe) is the easiest thing. Couldn't pypi record the needed info for this? Thomas From amk at amk.ca Thu Oct 21 15:22:04 2004 From: amk at amk.ca (A.M. Kuchling) Date: Thu Oct 21 15:24:34 2004 Subject: [Python-Dev] Re: Library packaging (was: SQLite module) In-Reply-To: <3c08nran.fsf@python.net> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> <20041021123427.GA2991@rogue.amk.ca> <3c08nran.fsf@python.net> Message-ID: <20041021132204.GB15482@rogue.amk.ca> On Thu, Oct 21, 2004 at 03:18:08PM +0200, Thomas Heller wrote: > It may be a crazy idea, but the more I switch platforms the more I have > the impression that getting software via cvs (or svn, maybe) is the > easiest thing. It wouldn't help Windows users much. Sensible platforms that come with CVS/SVN usually have their own packaging mechanisms. > Couldn't pypi record the needed info for this? It could be added. I'd like to add support for DOAP (http://usefulinc.com/doap) to PyPI at some point, which does include such information; it would make sense to add things that are in DOAP but not currently in PyPI. --amk From p.f.moore at gmail.com Thu Oct 21 16:23:28 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Thu Oct 21 16:23:32 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041021094130.GA12799@mylene.ghaering.de> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> <20041021094130.GA12799@mylene.ghaering.de> Message-ID: <79990c6b04102107236b590943@mail.gmail.com> On Thu, 21 Oct 2004 11:41:30 +0200, Gerhard Haering <gh@ghaering.de> wrote: > On Thu, Oct 21, 2004 at 11:26:12AM +0200, M.-A. Lemburg wrote: > > Gerhard Haering wrote: > > >My point is to include a usable DB-API 2.0 implementation that people > > >can use as a starting point when developing applications that need a > > >relational database. Other languages do the same btw. Java (win32?) > > >includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. > > [...] > > If you are just after a "usable database driver", then I have to > > agree with Skip: any of the other available drivers would fit in > > just as well. Please clarify this. > > I'm aiming at a usable DB-API implementation in the stdlib that does > not need a server. I want Python to have an RDBMS interface that works > OOTB, no administration required. SQLite seems the obvious choice to > me, haven't looked at Gadfly in a while, and MySQLdb/MySQL embedded > (GPL) has licensing issues (and adds megabytes to the Python binary > download, instead of ca. 270 kB uncompressed as for SQLite). I'm +1 on including PySQLite in the core. It would fit in the same space as Berkeley DB, *not* client-server databases like MySQL, PostgreSQL, Oracle, etc. However, it conforms to 2 important standards, SQL and the Python DB API, where Berkeley DB does not. This matters where people are looking for a more "portable" solution (whether that means scaling up to a full RDBMS at a later stage, or scaling *down* from such a thing, for a more standalone application, or just leveraging existing expertise). I don't think that the issue of batteries included vs easier package installation is relevant here - at the moment, Python *is* "batteries included". While a better package management solution is a laudable goal, until someone comes up and produces something, it doesn't affect the situation - when such a thing exists, I would assume that it would be appropriate to *un*bundle parts of the current stdlib (BSDDB, XML come to mind as "big" areas). Having to also unbundle PySQLite again shouldn't be too much of a chore. (If we're not willing to unbundle, the message is that the packaging solution is good enough for others, but not for "us" - a message I wouldn't feel happy supporting...) Paul. From skip at pobox.com Thu Oct 21 16:23:52 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 21 16:24:00 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> Message-ID: <16759.50808.382307.514375@montanaro.dyndns.org> >> By including PySQLite we'd somehow be blessing it as a better SQL >> solution than the other options. That means it will almost certainly >> be stretched beyond its limits and used in situations where it isn't >> appropriate (multiple writers, writers that hold the database for a >> long time, etc). That will reflect badly on both SQLite and Python. Bob> I can't imagine how that rather contrived scenario could reflect Bob> badly on Python or SQLite. You assume it was contrived, but it wasn't at all. We hit exactly these problems almost upon first use. We were in the process of copying a large amount of data from our corporate Sybase database. Because SQLite's lock granularity is the entire file, the SQLite database was unusable until the entire update process was complete, even though many tables were completely updated long before the update process finished. We also encountered a major performance problem almost immediately. It seems that using BETWEEN is much worse (order of magnitude worse) than two comparison clauses using >=, <, etc. We are in the process of deciding which server-based SQL solution to move to. Skip From misa at redhat.com Thu Oct 21 16:29:30 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Thu Oct 21 16:27:35 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? Message-ID: <20041021142930.GB10408@abulafia.devel.redhat.com> Hello, Here's an interesting bug: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 Is anyone aware of copyright issues on the Monty Python audio file? Thanks, Misa From bob at redivi.com Thu Oct 21 16:32:38 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Oct 21 16:33:16 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16759.50808.382307.514375@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> <16759.50808.382307.514375@montanaro.dyndns.org> Message-ID: <0EA0B903-236E-11D9-A940-000A95BA5446@redivi.com> On Oct 21, 2004, at 10:23, Skip Montanaro wrote: > >>> By including PySQLite we'd somehow be blessing it as a better SQL >>> solution than the other options. That means it will almost certainly >>> be stretched beyond its limits and used in situations where it isn't >>> appropriate (multiple writers, writers that hold the database for a >>> long time, etc). That will reflect badly on both SQLite and Python. > > Bob> I can't imagine how that rather contrived scenario could > reflect > Bob> badly on Python or SQLite. > > You assume it was contrived, but it wasn't at all. We hit exactly > these > problems almost upon first use. We were in the process of copying a > large > amount of data from our corporate Sybase database. Because SQLite's > lock > granularity is the entire file, the SQLite database was unusable until > the > entire update process was complete, even though many tables were > completely > updated long before the update process finished. We also encountered a > major performance problem almost immediately. It seems that using > BETWEEN > is much worse (order of magnitude worse) than two comparison clauses > using >> =, <, etc. > > We are in the process of deciding which server-based SQL solution to > move > to. The concurrency problem makes it sound like you were using SQLite 2.x, not SQLite 3.x. If it was SQLite 3.x, then you could've used separate files for each table: """ A limited form of table-level locking is now also available in SQLite. If each table is stored in a separate database file, those separate files can be attached to the main database (using the ATTACH command) and the combined databases will function as one. But locks will only be acquired on individual files as needed. So if you redefine "database" to mean two or more database files, then it is entirely possible for two processes to be writing to the same database at the same time. To further support this capability, commits of transactions involving two or more ATTACHed database are now atomic. """ ( from http://www.sqlite.org/version3.html -- see also http://www.sqlite.org/lockingv3.html ) -bob From p.f.moore at gmail.com Thu Oct 21 16:35:05 2004 From: p.f.moore at gmail.com (Paul Moore) Date: Thu Oct 21 16:35:08 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <16759.50808.382307.514375@montanaro.dyndns.org> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> <16759.50808.382307.514375@montanaro.dyndns.org> Message-ID: <79990c6b041021073521ab71b7@mail.gmail.com> On Thu, 21 Oct 2004 09:23:52 -0500, Skip Montanaro <skip@pobox.com> wrote: > You assume it was contrived, but it wasn't at all. We hit exactly these > problems almost upon first use. We were in the process of copying a large > amount of data from our corporate Sybase database. Getting very off-topic here, but I'm surprised you considered SQLite in this situation, as an "equivalent" to Sybase. I'd certainly never seen it as catering for that sort of application. I see it more related to something like MS Access (which I hope no-one would consider for serious sized corporate applications - even though I know some people do :-() Of course, if that perception is common, then I think that the library documentation should be very clear about where SQLite is appropriate, and where it is not... Paul. From Michael.Sparks at rd.bbc.co.uk Thu Oct 21 16:35:49 2004 From: Michael.Sparks at rd.bbc.co.uk (Michael Sparks) Date: Thu Oct 21 17:02:55 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <20041021142930.GB10408@abulafia.devel.redhat.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> Message-ID: <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> On Thursday 21 Oct 2004 15:29, Mihai Ibanescu wrote: > Hello, > > Here's an interesting bug: > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 Hi Mihai, That system requires a login in order to access that bug, and after registering I can't access the bug ("You are not authorized..."), and I doubt I'm alone in this. Could you elaborate further? > Is anyone aware of copyright issues on the Monty Python audio file? Since Sourceforge is hosted in the US, I _suspect_ at minimum if the file is sufficiently small that it's covered by fair use there, and in many countries (also whilst UK copyright law doesn't have that concept unfortunately). If it isn't I'd be willing to try and get in contact with the appropriate people. I'm not a lawyer though, nor able to grant permission if none hasn't already been, but I might be able to help *IF* there is an issue :) Best Regards, Michael. -- Michael Sparks, Senior R&D Engineer, Digital Media Group Michael.Sparks@rd.bbc.co.uk, British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This e-mail may contain personal views which are not the views of the BBC. From gvanrossum at gmail.com Thu Oct 21 17:11:27 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu Oct 21 17:11:30 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> References: <20041021142930.GB10408@abulafia.devel.redhat.com> <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> Message-ID: <ca471dc2041021081134f53d1e@mail.gmail.com> On Thu, 21 Oct 2004 15:35:49 +0100, Michael Sparks <michael.sparks@rd.bbc.co.uk> wrote: > On Thursday 21 Oct 2004 15:29, Mihai Ibanescu wrote: > > Hello, > > > > Here's an interesting bug: > > > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 > > Hi Mihai, > > That system requires a login in order to access that bug, and after > registering I can't access the bug ("You are not authorized..."), and I > doubt I'm alone in this. Could you elaborate further? > > > Is anyone aware of copyright issues on the Monty Python audio file? > > Since Sourceforge is hosted in the US, I _suspect_ at minimum if the file is > sufficiently small that it's covered by fair use there, and in many countries > (also whilst UK copyright law doesn't have that concept unfortunately). If it > isn't I'd be willing to try and get in contact with the appropriate people. > > I'm not a lawyer though, nor able to grant permission if none hasn't already > been, but I might be able to help *IF* there is an issue :) I have the same problem (tried to register but still waiting for the password to arrive via email) so I can only guess what this bug report is about. If it's some random user worrying about copyrighted materials, please stop worrying. There are thousands of copies of the entire works of Monty Python on the web; even if M.P. were somehow to decide to send lawyers after each of those (which they don't), I still doubt that they would care about a single 3 second fragment in telephone quality. If we replaced it with a clip of *me* saying the same words I doubt that most people would be able to tell which clip was the original. But all that is irrelevant; only a lawyer's opinion counts. However, my judgement is that it's not worth it for the PSF to spend money to talk to a lawyer about; if someone already has a lawyer's opinion on this I'd like to hear it. Non-lawyer opinions are useless in this (even of lay-people who know a lot about the law) because it's not the legal expertise that makes it valuable, it's the fact that it's a lawyer who says it; what lawyers say (when wearing their lawyer hats) has special significance to the law (at least in the US). Let me promise you this: the second the PSF receives a cease-and-desist letter about this from John Cleese's lawyer, we'll remove it from the distribution. Until then, it stays. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Thu Oct 21 17:17:27 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Thu Oct 21 17:17:30 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <41776668.7080507@colorstudy.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> <864d3709041020094016d2affa@mail.gmail.com> <ca471dc20410202141263225fc@mail.gmail.com> <41776668.7080507@colorstudy.com> Message-ID: <ca471dc204102108177868018f@mail.gmail.com> > >>...but -- to answer your question -- the point here isn't really the > >>'singleness' of the factory function, but the fact that it is > >>type-independent, which (in principle) would allow it to be extended > >>to handle arbitrary types by delegating some functionality to the > >>types themselves. > > > > This is all a nice generalization, but I don't see that much use for > > it. There's only a handful of types that are worth supporting here. So > > the cost of the generalization isn't worth the benefits. > > I definitely disagree. A common case is a constrained type, where only > a limited number of strings are allowed. Or an IP address, or domain > name, or an internationalized boolean converter ("si" -> True), or a > color specification, or a valid CSS class name, or... well, the list > goes on forever. > > The advantage of putting this in the parser is that you could have > better error messages when the values were malformed. If the parser > doesn't do the conversion, you are likely to have lost the location > information by the time you try to do the conversion. Good error > messages are one of the primary visible features for people who use the > configuration files. Sure, I agree with all of that. But my original (optint, optstr, optbool, optfloat) proposal can easily be extended the same way; in fact it is in some sense easier than an API that expects a type object. (Unless you have an adaptation framework in place; until we have a general one, inventing one just for this purpose definitely feels like overkill.) > An additional complication, though; if you plan to make the > configuration file writable, these types shouldn't just support > converting from a string to a Python type, but the other direction -- so > that ambiguous Python types (like a boolean; easily confused as an > integer) can be converted in specific ways to a configuration string. I > don't think __repr__ or __str__ of the value to be converted are > necessarily appropriate. Actually, repr() or str() probably *is* the right answer for this, even if calling the constructor with a string argument isn't the answer for parsing and validation. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Thu Oct 21 17:26:09 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Oct 21 17:26:11 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <ca471dc2041021081134f53d1e@mail.gmail.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> <ca471dc2041021081134f53d1e@mail.gmail.com> Message-ID: <1f7befae041021082626016d7b@mail.gmail.com> [Guido van Rossum] > ... > Let me promise you this: the second the PSF receives a > cease-and-desist letter about this from John Cleese's lawyer, we'll > remove it from the distribution. Until then, it stays. Well, I'd be inclined to remove it if John just asks us to, even in the absence of legal threats. But without the threats, he'd have to ask nicely <wink>. From barry at python.org Thu Oct 21 17:27:22 2004 From: barry at python.org (Barry Warsaw) Date: Thu Oct 21 17:27:32 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <ca471dc2041021081134f53d1e@mail.gmail.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> <ca471dc2041021081134f53d1e@mail.gmail.com> Message-ID: <1098372442.8514.17.camel@geddy.wooz.org> On Thu, 2004-10-21 at 11:11, Guido van Rossum wrote: > If it's some random user worrying about copyrighted materials, please > stop worrying. There are thousands of copies of the entire works of > Monty Python on the web; even if M.P. were somehow to decide to send > lawyers after each of those (which they don't), I still doubt that > they would care about a single 3 second fragment in telephone quality. > If we replaced it with a clip of *me* saying the same words I doubt > that most people would be able to tell which clip was the original. We can always go back to the clip of me saying "My hovercraft is full of eels". If you can find it, I'll happily donate it to the PSF. I could use the tax write-off of the full market value of that intellectual property <$1M wink>. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/34520aad/attachment.pgp From sjoerd at acm.org Thu Oct 21 17:28:41 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Thu Oct 21 17:28:49 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <20041021142930.GB10408@abulafia.devel.redhat.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> Message-ID: <4177D5A9.1070500@acm.org> Mihai Ibanescu wrote: > Hello, > > Here's an interesting bug: > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 > > Is anyone aware of copyright issues on the Monty Python audio file? I get this: "You are not authorized to access bug #136654." -- Sjoerd Mullender <sjoerd@acm.org> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 374 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/668bf227/signature.pgp From skip at pobox.com Thu Oct 21 17:44:37 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 21 17:44:52 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <79990c6b041021073521ab71b7@mail.gmail.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <CAC4935A-2317-11D9-A940-000A95BA5446@redivi.com> <16759.50808.382307.514375@montanaro.dyndns.org> <79990c6b041021073521ab71b7@mail.gmail.com> Message-ID: <16759.55653.816710.967678@montanaro.dyndns.org> >>>>> "Paul" == Paul Moore <p.f.moore@gmail.com> writes: Paul> On Thu, 21 Oct 2004 09:23:52 -0500, Skip Montanaro <skip@pobox.com> wrote: >> You assume it was contrived, but it wasn't at all. We hit exactly >> these problems almost upon first use. We were in the process of >> copying a large amount of data from our corporate Sybase database. Paul> Getting very off-topic here, but I'm surprised you considered Paul> SQLite in this situation, as an "equivalent" to Sybase. Again, people assume lots about what we are doing. I'm not interested in getting into all the details here for many reasons, but I don't believe I said anything about SQLite/Sybase equivalency. I said we were copying a large amount of data from Sybase to SQLite. Paul> Of course, if that perception is common, then I think that the Paul> library documentation should be very clear about where SQLite is Paul> appropriate, and where it is not... Here we are coming back around to what I initially indicated. SQLite will be stretched beyond its limits very quickly. We certainly did (yes, we were using v2, but there is no v3 Python binding yet, right?). The absence of any indication what those limits are will shine a bad light on both SQLite and Python when things don't work as expected. I'm done with this thread. I've registered by concerns about adding PySQLite to the standard distribution. Skip From misa at redhat.com Thu Oct 21 18:03:46 2004 From: misa at redhat.com (Mihai Ibanescu) Date: Thu Oct 21 18:01:45 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> References: <20041021142930.GB10408@abulafia.devel.redhat.com> <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> Message-ID: <20041021160346.GD10408@abulafia.devel.redhat.com> On Thu, Oct 21, 2004 at 03:35:49PM +0100, Michael Sparks wrote: > On Thursday 21 Oct 2004 15:29, Mihai Ibanescu wrote: > > Hello, > > > > Here's an interesting bug: > > > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 > > Hi Mihai, > > > That system requires a login in order to access that bug, and after > registering I can't access the bug ("You are not authorized..."), and I > doubt I'm alone in this. Could you elaborate further? Sorry folks, I am an idiot, I didn't notice the bug was marked as internal. I opened access to it, and for the ones who don't have accounts on bugzilla: <quote> >From an internal list: Are we really allowed to ship: /usr/lib/python2.3/email/test/data/audiotest.au Thats a clip from monthy python, and its surely copyrighted. ... Might as well nuke it to be safe ; it's not like it's part of actual functionality AFAICT. </quote> The copyright owner (if there is one) would probably not care about PSF that much, because PSF is not commercial. But software vendors/packagers are probably at risk more than PSF is. Misa From ianb at colorstudy.com Thu Oct 21 19:26:24 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu Oct 21 19:26:50 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc204102108177868018f@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> <864d3709041020094016d2affa@mail.gmail.com> <ca471dc20410202141263225fc@mail.gmail.com> <41776668.7080507@colorstudy.com> <ca471dc204102108177868018f@mail.gmail.com> Message-ID: <4177F140.7000308@colorstudy.com> Guido van Rossum wrote: >>>>...but -- to answer your question -- the point here isn't really the >>>>'singleness' of the factory function, but the fact that it is >>>>type-independent, which (in principle) would allow it to be extended >>>>to handle arbitrary types by delegating some functionality to the >>>>types themselves. >>> >>>This is all a nice generalization, but I don't see that much use for >>>it. There's only a handful of types that are worth supporting here. So >>>the cost of the generalization isn't worth the benefits. >> >>I definitely disagree. A common case is a constrained type, where only >>a limited number of strings are allowed. Or an IP address, or domain >>name, or an internationalized boolean converter ("si" -> True), or a >>color specification, or a valid CSS class name, or... well, the list >>goes on forever. >> >>The advantage of putting this in the parser is that you could have >>better error messages when the values were malformed. If the parser >>doesn't do the conversion, you are likely to have lost the location >>information by the time you try to do the conversion. Good error >>messages are one of the primary visible features for people who use the >>configuration files. > > > Sure, I agree with all of that. But my original (optint, optstr, > optbool, optfloat) proposal can easily be extended the same way; in > fact it is in some sense easier than an API that expects a type > object. (Unless you have an adaptation framework in place; until we > have a general one, inventing one just for this purpose definitely > feels like overkill. OK. I guess you could subclass opt* to get a new type; I wasn't thinking of that. I shy away from subclassing, but it might be appropriate here. It makes it easier to hang different parameters onto the type as well, like not_empty (for strings), max, min, etc. It would even be easier to hang serialization onto it. I don't think adaptation fits well here, since adaptation seems to generally be context-insensitive, and this conversion process is done in the context of a specific type declaration. >>An additional complication, though; if you plan to make the >>configuration file writable, these types shouldn't just support >>converting from a string to a Python type, but the other direction -- so >>that ambiguous Python types (like a boolean; easily confused as an >>integer) can be converted in specific ways to a configuration string. I >>don't think __repr__ or __str__ of the value to be converted are >>necessarily appropriate. > > > Actually, repr() or str() probably *is* the right answer for this, > even if calling the constructor with a string argument isn't the > answer for parsing and validation. In my experience, this stops working as the types become more complex. For instance, consider a converter that takes a string that has comma-separated names and creates a list of strings; there is a specific way to convert back to that representation (','.join(value)), and both repr() and str() will be incorrect. Potentially you could create a list subclass that has the right repr(), but that seems prone to error. repr() only gives an estimated Python representation of an object -- it is neither reliable (since obj == eval(repr(obj)) isn't true for a large number of objects), nor is it appropriate, since we're trying to generate configuration expressions that are tightly bound to a context, not Python expressions. In the case of generating a config file, if the conversion isn't reliable or is ambiguous it should be an error (which doesn't happen with repr() or str()). -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From bac at OCF.Berkeley.EDU Thu Oct 21 19:30:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Thu Oct 21 19:31:04 2004 Subject: Need better packaging (was Re: [Python-Dev] SQLite module for Python 2.5) In-Reply-To: <41779EFE.30904@zope.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> <41779EFE.30904@zope.com> Message-ID: <4177F234.1090202@ocf.berkeley.edu> Jim Fulton wrote: > M.-A. Lemburg wrote: > ... > >> Maybe we should have this discussion on more general grounds: >> do we really want a fat Python distribution or should we focus >> more on making installation of third-party tools easier ? > > > I think you hit the nail on the head. For myself, I think we want to > make it *much* easier to install 3rd-party tools. I also think we need > to make it *much* easier to *update* the modules that come with Python > without having to wait for a new Python release (much as it is easy to > update packages in a linux distribution). > OK, so is everyone just talking about coming up with our own version of CPAN, or does this also include replacing Distutils? And for those of you not in the Mac world, MacPython already has something called packman that downloads 3rd-party apps and installs them by running their setup.py scripts. -Brett From carribeiro at gmail.com Thu Oct 21 20:02:35 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Oct 21 20:02:38 2004 Subject: [Python-Dev] Fat vs lean distribution (was: SQLite) In-Reply-To: <41777041.8040702@egenix.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> Message-ID: <864d370904102111022770f330@mail.gmail.com> On Thu, 21 Oct 2004 10:16:01 +0200, M.-A. Lemburg <mal@egenix.com> wrote: > Maybe we should have this discussion on more general grounds: > do we really want a fat Python distribution or should we focus > more on making installation of third-party tools easier ? Let me toss my .02 on this. A lean distribution has one distinct feature: it not only allows, but _forces_ the programmer to make explicit choices about what extensions to use. Easy of installation from the network is not a big problem IMHO -- I particularly see no problem with downloading packages for my own development machine -- but *packaging for distribution* is much more important to allow for easy deployment. On the other hand, fat distributions tend to force people (consciously or not) to use only the standard library, if only to avoid problems when deploying the software, and even when the standard option is sub-optimal (MS JET is just one of many examples available). In this sense, I think that there is a market for a lean Python distribution, and a market for a fat distribution. In this scenario, the standard Python distribution should be kept more-or-less as it is -- with some polish, of course, and perhaps including a few extra modules (maybe even Pysqlite?). But it would still be fairly small, comparing with other language distributions and commercial products. The fat distribution (in my opinion) is a job for a commercial company. Perhaps someone as ActiveState could do it, or some of the commercial IDE makers. This distribution could include a bigger selection of modules and packages, including full client-server databsae engines, GUI libraries, UI designers, a report library -- the kind of stuff that makes the life of VB or Delphi programmers easier. The base language would still be the same, but the environment would be richer in features. Best for some developers, not so good for others. In this scenario, more than one company could offer their own 'framework', built on Python and using a different selection of libraries. The main problem here is that market for such frameworks has reduced considerably over the past decade, partly due to Microsoft's nearly absolute dominance (In this sense, it was an admirable feat that Sun managed to go so far with Java). In economic terms, it's not as attractive as it was a few years back, which may explain why such offering is still to materialize. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From jim at zope.com Thu Oct 21 20:04:46 2004 From: jim at zope.com (Jim Fulton) Date: Thu Oct 21 20:04:50 2004 Subject: Need better packaging (was Re: [Python-Dev] SQLite module for Python 2.5) In-Reply-To: <4177F234.1090202@ocf.berkeley.edu> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> <41779EFE.30904@zope.com> <4177F234.1090202@ocf.berkeley.edu> Message-ID: <4177FA3E.4020004@zope.com> Brett C. wrote: > Jim Fulton wrote: > >> M.-A. Lemburg wrote: >> ... >> >>> Maybe we should have this discussion on more general grounds: >>> do we really want a fat Python distribution or should we focus >>> more on making installation of third-party tools easier ? >> >> >> >> I think you hit the nail on the head. For myself, I think we want to >> make it *much* easier to install 3rd-party tools. I also think we need >> to make it *much* easier to *update* the modules that come with Python >> without having to wait for a new Python release (much as it is easy to >> update packages in a linux distribution). >> > > OK, so is everyone So far, we're far from everyone. :) > just talking about coming up with our own version of > CPAN, or does this also include replacing Distutils? I see this building on or working with distutils. > And for those of you not in the Mac world, MacPython already has > something called packman that downloads 3rd-party apps and installs them > by running their setup.py scripts. Right, I refered to that in my note. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pinard at iro.umontreal.ca Thu Oct 21 20:02:27 2004 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Thu Oct 21 20:08:36 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <20041021142930.GB10408@abulafia.devel.redhat.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> Message-ID: <20041021180227.GA9876@titan.progiciels-bpi.ca> [Mihai Ibanescu] > Here's an interesting bug: > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=136654 You are not authorized to access bug #136654. To see this bug, you must first log in to an account with the appropriate permissions. So, what is the rationale and interest of requesting the creation of an account and logging in, merely for browseing a bug report? -- Fran?ois Pinard http://pinard.progiciels-bpi.ca From lars at gustaebel.de Thu Oct 21 20:29:29 2004 From: lars at gustaebel.de (Lars =?iso-8859-15?q?Gust=E4bel?=) Date: Thu Oct 21 20:29:42 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <20041020115239.GA27615@rogue.amk.ca> References: <20041020115239.GA27615@rogue.amk.ca> Message-ID: <200410212029.34349.lars@gustaebel.de> Hello! Since I am the subject of this discussion, I think it's necessary to give my view on this too. I am reluctant on this topic. I don't mean to stab Andrew in the back, but he brought up this issue by himself, I was surprised and (very) flattered as I read it today - who wouldn't like to be a Python developer :-) This explains why I didn't ask for it on python-dev myself. I do not think that it is necessary at this point in time to add me as a developer. There is no open tarfile bug left in the bug tracker and the two open tarfile patches both introduce new features which will have to wait until 2.5 anyway. The first big bug wave since the 2.3 release seems to be over. In his post, Martin said that he believes I like my patches be reviewed by others before they're applied and I agree - I like a second opinion, it makes me feel more comfortable even if it slows down the process. I made positive experiences with Neil Norwitz, Martin and Andrew in this respect. I still see myself as the main developer of tarfile. As Andrew mentioned, I maintain a separate distribution on my website and have a lot of user feedback there, too. I search the SF tracker for tarfile bugs on a regular basis and try to help out whenever I can. This is not supposed to change in the future. If it were common Python development policy to add a contributor to the developers list for further maintenance of his module, I would be delighted to step in. But I don't think this is currently the case. BTW, The "Open issues" section of PEP #2 proposes a practice for module maintenance. It's about having a list of module maintainers that are notified each time there are issues with their module. Maybe there needs to be a more general discussion on this whole topic. -- Lars Gust?bel lars@gustaebel.de -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/c100f47b/attachment.pgp From carribeiro at gmail.com Thu Oct 21 20:48:26 2004 From: carribeiro at gmail.com (Carlos Ribeiro) Date: Thu Oct 21 20:48:29 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <4177F140.7000308@colorstudy.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <864d370904102006044f0e1f36@mail.gmail.com> <ca471dc204102008251a91e30e@mail.gmail.com> <864d3709041020094016d2affa@mail.gmail.com> <ca471dc20410202141263225fc@mail.gmail.com> <41776668.7080507@colorstudy.com> <ca471dc204102108177868018f@mail.gmail.com> <4177F140.7000308@colorstudy.com> Message-ID: <864d3709041021114852674950@mail.gmail.com> On Thu, 21 Oct 2004 12:26:24 -0500, Ian Bicking <ianb@colorstudy.com> wrote: > > Sure, I agree with all of that. But my original (optint, optstr, > > optbool, optfloat) proposal can easily be extended the same way; in > > fact it is in some sense easier than an API that expects a type > > object. (Unless you have an adaptation framework in place; until we > > have a general one, inventing one just for this purpose definitely > > feels like overkill. > > OK. I guess you could subclass opt* to get a new type; I wasn't > thinking of that. I shy away from subclassing, but it might be > appropriate here. It makes it easier to hang different parameters onto > the type as well, like not_empty (for strings), max, min, etc. It would > even be easier to hang serialization onto it. It may be a matter of style; I also tend to shy away from subclassing, because often it leads to a 'parallel' class hierarchy to handle things that could be represented by different interfaces, adaptations or aspects of the original classes. In this case: IntType -> optint IPAdddressType -> optipaddress CustomBooleanType -> optcustomboolean Of course, over-generalization in this case can lead to overly complex classes that try to do too much stuff for every possible situation. But simple interfaces -- as the ones required for configuration support -- don't add that much complexity to the native object anyway. In fact, conversion to & from strings is a so useful extension that I tend to provide it for many of my classes, albeit not with a true standard interface. (In the examples above, there is another caveat -- some types, as BooleanType, can't be subclassed, and this makes things more difficult to generalize under my proposal) > > Actually, repr() or str() probably *is* the right answer for this, > > even if calling the constructor with a string argument isn't the > > answer for parsing and validation. > > In my experience, this stops working as the types become more complex. > For instance, consider a converter that takes a string that has > comma-separated names and creates a list of strings; there is a specific > way to convert back to that representation (','.join(value)), and both > repr() and str() will be incorrect. > > Potentially you could create a list subclass that has the right repr(), > but that seems prone to error. repr() only gives an estimated Python > representation of an object -- it is neither reliable (since obj == > eval(repr(obj)) isn't true for a large number of objects), nor is it > appropriate, since we're trying to generate configuration expressions > that are tightly bound to a context, not Python expressions. In the > case of generating a config file, if the conversion isn't reliable or is > ambiguous it should be an error (which doesn't happen with repr() or str()). This is one of the situations where 'practicality beats purity', IMHO, because the standard Python prompt returns the repr() of the object. If all objects _always_ returned full reversible representations to repr() -- which should be possible, given enough care -- then the command prompt would be unusable for complex objects (just calling a method that return a complex object would cause a lot of stuff to be printed out). Perhaps the standard prompt could be changed to return str() instead of repr(), so repr could _really_ be used for the generic representation case... but it's probably too late to change it. -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: carribeiro@gmail.com mail: carribeiro@yahoo.com From DavidA at ActiveState.com Thu Oct 21 22:03:27 2004 From: DavidA at ActiveState.com (David Ascher) Date: Thu Oct 21 22:14:43 2004 Subject: [Python-Dev] audiotest.au and possible copyright issues? In-Reply-To: <1f7befae041021082626016d7b@mail.gmail.com> References: <20041021142930.GB10408@abulafia.devel.redhat.com> <200410211535.49171.Michael.Sparks@rd.bbc.co.uk> <ca471dc2041021081134f53d1e@mail.gmail.com> <1f7befae041021082626016d7b@mail.gmail.com> Message-ID: <4178160F.9010506@ActiveState.com> Tim Peters wrote: > [Guido van Rossum] > >>... >>Let me promise you this: the second the PSF receives a >>cease-and-desist letter about this from John Cleese's lawyer, we'll >>remove it from the distribution. Until then, it stays. > > > Well, I'd be inclined to remove it if John just asks us to, even in > the absence of legal threats. But without the threats, he'd have to > ask nicely <wink>. Especially if he asks by sending us a creative-commons licensed audio sample. "Hi, this is John Cleese, and I approve of this distribution" Sorry, getting political. --david From jlg at dds.nl Thu Oct 21 22:47:22 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Thu Oct 21 22:45:07 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc20410190800766db981@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> Message-ID: <20041021204722.GA4997@surfboy> On Tue, Oct 19, 2004 at 08:00:23AM -0700, Guido van Rossum wrote: > Let me suggest two variations that I have used successfully in my day > job (which is also my night job :). > > 1. For parsing .ini files, I wrote a wrapper around ConfigParser. The > Python-level API looks like this (anonymized and idealized): > > from XXX import MyConfigWrapper, optstr, optint, optbool, optfloat > > class Config(MyConfigWrapper): > poll_time = optfloat("network-parameters", "poll-time") > use_ssl = optbool("network-parameters", "use-ssl") > window_title = optstr("ui-parameters", "window-title") > # etc. I'm surprised no one has mentioned optparse yet. It already has all the features you use in this example. Maybe a similar API for configuration file parsing would be nice, if only for the sake of consistency: parser = ConfigParser() parser.add_option("network-parameters", "poll-time", type="float", dest="poll_time") parser.add_option("network-parameters", "use-ssl", type="bool", dest="use_ssl") parser.add_option("ui-parameters", "window-title", type="float", dest="window_title") options = parser.parse_file('foo.conf') print options.window_title print options.use_ssl # etc. Bonus points if the implementation allows me to specify a command-line option and configuration file option with one call, as in docutils [1]. More bonus points for reusing optparse code. Cheers, Johannes [1] http://cvs.sourceforge.net/viewcvs.py/docutils/docutils/docutils/frontend.py?rev=HEAD&view=auto From barry at python.org Fri Oct 22 00:28:34 2004 From: barry at python.org (Barry Warsaw) Date: Fri Oct 22 00:28:42 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <ca471dc20410190800766db981@mail.gmail.com> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> Message-ID: <1098397714.8514.76.camel@geddy.wooz.org> On Tue, 2004-10-19 at 11:00, Guido van Rossum wrote: > 2. We're handling modest amounts of XML, all using home-grown DTDs and > with no specific requirements to interface to other apps or XML tools. > I wrote a metaclass which lets me specify the DTD using Python syntax. Sounds like my recent situation. I've done enough custom XML-ing lately that I've been thinking alone similar lines as you. Note that most of what I've written lately uses minidom, although I do have one particular application that uses sax. Both are powerful enough to do the job, but neither are that intuitive, IMO. > Again, my approach is slightly lower-level than previous proposals > here but has the advantage of letting you be explicit about the > mapping between Python and XML names, both for attributes and for > subelements. The metaclass handles reading and writing. It supports > elements containing text (is that CDATA? I never know) I'm no XML guru, but I think they're different. In the one case you have something like: <node>text for the node</node> and in the other you have: <node><![CDATA[cdata, er, um, data]]></node> The differences being that the CDATA stuff shows up in a subnode of <node> and has less restriction on what data can be included within the delimiters. My applications use both. > or > sub-elements, but not both. For sub-elements, it supports cases where > one element has any number of sub-elements of a certain type, which > are then collected in a list, so you can refer to them using Python > sequence indexing/slicing notation. It also supports elements that > have zero or one sub-element of a certain type; absence is indicated > by setting the corresponding attribute to None. I don't support > namespaces, although I expect it would be easy enough to add them. I > don't support unrecognized elements or attributes: while everything > can be omitted (and defaults to None), unrecognized attributes or > elements are always rejected. (I suppose that could be fixed too if > desired.) I have use cases for both behaviors. OT1H, I generally want to reject unknown elements or attributes, reject duplicate elements where my "DTD" doesn't allow them, etc. In at least one case I'm doing something that's probably evil, where sub-elements name email headers and the text inside provide the data for the header. I'm sure XML experts cringe at that and suggest I use something like: <header name="to">value</header> or somesuch instead. > Here's an example: [deleted] That actually doesn't look too bad. Do you think you'll be able to release your stuff? I don't have anything generic enough to be useful yet, but I probably could release stuff if/when I do. > I'm undecided on whether I like the approach with lists of (name, > type) tuples better than the approach with property factories like in > the first example; the list approach allows me to order the attributes > and sub-elements consistently upon rendering, but I'm not particularly > keen on typing string quotes around Python identifiers. The property factories are nice, and I have the same aversion to string quoting Python identifiers. I personally have not had a use case for retaining sub-element order. I may play with my own implementation of your spec and see how far I can get. I definitely would like to see /something/ at a higher abstraction than minidom though. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/ccdf018e/attachment.pgp From barry at python.org Fri Oct 22 00:33:15 2004 From: barry at python.org (Barry Warsaw) Date: Fri Oct 22 00:33:20 2004 Subject: [Python-Dev] ConfigParser shootout, preliminary entry In-Reply-To: <20041021204722.GA4997@surfboy> References: <20041018213947.4730.1086801744.divmod.quotient.1417@ohm> <200410182221.43632.fdrake@acm.org> <ca471dc20410190800766db981@mail.gmail.com> <20041021204722.GA4997@surfboy> Message-ID: <1098397995.8509.78.camel@geddy.wooz.org> On Thu, 2004-10-21 at 16:47, Johannes Gijsbers wrote: > I'm surprised no one has mentioned optparse yet. It already has all the > features you use in this example. Maybe a similar API for configuration file > parsing would be nice, if only for the sake of consistency: > > parser = ConfigParser() > parser.add_option("network-parameters", "poll-time", type="float", > dest="poll_time") > parser.add_option("network-parameters", "use-ssl", type="bool", > dest="use_ssl") > parser.add_option("ui-parameters", "window-title", type="float", > dest="window_title") > options = parser.parse_file('foo.conf') > > print options.window_title > print options.use_ssl > # etc. > > Bonus points if the implementation allows me to specify a command-line option > and configuration file option with one call, as in docutils [1]. More bonus > points for reusing optparse code. This is a very intriguing idea. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/8584e8e3/attachment.pgp From barry at python.org Fri Oct 22 00:36:24 2004 From: barry at python.org (Barry Warsaw) Date: Fri Oct 22 00:36:27 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041021084630.GB12496@mylene.ghaering.de> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> Message-ID: <1098398184.8515.81.camel@geddy.wooz.org> On Thu, 2004-10-21 at 04:46, Gerhard Haering wrote: > WAIT! > > I *can* implement something that is smarter than always converting to > unicode/string, and that is, I can ask the SQLite engine which type a > column has, but the limitation is it will only return its internal > types: > > #define SQLITE_INTEGER 1 > #define SQLITE_FLOAT 2 > #define SQLITE_TEXT 3 > #define SQLITE_BLOB 4 > #define SQLITE_NULL 5 I think that would be a neat idea as a default. Still, I want what the MySQL python binding has -- a way to provide a mapping of column names to converters. IIRC, the interface for that was a bit clunky, but it was definitely usable, so it might be better to be consistent, than better. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/77e60723/attachment.pgp From barry at python.org Fri Oct 22 00:38:18 2004 From: barry at python.org (Barry Warsaw) Date: Fri Oct 22 00:38:22 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <79990c6b04102107236b590943@mail.gmail.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> <20041021094130.GA12799@mylene.ghaering.de> <79990c6b04102107236b590943@mail.gmail.com> Message-ID: <1098398298.8506.84.camel@geddy.wooz.org> On Thu, 2004-10-21 at 10:23, Paul Moore wrote: > I'm +1 on including PySQLite in the core. It would fit in the same > space as Berkeley DB, *not* client-server databases > > I don't think that the issue of batteries included vs easier package > installation is relevant here - at the moment, Python *is* "batteries > included". While a better package management solution is a laudable > goal, until someone comes up and produces something, it doesn't affect > the situation I agree with both points. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041021/32a0a455/attachment.pgp From bac at OCF.Berkeley.EDU Fri Oct 22 08:14:11 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Oct 22 08:14:28 2004 Subject: Need better packaging (was Re: [Python-Dev] SQLite module for Python 2.5) In-Reply-To: <4177FA3E.4020004@zope.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <16758.65368.937378.577791@montanaro.dyndns.org> <41777041.8040702@egenix.com> <41779EFE.30904@zope.com> <4177F234.1090202@ocf.berkeley.edu> <4177FA3E.4020004@zope.com> Message-ID: <4178A533.2000205@ocf.berkeley.edu> Jim Fulton wrote: > Brett C. wrote: > >> Jim Fulton wrote: >> >>> M.-A. Lemburg wrote: >>> ... >>> >>>> Maybe we should have this discussion on more general grounds: >>>> do we really want a fat Python distribution or should we focus >>>> more on making installation of third-party tools easier ? >>> >>> >>> >>> >>> I think you hit the nail on the head. For myself, I think we want to >>> make it *much* easier to install 3rd-party tools. I also think we need >>> to make it *much* easier to *update* the modules that come with Python >>> without having to wait for a new Python release (much as it is easy to >>> update packages in a linux distribution). >>> >> >> OK, so is everyone > > > So far, we're far from everyone. :) > I have a tendency of talking too broadly when it comes to referring to others as a group. > > just talking about coming up with our own version of > >> CPAN, or does this also include replacing Distutils? > > > I see this building on or working with distutils. > OK. >> And for those of you not in the Mac world, MacPython already has >> something called packman that downloads 3rd-party apps and installs >> them by running their setup.py scripts. > > > Right, I refered to that in my note. > Oops. Sorry. Been doing a networking assignment this week and it has thoroughly fried my brain. -Brett From ncoghlan at iinet.net.au Fri Oct 22 15:04:38 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 22 15:03:14 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <200410212029.34349.lars@gustaebel.de> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> Message-ID: <41790566.5020604@iinet.net.au> Lars Gust?bel wrote: > BTW, The "Open issues" section of PEP #2 proposes a practice for module > maintenance. It's about having a list of module maintainers that are notified > each time there are issues with their module. Maybe there needs to be a more > general discussion on this whole topic. Is it possible to give developers privileges on SF *without* granting CVS access to those that don't want it? Then (e.g. Lars) could be added to the list of people that bugs could be assigned to, without any (even imagined) pressure to commit changes directly to the source tree. This could also be useful for having a few more people who can handle tracker items on SF (e.g. by assigning bugs to those likely to be knowledgeable about them), without having to give them all CVS access. I'd be interested in having more ability to help handle bugs, but I see no reason at all to give me write access to CVS. However, I don't know enough about Source Forge to know if either idea is possible. Regards, Nick. From martin at v.loewis.de Fri Oct 22 19:04:25 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Oct 22 19:04:25 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <41790566.5020604@iinet.net.au> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> <41790566.5020604@iinet.net.au> Message-ID: <41793D99.1080005@v.loewis.de> Nick Coghlan wrote: > However, I don't know enough about Source Forge to know if either idea > is possible. This wasn't possible in the past, but is possible now. It would be a change of policy, though, since in Python, we have given all SF developers same rights (except for project admins, of course). I would not like to see a change in that policy, as it complicates matters, and it is difficult to tell whether a lack of access rights was by intention or by mistake. In the specific case, I don't find it too difficult to send an email message to Lars whenever I see a tarfile bug that he hasn't noticed yet. Regards, Martin From fdrake at acm.org Fri Oct 22 19:53:37 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 22 19:54:58 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <200410212029.34349.lars@gustaebel.de> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> Message-ID: <200410221353.37179.fdrake@acm.org> On Thursday 21 October 2004 02:29 pm, Lars Gust?bel wrote: > If it were common Python development policy to add a contributor to the > developers list for further maintenance of his module, I would be > delighted to step in. But I don't think this is currently the case. I think there are lots of people on python-dev who aren't listed as developers on SourceForge. I, for one, have no problem with contributors being on this list. I also don't recall hearing any objections from others. > BTW, The "Open issues" section of PEP #2 proposes a practice for module > maintenance. It's about having a list of module maintainers that are > notified each time there are issues with their module. Maybe there needs > to be a more general discussion on this whole topic. Perhaps. I think setting up a category for related bugs/patches, adding you to the project on SF, and having such tracker entries automatically assigned to you (on the basis of category) would pretty much take care of this. Martin expressed some reservation about having different people on the project have different permissions, which I'd be more concerned about if it were a matter of denying you a permission you wanted or needed. Given that you specifically prefer not to have commit privs, I don't have a problem with your permissions being set up that way. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From martin at v.loewis.de Fri Oct 22 20:25:30 2004 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Oct 22 20:25:29 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <200410221353.37179.fdrake@acm.org> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> <200410221353.37179.fdrake@acm.org> Message-ID: <4179509A.3050308@v.loewis.de> Fred L. Drake, Jr. wrote: > Martin expressed some reservation about having different people on the project > have different permissions, which I'd be more concerned about if it were a > matter of denying you a permission you wanted or needed. Given that you > specifically prefer not to have commit privs, I don't have a problem with > your permissions being set up that way. I'm just trying to say that the permissions won't stay that way. At some point, somebody will wonder about the inconsistency, and "adjust" them, to make them all same again. Regards, Martin From lars at gustaebel.de Fri Oct 22 20:36:53 2004 From: lars at gustaebel.de (Lars =?iso-8859-15?q?Gust=E4bel?=) Date: Fri Oct 22 20:37:09 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <41793D99.1080005@v.loewis.de> References: <20041020115239.GA27615@rogue.amk.ca> <41790566.5020604@iinet.net.au> <41793D99.1080005@v.loewis.de> Message-ID: <200410222036.58916.lars@gustaebel.de> ["Martin v. L?wis"] > In the specific case, I don't find it too difficult to send an email > message to Lars whenever I see a tarfile bug that he hasn't noticed > yet. That's okay with me. But if it were possible to assign bugs to me, others could see that these bugs are already taken care of, so they wouldn't have to bother. Wouldn't this save some time? -- Lars Gust?bel lars@gustaebel.de -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041022/a7a1b3af/attachment.pgp From fdrake at acm.org Fri Oct 22 20:44:17 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 22 20:44:43 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <4179509A.3050308@v.loewis.de> References: <20041020115239.GA27615@rogue.amk.ca> <200410221353.37179.fdrake@acm.org> <4179509A.3050308@v.loewis.de> Message-ID: <200410221444.17353.fdrake@acm.org> On Friday 22 October 2004 02:25 pm, Martin v. L?wis wrote: > I'm just trying to say that the permissions won't stay that way. At some > point, somebody will wonder about the inconsistency, and "adjust" them, > to make them all same again. Whoever's doing that has too much time on their hands. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From ncoghlan at iinet.net.au Sat Oct 23 06:13:26 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sat Oct 23 06:11:57 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <4179509A.3050308@v.loewis.de> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> <200410221353.37179.fdrake@acm.org> <4179509A.3050308@v.loewis.de> Message-ID: <4179DA66.5020607@iinet.net.au> Martin v. L?wis wrote: > Fred L. Drake, Jr. wrote: > >> Martin expressed some reservation about having different people on the >> project have different permissions, which I'd be more concerned about >> if it were a matter of denying you a permission you wanted or needed. >> Given that you specifically prefer not to have commit privs, I don't >> have a problem with your permissions being set up that way. > > > I'm just trying to say that the permissions won't stay that way. At some > point, somebody will wonder about the inconsistency, and "adjust" them, > to make them all same again. Although, with the current policy being "Don't add Python developers to SF without their asking for access", surely it would be simple enough to add an unwritten addendum to that unwritten policy to say "Don't adjust a developer's SF permissions without asking them first". Cheers, Nick. From Ludovic.Aubry at logilab.fr Wed Oct 20 17:07:59 2004 From: Ludovic.Aubry at logilab.fr (Ludovic Aubry) Date: Sat Oct 23 06:15:25 2004 Subject: [Python-Dev] Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 In-Reply-To: <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> References: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> Message-ID: <20041020150759.GD968@logilab.fr> Hi, I would say it is not safe: f=file("hello") from os import fork, execl if fork()!=0: execl("python", "python" ) This leaves fd=3 open on linux On Wed, Oct 20, 2004 at 03:10:28PM +0200, Peter Astrand wrote: > > Comments? Is it safe or not to assume that by the time the Python has > started, only fds 0,1,2 are open? > > /Peter > > > I think this is the core of the problem. The test_close_fds > > test works like this: > > > > All file descriptors in the forked child (except 0,1,2) are > > closed. Then the Python binary is executed via execvp(). A > > small test program is passed to the Python binary via the -c > > command line option. If the OS and subprocess module works > > correctly, we can be sure of that by the time of the > > execve() system call, only file descriptors (0,1,2) are open > > (well, the errpipe as well, but let's leave that out for > > now). But, by the time the Python binary starts executing > > the small program, all sorts of things may have happened. > > I'm not really sure we can trust Python not to open files > > during startup. For example, if we have a PYTHONSTARTUP > > file, that open file will have a file descriptor, perhaps 3. > > > > On one hand, this bug could indicate a bug in the Python > > interpreter itself: perhaps a file descriptor leak. On the > > other hand, this test might be a bit too unsafe. > > > > So probably, this test should be removed. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ludal%40logilab.fr > -- Ludovic Aubry LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org From martin at v.loewis.de Sat Oct 23 12:58:13 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Oct 23 12:58:12 2004 Subject: [Python-Dev] Adding a developer In-Reply-To: <4179DA66.5020607@iinet.net.au> References: <20041020115239.GA27615@rogue.amk.ca> <200410212029.34349.lars@gustaebel.de> <200410221353.37179.fdrake@acm.org> <4179509A.3050308@v.loewis.de> <4179DA66.5020607@iinet.net.au> Message-ID: <417A3945.3000806@v.loewis.de> Nick Coghlan wrote: > Although, with the current policy being "Don't add Python developers to > SF without their asking for access", surely it would be simple enough to > add an unwritten addendum to that unwritten policy to say "Don't adjust > a developer's SF permissions without asking them first". Certainly. Regards, Martin From jepler at unpythonic.net Sat Oct 23 17:15:58 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Sat Oct 23 17:16:03 2004 Subject: [Python-Dev] Re: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 In-Reply-To: <20041020150759.GD968@logilab.fr> References: <E1CKGB6-0007cL-Pc@sc8-sf-web3.sourceforge.net> <Pine.GSO.4.51L2.0410201508100.13884@koeberg.lysator.liu.se> <20041020150759.GD968@logilab.fr> Message-ID: <20041023151558.GB728@unpythonic.net> On Wed, Oct 20, 2004 at 05:07:59PM +0200, Ludovic Aubry wrote: > Hi, > > I would say it is not safe: .. or in this example: $ python -c 'import os; os.write(4, "the text\n")' 4>the_file $ cat the_file the text If Python changed to break this behavior, it would be a tragedy. Jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20041023/95f2e984/attachment.pgp From tismer at stackless.com Sun Oct 24 04:37:47 2004 From: tismer at stackless.com (Christian Tismer) Date: Sun Oct 24 04:37:51 2004 Subject: [Python-Dev] Paid Research Project on Stackless 3.1 Message-ID: <417B157B.2030705@stackless.com> (re-sent and modified, after I recognized that my hardware-clock is broken, need a new note-buck) Dear community, I would love to publish Stackless 3.1, of course. Also I know that there is some inherent bug in it. This is the state of the art sine four months. I am currently in a very tight project and have no time to dig into this problem. BUT IT IS URGENT! I'm seeking for a person who would take the job to find the buglet. He would need to debug and nail down a commercial application, which I cannot make public. (S)He would need to sign an NDA with me. The success payment would be $500, minimum. If the problem shows up to be very hard (to some undefined definition of very hard, to be negotiated), it can be increased to $1000. If my app works afterwards, Stackless 3.1 is just fine and can go out to the public.. If it doesn't work, no payment happens. The identified problem needs to be documented by a reproducible test case. If somebody is interested, please contact me privately. And be aware, this is really no easy stuff. You need to be a real hardcore system hacker with many years of experience. (Armin, Bob, Jeff, Lutz, Richard, Stefan, Stephan?) Here is the CVS path to the dev trunk: CVSROOT=:pserver:anonymous@stackless.com:/home/cvs cvs co slpdev/src/2.3/dev The cheapest complete solution wins. Hurry up :-) Sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Carmerstr. 2 : *Starship* http://starship.python.net/ 10623 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 31 86 04 18 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Wed Oct 20 19:51:31 2004 From: tismer at stackless.com (Christian Tismer) Date: Sun Oct 24 04:39:07 2004 Subject: [Python-Dev] Paid Research Project on Stackless 3.1 Message-ID: <4176A5A3.6010101@stackless.com> Dear community, I would love to publish Stackless 3.1, of course. Also I know that there is some inherent bug in it. I'm seeking a person who would take the job to find the buglet. He would need to debug and nail down a commercial application, which I cannot make public. (S)He would need to sign an NDA with me. The success payment would be $500. If the problem shows up to be very hard (to some undefined definition of very hard, to be negotiated), it can be increased to $1000. If my app works afterwards, Stackless 3.1 is just fine. If it doesn't work, no payment happens. The identified problem needs to be documented by a reprodicuble test case. If somebody is interested, please contact me privately. And be aware, this is really no easy stuff. You need to be a real hardcore system hacker with many years of experience. Here is the CVS path to the dev trunk: CVSROOT=:pserver:anonymous@stackless.com:/home/cvs cvs co slpdev/src/2.3/dev The cheapest complete solution wins. Hurry up :-) Sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Carmerstr. 2 : *Starship* http://starship.python.net/ 10623 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 31 86 04 18 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From stuart at stuartbishop.net Sun Oct 24 07:51:13 2004 From: stuart at stuartbishop.net (Stuart Bishop) Date: Sun Oct 24 07:52:21 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <417780B4.3070107@egenix.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> Message-ID: <417B42D1.5000304@stuartbishop.net> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 M.-A. Lemburg wrote: | Gerhard Haering wrote: | |> My point is to include a usable DB-API 2.0 implementation that people |> can use as a starting point when developing applications that need a |> relational database. Other languages do the same btw. Java (win32?) |> includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. | | | Note that JDBC and ODBC are database driver interfaces much like | the Python DB API, not database drivers. You still need to add | a JDBC or ODBC driver in order to talk to the database backend | of your choice (just like you have to do with the DB API). | | Adding an SQLite interface goes beyond that since it is a | database driver for a specific database backend. | | If you are just after a "usable database driver", then I have to | agree with Skip: any of the other available drivers would fit in | just as well. Please clarify this. | I doubt that anything except Gadfly or SQLite could be built from a Python source distribution. The others, such as PostgreSQL or Oracle drivers, would require a chunk of the database distribution or licences to build and would lock you using a particular release of the database backend. The other product suitable for inclusion would be SQLRelay if its maintainers and its release cycle agree. If I remember correctly, Gadfly would have been in 2.4 if anybody had been able to commit to the long term maintenance of it, so I thought the 'should we' question had already been answered. - -- Stuart Bishop <stuart@stuartbishop.net> http://www.stuartbishop.net/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (GNU/Linux) iD8DBQFBe0LRAfqZj7rGN0oRAqK1AJ4zX938kzA0dr3f52/fIBvfr+ymYgCggjnh DwvnLwRHny6n6UBHZhim9No= =/IG7 -----END PGP SIGNATURE----- From gvanrossum at gmail.com Sun Oct 24 18:43:16 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Sun Oct 24 18:43:19 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python compile.c, 2.330, 2.331 In-Reply-To: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> Message-ID: <ca471dc2041024094332677225@mail.gmail.com> Haven't seen the bug report, but you do realize that comparing code objects has other applications, and this pretty much kills that. On Sat, 23 Oct 2004 17:10:09 -0700, rhettinger@users.sourceforge.net <rhettinger@users.sourceforge.net> wrote: > Update of /cvsroot/python/python/dist/src/Python > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv30133/Python > > Modified Files: > compile.c > Log Message: > SF bug #1048870: call arg of lambda not updating > > Index: compile.c > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Python/compile.c,v > retrieving revision 2.330 > retrieving revision 2.331 > diff -u -d -r2.330 -r2.331 > --- compile.c 29 Sep 2004 23:54:08 -0000 2.330 > +++ compile.c 24 Oct 2004 00:10:05 -0000 2.331 > @@ -261,6 +261,8 @@ > if (cmp) return (cmp<0)?-1:1; > cmp = co->co_flags - cp->co_flags; > if (cmp) return (cmp<0)?-1:1; > + cmp = co->co_firstlineno - cp->co_firstlineno; > + if (cmp) return (cmp<0)?-1:1; > cmp = PyObject_Compare(co->co_code, cp->co_code); > if (cmp) return cmp; > cmp = PyObject_Compare(co->co_consts, cp->co_consts); > > _______________________________________________ > Python-checkins mailing list > Python-checkins@python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From astrand at lysator.liu.se Sun Oct 24 23:11:40 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Sun Oct 24 23:11:57 2004 Subject: [Python-Dev] Bug [ 959379 ] Implicit close() should check for errors Message-ID: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> I'd like to resolve bug 959379. Here's a naive fix. Comments? diff -u -r2.192 fileobject.c --- Objects/fileobject.c 11 Jun 2004 04:49:03 -0000 2.192 +++ Objects/fileobject.c 24 Oct 2004 20:58:49 -0000 @@ -304,7 +304,8 @@ PyObject_ClearWeakRefs((PyObject *) f); if (f->f_fp != NULL && f->f_close != NULL) { Py_BEGIN_ALLOW_THREADS - (*f->f_close)(f->f_fp); + if ((*f->f_close)(f->f_fp) != 0) + perror("close failed"); Py_END_ALLOW_THREADS } PyMem_Free(f->f_setbuf); /Peter ?strand <astrand@lysator.liu.se> From python at rcn.com Mon Oct 25 01:52:28 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon Oct 25 01:54:16 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python compile.c, 2.330, 2.331 In-Reply-To: <ca471dc2041024094332677225@mail.gmail.com> Message-ID: <001601c4ba24$85882e60$28ba2c81@oemcomputer> > Haven't seen the bug report, but you do realize that comparing code > objects has other applications, and this pretty much kills that. In the OP's situation, two distinct function objects with differing default values were showing up with a non-unique code object. As a result, when the OP's code failed, the traceback message pointed to the wrong like in the file (the first seen function rather than the one executing). If some other code comparison application is more important, I would be happy to reverse this patch and mark the bug as WON'T FIX. While the OP's situation did sent him on a wild goose chase (trying to debug the part of his code that did work), I can't imagine that it comes up often enough to care much about. Raymond From sandeep_gohad at yahoo.co.in Mon Oct 25 06:58:50 2004 From: sandeep_gohad at yahoo.co.in (Sandeep Gohad) Date: Mon Oct 25 06:58:54 2004 Subject: [Python-Dev] python with database connectivity Message-ID: <20041025045850.76626.qmail@web8506.mail.in.yahoo.com> Hello I am little new with the python.... If I want to develope any database related application,with options like add,save,modify,delete,previous,last,next,first for maintaining Employee's records.I want to use .mdb file or d-base database as a backend. In VB we can use ADO or DAO, In java we use JDBC, so for python please guide me for developing any basic application.... Regards Sandeep Yahoo! India Matrimony: Find your life partneronline. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041025/e4b00267/attachment.htm From mwh at python.net Mon Oct 25 13:12:43 2004 From: mwh at python.net (Michael Hudson) Date: Mon Oct 25 13:12:45 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python compile.c, 2.330, 2.331 In-Reply-To: <ca471dc2041024094332677225@mail.gmail.com> (Guido van Rossum's message of "Sun, 24 Oct 2004 09:43:16 -0700") References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> Message-ID: <2mvfczaw5w.fsf@starship.python.net> Guido van Rossum <gvanrossum@gmail.com> writes: > Haven't seen the bug report, but you do realize that comparing code > objects has other applications, and this pretty much kills that. Really? It certainly seemed to me that two code objects from different places in the source ought to compare differently... it also lead to a real problem (coalescing them in co_consts). I'm genuinely curious as to what other applications comparing code objects might have. Cheers, mwh (This was my patch that I didn't get around to applying myself). -- In short, just business as usual in the wacky world of floating point <wink>. -- Tim Peters, comp.lang.python From tcdelaney at optusnet.com.au Mon Oct 25 15:01:39 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Mon Oct 25 15:01:37 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net><ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> Message-ID: <012201c4ba92$c482e240$0200a8c0@ryoko> Michael Hudson wrote: > Guido van Rossum <gvanrossum@gmail.com> writes: > >> Haven't seen the bug report, but you do realize that comparing code >> objects has other applications, and this pretty much kills that. > > Really? It certainly seemed to me that two code objects from > different places in the source ought to compare differently... it also > lead to a real problem (coalescing them in co_consts). I'm genuinely > curious as to what other applications comparing code objects might > have. Well, one application which *did* turn up problems with comparing code objects in it's original form was my autosuper recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286195 Specifically, the way I work out what function I'm in: frame = sys._getframe().f_back code = frame.f_code name = code.co_name # Find the method we're currently running by scanning the MRO and comparing # the code objects - when we find a match, that's the class whose method # we're currently executing. for c in type(self).__mro__: try: m = getattr(c, name) except AttributeError: continue if m.func_code is code: return _super(super(c, self), name) I originally compared the code objects using ==, which did give false positives - hence the change to `is`. Using `is` is correct in this case no matter what, but if co_firstlineno was checked in the comparison, == would have also have worked (although slower). BTW, I've just realised that this may not be a sufficient check in all cases, since you can change the code object a function uses - I think I'll write a new test that does just that ... I can't think of an application where the current behaviour would be of great benefit. It's always possible to create a tuple of the fields you want in the comparison. Unless it caused a drop in performance, I'd be in favour of only having identity comparison for code objects. Tim Delaney From aahz at pythoncraft.com Mon Oct 25 15:39:53 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Oct 25 15:39:56 2004 Subject: [Python-Dev] python with database connectivity In-Reply-To: <20041025045850.76626.qmail@web8506.mail.in.yahoo.com> References: <20041025045850.76626.qmail@web8506.mail.in.yahoo.com> Message-ID: <20041025133953.GB9889@panix.com> On Mon, Oct 25, 2004, Sandeep Gohad wrote: > > I am little new with the python.... > If I want to develope any database related application,with options > like add,save,modify,delete,previous,last,next,first for maintaining > Employee's records.I want to use .mdb file or d-base database as a > backend. In VB we can use ADO or DAO, In java we use JDBC, so for > python please guide me for developing any basic application.... I'm sorry, this is a list for people developing core Python, not for people writing Python programs. Please use comp.lang.python or one of the other mailing lists (such as python-list). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From gvanrossum at gmail.com Mon Oct 25 18:36:18 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Mon Oct 25 18:36:24 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <012201c4ba92$c482e240$0200a8c0@ryoko> References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> Message-ID: <ca471dc204102509361880e6e7@mail.gmail.com> > I can't think of an application where the current behaviour would be of > great benefit. It's always possible to create a tuple of the fields you want > in the comparison. Unless it caused a drop in performance, I'd be in favour > of only having identity comparison for code objects. Well, then perhaps code object comparison (and function object comparison) ought to work the same as 'is', not try to do something clever? This would be ironic since that's what it *used* to do long ago, and then I thought it would be "better" if "equivalent" code objects compared equal. It seems that there are no real use cases for having a code object equivalence test, so we might as well save the effort. Right? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Mon Oct 25 20:12:29 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Oct 25 19:15:28 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <ca471dc204102509361880e6e7@mail.gmail.com> References: <012201c4ba92$c482e240$0200a8c0@ryoko> <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> Message-ID: <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> At 09:36 AM 10/25/04 -0700, Guido van Rossum wrote: > > I can't think of an application where the current behaviour would be of > > great benefit. It's always possible to create a tuple of the fields you > want > > in the comparison. Unless it caused a drop in performance, I'd be in favour > > of only having identity comparison for code objects. > >Well, then perhaps code object comparison (and function object >comparison) ought to work the same as 'is', not try to do something >clever? Isn't that what function objects do now? (for some value of "now") Python 2.3.4 (#1, Jun 13 2004, 11:21:03) [GCC 3.3.1 (cygming special)] on cygwin Type "help", "copyright", "credits" or "license" for more information. >>> l = [lambda:None for i in 0,1] >>> l[0]==l[1] False >>> l[0].func_code is l[1].func_code True It would seem that two lambdas with no default arguments, no free variables, and identical code objects would be "equal" if we were using value comparison, rather than identity. From jhylton at gmail.com Mon Oct 25 20:05:39 2004 From: jhylton at gmail.com (Jeremy Hylton) Date: Mon Oct 25 22:52:21 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <ca471dc204102509361880e6e7@mail.gmail.com> References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> <ca471dc204102509361880e6e7@mail.gmail.com> Message-ID: <e8bf7a5304102511054f90b681@mail.gmail.com> On Mon, 25 Oct 2004 09:36:18 -0700, Guido van Rossum <gvanrossum@gmail.com> wrote: > > I can't think of an application where the current behaviour would be of > > great benefit. It's always possible to create a tuple of the fields you want > > in the comparison. Unless it caused a drop in performance, I'd be in favour > > of only having identity comparison for code objects. > > Well, then perhaps code object comparison (and function object > comparison) ought to work the same as 'is', not try to do something > clever? This would be ironic since that's what it *used* to do long > ago, and then I thought it would be "better" if "equivalent" code > objects compared equal. It seems that there are no real use cases for > having a code object equivalence test, so we might as well save the > effort. Right? +1 I can't guess why someone would want to compare code objects for something other than object identity. If someone has a specific use case, there is enough introspection available that they can compare the bytecodes, constants, &c and produce whatever result they want. Jeremy From tcdelaney at optusnet.com.au Mon Oct 25 22:59:44 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Mon Oct 25 22:59:41 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 References: <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> <ca471dc204102509361880e6e7@mail.gmail.com> Message-ID: <013601c4bad5$8ddc9360$0200a8c0@ryoko> Guido van Rossum wrote: > Well, then perhaps code object comparison (and function object > comparison) ought to work the same as 'is', not try to do something > clever? This would be ironic since that's what it *used* to do long > ago, and then I thought it would be "better" if "equivalent" code > objects compared equal. It seems that there are no real use cases for > having a code object equivalence test, so we might as well save the > effort. Right? +1 (that's of course what I meant by identity comparison). Were there any use cases you thought of when you made the change? Tim Delaney From jim at zope.com Mon Oct 25 23:18:32 2004 From: jim at zope.com (Jim Fulton) Date: Mon Oct 25 23:18:56 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <41728467.2090504@interlink.com.au> References: <41728467.2090504@interlink.com.au> Message-ID: <417D6DA8.90702@zope.com> Anthony Baxter wrote: > The first beta is out, so the trunk is unfrozen, and available for > checkins. > > Now that we're in beta, we shouldn't see any new features or behaviour > changing fixes going into the trunk, unless it's been seen and agreed > to on python-dev. Why not create a release branch? That would free up the trunk for new work and reduce the chance of new features creeping in. I realize this is discussed briefly in PEP 101, but that PEP doesn't give a rational for waiting until the final release to make a release branch. It makes more sense to me to make the branch when features are frozen, which means the beta. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From db3l at fitlinxx.com Mon Oct 25 23:27:24 2004 From: db3l at fitlinxx.com (David Bolen) Date: Mon Oct 25 23:27:30 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> Message-ID: <ulldubi9v.fsf@fitlinxx.com> Peter Astrand <astrand@lysator.liu.se> writes: > I'd like to resolve bug 959379. Here's a naive fix. Comments? If you're going to match the normal close() processing, shouldn't that check the result against EOF and not just != 0? If for example, the file object is being used for a pipe, the result of the close might be the result code of the child process which need not be an error. I think that only EOF should signify a failure of the close(), and only in that case is errno probably appropriate (which perror will pick up). -- David From python at rcn.com Tue Oct 26 02:58:49 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Oct 26 03:00:47 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417D6DA8.90702@zope.com> Message-ID: <005301c4baf6$f57f5360$28ba2c81@oemcomputer> > Why not create a release branch? That would free up the trunk for new > work > and reduce the chance of new features creeping in. I prefer the current approach. New features can wait a bit. Anyone working on new stuff should Probably be focused on bug reports right now. We're not that far away from having Py2.4 done anyway. Adding a branch would make it a tiny bit more difficult to fix bugs. This is doubly true for non-developers who want to submit patches. Also, branching creates extra work because at some point the bug fixes and new features would need to be merged. Raymond From astrand at lysator.liu.se Tue Oct 26 07:53:02 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Tue Oct 26 07:56:59 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <ulldubi9v.fsf@fitlinxx.com> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> Message-ID: <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> On Mon, 25 Oct 2004, David Bolen wrote: > > I'd like to resolve bug 959379. Here's a naive fix. Comments? > > If you're going to match the normal close() processing, shouldn't that > check the result against EOF and not just != 0? If for example, the Yes. Thanks. Here's the new patch. OK to commit? diff -u -r2.192 fileobject.c --- fileobject.c 11 Jun 2004 04:49:03 -0000 2.192 +++ fileobject.c 26 Oct 2004 05:50:12 -0000 @@ -304,7 +304,8 @@ PyObject_ClearWeakRefs((PyObject *) f); if (f->f_fp != NULL && f->f_close != NULL) { Py_BEGIN_ALLOW_THREADS - (*f->f_close)(f->f_fp); + if ((*f->f_close)(f->f_fp) == EOF) + perror("close failed"); Py_END_ALLOW_THREADS } PyMem_Free(f->f_setbuf); /Peter ?strand <astrand@lysator.liu.se> From mwh at python.net Tue Oct 26 13:31:44 2004 From: mwh at python.net (Michael Hudson) Date: Tue Oct 26 13:31:47 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> (Phillip J. Eby's message of "Mon, 25 Oct 2004 13:12:29 -0500") References: <012201c4ba92$c482e240$0200a8c0@ryoko> <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> Message-ID: <2mk6tdbtr3.fsf@starship.python.net> "Phillip J. Eby" <pje@telecommunity.com> writes: > At 09:36 AM 10/25/04 -0700, Guido van Rossum wrote: >> > I can't think of an application where the current behaviour would be of >> > great benefit. It's always possible to create a tuple of the >> fields you want >> > in the comparison. Unless it caused a drop in performance, I'd be in favour >> > of only having identity comparison for code objects. >> >>Well, then perhaps code object comparison (and function object >>comparison) ought to work the same as 'is', not try to do something >>clever? +1 > Isn't that what function objects do now? (for some value of "now") Probably. I don't see the relavence, though. > Python 2.3.4 (#1, Jun 13 2004, 11:21:03) > [GCC 3.3.1 (cygming special)] on cygwin > Type "help", "copyright", "credits" or "license" for more information. > >>> l = [lambda:None for i in 0,1] > >>> l[0]==l[1] > False > >>> l[0].func_code is l[1].func_code > True What's happened *here* is that two code objects got compiled up for the two lambdas, but they compare equal so only one goes into co_consts. > It would seem that two lambdas with no default arguments, no free > variables, and identical code objects would be "equal" if we were > using value comparison, rather than identity. Yes. Cheers, mwh -- GET *BONK* BACK *BONK* IN *BONK* THERE *BONK* -- Naich using the troll hammer in cam.misc From mwh at python.net Tue Oct 26 13:34:30 2004 From: mwh at python.net (Michael Hudson) Date: Tue Oct 26 13:34:37 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417D6DA8.90702@zope.com> (Jim Fulton's message of "Mon, 25 Oct 2004 17:18:32 -0400") References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> Message-ID: <2mfz41btmh.fsf@starship.python.net> Jim Fulton <jim@zope.com> writes: > Anthony Baxter wrote: >> The first beta is out, so the trunk is unfrozen, and available for >> checkins. >> Now that we're in beta, we shouldn't see any new features or >> behaviour >> changing fixes going into the trunk, unless it's been seen and agreed >> to on python-dev. > > Why not create a release branch? That would free up the trunk for new work > and reduce the chance of new features creeping in. > > I realize this is discussed briefly in PEP 101, but that PEP doesn't > give a rational for waiting until the final release to make a release branch. > It makes more sense to me to make the branch when features are frozen, > which means the beta. Didn't this get tried before? Are you volunteering to port all the various bugfixy type things from whichever branch they get committed on to the other? I'm sure as hell not. Cheers, mwh -- We did requirements and task analysis, iterative design, and user testing. You'd almost think programming languages were an interface between people and computers. -- Steven Pemberton (one of the designers of Python's direct ancestor ABC) From jim at zope.com Tue Oct 26 15:28:36 2004 From: jim at zope.com (Jim Fulton) Date: Tue Oct 26 15:28:45 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <2mfz41btmh.fsf@starship.python.net> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <2mfz41btmh.fsf@starship.python.net> Message-ID: <417E5104.6010204@zope.com> Michael Hudson wrote: > Jim Fulton <jim@zope.com> writes: > > >>Anthony Baxter wrote: >> >>>The first beta is out, so the trunk is unfrozen, and available for >>>checkins. >>>Now that we're in beta, we shouldn't see any new features or >>>behaviour >>>changing fixes going into the trunk, unless it's been seen and agreed >>>to on python-dev. >> >>Why not create a release branch? That would free up the trunk for new work >>and reduce the chance of new features creeping in. >> >>I realize this is discussed briefly in PEP 101, but that PEP doesn't >>give a rational for waiting until the final release to make a release branch. >>It makes more sense to me to make the branch when features are frozen, >>which means the beta. > > > Didn't this get tried before? I dunno. This is what we do for Zope, and I think it works well. > Are you volunteering to port all the > various bugfixy type things from whichever branch they get committed > on to the other? I'm not talking about any new branches. I'm just suggesting that the branch should be made when features are frozen, rather than waiting until the final release. BTW, if we switched to subversion, porting fixes accross branches would become a lot easier. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From pje at telecommunity.com Tue Oct 26 18:01:29 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Oct 26 17:00:44 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <2mk6tdbtr3.fsf@starship.python.net> References: <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> <012201c4ba92$c482e240$0200a8c0@ryoko> <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20041026105659.02122bc0@mail.telecommunity.com> At 12:31 PM 10/26/04 +0100, Michael Hudson wrote: >"Phillip J. Eby" <pje@telecommunity.com> writes: > > > At 09:36 AM 10/25/04 -0700, Guido van Rossum wrote: > >>Well, then perhaps code object comparison (and function object > >>comparison) ought to work the same as 'is', not try to do something > >>clever? > >+1 > > > Isn't that what function objects do now? (for some value of "now") > >Probably. I don't see the relavence, though. Because Guido sounded like he was saying that function objects didn't already do that. > > Python 2.3.4 (#1, Jun 13 2004, 11:21:03) > > [GCC 3.3.1 (cygming special)] on cygwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> l = [lambda:None for i in 0,1] > > >>> l[0]==l[1] > > False > > >>> l[0].func_code is l[1].func_code > > True > >What's happened *here* is that two code objects got compiled up for >the two lambdas, but they compare equal so only one goes into >co_consts. Actually, no. There's only one code object, otherwise this: [lambda:None for i in range(10000)] would somehow create 10000 code objects, which makes no sense. There's simply a 1:1 mapping between function and class suites in a source text, and code objects created in the module's co_consts. Code comparison has nothing to do with it. From mwh at python.net Tue Oct 26 17:05:31 2004 From: mwh at python.net (Michael Hudson) Date: Tue Oct 26 17:05:32 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Pythoncompile.c, 2.330, 2.331 In-Reply-To: <5.1.1.6.0.20041026105659.02122bc0@mail.telecommunity.com> (Phillip J. Eby's message of "Tue, 26 Oct 2004 11:01:29 -0500") References: <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> <012201c4ba92$c482e240$0200a8c0@ryoko> <E1CLVxZ-0007tb-9Q@sc8-pr-cvs1.sourceforge.net> <ca471dc2041024094332677225@mail.gmail.com> <2mvfczaw5w.fsf@starship.python.net> <012201c4ba92$c482e240$0200a8c0@ryoko> <5.1.1.6.0.20041025130840.02d18ec0@mail.telecommunity.com> <5.1.1.6.0.20041026105659.02122bc0@mail.telecommunity.com> Message-ID: <2m7jpda5ac.fsf@starship.python.net> "Phillip J. Eby" <pje@telecommunity.com> writes: > At 12:31 PM 10/26/04 +0100, Michael Hudson wrote: >>"Phillip J. Eby" <pje@telecommunity.com> writes: >> >> > At 09:36 AM 10/25/04 -0700, Guido van Rossum wrote: >> >>Well, then perhaps code object comparison (and function object >> >>comparison) ought to work the same as 'is', not try to do something >> >>clever? >> >>+1 >> >> > Isn't that what function objects do now? (for some value of "now") >> >>Probably. I don't see the relavence, though. > > Because Guido sounded like he was saying that function objects didn't > already do that. Uh, yeah, I clearly wrote that post before I'd drunk enough coffee... Cheers, mwh -- Java sucks. [...] Java on TV set top boxes will suck so hard it might well inhale people from off their sofa until their heads get wedged in the card slots. --- Jon Rabone, ucam.chat From martin at v.loewis.de Tue Oct 26 18:49:16 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 26 18:49:08 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417D6DA8.90702@zope.com> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> Message-ID: <417E800C.9050400@v.loewis.de> Jim Fulton wrote: > Why not create a release branch? That would free up the trunk for new work > and reduce the chance of new features creeping in. What specific features are you thinking of that you want to creep into HEAD before 2.4 is released? (without including them in 2.4, of course). The release may happen before December, after all. Regards, Martin From martin at v.loewis.de Tue Oct 26 18:56:17 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 26 18:56:09 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> Message-ID: <417E81B1.8080601@v.loewis.de> Peter Astrand wrote: > Yes. Thanks. Here's the new patch. OK to commit? No. I feel that perror is inappropriate; PySys_WriteStderr might be slightly better. However, it might be that even raising an exception has some value. Regards, Martin From jim at zope.com Tue Oct 26 19:30:56 2004 From: jim at zope.com (Jim Fulton) Date: Tue Oct 26 19:31:23 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417E800C.9050400@v.loewis.de> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> Message-ID: <417E89D0.80305@zope.com> Martin v. L?wis wrote: > Jim Fulton wrote: > >> Why not create a release branch? That would free up the trunk for new >> work >> and reduce the chance of new features creeping in. > > > What specific features are you thinking of that you want to creep into > HEAD before 2.4 is released? (without including them in 2.4, of course). > The release may happen before December, after all. No features. I just think it makes more sense to create the release branch when you make the first beta. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Tue Oct 26 19:41:37 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 26 19:41:29 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417E89D0.80305@zope.com> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> Message-ID: <417E8C51.5090506@v.loewis.de> Jim Fulton wrote: > No features. I just think it makes more sense to create the release > branch when you make the first beta. That is common in other projects. It does have the problem that you have to apply bug-fixes twice, which is tedious to do. Not branching only hurts if there are significant sub-projects that need to share up-to-date sources, which is not the case for Python. We don't even branch for the final release, which I consider a (minor) flaw. Instead, the maintenance branch is created *after* the release. Regards, Martin From barry at python.org Tue Oct 26 20:02:04 2004 From: barry at python.org (Barry Warsaw) Date: Tue Oct 26 20:02:09 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417E8C51.5090506@v.loewis.de> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> Message-ID: <1098813724.20432.23.camel@geddy.wooz.org> On Tue, 2004-10-26 at 13:41, "Martin v. L?wis" wrote: > That is common in other projects. It does have the problem that you > have to apply bug-fixes twice, which is tedious to do. Not branching > only hurts if there are significant sub-projects that need to share > up-to-date sources, which is not the case for Python. > > We don't even branch for the final release, which I consider a > (minor) flaw. Instead, the maintenance branch is created *after* the > release. /me is pining for subversion on sourceforge ;) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041026/4127a518/attachment-0001.pgp From jim at zope.com Tue Oct 26 20:47:40 2004 From: jim at zope.com (Jim Fulton) Date: Tue Oct 26 20:47:51 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417E8C51.5090506@v.loewis.de> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> Message-ID: <417E9BCC.3080702@zope.com> Martin v. L?wis wrote: > Jim Fulton wrote: > >> No features. I just think it makes more sense to create the release >> branch when you make the first beta. > > > That is common in other projects. It does have the problem that you > have to apply bug-fixes twice, That only makes a difference for a short time. Eventually there will be a maintenance branch. > which is tedious to do. It is *much* less so with subversion. We should switch. :) > Not branching > only hurts if there are significant sub-projects that need to share > up-to-date sources, which is not the case for Python. I think that branching would tend to enforce the feature freeze. I don't feel strongly about this. I was just a little surprised. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From astrand at lysator.liu.se Tue Oct 26 20:48:42 2004 From: astrand at lysator.liu.se (Peter Astrand) Date: Tue Oct 26 20:48:58 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <417E81B1.8080601@v.loewis.de> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> <417E81B1.8080601@v.loewis.de> Message-ID: <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> On Tue, 26 Oct 2004, "Martin v. L?wis" wrote: > > Yes. Thanks. Here's the new patch. OK to commit? > > No. I feel that perror is inappropriate; PySys_WriteStderr > might be slightly better. Here's a patch with PySys_WriteStderr: diff -u -r2.192 fileobject.c --- Objects/fileobject.c 11 Jun 2004 04:49:03 -0000 2.192 +++ Objects/fileobject.c 26 Oct 2004 18:44:19 -0000 @@ -300,12 +300,15 @@ static void file_dealloc(PyFileObject *f) { + int sts = 0; if (f->weakreflist != NULL) PyObject_ClearWeakRefs((PyObject *) f); if (f->f_fp != NULL && f->f_close != NULL) { Py_BEGIN_ALLOW_THREADS - (*f->f_close)(f->f_fp); + sts = (*f->f_close)(f->f_fp); Py_END_ALLOW_THREADS + if (sts == EOF) + PySys_WriteStderr("close failed: %s\n", strerror(errno)); } PyMem_Free(f->f_setbuf); Py_XDECREF(f->f_name); >However, it might be that even > raising an exception has some value. Is it really possible to raise an exception in response to something triggered by the GC? /Peter ?strand <astrand@lysator.liu.se> From tim.peters at gmail.com Tue Oct 26 21:35:38 2004 From: tim.peters at gmail.com (Tim Peters) Date: Tue Oct 26 21:35:41 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> <417E81B1.8080601@v.loewis.de> <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> Message-ID: <1f7befae04102612351c2b7c99@mail.gmail.com> [Peter Astrand] ... > Is it really possible to raise an exception in response to something > triggered by the GC? It's possible but wholly useless. If you try, gc will detect that an exception was raised, and kill the Python process ungracefully, via Py_FatalError("unexpected exception during garbage collection"); You don't want to do that <wink>. From martin at v.loewis.de Tue Oct 26 22:01:23 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 26 22:01:14 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417E9BCC.3080702@zope.com> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> Message-ID: <417EAD13.3090309@v.loewis.de> Jim Fulton wrote: > That only makes a difference for a short time. Eventually there will > be a maintenance branch. Yes. However, between a beta an the final release, there are likely many bug fixes in a short period of time, whereas (hopefully) the frequency of bug fixes goes down after a release. > > which is tedious to do. > > It is *much* less so with subversion. We should switch. :) Both Barry and you make this claim, which makes me curious. How precisely is it more easy to apply the same change to two branches in subversion? Being a long time subversion user, I'ld normally just use the same procedure I use in CVS, i.e. just apply the patch twice, and commit it twice. > I think that branching would tend to enforce the feature freeze. Hmm. I'm both uncertain whether that would indeed be the case, and whether it would be a good thing. You seem the be saying that there would be less changes on the branch. That might be the case. In projects that do use this approach, it often means that the burden on the release manager is increased, who now has to make sure that bugs get not only fixed in the trunk, but also on the branch. I don't think I want that. Regards, Martin From martin at v.loewis.de Tue Oct 26 22:03:20 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Oct 26 22:03:11 2004 Subject: [Python-Dev] Re: Bug [ 959379 ] Implicit close() should check for errors In-Reply-To: <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> References: <Pine.GSO.4.51L2.0410242311270.5858@koeberg.lysator.liu.se> <ulldubi9v.fsf@fitlinxx.com> <Pine.GSO.4.51L2.0410260750530.29046@koeberg.lysator.liu.se> <417E81B1.8080601@v.loewis.de> <Pine.GSO.4.51L2.0410262044200.14637@koeberg.lysator.liu.se> Message-ID: <417EAD88.7080105@v.loewis.de> Peter Astrand wrote: > + PySys_WriteStderr("close failed: %s\n", strerror(errno)); This should use HAVE_STRERROR. Regards, Martin From jim at zope.com Tue Oct 26 22:14:12 2004 From: jim at zope.com (Jim Fulton) Date: Tue Oct 26 22:14:29 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417EAD13.3090309@v.loewis.de> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> Message-ID: <417EB014.4050604@zope.com> Martin v. L?wis wrote: > Jim Fulton wrote: ... >> > which is tedious to do. >> >> It is *much* less so with subversion. We should switch. :) > > > Both Barry and you make this claim, which makes me curious. > How precisely is it more easy to apply the same change to two > branches in subversion? Being a long time subversion user, > I'ld normally just use the same procedure I use in CVS, i.e. > just apply the patch twice, and commit it twice. With CVS either: - When creating the branch, I have to be very carreful about tags, so that I can tell CVS to apply differences made between the 2 tags, or - I have to apply changes to each individual file affected. With subversion, every revision is is effectively a tag. Merging is usually just a matter of merging differences between 2 repository revision numbers. This makes a huge difference when multiple files are involved. >> I think that branching would tend to enforce the feature freeze. > > > Hmm. I'm both uncertain whether that would indeed be the case, I find that having a separate branch makes it absolutely unambiguous that only bug fixes should be created there. <shrug> > and > whether it would be a good thing. You seem the be saying that there > would be less changes on the branch. That might be the case. I'm saying that there would be fewer feature changes made. This definately seems like a good thing if features are supposed to be frozen. Python hasn't always done a good job of avoiding feature changes during beta cycles or on maintenence branches. If avoiding changes in such situations is desireable, as I think it is, then extra process to discourage such changes would be good. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From martin at v.loewis.de Wed Oct 27 00:15:54 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 27 00:15:46 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417EB014.4050604@zope.com> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> Message-ID: <417ECC9A.5060200@v.loewis.de> Jim Fulton wrote: > With subversion, every revision is is effectively a tag. Merging > is usually just a matter of merging differences between 2 > repository revision numbers. This makes a huge difference when > multiple files are involved. Ah, that. I don't worry about this difference too much. When I apply a change, it usually comes from SF, where it usually is in the form of a patch. So applying the patch to both the HEAD and a maintenance branch is: cd py2.4 cvs up patch -p0 < /tmp/name_of_patch (make, test, etc) cvs commit cd ../py2.3 cvs up patch -p0 < /tmp/name_of_patch (make, test, etc) cvs commit To my understanding, you are saying it is very much the same for svn. The only difference is that back-porting of patches afterwards is easier for svn - that may be true, but is irrelevant for applying SF patches. > I find that having a separate branch makes it absolutely > unambiguous that only bug fixes should be created there. <shrug> I don't think it makes it so for all developers. Between branching the beta, and the final release, it might be still reasonable to add features (in a strict sense) - the same is actually true after the release. Whether a branch is created or not would be immaterial to me. > Python hasn't always done a good job of avoiding feature changes > during beta cycles or on maintenence branches. If avoiding changes > in such situations is desireable, as I think it is, then extra process > to discourage such changes would be good. There is an ongoing debate within both the Python developers, and the Python users, whether strict rejection of new features is a good thing. I've personally come to the conclusion that I will bow to whatever decision the release manager decides. Regards, Martin From jim at zope.com Wed Oct 27 00:44:20 2004 From: jim at zope.com (Jim Fulton) Date: Wed Oct 27 00:44:45 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417ECC9A.5060200@v.loewis.de> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417ECC9A.5060200@v.loewis.de> Message-ID: <417ED344.9050306@zope.com> Martin v. L?wis wrote: > Jim Fulton wrote: > >> With subversion, every revision is is effectively a tag. Merging >> is usually just a matter of merging differences between 2 >> repository revision numbers. This makes a huge difference when >> multiple files are involved. > > > Ah, that. I don't worry about this difference too much. When I apply > a change, it usually comes from SF, where it usually is in the form > of a patch. ... > The only difference is that back-porting of patches afterwards is > easier for svn - that may be true, but is irrelevant for applying > SF patches. Well, I never backport SF patches. :) >> I find that having a separate branch makes it absolutely >> unambiguous that only bug fixes should be created there. <shrug> > > > I don't think it makes it so for all developers. Between branching > the beta, and the final release, it might be still reasonable to > add features (in a strict sense) - the same is actually true after > the release. Whether a branch is created or not would be immaterial > to me. Then we disagree strongly about what should happen. I don't see any point in a feature freeze if you are going to then change features. ;) IMO, it is really bad to make feature changes after a beta release or in bug-fix releases. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From barry at python.org Wed Oct 27 00:59:32 2004 From: barry at python.org (Barry Warsaw) Date: Wed Oct 27 00:59:39 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417ED344.9050306@zope.com> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417ECC9A.5060200@v.loewis.de> <417ED344.9050306@zope.com> Message-ID: <1098831572.22871.21.camel@geddy.wooz.org> On Tue, 2004-10-26 at 18:44, Jim Fulton wrote: > IMO, it is really bad to make feature changes after a beta release or in > bug-fix releases. Don't worry, we won't need overly elaborate processes <wink> to keep this under control. Anthony's enough of a hard-ass to strongly discourage new features by public humiliation and intimidation. that's-why-we-love-him-or-better-him-than-me-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20041026/6743ac34/attachment-0001.pgp From anthony at interlink.com.au Wed Oct 27 03:07:04 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 27 03:07:37 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <1098831572.22871.21.camel@geddy.wooz.org> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <4 Message-ID: <417EF4B8.8060100@interlink.com.au> Barry Warsaw wrote: > Don't worry, we won't need overly elaborate processes <wink> to keep > this under control. Anthony's enough of a hard-ass to strongly > discourage new features by public humiliation and intimidation. I prefer to think of it as a campaign of education and mockery, in relevant proportions. As far as the original point (branching at the beta point, rather than post-final-release) - I've seen little demand for making this happen, and it increases both the workload and (much more importantly) the chance for a visit from Mr Cockup. But it's something we could do now if there's a real groundswell of people who want to get stuff into the trunk for 2.5 that extra month early <wink>. I'm also planning for more frequent releases during the beta cycle, so if all things go perfectly, I anticipate something like: beta2: Wednesday, November 3rd RC1: Wednesday, November 18th final: Tuesday, November 30th Obviously this is also dependent on Martin and Fred's time, nd any other feedback people have to offer. I think this is a realistic schedule... Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From amk at amk.ca Wed Oct 27 03:41:35 2004 From: amk at amk.ca (A.M. Kuchling) Date: Wed Oct 27 03:44:21 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417EF4B8.8060100@interlink.com.au> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417EF4B8.8060100@interlink.com.au> Message-ID: <20041027014135.GA15116@rogue.amk.ca> On Wed, Oct 27, 2004 at 11:07:04AM +1000, Anthony Baxter wrote: > beta2: Wednesday, November 3rd > RC1: Wednesday, November 18th > final: Tuesday, November 30th Any room in there for a bug day? Or do you think it wouldn't be advisable? Bug day = lots of changes, so maybe we don't want that to happen at the beta stage. The best weekends for a bug day might be Nov. 6 or 13th (between beta2 and RC1), or this coming weekend (before beta2). I still won't have time to run it, so someone will have to jump in. --amk From trentm at ActiveState.com Wed Oct 27 04:05:42 2004 From: trentm at ActiveState.com (Trent Mick) Date: Wed Oct 27 04:06:05 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 Message-ID: <417F0276.7000209@activestate.com> There is a subtlety in CreateProcess in the Win32 API in that if one specifies an environment (via the lpEnvironment argument), the environment strings (A) must be sorted alphabetically and (B) that sort must be case-insensitive. See the Remarks section on: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/createprocess.asp If this is not done, then surprises can happen with the use of {Get|Set}EnvironmentVariable in the created process: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/getenvironmentvariable.asp Neither _subprocess.pyd (supporting the new subprocess.py module on Windows) nor PyWin32's CreateProcess binding do this. I haven't done so yet, but I should be able to put together a test case for subprocess.py for this. We came across such a surprise when using my process.py module that uses this PyWin32 code (which it looks like _subprocess.c borrowed). Fixing (A) is easy with a "PyList_Sort(keys)" and some other minor changes to _subprocess.c::getenvironment() -- and to win32process.i::CreateEnvironmentString() in PyWin32. However, I'd like some guidance on the best way to case-insensitively sort a Python list in C code to fix (B). The best thing I see would be to expose PyString_Lower/PyUnicode_Lower and/or PyString_Upper/PyUnicode_Upper so they can be used to .lower()/.upper() the given environment mapping keys for sorting. Does that sound reasonable? Is there some problem to this approach that anyone can see? Trent -- Trent Mick trentm@activestate.com From anthony at interlink.com.au Wed Oct 27 04:07:42 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Oct 27 04:08:02 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <20041027014135.GA15116@rogue.amk.ca> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417EF Message-ID: <417F02EE.7040903@interlink.com.au> A.M. Kuchling wrote: > Any room in there for a bug day? Or do you think it wouldn't be > advisable? Bug day = lots of changes, so maybe we don't want that to > happen at the beta stage. I'm not sure. If there was one, I'd hope it could be tightly focussed on only bugs relevant to the release - this would probably require someone to keep an eye on the bugday channel to make sure that this is handled appropriately. I don't have time to do this, unfortunately. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood. From fred at zope.com Wed Oct 27 05:15:05 2004 From: fred at zope.com (Fred Drake) Date: Wed Oct 27 05:15:27 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <417EF4B8.8060100@interlink.com.au> References: <41728467.2090504@interlink.com.au> <4 <417EF4B8.8060100@interlink.com.au> Message-ID: <200410262315.05513.fred@zope.com> On Tuesday 26 October 2004 09:07 pm, Anthony Baxter wrote: > As far as the original point (branching at the beta point, rather > than post-final-release) - I've seen little demand for making > this happen, and it increases both the workload and (much more > importantly) the chance for a visit from Mr Cockup. But it's Indeed it does. We don't need that. > something we could do now if there's a real groundswell of > people who want to get stuff into the trunk for 2.5 that extra > month early <wink>. As someone who works daily using the Zope 3 Way, I don't think it makes much sense for Python. No matter how easy Subversion makes moving patches around, it is another opportunity for a screwup. I don't see any reason to make the branch earlier than the first release candidate. We don't have a history of accidental inclusion of new features; when we've added things, it was done intentionally. (I'm not endorsing a policy on *that* here, just noting that we don't have a problem caused by the lack of an early branch.) > beta2: Wednesday, November 3rd > RC1: Wednesday, November 18th > final: Tuesday, November 30th This looks good for me at this point. -Fred -- Fred L. Drake, Jr. <fred at zope.com> Zope Corporation From martin at v.loewis.de Wed Oct 27 08:59:47 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 27 08:59:36 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <417F0276.7000209@activestate.com> References: <417F0276.7000209@activestate.com> Message-ID: <417F4763.4020703@v.loewis.de> Trent Mick wrote: > However, I'd like some guidance on the best way to case-insensitively > sort a Python list in C code to fix (B). The best thing I see would be > to expose PyString_Lower/PyUnicode_Lower and/or > PyString_Upper/PyUnicode_Upper so they can be used to .lower()/.upper() > the given environment mapping keys for sorting. > > Does that sound reasonable? Is there some problem to this approach that > anyone can see? Here we go. Exposing PyString_Lower would be a new feature, we are in beta, so new features are not acceptable. That said, I personally could accept such a feature (exposing more C API) quite well between the beta and the final release. That said, I think exposing PyString_Lower would not be desirable (whether now or later). Instead, you should use PyObject_CallMethod to invoke .lower(). Regards, Martin From mal at egenix.com Wed Oct 27 10:10:33 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed Oct 27 10:10:36 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <417F0276.7000209@activestate.com> References: <417F0276.7000209@activestate.com> Message-ID: <417F57F9.30408@egenix.com> Trent Mick wrote: > > There is a subtlety in CreateProcess in the Win32 API in that if one > specifies an environment (via the lpEnvironment argument), the > environment strings (A) must be sorted alphabetically and (B) that sort > must be case-insensitive. See the Remarks section on: > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/createprocess.asp > > > If this is not done, then surprises can happen with the use of > {Get|Set}EnvironmentVariable in the created process: > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/getenvironmentvariable.asp > > > Neither _subprocess.pyd (supporting the new subprocess.py module on > Windows) nor PyWin32's CreateProcess binding do this. I haven't done so > yet, but I should be able to put together a test case for subprocess.py > for this. We came across such a surprise when using my process.py module > that uses this PyWin32 code (which it looks like _subprocess.c borrowed). > > Fixing (A) is easy with a "PyList_Sort(keys)" and some other minor > changes to _subprocess.c::getenvironment() -- and to > win32process.i::CreateEnvironmentString() in PyWin32. > > However, I'd like some guidance on the best way to case-insensitively > sort a Python list in C code to fix (B). The best thing I see would be > to expose PyString_Lower/PyUnicode_Lower and/or > PyString_Upper/PyUnicode_Upper so they can be used to .lower()/.upper() > the given environment mapping keys for sorting. > > Does that sound reasonable? Is there some problem to this approach that > anyone can see? If you want to sort the list in C, it's better to provide a C sorting function. That function can then use Py_UNICODE_TOUPPER(ch) and toupper() for the comparison. Calling the .upper() method on the object would be much too expensive. Dito for creating new objects just for the purpose of comparing two objects. I think it would be worthwhile to consider replacing the string implementation's direct usage of toupper(), tolower() etc. with a Py_STRING_TOUPPER(ch) et al. approach much like the Unicode object does - at least for consistency reasons. In the future, I think it would be best to move away from the C lib implementation of toupper(), tolower() et al. because these are affected by the current locale settings which is not what you'd normally expect in Python. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 27 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From python at rcn.com Wed Oct 27 10:47:36 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Oct 27 10:49:24 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <417F0276.7000209@activestate.com> Message-ID: <000201c4bc01$9c331880$7c09a044@oemcomputer> > However, I'd like some guidance on the best way to case-insensitively > sort a Python list in C code to fix (B). The best thing I see would be > to expose PyString_Lower/PyUnicode_Lower and/or > PyString_Upper/PyUnicode_Upper so they can be used to .lower()/.upper() > the given environment mapping keys for sorting. Why not mimic the pure python approach? lowerfunc = PyObject_GetAttrString(&PyUnicode_Type, "lower"); PyObject_CallMethod(mylist, "sort", "OO", Py_None, lowerfunc); Raymond From jim at zope.com Wed Oct 27 15:24:48 2004 From: jim at zope.com (Jim Fulton) Date: Wed Oct 27 15:24:51 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <1098831572.22871.21.camel@geddy.wooz.org> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417ECC9A.5060200@v.loewis.de> <417ED344.9050306@zope.com> <1098831572.22871.21.camel@geddy.wooz.org> Message-ID: <417FA1A0.7080103@zope.com> Barry Warsaw wrote: > On Tue, 2004-10-26 at 18:44, Jim Fulton wrote: > > >>IMO, it is really bad to make feature changes after a beta release or in >>bug-fix releases. > > > Don't worry, we won't need overly elaborate processes <wink> to keep > this under control. Anthony's enough of a hard-ass to strongly > discourage new features by public humiliation and intimidation. I dunno. It doesn't seem as though there is much agreement on this. If we can't agree on the goal, it seems hard for Anthony to enforce or encourage it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From FBatista at uniFON.com.ar Wed Oct 27 15:30:38 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Oct 27 15:32:13 2004 Subject: [Python-Dev] On when making modifications to the CVS code Message-ID: <A128D751272CD411BC9200508BC2194D053C7AEF@escpl.tcp.com.ar> People: I have these doubts from a while ago, and while a learned a lot about this through Raymond Hettinger, I still have some loose ends. Don't know if there's an official position or it's just developer common sense (which I still don't have), but I didn't find an article/PEP about this. Such paper exists? For now, I'll ask you about a specific issue: there's this bug open about the reindent.py tool, which has an issue about the reindented code file's metadata (more specific: permissions). So I came up with a solution (small patch, three lines), which leaves the reindented file with the same permissions that the original one. My problem is that I can not decide to commit those changes. We're in beta, and don't know if the changes will be tested enough before the final release, in enough platforms (for now, it's tested on Linux, Win2k and MacOS X). And if I don't commit these now, when? Thank you all! . Facundo -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041027/a75e0646/attachment.htm From dgm at ecs.soton.ac.uk Wed Oct 27 15:53:13 2004 From: dgm at ecs.soton.ac.uk (David G Mills) Date: Wed Oct 27 15:53:19 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? Message-ID: <Pine.LNX.4.44.0410271437170.5154-100000@login.ecs.soton.ac.uk> Hey there, I've noticed a potential bug and wanted to see if anyone else can replicate it, or has also seen it. When creating a socket with getaddrinfo, with the address family set to AF_UNSPEC and the host set to localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of the machine it binds to IPv6. I've tested the same code on two different machines, and the one running python 2.3.3 works as it should, but the python 2.3.4 doesn't. Anyone seen this? Can you replicate the error? Any help would be appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat 8 on the 2.3.3 machine. Cheers, David. From tcdelaney at optusnet.com.au Wed Oct 27 15:57:13 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Wed Oct 27 15:57:24 2004 Subject: [Python-Dev] On when making modifications to the CVS code References: <A128D751272CD411BC9200508BC2194D053C7AEF@escpl.tcp.com.ar> Message-ID: <01c801c4bc2c$dcd38890$0200a8c0@ryoko> Batista, Facundo wrote: >> My problem is that I can not decide to commit those changes. We're >> in beta, >> and don't know if the changes will be tested enough before the final >> release, in enough platforms (for now, it's tested on Linux, Win2k >> and MacOS X). I think those three platforms are sufficiently representative of Python users, so if it works on them, and the code looks good to a reviewer, it should be committed. It's not exactly a large patch after all ... What's the bug number? I've got a FreeBSD (5.2.1) virtual machine sitting around I could try it on (tomorrow - bed time now ;). Cheers. Tim Delaney From FBatista at uniFON.com.ar Wed Oct 27 15:58:57 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Oct 27 16:00:32 2004 Subject: [Python-Dev] On when making modifications to the CVS code Message-ID: <A128D751272CD411BC9200508BC2194D053C7AF2@escpl.tcp.com.ar> [Tim Delaney] #- I think those three platforms are sufficiently #- representative of Python #- users, so if it works on them, and the code looks good to a #- reviewer, it #- should be committed. It's not exactly a large patch after all ... Do you want to take a look at it? ;) #- What's the bug number? I've got a FreeBSD (5.2.1) virtual #- machine sitting #- around I could try it on (tomorrow - bed time now ;). The bug is 1050828. Thanks! . Facundo From skip at pobox.com Wed Oct 27 16:50:46 2004 From: skip at pobox.com (Skip Montanaro) Date: Wed Oct 27 16:50:50 2004 Subject: [Python-Dev] To run all exit handlers or not? Message-ID: <16767.46534.220081.62255@montanaro.dyndns.org> The current behavior of the atexit module is that if any of the exit handlers raises an exception the remaining handlers are not run. Greg Chapman posted a bug report about this: http://www.python.org/sf/1052242 Greg proposed catching any exceptions and continuing so that all exit handlers at least have a chance to run and Raymond agrees with him. I attached a patch to the ticket to add a flag to determine the behavior on the principle that atexit has been around long enough that someone out there probably relies on the early exit behavior. This is the old Python chestnut of using a flag to preserve existing behavior as the default while allowing users to set the flag to get the new behavior. I'm happy to go either way, but thought perhaps a quick poll of the troops might be in order, hence this note. Skip From gvanrossum at gmail.com Wed Oct 27 17:58:07 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed Oct 27 17:58:14 2004 Subject: [Python-Dev] To run all exit handlers or not? In-Reply-To: <16767.46534.220081.62255@montanaro.dyndns.org> References: <16767.46534.220081.62255@montanaro.dyndns.org> Message-ID: <ca471dc204102708586a2dcd9@mail.gmail.com> On Wed, 27 Oct 2004 09:50:46 -0500, Skip Montanaro <skip@pobox.com> wrote: > The current behavior of the atexit module is that if any of the exit > handlers raises an exception the remaining handlers are not run. Greg > Chapman posted a bug report about this: > > http://www.python.org/sf/1052242 > > Greg proposed catching any exceptions and continuing so that all exit > handlers at least have a chance to run and Raymond agrees with him. I > attached a patch to the ticket to add a flag to determine the behavior on > the principle that atexit has been around long enough that someone out there > probably relies on the early exit behavior. This is the old Python chestnut > of using a flag to preserve existing behavior as the default while allowing > users to set the flag to get the new behavior. > > I'm happy to go either way, but thought perhaps a quick poll of the troops > might be in order, hence this note. If I were a user of the atexit module and aware of this behavior, I'd consider it a bug write my atexit handler to avoid ever raising an exception, because I didn't want to jeopardize other atexit handlers. So I think preserving backwards-compatible behavior is unnecessary. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvanrossum at gmail.com Wed Oct 27 17:59:42 2004 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed Oct 27 17:59:46 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <000201c4bc01$9c331880$7c09a044@oemcomputer> References: <417F0276.7000209@activestate.com> <000201c4bc01$9c331880$7c09a044@oemcomputer> Message-ID: <ca471dc2041027085957dd696b@mail.gmail.com> > Why not mimic the pure python approach? > > lowerfunc = PyObject_GetAttrString(&PyUnicode_Type, "lower"); > PyObject_CallMethod(mylist, "sort", "OO", Py_None, lowerfunc); unicode.lower() doesn't like non-unicode objects, so this would only work if you know for sure there are no str objects in the list. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Wed Oct 27 19:53:26 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Oct 27 19:53:16 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? In-Reply-To: <Pine.LNX.4.44.0410271437170.5154-100000@login.ecs.soton.ac.uk> References: <Pine.LNX.4.44.0410271437170.5154-100000@login.ecs.soton.ac.uk> Message-ID: <417FE096.5070607@v.loewis.de> David G Mills wrote: > I've noticed a potential bug and wanted to see if anyone else can > replicate it, or has also seen it. When creating a socket with > getaddrinfo, with the address family set to AF_UNSPEC and the host set to > localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of > the machine it binds to IPv6. I've tested the same code on two different > machines, and the one running python 2.3.3 works as it should, but the > python 2.3.4 doesn't. Can you show the code fragment that you use? socket.getaddrinfo does never ever create sockets; only socket.socket does. So you apparently somehow use the results of getaddrinfo to create the socket. As getaddrinfo may return multiple results, you somehow need to select one of the results to create the socket, but you don't explain which result you use. A code snippet would answer the question. > Anyone seen this? Can you replicate the error? Any help would be > appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat > 8 on the 2.3.3 machine. I can imagine a code fragment where this happens, but I hardly can see an error here. Regards, Martin From ejones at uwaterloo.ca Wed Oct 27 20:25:36 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Wed Oct 27 20:25:44 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? In-Reply-To: <Pine.LNX.4.44.0410271437170.5154-100000@login.ecs.soton.ac.uk> References: <Pine.LNX.4.44.0410271437170.5154-100000@login.ecs.soton.ac.uk> Message-ID: <1098901536.4301.100.camel@localhost.localdomain> Welcome to the fun land of IPv6: Dealing with buggy and changing implementations. As far as I am aware, Python's socket module is basically just a wrapper around the operating system. The operating system behaviour changed, not Python's. On Wed, 2004-10-27 at 09:53, David G Mills wrote: > I've noticed a potential bug and wanted to see if anyone else can > replicate it, or has also seen it. When creating a socket with > getaddrinfo, with the address family set to AF_UNSPEC and the host set to > localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of > the machine it binds to IPv6. I've tested the same code on two different > machines, and the one running python 2.3.3 works as it should, but the > python 2.3.4 doesn't. What do you expect getaddrinfo to do? Return the IPv6 address first? That is very much up to the implementation. The man page states: > PF_UNSPEC in ai_family specifies any protocol > family (either IPv4 or IPv6, for example) This would imply that returning either an IPv4 or an IPv6 address would be correct. On my system, the following code returns a list of ipv6 addresses: import socket stuff = socket.getaddrinfo( "evanjones.ca", "http", socket.AF_UNSPEC ) print stuff [(10, 1, 6, '', ('2002:42a0:8738:1::1', 80, 0, 0)), (10, 1, 6, '', ('2001:470:1f01:ffff::151', 80, 0, 0)), (2, 1, 6, '', ('66.160.135.56', 80)) It does return the IPv6 addresses first, but I'm not sure that is intentional, or if it is pure chance. In your case, if you are looking up the name "localhost," it could be that /etc/hosts on one machine does not map "::1" to the name "localhost." Compare the contents of these files on both machines. My new Debian machine has "ip6-localhost" as "::1." If I change it so "::1" is also localhost, Python returns both addresses. The one thing that *is* a Python bug is that it does not include "IN6ADDR_LOOPBACK_INIT" or "in6addr_loopback" as constants in the socket module. Hmm... Maybe I should fix this... > Anyone seen this? Can you replicate the error? Any help would be > appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat > 8 on the 2.3.3 machine. These machines have different versions of glibc, which could also explain the behaviour change. In fact, there are very recent bug fixes to glibc. For example, this mailing list thread is a fix on top of glibc-2.3.3. My Debian Unstable machine is only running 2.3.2: http://sources.redhat.com/ml/libc-hacker/2004-09/msg00060.html On my system, /usr/share/doc/libc6/BUGS states: > [ *] Some of the functions which also handled IPv6 are currently broken. > IPv6 and IPv4 lookups occasionally happen when not needed. This > happens in getaddrinfo() and getnameinfo(). IPv4 handling of > these functions is OK though and there are patches available to fix > the IPv6 code as well. So basically, IPv6 is still very much a "work in progress." I suggest not relying on specific system behaviour. Hope this helps, Evan Jones From Scott.Daniels at Acm.Org Thu Oct 28 00:17:04 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu Oct 28 00:15:49 2004 Subject: [Python-Dev] Re: TRUNK UNFROZEN (release is done) In-Reply-To: <20041027014135.GA15116@rogue.amk.ca> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417EF4B8.8060100@interlink.com.au> <20041027014135.GA15116@rogue.amk.ca> Message-ID: <clp6mg$3kf$1@sea.gmane.org> A.M. Kuchling wrote: > The best weekends for a bug day might be Nov. 6 or 13th (between beta2 > and RC1), or this coming weekend (before beta2). I still won't have > time to run it, so someone will have to jump in. I suspect many of us in the USA may be doing some election work, so the 6th or 13th would be a lot better. -- -- Scott David Daniels Scott.Daniels@Acm.Org From tim.peters at gmail.com Thu Oct 28 06:31:52 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Oct 28 06:31:58 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <417F0276.7000209@activestate.com> References: <417F0276.7000209@activestate.com> Message-ID: <1f7befae04102721314c1d4feb@mail.gmail.com> [Trent Mick] > There is a subtlety in CreateProcess in the Win32 API in that if one > specifies an environment (via the lpEnvironment argument), the > environment strings (A) must be sorted alphabetically and (B) that sort > must be case-insensitive. Well, the docs are lying about Win95 (it doesn't sort envar blocks). On NT+, the sort must also be in "Unicode order, without regard to locale" -- WTF that means: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/changing_environment_variables.asp The best discussion I found: http://www.mail-archive.com/cygwin@cygwin.com/msg15239.html That claims NT+ uses RTLCompareUnicodeString to search for envars (and that function has a case-insensitive option). From martin at v.loewis.de Thu Oct 28 07:44:15 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Oct 28 07:44:04 2004 Subject: [Python-Dev] adding Py{String|Unicode}_{Lower|Upper} and fixing CreateProcess in _subprocess.pyd and PyWin32 In-Reply-To: <1f7befae04102721314c1d4feb@mail.gmail.com> References: <417F0276.7000209@activestate.com> <1f7befae04102721314c1d4feb@mail.gmail.com> Message-ID: <4180872F.3030500@v.loewis.de> Tim Peters wrote: > Well, the docs are lying about Win95 (it doesn't sort envar blocks). > On NT+, the sort must also be in "Unicode order, without regard to > locale" -- WTF that means: It means that the sorting uses ordinal character numbers, not the order characters have in the German alphabet. The interesting question then is what case-insensitive sorting by ordinal number means: you convert either to lower case, or to upper case (again, independent of locale), and sort by ordinal number after that conversion. Then the question is: lower case or upper case? as the order depends on that choice. I believe Windows always uses upper case for this kind of thing. At least NTFS sorts by upper case, by means of the $UpCase file. > That claims NT+ uses RTLCompareUnicodeString to search for envars (and > that function has a case-insensitive option). That sounds reasonable, and it should be used, then, because it will be impossible to emulate that function with Python library functions (if for no other reason that RTLCompareUnicodeString probably assumes Unicode 2.2, whereas Python only has the Unicode 3.2 database). Regards, Martin From greg at electricrain.com Thu Oct 28 10:09:01 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Thu Oct 28 10:09:06 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <41755658.8080806@v.loewis.de> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <41755658.8080806@v.loewis.de> Message-ID: <20041028080901.GA28925@zot.electricrain.com> On Tue, Oct 19, 2004 at 08:00:56PM +0200, "Martin v. L?wis" wrote: > Evan Jones wrote: > >Some posts to various lists [1] have stated that this is not a real > >problem because virtual memory takes care of it. This is fair if you > >are talking about a couple megabytes. In my case, I'm talking about > >~700 MB of wasted RAM, which is a problem. > > This is not true. The RAM is not wasted. As you explain later, the > pages will be swapped out to swap space, making the RAM available > again for other tasks. > > >First, this is wasting space > >which could be used for disk cache, which would improve the performance > >of my system. > > And indeed, this is what the operating system does for you: free the > memory (by swapping it out), then using the memory for disk cache, thus > improving performance of your system. In the long run on a system the RAM may not be wasted once the OS happens to have swapped it out but the address space is still used. You're still consuming ~700 MB of your OS's total address space with swapped garbage. The fact that ultimately a lot of it ends up on disk as swap is not nice to other processes wanting memory (and disk space for oses using a dynamic swap). That said, here's a workaround for avoiding permanent huge memory consumption in known workloads: fork() before doing the part that consumes a ton of memory. afterwards return the results, post huge memory consumption, via pipe to the waiting parent process and exit the child so the parent can continue on not consuming 700mb. From greg at electricrain.com Thu Oct 28 10:16:14 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Thu Oct 28 10:16:17 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 15, Issue 46 In-Reply-To: <B8D0DCCE-21DA-11D9-B920-0003938016AE@uwaterloo.ca> References: <20041019100214.297BB1E4013@bag.python.org> <c56e219d0410190532555e3381@mail.gmail.com> <6E2F71BA-21D1-11D9-B920-0003938016AE@uwaterloo.ca> <41751A4A.5070003@iinet.net.au> <B8D0DCCE-21DA-11D9-B920-0003938016AE@uwaterloo.ca> Message-ID: <20041028081614.GB28925@zot.electricrain.com> On Tue, Oct 19, 2004 at 10:25:27AM -0400, Evan Jones wrote: > 1. How can we run such an occasional task? It should be called > regularly, if Python is actively executing or not, just like a garbage > collector thread would be. I don't think making this collection part of > the normal call to allocate/free objects is a good idea, because that > would add some expensive overhead in what needs to be a fast path for > Python. Would attaching it to a less frequent level 1 garbage collection call work? (or even a level 2 if 1 is called too often) From greg at electricrain.com Thu Oct 28 10:40:01 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Thu Oct 28 10:40:37 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <79990c6b04102107236b590943@mail.gmail.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> <20041021094130.GA12799@mylene.ghaering.de> <79990c6b04102107236b590943@mail.gmail.com> Message-ID: <20041028084001.GC28925@zot.electricrain.com> On Thu, Oct 21, 2004 at 03:23:28PM +0100, Paul Moore wrote: > On Thu, 21 Oct 2004 11:41:30 +0200, Gerhard Haering <gh@ghaering.de> wrote: > > On Thu, Oct 21, 2004 at 11:26:12AM +0200, M.-A. Lemburg wrote: > > > Gerhard Haering wrote: > > > >My point is to include a usable DB-API 2.0 implementation that people > > > >can use as a starting point when developing applications that need a > > > >relational database. Other languages do the same btw. Java (win32?) > > > >includes a JDBC driver or ODBC, and PHP5 includes a SQLite module. > > > [...] > > > If you are just after a "usable database driver", then I have to > > > agree with Skip: any of the other available drivers would fit in > > > just as well. Please clarify this. > > > > I'm aiming at a usable DB-API implementation in the stdlib that does > > not need a server. I want Python to have an RDBMS interface that works > > OOTB, no administration required. SQLite seems the obvious choice to > > me, haven't looked at Gadfly in a while, and MySQLdb/MySQL embedded > > (GPL) has licensing issues (and adds megabytes to the Python binary > > download, instead of ca. 270 kB uncompressed as for SQLite). > > I'm +1 on including PySQLite in the core. It would fit in the same > space as Berkeley DB, *not* client-server databases like MySQL, > PostgreSQL, Oracle, etc. However, it conforms to 2 important > standards, SQL and the Python DB API, where Berkeley DB does not. Agreed. Hopefully including it would encourage the random people who have found the undocumented bsddb.dbtables module to use something saner. :) Along the same lines of including PySQLite it'd also be nice to consider a good database object abstraction module such as SqlObject (http://sqlobject.sf.net/). Anything to encourage people -not- to write raw SQL inline in their code is a good thing (and makes the app much more readable and even more portable as SQL is only partially so). > I don't think that the issue of batteries included vs easier package > installation is relevant here - at the moment, Python *is* "batteries > included". Also agreed. I personally think the pysqlite module bundled would get more use by more people than bsddb. -g From greg at electricrain.com Thu Oct 28 10:44:43 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Thu Oct 28 10:44:47 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format Message-ID: <20041028084443.GD28925@zot.electricrain.com> On Thu, Oct 21, 2004 at 12:49:19PM +0400, Oleg Broytmann wrote: > > BTW, just installing is not enough, even when it is come with Python > distribution. Installing a newer version of BerkeleyDB breaks older > databases due to incompatible file formats. fwiw its possible to fix this in the bsddb code. someone willing to do the exception catching database backing-up + auto db_upgrade() work should submit a patch. (i added backing-up as its rude to auto-convert someones database for them to a version their old software can no longer read) -g From tim.one at comcast.net Thu Oct 28 11:21:47 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Oct 28 11:21:54 2004 Subject: [Python-Dev] Pre-existing bug of the millennium (so far) Message-ID: <20041028092153.5B7371E4003@bag.python.org> This one came up while working on ZODB: weakref callback vs. gc vs. threads http://www.python.org/sf/1055820 Short course: in the presence of weakrefs, cyclic gc is still hosed (it turns out that neither threads nor weakref callbacks are necessary to get hosed). temp2a.py there demonstrates there's a problem, but in an unclear way (hundreds of objects, hundreds of weakrefs and weakref callbacks (all via WeakValueDictionary internals), 3 threads). OTOH, there's nothing clever or tricky about it. Sooner or later, it just fails (an accessible instance of a user-defined class gets its __dict__ cleared "by magic"). temp2b.py reduces it to 2 objects and 1 thread. This is contrived, but is deterministic. temp2c.py boosts it to 3 objects, and is a nightmare: it shows that the problem can occur during a gc collection that doesn't see *any* objects having a weakref with a callback. There is a weakref with a callback here, but it's attached to an object in an older generation, and collection of a younger generation triggers that callback indirectly. Because this is such a nasty case (no amount of analysis of the objects in the generation being collected can deduce that it's possible for a weakref callback to run), there are extensive comments and an ASCII-art diagram in the file. Even worse, temp2d.py shows we can get in trouble even if there's never a weakref with a callback. It's enough to have one weakref (without a callback), and one object with a __del__ method. Offhand, I don't have a plausible solution that will work. The elegant <wink> analysis in gc_weakref.txt missed what should have been obvious even then: cyclic trash is still potentially reachable via "global" weakrefs, so if any Python code whatsoever can run while gc is breaking cycles (whether via __del__ or via wr callback), global weakrefs can resurrect cyclic trash. That suggests some Draconian approaches. Anyone have a bright idea? It's remarkable how long we've managed to go without noticing that everything is disastrously broken here <0.9 wink>. From dgm at ecs.soton.ac.uk Thu Oct 28 11:37:09 2004 From: dgm at ecs.soton.ac.uk (David G Mills) Date: Thu Oct 28 11:37:21 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? In-Reply-To: <1098901536.4301.100.camel@localhost.localdomain> Message-ID: <Pine.LNX.4.44.0410281025390.5154-100000@login.ecs.soton.ac.uk> okay, here's my code: HOST = 'localhost' PORT = 55555 s = None for res in socket.getaddrinfo(HOST, PORT, socket.AF_UNSPEC, socket.SOCK_STREAM, 0, socket.AI_PASSIVE): af, socktype, proto, canonname, sa = res try: s = socket.socket(af, socktype, proto) except socket.error, msg: s = None continue try: s.bind(sa) s.listen(5) except socket.error, msg: s.close() s = None continue break First off I'd like to point out that when host is set to a fully qualified domain name, comp.foo.org for example, the correct behaviour is seen, getaddrinfo returns an IPv4 and IPv6 address, and when the socket is created, it binds to the IPv6 address. Only when localhost is used does it fail to find an IPv6 address. Now there's nothing wrong with the /etc/hosts file, as I tested with the one from the working machine. Also when using AF_INET6 and host=localhost it works fine, which is why I am puzzled. Thanks for the glibc information, I shall investigate down that path and see what I can discover, but if you can think of anything else that might be causing this I'd like to hear it. =) Regards, David. On Wed, 27 Oct 2004, Evan Jones wrote: > Welcome to the fun land of IPv6: Dealing with buggy and changing > implementations. As far as I am aware, Python's socket module is > basically just a wrapper around the operating system. The operating > system behaviour changed, not Python's. > > On Wed, 2004-10-27 at 09:53, David G Mills wrote: > > I've noticed a potential bug and wanted to see if anyone else can > > replicate it, or has also seen it. When creating a socket with > > getaddrinfo, with the address family set to AF_UNSPEC and the host set to > > localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of > > the machine it binds to IPv6. I've tested the same code on two different > > machines, and the one running python 2.3.3 works as it should, but the > > python 2.3.4 doesn't. > > What do you expect getaddrinfo to do? Return the IPv6 address first? > That is very much up to the implementation. The man page states: > > > PF_UNSPEC in ai_family specifies any protocol > > family (either IPv4 or IPv6, for example) > > This would imply that returning either an IPv4 or an IPv6 address would > be correct. On my system, the following code returns a list of ipv6 > addresses: > > import socket > stuff = socket.getaddrinfo( "evanjones.ca", "http", socket.AF_UNSPEC ) > print stuff > > [(10, 1, 6, '', ('2002:42a0:8738:1::1', 80, 0, 0)), (10, 1, 6, '', > ('2001:470:1f01:ffff::151', 80, 0, 0)), (2, 1, 6, '', ('66.160.135.56', > 80)) > > It does return the IPv6 addresses first, but I'm not sure that is > intentional, or if it is pure chance. In your case, if you are looking > up the name "localhost," it could be that /etc/hosts on one machine does > not map "::1" to the name "localhost." Compare the contents of these > files on both machines. My new Debian machine has "ip6-localhost" as > "::1." If I change it so "::1" is also localhost, Python returns both > addresses. > > The one thing that *is* a Python bug is that it does not include > "IN6ADDR_LOOPBACK_INIT" or "in6addr_loopback" as constants in the socket > module. Hmm... Maybe I should fix this... > > > Anyone seen this? Can you replicate the error? Any help would be > > appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat > > 8 on the 2.3.3 machine. > > These machines have different versions of glibc, which could also > explain the behaviour change. In fact, there are very recent bug fixes > to glibc. For example, this mailing list thread is a fix on top of > glibc-2.3.3. My Debian Unstable machine is only running 2.3.2: > > http://sources.redhat.com/ml/libc-hacker/2004-09/msg00060.html > > On my system, /usr/share/doc/libc6/BUGS states: > > > [ *] Some of the functions which also handled IPv6 are currently broken. > > IPv6 and IPv4 lookups occasionally happen when not needed. This > > happens in getaddrinfo() and getnameinfo(). IPv4 handling of > > these functions is OK though and there are patches available to fix > > the IPv6 code as well. > > So basically, IPv6 is still very much a "work in progress." I suggest > not relying on specific system behaviour. > > Hope this helps, > > Evan Jones > > From phd at mail2.phd.pp.ru Thu Oct 28 11:54:39 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Oct 28 11:54:42 2004 Subject: [Python-Dev] SQLObject module for Python 2.5 In-Reply-To: <20041028084001.GC28925@zot.electricrain.com> References: <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> <20041021094130.GA12799@mylene.ghaering.de> <79990c6b04102107236b590943@mail.gmail.com> <20041028084001.GC28925@zot.electricrain.com> Message-ID: <20041028095439.GB19204@phd.pp.ru> On Thu, Oct 28, 2004 at 01:40:01AM -0700, Gregory P. Smith wrote: > Along the same lines of including PySQLite it'd also be nice to > consider a good database object abstraction module such as SqlObject > (http://sqlobject.sf.net/). -1 for now. SQLObject shows a great promise, but the promise has to be fulfiled (yes, I did send a few patches)... Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From phd at mail2.phd.pp.ru Thu Oct 28 12:05:13 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Oct 28 12:05:18 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format In-Reply-To: <20041028084443.GD28925@zot.electricrain.com> References: <20041028084443.GD28925@zot.electricrain.com> Message-ID: <20041028100513.GA19996@phd.pp.ru> On Thu, Oct 28, 2004 at 01:44:43AM -0700, Gregory P. Smith wrote: > On Thu, Oct 21, 2004 at 12:49:19PM +0400, Oleg Broytmann wrote: > > BTW, just installing is not enough, even when it is come with Python > > distribution. Installing a newer version of BerkeleyDB breaks older > > databases due to incompatible file formats. > > fwiw its possible to fix this in the bsddb code. someone willing to do > the exception catching database backing-up + auto db_upgrade() What's db_upgrade()? I can find only a reference to a program called db_upgrade, and I don't have one on my computer. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From jim at zope.com Thu Oct 28 13:06:44 2004 From: jim at zope.com (Jim Fulton) Date: Thu Oct 28 13:06:48 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format In-Reply-To: <20041028084443.GD28925@zot.electricrain.com> References: <20041028084443.GD28925@zot.electricrain.com> Message-ID: <4180D2C4.60008@zope.com> Gregory P. Smith wrote: > On Thu, Oct 21, 2004 at 12:49:19PM +0400, Oleg Broytmann wrote: > >> BTW, just installing is not enough, even when it is come with Python >>distribution. Installing a newer version of BerkeleyDB breaks older >>databases due to incompatible file formats. > > > fwiw its possible to fix this in the bsddb code. someone willing to do > the exception catching database backing-up + auto db_upgrade() work should > submit a patch. (i added backing-up as its rude to auto-convert someones > database for them to a version their old software can no longer read) Automatically converting the database is a bad idea. bsddb does too much implicitly already, like installing. Wishing I could run the tests without segfaultingly yours, Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From skip at pobox.com Thu Oct 28 13:21:45 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 28 13:21:54 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format In-Reply-To: <20041028100513.GA19996@phd.pp.ru> References: <20041028084443.GD28925@zot.electricrain.com> <20041028100513.GA19996@phd.pp.ru> Message-ID: <16768.54857.137255.972793@montanaro.dyndns.org> >>>>> "Oleg" == Oleg Broytmann <phd@oper.phd.pp.ru> writes: Oleg> What's db_upgrade()? I can find only a reference to a program Oleg> called db_upgrade, and I don't have one on my computer. It's a program that comes with recent versions of Berkeley DB to upgrade files written in older formats to the current format (the format associated with the version of db_upgrade being run). Skip From skip at pobox.com Thu Oct 28 13:23:09 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 28 13:23:12 2004 Subject: [Python-Dev] To run all exit handlers or not? In-Reply-To: <ca471dc204102708586a2dcd9@mail.gmail.com> References: <16767.46534.220081.62255@montanaro.dyndns.org> <ca471dc204102708586a2dcd9@mail.gmail.com> Message-ID: <16768.54941.259265.283310@montanaro.dyndns.org> Guido> If I were a user of the atexit module and aware of this behavior, Guido> I'd consider it a bug [and] write my atexit handler to avoid ever Guido> raising an exception, because I didn't want to jeopardize other Guido> atexit handlers. So I think preserving backwards-compatible Guido> behavior is unnecessary. Okay. Will take that as a pronouncement... Skip From jim at zope.com Thu Oct 28 13:36:35 2004 From: jim at zope.com (Jim Fulton) Date: Thu Oct 28 13:36:39 2004 Subject: [Python-Dev] Re: Pre-existing bug of the millennium (so far) In-Reply-To: <20041028092152.5553F600A2@ns1.zope.com> References: <20041028092152.5553F600A2@ns1.zope.com> Message-ID: <4180D9C3.9090206@zope.com> Tim Peters wrote: ... > Anyone have a bright idea? It's remarkable how long we've managed to go > without noticing that everything is disastrously broken here <0.9 wink>. Sure. Clearing cyclic trash can call Python code. If there are weakrefs to any of the cyclic trash, then those wekrefs can be used to resurrect the objects. Therefore, *before* clearing cyclic trash, we need to remove any weakrefs. If any of the weakrefs being removed have callbacks, then we need to save the callbacks and call them *after* all of the weakrefs have been cleared. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From phd at mail2.phd.pp.ru Thu Oct 28 13:39:48 2004 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Thu Oct 28 13:39:51 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format In-Reply-To: <16768.54857.137255.972793@montanaro.dyndns.org> References: <20041028084443.GD28925@zot.electricrain.com> <20041028100513.GA19996@phd.pp.ru> <16768.54857.137255.972793@montanaro.dyndns.org> Message-ID: <20041028113948.GB23644@phd.pp.ru> On Thu, Oct 28, 2004 at 06:21:45AM -0500, Skip Montanaro wrote: > >>>>> "Oleg" == Oleg Broytmann <phd@oper.phd.pp.ru> writes: > Oleg> What's db_upgrade()? > > It's a program that comes with recent versions of Berkeley DB to upgrade > files written in older formats to the current format (the format associated > with the version of db_upgrade being run). So it has to be distributed with Python. I am very nervous about automagically running a program in the background upon catching a data format exception in bsddb module. What if the program fails? Where the module will put its stderr? popen+raise FormatError(caught_error_string)? I am in favour of including the program in the Python distribution (if its license allows inclusion) but not to run it automatically. This is the job for an application, not for the bsddb module. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From ndbecker2 at verizon.net Thu Oct 28 14:34:09 2004 From: ndbecker2 at verizon.net (Neal D. Becker) Date: Thu Oct 28 14:34:16 2004 Subject: [Python-Dev] logging needs better documentation Message-ID: <clqp01$h6o$1@sea.gmane.org> There is only a single example in the logging module documentation. Please add more examples! I'm trying to use DatagramHandler. The documentation on this is not adequate. There is a confusing mention of makeLogRecord. No example. I finally stumbled across this: data, address = sock.recvfrom (8192) rec = logging.makeLogRecord (pickle.loads(data[4:])) I can't find _anywhere_ a mention that you need to leave off the first 4 bytes in order for unpickle to work. From ejones at uwaterloo.ca Thu Oct 28 14:34:38 2004 From: ejones at uwaterloo.ca (Evan Jones) Date: Thu Oct 28 14:34:49 2004 Subject: [Python-Dev] Changing pymalloc behaviour for long running processes In-Reply-To: <20041028080901.GA28925@zot.electricrain.com> References: <746A537C-2183-11D9-B920-0003938016AE@uwaterloo.ca> <41755658.8080806@v.loewis.de> <20041028080901.GA28925@zot.electricrain.com> Message-ID: <1098966878.4277.5.camel@localhost.localdomain> On Thu, 2004-10-28 at 04:09, Gregory P. Smith wrote: > That said, here's a workaround for avoiding permanent huge memory > consumption in known workloads: > > fork() before doing the part that consumes a ton of memory. afterwards > return the results, post huge memory consumption, via pipe to the waiting > parent process and exit the child so the parent can continue on not > consuming 700mb. Right: This certainly is an effective workaround, but only for very specific, memory consuming tasks. Python should be better than to inflict this kind of hack on programmers. We shouldn't have to worry about an implementation detail of the Python interpreter. I don't believe that Jython has this problem, but it has been a while since I looked at it, so I could be wrong. I'm still planning on working on this issue: I just need to find the time. Evan Jones From dgm at ecs.soton.ac.uk Thu Oct 28 16:02:04 2004 From: dgm at ecs.soton.ac.uk (David G Mills) Date: Thu Oct 28 16:02:07 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? - found the real cause In-Reply-To: <1098901536.4301.100.camel@localhost.localdomain> Message-ID: <Pine.LNX.4.44.0410281457160.6858-100000@login.ecs.soton.ac.uk> As you suggested I investigated glibc bugs and found that when getaddrinfo uses AF_UNSPEC and looks up an address, it first looks in /etc/hosts and if it finds an IPv4 address it will just return that and never return the IPv6 address. The stop gap fix, before patching glbic, is to just remove any IPv4 entries in /etc/hosts forcing it to do a DNS lookup. Thanks for your help in this matter. Regards, David. On Wed, 27 Oct 2004, Evan Jones wrote: > Welcome to the fun land of IPv6: Dealing with buggy and changing > implementations. As far as I am aware, Python's socket module is > basically just a wrapper around the operating system. The operating > system behaviour changed, not Python's. > > On Wed, 2004-10-27 at 09:53, David G Mills wrote: > > I've noticed a potential bug and wanted to see if anyone else can > > replicate it, or has also seen it. When creating a socket with > > getaddrinfo, with the address family set to AF_UNSPEC and the host set to > > localhost, the socket binds to IPv4 instead of IPv6. When using the FQD of > > the machine it binds to IPv6. I've tested the same code on two different > > machines, and the one running python 2.3.3 works as it should, but the > > python 2.3.4 doesn't. > > What do you expect getaddrinfo to do? Return the IPv6 address first? > That is very much up to the implementation. The man page states: > > > PF_UNSPEC in ai_family specifies any protocol > > family (either IPv4 or IPv6, for example) > > This would imply that returning either an IPv4 or an IPv6 address would > be correct. On my system, the following code returns a list of ipv6 > addresses: > > import socket > stuff = socket.getaddrinfo( "evanjones.ca", "http", socket.AF_UNSPEC ) > print stuff > > [(10, 1, 6, '', ('2002:42a0:8738:1::1', 80, 0, 0)), (10, 1, 6, '', > ('2001:470:1f01:ffff::151', 80, 0, 0)), (2, 1, 6, '', ('66.160.135.56', > 80)) > > It does return the IPv6 addresses first, but I'm not sure that is > intentional, or if it is pure chance. In your case, if you are looking > up the name "localhost," it could be that /etc/hosts on one machine does > not map "::1" to the name "localhost." Compare the contents of these > files on both machines. My new Debian machine has "ip6-localhost" as > "::1." If I change it so "::1" is also localhost, Python returns both > addresses. > > The one thing that *is* a Python bug is that it does not include > "IN6ADDR_LOOPBACK_INIT" or "in6addr_loopback" as constants in the socket > module. Hmm... Maybe I should fix this... > > > Anyone seen this? Can you replicate the error? Any help would be > > appreciated. Oh yeah, I'm running on FC2 on the 2.3.4 machine and redhat > > 8 on the 2.3.3 machine. > > These machines have different versions of glibc, which could also > explain the behaviour change. In fact, there are very recent bug fixes > to glibc. For example, this mailing list thread is a fix on top of > glibc-2.3.3. My Debian Unstable machine is only running 2.3.2: > > http://sources.redhat.com/ml/libc-hacker/2004-09/msg00060.html > > On my system, /usr/share/doc/libc6/BUGS states: > > > [ *] Some of the functions which also handled IPv6 are currently broken. > > IPv6 and IPv4 lookups occasionally happen when not needed. This > > happens in getaddrinfo() and getnameinfo(). IPv4 handling of > > these functions is OK though and there are patches available to fix > > the IPv6 code as well. > > So basically, IPv6 is still very much a "work in progress." I suggest > not relying on specific system behaviour. > > Hope this helps, > > Evan Jones > > From ianb at colorstudy.com Thu Oct 28 17:41:40 2004 From: ianb at colorstudy.com (Ian Bicking) Date: Thu Oct 28 17:42:31 2004 Subject: [Python-Dev] SQLite module for Python 2.5 In-Reply-To: <20041028084001.GC28925@zot.electricrain.com> References: <FE9E6BEA-22BB-11D9-A2F8-000A95BA5446@redivi.com> <20041020162905.GA9843@mylene.ghaering.de> <41769AC1.4020603@egenix.com> <5.1.1.6.0.20041020205042.0260a280@mail.telecommunity.com> <16759.12986.759165.964094@montanaro.dyndns.org> <4177685B.4060905@colorstudy.com> <20041021084630.GB12496@mylene.ghaering.de> <417780B4.3070107@egenix.com> <20041021094130.GA12799@mylene.ghaering.de> <79990c6b04102107236b590943@mail.gmail.com> <20041028084001.GC28925@zot.electricrain.com> Message-ID: <41811334.4070601@colorstudy.com> Gregory P. Smith wrote: > Along the same lines of including PySQLite it'd also be nice to > consider a good database object abstraction module such as SqlObject > (http://sqlobject.sf.net/). Anything to encourage people -not- to write > raw SQL inline in their code is a good thing (and makes the app much > more readable and even more portable as SQL is only partially so). SQLObject isn't really mature enough, and seems too complex to really be right for the standard library. Maybe something like dbrow, though. But most low-level database libraries or extensions really belong as part of the DB API, and the DB API is implement on a per-backend basis. Most high-level libraries are outside the scope of the standard library. dbrow is kind of an anomoly, being both low-level and fairly database-neutral. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org From mchermside at ingdirect.com Thu Oct 28 18:23:16 2004 From: mchermside at ingdirect.com (Chermside, Michael) Date: Thu Oct 28 18:23:17 2004 Subject: [Python-Dev] logging needs better documentation Message-ID: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> Neal Becker writes: > There is only a single example in the logging module documentation. Please > add more examples! You are completely right... I am sure that those docs need to be improved. Would you be willing to write up some improved docs? If so, submit them to the patch manager on SourceForge, and we'll incorporate them into the next release possible (probably Python 2.5). If not, then it may have to wait until someone else has time to address it. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. From fdrake at acm.org Thu Oct 28 18:30:55 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Oct 28 18:31:04 2004 Subject: [Python-Dev] logging needs better documentation In-Reply-To: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> Message-ID: <200410281230.55464.fdrake@acm.org> On Thursday 28 October 2004 12:23 pm, Chermside, Michael wrote: > Would you be willing to write up some improved docs? If so, submit them > to the patch manager on SourceForge, and we'll incorporate them into the > next release possible (probably Python 2.5). If not, then it may have to > wait until someone else has time to address it. Wrong or missing documentation is considered a bug, so improvements can (and should) be incorporated into 2.4 and backported to relevant maintenance branches. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From fdrake at acm.org Thu Oct 28 18:43:40 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Oct 28 18:43:50 2004 Subject: [Python-Dev] logging needs better documentation In-Reply-To: <clqp01$h6o$1@sea.gmane.org> References: <clqp01$h6o$1@sea.gmane.org> Message-ID: <200410281243.40850.fdrake@acm.org> On Thursday 28 October 2004 08:34 am, Neal D. Becker wrote: > There is only a single example in the logging module documentation. > Please add more examples! If you'd like to provide some that would have helped you, please pass them along. > I'm trying to use DatagramHandler. The documentation on this is not > adequate. There is a confusing mention of makeLogRecord. No example. > > I finally stumbled across this: > > data, address = sock.recvfrom (8192) > rec = logging.makeLogRecord (pickle.loads(data[4:])) > > I can't find _anywhere_ a mention that you need to leave off the first 4 > bytes in order for unpickle to work. Regardless of whether you have time to provide specific documentation patches, a bug report should be filed describing this omission and providing any specific suggestions you can make about improvements you'd like to see. Documentation bug reports can be filed in the Python bug tracker: http://sourceforge.net/tracker/?group_id=5470&atid=105470 -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From skip at pobox.com Thu Oct 28 19:19:24 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 28 19:19:47 2004 Subject: [Python-Dev] logging needs better documentation In-Reply-To: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> Message-ID: <16769.10780.180658.635437@montanaro.dyndns.org> Michael> Neal Becker writes: >> There is only a single example in the logging module documentation. >> Please add more examples! Michael> You are completely right... I am sure that those docs need to Michael> be improved. Michael> Would you be willing to write up some improved docs? In an attempt to not discourage Neal, wholesale rewrites are not necessary. If you can submit plain text (or ReST) of an (expanded?) Examples section that would be great. One of us python-dev peons can convert your submission to LaTeX easily enough. (And Fred can correct our mistakes.) Skip From ndbecker2 at verizon.net Thu Oct 28 19:25:07 2004 From: ndbecker2 at verizon.net (Neal D. Becker) Date: Thu Oct 28 19:25:34 2004 Subject: [Python-Dev] RE: logging needs better documentation References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> <16769.10780.180658.635437@montanaro.dyndns.org> Message-ID: <clra1j$8td$2@sea.gmane.org> Skip Montanaro wrote: > > Michael> Neal Becker writes: > >> There is only a single example in the logging module documentation. > >> Please add more examples! > > Michael> You are completely right... I am sure that those docs need to > Michael> be improved. > > Michael> Would you be willing to write up some improved docs? > > In an attempt to not discourage Neal, wholesale rewrites are not > necessary. If you can submit plain text (or ReST) of an (expanded?) > Examples section > that would be great. One of us python-dev peons can convert your > submission > to LaTeX easily enough. (And Fred can correct our mistakes.) > You do realize, that this is the first time I even looked at logging, and I'm learning a few things with more effort than I think it should take. The fact remains, that I now have about 2 hours worth of experience with it, so don't you think I'm about the last person you should ask to write the doc? From skip at pobox.com Thu Oct 28 19:58:20 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Oct 28 19:58:29 2004 Subject: [Python-Dev] RE: logging needs better documentation In-Reply-To: <clra1j$8td$2@sea.gmane.org> References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> <16769.10780.180658.635437@montanaro.dyndns.org> <clra1j$8td$2@sea.gmane.org> Message-ID: <16769.13116.866605.646746@montanaro.dyndns.org> >> In an attempt to not discourage Neal, wholesale rewrites are not >> necessary. If you can submit plain text (or ReST) of an (expanded?) >> Examples section that would be great. One of us python-dev peons can >> convert your submission to LaTeX easily enough. (And Fred can >> correct our mistakes.) >> Neal> You do realize, that this is the first time I even looked at Neal> logging, and I'm learning a few things with more effort than I Neal> think it should take. The fact remains, that I now have about 2 Neal> hours worth of experience with it, so don't you think I'm about Neal> the last person you should ask to write the doc? Don't shortchange yourself. The sorts of examples that would be useful to a rank beginner are often the most useful. Skip From martin at v.loewis.de Thu Oct 28 20:10:42 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Oct 28 20:10:48 2004 Subject: [Python-Dev] IPv6 bug in 2.3.4?? In-Reply-To: <Pine.LNX.4.44.0410281025390.5154-100000@login.ecs.soton.ac.uk> References: <Pine.LNX.4.44.0410281025390.5154-100000@login.ecs.soton.ac.uk> Message-ID: <41813622.3010600@v.loewis.de> David G Mills wrote: > First off I'd like to point out that when host is set to a fully > qualified domain name, comp.foo.org for example, the correct behaviour is > seen, getaddrinfo returns an IPv4 and IPv6 address, and when the socket is > created, it binds to the IPv6 address. Only when localhost is used does it > fail to find an IPv6 address. Now there's nothing wrong with the > /etc/hosts file, as I tested with the one from the working machine. Also > when using AF_INET6 and host=localhost it works fine, which is why I am > puzzled. Ah, ok. It wasn't quite clear to me what specifically you consider an error. As Evan Jones says, Python returns the data in the very same way in which the C library returns them > Thanks for the glibc information, I shall investigate down that path > and see what I can discover, but if you can think of anything else that > might be causing this I'd like to hear it. =) On my system, it would be /etc/hosts, which reads 127.0.0.1 localhost mira ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts So I find it perfectly ok that "localhost" only resolves to IPv4. Regards, Martin From tim.peters at gmail.com Thu Oct 28 20:21:37 2004 From: tim.peters at gmail.com (Tim Peters) Date: Thu Oct 28 20:21:54 2004 Subject: [Python-Dev] Re: Pre-existing bug of the millennium (so far) In-Reply-To: <4180D9C3.9090206@zope.com> References: <20041028092152.5553F600A2@ns1.zope.com> <4180D9C3.9090206@zope.com> Message-ID: <1f7befae04102811213f7ec922@mail.gmail.com> [Tim] > ... >> Anyone have a bright idea? It's remarkable how long we've managed to go >> without noticing that everything is disastrously broken here <0.9 wink>. [Jim] > Sure. Clearing cyclic trash can call Python code. If there > are weakrefs to any of the cyclic trash, then those wekrefs can > be used to resurrect the objects. Therefore, *before* clearing cyclic > trash, we need to remove any weakrefs. If any of the weakrefs > being removed have callbacks, then we need to save the callbacks > and call them *after* all of the weakrefs have been cleared. Thanks! Neil Schemenauer and I got together today, and talked each other into believing this can be made to work, and that it's likely the best solution too. The devil's in the details, but this a generalization of the last round of weakref vs. gc bugfixes, which got the effect of what you suggest here except only for cyclic-trash weakref objects with weakref callbacks. It needs generalization to all weakly-referenced cyclic-trash objects. Still unsure how hard that will be to code up, but am sure it will fix all the test cases I concocted. From ncoghlan at iinet.net.au Fri Oct 29 03:48:22 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri Oct 29 03:46:39 2004 Subject: [Python-Dev] RE: logging needs better documentation In-Reply-To: <16769.13116.866605.646746@montanaro.dyndns.org> References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> <16769.10780.180658.635437@montanaro.dyndns.org> <clra1j$8td$2@sea.gmane.org> <16769.13116.866605.646746@montanaro.dyndns.org> Message-ID: <4181A166.8010604@iinet.net.au> Skip Montanaro wrote: > Neal> You do realize, that this is the first time I even looked at > Neal> logging, and I'm learning a few things with more effort than I > Neal> think it should take. The fact remains, that I now have about 2 > Neal> hours worth of experience with it, so don't you think I'm about > Neal> the last person you should ask to write the doc? > > Don't shortchange yourself. The sorts of examples that would be useful to a > rank beginner are often the most useful. I'm with Skip - even a clear & concise description of what you found most confusing can help with the update process. Updating the docs to clarify a couple of specific points (or add in a few specific usage examples) is much easier (and far more likely to get done) than a general attempt to "improve the logging documentation". Cheers, Nick. From fdrake at acm.org Fri Oct 29 04:00:23 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 29 04:00:43 2004 Subject: [Python-Dev] RE: logging needs better documentation In-Reply-To: <clra1j$8td$2@sea.gmane.org> References: <0CFFADBB825C6249A26FDF11C1772AE10167A97B@ingdexj1.ingdirect.com> <16769.10780.180658.635437@montanaro.dyndns.org> <clra1j$8td$2@sea.gmane.org> Message-ID: <200410282200.23214.fdrake@acm.org> On Thursday 28 October 2004 01:25 pm, Neal D. Becker wrote: > You do realize, that this is the first time I even looked at logging, and > I'm learning a few things with more effort than I think it should take. > The fact remains, that I now have about 2 hours worth of experience with > it, so don't you think I'm about the last person you should ask to write > the doc? Or perhaps the first? You know what you had trouble with, and what information would have helped you. If you manage to answer your questions about logging (with or without help), you'll know what it takes to answer the questions you had that weren't answered by the documentation. Having working answers in the docs is better than not having answers; we can refine later as needed. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From anthony at interlink.com.au Fri Oct 29 07:58:04 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri Oct 29 07:58:36 2004 Subject: [Python-Dev] Re: Pre-existing bug of the millennium (so far) In-Reply-To: <1f7befae04102811213f7ec922@mail.gmail.com> References: <20041028092152.5553F600A2@ns1.zope.com> <4180D9C3.9090206@zope.com> <1f7befae04102811213f7ec922@mail.gmail.com> Message-ID: <4181DBEC.1060302@interlink.com.au> It's not clear to me as to how severe a problem this is - given that beta2 is scheduled for next Wednesday, should I be delaying it? Anthony From tim.peters at gmail.com Fri Oct 29 08:24:20 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Oct 29 08:24:33 2004 Subject: [Python-Dev] Re: Pre-existing bug of the millennium (so far) In-Reply-To: <4181DBEC.1060302@interlink.com.au> References: <20041028092152.5553F600A2@ns1.zope.com> <4180D9C3.9090206@zope.com> <1f7befae04102811213f7ec922@mail.gmail.com> <4181DBEC.1060302@interlink.com.au> Message-ID: <1f7befae04102823246ce12b74@mail.gmail.com> [Anthony Baxter] > It's not clear to me as to how severe a problem this is - It's a critical problem -- that's why I set the priority to 9. It affects all versions of Python since weakrefs were introduced. If you don't use weakrefs, or use any libraries that use weakrefs, it won't bite you. If you do, symptoms range from nothing, through "impossible AttributeError"s at seemingly random times, to segfaults. > given that beta2 is scheduled for next Wednesday, should I be delaying it? I don't think so. On my home box, the Python test suite contains new tests for all the scenarios in the bug report, and I have a Python that passes the -uall test suite in debug and release builds. So it's looking good. I'm too punchy to check it in now, but if more tests pass I'll upload a patch to SF before I go to sleep, and assign it to Neil for review. The interactions between cyclic gc and weakrefs are arguably delicate <heh>. From vinay_sajip at red-dove.com Fri Oct 29 13:17:41 2004 From: vinay_sajip at red-dove.com (Vinay Sajip at Red Dove) Date: Fri Oct 29 13:18:02 2004 Subject: [Python-Dev] RE: logging needs better documentation Message-ID: <000801c4bda8$eb31ee90$0200a8c0@DELTA> Hi Neal, I quite agree that the documentation for the logging package can be improved. As others have said in response to your post, you can definitely help by indicating more specifically where you find the documentation lacking, with a patch if possible. For example, your initial post about the length prefix for pickles is documented in the docstring for SocketHandler.makePickle(): "Pickles the record in binary format with a length prefix, and returns it ready for transmission across the socket." I agree that the documentation does not mention specifically that the length is encoded as four bytes, or that it is packed using struct.pack(), or exactly how you unpack it. I also agree that more examples would be helpful. I will endeavour to improve the situation insofar as time allows. Patches and specific suggestions from users, especially new users like you, will be a definite help to me. Best Regards, Vinay Sajip -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20041029/a2235156/attachment.htm From ndbecker2 at verizon.net Fri Oct 29 13:30:50 2004 From: ndbecker2 at verizon.net (Neal D. Becker) Date: Fri Oct 29 13:30:58 2004 Subject: [Python-Dev] RE: logging needs better documentation References: <000801c4bda8$eb31ee90$0200a8c0@DELTA> Message-ID: <clt9la$1ll$1@sea.gmane.org> Vinay Sajip at Red Dove wrote: > Hi Neal, > > I quite agree that the documentation for the logging package can be > improved. As others have said in response to your post, you can definitely > help by indicating more specifically where you find the documentation > lacking, with a patch if possible. For example, your initial post about > the length prefix for pickles is documented in the docstring for > SocketHandler.makePickle(): > > "Pickles the record in binary format with a length prefix, and returns it > ready for transmission across the socket." > > I agree that the documentation does not mention specifically that the > length is encoded as four bytes, or that it is packed using struct.pack(), > or exactly how you unpack it. I also agree that more examples would be > helpful. I will endeavour to improve the situation insofar as time allows. > Patches and specific suggestions from users, especially new users like > you, will be a definite help to me. > Thanks. In this case at least, just an example was all I needed. Perhaps this example is a little larger than you would want? It is a system for collecting simulation results from a cluster and logging onto a server. The clients send periodic datagrams with fqdn, pid, argv and a result. The server updates a file whose name is .../fqdn:pid. The parts I had trouble with were: 1) the 4 extra bytes in the pickle 2) the fact that formatter needed to be on the server side, not the client 3) the apparantly undocumented fact that most (but not all!) handlers are in logging.handlers. ,----[ /disk1/nbecker/shannon2/nonlinear/logserver.py ] | #!/usr/bin/env python | | import socket | import logging | import pickle | import string | import os | | def Update (fname, msg): | newname = fname+".new" | f = open (newname, "w") | f.write (str(msg)) | f.close() | oldname = fname+".old" | try: | os.link (fname, oldname) | except: | pass | os.rename (newname, fname) | try: | os.unlink (oldname) | except: | pass | | class MyHandler (logging.Handler): | def __init__ (self, path): | logging.Handler.__init__ (self) | self.path = path | try: | os.makedirs (path, 0775) | except: | pass | | def emit (self, rec): | form = self.format (rec) | print form | els = string.split (form, "|") | date = els[0] | fqdn = els[2] | argv = els[3] | pid = els[4] | msg = els[5] | | fname = self.path+'/'+fqdn+':'+pid | Update (fname, [argv,msg]) | | sock = socket.socket (socket.AF_INET, socket.SOCK_DGRAM) | sock.bind (("", 8881)) | | logger = logging.getLogger() | hdlr = MyHandler("/tmp/logger") | formatter = logging.Formatter('%(asctime)s|%(levelname)s|%(message)s') | hdlr.setFormatter(formatter) | | try: | while (True): | data, address = sock.recvfrom (8192) | rec = logging.makeLogRecord (pickle.loads(data[4:])) | hdlr.emit (rec) | | finally: | sock.close() `---- ,----[ /disk1/nbecker/shannon2/nonlinear/logclient.py ] | #!/usr/bin/env python | | import logging | import logging.handlers | import sys | import os | import socket | | class logclient (object): | def __init__ (self): | self.logger = logging.getLogger() | hdlr = logging.handlers.DatagramHandler ('myhostname.com', | 8881) self.logger.addHandler(hdlr) | self.logger.setLevel(logging.INFO) | | def __call__(self, msg): | self.logger.info ("%s|%s|%s|%s" % | (socket.getfqdn(),sys.argv,os.getpid(),msg)) | | if (__name__ == "__main__"): | c = logclient() | c (str (["hello"])) `---- From jlg at dds.nl Fri Oct 29 16:51:35 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Fri Oct 29 16:49:23 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <20041027014135.GA15116@rogue.amk.ca> References: <41728467.2090504@interlink.com.au> <417D6DA8.90702@zope.com> <417E800C.9050400@v.loewis.de> <417E89D0.80305@zope.com> <417E8C51.5090506@v.loewis.de> <417E9BCC.3080702@zope.com> <417EAD13.3090309@v.loewis.de> <417EB014.4050604@zope.com> <417EF4B8.8060100@interlink.com.au> <20041027014135.GA15116@rogue.amk.ca> Message-ID: <20041029145135.GB10557@surfboy> On Tue, Oct 26, 2004 at 09:41:35PM -0400, A.M. Kuchling wrote: > The best weekends for a bug day might be Nov. 6 or 13th (between beta2 > and RC1), or this coming weekend (before beta2). I still won't have > time to run it, so someone will have to jump in. I finally decided to stop postponing doing this. I'll be running a bug day on November 7. I'll be in our office with some coworkers: any Dutch Pythoneers are welcome to join us for some 'real life' Python hacking! Cheers, Johannes From mcherm at mcherm.com Fri Oct 29 18:33:51 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Oct 29 18:33:40 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) Message-ID: <1099067631.418270ef81671@mcherm.com> Johannes writes: > I finally decided to stop postponing doing this. I'll be running a bug day on > November 7. I'll be in our office with some coworkers: any Dutch Pythoneers are > welcome to join us for some 'real life' Python hacking! Just a thought: should we consider doing a "patch review day" someday instead of a bug day? At a time like this we might be dividing patches into categories like "apply for 2.4", "apply for 2.5", "reject", "needs work", and "contraversial". -- Michael Chermside From nas at arctrix.com Fri Oct 29 19:24:34 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Fri Oct 29 19:24:37 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> Message-ID: <20041029172434.GA23030@mems-exchange.org> [Taking this to email. Carrying out discussions via the SF bug tracker sucks.] Comment By: Tim Peters (tim_one) > [Neil] > > I had to change _PyWeakref_ClearRef() since it was also > > clearing the weakref list of the trash object. > > That was really its *purpose*. If a trash weakref with a > callback isn't removed from the referent's list of weakrefs, > then the callback will trigger when PyObject_ClearWeakRefs() > is invoked on the referent. The purpose of > _PyWeakref_ClearRef() was to ensure that the callback never > triggers. But it's okay of the callback triggers, as long as the callback doesn't reference trash. > > Now it just sets wr_object to Py_None. > > That won't stop the callback from triggering. It also means > (see earlier comment) that PyObject_ClearWeakRefs() will > never removed the weakref from the list either, although I'm > not sure that does real harm. I'm trying to figure out PyObject_ClearWeakRefs() right now. > > I also made some serious simplifications to gcmodule by > > just treating trash weakref objects with callbacks the same > > as objects with __del__ methods (i.e. move them to the > > finalizers list and then do the transitive closure of that set). > > Does that mean they can end up in gc.garbage too? If so, I > don't think that's sellable. I think so. That can be easily changed though. What we can't do is invoke those callbacks. Neil From amk at amk.ca Fri Oct 29 19:45:53 2004 From: amk at amk.ca (A.M. Kuchling) Date: Fri Oct 29 19:48:47 2004 Subject: [Python-Dev] TRUNK UNFROZEN (release is done) In-Reply-To: <1099067631.418270ef81671@mcherm.com> References: <1099067631.418270ef81671@mcherm.com> Message-ID: <20041029174553.GA1688@rogue.amk.ca> On Fri, Oct 29, 2004 at 09:33:51AM -0700, Michael Chermside wrote: > Just a thought: should we consider doing a "patch review day" someday instead > of a bug day? At a time like this we might be dividing patches into categories > like "apply for 2.4", "apply for 2.5", "reject", "needs work", and > "contraversial". At past bug days people have both looked at bugs and processed some patches. Going through and marking selected patches as critical for 2.4 would be very useful right now, but I don't know if Anthony or someone else has already done that. --amk From tim.peters at gmail.com Fri Oct 29 20:10:08 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Oct 29 20:10:20 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <20041029172434.GA23030@mems-exchange.org> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <20041029172434.GA23030@mems-exchange.org> Message-ID: <1f7befae041029111046d2528@mail.gmail.com> [Neil] >>> I had to change _PyWeakref_ClearRef() since it was also >>> clearing the weakref list of the trash object. [Tim] >> That was really its *purpose*. If a trash weakref with a >> callback isn't removed from the referent's list of weakrefs, >> then the callback will trigger when PyObject_ClearWeakRefs() >> is invoked on the referent. The purpose of >> _PyWeakref_ClearRef() was to ensure that the callback never >> triggers. [Neil] > But it's okay of the callback triggers, as long as the callback > doesn't reference trash. Indeed, the callback must trigger if a weakref W's referent is in CT (cyclic trash) but W is not in CT (and W has a callback, of course). If W is also in CT, it's not necessary to invoke the callback, and it can be disastrous to invoke it (that was the key to fixing the *last* pile of bugs, which were subtler than this pile). The meaning of "clearing a weakref" has become ambiguous -- so far, to me (and I believe to Fred too) it's always meant both "set wr_object to Py_None" and "remove the weakref from its referent's list of weakrefs". You're sometimes using it to mean only the former. That may be a clever thing to do -- unsure yet. >>> Now it just sets wr_object to Py_None. >> That won't stop the callback from triggering. It also means >> (see earlier comment) that PyObject_ClearWeakRefs() will >> never remove the weakref from the list either, although I'm >> not sure that does real harm. > I'm trying to figure out PyObject_ClearWeakRefs() right now. For each weakref W in the object's list of weakrefs, it clears W (in the dual sense above), and invokes W's callback (if any). If W->wr_object is Py_None, the callback is still invoked, but W isn't removed from its referent's list of weakrefs then. I don't think Fred intended that wr_object could be None at the same time wr_callback is non-NULL (Fred?), so the consequences of doing "half a clear" are only half clear <wink>. >>> I also made some serious simplifications to gcmodule by >>> just treating trash weakref objects with callbacks the same >>> as objects with __del__ methods (i.e. move them to the >>> finalizers list and then do the transitive closure of that set). >> Does that mean they can end up in gc.garbage too? If so, I >> don't think that's sellable. > I think so. That can be easily changed though. What we can't do is > invoke those callbacks. Some callbacks, yes, but others must be invoked. For example, in any sensible use of a weak dictionary, it's vital that callbacks on weakrefs to cyclic trash get invoked; the weakrefs themselves aren't in CT in such cases (the weak dict holds strong references to the weakrefs, so for as long as the weak dict isn't trash, the weakrefs aren't trash either). It would be great if you've found a simpler way to get this all working. The most sensitive tests for this stuff are test_gc (with the diffs from my patch on SF), test_weakref, and test_descr. But they're not verifying that things don't end up in gc.garbage, so that's something to look out for. From tim.peters at gmail.com Fri Oct 29 20:36:20 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Oct 29 20:36:27 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <1f7befae041029111046d2528@mail.gmail.com> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <20041029172434.GA23030@mems-exchange.org> <1f7befae041029111046d2528@mail.gmail.com> Message-ID: <1f7befae041029113630087070@mail.gmail.com> [Tim] ... > The most sensitive tests for this stuff are test_gc (with the diffs from > my patch on SF), test_weakref, and test_descr. BTW, use a debug build of Python. Some of the relevant tests set up scenarios so excruciatingly delicate that the only reliable symptom when they fail is triggering a C-level assert (whatever->mro != NULL is most common). In a release build, failure may or may not be visible (it usually is, but sometimes an insane object can hide its insanity temporarily in platform-specific ways). From tim.peters at gmail.com Fri Oct 29 21:59:37 2004 From: tim.peters at gmail.com (Tim Peters) Date: Fri Oct 29 21:59:47 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <1f7befae041029111046d2528@mail.gmail.com> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <20041029172434.GA23030@mems-exchange.org> <1f7befae041029111046d2528@mail.gmail.com> Message-ID: <1f7befae0410291259db538cb@mail.gmail.com> [Tim] ... > The meaning of "clearing a weakref" has become ambiguous -- so far, to > me (and I believe to Fred too) it's always meant both "set wr_object > to Py_None" and "remove the weakref from its referent's list of > weakrefs". You're sometimes using it to mean only the former. That > may be a clever thing to do -- unsure yet. It has one downside that seems rare (the full Zope3 test suite provokes it, but not Python's): say a weakref W "is cleared" just by setting wr_object to Py_None. If W's refcount later falls to 0, weakref_dealloc() won't remove W from W's referent's list of weakrefs. That's the way clear_weakref() is written, and I think it has to be that way: if clear_weakref doesn't know what the referent is, it can't find the pointer to the head of the referent's weakref list, so it can't unlink W from the list correctly (unlike gc's doubly-linked lists, weakref lists aren't circular, and don't have list heads) This leaves free'd memory *in* the referent's weakref list. So if the referent's refcount falls to 0 after that, then PyObject_ClearWeakRefs() goes nuts trying to crawl over the freed memory. In a debug build that's obvious when it happens, because _PyWeakref_GetWeakrefCount() hits a weakref object full of 0xdb bytes. From nas at arctrix.com Fri Oct 29 22:22:11 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Fri Oct 29 22:22:15 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <1f7befae0410291259db538cb@mail.gmail.com> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <20041029172434.GA23030@mems-exchange.org> <1f7befae041029111046d2528@mail.gmail.com> <1f7befae0410291259db538cb@mail.gmail.com> Message-ID: <20041029202211.GA23502@mems-exchange.org> On Fri, Oct 29, 2004 at 03:59:37PM -0400, Tim Peters wrote: > if clear_weakref doesn't know what the referent is, it can't find > the pointer to the head of the referent's weakref list, so it > can't unlink W from the list correctly (unlike gc's doubly-linked > lists, weakref lists aren't circular, and don't have list heads) > This leaves free'd memory *in* the referent's weakref list. So if > the referent's refcount falls to 0 after that, then > PyObject_ClearWeakRefs() goes nuts trying to crawl over the freed > memory. Crap. That's got to be the problem that I've been bashing my head against for the past few hours. Okay, how about we first clearly specify what needs to happen. Afterwards we can figure how to do it while retaining binary compatibility. :-( Non-trash weakrefs to trash objects MUST have their callbacks invoked. Trash weakrefs MUST NOT have their callbacks invoked. Any weakref to a trash object MUST NOT reveal the trash (i.e. via the wr_object pointer). Does that sound correct? Neil From fdrake at acm.org Fri Oct 29 22:58:35 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Oct 29 22:58:59 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <20041029202211.GA23502@mems-exchange.org> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <1f7befae0410291259db538cb@mail.gmail.com> <20041029202211.GA23502@mems-exchange.org> Message-ID: <200410291658.35476.fdrake@acm.org> On Friday 29 October 2004 04:22 pm, Neil Schemenauer wrote: > Non-trash weakrefs to trash objects MUST have their callbacks > invoked. Trash weakrefs MUST NOT have their callbacks invoked. Any > weakref to a trash object MUST NOT reveal the trash (i.e. via the > wr_object pointer). Yes, that's right. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From tim.peters at gmail.com Fri Oct 29 23:59:54 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sat Oct 30 00:00:00 2004 Subject: [Python-Dev] Re: weakref callback vs gc vs threads In-Reply-To: <20041029202211.GA23502@mems-exchange.org> References: <E1CNaJH-0005mG-2T@sc8-sf-web3.sourceforge.net> <20041029172434.GA23030@mems-exchange.org> <1f7befae041029111046d2528@mail.gmail.com> <1f7befae0410291259db538cb@mail.gmail.com> <20041029202211.GA23502@mems-exchange.org> Message-ID: <1f7befae04102914593c567de5@mail.gmail.com> [Neil, on "half clear"ing] > Crap. That's got to be the problem that I've been bashing my head > against for the past few hours. I can believe that. It was attractive because otherwise (as my patch does) the guts of PyObject_ClearWeakRefs() get duplicated inline inside gc, spread across distinct passes. > Okay, how about we first clearly specify what needs to happen. Afterwards > we can figure how to do it while retaining binary compatibility. :-( It's a good plan. > Non-trash weakrefs to trash objects MUST have their callbacks > invoked. Yes. > Trash weakrefs MUST NOT have their callbacks invoked. That's actually a kind of optimization. It's never *necessary* to invoke a callback from a trash weakref, and it *may* be catastrophic to do so. So the cheapest/easiest route is to avoid analyzing whether this case is safe, and never do it. IOW, "MUST NOT" is a correct approach to this case, but looser approaches could suffice. > Any weakref to a trash object MUST NOT reveal the trash (i.e. via the > wr_object pointer). Yes. Other subtleties follow from those. Chief example: if you decide not to invoke a callback in a trash weakref to a trash object, that can't be accomplished correctly by decref'ing wr_callback then setting wr_callback to NULL. The problem is that wr_callback may itself be weakly referenced by a different weakref with a different callback, so decref'ing wr_callback may cause that other callback to run right away, and that may not be safe. So if setting wr_calback to NULL is the strategy for preventing wr_callback from running, it can't be carried out safely until after all weakrefs to trash have been cleared. That's why dealing with trash weakrefs required two passes when fixing the last pile of bugs. There are also subtleties about what "trash" means, exactly, here. gc has been ignoring the worst of these. For example, if the referent of a non-trash weakref is in the unreachable set, but the referent is *also* in a cycle and has a __del__ method, the referent may well end up in gc.garbage. It's arguably wrong to trigger the callback in that case. It's also arguable that the user deserves whatever they get in that case <0.7 wink>. All that said, the only problem I have with the patch I posted is that the abuse of a weakref's `hash` member breaks weak-key (but not weak-value) dictionaries. That doesn't cause any tests to fail, but isn't acceptable (weak-key dicts don't work as intended). Since I'm not convinced there's a squeaky-clean approach just waiting to be found here, I think I'll go back to repairing that glitch now. From greg at electricrain.com Sat Oct 30 00:33:55 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Sat Oct 30 00:34:00 2004 Subject: [Python-Dev] bsddb todo for someone - auto-upgrade db format In-Reply-To: <20041028100513.GA19996@phd.pp.ru> References: <20041028084443.GD28925@zot.electricrain.com> <20041028100513.GA19996@phd.pp.ru> Message-ID: <20041029223355.GA32510@zot.electricrain.com> On Thu, Oct 28, 2004 at 02:05:13PM +0400, Oleg Broytmann wrote: > On Thu, Oct 28, 2004 at 01:44:43AM -0700, Gregory P. Smith wrote: > > On Thu, Oct 21, 2004 at 12:49:19PM +0400, Oleg Broytmann wrote: > > > BTW, just installing is not enough, even when it is come with Python > > > distribution. Installing a newer version of BerkeleyDB breaks older > > > databases due to incompatible file formats. > > > > fwiw its possible to fix this in the bsddb code. someone willing to do > > the exception catching database backing-up + auto db_upgrade() > > What's db_upgrade()? I can find only a reference to a program called > db_upgrade, and I don't have one on my computer. http://www.sleepycat.com/docs/api_c/db_upgrade.html Its part of the sleepycat database API to upgrade from old formats to new. When you build BerkeleyDB it does create a standalone command line utility that'll call the function for you but its also available thru the C (and python) api. See the bsddb.db.DB.upgrade() method. From kbk at shore.net Sat Oct 30 16:20:38 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Sat Oct 30 16:20:44 2004 Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200410301420.i9UEKcYr011477@h006008a7bda6.ne.client2.attbi.com> Patch / Bug Summary ___________________ Patches : 243 open ( +3) / 2678 closed (+17) / 2921 total (+20) Bugs : 789 open (+17) / 4545 closed (+17) / 5334 total (+34) RFE : 156 open ( +1) / 131 closed ( +0) / 287 total ( +1) New / Reopened Patches ______________________ Fix various x86_64 build issues (2004-10-19) http://python.org/sf/1050475 opened by Bob Halley make distutils.core.run_setup re-entrant (2004-10-20) http://python.org/sf/1051216 opened by Bob Ippolito xml.dom missing API docs (bugs 1010196, 1013525) (2004-10-21) http://python.org/sf/1051321 opened by Mike Brown locale.getdefaultlocale does not return tuple in some OS (2004-10-21) http://python.org/sf/1051395 opened by Jiwon Seo Syntax error in Misc/NEWS (2004-10-22) CLOSED http://python.org/sf/1051866 opened by Felix Wiemann plistlib: add plst resource functionality, fix bugs (2004-10-22) CLOSED http://python.org/sf/1052399 opened by Bob Ippolito urllib2: better import ftplib and gopherlib etc late (2004-10-24) http://python.org/sf/1053150 opened by Robert Kiendl nntplib: add support for NNTP over SSL (2004-10-24) http://python.org/sf/1053365 opened by Reinhard Speyerer ftplib: add support for MDTM command (2004-10-24) http://python.org/sf/1053369 opened by Reinhard Speyerer Decimal speed up (2004-10-24) CLOSED http://python.org/sf/1053375 opened by Facundo Batista mingw compile (2004-10-25) http://python.org/sf/1053879 opened by Matthias Kramm fix inplace string concat hash error (2004-10-25) CLOSED http://python.org/sf/1054197 opened by Bob Halley automatic XMLRPC boxcarring via multicall for xmlrpclib (2004-10-26) http://python.org/sf/1054434 opened by Eric Ries Fix for Unix macro expansion in citetitle (2004-10-26) http://python.org/sf/1054715 opened by Richard Brodie bdist_deb - Debian packager (2004-10-26) http://python.org/sf/1054967 opened by Geoffrey T. Dairiki CGIHTTPServer can't redirect (2004-10-27) http://python.org/sf/1055159 opened by Thomas Guettler cgi.py does not correctly handle fields with ';' (2004-10-27) http://python.org/sf/1055234 opened by Vova typo in comment (unicodeobject.h) (2004-10-28) http://python.org/sf/1056231 opened by Baris Metin python / pythonw replacement in C (2004-10-29) http://python.org/sf/1056561 opened by Michiel de Hoon No ValueError for safe_substitute() (2004-10-29) http://python.org/sf/1056967 opened by Barry A. Warsaw Patches Closed ______________ __main__ for whichdb.py (2004-10-01) http://python.org/sf/1038388 closed by rhettinger Bug fixes and cleanup for decimal.py (2004-10-17) http://python.org/sf/1048728 closed by rhettinger tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 closed by akuchling replace() method in unicode objects (2003-02-05) http://python.org/sf/680863 closed by nnorwitz Remove __file__ after running $PYTHONSTARTUP (2003-04-11) http://python.org/sf/719777 closed by nnorwitz sre fixes for lastindex and minimizing repeats+assertions (2003-03-31) http://python.org/sf/712900 closed by nnorwitz Some bug fixes for regular ex code. (2003-04-14) http://python.org/sf/720991 closed by nnorwitz optimize eval_frame (2003-12-21) http://python.org/sf/864059 closed by nnorwitz Implementation for PEP 318 ([as classmethod] version) (2004-04-08) http://python.org/sf/932100 closed by nnorwitz Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 closed by perky New patch for Python 2.3 spec file. (2004-07-23) http://python.org/sf/996316 closed by jafo Syntax error in Misc/NEWS (2004-10-21) http://python.org/sf/1051866 closed by bcannon plistlib: add plst resource functionality, fix bugs (2004-10-22) http://python.org/sf/1052399 closed by jvr Decimal speed up (2004-10-24) http://python.org/sf/1053375 closed by rhettinger fix inplace string concat hash error (2004-10-25) http://python.org/sf/1054197 closed by rhettinger traceback module caches sources invalid (2003-05-14) http://python.org/sf/737473 closed by perky do not add directory of sys.argv[0] into sys.path (2004-05-02) http://python.org/sf/946373 closed by wrobell New / Reopened Bugs ___________________ 2.4b1 Win32 installer eliminates 2.3 assoc (2004-10-20) http://python.org/sf/1050486 opened by Mike C. Fletcher reindent.py strips execute permission on Unix (2004-10-20) http://python.org/sf/1050828 opened by David Ripton incorrect traceback filename from pyc (2004-10-21) http://python.org/sf/1051638 opened by Kevin Quick HTMLParser doesn't treat endtags in <script> tags as CDATA (2004-10-21) http://python.org/sf/1051840 opened by Luke Bradley Appending non-empty element to a non-empty list yields None (2004-10-22) CLOSED http://python.org/sf/1052036 opened by Peer Janssen Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 opened by Diedrich Vorberg page on python api down (2004-10-22) CLOSED http://python.org/sf/1052274 opened by mperpick pdb runcall should accept keyword arguments (2004-10-22) CLOSED http://python.org/sf/1052503 opened by Leonardo Rochael Almeida filelist.findall should use os.lstat (2004-10-23) http://python.org/sf/1052827 opened by Geoffrey T. Dairiki time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 reopened by panjunyong time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 opened by panjunyong Unjustified SyntaxWarning (2004-10-25) CLOSED http://python.org/sf/1053604 opened by Markus Gritsch Why is this expression = False? (2004-10-25) CLOSED http://python.org/sf/1053629 opened by Alexander R?dseth PyOS_InputHook not called in subprocess (2004-10-25) http://python.org/sf/1053687 opened by Michiel de Hoon Segfault in tuple_of_constants (2004-10-25) CLOSED http://python.org/sf/1053819 opened by Duncan Grisby help() nor in tutorial index (2004-10-25) CLOSED http://python.org/sf/1053956 opened by Guido van Rossum Python doesn't exit with proper resultcode on SIGINT (2004-10-25) http://python.org/sf/1054041 opened by James Y Knight serious string hashing error in 2.4b1 (2004-10-25) CLOSED http://python.org/sf/1054139 opened by Bob Halley RE '*.?' cores if len of found string exceeds 10000 (2004-10-26) http://python.org/sf/1054564 opened by Rob New failure in debug build (2004-10-26) http://python.org/sf/1054615 opened by Tim Peters difflib HtmlDiff() extra space on inserted 1 character lines (2004-10-26) http://python.org/sf/1054821 opened by Dan Gass Python may contain NFC/NFKC bug per Unicode PRI #29 (2004-10-26) http://python.org/sf/1054943 opened by Rick McGowan pdb.set_trace should go up one more frame (2004-10-27) http://python.org/sf/1055168 opened by Johannes Gijsbers bdist_wininst broken for pure Python distributions (2004-10-27) CLOSED http://python.org/sf/1055540 opened by Thomas Heller weakref callback vs gc vs threads (2004-10-27) http://python.org/sf/1055820 opened by Tim Peters HTMLParser not compliant to XHTML spec (2004-10-27) http://python.org/sf/1055864 opened by Luke Bradley dir() should only return strings (2004-10-28) http://python.org/sf/1056293 opened by Skip Montanaro subprocess fails for args="...", executable="..." (2004-10-28) http://python.org/sf/1056441 opened by Russell Owen Lambda in list comprehension (2004-10-28) CLOSED http://python.org/sf/1056483 opened by Doug Blank minor error in os.access (2004-10-29) http://python.org/sf/1056498 opened by edward ambiguity in os.tmpfile (2004-10-29) CLOSED http://python.org/sf/1056502 opened by edward ambiguity in os.tmpfile (2004-10-29) http://python.org/sf/1056515 opened by edward subprocess works poorly on Windows with Python 2.3 (2004-10-29) http://python.org/sf/1057048 opened by Russell Owen subprocess on Windows 2: unexpected failure (2004-10-29) http://python.org/sf/1057052 opened by Russell Owen subprocess on Windows: extra windows (2004-10-29) http://python.org/sf/1057061 opened by Russell Owen Bugs Closed ___________ relative on-disk SYSTEM id handling is incorrect (2004-02-22) http://python.org/sf/902037 closed by rhettinger filemode() in tarfile.py makes wrong file mode strings (2004-08-27) http://python.org/sf/1017553 closed by akuchling test_subprocess 2.4b1 fails on FreeBSD 5.2 (2004-10-17) http://python.org/sf/1048808 closed by astrand No command line args when script run without python.exe (2004-09-16) http://python.org/sf/1029047 closed by facundobatista Appending non-empty element to a non-empty list yields None (2004-10-22) http://python.org/sf/1052036 closed by peerjanssen page on python api down (2004-10-22) http://python.org/sf/1052274 closed by loewis pdb runcall should accept keyword arguments (2004-10-22) http://python.org/sf/1052503 closed by rhettinger call arg of lambda not updating (2004-10-17) http://python.org/sf/1048870 closed by rhettinger time module returns wrong timezone under windows (2004-10-25) http://python.org/sf/1053539 closed by tim_one Unjustified SyntaxWarning (2004-10-25) http://python.org/sf/1053604 closed by rhettinger Why is this expression = False? (2004-10-25) http://python.org/sf/1053629 closed by doerwalter Segfault in tuple_of_constants (2004-10-25) http://python.org/sf/1053819 closed by rhettinger help() nor in tutorial index (2004-10-25) http://python.org/sf/1053956 closed by rhettinger serious string hashing error in 2.4b1 (2004-10-25) http://python.org/sf/1054139 closed by rhettinger strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 closed by bcannon bdist_wininst broken for pure Python distributions (2004-10-28) http://python.org/sf/1055540 closed by mhammond Lambda in list comprehension (2004-10-28) http://python.org/sf/1056483 closed by rhettinger ambiguity in os.tmpfile (2004-10-28) http://python.org/sf/1056502 closed by facundobatista New / Reopened RFE __________________ Attempt to run all atexit handlers (2004-10-22) http://python.org/sf/1052242 opened by Greg Chapman Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 reopened by loewis RFE Closed __________ Seting defaultencoding through an environment variable (2004-10-22) http://python.org/sf/1052098 closed by loewis From d+pydev at trit.org Sun Oct 31 16:13:27 2004 From: d+pydev at trit.org (Dima Dorfman) Date: Sun Oct 31 16:13:47 2004 Subject: [Python-Dev] Pickle breakage with reduce of recursive structures (was: AssertionError in pickle's memoize function) In-Reply-To: <yyoy8hp2nyx.fsf@hooknose.lbl.gov> References: <yyo3bzx4e4w.fsf@hooknose.lbl.gov> <mailman.5674.1099074661.5135.python-list@python.org> <yyoy8hp2nyx.fsf@hooknose.lbl.gov> Message-ID: <20041031151326.GB22539@trit.org> [Followups to python-dev, please.] [Michael Hohn] > > > under python 2.2, the pickle/unpickle sequence incorrectly restores > > > a larger data structure I have. > > > > > > Under Python 2.3, these structures now give an explicit exception from > > > Pickle.memoize(): > > > assert id(obj) not in self.memo > > > [Tim Peters] > > Assertions are never expected to fail, so "something impossible > > happened" when they do fail. [Michael Hohn] > Here is a code sample that shows the problem I ran into: Summary for the OP: This is a bug in Python. Using cPickle won't help, but if you don't subclass builtin container types others than list, dict, and tuple, using pickle protocol 2 should work. The rest of this message is for python-dev. The simplest breaking case is: t = type('t', (list,), {}) obj = t() obj.append(obj) pickle.dumps(obj) [infinite recursion] The subclass causes save_reduce to be used instead of save_list. For proto < 2, copy_reg._reduce_ex returns (_reconstructor, list(a)), and the args--list(a)--cycle back through obj. Initially it looks like this should be okay, but the args are saved before obj is memoized, and obj can't be memoized until REDUCE can be executed with the args--and there's the cycle. It is even more obviously impossible from the unpickler's perspective because it has to call _reconstructor([obj]) to create obj! There are two separate problems: 1. Any __reduce__ implementation that returns args that cycle back through the object it tried to reduce hasn't done its job. As described above, _reduce_ex is one such implementation. reduce_2 avoids this by using the listitems and dictitems parameters. Since that's a pickler-side feature it can be used in _reduce_ex too. The basetype(obj) hook (documented in PEP 307) would remain for immutable bases; it doesn't work for containers, but user-defined containers already have to implement their own reduce functions. POC patch: http://www.trit.org/~dima/home/reduce_ex.diff At least the set and deque types also have this problem. 2. The pickle implementations don't detect reduction cycles. Pickling an instance of this obviously broken class causes an infinite recursion: class evil(object): def __reduce__(self): return evil, (self,) It's easy to detect this case. POC patch for the pickle module: http://www.trit.org/~dima/home/redcycle.diff BTW, the failed assert the OP is seeing happens when the cycle goes through another object: t = type('t', (list,), {}) obj = t() d = {'obj': obj} obj.append(d) pickle.dumps(obj) [AssertionError] cPickle has the same problem, but it lacks the assert, so it writes garbage instead: new = cPickle.loads(cPickle.dumps(obj)) new[0]['obj'] is new -> False # wrong obj[0]['obj'] is obj -> True # right This makes the reduction cycle check (#2 above) more than just cosmetic since if cPickle had that assert (it should) it would've been a crash. Right now it's garbage output instead, which is arguably worse. Formally complete versions of the above patches will be on SF tomorrow unless someone suggests better alternatives. Dima. From aahz at pythoncraft.com Sun Oct 31 17:12:10 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Oct 31 17:12:12 2004 Subject: [Python-Dev] Pickle breakage with reduce of recursive structures (was: AssertionError in pickle's memoize function) In-Reply-To: <20041031151326.GB22539@trit.org> References: <yyo3bzx4e4w.fsf@hooknose.lbl.gov> <mailman.5674.1099074661.5135.python-list@python.org> <yyoy8hp2nyx.fsf@hooknose.lbl.gov> <20041031151326.GB22539@trit.org> Message-ID: <20041031161210.GB2013@panix.com> On Sun, Oct 31, 2004, Dima Dorfman wrote: > > Summary for the OP: This is a bug in Python. Using cPickle won't help, > but if you don't subclass builtin container types others than list, > dict, and tuple, using pickle protocol 2 should work. The rest of this > message is for python-dev. Very nice! Please file a bug report on SF. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ WiFi is the SCSI of the 21st Century -- there are fundamental technical reasons for sacrificing a goat. (with no apologies to John Woods) From tim.peters at gmail.com Sun Oct 31 19:39:03 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 31 19:39:12 2004 Subject: [Python-Dev] Windows: PyBanner048.gif damage repaired Message-ID: <1f7befae04103110396aece63d@mail.gmail.com> Lib\email\test\data\PyBanner048.gif wasn't marked binary in CVS. It is now. If you're on Windows, delete your copy, then cvs up to refetch an unmangled copy. From tim.peters at gmail.com Sun Oct 31 20:27:18 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 31 20:27:20 2004 Subject: [Python-Dev] Tests broken on Windows Message-ID: <1f7befae0410311127b31675c@mail.gmail.com> Current test_shutil is apparently trying to do something that can't work on Windows, so it fails: test_shutil test test_shutil failed -- Traceback (most recent call last): File "C:\Code\python\lib\test\test_shutil.py", line 28, in test_on_error shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) File "C:\Code\python\lib\shutil.py", line 168, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "C:\Code\python\lib\test\test_shutil.py", line 36, in check_args_to_onerror self.assertEqual(func, os.remove) AssertionError: <built-in function rmdir> != <built-in function remove> 1 test failed: test_shutil After that, it leaves behind temp directories that cause other tests to fail. Sorry, I've got other problems on my stack, and can't make time for this one. From jlg at dds.nl Sun Oct 31 20:52:18 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Sun Oct 31 20:50:56 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <1f7befae0410311127b31675c@mail.gmail.com> References: <1f7befae0410311127b31675c@mail.gmail.com> Message-ID: <20041031195218.GA7395@surfboy> On Sun, Oct 31, 2004 at 02:27:18PM -0500, Tim Peters wrote: > Current test_shutil is apparently trying to do something that can't > work on Windows, so it fails: > > test_shutil > test test_shutil failed -- Traceback (most recent call last): > File "C:\Code\python\lib\test\test_shutil.py", line 28, in test_on_error > shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) > File "C:\Code\python\lib\shutil.py", line 168, in rmtree > onerror(os.rmdir, path, sys.exc_info()) > File "C:\Code\python\lib\test\test_shutil.py", line 36, in > check_args_to_onerror > self.assertEqual(func, os.remove) > AssertionError: <built-in function rmdir> != <built-in function remove> > > 1 test failed: > test_shutil Hum, I didn't realize os.chmod() worked this differently on Windows. Seems like I bumped into bug #755617 (os module: Need a better description of "mode"). Per that bug, I googled for '_chmod msdn' and came up with the attached patch. However, I don't have a Windows machine handy to test. Could someone with a Windows machine try the patch? Cheers, Johannes From tim.peters at gmail.com Sun Oct 31 22:00:43 2004 From: tim.peters at gmail.com (Tim Peters) Date: Sun Oct 31 22:00:49 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <20041031195218.GA7395@surfboy> References: <1f7befae0410311127b31675c@mail.gmail.com> <20041031195218.GA7395@surfboy> Message-ID: <1f7befae04103113007e083d7c@mail.gmail.com> [Johannes Gijsbers] > Hum, I didn't realize os.chmod() worked this differently on Windows. Seems like > I bumped into bug #755617 (os module: Need a better description of "mode"). Per > that bug, I googled for '_chmod msdn' and came up with the attached patch. > However, I don't have a Windows machine handy to test. Could someone with a > Windows machine try the patch? I'll try it, but there were no attachments. Distinctions among "user", "group" and "world" are unique to Unix. Windows has a much more elaborate permission scheme than that, but at chmod()'s feeble level it's essentially just "nobody can delete me" or "anyone can delete me". From jlg at dds.nl Sun Oct 31 22:06:10 2004 From: jlg at dds.nl (Johannes Gijsbers) Date: Sun Oct 31 22:03:52 2004 Subject: [Python-Dev] Tests broken on Windows In-Reply-To: <1f7befae04103113007e083d7c@mail.gmail.com> References: <1f7befae0410311127b31675c@mail.gmail.com> <20041031195218.GA7395@surfboy> <1f7befae04103113007e083d7c@mail.gmail.com> Message-ID: <20041031210610.GE7395@surfboy> On Sun, Oct 31, 2004 at 04:00:43PM -0500, Tim Peters wrote: > [Johannes Gijsbers] > > Hum, I didn't realize os.chmod() worked this differently on Windows. Seems like > > I bumped into bug #755617 (os module: Need a better description of "mode"). Per > > that bug, I googled for '_chmod msdn' and came up with the attached patch. > > However, I don't have a Windows machine handy to test. Could someone with a > > Windows machine try the patch? > > I'll try it, but there were no attachments. Oops. Here you go. Johannes -------------- next part -------------- *** test_shutil.py 31 okt 2004 12:49:56 +0100 1.8 --- test_shutil.py 31 okt 2004 20:47:57 +0100 *************** *** 22,34 **** os.mkdir(TESTFN) f = open(os.path.join(TESTFN, 'a'), 'w') f.close() # Make TESTFN unwritable. ! os.chmod(TESTFN, stat.S_IRUSR) shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) # Make TESTFN writable again. ! os.chmod(TESTFN, stat.S_IRWXU) shutil.rmtree(TESTFN) def check_args_to_onerror(self, func, arg, exc): --- 22,35 ---- os.mkdir(TESTFN) f = open(os.path.join(TESTFN, 'a'), 'w') f.close() + oldmode = os.stat(TESTFN).st_mode # Make TESTFN unwritable. ! os.chmod(TESTFN, stat.S_IREAD) shutil.rmtree(TESTFN, onerror=self.check_args_to_onerror) # Make TESTFN writable again. ! os.chmod(TESTFN, oldmode) shutil.rmtree(TESTFN) def check_args_to_onerror(self, func, arg, exc):