From dieter at handshake.de Thu Mar 1 02:11:03 2018 From: dieter at handshake.de (dieter) Date: Thu, 01 Mar 2018 08:11:03 +0100 Subject: RFC: Proposal: Deterministic Object Destruction References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: <87r2p41da0.fsf@handshake.de> Rick Johnson writes: > ... > Can you provide a real world example in which you need an > object which circularly references _itself_? Circular structures occur in real life. One example are tracebacks (which are very helpful for debugging reasons). Keeping a traceback in a variable can easily introduce a cycle. Another example is a tree structure where you need both efficient access to children and ancestors of a node (see the DOM standard). From storchaka at gmail.com Thu Mar 1 02:54:04 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Mar 2018 09:54:04 +0200 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: 01.03.18 04:46, Rick Johnson ????: > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: > >> Here's one example: reference cycles. When do they get detected? >> Taking a really simple situation: >> >> class Foo: >> def __init__(self): >> self.self = self > > *shudders* > > Can you provide a real world example in which you need an > object which circularly references _itself_? This looks like > a highly contrived example used to (1) merely win the > argument, and (2) Bump fib() up one position from it's > current position as "the worst introductory example of how > to write a function in the history of programming tutorials" Every global function (or method of global class) creates a reference cycle. def f(): pass f.__globals__['f'] is f From greg.ewing at canterbury.ac.nz Thu Mar 1 04:34:47 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 01 Mar 2018 22:34:47 +1300 Subject: Is there are good DRY fix for this painful design pattern? In-Reply-To: References: <20180227114713.6582f82c@bigbox.christie.dr> Message-ID: After this thread, the term "docstring" is never going to mean quite the same thing to me again. -- Greg From tomuxiong at gmx.com Thu Mar 1 04:57:11 2018 From: tomuxiong at gmx.com (Thomas Nyberg) Date: Thu, 1 Mar 2018 10:57:11 +0100 Subject: Functions unnecessarily called in Python/pylifecycle.c:_Py_InitializeCore() ? Message-ID: <33856b32-6549-6ed9-4d7c-cde4d79996ce@gmx.com> Hello, I was playing around with cpython and noticed the following. The `_PyFrame_Init()` and `PyByteArray_Init()` functions are called in these two locations: https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L693-L694 https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L699-L700 But here are their function definitions and they appear to do nothing: https://github.com/python/cpython/blob/master/Objects/frameobject.c#L555-L561 https://github.com/python/cpython/blob/master/Objects/bytearrayobject.c#L24-L28 I can understand leaving the functions in the source for backwards-compatibility, but why are they still being called in `_Py_InitializeCore()`? Seems like it just adds noise for those new to the cpython internals. Is there some consistency doc that requires this or something? Maybe this isn't the right place to ask, but I'm not sure where is. Thanks for any enlightenment! Cheers, Thomas From zhangyangyu0614 at gmail.com Thu Mar 1 05:41:14 2018 From: zhangyangyu0614 at gmail.com (Xiang Zhang) Date: Thu, 1 Mar 2018 02:41:14 -0800 (PST) Subject: In Python2, does it need to wrap imp.find/load_module with imp_acquire/release_lock? In-Reply-To: References: <71381ce7-213b-460e-b5d2-07337158d1ff@googlegroups.com> <87d10p38k3.fsf@handshake.de> Message-ID: <5cef6d1c-6da9-4e38-bbc6-76325f040709@googlegroups.com> On Wednesday, February 28, 2018 at 3:02:38 PM UTC+8, dieter wrote: > Xiang Zhang writes: > > > Just like the title. It seems to me it is needed from the source code but codes in stdlib all doesn't do that. > > The "import" machinery uses locks of its own (to protect "sys.modules"). > > I assume that "load_module" will use those locks internally. > "find" might be considered to usually see quite a static situation; > should your file system be modified by concurrent threads, it might > be necessary that you use your own locks to protect against this. I know. But my confusion is that I can't confirm your assumption. I can't find where the import lock is acquired in its code path. From bc at freeuk.com Thu Mar 1 06:46:41 2018 From: bc at freeuk.com (bartc) Date: Thu, 1 Mar 2018 11:46:41 +0000 Subject: Functions unnecessarily called in Python/pylifecycle.c:_Py_InitializeCore() ? In-Reply-To: References: <33856b32-6549-6ed9-4d7c-cde4d79996ce@gmx.com> Message-ID: On 01/03/2018 09:57, Thomas Nyberg wrote: > Hello, > > I was playing around with cpython and noticed the following. The > `_PyFrame_Init()` and `PyByteArray_Init()` functions are called in these > two locations: > > https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L693-L694 > https://github.com/python/cpython/blob/master/Python/pylifecycle.c#L699-L700 > > But here are their function definitions and they appear to do nothing: > > https://github.com/python/cpython/blob/master/Objects/frameobject.c#L555-L561 > https://github.com/python/cpython/blob/master/Objects/bytearrayobject.c#L24-L28 > > I can understand leaving the functions in the source for > backwards-compatibility, but why are they still being called in > `_Py_InitializeCore()`? Seems like it just adds noise for those new to > the cpython internals. Is there some consistency doc that requires this > or something? If they're only called once, then it probably doesn't matter too much in terms of harming performance. As for leaving them in, there might be a number of reasons. One, if one day some special initialisation does need to be done, then this gives a place to put it. I quite often have an initialisation routine for a module, that sometimes ends up empty, but I keep it in anyway as often things can get added back. (Old CPython source I have does do something in those functions. For example: int _PyFrame_Init() { builtin_object = PyUnicode_InternFromString("__builtins__"); if (builtin_object == NULL) return 0; return 1; } ) -- Bartc From Richard at Damon-Family.org Thu Mar 1 07:34:56 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Thu, 1 Mar 2018 07:34:56 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <1519876799l.9175092l.0l@psu.edu> References: <1519876799l.9175092l.0l@psu.edu> Message-ID: <2cc02018-b7db-b976-1134-e4394fce43f0@Damon-Family.org> On 2/28/18 11:00 PM, ROGER GRAYDON CHRISTMAN wrote: > On Wed, Feb 28, 2018, Rick Johnson wrote: > > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: >>> Here's one example: reference cycles. When do they get detected? >>> Taking a really simple situation: >>> >>> class Foo: >>> def __init__(self): >>> self.self = self >> *shudders* >> >> Can you provide a real world example in which you need an >> object which circularly references_itself_? This looks like >> a highly contrived example used to (1) merely win the >> argument, and (2) Bump fib() up one position from it's >> current position as "the worst introductory example of how >> to write a function in the history of programming tutorials" > If you want something that looks like a real world example, > consider the very common doubly-linked list: > [ 1 ] <---> [ 2 ] <---> [ 3 ] <--.....--> [ N ] > > This is chock-full of reference cycles, everywhere you see a <--> > > Then you have a wrapper List object that has a head referencing > the 1 node and a tail referencing the N node. > > And you decide you don't need that List any more. > > You could: > 1) Take the time the code to carefully delete every single node > in this linked list, taking the time to set all of the internal references to > None, or > 2) Just discard your List object, and let the GC take care of the rest. > > Note that removing the list alone does not clear any of the > reference counts in the picture above, because those internal > <--> references are all still there. You either have to manually > remove all the references, or rely on the garbage collector. > > Oh, and for those who claim that reference counting is 'efficient', > do bear in mind that every single assignment statement increases > the reference count of one value and possibly reduces the reference > count of another. So down in the machine or byte-code world, > every assignment operation is really two arithmetic computations > and three assignments, which takes 5 times as many operations > as not doing any reference counting at all. > > So if you have a program that can get by leaving unused memory > allocated, and can still terminate the program before all the memory > is claimed, your reference count system is far less efficient than > relying on a garbage collector that happens to not get activated > before your program ends. > > Roger Christman > Pennsylvania State University Now, of course in a language that WANT'S to make reference counting work, and allow for deterministic destruction of object happen, all you need to do is? make one direction of the pointer into a 'weak reference', and the list control object have a strong pointer to the end that starts the strong chain and a weak reference to the other, and when the list control object goes away, so does all the list elements. Yes, there is a slight speed cost in keeping the reference counts, but there is also a significant cost to the processing that is needed to run the garbage collection. And that doesn't include the cost of having to explicitly manage all resources that aren't memory as you have broken the RAII concept. I will agree that many programs many programs don't need precise destruction for most of their objects, but many can be helped by it for at least a specific set of them. I am not sure how hard it would be to implement a really useful reference counting system in Python, as first you would need a concept of a weak pointer to deal with cases like your list above (which I don't think Python has anything like that concept). You would also need to keep the current garbage collection, to handle the existing cases of circular references (I disagree with the original complaint that these are always 'errors', if you know you have garbage collection, the allowance of cycles knowing they will still get cleaned up is a useful simplification if you don't need the immediate clean up). -- Richard Damon From tomuxiong at gmx.com Thu Mar 1 07:40:16 2018 From: tomuxiong at gmx.com (Thomas Nyberg) Date: Thu, 1 Mar 2018 13:40:16 +0100 Subject: Functions unnecessarily called in Python/pylifecycle.c:_Py_InitializeCore() ? In-Reply-To: References: <33856b32-6549-6ed9-4d7c-cde4d79996ce@gmx.com> Message-ID: <56c81fba-0793-e47b-bda4-cbb88e640aaf@gmx.com> On 03/01/2018 12:46 PM, bartc wrote: > If they're only called once, then it probably doesn't matter too much in > terms of harming performance. Oh yeah there's no way this has any affect on performance. A smart compiler might even be able optimize the call away entirely. Even if it couldn't, it's about as fast as an operation could possibly be. > As for leaving them in, there might be a number of reasons. One, if one > day some special initialisation does need to be done, then this gives a > place to put it. > > I quite often have an initialisation routine for a module, that > sometimes ends up empty, but I keep it in anyway as often things can get > added back. Yeah I figure this is the reasoning. Personally I don't like having noops for this sort of code structure reasoning, but I can see how rules like "everything gets init'ed" is also a fair strategy. Half of my question was simply whether that is the case or if possible it was just a mistake when older versions that did something could be removed (e.g. the code example you put int). Thanks for the response! Cheers, Thomas From jugurtha.hadjar at gmail.com Thu Mar 1 07:56:38 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Thu, 1 Mar 2018 13:56:38 +0100 Subject: Is there are good DRY fix for this painful design pattern? In-Reply-To: References: <20180227114713.6582f82c@bigbox.christie.dr> Message-ID: <57cba2d2-a261-e496-f405-e0ef27ee2fa0@gmail.com> On 03/01/2018 10:34 AM, Gregory Ewing wrote: > After this thread, the term "docstring" is never going > to mean quite the same thing to me again. > I still feel that the following is quite readable: <-----------------------------------> import inspect def snatch(func): ??? def snatched(self, *args, **kwargs): ??????? signature = inspect.signature(func) ??????? for name in signature.parameters: ??????????? if name != 'self': ??????????????? kwargs.setdefault(name, getattr(self, name)) ??????? return func(self, *args, **kwargs) ??? return snatched class Foo(object): ??? def __init__(self, sleepy, groggy): ??????? self.sleepy = sleepy ??????? self.groggy = groggy ??? @snatch ??? def spam(self, sleepy=None): ??????? print("I am spam. sleepy=", sleepy) ??? @snatch ??? def ham(self, sleepy=None, groggy=None): ??????? print("I am ham. sleepy=", sleepy, " and groggy=", groggy) <--------------------------------------> -- ~ Jugurtha Hadjar, From Cecil at decebal.nl Thu Mar 1 08:24:13 2018 From: Cecil at decebal.nl (Cecil Westerhof) Date: Thu, 01 Mar 2018 14:24:13 +0100 Subject: Strange problem with pip2 Message-ID: <87po4o2aki.fsf@munus.decebal.nl> There are three pip2 packages that should be updated: apsw (3.13.0.post1) - Latest: 3.9.2.post1 [sdist] mysql-utilities (1.6.4) - Latest: 1.4.3 [sdist] pygobject (3.22.0) - Latest: 3.27.4 [sdist] But the strange thing is that the installed version is newer as the new version. And when I try: pip2 install --upgrade apsw I get: Requirement already up-to-date: apsw in /usr/lib/python2.7/dist-packages But with pygobject the problem is even stranger, when I try to update this, it end with: command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-vY4LXH/pycairo/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-40OPWv-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-vY4LXH/pycairo/ What could be happening here? And how do I solve this? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof From julien at editx.eu Thu Mar 1 08:36:42 2018 From: julien at editx.eu (Julien Carlier) Date: Thu, 1 Mar 2018 14:36:42 +0100 Subject: Python Challenge Online - 30 Questions Message-ID: Hello, Cisco & Dimension Data organize a Python Challenge on EDITx. It is a good way to test your skills & have fun. -> https://editx.eu/it-challenge/python-challenge-2018-cisco-and-dimension-data Rules are simple: 15 minutes online, 30 questions & 3 jokers. Everyone is allowed to participate but only belgian residents can win the prizes. Do not hesitate to share it with people that might be interested. Feedbacks are welcome :) Regards, Julien -- Frankenstraat - Rue des Francs 79 1000 Brussels From rustompmody at gmail.com Thu Mar 1 08:49:22 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Thu, 1 Mar 2018 05:49:22 -0800 (PST) Subject: help me ? (Posting On Python-List Prohibited) In-Reply-To: References: <2edd557f-91ce-4b14-b79e-5a524596fd40@googlegroups.com> <04591b86-19e7-46c8-9ffc-0b3980769213@googlegroups.com> <540c622c-e40d-4f29-a6cd-577c535b248e@googlegroups.com> <158fa187-51f4-4368-926e-0a002440c0cf@googlegroups.com> Message-ID: <02ee54d9-e781-447d-9500-1c3855fd0667@googlegroups.com> On Thursday, March 1, 2018 at 5:37:28 AM UTC+5:30, Steven D'Aprano wrote: > On Wed, 28 Feb 2018 09:58:24 -0800, Aktive wrote: > > > what the hell do you care about cheating.. > > > > the world doest care about cheating. > > > > its about skill. > > Because cheaters don't have skill. That's why they cheat. > > > > You guys been too much in school > > Ah, spoken like a cheater. And in case you happen to want to ?get real? http://www.latimes.com/opinion/op-ed/la-oe-caplan-education-credentials-20180211-story.html From neilc at norwich.edu Thu Mar 1 08:57:08 2018 From: neilc at norwich.edu (Neil Cerutti) Date: Thu, 1 Mar 2018 13:57:08 +0000 (UTC) Subject: csv module and NULL data byte References: <20180228205259.39594497@bigbox.christie.dr> Message-ID: On 2018-03-01, Tim Chase wrote: > On 2018-02-28 21:38, Dennis Lee Bieber wrote: >> > ??? with open( fname, 'rt', encoding='iso-8859-1' ) as csvfile: >> >> Pardon? Has the CSV module changed in the last year or so? >> >> Last time I read the documentation, it was recommended that >> the file be opened in BINARY mode ("rb"). > > It recommends binary mode, but seems to largely work fine with > text/ascii mode or even arbitrary iterables. I've not seen the > rationale behind the binary recommendation, but in 10+ years of using > the csv module, I've not found any issues in using text/ascii mode > that were solved by switching to using binary mode. Binary mode was recommended for Python 2, but not 3, where you open in text mode but use newline=''. -- Neil Cerutti From ned at nedbatchelder.com Thu Mar 1 10:58:44 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 1 Mar 2018 10:58:44 -0500 Subject: Functions unnecessarily called in Python/pylifecycle.c:_Py_InitializeCore() ? In-Reply-To: <56c81fba-0793-e47b-bda4-cbb88e640aaf@gmx.com> References: <33856b32-6549-6ed9-4d7c-cde4d79996ce@gmx.com> <56c81fba-0793-e47b-bda4-cbb88e640aaf@gmx.com> Message-ID: <8b36a44b-0b6f-5d4a-7d81-a0b0ecc87d7e@nedbatchelder.com> On 3/1/18 7:40 AM, Thomas Nyberg wrote: > On 03/01/2018 12:46 PM, bartc wrote: >> If they're only called once, then it probably doesn't matter too much in >> terms of harming performance. > Oh yeah there's no way this has any affect on performance. A smart > compiler might even be able optimize the call away entirely. Even if it > couldn't, it's about as fast as an operation could possibly be. > >> As for leaving them in, there might be a number of reasons. One, if one >> day some special initialisation does need to be done, then this gives a >> place to put it. >> >> I quite often have an initialisation routine for a module, that >> sometimes ends up empty, but I keep it in anyway as often things can get >> added back. > Yeah I figure this is the reasoning. Personally I don't like having > noops for this sort of code structure reasoning, but I can see how rules > like "everything gets init'ed" is also a fair strategy. Half of my > question was simply whether that is the case or if possible it was just > a mistake when older versions that did something could be removed (e.g. > the code example you put int). > > This sounds like it could make a good contribution to CPython :) --Ned. From p.f.moore at gmail.com Thu Mar 1 11:40:56 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Mar 2018 16:40:56 +0000 Subject: Strange problem with pip2 In-Reply-To: References: <87po4o2aki.fsf@munus.decebal.nl> Message-ID: On 1 March 2018 at 16:32, Dennis Lee Bieber wrote: > Note that a google search shows lots of problems apparently caused by > the --single-version-externally-managed option. --single-version-externally-managed is how pip calls setuptools, and is required. It doesn't cause any problems the way pip uses it (that's roughly what "externally managed" means, it's pip that's managing the install). I guess the google hits were about interactions with easy_install's (old, and as far as I know no longer recommended by anyone) support for multiple versions being installed in parallel. So I don't think --single-version-externally-managed is the issue here. Unfortunately, I don't know what is (other than a general note that if the OP is using Debian/Ubunto (they don't state either way) then I know that those distros patch their versions of pip, so starting with a question on a distro-specific forum might be a good idea. Paul From tomuxiong at gmx.com Thu Mar 1 12:07:45 2018 From: tomuxiong at gmx.com (Thomas Nyberg) Date: Thu, 1 Mar 2018 18:07:45 +0100 Subject: Functions unnecessarily called in Python/pylifecycle.c:_Py_InitializeCore() ? In-Reply-To: <8b36a44b-0b6f-5d4a-7d81-a0b0ecc87d7e@nedbatchelder.com> References: <33856b32-6549-6ed9-4d7c-cde4d79996ce@gmx.com> <56c81fba-0793-e47b-bda4-cbb88e640aaf@gmx.com> <8b36a44b-0b6f-5d4a-7d81-a0b0ecc87d7e@nedbatchelder.com> Message-ID: <6c97adcb-8020-cc30-4f95-0427bae3eacc@gmx.com> On 03/01/2018 04:58 PM, Ned Batchelder wrote: > This sounds like it could make a good contribution to CPython :) > > --Ned. Thanks for the recommendation. Issue/PR created: https://bugs.python.org/issue32980 https://github.com/python/cpython/pull/5953 Cheers, Thomas From gheskett at shentel.net Thu Mar 1 13:44:27 2018 From: gheskett at shentel.net (Gene Heskett) Date: Thu, 1 Mar 2018 13:44:27 -0500 Subject: Problem: Need galileo running on debian wheezy Message-ID: <201803011344.27353.gheskett@shentel.net> I know its supposed to be in the debian stretch repo's. I've been told to get a fitbit, but they don't support linux of any flavor, and that leaves galileo as the possible solution? So how should I proceed since the only stretch machine I have ATM is an arm64, aka a rock64. -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From ooomzay at gmail.com Thu Mar 1 14:42:25 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Thu, 1 Mar 2018 11:42:25 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <1519876799l.9175092l.0l@psu.edu> <2cc02018-b7db-b976-1134-e4394fce43f0@Damon-Family.org> Message-ID: On Thursday, March 1, 2018 at 12:35:38 PM UTC, Richard Damon wrote: > [snip] > I disagree with the original complaint that these > are always 'errors', if you know you have garbage collection, the > allowance of cycles knowing they will still get cleaned up is a useful > simplification if you don't need the immediate clean up. I (O.P.) regret making that complaint, it was flippant and has distracted from the essence of the PEP. Whilst my informal introduction was deliberately a little provocative, my PEP is sincere and does not affect the ability to create orphan cycles. From HooDunnit at didly42KahZidly.net Thu Mar 1 15:31:32 2018 From: HooDunnit at didly42KahZidly.net (Cousin Stanley) Date: Thu, 01 Mar 2018 13:31:32 -0700 Subject: Problem: Need galileo running on debian wheezy References: <201802282130.07622.gheskett@shentel.net> Message-ID: Gene Heskett wrote: > I know its supposed to be in the debian stretch repo's. > > I've been told to get a fitbit, but they don't support linux > of any flavor, and that leaves galileo as the possible solution? > > So how should I proceed since the only stretch machine I have ATM > is an arm64, aka a rock64. The galileo package is available under debian stretch on my rock64 .... # apt-cache policy galileo galileo: Installed: (none) Candidate: 0.5.1-4 Version table: 0.5.1-4 500 500 http://approx:9999/debian stretch/main arm64 Packages Package details .... # apt-cache show galileo However, I have no clue as to its usage or effectiveness .... -- Stanley C. Kitching Human Being Phoenix, Arizona From best_lay at yahoo.com Thu Mar 1 17:29:27 2018 From: best_lay at yahoo.com (Wildman) Date: Thu, 01 Mar 2018 16:29:27 -0600 Subject: Problem: Need galileo running on debian wheezy References: <201803011344.27353.gheskett@shentel.net> Message-ID: On Thu, 01 Mar 2018 13:44:27 -0500, Gene Heskett wrote: > I know its supposed to be in the debian stretch repo's. > > I've been told to get a fitbit, but they don't support linux of any > flavor, and that leaves galileo as the possible solution? > > So how should I proceed since the only stretch machine I have ATM is an > arm64, aka a rock64. > > -- > Cheers, Gene Heskett I would suggest to download it from stretch's repo and open it with gdebi. It will tell you if the depends are met. -- GNU/Linux user #557453 The cow died so I don't need your bull! From rgaddi at highlandtechnology.invalid Thu Mar 1 17:44:31 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Thu, 1 Mar 2018 14:44:31 -0800 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> Message-ID: On 03/01/2018 02:24 PM, Lawrence D?Oliveiro wrote: > On Thursday, March 1, 2018 at 6:44:39 PM UTC+13, Paul Rubin wrote: >> DOM trees are a classic example (see the various DOM modules in the >> Python stdlib). Non-leaf nodes have a list of child nodes, child nodes >> have pointers back upwards to their parent, and each child node has >> pointers to its left and right siblings if they exist. It's all very >> circular. > > This is why you have weak refs. > But making sure that the failure to use weakrefs doesn't memory leak the program into the pavement is why you have background garbage collection with intelligent cyclical reference breaking. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From rantingrickjohnson at gmail.com Thu Mar 1 17:49:30 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 14:49:30 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: <383961ab-a7b4-46c9-8d31-e73e81ade32a@googlegroups.com> On Wednesday, February 28, 2018 at 9:00:37 PM UTC-6, Chris Angelico wrote: > On Thu, Mar 1, 2018 at 1:46 PM, Rick Johnson > wrote: > > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: > > > >> Here's one example: reference cycles. When do they get detected? > >> Taking a really simple situation: > >> > >> class Foo: > >> def __init__(self): > >> self.self = self > > > > *shudders* > > > > Can you provide a real world example in which you need an > > object which circularly references _itself_? [...] > > Not off hand, but I can provide an EXTREMELY real-world > example of a fairly tight loop: exceptions. An exception > has a reference to the local variables it came from, and > those locals may well include the exception itself: > > try: > 1/0 > except Exception as e: > print(e) > > The ZeroDivisionError has a reference to the locals, and > 'e' in the locals refers to that very exception object. AFAIK, Python allows access to the "locals namespace" in every scope, so i'm not sure how this specific case is any more unique than any of the innumerable other cases that could be presented. (but if i'm missing something, feel free explain) Also, i'm not convinced that a reference which is encapsulated by the current local namespace (even if the containing dict is an attribute of some object!) is the equivalent of `self.bar = bar`. (NOTE: i used `bar` instead of `self` so the comparison would be more clear) ## Python 2.x ## >>> class Foo(object): ... def __init__(self): ... self.bar = "bar" ... print locals() ... >>> foo = Foo() {'self': <__main__.Foo object at 0x027A6890>} >>> foo.__dict__ {'bar': 'bar'} So, yeah. The challenge remains: "Produce a real world example in which an object stores a reference of itself, and then provide an acceptable justification for such a thing." From rosuav at gmail.com Thu Mar 1 18:00:46 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 10:00:46 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <383961ab-a7b4-46c9-8d31-e73e81ade32a@googlegroups.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <383961ab-a7b4-46c9-8d31-e73e81ade32a@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 9:49 AM, Rick Johnson wrote: > On Wednesday, February 28, 2018 at 9:00:37 PM UTC-6, Chris Angelico wrote: >> On Thu, Mar 1, 2018 at 1:46 PM, Rick Johnson >> wrote: >> > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: >> > >> >> Here's one example: reference cycles. When do they get detected? >> >> Taking a really simple situation: >> >> >> >> class Foo: >> >> def __init__(self): >> >> self.self = self >> > >> > *shudders* >> > >> > Can you provide a real world example in which you need an >> > object which circularly references _itself_? [...] >> >> Not off hand, but I can provide an EXTREMELY real-world >> example of a fairly tight loop: exceptions. An exception >> has a reference to the local variables it came from, and >> those locals may well include the exception itself: >> >> try: >> 1/0 >> except Exception as e: >> print(e) >> >> The ZeroDivisionError has a reference to the locals, and >> 'e' in the locals refers to that very exception object. > > AFAIK, Python allows access to the "locals namespace" in > every scope, so i'm not sure how this specific case is any > more unique than any of the innumerable other cases that > could be presented. (but if i'm missing something, feel free > explain) > > Also, i'm not convinced that a reference which is > encapsulated by the current local namespace (even if the > containing dict is an attribute of some object!) is the > equivalent of `self.bar = bar`. (NOTE: i used `bar` instead > of `self` so the comparison would be more clear) > > ## Python 2.x ## > >>> class Foo(object): > ... def __init__(self): > ... self.bar = "bar" > ... print locals() > ... > >>> foo = Foo() > {'self': <__main__.Foo object at 0x027A6890>} > >>> foo.__dict__ > {'bar': 'bar'} > > > So, yeah. The challenge remains: "Produce a real world > example in which an object stores a reference of itself, and > then provide an acceptable justification for such a thing." The exact same example I already mentioned. Here, I'll make it perfectly clear for you. >>> def exc(): ... try: 1/0 ... except Exception as e: return e ... >>> ex = exc() >>> ex.__traceback__.tb_frame.f_globals["ex"] is ex True Is it real-world? Yep. EVERY SINGLE EXCEPTION has this. Acceptable justification? Printing out tracebacks that involve objects from your local or global scope. Got it now? ChrisA From rantingrickjohnson at gmail.com Thu Mar 1 18:35:14 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 15:35:14 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <1519876799l.9175092l.0l@psu.edu> Message-ID: <37811160-249b-4fe3-a42a-dd867d201441@googlegroups.com> On Wednesday, February 28, 2018 at 10:03:56 PM UTC-6, ROGER GRAYDON CHRISTMAN wrote: [...] > If you want something that looks like a real world example, > consider the very common doubly-linked list: > > [ 1 ] <---> [ 2 ] <---> [ 3 ] <--.....--> [ N ] > > This is chock-full of reference cycles, everywhere you see > a <--> Then you have a wrapper List object that has a head > referencing the 1 node and a tail referencing the N node. > And you decide you don't need that List any more. > > You could: > > 1) Take the time the code to carefully delete > every single node in this linked list, taking the time to > set all of the internal references to None, or Hmm, not pythonic though. > 2) Just discard your List object, and let the GC take care > of the rest. Agreed. And just for the record: I'm not arguing _against_ garbage collection, more that, since I have always been disturbed by a Python reality in which __del__ is virtually useless, my intent here was simply to throw my support behind the general concept of "Deterministic Object Destruction". Now, whether or not DOD can be achieved is another matter altogether. O:-) Sometimes certain design architectures force us to make compromises that we'd prefer not to make (GIL, for instance). And it may be that __del__ is an unfortunate side-effect of another helpful feature that we cannot bring our pythonic selfs to live without, and therefore, __del__ was sacrificed in favor of that feature. And if that is the case, then i can live with it. From johnpote at jptechnical.co.uk Thu Mar 1 18:37:54 2018 From: johnpote at jptechnical.co.uk (John Pote) Date: Thu, 1 Mar 2018 23:37:54 +0000 Subject: csv module and NULL data byte In-Reply-To: References: Message-ID: <4b0e7c15-81ac-2736-a750-bff0b4cce3c5@jptechnical.co.uk> On 01/03/2018 02:38, Dennis Lee Bieber wrote: > On Wed, 28 Feb 2018 23:40:41 +0000, John Pote > declaimed the following: > > >> ??? with open( fname, 'rt', encoding='iso-8859-1' ) as csvfile: > Pardon? Has the CSV module changed in the last year or so? Python 3.6 docs say csv reader has to be given an iterator that supplies strings. That requires the file to be opened 'rt' mode as 'rb' returns bytes objects and the csv reader throws an exception. I did a test to check and with the encoding='iso-8859-1' the file obj does pass through all 256 possible byte values without exception. Interesting point - how are the 'bytes' from the file translated to Python's unicode strings. Only doing this for 3.6 so have not checked 2.7 docs. > > Last time I read the documentation, it was recommended that the file be > opened in BINARY mode ("rb"). Using the 'rb' mode results in the helpful exception, ??? >>python36 csvTest.py ??? Traceback (most recent call last): ??? ? File "csvTest.py", line 25, in ??? ??? data = list( csvreader ) ??? _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?) Good point so thanks anyway, John From ooomzay at gmail.com Thu Mar 1 18:38:15 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Thu, 1 Mar 2018 15:38:15 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <871sh4ps9b.fsf@nightsong.com> References: <871sh4ps9b.fsf@nightsong.com> Message-ID: <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> On Thursday, March 1, 2018 at 12:15:57 AM UTC, Paul Rubin wrote: > RAII is similar to Python's "with" statement. So it sounds like OP > wants to replace one "malignant carbuncle" with another one. I would like to understand why you think RAII is not substantially more pythonic than "With". Below I have sketched out a File-like resource management scenario using both PEP343 and RAII idioms for comparison. First lets look at the implementation of the competing resource management classes:- class PEP343FileAccess(): ''' A File Access-like resource with self-contained Context Manager for use with "with". This could be acheived with two separate classes but I don't think that adds anything except lines of code. ''' # Structors def __init__(self, filename, mode): self.filename = filename self.mode = mode self.handle = None # dummy file content accessible only when handle is not None self.lines = ['one', 'two', 'three'] # PEP343 Context Manager compliance def __enter__(self): self.handle = low_level_file_open(self.filename, self.mode) # fictitious return self def __exit__(self, extype, exvalue, extraceback): low_level_file_close(self.handle) # fictitious function self.handle = None # Example methods def __iter__(self): assert self.handle, "File has been closed" for line in self.lines: yield line def write(self, line): assert self.handle, "File has been closed" self.lines.append(line) class RAIIFileAccess(): '''File Access-like Resource using RIAA idiom''' # Structors def __init__(self, filename, mode): self.handle = low_level_file_open(filename, mode) # fictitious # dummy content accessible as long as the object exists (invariant) self.lines = ['one', 'two', 'three'] def __del__(self): low_level_file_close(self.handle) # fictitious function # Example methods def __iter__(self): for line in self.lines: yield line def write(self, line): self.lines.append(line) What I see is that PEP343 requires two new methods: __enter__ & __exit__. RIAA requires no new methods. RIAA resources are invariant: If you have a reference to it you can use it. PEP343 resources can not be invariant: To be robust the enter/exit state must be tracked and checked. (assert self.handle in the example) Now lets look at example resource usage:- def pep343_example(): with PEP343FileAccess("src.txt", 'r') as src, PEP343FileAccess("dst.txt", 'w') as dst: for line in src: dst.write(line) def raii_example(): src = RAIIFileAccess("src.txt", 'r') dst = RAIIFileAccess("dst.txt", 'w') for line in src: dst.write(line) PEP343 requires specialised "with" syntax, RIAA requires no new syntax. Furthermore, although src & dst objects are still accessible outside the PEP343 "with" block they are not in a usable state (not invariant). In the RIAA case the resources are guaranteed to be in a usable state as long as any reference exists (invariant). References can also be safely passed around.The resource will be freed/closed when the last man has finished with it, even in the face of exceptions. From johnpote at jptechnical.co.uk Thu Mar 1 18:57:09 2018 From: johnpote at jptechnical.co.uk (John Pote) Date: Thu, 1 Mar 2018 23:57:09 +0000 Subject: csv module and NULL data byte In-Reply-To: <20180228193535.144393fb@bigbox.christie.dr> References: <20180228193535.144393fb@bigbox.christie.dr> Message-ID: <359622c5-d293-3774-1be6-d87f5adc222b@jptechnical.co.uk> On 01/03/2018 01:35, Tim Chase wrote: > While inelegant, I've "solved" this with a wrapper/generator > > f = file(fname, ?) > g = (line.replace('\0', '') for line in f) I wondered about something like this but thought if there's a way of avoiding the extra step it would keep the execution speed up. My next thought was to pass a custom encoder to the open() that translates NULLs to, say, 0x01. It won't make any difference to change one corrupt value to a different corrupt value. > reader = csv.reader(g, ?) > for row in reader: > process(row) Regards, John From rantingrickjohnson at gmail.com Thu Mar 1 18:58:48 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 15:58:48 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: <7cf6f26f-60b1-4a20-86d2-951ea9bf0115@googlegroups.com> On Wednesday, February 28, 2018 at 10:26:26 PM UTC-6, Steven D'Aprano wrote: > On Wed, 28 Feb 2018 18:46:05 -0800, Rick Johnson wrote: > > > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico > > wrote: > > > >> Here's one example: reference cycles. When do they get detected? Taking > >> a really simple situation: > >> > >> class Foo: > >> def __init__(self): > >> self.self = self > > > > *shudders* > > > > Can you provide a real world example in which you need an object which > > circularly references _itself_? This looks like a highly contrived > > example used to (1) merely win the argument, and (2) Bump fib() up one > > position from it's current position as "the worst introductory example > > of how to write a function in the history of programming tutorials" > > > Of course it is a contrived example: it is close to the > simplest possible example of a reference cycle. Its not > intended as an example of useful code. I support your appeal to simplicity, but surely you will admit that circular references (CR) of this type are extremely rare. The most common usage of CR is the node tree. > (Useful or not, your interpreter better not crash when > faced with a cycle like this.) Well, if you want your language to have CR as a feature, then yes. Some folks believe that CRs are evil. But i'm not one of those people. I believe CRs are useful in some specific cases. But a programmer must use them wisely. > But of course there are uses for objects to hold a > reference to themselves. If you're writing a general > purpose collection (say, a list or dict) then you better be > able to cope with the case that somebody puts your list in > itself: > > L = [] > L.append(L) > > either directly, as above, or indirectly. A good example. And one that is easy to overlook. > And that's the point really: cycles don't just occur when > an object holds a reference to itself. They occur when one > object holds a reference to another object, which holds a > reference to a third, which holds a reference to a fourth > (and so on...), which eventually holds a reference to the > original again. > > Cycles are useful. For example, you might think of a window > which holds a reference to a control, which in turn needs a > reference to its owning window. Or a network which contains > loops, say a network of streets, or networks of disease > transmission [...] Agreed. I don't buy into the religion that _all_ CRs are evil. Those who make such claims are dealing in absolutes. And as Obi- wan warned Anakin, only a Sith deals in absolutes. ;-) From rosuav at gmail.com Thu Mar 1 18:59:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 10:59:11 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 10:38 AM, wrote: > def raii_example(): > > src = RAIIFileAccess("src.txt", 'r') > dst = RAIIFileAccess("dst.txt", 'w') > > for line in src: > dst.write(line) > What happens if we make this change? def raii_example(): global dst src = RAIIFileAccess("src.txt", 'r') dst = RAIIFileAccess("dst.txt", 'w') for line in src: dst.write(line) When does the destination file get closed? ChrisA From ooomzay at gmail.com Thu Mar 1 19:02:36 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Thu, 1 Mar 2018 16:02:36 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <80de5e21-ae11-436a-a91f-973dace25d75@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <80de5e21-ae11-436a-a91f-973dace25d75@googlegroups.com> Message-ID: <3fa7e739-b822-4cd2-b163-675d7e367929@googlegroups.com> On Thursday, March 1, 2018 at 11:51:50 PM UTC, Lawrence D?Oliveiro wrote: > On Friday, March 2, 2018 at 12:39:01 PM UTC+13, ooo... at gmail.com wrote: > > class RAIIFileAccess(): > > '''File Access-like Resource using [RAII] idiom''' > > > > ... > > > > def __del__(self): > > low_level_file_close(self.handle) # fictitious function > > This may be OK for files opening for reading, not so good for writing. Please could you explain the issue you perceive with writing? From rosuav at gmail.com Thu Mar 1 19:05:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 11:05:44 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <7cf6f26f-60b1-4a20-86d2-951ea9bf0115@googlegroups.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <7cf6f26f-60b1-4a20-86d2-951ea9bf0115@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 10:58 AM, Rick Johnson wrote: > I don't buy into the religion that _all_ CRs are evil. Those > who make such claims are dealing in absolutes. And as Obi- > wan warned Anakin, only a Sith deals in absolutes. ;-) So mathematicians are all Sith? ChrisA From ooomzay at gmail.com Thu Mar 1 19:07:06 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Thu, 1 Mar 2018 16:07:06 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> Message-ID: On Thursday, March 1, 2018 at 11:59:26 PM UTC, Chris Angelico wrote: > On Fri, Mar 2, 2018 at 10:38 AM, ooomzay wrote: > > def raii_example(): > > > > src = RAIIFileAccess("src.txt", 'r') > > dst = RAIIFileAccess("dst.txt", 'w') > > > > for line in src: > > dst.write(line) > > What happens if we make this change? > > def raii_example(): > global dst > src = RAIIFileAccess("src.txt", 'r') > dst = RAIIFileAccess("dst.txt", 'w') > > for line in src: > dst.write(line) > > When does the destination file get closed? When you execute:- del dst or:- dst = something_else From rosuav at gmail.com Thu Mar 1 19:14:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 11:14:36 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 11:07 AM, wrote: > On Thursday, March 1, 2018 at 11:59:26 PM UTC, Chris Angelico wrote: >> On Fri, Mar 2, 2018 at 10:38 AM, ooomzay wrote: >> > def raii_example(): >> > >> > src = RAIIFileAccess("src.txt", 'r') >> > dst = RAIIFileAccess("dst.txt", 'w') >> > >> > for line in src: >> > dst.write(line) >> >> What happens if we make this change? >> >> def raii_example(): >> global dst >> src = RAIIFileAccess("src.txt", 'r') >> dst = RAIIFileAccess("dst.txt", 'w') >> >> for line in src: >> dst.write(line) >> >> When does the destination file get closed? > > When you execute:- > > del dst > > or:- > > dst = something_else > What if you don't? ChrisA From rantingrickjohnson at gmail.com Thu Mar 1 19:14:58 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 16:14:58 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <87r2p4z6xv.fsf@nightsong.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> Message-ID: On Wednesday, February 28, 2018 at 11:44:39 PM UTC-6, Paul Rubin wrote: > Rick Johnson writes: > > Can you provide a real world example in which you need an > > object which circularly references _itself_? > > DOM trees are a classic example (see the various DOM > modules in the Python stdlib). Non-leaf nodes have a list > of child nodes, child nodes have pointers back upwards to > their parent, and each child node has pointers to its left > and right siblings if they exist. It's all very circular. Hmm, Yes. I am familiar with these types of tree structures. But i don't remember encountering any nodes which explicitly referenced _themselves_ in the manner that Chris presented: class Node(object): def __init__(self): self.self = self self.children = [] ... True, each node may possibly save a reference to a parent node, and also contain N child nodes (either proxies or actual instances) but not something akin to what Chris presented earlier in this thread. But if i missed something, feel free to elaborate... From python.list at tim.thechases.com Thu Mar 1 19:15:50 2018 From: python.list at tim.thechases.com (Tim Chase) Date: Thu, 1 Mar 2018 18:15:50 -0600 Subject: csv module and NULL data byte In-Reply-To: <359622c5-d293-3774-1be6-d87f5adc222b@jptechnical.co.uk> References: <20180228193535.144393fb@bigbox.christie.dr> <359622c5-d293-3774-1be6-d87f5adc222b@jptechnical.co.uk> Message-ID: <20180301181550.1328231f@bigbox.christie.dr> On 2018-03-01 23:57, John Pote wrote: > On 01/03/2018 01:35, Tim Chase wrote: > > While inelegant, I've "solved" this with a wrapper/generator > > > > f = file(fname, ?) > > g = (line.replace('\0', '') for line in f) > I wondered about something like this but thought if there's a way > of avoiding the extra step it would keep the execution speed up. There shouldn't be noticeable performance issues with using a generator. It's also lazy so it's not like it's pulling the entire file into memory; no more than one line at a time. > My next thought was to pass a custom encoder to the open() that > translates NULLs to, say, 0x01. It won't make any difference to > change one corrupt value to a different corrupt value. > > reader = csv.reader(g, ?) > > for row in reader: > > process(row) ...which is pretty much exactly what my generator solution does: putting a translating encoder between the open() and the csv.reader() call. -tkc From python at mrabarnett.plus.com Thu Mar 1 19:21:55 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 2 Mar 2018 00:21:55 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> Message-ID: <7fe6dd8d-fcd9-5863-fb5a-06ce18adf8fc@mrabarnett.plus.com> On 2018-03-01 23:38, ooomzay at gmail.com wrote: > On Thursday, March 1, 2018 at 12:15:57 AM UTC, Paul Rubin wrote: >> RAII is similar to Python's "with" statement. So it sounds like OP >> wants to replace one "malignant carbuncle" with another one. > > I would like to understand why you think RAII is not substantially more > pythonic than "With". Below I have sketched out a File-like resource > management scenario using both PEP343 and RAII idioms for comparison. > > First lets look at the implementation of the competing resource management > classes:- > > > class PEP343FileAccess(): > ''' > A File Access-like resource with self-contained Context Manager for use > with "with". > > This could be acheived with two separate classes but I don't think that > adds anything except lines of code. > ''' > > # Structors > > def __init__(self, filename, mode): > self.filename = filename > self.mode = mode > self.handle = None > # dummy file content accessible only when handle is not None > self.lines = ['one', 'two', 'three'] > > # PEP343 Context Manager compliance > > def __enter__(self): > self.handle = low_level_file_open(self.filename, self.mode) # fictitious > return self > > def __exit__(self, extype, exvalue, extraceback): > low_level_file_close(self.handle) # fictitious function > self.handle = None > > # Example methods > > def __iter__(self): > assert self.handle, "File has been closed" > for line in self.lines: > yield line > > def write(self, line): > assert self.handle, "File has been closed" > self.lines.append(line) > > > class RAIIFileAccess(): > '''File Access-like Resource using RIAA idiom''' > > # Structors > > def __init__(self, filename, mode): > self.handle = low_level_file_open(filename, mode) # fictitious > # dummy content accessible as long as the object exists (invariant) > self.lines = ['one', 'two', 'three'] > > def __del__(self): > low_level_file_close(self.handle) # fictitious function > > # Example methods > > def __iter__(self): > for line in self.lines: > yield line > > def write(self, line): > self.lines.append(line) > > > What I see is that PEP343 requires two new methods: __enter__ & __exit__. > RIAA requires no new methods. > > RIAA resources are invariant: If you have a reference to it you can use it. > > PEP343 resources can not be invariant: To be robust the enter/exit state > must be tracked and checked. (assert self.handle in the example) > > Now lets look at example resource usage:- > > > def pep343_example(): > > with PEP343FileAccess("src.txt", 'r') as src, > PEP343FileAccess("dst.txt", 'w') as dst: > > for line in src: > dst.write(line) > > def raii_example(): > > src = RAIIFileAccess("src.txt", 'r') > dst = RAIIFileAccess("dst.txt", 'w') > > for line in src: > dst.write(line) > > > PEP343 requires specialised "with" syntax, RIAA requires no new syntax. > > Furthermore, although src & dst objects are still accessible outside the > PEP343 "with" block they are not in a usable state (not invariant). > > In the RIAA case the resources are guaranteed to be in a usable state as long > as any reference exists (invariant). References can also be safely > passed around.The resource will be freed/closed when the last man has finished > with it, even in the face of exceptions. > What's the difference between 'RAIIFileAccess' and 'open'? From ooomzay at gmail.com Thu Mar 1 19:26:47 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Thu, 1 Mar 2018 16:26:47 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> Message-ID: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> On Friday, March 2, 2018 at 12:14:53 AM UTC, Chris Angelico wrote: > On Fri, Mar 2, 2018 at 11:07 AM, ooomzay wrote: > > On Thursday, March 1, 2018 at 11:59:26 PM UTC, Chris Angelico wrote: > >> On Fri, Mar 2, 2018 at 10:38 AM, ooomzay wrote: > >> > def raii_example(): > >> > > >> > src = RAIIFileAccess("src.txt", 'r') > >> > dst = RAIIFileAccess("dst.txt", 'w') > >> > > >> > for line in src: > >> > dst.write(line) > >> > >> What happens if we make this change? > >> > >> def raii_example(): > >> global dst > >> src = RAIIFileAccess("src.txt", 'r') > >> dst = RAIIFileAccess("dst.txt", 'w') > >> > >> for line in src: > >> dst.write(line) > >> > >> When does the destination file get closed? > > > > When you execute:- > > > > del dst > > > > or:- > > > > dst = something_else > > What if you don't? Then the resource will remain open until your script exits at which point it is probably not very well defined exactly when or even if the destructor/__del__ will be called. I.e. Don't do this! Did you have some realistic case in mind or are you just probing the behaviour? From rantingrickjohnson at gmail.com Thu Mar 1 19:31:14 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 16:31:14 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <7cf6f26f-60b1-4a20-86d2-951ea9bf0115@googlegroups.com> Message-ID: <91500c31-6a88-4cd1-bef6-6bc2a5e1263b@googlegroups.com> On Thursday, March 1, 2018 at 6:10:45 PM UTC-6, Chris Angelico wrote: > On Fri, Mar 2, 2018 at 10:58 AM, Rick Johnson > wrote: > > I don't buy into the religion that _all_ CRs are evil. Those > > who make such claims are dealing in absolutes. And as Obi- > > wan warned Anakin, only a Sith deals in absolutes. ;-) > > So mathematicians are all Sith? Yes. They always want you to "feel the power of the dark side". From rantingrickjohnson at gmail.com Thu Mar 1 19:49:35 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 16:49:35 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Thursday, March 1, 2018 at 1:54:40 AM UTC-6, Serhiy Storchaka wrote: [...] > Every global function (or method of global class) creates a > reference cycle. > > def f(): pass > > f.__globals__['f'] is f (Note: This is also a response to dieter) This is true, but it does not answer the challenge directly because function CRs are a consequence of Python _internals_ *NOT* consequences of a custom class written by a programmer which references itself _explicitly_. Remember, the challenge is _not_ simply a matter of circular references (there is literally tons of Python code out there which creates CRs for various reasons), no, the challenge is to create a custom class which references _itself_, and then explain (or demonstrate) how such a thing serves a practical purpose. @Steven. Whilst your "list that appends itself" was very clever, it only indirectly answered the challenge. So if you want the gold metal, i need to see a _direct_ solution for the challenge. From songofacandy at gmail.com Thu Mar 1 19:57:27 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 2 Mar 2018 09:57:27 +0900 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: > > Remember, the challenge is _not_ simply a matter of circular > references (there is literally tons of Python code out there > which creates CRs for various reasons), no, the challenge is > to create a custom class which references _itself_, and then > explain (or demonstrate) how such a thing serves a practical > purpose. I can't agree with it. Since this thread is about robust RAII implementation, the main challenge is "Is it possible to make **all** Python interpreter implement it without significant overhead?" In CPython case, the challenge is "Can we removing all possible reference cycles?". Even if you can't agree some examples explained is "practical", it can be enough reason for we don't go to proposed RAII way. Regards, -- INADA Naoki From gheskett at shentel.net Thu Mar 1 20:03:28 2018 From: gheskett at shentel.net (Gene Heskett) Date: Thu, 1 Mar 2018 20:03:28 -0500 Subject: Problem: Need galileo running on debian wheezy In-Reply-To: References: <201802282130.07622.gheskett@shentel.net> Message-ID: <201803012003.28272.gheskett@shentel.net> On Thursday 01 March 2018 15:31:32 Cousin Stanley wrote: > Gene Heskett wrote: > > I know its supposed to be in the debian stretch repo's. > > > > I've been told to get a fitbit, but they don't support linux > > of any flavor, and that leaves galileo as the possible solution? > > > > So how should I proceed since the only stretch machine I have ATM > > is an arm64, aka a rock64. > > The galileo package is available under debian stretch > on my rock64 .... > > # apt-cache policy galileo > > galileo: > Installed: (none) > Candidate: 0.5.1-4 > Version table: > 0.5.1-4 500 > 500 http://approx:9999/debian stretch/main arm64 Packages > > > Package details .... > > # apt-cache show galileo > > > However, I have no clue as to its usage or effectiveness .... Its supposed to be an interface that can talk to a fitbit. But the one I bought is wifi only I think since the usb clipon intended to be used for recharging it, has only 2 contacts. And the rock64 doesn't have wifi hardware that I know of. One of the reasons I bought a couple of them, as its required it be turned off on a pi for anything like rt-preempt use on a pi as its noise interferes with everything else but then found there is no simple "make install" for a realtime kernel thats doesn't just brick the sdcard. And everytime someone on the pine64 forum points me at another way to do it, I grab the sw, only to find its not been updated since 2012? I've wasted $250 so far trying. Anyway, I'll bring in the 2nd one and see if an lshw shows a wifi with a minimal stretch +xfce install. Quick answer, no. That leaves an r-pi3b, which does have wifi. But since its busy drowning frogs outside, I'll see if I can find a stretch install for it tomorrow. Currently has a jessie install. So, is galileo shown for armhf's? The reason for posting here is that galileo is supposedly written in python, but don't know if 2 or 3. Thanks, Stanley. > > -- > Stanley C. Kitching > Human Being > Phoenix, Arizona -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From gheskett at shentel.net Thu Mar 1 20:14:48 2018 From: gheskett at shentel.net (Gene Heskett) Date: Thu, 1 Mar 2018 20:14:48 -0500 Subject: Problem: Need galileo running on debian wheezy In-Reply-To: References: <201803011344.27353.gheskett@shentel.net> Message-ID: <201803012014.48019.gheskett@shentel.net> On Thursday 01 March 2018 17:29:27 Wildman via Python-list wrote: > On Thu, 01 Mar 2018 13:44:27 -0500, Gene Heskett wrote: > > I know its supposed to be in the debian stretch repo's. > > > > I've been told to get a fitbit, but they don't support linux of any > > flavor, and that leaves galileo as the possible solution? > > > > So how should I proceed since the only stretch machine I have ATM is > > an arm64, aka a rock64. > > > > -- > > Cheers, Gene Heskett > > I would suggest to download it from stretch's repo > and open it with gdebi. It will tell you if the > depends are met. Or better yet since this elderly Asus board doesn't have a wifi, and all slots are full, lashup an r-pi 3b with a copy of stretch tomorrow, as it does have a wifi. Right now its busy drowning frogs out. With mixed intermittent horizontal rain. > -- > GNU/Linux user #557453 > The cow died so I don't need your bull! Thanks Wildman. -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From steve+comp.lang.python at pearwood.info Thu Mar 1 23:11:20 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 2 Mar 2018 04:11:20 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Thu, 01 Mar 2018 16:49:35 -0800, Rick Johnson wrote: > Remember, the challenge is _not_ simply a matter of circular references > (there is literally tons of Python code out there which creates CRs for > various reasons), no, the challenge is to create a custom class which > references _itself_, and then explain (or demonstrate) how such a thing > serves a practical purpose. Nobody cares about this challenge, since nobody but yourself seems to think that it is important or necessary whether "self.foo = self" specifically gets used or not. It's just an illustration of an entire class of cycles. If your language can deal with cycles, then it can naturally deal with the case `self.foo = self` too, and without needing any special handling. There's no benefit to banning this one special case when an infinite variety of other cycles are possible. We could prohibit it, by adding the check: if value is self: raise RickError('self.foo = self not allowed') to every attribute assignment, but for what purpose? Doing so doesn't prevent cycles, it doesn't avoid a common source of bugs, and it needlessly slows down every attribute assignment. And for the record, consider a tree of nodes, where each node points back at the root of the tree, which is also a node. So what does the root node point back at? -- Steve From steve+comp.lang.python at pearwood.info Thu Mar 1 23:33:10 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 2 Mar 2018 04:33:10 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> Message-ID: On Thu, 01 Mar 2018 16:26:47 -0800, ooomzay wrote: >> >> When does the destination file get closed? >> > >> > When you execute:- >> > >> > del dst >> > >> > or:- >> > >> > dst = something_else >> >> What if you don't? > > Then the resource will remain open until your script exits at which > point it is probably not very well defined exactly when or even if the > destructor/__del__ will be called. > > I.e. Don't do this! Did you have some realistic case in mind or are you > just probing the behaviour? If you're going to *require* the programmer to explicitly del the reference: f = open("file") text = f.read() del f then you might as well require them to explicitly close the file: f = open("file") text = f.read() f.close() which we know from many years experience is not satisfactory except for the simplest scripts that don't need to care about resource management. That's the fatal flaw in RAII: for resources that you care about their *timely* release, the problem is that the lifespan of the resource may not be the same as the lifetime of the object. Especially for files, the problem is that the lifespan of resource (the time you are actually using it) may be significantly less than the lifespan of the object holding onto that resource. Since there's no way for the interpreter to know whether or not you have finished with the resource, you have a choice: - close the resource yourself (either explicitly with file.close(), or implicitly with a context manager); - or keep the resource open indefinitely, until such eventual time that the object is garbage collected and the resource closed. -- Steve From ian.g.kelly at gmail.com Fri Mar 2 00:16:29 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 1 Mar 2018 22:16:29 -0700 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Wed, Feb 28, 2018 at 8:00 PM, Chris Angelico wrote: > On Thu, Mar 1, 2018 at 1:46 PM, Rick Johnson > wrote: >> On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: >> >>> Here's one example: reference cycles. When do they get detected? >>> Taking a really simple situation: >>> >>> class Foo: >>> def __init__(self): >>> self.self = self >> >> *shudders* >> >> Can you provide a real world example in which you need an >> object which circularly references _itself_? This looks like >> a highly contrived example used to (1) merely win the >> argument, and (2) Bump fib() up one position from it's >> current position as "the worst introductory example of how >> to write a function in the history of programming tutorials" > > Not off hand, but I can provide an EXTREMELY real-world example of a > fairly tight loop: exceptions. An exception has a reference to the > local variables it came from, and those locals may well include the > exception itself: > > try: > 1/0 > except Exception as e: > print(e) > > The ZeroDivisionError has a reference to the locals, and 'e' in the > locals refers to that very exception object. The problem with this example of course is that the variable 'e' is scoped to the except block and automatically del'ed when it exits. From rustompmody at gmail.com Fri Mar 2 00:29:16 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Thu, 1 Mar 2018 21:29:16 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> Message-ID: <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> On Friday, March 2, 2018 at 10:05:41 AM UTC+5:30, Steven D'Aprano wrote: > On Thu, 01 Mar 2018 16:26:47 -0800, ooomzay wrote: > > >> >> When does the destination file get closed? > >> > > >> > When you execute:- > >> > > >> > del dst > >> > > >> > or:- > >> > > >> > dst = something_else > >> > >> What if you don't? > > > > Then the resource will remain open until your script exits at which > > point it is probably not very well defined exactly when or even if the > > destructor/__del__ will be called. > > > > I.e. Don't do this! Did you have some realistic case in mind or are you > > just probing the behaviour? > > > If you're going to *require* the programmer to explicitly del the > reference: > > f = open("file") > text = f.read() > del f > > then you might as well require them to explicitly close the file: > > f = open("file") > text = f.read() > f.close() > > which we know from many years experience is not satisfactory except for > the simplest scripts that don't need to care about resource management. > > That's the fatal flaw in RAII: for resources that you care about their > *timely* release, the problem is that the lifespan of the resource may > not be the same as the lifetime of the object. Especially for files, the > problem is that the lifespan of resource (the time you are actually using > it) may be significantly less than the lifespan of the object holding > onto that resource. Since there's no way for the interpreter to know > whether or not you have finished with the resource, you have a choice: > > - close the resource yourself (either explicitly with file.close(), > or implicitly with a context manager); > > - or keep the resource open indefinitely, until such eventual time > that the object is garbage collected and the resource closed. Please excuse if this has been addressed above and/or its too basic: What's the difference between RAII and python's with/context-managers? From ian.g.kelly at gmail.com Fri Mar 2 00:32:34 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 1 Mar 2018 22:32:34 -0700 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <1519876799l.9175092l.0l@psu.edu> References: <1519876799l.9175092l.0l@psu.edu> Message-ID: On Wed, Feb 28, 2018 at 9:00 PM, ROGER GRAYDON CHRISTMAN wrote: > > On Wed, Feb 28, 2018, Rick Johnson wrote: > > On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: >> >>> Here's one example: reference cycles. When do they get detected? >>> Taking a really simple situation: >>> >>> class Foo: >>> def __init__(self): >>> self.self = self >> >>*shudders* >> >>Can you provide a real world example in which you need an >>object which circularly references _itself_? This looks like >>a highly contrived example used to (1) merely win the >>argument, and (2) Bump fib() up one position from it's >>current position as "the worst introductory example of how >>to write a function in the history of programming tutorials" > > If you want something that looks like a real world example, > consider the very common doubly-linked list: > [ 1 ] <---> [ 2 ] <---> [ 3 ] <--.....--> [ N ] > > This is chock-full of reference cycles, everywhere you see a <--> > > Then you have a wrapper List object that has a head referencing > the 1 node and a tail referencing the N node. > > And you decide you don't need that List any more. > > You could: > 1) Take the time the code to carefully delete every single node > in this linked list, taking the time to set all of the internal references to > None, or > 2) Just discard your List object, and let the GC take care of the rest. 3) Use strong references for the left-to-right links and weak references for the right-to-left links. Poof, no reference cycle. Even a cyclical graph without ownership relationships can be expressed without reference cycles. One approach is to build a spanning tree of strong references and let all the redundant edges be weak. Just be careful not to cleave your spanning tree when deleting edges. Another, perhaps simpler, approach is to just have a collection containing all the nodes and then let *all* the edges be weak. From rosuav at gmail.com Fri Mar 2 00:35:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 16:35:44 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 4:16 PM, Ian Kelly wrote: > On Wed, Feb 28, 2018 at 8:00 PM, Chris Angelico wrote: >> On Thu, Mar 1, 2018 at 1:46 PM, Rick Johnson >> wrote: >>> On Wednesday, February 28, 2018 at 5:02:17 PM UTC-6, Chris Angelico wrote: >>> >>>> Here's one example: reference cycles. When do they get detected? >>>> Taking a really simple situation: >>>> >>>> class Foo: >>>> def __init__(self): >>>> self.self = self >>> >>> *shudders* >>> >>> Can you provide a real world example in which you need an >>> object which circularly references _itself_? This looks like >>> a highly contrived example used to (1) merely win the >>> argument, and (2) Bump fib() up one position from it's >>> current position as "the worst introductory example of how >>> to write a function in the history of programming tutorials" >> >> Not off hand, but I can provide an EXTREMELY real-world example of a >> fairly tight loop: exceptions. An exception has a reference to the >> local variables it came from, and those locals may well include the >> exception itself: >> >> try: >> 1/0 >> except Exception as e: >> print(e) >> >> The ZeroDivisionError has a reference to the locals, and 'e' in the >> locals refers to that very exception object. > > The problem with this example of course is that the variable 'e' is > scoped to the except block and automatically del'ed when it exits. Or, to be more accurate: The language design acknowledges that this reference cycle is a fundamental problem, and the *solution* is that there's an implicit "e = None; del e" at the end of the except block. You can easily defeat that protection with "except Exception as ee: e = ee", if you want to demonstrate the cycle. ChrisA From ian.g.kelly at gmail.com Fri Mar 2 00:46:56 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Thu, 1 Mar 2018 22:46:56 -0700 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Thu, Mar 1, 2018 at 10:35 PM, Chris Angelico wrote: > On Fri, Mar 2, 2018 at 4:16 PM, Ian Kelly wrote: >> On Wed, Feb 28, 2018 at 8:00 PM, Chris Angelico wrote: >>> Not off hand, but I can provide an EXTREMELY real-world example of a >>> fairly tight loop: exceptions. An exception has a reference to the >>> local variables it came from, and those locals may well include the >>> exception itself: >>> >>> try: >>> 1/0 >>> except Exception as e: >>> print(e) >>> >>> The ZeroDivisionError has a reference to the locals, and 'e' in the >>> locals refers to that very exception object. >> >> The problem with this example of course is that the variable 'e' is >> scoped to the except block and automatically del'ed when it exits. > > Or, to be more accurate: The language design acknowledges that this > reference cycle is a fundamental problem, and the *solution* is that > there's an implicit "e = None; del e" at the end of the except block. > > You can easily defeat that protection with "except Exception as ee: e > = ee", if you want to demonstrate the cycle. Yes, but how often does this happen in practice? This situation was billed as "an EXTREMELY real-world example". In the real world, most of the time when you have an exception, you log it or handle it, and then you discard it. From steve+comp.lang.python at pearwood.info Fri Mar 2 00:57:58 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 2 Mar 2018 05:57:58 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Thu, 01 Mar 2018 22:46:56 -0700, Ian Kelly wrote: > On Thu, Mar 1, 2018 at 10:35 PM, Chris Angelico > wrote: >> On Fri, Mar 2, 2018 at 4:16 PM, Ian Kelly >> wrote: >>> On Wed, Feb 28, 2018 at 8:00 PM, Chris Angelico >>> wrote: >>>> Not off hand, but I can provide an EXTREMELY real-world example of a >>>> fairly tight loop: exceptions. An exception has a reference to the >>>> local variables it came from, and those locals may well include the >>>> exception itself: >>>> >>>> try: >>>> 1/0 >>>> except Exception as e: >>>> print(e) >>>> >>>> The ZeroDivisionError has a reference to the locals, and 'e' in the >>>> locals refers to that very exception object. >>> >>> The problem with this example of course is that the variable 'e' is >>> scoped to the except block and automatically del'ed when it exits. >> >> Or, to be more accurate: The language design acknowledges that this >> reference cycle is a fundamental problem, and the *solution* is that >> there's an implicit "e = None; del e" at the end of the except block. >> >> You can easily defeat that protection with "except Exception as ee: e = >> ee", if you want to demonstrate the cycle. > > Yes, but how often does this happen in practice? This situation was > billed as "an EXTREMELY real-world example". In the real world, most of > the time when you have an exception, you log it or handle it, and then > you discard it. What difference does that make it? For the period of time you are in the except block, you have a cycle. Nobody said the cycle has to persist for a long time. In other words, every single time you catch an exception in Python 3, you have the situation Chris was asked to demonstrate. That seems very common to me :-) -- Steve From rantingrickjohnson at gmail.com Fri Mar 2 00:58:15 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Thu, 1 Mar 2018 21:58:15 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Thursday, March 1, 2018 at 10:13:51 PM UTC-6, Steven D'Aprano wrote: [...] > And for the record, consider a tree of nodes, where each > node points back at the root of the tree, which is also a > node. So what does the root node point back at? Finally! A practical solution is offered that answers the challenge _directly_. And even though we had to wade through a swamp of waist-deep toxic goo to get here -- i must say -- i am relieved. Mr. D'Aprano, you win the gold. Now. Let's get back to Deterministic Object Destruction, shall we? From rosuav at gmail.com Fri Mar 2 01:02:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 17:02:02 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: On Fri, Mar 2, 2018 at 4:46 PM, Ian Kelly wrote: > On Thu, Mar 1, 2018 at 10:35 PM, Chris Angelico wrote: >> On Fri, Mar 2, 2018 at 4:16 PM, Ian Kelly wrote: >>> On Wed, Feb 28, 2018 at 8:00 PM, Chris Angelico wrote: >>>> Not off hand, but I can provide an EXTREMELY real-world example of a >>>> fairly tight loop: exceptions. An exception has a reference to the >>>> local variables it came from, and those locals may well include the >>>> exception itself: >>>> >>>> try: >>>> 1/0 >>>> except Exception as e: >>>> print(e) >>>> >>>> The ZeroDivisionError has a reference to the locals, and 'e' in the >>>> locals refers to that very exception object. >>> >>> The problem with this example of course is that the variable 'e' is >>> scoped to the except block and automatically del'ed when it exits. >> >> Or, to be more accurate: The language design acknowledges that this >> reference cycle is a fundamental problem, and the *solution* is that >> there's an implicit "e = None; del e" at the end of the except block. >> >> You can easily defeat that protection with "except Exception as ee: e >> = ee", if you want to demonstrate the cycle. > > Yes, but how often does this happen in practice? This situation was > billed as "an EXTREMELY real-world example". In the real world, most > of the time when you have an exception, you log it or handle it, and > then you discard it. The exception object itself is the cycle. The fact that the language has a default way of handling this doesn't change that. What happens, for instance, when you have another exception inside the except block, and they chain? Once the top-level exception is freed, there's still a definite reference cycle in the original. Or what if you have an interactive debugger that lets you inspect objects at point of crash? That's going to work by retaining a reference to the exception. So, yep, this is stuff that happens every day. Not contrived, not uncommon. ChrisA From songofacandy at gmail.com Fri Mar 2 01:09:24 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 2 Mar 2018 15:09:24 +0900 Subject: Homebrew changed default Python to python 3! In-Reply-To: References: Message-ID: https://github.com/Homebrew/homebrew-core/pull/24604 /use/local/bin/python is symlink to python3. vim is built with python3. You can install it from bottle. Thanks to Homebrew maintainers!! From storchaka at gmail.com Fri Mar 2 01:35:48 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 2 Mar 2018 08:35:48 +0200 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> Message-ID: 02.03.18 02:49, Rick Johnson ????: > This is true, but it does not answer the challenge directly > because function CRs are a consequence of Python _internals_ > *NOT* consequences of a custom class written by a programmer > which references itself _explicitly_. This doesn't matter. You question was "Can you provide a real world example in which you need an object which circularly references _itself_?" Global functions are such objects and unlikely you can write any reasonable program without using global functions. And if you have reference cycles in such fundamental objects you can't avoid detecting and breaking reference cycles in GC. If you want to see an example that creates a reference cycle in Python code, here is yet one real-word example: class C: def __init__(self, ...): ... self.foo = self.bar if cond else self.baz ... def bar(self, ...): ... def baz(self, ...): ... Depending on condition you need to call either method bar or method baz. You save a reference to method as instance attribute, but a method has a reference to the instance. This creates a cycle. From dieter at handshake.de Fri Mar 2 01:51:44 2018 From: dieter at handshake.de (dieter) Date: Fri, 02 Mar 2018 07:51:44 +0100 Subject: Strange problem with pip2 References: <87po4o2aki.fsf@munus.decebal.nl> Message-ID: <878tbbq8an.fsf@handshake.de> Cecil Westerhof writes: > There are three pip2 packages that should be updated: > apsw (3.13.0.post1) - Latest: 3.9.2.post1 [sdist] > mysql-utilities (1.6.4) - Latest: 1.4.3 [sdist] > pygobject (3.22.0) - Latest: 3.27.4 [sdist] > > But the strange thing is that the installed version is newer as the > new version. And when I try: > pip2 install --upgrade apsw > > I get: > Requirement already up-to-date: apsw in /usr/lib/python2.7/dist-packages Playing recently with a daily Ubuntu 18.4 preview, I have seen something similar: I was told that a Python package must be updated but Python's installation tools reported: current version already installed. I have not investigated (thinking it was a fault of the early Ubuntu version). From alenkabor129769 at gmail.com Fri Mar 2 02:40:07 2018 From: alenkabor129769 at gmail.com (alenkabor129769 at gmail.com) Date: Thu, 1 Mar 2018 23:40:07 -0800 (PST) Subject: Help me with the Python! ODE system. Message-ID: <02a5c438-025a-4bda-b8dc-58c3f224e5a3@googlegroups.com> I can not find an example with this function: https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.RK45.html#scipy.integrate.RK45. From bc at freeuk.com Fri Mar 2 07:05:04 2018 From: bc at freeuk.com (bartc) Date: Fri, 2 Mar 2018 12:05:04 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <87tvtyki4l.fsf@nightsong.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> <87tvtyki4l.fsf@nightsong.com> Message-ID: On 02/03/2018 08:15, Paul Rubin wrote: > If someone says "but > limited memory", consider that MicroPython runs on the BBC Micro:bit > board which has 16k of ram, and it uses gc. The specs say it also has 256KB of flash memory (ie. 'ROM'), so I suspect much of the program code resides there. -- bartc From bfaruq at ymail.com Fri Mar 2 08:04:50 2018 From: bfaruq at ymail.com (Faruq Bashir) Date: Fri, 2 Mar 2018 13:04:50 +0000 (UTC) Subject: APPLICATION NOT RUNNING. References: <1597615689.9157921.1519995890597.ref@mail.yahoo.com> Message-ID: <1597615689.9157921.1519995890597@mail.yahoo.com> I try to run an application with the latest version of python that is python 3.6.4 (32-bit) ., instead of running the application it only shows feel free to mail python-list at python.org if you continue to encounter issues,Please help me out thanks. From ooomzay at gmail.com Fri Mar 2 08:55:42 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Fri, 2 Mar 2018 05:55:42 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <7fe6dd8d-fcd9-5863-fb5a-06ce18adf8fc@mrabarnett.plus.com> Message-ID: <660b2489-faf6-4ef5-9115-1c5cacc49fea@googlegroups.com> On Friday, March 2, 2018 at 12:22:13 AM UTC, MRAB wrote: > On 2018-03-01 23:38, ooomzay wrote: [Snip] > > PEP343 requires two new methods: __enter__ & __exit__. > > RIAA requires no new methods. > > > > RIAA resources are invariant: If you have a reference to it you can use it. > > PEP343 resources can not be invariant: To be robust the enter/exit state > > must be tracked and checked. (assert self.handle in the example) [Snip] > > PEP343 requires specialised "with" syntax, RIAA requires no new syntax. > > Furthermore, although src & dst objects are still accessible outside the > > PEP343 "with" block they are not in a usable state (not invariant). > > > > In the RIAA case the resources are guaranteed to be in a usable state as long > > as any reference exists (invariant). References can also be safely > > passed around.The resource will be freed/closed when the last man has finished > > with it, even in the face of exceptions. > > [snip] > > > What's the difference between 'RAIIFileAccess' and 'open'? I listed all the differences I know of in my preceding post. (I have snipped the other stuff so they are easy to see). From ooomzay at gmail.com Fri Mar 2 09:18:46 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Fri, 2 Mar 2018 06:18:46 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <87tvtyki4l.fsf@nightsong.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> <87tvtyki4l.fsf@nightsong.com> Message-ID: On Friday, March 2, 2018 at 8:16:22 AM UTC, Paul Rubin wrote:[snip] > controlling stuff like file handles > with scopes (like with "with") is fine. How does with work for non-trivial/composite objects that represent/reference multiple resources or even a hierarchy of such objects where all the resources held must be released in a timely fashion when finished with? From ooomzay at gmail.com Fri Mar 2 09:32:03 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Fri, 2 Mar 2018 06:32:03 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <62f998f4-7077-4ba3-967f-6bbe893c69aa@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <80de5e21-ae11-436a-a91f-973dace25d75@googlegroups.com> <3fa7e739-b822-4cd2-b163-675d7e367929@googlegroups.com> <62f998f4-7077-4ba3-967f-6bbe893c69aa@googlegroups.com> Message-ID: On Friday, March 2, 2018 at 1:59:02 AM UTC, Lawrence D?Oliveiro wrote: > On Friday, March 2, 2018 at 1:03:08 PM UTC+13, ooo... at gmail.com wrote: > > On Thursday, March 1, 2018 at 11:51:50 PM UTC, Lawrence D?Oliveiro wrote: > >> On Friday, March 2, 2018 at 12:39:01 PM UTC+13, ooo... at gmail.com wrote: > >>> > >>> class RAIIFileAccess(): > >>> '''File Access-like Resource using [RAII] idiom''' > >>> ... > >>> def __del__(self): > >>> low_level_file_close(self.handle) # fictitious function > >> > >> This may be OK for files opening for reading, not so good for writing. > > > > Please could you explain the issue you perceive with writing? > > Compare the difference in behaviour between > > f = open("/dev/full", "w") > f.write("junk") > f.close() > > and > > f = open("/dev/full", "w") > f.write("junk") > f = None > > Should there be a difference in behaviour? Could you please be very explicit about why you think the PEP would be more suitable when reading a file than when writing a file? You have responded with another question that I just can't see the relevance of (yet). From rosuav at gmail.com Fri Mar 2 09:42:35 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 01:42:35 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> <87tvtyki4l.fsf@nightsong.com> Message-ID: On Sat, Mar 3, 2018 at 1:18 AM, wrote: > On Friday, March 2, 2018 at 8:16:22 AM UTC, Paul Rubin wrote:[snip] >> controlling stuff like file handles >> with scopes (like with "with") is fine. > > How does with work for non-trivial/composite objects that represent/reference multiple resources or even a hierarchy of such objects where all the resources held must be released in a timely fashion when finished with? > Can you give me an example that works with RAII but doesn't work in a 'with' statement? ChrisA From ooomzay at gmail.com Fri Mar 2 10:09:19 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Fri, 2 Mar 2018 07:09:19 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> Message-ID: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> On Friday, March 2, 2018 at 4:35:41 AM UTC, Steven D'Aprano wrote: > On Thu, 01 Mar 2018 16:26:47 -0800, ooomzay wrote: > > >> >> When does the destination file get closed? > >> > > >> > When you execute:- > >> > > >> > del dst > >> > > >> > or:- > >> > > >> > dst = something_else > >> > >> What if you don't? > > > > Then the resource will remain open until your script exits at which > > point it is probably not very well defined exactly when or even if the > > destructor/__del__ will be called. > > > > I.e. Don't do this! Did you have some realistic case in mind or are you > > just probing the behaviour? > > > If you're going to *require* the programmer to explicitly del the > reference: > > f = open("file") > text = f.read() > del f But I am not! On the contrary RAII frees the programmer from even having to remember to close the file. The poster asked what would happen if the resource was deliberately kept open by storing a reference at global scope. In practice CPython destroys it cleanly on exit - but I am not sure the language guarantees this - in any case RAII won't make things any worse in this respect. (Logfiles are a common example of such global resource.) > then you might as well require them to explicitly close the file: > > f = open("file") > text = f.read() > f.close() > > which we know from many years experience is not satisfactory except for > the simplest scripts that don't need to care about resource management. > That's the fatal flaw in RAII: We must be discussing a different RAII. That is the raison d'etre of RAII: RAII directly addresses this problem in an exception-safe way that does not burden the resource user at all. > the problem is that the lifespan of the resource may > not be the same as the lifetime of the object. > > Especially for files, the > problem is that the lifespan of resource (the time you are actually using > it) may be significantly less than the lifespan of the object holding > onto that resource. Since there's no way for the interpreter to know > whether or not you have finished with the resource, you have a choice: > > - close the resource yourself (either explicitly with file.close(), > or implicitly with a context manager); > > - or keep the resource open indefinitely, until such eventual time > that the object is garbage collected and the resource closed. Hence my PEP! It enables RAII. The interpreter merely has to call __del__ as soon as the object is no longer referenced (as does CPYthon). From HooDunnit at didly42KahZidly.net Fri Mar 2 10:27:57 2018 From: HooDunnit at didly42KahZidly.net (Cousin Stanley) Date: Fri, 02 Mar 2018 08:27:57 -0700 Subject: Problem: Need galileo running on debian wheezy References: <201802282130.07622.gheskett@shentel.net> <201803012003.28272.gheskett@shentel.net> Message-ID: Gene Heskett wrote: > .... > And the rock64 doesn't have wifi hardware > that I know of. > .... I did manage to get wifi working on my rock64 using a usb wifi dongle ordered from their store .... It was a bit fiddly to set up, somewhat shaky connection-wise, and slower than I was comfortable with so I went with an ethernet connection which is quick and stable .... > So, is galileo shown for armhf's ? The galileo package information doesn't explicitly mention arm .... However, I've had no problem with any other python package on the rock64 here .... dpkg-print-architecture indicates that it is arm64 and I pull the debian stretch packages and updates from .... o http://mirrors.kernel.org/debian/ stretch main non-free contrib o http://mirrors.kernel.org/debian/ stretch-updates main contrib non-free o http://ppa.launchpad.net/ayufan/rock64-ppa/ubuntu xenial main > The reason for posting here is that galileo > is supposedly written in python, > but don't know if 2 or 3. The stretch package depends information for galileo indicates it is for python3 .... $ apt-cache show galileo .... Depends: python3-requests (>= 2), python3-usb (>= 1.0.0~b2), python3:any (>= 3.4~) .... -- Stanley C. Kitching Human Being Phoenix, Arizona From p.f.moore at gmail.com Fri Mar 2 10:36:58 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 15:36:58 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On 2 March 2018 at 15:09, wrote: > We must be discussing a different RAII. That is the raison d'etre of RAII: RAII directly addresses this problem in an exception-safe way that does not burden the resource user at all. RAII works in C++ (where it was initially invented) because it's used with stack-allocated variables that have clearly-defined and limited scope. In my experience writing C++, nobody uses RAII with heap-allocated variables - those require explicit allocation and deallocation and so are equivalent to having an explicit "close()" method in Python (or using __del__ in CPython as it currently exists). Python doesn't have stack allocation, nor does it have a deterministic order of deletion of objects when their last reference goes out of scope (which can happen simultaneously for many objects): class Tracker: def __init__(self, n): self.n = n def __del__(self): print("Deleting instance", self.n) def f(): a = Tracker(1) b = Tracker(2) f() The language doesn't guarantee that a is removed before b. Are you proposing to make that change to the language as well? Also Python only has function scope, so variables local to a smaller-than-the-function block of code aren't possible. That's something that is used in C++ a lot to limit the lifetime of resources under RAII. How do you propose to address that (without needing explicit del statements)? That's why the with statement exists, to clearly define lifetimes smaller than "the enclosing function". Your proposal doesn't offer any equivalent (other than an extra function). Consider C++: void fn() { for (i = 0; i < 10000; ++i) { char name[100]; sprintf(name, "file%d.txt, i); File f(name); // I don't think std::ofstream doesn't support RAII f << "Some text"; } } Or (real Python): def fn(): for i in range(10000): with open(f"file{i}.txt", "w") as f: f.write("Some text") How would you write this in your RAII style - without leaving 10,000 file descriptors open until the end of the function? def loop_body(i): f = open(f"file{i}.txt", "w") f.write("Some text") def fn(): for i in range(10000): loop_body(i) That's both less efficient (function calls have a cost) and less maintainable than the with-statement version. Paul From ooomzay at gmail.com Fri Mar 2 10:50:55 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Fri, 2 Mar 2018 07:50:55 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> <87tvtyki4l.fsf@nightsong.com> Message-ID: <83d6fd63-b1a4-4f6f-b11c-f2057a5d441f@googlegroups.com> On Friday, March 2, 2018 at 2:43:09 PM UTC, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 1:18 AM, ooomzay wrote: > > On Friday, March 2, 2018 at 8:16:22 AM UTC, Paul Rubin wrote:[snip] > >> controlling stuff like file handles > >> with scopes (like with "with") is fine. > > > > How does with work for non-trivial/composite objects that represent/reference multiple resources or even a hierarchy of such objects where all the resources held must be released in a timely fashion when finished with? > > > > Can you give me an example that works with RAII but doesn't work in a > 'with' statement? My claim is about the relative elegance/pythonic nature of RAII w.r.t. 'with'. Probably with enough coding and no ommissions or mistakes 'with' could do achieve the same result that RAII does automatically without any burden on the user. Consider the hierarchy/tree of resource-holding objects postulated above... Presumably __enter__, __exit__ (or functional equivalent) would have to be implemented at every layer breaking the invariance. Thus complicating and making less robust every object it touches. From gheskett at shentel.net Fri Mar 2 11:14:57 2018 From: gheskett at shentel.net (Gene Heskett) Date: Fri, 2 Mar 2018 11:14:57 -0500 Subject: Problem: Need galileo running on debian wheezy In-Reply-To: References: <201802282130.07622.gheskett@shentel.net> Message-ID: <201803021114.57997.gheskett@shentel.net> On Friday 02 March 2018 10:27:57 Cousin Stanley wrote: > Gene Heskett wrote: > > .... > > And the rock64 doesn't have wifi hardware > > that I know of. > > .... > > I did manage to get wifi working on my rock64 > using a usb wifi dongle ordered from their store .... fitbit says its BTLE, claims a 20 foot range. > It was a bit fiddly to set up, somewhat shaky > connection-wise, and slower than I was comfortable with > so I went with an ethernet connection which is quick > and stable .... > > > So, is galileo shown for armhf's ? > > The galileo package information > doesn't explicitly mention arm .... > > However, I've had no problem > with any other python package > on the rock64 here .... > > dpkg-print-architecture indicates that it is arm64 > and I pull the debian stretch packages and updates from .... > > o http://mirrors.kernel.org/debian/ stretch main non-free contrib > > o http://mirrors.kernel.org/debian/ stretch-updates main contrib > non-free > > o http://ppa.launchpad.net/ayufan/rock64-ppa/ubuntu xenial main > So I have several choices, not all of which are arm64 based. I also have an old hp lappy I could put stretch on. And I found a BT dongle but no clue yet if compatible with the BTLE of the fitbit. More checking to see if it can pair. But I just now installed the rest of the BT toy's, but not checked for function yet. Might have to buy another dongle. And/or find a place to put things in this midden heap. Too many years in the same house, but its paid off too. Thanks. > > The reason for posting here is that galileo > > is supposedly written in python, > > but don't know if 2 or 3. > > The stretch package depends information for galileo > indicates it is for python3 .... > > $ apt-cache show galileo > .... > Depends: python3-requests (>= 2), python3-usb (>= 1.0.0~b2), > python3:any (>= 3.4~) .... > > > -- > Stanley C. Kitching > Human Being > Phoenix, Arizona -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From jsf80238 at gmail.com Fri Mar 2 12:35:07 2018 From: jsf80238 at gmail.com (Jason Friedman) Date: Fri, 2 Mar 2018 10:35:07 -0700 Subject: APPLICATION NOT RUNNING. In-Reply-To: <1597615689.9157921.1519995890597@mail.yahoo.com> References: <1597615689.9157921.1519995890597.ref@mail.yahoo.com> <1597615689.9157921.1519995890597@mail.yahoo.com> Message-ID: > > I try to run an application with the latest version of python that is > python 3.6.4 (32-bit) ., instead of running the application it only shows > feel free to mail python-list at python.org if you continue to encounter > issues,Please help me out thanks. > Hello, you might have more success if you restate your question. Read http://www.catb.org/esr/faqs/smart-questions.html#intro for information about how to ask in a way that makes it more likely someone will help. TL;DR: Tell us the operating system. Copy-and-paste into your question what you typed. Copy-and-paste into your question what the computer typed in return. Don't attach images. From bsferrazza at avnera.com Fri Mar 2 12:41:22 2018 From: bsferrazza at avnera.com (bsferrazza at avnera.com) Date: Fri, 2 Mar 2018 09:41:22 -0800 (PST) Subject: Python 3.6 fails to install to non-standard directory under Linux Message-ID: <14bd73a2-3cb0-4e49-ae4b-88a89448ebea@googlegroups.com> I posted this to Stackoverflow and the original post can be seen here. I'll try and copy and past the contents below. Thank you for your help! https://stackoverflow.com/questions/49074327/python-3-6-fails-to-install-to-non-standard-directory-under-linux I have a completely insulated boostrapped toolchain+binaries setup (located under /home/tools) that has been able to build everything that I throw at it. That includes Python 2.7.14, which built without issue with a simple --prefix. The lone exception is Python 3.6.4. I had to edit the configure and setup.py script to patch some of the hardcoded paths that were set to standard /usr directories (the link to ncursesw was notable). It completely compiled and built correctly, except failed during the 'make install' installation of pip. The installation scripts create a pip-build-* directory under /tmp and the source of the failure seems to be centered around that. It appears when the directory is first created, it's made with the proper permission mask. But after failure, I can see it's only user writable (not even readable or executable). *************** d-w------- 4 myuser eng 4096 Mar 1 00:56 pip-build-xh7onsny/ *************** Here's the section of the strace where it first creates the pip-build-* directory. *************** [pid 19771] lstat("/home/myuser/.cache/pip", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 [pid 19771] geteuid() = 10345 [pid 19771] access("/home/myuser/.cache/pip", W_OK) = 0 [pid 19771] mkdir("/tmp/pip-build-xh7onsny", 0700) = 0 [pid 19771] lstat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=28672, ...}) = 0 [pid 19771] lstat("/tmp/pip-build-xh7onsny", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 [pid 19771] getcwd("/nfs/home/myuser/lfs/sources/Python-3.6.4"..., 1024) = 46 ************************ Here's the end of the strace that shows where it Error'ed out. ************************ [pid 19771] munmap(0x2ac5506c0000, 4096) = 0 [pid 19771] stat("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", {st_mode=S_IFREG|0644, st_size=40227, ...}) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=40227, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a58460) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Utility functions for copying"..., 32768) = 32768 [pid 19771] lseek(4, 0, SEEK_CUR) = 32768 [pid 19771] read(4, "e, extract_dir=None, format=None"..., 8192) = 7459 [pid 19771] read(4, "", 8192) = 0 [pid 19771] close(4) = 0 [pid 19771] ioctl(2, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a59f40) = -1 EINVAL (Invalid argument) [pid 19771] write(2, "Exception:\nTraceback (most recen"..., 1739Exception: Traceback (most recent call last): File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 385, in run requirement_set.cleanup_files() File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/utils/build.py", line 38, in __exit__ self.cleanup() File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/utils/build.py", line 42, in cleanup rmtree(self.name) File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 212, in call raise attempt.get() File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/six.py", line 686, in reraise raise value File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/tmp/tmpojuc47gy/pip-9.0.1-py2.py3-none-any.whl/pip/utils/__init__.py", line 102, in rmtree onerror=rmtree_errorhandler) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 476, in rmtree onerror(os.lstat, path, sys.exc_info()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 474, in rmtree fd = os.open(path, os.O_RDONLY) PermissionError: [Errno 13] Permission denied: '/tmp/pip-build-xh7onsny' ) = 1739 [pid 19771] lstat("/tmp/tmpojuc47gy", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 [pid 19771] open("/tmp/tmpojuc47gy", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=11959, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"runpy.py - locating and runni"..., 8192) = 8192 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=11959, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"runpy.py - locating and runni"..., 8192) = 8192 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__main__.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=88, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "import ensurepip\nimport sys\n\nif "..., 8192) = 88 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=6391, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "import os\nimport os.path\nimport "..., 8192) = 6391 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=6391, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "import os\nimport os.path\nimport "..., 8192) = 6391 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=26635, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Temporary files.\n\nThis module"..., 8192) = 8192 [pid 19771] read(4, " continue # try again\n "..., 8192) = 8192 [pid 19771] read(4, "\n \"\"\"\n Close the t"..., 8192) = 8192 [pid 19771] read(4, "emporaryFileArgs['mode']:\n "..., 8192) = 2059 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=26635, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Temporary files.\n\nThis module"..., 8192) = 8192 [pid 19771] read(4, " continue # try again\n "..., 8192) = 8192 [pid 19771] read(4, "\n \"\"\"\n Close the t"..., 8192) = 8192 [pid 19771] read(4, "emporaryFileArgs['mode']:\n "..., 8192) = 2059 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=40227, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Utility functions for copying"..., 8192) = 8192 [pid 19771] read(4, " that are used to exclude fil"..., 8192) = 8192 [pid 19771] read(4, " = os.open(path, os.O_RDONLY)\n "..., 8192) = 8192 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=40227, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Utility functions for copying"..., 8192) = 8192 [pid 19771] read(4, " that are used to exclude fil"..., 8192) = 8192 [pid 19771] close(4) = 0 [pid 19771] open("/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", O_RDONLY|0x80000 /* O_??? */) = 4 [pid 19771] ioctl(4, FIOCLEX) = 0 [pid 19771] fstat(4, {st_mode=S_IFREG|0644, st_size=40227, ...}) = 0 [pid 19771] ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff38a5b0d0) = -1 ENOTTY (Inappropriate ioctl for device) [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] fcntl(4, 0x406 /* F_??? */, 0) = -1 EINVAL (Invalid argument) [pid 19771] lseek(4, 0, SEEK_SET) = 0 [pid 19771] lseek(4, 0, SEEK_CUR) = 0 [pid 19771] read(4, "\"\"\"Utility functions for copying"..., 8192) = 8192 [pid 19771] read(4, " that are used to exclude fil"..., 8192) = 8192 [pid 19771] close(4) = 0 [pid 19771] write(2, "Traceback (most recent call last"..., 1425Traceback (most recent call last): File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__main__.py", line 5, in sys.exit(ensurepip._main()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", line 204, in _main default_pip=args.default_pip, File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", line 117, in _bootstrap return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", line 807, in __exit__ self.cleanup() File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", line 811, in cleanup _shutil.rmtree(self.name) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 402, in _rmtree_safe_fd onerror(os.listdir, path, sys.exc_info()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 399, in _rmtree_safe_fd names = os.listdir(topfd) OSError: [Errno 22] Invalid argument: '/tmp/tmpojuc47gy' ) = 1425 [pid 19771] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x2ac548950ab0}, {0x2ac5485f8ca0, [], SA_RESTORER, 0x2ac548950ab0}, 8) = 0 [pid 19771] munmap(0x2ac54c1df000, 1257472) = 0 [pid 19771] munmap(0x2ac54f203000, 262144) = 0 [pid 19771] sigaltstack(NULL, {ss_sp=0xc877ce0, ss_flags=0, ss_size=8192}) = 0 [pid 19771] sigaltstack({ss_sp=0, ss_flags=SS_DISABLE, ss_size=0}, NULL) = 0 [pid 19771] close(3) = 0 [pid 19771] futex(0x2ac54b6f78ac, FUTEX_WAKE_PRIVATE, 2147483647) = 0 [pid 19771] exit_group(1) = ? Process 19770 resumed Process 19771 detached [pid 19770] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 19771 [pid 19770] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 19770] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 19770] wait4(-1, 0x7fff47da9a04, WNOHANG, NULL) = -1 ECHILD (No child processes) [pid 19770] rt_sigreturn(0xffffffffffffffff) = 0 [pid 19770] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x3c31a30030}, {0x436f50, [], SA_RESTORER, 0x3c31a30030}, 8) = 0 [pid 19770] exit_group(1) = ? Process 16346 resumed Process 19770 detached [pid 16346] <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 19770 [pid 16346] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 16346] rt_sigreturn(0xffffffff) = 19770 [pid 16346] write(2, "make: *** [Makefile:1099: instal"..., 43make: *** [Makefile:1099: install] Error 1 ) = 43 *************** Here's the stdout/stderr output from the 'make install' process. It was from another run, so the directories are slightly different. *************** Installing collected packages: setuptools, pip Successfully installed pip-9.0.1 setuptools-28.8.0 Exception: Traceback (most recent call last): File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/commands/install.py", line 385, in run requirement_set.cleanup_files() File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/utils/build.py", line 38, in __exit__ self.cleanup() File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/utils/build.py", line 42, in cleanup rmtree(self.name) File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 49, in wrapped_f return Retrying(*dargs, **dkw).call(f, *args, **kw) File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 212, in call raise attempt.get() File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 247, in get six.reraise(self.value[0], self.value[1], self.value[2]) File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/six.py", line 686, in reraise raise value File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/_vendor/retrying.py", line 200, in call attempt = Attempt(fn(*args, **kwargs), attempt_number, False) File "/tmp/tmpkmstr4wc/pip-9.0.1-py2.py3-none-any.whl/pip/utils/__init__.py", line 102, in rmtree onerror=rmtree_errorhandler) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 476, in rmtree onerror(os.lstat, path, sys.exc_info()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 474, in rmtree fd = os.open(path, os.O_RDONLY) PermissionError: [Errno 13] Permission denied: '/tmp/pip-build-in_asosn' Traceback (most recent call last): File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__main__.py", line 5, in sys.exit(ensurepip._main()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", line 204, in _main default_pip=args.default_pip, File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/ensurepip/__init__.py", line 117, in _bootstrap return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", line 807, in __exit__ self.cleanup() File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/tempfile.py", line 811, in cleanup _shutil.rmtree(self.name) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 402, in _rmtree_safe_fd onerror(os.listdir, path, sys.exc_info()) File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 399, in _rmtree_safe_fd names = os.listdir(topfd) OSError: [Errno 22] Invalid argument: '/tmp/tmpkmstr4wc' make: *** [Makefile:1109: altinstall] Error 1 It's worth noting that it actually does install the pip binaries to my $prefix/bin directory, but installing subsequent packages, like meson, fail in a similar fashion, so I'm presuming the Python 3.6 / pip installation didn't fully succeed. Seems to also comes from shutil.py. Here's the strace for meson installation. write(2, "Traceback (most recent call last"..., 1805Traceback (most recent call last): File "/home/tools/lib/python3.6/shutil.py", line 399, in _rmtree_safe_fd names = os.listdir(topfd) OSError: [Errno 22] Invalid argument: '/tmp/easy_install-0o0par1r' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "setup.py", line 98, in complete DSL.''') File "/home/tools/lib/python3.6/distutils/core.py", line 148, in setup dist.run_commands() File "/home/tools/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/home/tools/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/home/tools/lib/python3.6/site-packages/setuptools/command/install.py", line 67, in run self.do_egg_install() File "/home/tools/lib/python3.6/site-packages/setuptools/command/install.py", line 117, in do_egg_install cmd.run() File "/home/tools/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 410, in run self.easy_install(spec, not self.no_deps) File "/home/tools/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 669, in easy_instal l rmtree(tmpdir) File "/home/tools/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 2227, in rmtree return shutil.rmtree(path, ignore_errors, onerror) File "/home/tools/lib/python3.6/shutil.py", line 480, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/home/tools/lib/python3.6/shutil.py", line 402, in _rmtree_safe_fd onerror(os.listdir, path, sys.exc_info()) File "/home/tools/lib/python3.6/site-packages/setuptools/command/easy_install.py", line 1677, in auto_chmod six.reraise(et, (ev[0], ev[1] + (" %s %s" % (func, arg)))) TypeError: 'OSError' object is not subscriptable ) = 1805 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x2b2b4caa4ab0}, {0x2b2b4c74cca0, [], SA_RESTORER, 0x2b2b4caa 4ab0}, 8) = 0 ************************ I've tried changing the /tmp directory one under my own home directory, and the same Error occurs and also has the odd write only permissions on the pip-build-* directory. I'd appreciate any help in debugging this issue and getting Python 3.6.4 to properly install. Thank you! Background: I have setup a separate, bootstrapped toolchain and binaries that are completely insulated from the host libraries on a Linux server at work, following much of the Linux From Scratch book. It's running an old Red Hat / CentOS 5 distribution with kernel 2.6.18, that can't be updated due to support for older CAD software. But I now have a fully working environment that has the latest Glibc supported by the kernel (2.19), GCC (7.3.0), binutils (2.30), etc. I've gotten everything I've thrown at it to build correctly, even X libraries and gtk applications, all directly referencing my set of libraries through an rpath. From bsferrazza at avnera.com Fri Mar 2 14:48:04 2018 From: bsferrazza at avnera.com (bsferrazza at avnera.com) Date: Fri, 2 Mar 2018 11:48:04 -0800 (PST) Subject: Python 3.6 fails to install to non-standard directory under Linux In-Reply-To: <14bd73a2-3cb0-4e49-ae4b-88a89448ebea@googlegroups.com> References: <14bd73a2-3cb0-4e49-ae4b-88a89448ebea@googlegroups.com> Message-ID: Here's my configure I need to set ac_cv_fun_utimensat=no and ac_cv_func_futimens=no because presumably the file-system or kernel on my system doesn't support nanosecond timestamps. With these options, and patching the configure/setup.py files to remove references to /usr/lib/ncursesw and replace with my /home/tools/lib path, everything builds just fine. It's during the 'make install' phase of pip that it fails. ./configure --prefix=/home/tools \ --enable-shared \ --with-system-expat \ --with-system-ffi \ --with-ensurepip=install \ ac_cv_func_utimensat=no \ ac_cv_func_futimens=no From bsferrazza at avnera.com Fri Mar 2 15:45:34 2018 From: bsferrazza at avnera.com (bsferrazza at avnera.com) Date: Fri, 2 Mar 2018 12:45:34 -0800 (PST) Subject: Python 3.6 fails to install to non-standard directory under Linux (Posting On Python-List Prohibited) In-Reply-To: References: <14bd73a2-3cb0-4e49-ae4b-88a89448ebea@googlegroups.com> Message-ID: On Friday, March 2, 2018 at 12:27:17 PM UTC-8, Lawrence D?Oliveiro wrote: > On Saturday, March 3, 2018 at 6:42:05 AM UTC+13, bsfer... at avnera.com wrote: > > > ... > > File "/nfs/home/myuser/lfs/sources/Python-3.6.4/Lib/shutil.py", line 476, in rmtree > > ... > > NFS trouble? > > I have had builds give trouble when done on directories mounted via NFS, yet work perfectly on local directories. Hmm. That could very well be the case. I know the /home directory is NFS mounted, though not sure about /tmp. I'm starting to get the impression that everything is actually OK, and that these errors are only on the cleanup after installation. Even with the meson install. This bugs.python.org/issue23346 bug got me in the right path. When I comment out the _use_fd_functions line that ends up setting it to True and replace it with a direct assignment to False, everything works ok. The rmtree function instead returns _rmtree_unsafe(path, onerror) when _use_fd_functions is false and that eliminates the error I was seeing. From python at bladeshadow.org Fri Mar 2 15:45:55 2018 From: python at bladeshadow.org (Python) Date: Fri, 2 Mar 2018 14:45:55 -0600 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> Message-ID: <20180302204555.GL10273@bladeshadow.org> On Mon, Feb 26, 2018 at 09:57:06AM +0000, Steven D'Aprano wrote: > Besides, if you want Python with no GIL so you can run threaded code, why > aren't you using IronPython or Jython? But this is just another oversimplified argument. In the real world there rather often exist constraints which have nothing to do with what is technically possible, but which make impractical or infeasible what may be technically possible. Python is often a preferred solution because it is often fantastic for rapid implementation and maintainability. The GIL's interference with threaded code performance has, for me at least, on several occasions been... disappointing (perf costs of removing it aside) because it gets in the way of choosing Python for such solutions. Jython and IronPython are simply not feasible options for me, for multiple reasons that have zero to do with their technical suitability. But back to the point of the thread, I think it's ironic that you're basically making the benchmarks' argument: Choose the tool which suits the solution you need. You argue we should use IronPython or Jython when threads are needed, and the Julia benchmarks *imply* (but never actually argue, AFAICT) that if you need a solution which relies on heavy use of function calls, since Python will be slower than Julia, you should therefore choose Julia instead. But that's only a sensible conclusion if all other things be equal, which of course they are not, neither for Julia vs. Python, nor for Python vs. Jython or IronPython (though likely the latter case is slightly closer to being true). From rosuav at gmail.com Fri Mar 2 16:18:03 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 08:18:03 +1100 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: <20180302204555.GL10273@bladeshadow.org> References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> Message-ID: On Sat, Mar 3, 2018 at 7:45 AM, Python wrote: > On Mon, Feb 26, 2018 at 09:57:06AM +0000, Steven D'Aprano wrote: >> Besides, if you want Python with no GIL so you can run threaded code, why >> aren't you using IronPython or Jython? > > But this is just another oversimplified argument. In the real world > there rather often exist constraints which have nothing to do with > what is technically possible, but which make impractical or infeasible > what may be technically possible. > > Python is often a preferred solution because it is often fantastic for > rapid implementation and maintainability. The GIL's interference > with threaded code performance has, for me at least, on several > occasions been... disappointing (perf costs of removing it aside) > because it gets in the way of choosing Python for such solutions. > Jython and IronPython are simply not feasible options for me, for > multiple reasons that have zero to do with their technical > suitability. Have you actually tried it and run into problems, or do you succumb to the FUD and give up before implementing? I can think of only one time when I tried to speed up a program by multithreading it and was unable to due to locking. (And actually, it wasn't even the GIL that was the problem - it was a non-thread-safe library I was calling on, and *that* meant everything serialized.) "One time, people!" -- Wasabi ChrisA From steve+comp.lang.python at pearwood.info Fri Mar 2 17:41:13 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 2 Mar 2018 22:41:13 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On Fri, 02 Mar 2018 07:09:19 -0800, ooomzay wrote: [...] >> If you're going to *require* the programmer to explicitly del the >> reference: >> >> f = open("file") >> text = f.read() >> del f > > But I am not! On the contrary RAII frees the programmer from even having > to remember to close the file. The poster asked what would happen if the > resource was deliberately kept open by storing a reference at global > scope. You say that as if it were difficult to do, requiring the programmer to take extraordinary steps of heroic proportion. It doesn't. It is unbelievably easy to store the reference at global scope, which means that the programmer needs to remember to close the file explicitly and RAII doesn't help. > In practice CPython destroys it cleanly on exit - but I am not sure the > language guarantees this Typically the OS guarantees that any open files will be closed when the application exits. But that *absolutely does not* apply to other resources that may need closing, and Python at least doesn't guarantee to run __del__ during application shutdown as it may not be able to. (It is better now than it used to be, but there are still scenarios where the resources needed to run __del__ are gone before __del__ is called.) > - in any case RAII won't make things any worse > in this respect. (Logfiles are a common example of such global > resource.) I didn't say RAII will make it worse, but neither will it make it better, nor make "with" statements obsolete. Your justification for requiring RAII is to ensure the timely closure of resources -- but to do that, you have to explicitly close or delete the resource. It simply isn't true that "RAII frees the programmer from even having to remember to close the file". >> then you might as well require them to explicitly close the file: >> >> f = open("file") >> text = f.read() >> f.close() >> >> which we know from many years experience is not satisfactory except for >> the simplest scripts that don't need to care about resource management. >> That's the fatal flaw in RAII: > > We must be discussing a different RAII. That is the raison d'etre of > RAII: RAII directly addresses this problem in an exception-safe way that > does not burden the resource user at all. But as you said yourself, if the resource is held open in a global reference, it will stay open indefinitely. And remember, global in this context doesn't just mean the main module of your application, but *every* module you import. I think you have just put your finger on the difference between what RAII *claims* to do and what it *actually* can do. -- Steve From jladasky at itu.edu Fri Mar 2 18:15:33 2018 From: jladasky at itu.edu (jladasky at itu.edu) Date: Fri, 2 Mar 2018 15:15:33 -0800 (PST) Subject: Redundant features in python library, PyQt Message-ID: Python's standard library has (to take three examples) threads, processes, and datetime functions. Meanwhile, PyQt has QThread, QProcess, and QDateTime. Does this redundancy exist for C++ programmers who are programming Qt directly, and who may lack a standard C++ library with these features? Are the Python wrappers simply provided for completeness, or do these libraries truly function differently? When writing a PyQt application, how does one choose between the standard library and the PyQt library? From steve+comp.lang.python at pearwood.info Fri Mar 2 18:44:04 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 2 Mar 2018 23:44:04 +0000 (UTC) Subject: How to make Python run as fast (or faster) than Julia References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> Message-ID: On Fri, 02 Mar 2018 14:45:55 -0600, Python wrote: > On Mon, Feb 26, 2018 at 09:57:06AM +0000, Steven D'Aprano wrote: >> Besides, if you want Python with no GIL so you can run threaded code, >> why aren't you using IronPython or Jython? > > But this is just another oversimplified argument. In the real world > there rather often exist constraints which have nothing to do with what > is technically possible, but which make impractical or infeasible what > may be technically possible. Of course it is simplified. As is the argument that the GIL is the source of all evil, er, why Python threading is slow. In reality, we may have good reasons for not using IronPython or Jython, such as the desire for the latest 3.x version, or because we rely on C external libraries that don't work under the CLR or JVM. [...] > But back to the point of the thread, I think it's ironic that you're > basically making the benchmarks' argument: Choose the tool which suits > the solution you need. You argue we should use IronPython or Jython > when threads are needed, and the Julia benchmarks *imply* (but never > actually argue, AFAICT) that if you need a solution which relies on > heavy use of function calls, since Python will be slower than Julia, you > should therefore choose Julia instead. Indeed. While I haven't said so explicitly in this thread, I agree entirely with everything you say here. One of the things I'm very interested in is the possibility of writing number-crunching code in Julia and calling it from Python. I haven't tried it yet, but it is on my bucket-list :-) > But that's only a sensible > conclusion if all other things be equal, which of course they are not, > neither for Julia vs. Python, nor for Python vs. Jython or IronPython > (though likely the latter case is slightly closer to being true). There are always constraints on what language you might use. Foo-lang might be the best language for the job, but since I don't know the language (or even heard of it), I can't use it. You may be constrained by familiarity, shop politics, customer requirements, the availability of staff, personal preferences, legal requirements, etc. But taking all of those things into consideration, the conclusion still fits: use the language that suits you and the job best. If that language is Python, then this is how you can get the best performance if and when needed. -- Steve From torriem at gmail.com Fri Mar 2 19:58:29 2018 From: torriem at gmail.com (Michael Torrie) Date: Fri, 2 Mar 2018 17:58:29 -0700 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On 03/02/2018 08:36 AM, Paul Moore wrote: > On 2 March 2018 at 15:09, wrote: >> We must be discussing a different RAII. That is the raison d'etre of RAII: RAII directly addresses this problem in an exception-safe way that does not burden the resource user at all. > > RAII works in C++ (where it was initially invented) because it's used > with stack-allocated variables that have clearly-defined and limited > scope. In my experience writing C++, nobody uses RAII with > heap-allocated variables - those require explicit allocation and > deallocation and so are equivalent to having an explicit "close()" > method in Python (or using __del__ in CPython as it currently exists). Very good point. I'm not sure the OP considered this when proposing this idea. That said, In C++, there are a number of ways of making heap-allocated objects more RAII-like. These include smart pointers that do reference counting and reference borrowing. They work pretty well when used as designed, but they do require some programmer understanding and intervention to use correctly and leaks can and do happen. Smart pointers are pretty good hacks as far as they go, without a garbage collector. From torriem at gmail.com Fri Mar 2 20:11:10 2018 From: torriem at gmail.com (Michael Torrie) Date: Fri, 2 Mar 2018 18:11:10 -0700 Subject: Redundant features in python library, PyQt In-Reply-To: References: Message-ID: On 03/02/2018 04:15 PM, jladasky at itu.edu wrote: > Python's standard library has (to take three examples) threads, > processes, and datetime functions. Meanwhile, PyQt has QThread, > QProcess, and QDateTime. > > Does this redundancy exist for C++ programmers who are programming Qt > directly, and who may lack a standard C++ library with these > features? Are the Python wrappers simply provided for completeness, > or do these libraries truly function differently? When writing a > PyQt application, how does one choose between the standard library > and the PyQt library Yes, when using Qt in C++, you'll be using QThread, QProcess, QDateTime, etc, even though there are equivalents in the C++ standard library. This is largely because Qt was created long before the C++ standard library had many of these things. But it's also because Qt's versions are tightly integrated with the Qt object and event machinery. A QThread sets up the necessary message queues, for example, whereas a STL thread will not. When programming in PyQt, you will also want to use the Qt versions of primitives when they exist. The primary reason is that the Qt versions integrate with the Qt event-handling machinery. As Qt is a C++ library, it's necessary to wrap each and every Qt class with a Python class. Thus PyQt often really is just C++ with a Python dialect. In other words often times you'll be using C++ idioms and C++ data structures even though Python's idioms and versions would be cleaner and more flexible. This is the price you pay for using a C++ library that's wrapped using automated tools for use in Python. So you'll find api calls that return a QString, for example, when a python string would be preferred. Or a QArray when a list would be better. This is not just a PyQt thing. Any time you have thin wrappers around C or C++ libraries, the underlying language idioms are going to bleed through a bit into Python. From rosuav at gmail.com Fri Mar 2 22:28:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 14:28:18 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On Sat, Mar 3, 2018 at 11:58 AM, Michael Torrie wrote: > On 03/02/2018 08:36 AM, Paul Moore wrote: >> On 2 March 2018 at 15:09, wrote: >>> We must be discussing a different RAII. That is the raison d'etre of RAII: RAII directly addresses this problem in an exception-safe way that does not burden the resource user at all. >> >> RAII works in C++ (where it was initially invented) because it's used >> with stack-allocated variables that have clearly-defined and limited >> scope. In my experience writing C++, nobody uses RAII with >> heap-allocated variables - those require explicit allocation and >> deallocation and so are equivalent to having an explicit "close()" >> method in Python (or using __del__ in CPython as it currently exists). > > Very good point. I'm not sure the OP considered this when proposing > this idea. > > That said, In C++, there are a number of ways of making heap-allocated > objects more RAII-like. These include smart pointers that do reference > counting and reference borrowing. They work pretty well when used as > designed, but they do require some programmer understanding and > intervention to use correctly and leaks can and do happen. Smart > pointers are pretty good hacks as far as they go, without a garbage > collector. Ref-counting smart pointers? They might save you the trouble of calling (the equivalents of) Py_INCREF and Py_DECREF manually, but they still don't solve reference loops, and they don't deal with the termination cleanup problem. Smart pointers don't change heap allocation semantics and don't remove the garbage collector; all they do is change the way you do refcounting. (Which is not a bad thing, btw. I've used smart pointers, and they can be very helpful. But they're no panacea.) ChrisA From youta.t at gmail.com Fri Mar 2 23:28:24 2018 From: youta.t at gmail.com (=?UTF-8?B?6auY5bKh6Zm95aSq?=) Date: Sat, 03 Mar 2018 04:28:24 +0000 Subject: "except" and "subclasscheck" changed between CPython2 and 3 Message-ID: Hello, I found a difference of behavior about `except` statement between CPython 2.7 and 3.x . `except EXC_CLASS:` calls `__subclasscheck__` in 2.7, but does not in 3.x . Let me show you an example. Now, define a class "ExceptionLike" (with metaclass "ExceptionLikeMeta") below. class ExceptionLikeMeta(type): def __subclasscheck__(cls, other): return other.__name__.endswith('Exception') ExceptionLike = ExceptionLikeMeta('ExceptionLike', (Exception,), {}) try: raise Exception() except ExceptionLike: print('catch it!') except: assert False, 'not caught!' The exception is caught with `ExceptionLike` in 2.7, but not in 3.x . (I tested it with CPython 3.6.2 and 2.7.13 on mac) The difference seems to be introduced with this commit [ https://github.com/python/cpython/commit/ec569b794737be248671d0dfac11b664fc930eef#diff-73da65b698644ee286bc60b703916bbbL163] and this ticket [https://bugs.python.org/issue2534]. The behavior of CPython 3 may have some advantages. - `except EXC_TYPE:` does never raise an exception caused by `__subclasscheck__`. - (may be) fast, because it just scan mro. So, the diff can be intended one. But the ticket itself seems not. Should not I expect that `__subclasscheck__` is used for subclass checking anywhere? It effects only `issubclass` anymore? (or, should I go to other list?) --- youta.t at gmail.com From greg.ewing at canterbury.ac.nz Fri Mar 2 23:57:37 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sat, 03 Mar 2018 17:57:37 +1300 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <87efl33vaj.fsf@nightsong.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> Message-ID: Paul Rubin wrote: > So you want the programmer to put more head scratching into figuring out > which reference should be strong and which should be weak? Also, sometimes weak references don't really solve the problem, e.g. if you have a graph where you can't identify any particular node as a "root" node, and you want the graph to stay around as long as any of its nodes are referenced. -- Greg From steve+comp.lang.python at pearwood.info Sat Mar 3 00:42:02 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 3 Mar 2018 05:42:02 +0000 (UTC) Subject: I'm not seeing Paul Rubin's posts Message-ID: I see people replying to Paul Rubin, but I'm not seeing his posts. I've reading this through gmane. Does anyone know what's going on? Thanks, Steve From rosuav at gmail.com Sat Mar 3 00:52:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 16:52:18 +1100 Subject: I'm not seeing Paul Rubin's posts In-Reply-To: References: Message-ID: On Sat, Mar 3, 2018 at 4:42 PM, Steven D'Aprano wrote: > I see people replying to Paul Rubin, but I'm not seeing his posts. > > I've reading this through gmane. > > Does anyone know what's going on? I'm not seeing them either. Usually that means someone's been banned from the mailing list, or else has self-banned by saying "don't archive". Which, to me, seems like going out in public and talking to people, and then saying "But that was off the record, so make sure you forget that I said that". ChrisA From patatetom at gmail.com Sat Mar 3 05:37:09 2018 From: patatetom at gmail.com (patatetom at gmail.com) Date: Sat, 3 Mar 2018 02:37:09 -0800 (PST) Subject: read function of fuse Message-ID: hello, my xb360hd module allows me to browse a xtaf partition (xbox360) and extract files. a file can be large (and discontinuous), so the readFile function of the module returns a generator. from a console, I can "easily" copy a file with this piece of code: partition = xb360hd.Xtaf('/dev/sdb') with open ('/tmp/copy',' wb') as copy: for data in partition.readFile(partition.getEntry('/path/to/un/file'): copy.write(data) So far so good. I'm now trying to make the contents of the partition accessible "naturally" with my xtaffuse module, but I'm blocking on the read function of fusepy and don't see how to return the content of the file without having to read it completely and store it temporarily in variable, and therefore in memory. thank for your help. (If the read function of the xtaffuse module returns directly the generator from the readFile function of the xb360hd module, I have the error "object of type' generator' has no len ()" / I don't want to use a temporary file between readFile of the xb360hd module and read of the xtaffuse module) xb360hd: https://github.com/patatetom/xtaf/blob/master/xb360hd.py xtaffuse: https://github.com/patatetom/xtaf/blob/master/xtaffuse.py fusepy: https://github.com/fusepy/fusepy xtaf: http://www.free60.org/wiki/FATX Translated with www.DeepL.com/Translator From patatetom at gmail.com Sat Mar 3 08:49:07 2018 From: patatetom at gmail.com (patatetom at gmail.com) Date: Sat, 3 Mar 2018 05:49:07 -0800 (PST) Subject: read function of fuse In-Reply-To: References: Message-ID: after a few tests, it appears that the file is read in pieces: read[0] 131072 bytes from 0 flags: 0x8000 read[0] 131072 bytes from 131072 flags: 0x8000 read[0] 131072 bytes from 262144 flags: 0x8000 ... read[0] 8192 bytes from 917504 flags: 0x8000 read[0] 4096 bytes from 925696 flags: 0x8000 so the problem I was raising is not a problem in practice. I would appreciate your feedback on my xtaffuse module ;-) From ian.g.kelly at gmail.com Sat Mar 3 09:03:42 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 3 Mar 2018 07:03:42 -0700 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> Message-ID: On Fri, Mar 2, 2018 at 9:57 PM, Gregory Ewing wrote: > Paul Rubin wrote: >> >> So you want the programmer to put more head scratching into figuring out >> which reference should be strong and which should be weak? > > > Also, sometimes weak references don't really solve the > problem, e.g. if you have a graph where you can't identify > any particular node as a "root" node, and you want the graph > to stay around as long as any of its nodes are referenced. I offered two possible solutions for the graph problem elsewhere in this thread. For the record I'm not in favor of the OP's proposal (although making the different Python implementations more consistent in their garbage collection semantics would not be a bad thing per se) but I am somewhat mystified as to why people seem to think that weak refs are difficult to use correctly. From ooomzay at gmail.com Sat Mar 3 11:02:35 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Sat, 3 Mar 2018 08:02:35 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On Friday, March 2, 2018 at 10:43:57 PM UTC, Steven D'Aprano wrote: > On Fri, 02 Mar 2018 07:09:19 -0800, ooomzay wrote: > [...] > >> If you're going to *require* the programmer to explicitly del the > >> reference: > >> > >> f = open("file") > >> text = f.read() > >> del f > > > > But I am not! On the contrary RAII frees the programmer from even having > > to remember to close the file. The poster asked what would happen if the > > resource was deliberately kept open by storing a reference at global > > scope. > > You say that as if it were difficult to do, requiring the programmer to > take extraordinary steps of heroic proportion. It doesn't. > It is unbelievably easy to store the reference at global scope, which > means that the programmer needs to remember to close the file explicitly > and RAII doesn't help. Can you expand on what you mean by "unbelievably easy to store the reference at global scope". I will assume you are claiming that it is easy to _inadvertently_ store a global reference. Can you give an example of how this might happen? (Apart from writing leaky exception handlers, which I trust no one thinks is a good idea) > [...] > Your justification for requiring RAII is to ensure the timely closure of > resources -- but to do that, you have to explicitly close or delete the > resource. It simply isn't true that "RAII frees the programmer from even > having to remember to close the file". In the special case (keyword global) that you reference resources from global scope - then yes, you must remember to del the global references before you exit to avoid relying on the vagaries of exit processing. This problem with destruction of globals on exit is not unique to python. There are patterns to manage this for code you control, such as putting them all in one global "bucket" object and explicitly deleting it before exit. > >> then you might as well require them to explicitly close the file: > >> > >> f = open("file") > >> text = f.read() > >> f.close() > >> > >> which we know from many years experience is not satisfactory except for > >> the simplest scripts that don't need to care about resource management. > >> That's the fatal flaw in RAII: > > > > We must be discussing a different RAII. That is the raison d'etre of > > RAII: RAII directly addresses this problem in an exception-safe way that > > does not burden the resource user at all. > > But as you said yourself, if the resource is held open in a global > reference, it will stay open indefinitely. And remember, global in this > context doesn't just mean the main module of your application, but > *every* module you import. Can you give an example of such a case where an application-managed (acquired/opened & released/closed) resource is referenced strongly by a 3rd party module at global scope? ...in my experience this pattern usually smells. > I think you have just put your finger on the difference between what RAII > *claims* to do and what it *actually* can do. I can assure you that RAII does what it says on the tin and is relied on in many critical systems to release resources robustly ... given the pre-requisite deterministic destruction. From Richard at Damon-Family.org Sat Mar 3 11:19:51 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 11:19:51 -0500 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> Message-ID: <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> On 3/3/18 9:03 AM, Ian Kelly wrote: > On Fri, Mar 2, 2018 at 9:57 PM, Gregory Ewing > wrote: >> Paul Rubin wrote: >>> So you want the programmer to put more head scratching into figuring out >>> which reference should be strong and which should be weak? >> >> Also, sometimes weak references don't really solve the >> problem, e.g. if you have a graph where you can't identify >> any particular node as a "root" node, and you want the graph >> to stay around as long as any of its nodes are referenced. > I offered two possible solutions for the graph problem elsewhere in this thread. > > For the record I'm not in favor of the OP's proposal (although making > the different Python implementations more consistent in their garbage > collection semantics would not be a bad thing per se) but I am > somewhat mystified as to why people seem to think that weak refs are > difficult to use correctly. As I think about this, and coming from a background where I have found the ability to reliably use RAII useful in some cases, I can see some merit to the proposal. One big issue is that since Python doesn't have sub-function level lifetime scopes for locals, it is perhaps not quite as useful in some cases. One idea does come to mind though, would it be reasonable, and somewhat Pythonic, for a class to define member functions like __ref__ and __unref__ (or perhaps some other name) that if defined, would be called every time a name was bound or unbound to an object? (the assignment to a name should __ref__ the new value, then __unref__ the old to avoid possible issues with rebinding an object to the last name it was bound to). This would allow those (limited) objects that want to be destroyed as soon as possible to do so, but most other objects wouldn't have any significant overhead added (just a check for this method). Objects with these methods would still be subject to being cleaned up with garbage collection in the case they were kept alive via a cycle, having the cycle just makes it so that you don't get the immediate distruction. -- Richard Damon From torriem at gmail.com Sat Mar 3 11:33:33 2018 From: torriem at gmail.com (Michael Torrie) Date: Sat, 3 Mar 2018 09:33:33 -0700 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: > I can assure you that RAII does what it says on the tin and is relied on in > many critical systems to release resources robustly ... given the > pre-requisite deterministic destruction. Sure but did you read what Paul Moore wrote? He said RAII works in C++ because objects are allocated on the *stack* with strict lifetimes and scopes. They won't ever have cycles and they are guaranteed to be destroyed no matter what as the stack is unwound. Python has no stack-allocated objects. In C++, Heap-allocated objects must still be managed manually, without the benefit of RAII, for much of the same reasons as people are giving here for why RAII is not a good fit for Python. There are smart pointer objects that try to give RAII semantics to heap-allocated objects, with varying degrees of success. In other words there are some limitations. Python does not have stack-allocated objects, so the same issues that prevent RAII from automatically applying in C++ to heap objects exist here. From maillist at schwertberger.de Sat Mar 3 11:59:12 2018 From: maillist at schwertberger.de (Dietmar Schwertberger) Date: Sat, 3 Mar 2018 17:59:12 +0100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: Message-ID: On 2/28/2018 11:51 PM, ooomzay at gmail.com wrote: > This PEP proposes that valid python interpreters *must* synchronously destroy objects when the last reference to an object goes out of scope. This interpreter behaviour is currently permitted and exhibited by the reference implementation [CPython], but it is optional. CPython does *not* guarantee destruction when the object reference goes out of scope, even if there are no other references. I would very much appreciate such a deterministic behaviour, at least with CPython. I recently had to debug an issue in the matplotlib wx backend (*). Under certain conditions, the wx device context was not destroyed when the reference went out of scope. Adding a del to the end of the method or calling the Destroy method of the context did fix the issue. (There was also a hidden reference, but avoiding this was not sufficient. The del was still required.) (*) https://github.com/matplotlib/matplotlib/issues/10174 Regards, Dietmar From ian.g.kelly at gmail.com Sat Mar 3 12:01:43 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 3 Mar 2018 10:01:43 -0700 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: On Sat, Mar 3, 2018 at 9:19 AM, Richard Damon wrote: > One idea does come to mind though, would it be reasonable, and somewhat > Pythonic, for a class to define member functions like __ref__ and __unref__ > (or perhaps some other name) that if defined, would be called every time a > name was bound or unbound to an object? (the assignment to a name should > __ref__ the new value, then __unref__ the old to avoid possible issues with > rebinding an object to the last name it was bound to). This would allow > those (limited) objects that want to be destroyed as soon as possible to do > so, but most other objects wouldn't have any significant overhead added > (just a check for this method). > > Objects with these methods would still be subject to being cleaned up with > garbage collection in the case they were kept alive via a cycle, having the > cycle just makes it so that you don't get the immediate distruction. This sounds like a nightmare for performance. Ref counting is enough of a drain already when it's done in C, using macros for in-lining. Now every time the count would be incremented or decremented, we'd also need to check the class __dict__ to see if there's a corresponding Python method, build a frame object, and then call it in the interpreter. And note that it would have to be basically any time the CPython implementation currently adjusts the ref count, not just when a name is bound or unbound, as there are lots of references that aren't related to name bindings, for example: * Objects stored in collections * Function argument default values * Function annotations * References temporarily held by C-API code just to make sure that the object they're manipulating doesn't suddenly become invalid during a function call It would be much more efficient to spare all that by doing the ref-counting in the interpreter and just call a method when it hits zero. Which is just what CPython already does. From rosuav at gmail.com Sat Mar 3 12:02:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 04:02:36 +1100 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: On Sun, Mar 4, 2018 at 3:19 AM, Richard Damon wrote: > On 3/3/18 9:03 AM, Ian Kelly wrote: >> >> On Fri, Mar 2, 2018 at 9:57 PM, Gregory Ewing >> wrote: >>> >>> Paul Rubin wrote: >>>> >>>> So you want the programmer to put more head scratching into figuring out >>>> which reference should be strong and which should be weak? >>> >>> >>> Also, sometimes weak references don't really solve the >>> problem, e.g. if you have a graph where you can't identify >>> any particular node as a "root" node, and you want the graph >>> to stay around as long as any of its nodes are referenced. >> >> I offered two possible solutions for the graph problem elsewhere in this >> thread. >> >> For the record I'm not in favor of the OP's proposal (although making >> the different Python implementations more consistent in their garbage >> collection semantics would not be a bad thing per se) but I am >> somewhat mystified as to why people seem to think that weak refs are >> difficult to use correctly. > > > As I think about this, and coming from a background where I have found the > ability to reliably use RAII useful in some cases, I can see some merit to > the proposal. One big issue is that since Python doesn't have sub-function > level lifetime scopes for locals, it is perhaps not quite as useful in some > cases. > > One idea does come to mind though, would it be reasonable, and somewhat > Pythonic, for a class to define member functions like __ref__ and __unref__ > (or perhaps some other name) that if defined, would be called every time a > name was bound or unbound to an object? (the assignment to a name should > __ref__ the new value, then __unref__ the old to avoid possible issues with > rebinding an object to the last name it was bound to). This would allow > those (limited) objects that want to be destroyed as soon as possible to do > so, but most other objects wouldn't have any significant overhead added > (just a check for this method). > > Objects with these methods would still be subject to being cleaned up with > garbage collection in the case they were kept alive via a cycle, having the > cycle just makes it so that you don't get the immediate distruction. I'm not sure what __unref__ gives you that __del__ doesn't, aside from mandating that every Python implementation from here to eternity MUST use reference counting. ChrisA From ned at nedbatchelder.com Sat Mar 3 12:15:49 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sat, 3 Mar 2018 12:15:49 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: <559003cd-23e4-7b13-e284-6b82901c273c@nedbatchelder.com> On 3/2/18 10:36 AM, Paul Moore wrote: > Or (real Python): > > def fn(): > for i in range(10000): > with open(f"file{i}.txt", "w") as f: > f.write("Some text") > > How would you write this in your RAII style - without leaving 10,000 > file descriptors open until the end of the function? IIUC, if the OP's proposal were accepted, the __del__ method would be called as soon as the *value*'s reference count went to zero. That means this wouldn't leave 10,000 files open, since each open() would assign a new file object to f, which would make the previous file object's ref count be zero, and it would be closed. --Ned. From ned at nedbatchelder.com Sat Mar 3 12:26:09 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sat, 3 Mar 2018 12:26:09 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <6ad254cb-7318-4e24-8515-cedf9ab9feb6@googlegroups.com> Message-ID: <8cf5f221-74f2-73a1-9de3-200c5ebc033f@nedbatchelder.com> On 2/28/18 6:53 PM, ooomzay at gmail.com wrote: > On Wednesday, February 28, 2018 at 11:45:24 PM UTC, ooo... at gmail.com wrote: >> On Wednesday, February 28, 2018 at 11:02:17 PM UTC, Chris Angelico wrote: >>> On Thu, Mar 1, 2018 at 9:51 AM, ooomzay wrote: >>> [snip] >>> Taking a really simple situation: >>> >>> class Foo: >>> def __init__(self): >>> self.self = self >>> print("Creating a Foo") >>> def __del__(self): >>> print("Disposing of a Foo") >>> >>> foo = Foo() >>> foo = 1 >>> >>> When do you expect __del__ to be called? >> At the point of (re)assignment: "foo = 1" > Oh... I now see there is a (non-weak) self reference in there. > > So in this case it would be orphaned. It is a design error and should be recoded. I don't care how it is detected for current purposes. There are many ways to create a cycle. You can't simply dismiss them all as design errors and put the burden on the developer to rewrite their code. For example, objects have a reference to their classes.? So any time you write a class that references its objects, you have a cycle.? Or perhaps you have a tree structure where nodes have a list of their children, but also children have a reference to their parent?? I'm not even sure where in the inner workings of Python there are cycles. There's a reason Python doesn't guarantee synchronous execution of __del__.? If you want your proposal to be taken seriously, you need to understand that reason, and propose a realistic alternative. --Ned. From rosuav at gmail.com Sat Mar 3 12:35:52 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 04:35:52 +1100 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: On Sun, Mar 4, 2018 at 3:19 AM, Richard Damon wrote: > One idea does come to mind though, would it be reasonable, and somewhat > Pythonic, for a class to define member functions like __ref__ and __unref__ > (or perhaps some other name) that if defined, would be called every time a > name was bound or unbound to an object? Just a thought, here: would __ref__ be called every time a new function-local name is bound to an object? For instance, when a method is called, and its 'self' parameter gets set to that object? That miiiiiight be a little problematic. ChrisA From Richard at Damon-Family.org Sat Mar 3 12:37:08 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 12:37:08 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On 3/3/18 11:33 AM, Michael Torrie wrote: > On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: >> I can assure you that RAII does what it says on the tin and is relied on in >> many critical systems to release resources robustly ... given the >> pre-requisite deterministic destruction. > Sure but did you read what Paul Moore wrote? He said RAII works in C++ > because objects are allocated on the *stack* with strict lifetimes and > scopes. They won't ever have cycles and they are guaranteed to be > destroyed no matter what as the stack is unwound. Python has no > stack-allocated objects. > > In C++, Heap-allocated objects must still be managed manually, without > the benefit of RAII, for much of the same reasons as people are giving > here for why RAII is not a good fit for Python. There are smart pointer > objects that try to give RAII semantics to heap-allocated objects, with > varying degrees of success. In other words there are some limitations. > > Python does not have stack-allocated objects, so the same issues that > prevent RAII from automatically applying in C++ to heap objects exist here. Yes, stack allocated object in C++ have a nice lifetime to allow RAII to work, but it doesn't just work with stack allocated objects. A lot of RAII objects are members of a class object that may well be allocated on the heap, and RAII makes sure that all the needed cleanup gets done when that object gets destroyed. Yes, I suspect that directly creating a RAII control object as the sole object on the heap is unusual, but that doesn't make the never appear on the heap. When the resource is memory, since one chunk of memory tends to be just as usable as another, garbage collection works great. Other resources can be quite precious, and delays in releasing them can have significant effects. With RAII and immediate destruction on end of scope, we can automate the release, without it and you need a lot of explicit code to manage these resources. Maybe the issue is that most Pythonic code doesn't need to deal with these sorts of resources where this becomes important to be dealt with automatically, either being held for the whole program so OS cleanup handles it, or the are so fine grained that the explicitness isn't an issue. -- Richard Damon From rosuav at gmail.com Sat Mar 3 12:43:40 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 04:43:40 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon wrote: > On 3/3/18 11:33 AM, Michael Torrie wrote: >> >> On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: >>> >>> I can assure you that RAII does what it says on the tin and is relied on >>> in >>> many critical systems to release resources robustly ... given the >>> pre-requisite deterministic destruction. >> >> Sure but did you read what Paul Moore wrote? He said RAII works in C++ >> because objects are allocated on the *stack* with strict lifetimes and >> scopes. They won't ever have cycles and they are guaranteed to be >> destroyed no matter what as the stack is unwound. Python has no >> stack-allocated objects. >> >> In C++, Heap-allocated objects must still be managed manually, without >> the benefit of RAII, for much of the same reasons as people are giving >> here for why RAII is not a good fit for Python. There are smart pointer >> objects that try to give RAII semantics to heap-allocated objects, with >> varying degrees of success. In other words there are some limitations. >> >> Python does not have stack-allocated objects, so the same issues that >> prevent RAII from automatically applying in C++ to heap objects exist >> here. > > > Yes, stack allocated object in C++ have a nice lifetime to allow RAII to > work, but it doesn't just work with stack allocated objects. A lot of RAII > objects are members of a class object that may well be allocated on the > heap, and RAII makes sure that all the needed cleanup gets done when that > object gets destroyed. How do you guarantee that the heap object is properly disposed of when you're done with it? Your RAII object depends 100% on the destruction of the heap object. ChrisA From Richard at Damon-Family.org Sat Mar 3 13:22:27 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 13:22:27 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On 3/3/18 12:43 PM, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon wrote: >> On 3/3/18 11:33 AM, Michael Torrie wrote: >>> On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: >>>> I can assure you that RAII does what it says on the tin and is relied on >>>> in >>>> many critical systems to release resources robustly ... given the >>>> pre-requisite deterministic destruction. >>> Sure but did you read what Paul Moore wrote? He said RAII works in C++ >>> because objects are allocated on the *stack* with strict lifetimes and >>> scopes. They won't ever have cycles and they are guaranteed to be >>> destroyed no matter what as the stack is unwound. Python has no >>> stack-allocated objects. >>> >>> In C++, Heap-allocated objects must still be managed manually, without >>> the benefit of RAII, for much of the same reasons as people are giving >>> here for why RAII is not a good fit for Python. There are smart pointer >>> objects that try to give RAII semantics to heap-allocated objects, with >>> varying degrees of success. In other words there are some limitations. >>> >>> Python does not have stack-allocated objects, so the same issues that >>> prevent RAII from automatically applying in C++ to heap objects exist >>> here. >> >> Yes, stack allocated object in C++ have a nice lifetime to allow RAII to >> work, but it doesn't just work with stack allocated objects. A lot of RAII >> objects are members of a class object that may well be allocated on the >> heap, and RAII makes sure that all the needed cleanup gets done when that >> object gets destroyed. > How do you guarantee that the heap object is properly disposed of when > you're done with it? Your RAII object depends 100% on the destruction > of the heap object. > > ChrisA Yes, the heap object 'owns' a resource, and as long as that heap object exists, the resource needs to exist, but as soon as it doesn't, you want that resource freed. That heap object might well be controlled by some other RAII object (maybe a smart pointer with a ref count) or maybe that object is being manually controlled by the program. But the key is that when it goes away, all of the resources it controls via RAII are automatically cleaned up, and you don't need all that cleanup made explicit in the owning class. If the resource is plentiful, like memory, we can simplify the life management by using something like garbage collection, so if some is used longer than needed it isn't a big issue. Other resources are much more valuable and limited, and it is important to release them when no longer needed. Without some method to make sure that happens 'automatically' as soon as possible, it requires that the programmer explicitly deal with it in full detail. Now, a lot of talk has been throw around about things like reference cycles getting in the way of this, but practically, the resources that this would be needed for, then not to get involved in those sorts of issues, but tend to live in more simple structures, so the fact that objects in a cycle might need for the garbage collector to determine they aren't used anymore isn't an issue (and the presence of garbage collection says that these more complicated structures, which only hold resources that are plentiful, don't need to be precisely managed will RAII controls). -- Richard Damon From rosuav at gmail.com Sat Mar 3 13:28:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 05:28:36 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sun, Mar 4, 2018 at 5:22 AM, Richard Damon wrote: > On 3/3/18 12:43 PM, Chris Angelico wrote: >> >> On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon >> wrote: >>> >>> On 3/3/18 11:33 AM, Michael Torrie wrote: >>>> >>>> On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: >>>>> >>>>> I can assure you that RAII does what it says on the tin and is relied >>>>> on >>>>> in >>>>> many critical systems to release resources robustly ... given the >>>>> pre-requisite deterministic destruction. >>>> >>>> Sure but did you read what Paul Moore wrote? He said RAII works in C++ >>>> because objects are allocated on the *stack* with strict lifetimes and >>>> scopes. They won't ever have cycles and they are guaranteed to be >>>> destroyed no matter what as the stack is unwound. Python has no >>>> stack-allocated objects. >>>> >>>> In C++, Heap-allocated objects must still be managed manually, without >>>> the benefit of RAII, for much of the same reasons as people are giving >>>> here for why RAII is not a good fit for Python. There are smart pointer >>>> objects that try to give RAII semantics to heap-allocated objects, with >>>> varying degrees of success. In other words there are some limitations. >>>> >>>> Python does not have stack-allocated objects, so the same issues that >>>> prevent RAII from automatically applying in C++ to heap objects exist >>>> here. >>> >>> >>> Yes, stack allocated object in C++ have a nice lifetime to allow RAII to >>> work, but it doesn't just work with stack allocated objects. A lot of >>> RAII >>> objects are members of a class object that may well be allocated on the >>> heap, and RAII makes sure that all the needed cleanup gets done when that >>> object gets destroyed. >> >> How do you guarantee that the heap object is properly disposed of when >> you're done with it? Your RAII object depends 100% on the destruction >> of the heap object. >> >> ChrisA > > > Yes, the heap object 'owns' a resource, and as long as that heap object > exists, the resource needs to exist, but as soon as it doesn't, you want > that resource freed. > > That heap object might well be controlled by some other RAII object (maybe a > smart pointer with a ref count) or maybe that object is being manually > controlled by the program. But the key is that when it goes away, all of the > resources it controls via RAII are automatically cleaned up, and you don't > need all that cleanup made explicit in the owning class. You're trying to handwave away the *exact point* that we are all making: EVERY object in Python is a heap object. So how is your magic wand going to make any of this work? You have to do it exclusively with heap objects because there is nothing else available. ChrisA From phd at phdru.name Sat Mar 3 13:54:49 2018 From: phd at phdru.name (Oleg Broytman) Date: Sat, 3 Mar 2018 19:54:49 +0100 Subject: Cheetah 3.1.0 Message-ID: <20180303185449.bqg3pxk56ahu6lbl@phdru.name> Hello! I'm pleased to announce version 3.1.0, the first stable release of branch 3.1 of CheetahTemplate3. What's new in CheetahTemplate3 ============================== Contributors for this release is Mathias Stearn. Features: - Fix Cheetah to work with PyPy. Pull request by Mathias Stearn. Minor features: - Code cleanup: fix code style to satisfy flake8 linter. Documentation: - Rename www directory to docs. Tests: - Run pypy tests at AppVeyor. - Use remove-old-files.py from ppu to cleanup pip cache at Travis and AppVeyor. What is CheetahTemplate3 ======================== Cheetah3 is a free and open source template engine. It's a fork of the original CheetahTemplate library. Python 2.7 or 3.3+ is required. Where is CheetahTemplate3 ========================= Site: http://cheetahtemplate.org/ Development: https://github.com/CheetahTemplate3 Download: https://pypi.python.org/pypi/Cheetah3/3.1.0 News and changes: http://cheetahtemplate.org/news.html StackOverflow: https://stackoverflow.com/questions/tagged/cheetah Example ======= Below is a simple example of some Cheetah code, as you can see it's practically Python. You can import, inherit and define methods just like in a regular Python module, since that's what your Cheetah templates are compiled to :) :: #from Cheetah.Template import Template #extends Template #set $people = [{'name' : 'Tom', 'mood' : 'Happy'}, {'name' : 'Dick', 'mood' : 'Sad'}, {'name' : 'Harry', 'mood' : 'Hairy'}] How are you feeling?
    #for $person in $people
  • $person['name'] is $person['mood']
    • #end for
Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From Richard at Damon-Family.org Sat Mar 3 14:01:48 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 14:01:48 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On 3/3/18 1:28 PM, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 5:22 AM, Richard Damon wrote: >> On 3/3/18 12:43 PM, Chris Angelico wrote: >>> On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon >>> wrote: >>>> On 3/3/18 11:33 AM, Michael Torrie wrote: >>>>> On 03/03/2018 09:02 AM, ooomzay at gmail.com wrote: >>>>>> I can assure you that RAII does what it says on the tin and is relied >>>>>> on >>>>>> in >>>>>> many critical systems to release resources robustly ... given the >>>>>> pre-requisite deterministic destruction. >>>>> Sure but did you read what Paul Moore wrote? He said RAII works in C++ >>>>> because objects are allocated on the *stack* with strict lifetimes and >>>>> scopes. They won't ever have cycles and they are guaranteed to be >>>>> destroyed no matter what as the stack is unwound. Python has no >>>>> stack-allocated objects. >>>>> >>>>> In C++, Heap-allocated objects must still be managed manually, without >>>>> the benefit of RAII, for much of the same reasons as people are giving >>>>> here for why RAII is not a good fit for Python. There are smart pointer >>>>> objects that try to give RAII semantics to heap-allocated objects, with >>>>> varying degrees of success. In other words there are some limitations. >>>>> >>>>> Python does not have stack-allocated objects, so the same issues that >>>>> prevent RAII from automatically applying in C++ to heap objects exist >>>>> here. >>>> >>>> Yes, stack allocated object in C++ have a nice lifetime to allow RAII to >>>> work, but it doesn't just work with stack allocated objects. A lot of >>>> RAII >>>> objects are members of a class object that may well be allocated on the >>>> heap, and RAII makes sure that all the needed cleanup gets done when that >>>> object gets destroyed. >>> How do you guarantee that the heap object is properly disposed of when >>> you're done with it? Your RAII object depends 100% on the destruction >>> of the heap object. >>> >>> ChrisA >> >> Yes, the heap object 'owns' a resource, and as long as that heap object >> exists, the resource needs to exist, but as soon as it doesn't, you want >> that resource freed. >> >> That heap object might well be controlled by some other RAII object (maybe a >> smart pointer with a ref count) or maybe that object is being manually >> controlled by the program. But the key is that when it goes away, all of the >> resources it controls via RAII are automatically cleaned up, and you don't >> need all that cleanup made explicit in the owning class. > You're trying to handwave away the *exact point* that we are all > making: EVERY object in Python is a heap object. So how is your magic > wand going to make any of this work? You have to do it exclusively > with heap objects because there is nothing else available. > > ChrisA Because while every OBJECT in Python is on the heap, not every THING is, as you do have local and global names bound to those objects. This is like in C++, in C++ there is no way to directly reference an object on the heap, but need to use a stack or statically allocated object or reference (which has a name) to get to it. The fact that all objects exist on the heap actually makes some things simpler, in C++ you need to be careful with smart pointers that they only point to objects on the heap, and thus are delete-able. If you accidentally put the address of a static or stack based object into one of them (and don't add a fake reference to the count) you will run into an issue when the reference count goes to zero and you try to destroy and then free the memory for the object. In Python, that can't happen. -- Richard Damon From Richard at Damon-Family.org Sat Mar 3 17:26:41 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 17:26:41 -0500 Subject: cute interview problem In-Reply-To: References: <87a7vv71zg.fsf@nightsong.com> <7c9db8d0-64d3-4793-b9a1-c3274f602f85@googlegroups.com> Message-ID: On 2/28/18 3:51 PM, Ian Kelly wrote: > On Wed, Feb 28, 2018 at 12:55 PM, wrote: >> On Tuesday, 27 February 2018 00:42:02 UTC+1, Paul Rubin wrote: >>> Ron Aaron posted the below url on comp.lang.forth. It points to what I >>> thought was a cute problem, along with his solution in his Forth dialect >>> 8th: >>> >>> https://8th-dev.com/forum/index.php/topic,1584.msg8720.html >>> >>> I wrote a solution in Forth and additional ones in Python and Haskell >>> and thought people here might like trying it themselves. I'll post my >>> Python version here in a few days if anyone wants to see it. Time limit >>> for the problem is supposed to be 45 minutes. I spent a lot longer >>> because I ended up writing several versions in various languages. >> >> I dont think its cute at all. >> how did the interview go? > > Yeah, seems to me this is basically just asking "have you ever heard > of an interval tree (or can you invent one on the fly)". > An interval tree sounds a bit more complicated than you really need (there is no need to keep the original list of intervals intact, we just need to know if a number was in ANY forbidden interval). All you really need to do is build a binary search tree of disjoint intervals, where before actually adding the next interval in the list you find the intervals it is between and merge rather than add if it overlaps (or is adjacent), and merging the two intervals (the prior and next) if it fills all the space between. The output is just a simple walking of the tree and processing the gaps between recorded forbidden intervals. You also could just build up a tree of forbidden intervals, sorted by starting value, and not consider overlaps, and in the output walk just keep track of the highest end of interval encountered so far to effectively merge them at readout time. From ooomzay at gmail.com Sat Mar 3 18:05:10 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Sat, 3 Mar 2018 15:05:10 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Saturday, March 3, 2018 at 4:33:59 PM UTC, Michael Torrie wrote: > On 03/03/2018 09:02 AM, ooomzay wrote: > > I can assure you that RAII does what it says on the tin and is relied on in > > many critical systems to release resources robustly ... given the > > pre-requisite deterministic destruction. > > Sure but did you read what Paul Moore wrote? Yes. > He said RAII works in C++ > because objects are allocated on the *stack* with strict lifetimes and > scopes. Well he was not telling you the whole story: RAII works just as well with heap objects using smart pointers (unique_ptr and friends) which are a closer analogy to python object references. > In C++, Heap-allocated objects must still be managed manually, without > the benefit of RAII, No one should be manually managing resources on the heap in C++. They should be using smart pointers. > for much of the same reasons as people are giving > here for why RAII is not a good fit for Python. ...for much the same reasons I am giving here for why RAII could be a very good fit for python. > There are smart pointer > objects that try to give RAII semantics to heap-allocated objects, with > varying degrees of success. In other words there are some limitations. Not sure what limitations you are evoking here but I have not had to write a delete or suffered a resource leak in C++ for many years. (we rolled our own smart pointers before they were standardised). > Python does not have stack-allocated objects, so the same issues that > prevent RAII from automatically applying in C++ to heap objects exist here. False premise, false conclusion. From ian.g.kelly at gmail.com Sat Mar 3 18:47:25 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sat, 3 Mar 2018 16:47:25 -0700 Subject: cute interview problem In-Reply-To: References: <87a7vv71zg.fsf@nightsong.com> <7c9db8d0-64d3-4793-b9a1-c3274f602f85@googlegroups.com> Message-ID: On Sat, Mar 3, 2018 at 3:26 PM, Richard Damon wrote: > On 2/28/18 3:51 PM, Ian Kelly wrote: >> >> On Wed, Feb 28, 2018 at 12:55 PM, wrote: >>> >>> On Tuesday, 27 February 2018 00:42:02 UTC+1, Paul Rubin wrote: >>>> >>>> Ron Aaron posted the below url on comp.lang.forth. It points to what I >>>> thought was a cute problem, along with his solution in his Forth dialect >>>> 8th: >>>> >>>> https://8th-dev.com/forum/index.php/topic,1584.msg8720.html >>>> >>>> I wrote a solution in Forth and additional ones in Python and Haskell >>>> and thought people here might like trying it themselves. I'll post my >>>> Python version here in a few days if anyone wants to see it. Time limit >>>> for the problem is supposed to be 45 minutes. I spent a lot longer >>>> because I ended up writing several versions in various languages. >>> >>> >>> I dont think its cute at all. >>> how did the interview go? >> >> >> Yeah, seems to me this is basically just asking "have you ever heard >> of an interval tree (or can you invent one on the fly)". >> > > An interval tree sounds a bit more complicated than you really need (there > is no need to keep the original list of intervals intact, we just need to > know if a number was in ANY forbidden interval). All you really need to do > is build a binary search tree of disjoint intervals, where before actually > adding the next interval in the list you find the intervals it is between > and merge rather than add if it overlaps (or is adjacent), and merging the > two intervals (the prior and next) if it fills all the space between. > > The output is just a simple walking of the tree and processing the gaps > between recorded forbidden intervals. > > You also could just build up a tree of forbidden intervals, sorted by > starting value, and not consider overlaps, and in the output walk just keep > track of the highest end of interval encountered so far to effectively merge > them at readout time. For that matter, I suppose that you don't even really need a tree. Just put all the intervals in a list, sort the list by the low endpoint of each interval, and iterate as above. Maybe it's a better interview question that I first thought, due to the propensity to overthink it. Quick and dirty Python implementation, assuming that the intervals have already been read (because file I/O is boring): def number_allowed_and_smallest(minval, maxval, forbidden): allowed = 0 smallest = None previous_high = minval - 1 for (low, high) in sorted(forbidden): if low > previous_high + 1: allowed += low - 1 - previous_high if smallest is None: smallest = previous_high + 1 previous_high = max(high, previous_high) if maxval > previous_high: allowed += maxval - previous_high if smallest is None: smallest = previous_high + 1 return allowed, smallest From steve+comp.lang.python at pearwood.info Sat Mar 3 18:49:51 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 3 Mar 2018 23:49:51 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sat, 03 Mar 2018 12:37:08 -0500, Richard Damon wrote: > With RAII and immediate destruction on end of scope, we can automate the > release, without it and you need a lot of explicit code to manage these > resources. Not so much. with resource_i_care_about() as rsrc: process(rsrc) is hardly what I call "a lot of explicit code". Yes, there's some code in the context manager, but you write that *once* (if it isn't already written for you!) and you're done. I know that laziness and hubris are programmer virtues, but there is still such a thing as *too much laziness*. RAII works in C++ where instances are allocated in the stack, but even there, if you have an especially long-lived function, your resources won't be closed promptly. In Python terms: def function(): x = open_resource() process(x) # and we're done with x now, but too lazy to explicitly close it sleep(10000) # Simulate some more work. Lots of work. return # and finally x is closed (2.8 hours after you finished using it) The answer in C++ is "well don't do that then". The answer is Python is, "don't be so lazy, just use a with statement". If you want deterministic closing of resources, with statements are the way to do it. def function(): with open_resource() as x: process(x) # and x is guaranteed to be closed sleep(10000) # Simulate some more work. Lots of work. return -- Steve From steve+comp.lang.python at pearwood.info Sat Mar 3 18:57:01 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 3 Mar 2018 23:57:01 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: On Sat, 03 Mar 2018 10:01:43 -0700, Ian Kelly wrote: > On Sat, Mar 3, 2018 at 9:19 AM, Richard Damon > wrote: >> One idea does come to mind though, would it be reasonable, and somewhat >> Pythonic, for a class to define member functions like __ref__ and >> __unref__ (or perhaps some other name) that if defined, would be called >> every time a name was bound or unbound to an object? (the assignment to >> a name should __ref__ the new value, then __unref__ the old to avoid >> possible issues with rebinding an object to the last name it was bound >> to). This would allow those (limited) objects that want to be destroyed >> as soon as possible to do so, but most other objects wouldn't have any >> significant overhead added (just a check for this method). >> >> Objects with these methods would still be subject to being cleaned up >> with garbage collection in the case they were kept alive via a cycle, >> having the cycle just makes it so that you don't get the immediate >> distruction. > > This sounds like a nightmare for performance. Ref counting is enough of > a drain already when it's done in C, using macros for in-lining. Now > every time the count would be incremented or decremented, we'd also need > to check the class __dict__ to see if there's a corresponding Python > method, build a frame object, and then call it in the interpreter. Not just the class __dict__. You would have to do a full search of the MRO looking for any superclass which defines such methods. Python code doesn't tend to go in for long inheritance hierarchies, but still, you could easily be checking four or five classes on *every* binding. Richard's assertion that this wouldn't add significant overhead isn't right. [...] > It would be much more efficient to spare all that by doing the > ref-counting in the interpreter and just call a method when it hits > zero. Which is just what CPython already does. Indeed. I'm afraid I don't understand what Richard's proposal is actually meant to do. It sounds to me like he's just proposed adding a second, parallel, optional, *SLOW* reference counter to the CPython implementation, in order to force non-ref counter implementations to do something which they have no need to do. -- Steve From Richard at Damon-Family.org Sat Mar 3 19:30:59 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 19:30:59 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On 3/3/18 6:49 PM, Steven D'Aprano wrote: > On Sat, 03 Mar 2018 12:37:08 -0500, Richard Damon wrote: > >> With RAII and immediate destruction on end of scope, we can automate the >> release, without it and you need a lot of explicit code to manage these >> resources. > Not so much. > > with resource_i_care_about() as rsrc: > process(rsrc) > > > is hardly what I call "a lot of explicit code". Yes, there's some code in > the context manager, but you write that *once* (if it isn't already > written for you!) and you're done. The with statement handles the C++ case of using a unique_ptr, where the resource is aquired and release within a single block scope and thus a single stack based object is ultimately responsible for the lifetime of the resource. I am not sure how it would handle a resource that gets shared for a while with something like a shared_ptr, and needs to be released when all the people who is has been shared with are done with their copy. Admittedly, this case is much less likely to be using such a critical resource, but it might. In Python, currently, you would either need to keep an explicit usage count to allow you to delete it when done, or let things wait for the collector to find it. -- Richard Damon From Richard at Damon-Family.org Sat Mar 3 19:36:02 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 19:36:02 -0500 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: <39e8e791-7041-1fe9-c6f4-5eace7ebc26d@Damon-Family.org> On 3/3/18 6:57 PM, Steven D'Aprano wrote: > On Sat, 03 Mar 2018 10:01:43 -0700, Ian Kelly wrote: > >> On Sat, Mar 3, 2018 at 9:19 AM, Richard Damon >> wrote: >>> One idea does come to mind though, would it be reasonable, and somewhat >>> Pythonic, for a class to define member functions like __ref__ and >>> __unref__ (or perhaps some other name) that if defined, would be called >>> every time a name was bound or unbound to an object? (the assignment to >>> a name should __ref__ the new value, then __unref__ the old to avoid >>> possible issues with rebinding an object to the last name it was bound >>> to). This would allow those (limited) objects that want to be destroyed >>> as soon as possible to do so, but most other objects wouldn't have any >>> significant overhead added (just a check for this method). >>> >>> Objects with these methods would still be subject to being cleaned up >>> with garbage collection in the case they were kept alive via a cycle, >>> having the cycle just makes it so that you don't get the immediate >>> distruction. >> This sounds like a nightmare for performance. Ref counting is enough of >> a drain already when it's done in C, using macros for in-lining. Now >> every time the count would be incremented or decremented, we'd also need >> to check the class __dict__ to see if there's a corresponding Python >> method, build a frame object, and then call it in the interpreter. > Not just the class __dict__. You would have to do a full search of the > MRO looking for any superclass which defines such methods. > > Python code doesn't tend to go in for long inheritance hierarchies, but > still, you could easily be checking four or five classes on *every* > binding. > > Richard's assertion that this wouldn't add significant overhead isn't > right. > > [...] >> It would be much more efficient to spare all that by doing the >> ref-counting in the interpreter and just call a method when it hits >> zero. Which is just what CPython already does. > Indeed. > > I'm afraid I don't understand what Richard's proposal is actually meant > to do. It sounds to me like he's just proposed adding a second, parallel, > optional, *SLOW* reference counter to the CPython implementation, in > order to force non-ref counter implementations to do something which they > have no need to do. > > If checking for a method definiton is that slow, it wouldn't work. The idea was to have a way to mark that certain classes/objects request that they are reference counted so they get the __del__ called as soon as the last reference goes away, without needing to require that overhead for all objects in all implementations. -- Richard Damon From ooomzay at gmail.com Sat Mar 3 19:49:42 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Sat, 3 Mar 2018 16:49:42 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: <58d72ea2-149b-4f0e-9975-a15d79ce77c5@googlegroups.com> On Friday, March 2, 2018 at 3:37:25 PM UTC, Paul Moore wrote: > [...] > RAII works in C++ (where it was initially invented) because it's used > with stack-allocated variables that have clearly-defined and limited > scope. RAII also works with smart pointers, which are a closer analogue to python references. > In my experience writing C++, nobody uses RAII with > heap-allocated variables - those require explicit allocation and > deallocation and so are equivalent to having an explicit "close()" > method in Python (or using __del__ in CPython as it currently exists). In my experience no-one has been been explicitly deallocating heap-variables for many years: they have been using smart pointers and RAII. > Python doesn't have stack allocation, nor does it have a deterministic > order of deletion of objects when their last reference goes out of > scope (which can happen simultaneously for many objects): > > class Tracker: > def __init__(self, n): > self.n = n > def __del__(self): > print("Deleting instance", self.n) > > def f(): > a = Tracker(1) > b = Tracker(2) > > f() > > The language doesn't guarantee that a is removed before b. Are you > proposing to make that change to the language as well? It would improve things more, but I have not included such a proposal in the current PEP as this is not essential for the main benefit. > Also Python only has function scope, so variables local to a > smaller-than-the-function block of code aren't possible. That's > something that is used in C++ a lot to limit the lifetime of resources > under RAII. How do you propose to address that (without needing > explicit del statements)? Create a sub function. > That's why the with statement exists, to > clearly define lifetimes smaller than "the enclosing function". It does that in a half-hearted and smelly way compared to a function: The object references are left dangling in a broken state (not invariant). In fact you would do well to wrap your "with"s in a function to contain the smell. > Your proposal doesn't offer any equivalent (other than an extra function). An equivalent is not needed. Using a function is the ideomatic and does not leak - unlike the "with" hack. (IMO block-scoping would enhance the langauge but is not essential so not in this PEP). > Consider C++: > > void fn() { > for (i = 0; i < 10000; ++i) { > char name[100]; > sprintf(name, "file%d.txt, i); > File f(name); // I don't think std::ofstream doesn't support RAII > f << "Some text"; > } > } > > Or (real Python): > > def fn(): > for i in range(10000): > with open(f"file{i}.txt", "w") as f: > f.write("Some text") > > How would you write this in your RAII style - without leaving 10,000 > file descriptors open until the end of the function? def write_some_text_to_file(fname): f = RAIIFileAccess(fname, 'w') f.write("Some text") def fn(): for i in range(10000): write_some_text_to_file(f"file{i}.txt") > That's both less efficient (function calls have a cost) Oh come now. No one is choosing to use python for its efficiency and functions that deal with real resources, such as files, are likely dominated by the os fopen/close etc. > and less maintainable than the with-statement version. I disagree. Even for such a trivial example, as soon as you start to change the detail of what you do then the usual maintainability benefits of small well factored functions comes into play. That's why we have functions in the language isn't it? Or do you just write one long script? From ooomzay at gmail.com Sat Mar 3 20:14:48 2018 From: ooomzay at gmail.com (ooomzay at gmail.com) Date: Sat, 3 Mar 2018 17:14:48 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <50a95cb7-1ce7-4c6a-b23a-42fa7bc599ef@googlegroups.com> Message-ID: <88195018-c4f9-4af7-9997-c05caf4463f4@googlegroups.com> On Friday, March 2, 2018 at 5:29:54 AM UTC, Rustom Mody wrote: > Please excuse if this has been addressed above and/or its too basic: > What's the difference between RAII and python's with/context-managers? They address the same problem but I am claiming that RAII achieves this in a significantly more elegant/pythonic way without involving any special keywords or methods. in summary _if_ the PEP was adopted and/or you are using CPython today then:- def riaa_file_copy(srcname, dstname): src = RAIIFile(srcname, 'r') dst = RAIIFile(dstname, 'w') for line in src: dst.write(line) becomes equivalent to: def pep343_file_copy(srcname, dstname): with open(srcname, 'r') as src, open(dstname, 'w') as dst: for line in src: dst.write(line) RAII resource management is also simpler to implement only requiring existing __init__ and __del__ methods (e.g. to open/close the underlying file) and the resource objects are invariant. Which means the objects/managers do not need to track the enter/exit state - as there is no way to access them when they are not "open" in RAII. From ooomzay at gmail.com Sat Mar 3 20:55:55 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sat, 3 Mar 2018 17:55:55 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <559003cd-23e4-7b13-e284-6b82901c273c@nedbatchelder.com> Message-ID: <38520792-f62d-4e32-b89b-79d6e16b736d@googlegroups.com> On Saturday, 3 March 2018 17:16:14 UTC, Ned Batchelder wrote: > On 3/2/18 10:36 AM, Paul Moore wrote: > > Or (real Python): > > > > def fn(): > > for i in range(10000): > > with open(f"file{i}.txt", "w") as f: > > f.write("Some text") > > > > How would you write this in your RAII style - without leaving 10,000 > > file descriptors open until the end of the function? > > IIUC, if the OP's proposal were accepted, the __del__ method would be > called as soon as the *value*'s reference count went to zero. That means > this wouldn't leave 10,000 files open, since each open() would assign a > new file object to f, which would make the previous file object's ref > count be zero, and it would be closed. You have understood correctly. Here is the equivalent RAII version:- def fn(): for i in range(10000): f = RAIIFile(f"file{i}.txt", "w") f.write("Some text") From greg.ewing at canterbury.ac.nz Sat Mar 3 20:57:44 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sun, 04 Mar 2018 14:57:44 +1300 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: ooomzay at gmail.com wrote: > Well he was not telling you the whole story: RAII works just as well with > heap objects using smart pointers (unique_ptr and friends) which are a closer > analogy to python object references. By that definition, *all* resource management in Python is based on RAII[1]. The only difference is that in Python the only way to destroy an object is to remove all references to it, whereas in C++ you have some other options (explicit delete, stack allocation, etc.) However, when people talk about RAII in C++ they're *usually* referring to stack-allocated objects. There are various ways of getting the same effect in Python: (a) Rely on CPython reference counting (b) Use a with-statement (c) Use a try-finally statement Of these, (a) is the closest in spirit to stack-allocated RAII in C++, but it's only guaranteed to work in current CPython. [1] Well, not quite -- you can use low-level OS calls to open files, etc. But the high-level wrappers for files, sockets, and so forth all use RAII. -- Greg From ooomzay at gmail.com Sat Mar 3 21:01:30 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sat, 3 Mar 2018 18:01:30 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> On Saturday, 3 March 2018 17:44:08 UTC, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon > > Yes, stack allocated object in C++ have a nice lifetime to allow RAII to > > work, but it doesn't just work with stack allocated objects. A lot of RAII > > objects are members of a class object that may well be allocated on the > > heap, and RAII makes sure that all the needed cleanup gets done when that > > object gets destroyed. > > How do you guarantee that the heap object is properly disposed of when > you're done with it? Your RAII object depends 100% on the destruction > of the heap object. Smart pointers (unique_ptr and friends) are used to manage heap object lifecycles n . These are analogous to python object references. From rosuav at gmail.com Sat Mar 3 21:10:39 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 13:10:39 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> Message-ID: On Sun, Mar 4, 2018 at 1:01 PM, Ooomzay wrote: > On Saturday, 3 March 2018 17:44:08 UTC, Chris Angelico wrote: >> On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon >> > Yes, stack allocated object in C++ have a nice lifetime to allow RAII to >> > work, but it doesn't just work with stack allocated objects. A lot of RAII >> > objects are members of a class object that may well be allocated on the >> > heap, and RAII makes sure that all the needed cleanup gets done when that >> > object gets destroyed. >> >> How do you guarantee that the heap object is properly disposed of when >> you're done with it? Your RAII object depends 100% on the destruction >> of the heap object. > > Smart pointers (unique_ptr and friends) are used to manage heap object lifecycles n . These are analogous to python object references. Yep, cool. Now do that with all of your smart pointers being on the heap too. You are not allowed to use ANY stack objects. ANY. Got it? ChrisA From greg.ewing at canterbury.ac.nz Sat Mar 3 21:13:41 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sun, 04 Mar 2018 15:13:41 +1300 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <87a7vobw62.fsf@nightsong.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> <87a7vobw62.fsf@nightsong.com> Message-ID: Paul Rubin wrote: > Richard Damon writes: > >>a class to define member functions like __ref__ >>and __unref__ (or perhaps some other name) that if defined, would be >>called every time a name was bound or unbound to an object? > > That sounds horrendous and wouldn't handle the case of a list element > creating a reference without binding a name. To be any use, they would have to be called any time the reference count changes, which happens *very* frequently -- so massive overhead. There's also a logical problem. Calling __ref__ would itself involve adding at least one reference to the object temporarily, thus requiring __ref__ to be called, which would... -- Greg From ooomzay at gmail.com Sat Mar 3 21:19:37 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sat, 3 Mar 2018 18:19:37 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> On Saturday, 3 March 2018 23:52:34 UTC, Steven D'Aprano wrote: > I know that laziness and hubris are programmer virtues, but there is > still such a thing as *too much laziness*. RAII works in C++ where > instances are allocated in the stack, but even there, if you have an > especially long-lived function, your resources won't be closed promptly. > In Python terms: > > def function(): > x = open_resource() > process(x) > # and we're done with x now, but too lazy to explicitly close it > sleep(10000) # Simulate some more work. Lots of work. > return > # and finally x is closed (2.8 hours after you finished using it) > > The answer in C++ is "well don't do that then". The answer is Python is, > "don't be so lazy, just use a with statement". The answer in C++ would be to say "don't be so lazy just put the x manipulation in a function or sub-block". The answer with Python + this PEP would be "don't be so lazy just put the x manipulation in a function or explicitly del x" ...no new syntax. > If you want deterministic closing of resources, with statements are the > way to do it. > > def function(): > with open_resource() as x: > process(x) > # and x is guaranteed to be closed What a palava! From greg.ewing at canterbury.ac.nz Sat Mar 3 21:24:32 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sun, 04 Mar 2018 15:24:32 +1300 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> Message-ID: Steven D'Aprano wrote: > Not just the class __dict__. You would have to do a full search of the > MRO looking for any superclass which defines such methods. That could be reduced a lot by making it a type slot. But it would still increase the overhead of every refcount change by at least a factor of two, as a rough estimate. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 21:27:45 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Sun, 04 Mar 2018 15:27:45 +1300 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <87efl33vaj.fsf@nightsong.com> <8e8b9ab4-bac4-cc1d-b6c6-34333f67891e@Damon-Family.org> <39e8e791-7041-1fe9-c6f4-5eace7ebc26d@Damon-Family.org> Message-ID: Richard Damon wrote: > The > idea was to have a way to mark that certain classes/objects request that > they are reference counted so they get the __del__ called as soon as the > last reference goes away, without needing to require that overhead for > all objects in all implementations. That could be done with a flag in the object or type header. But checking the flag every time a reference is added or removed is probably about as expensive as doing the refcount adjustment. -- Greg From ooomzay at gmail.com Sat Mar 3 21:28:49 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sat, 3 Mar 2018 18:28:49 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote: [snip] > def fn(): > for i in range(10000): > with open(f"file{i}.txt", "w") as f: > f.write("Some text") > > How would you write this in your RAII style - without leaving 10,000 > file descriptors open until the end of the function? def fn(): for i in range(10000): f = RAIIFile(f"file{i}.txt", "w") f.write("Some text") From Richard at Damon-Family.org Sat Mar 3 21:34:26 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 3 Mar 2018 21:34:26 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> Message-ID: <56f92be1-b80a-ce74-1519-b7161dff3fba@Damon-Family.org> On 3/3/18 9:10 PM, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 1:01 PM, Ooomzay wrote: >> On Saturday, 3 March 2018 17:44:08 UTC, Chris Angelico wrote: >>> On Sun, Mar 4, 2018 at 4:37 AM, Richard Damon >>>> Yes, stack allocated object in C++ have a nice lifetime to allow RAII to >>>> work, but it doesn't just work with stack allocated objects. A lot of RAII >>>> objects are members of a class object that may well be allocated on the >>>> heap, and RAII makes sure that all the needed cleanup gets done when that >>>> object gets destroyed. >>> How do you guarantee that the heap object is properly disposed of when >>> you're done with it? Your RAII object depends 100% on the destruction >>> of the heap object. >> Smart pointers (unique_ptr and friends) are used to manage heap object lifecycles n . These are analogous to python object references. > Yep, cool. Now do that with all of your smart pointers being on the > heap too. You are not allowed to use ANY stack objects. ANY. Got it? > > ChrisA Now write your Python without using anything not on the heap either (so no binding your objects to a name). As Chris said, the stack variables in C++ are the local names used to refer to the objects. -- Richard Damon From ooomzay at gmail.com Sat Mar 3 21:45:22 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sat, 3 Mar 2018 18:45:22 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sunday, 4 March 2018 01:58:02 UTC, Gregory Ewing wrote: > ooomzay wrote: > > Well he was not telling you the whole story: RAII works just as well with > > heap objects using smart pointers (unique_ptr and friends) which are a closer > > analogy to python object references. > > By that definition, *all* resource management in Python is > based on RAII[1]. I think not. Objects may have a close() in their __del__ method as a back-up - but currently this is not guaranteed to be called in a timely fashion. Hence the justification for "with" and my proposal to obviate it by actually guaranteeing that __del__ is always called in a timely fashion. From rosuav at gmail.com Sat Mar 3 21:59:56 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 13:59:56 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sun, Mar 4, 2018 at 1:45 PM, Ooomzay wrote: > On Sunday, 4 March 2018 01:58:02 UTC, Gregory Ewing wrote: >> ooomzay wrote: >> > Well he was not telling you the whole story: RAII works just as well with >> > heap objects using smart pointers (unique_ptr and friends) which are a closer >> > analogy to python object references. >> >> By that definition, *all* resource management in Python is >> based on RAII[1]. > > I think not. Objects may have a close() in their __del__ method as a back-up - but currently this is not guaranteed to be called in a timely fashion. Hence the > justification for "with" and my proposal to obviate it by actually guaranteeing > that __del__ is always called in a timely fashion. This thread is dead. The OP wants to wave a magic wand and say "__del__ is now guaranteed to be called immediately", without any explanation - and, from the look of things, without any understanding - of what that means for the language and the interpreters. Everyone else is saying "your magic wand is broken". This is not going to go anywhere. ChrisA From steve+comp.lang.python at pearwood.info Sat Mar 3 22:29:22 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 03:29:22 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On Sat, 03 Mar 2018 08:02:35 -0800, ooomzay wrote: [...] >> > But I am not! On the contrary RAII frees the programmer from even >> > having to remember to close the file. The poster asked what would >> > happen if the resource was deliberately kept open by storing a >> > reference at global scope. >> >> You say that as if it were difficult to do, requiring the programmer to >> take extraordinary steps of heroic proportion. It doesn't. It is >> unbelievably easy to store the reference at global scope, which means >> that the programmer needs to remember to close the file explicitly and >> RAII doesn't help. > > Can you expand on what you mean by "unbelievably easy to store the > reference at global scope". I will assume you are claiming that it is > easy to _inadvertently_ store a global reference. No. Copy and paste this line: --- %< --------- x = 1 --- %< --------- into a text file, and save it as "demo_global.py". Congratulations, you've just created a reference in the global scope! I told you it was easy. Do you put your imports at the top of the module, as recommended? Every one of those imported modules are references in the global scope. Do you define classes and functions at the top level of your module? Those too are references in the global scope. I'll accept that, *typically*, imported modules, functions and classes are unlikely to involve the sorts of resources that we care about closing in a timely fashion. But other kinds of top-level instances may. The point is, you want to *mandate* RAII as a *language feature* rather than an implementation detail which Python interpreters are free to choose or not choose on their own. This (probably?) will put a VERY high burden on Jython, IronPython and Stackless, as well as anyone else who tries making a Python interpreter. And for what benefit? Let's see: - RAII doesn't make reference cycles go away, so we still need a garbage collector that can break cycles; - RAII rarely helps for objects in the module global namespace, and many Python objects are in the global namespace; - to make RAII work for heap objects, every implementation needs to implement their own "smart pointers" or similar, otherwise the entire idea is a bust; this adds significant complexity to interpreters that aren't written in C++ (i.e. all of them apart from Nuitka); - to make the RAII guarantees of timely resource closure work, you have to impose significant mental burdens on the programmer: * no cycles (avoid them, or use weak references); * no globals; * if you must use a global, then you are responsible for manually deleting it in order to allow RAII to operate; - to make the RAII guarantees work *well*, you need to introduce new scoping rules (new to Python, not C++ ) where each indented block (loops, if...else blocks etc) are new scopes; otherwise the lifetime of a function scope could easily be too long. > This problem with destruction of globals on exit is not unique to > python. There are patterns to manage this for code you control, such as > putting them all in one global "bucket" object and explicitly deleting > it before exit. The problem we have is, we want the resource to be closed in a timely manner once we're done with it. And your solution is to put it in a "God Object" that collects a whole lot of unrelated resources, then wait until you are done with the last of the them, before closing any of them? That's not a pattern managing the problem, that's a work-around that fails to come even close to fixing the problem. [...] >> But as you said yourself, if the resource is held open in a global >> reference, it will stay open indefinitely. And remember, global in this >> context doesn't just mean the main module of your application, but >> *every* module you import. > > Can you give an example of such a case where an application-managed > (acquired/opened & released/closed) resource is referenced strongly by a > 3rd party module at global scope? ...in my experience this pattern > usually smells. Regardless of whether it smells or not, it is allowed. >> I think you have just put your finger on the difference between what >> RAII *claims* to do and what it *actually* can do. > > I can assure you that RAII does what it says on the tin and is relied on > in many critical systems to release resources robustly ... given the > pre-requisite deterministic destruction. Given the execution environment and design constraints of C++, I'm sure RAII works fine in C++ (and any other language which may copy it). But it is not a panacea that makes resource management go away. You yourself have stated at least one limitation: if you hold a global reference to the object then RAII won't help you. So your proposal requires major changes to not just one but all (present and future) Python interpreters, AND major changes to the way people write Python code, in order to solve a problem which is already solved by context managers and the with statement. -- Steve From steve+comp.lang.python at pearwood.info Sat Mar 3 23:20:31 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 04:20:31 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sat, 03 Mar 2018 18:19:37 -0800, Ooomzay wrote: >> def function(): >> x = open_resource() >> process(x) >> # and we're done with x now, but too lazy to explicitly close it >> sleep(10000) # Simulate some more work. Lots of work. >> return >> # and finally x is closed (2.8 hours after you finished using it) >> >> The answer in C++ is "well don't do that then". The answer is Python >> is, "don't be so lazy, just use a with statement". > > The answer in C++ would be to say "don't be so lazy just put the x > manipulation in a function or sub-block". Right -- so you still have to think about resource management. The premise of this proposal is that RAII means you don't have to think about resource management, it just happens, but that's not really the case. > The answer with Python + this > PEP would be "don't be so lazy just put the x manipulation in a function > or explicitly del x" ...no new syntax. Sure -- but it doesn't gain you anything we don't already have. It imposes enormous burdens on the maintainers of at least five interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which will need to be re-written to have RAII semantics guaranteed; it will probably have significant performance costs; and at the end of the day, the benefit over using a with statement is minor. (Actually, I'm not convinced that there is *any* benefit. If anything, I think it will be a reliability regression -- see below.) >> If you want deterministic closing of resources, with statements are the >> way to do it. >> >> def function(): >> with open_resource() as x: >> process(x) >> # and x is guaranteed to be closed > > What a palava! I don't know that word, and neither do any of my dictionaries. I think the word you might mean is "pelaver"? Palaver \Pa*la"ver\, n. [Sp. palabra, or Pg. palavra, fr. L. parabola a comparison, a parable, LL., a word. See Parable.] [1913 Webster] 1. Talk; conversation; esp., idle or beguiling talk; talk intended to deceive; flattery. In any case, you might not like with statements, but I think they're infinitely better than: def meaningless_function_that_exists_only_to_manage_resource(): x = open_resource() process(x) def function(): meaningless_function_that_exists_only_to_manage_resource() sleep(10000) # simulate a long-running function Or worse: def function(): if True: # Do this just to introduce a new scope, as in C++ # (but not Python today) x = open_resource() process(x) sleep(10000) # simulate a long-running function In other words, your solutions are just as much manual resource management as the with statement. The only differences are: - instead of explicitly using a dedicated syntax designed for resource management, you're implicitly using scope behaviour; - the with block is equivalent to a try...finally, and so it is guaranteed to close the resource even if an exception occurs; your solution isn't. If process(x) creates a non-local reference to x, and then raises an exception, and that exception is caught elsewhere, x will not go out of scope and won't be closed. A regression in the reliability of the code. (I'm moderately confident that this failure of RAII to deliver what it promises is possible in C++ too.) -- Steve From ooomzay at gmail.com Sun Mar 4 06:37:38 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 03:37:38 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: > On Sat, 03 Mar 2018 18:19:37 -0800, Ooomzay wrote: > > >> def function(): > >> x = open_resource() > >> process(x) > >> # and we're done with x now, but too lazy to explicitly close it > >> sleep(10000) # Simulate some more work. Lots of work. > >> return > >> # and finally x is closed (2.8 hours after you finished using it) > >> > >> The answer in C++ is "well don't do that then". The answer is Python > >> is, "don't be so lazy, just use a with statement". > > > > The answer in C++ would be to say "don't be so lazy just put the x > > manipulation in a function or sub-block". > > Right -- so you still have to think about resource management. The > premise of this proposal is that RAII means you don't have to think about > resource management, it just happens, but that's not really the case. That is not my main premise: Which is that RAII is a more elegant (no specialised syntax at all) and robust way to manage resources. Please consider the case of a composite resource: You need to implement __enter__, __exit__ and track the open/closed state at every level in your component hierarchy - even if some levels hold no resources directly. This is burdensome, breaks encapsulation, breaks invariance and is error prone ...very unpythonic. > > The answer with Python + this > > PEP would be "don't be so lazy just put the x manipulation in a function > > or explicitly del x" ...no new syntax. > > Sure -- but it doesn't gain you anything we don't already have. See above. > It imposes enormous burdens on the maintainers of at least five > interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which > will need to be re-written to have RAII semantics guaranteed; Yes. This is a substantial issue that will almost certainly see it rejected by HWCNBN on political, rather than linguistic, grounds. My PEP is about improving the linguistic integrity and fitness for resource management purpose of the language. > it will > probably have significant performance costs; and at the end of the day, > the benefit over using a with statement is minor. > > (Actually, I'm not convinced that there is *any* benefit. If anything, I > think it will be a reliability regression -- see below.) > > >> If you want deterministic closing of resources, with statements are the > >> way to do it. > >> > >> def function(): > >> with open_resource() as x: > >> process(x) > >> # and x is guaranteed to be closed > > > > What a palava! > > I don't know that word, and neither do any of my dictionaries. I think > the word you might mean is "pelaver"? I don't know that word, and neither do any of my dictionaries. I think the word you might mean is "Palaver"? Anyway Palava/Pelaver/Palaver/Palavra/Palabra derives from the word for "word", but in England it is often used idiomatically to mean a surfeit of words, or even more generally, a surfeit of effort. I intend it in both senses: The unnecessary addition of the words "with", "as", "__enter__" & "__exit__" to the language and the need implement the latter two methods all over the place. > In any case, you might not like with statements, but I think they're > infinitely better than: > > def meaningless_function_that_exists_only_to_manage_resource(): > x = open_resource() > process(x) > def function(): > meaningless_function_that_exists_only_to_manage_resource() > sleep(10000) # simulate a long-running function Why would you prefer a new construct? Functions _are_ pythons scoping context! Giving one a pejorative name does not change that. > In other words, your solutions are just as much manual resource > management as the with statement. The only differences are: > > - instead of explicitly using a dedicated syntax designed for > resource management, you're implicitly using scope behaviour; Excellent: With the benefit of automatic, exception safe, destruction of resources, including composite resources. > - the with block is equivalent to a try...finally, and so it is > guaranteed to close the resource even if an exception occurs; > your solution isn't. > If process(x) creates a non-local reference to x, and then raises an > exception, and that exception is caught elsewhere, x will not go out of > scope and won't be closed. > A regression in the reliability of the code. This PEP does not affect existing code. Peeps who are familiar with RAII understand that creating a global reference to an RAII resource is explicitly saying "I want this kept open at global scope" and that is the behaviour that they will be guaranteed. > (I'm moderately confident that this failure of RAII to deliver what it > promises is possible in C++ too.) (Well I bet the entire VOIP stack, AAA and accounting modules in a certain 99999 satellite network's base-stations against you) From rosuav at gmail.com Sun Mar 4 07:14:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 23:14:05 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sun, Mar 4, 2018 at 10:37 PM, Ooomzay wrote: > Please consider the case of a composite resource: You need to implement > __enter__, __exit__ and track the open/closed state at every level in > your component hierarchy - even if some levels hold no resources directly. > > This is burdensome, breaks encapsulation, breaks invariance and is error prone > ...very unpythonic. Why do you need to? I don't understand your complaint here - can you give an example of a composite resource that needs this kind of special management? ChrisA From ooomzay at gmail.com Sun Mar 4 07:37:46 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 04:37:46 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: > [...] > [This PEP] imposes enormous burdens on the maintainers of at least five > interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which > will need to be re-written to have RAII semantics guaranteed; Not so:- CPython, the reference interpreter, already implements the required behaviour, as mentioned in the PEP. From ooomzay at gmail.com Sun Mar 4 08:26:03 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 05:26:03 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <87a7voa5tr.fsf@nightsong.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On Sunday, 4 March 2018 03:16:31 UTC, Paul Rubin wrote: > Chris Angelico writes: > > Yep, cool. Now do that with all of your smart pointers being on the > > heap too. You are not allowed to use ANY stack objects. ANY. Got it? > > That's both overconstraining and not even that big a problem the way you > phrase it. > > 1) Python has both the "with" statement and try/finally. Both of these > run code at the exit from a syntactically defined scope. So they are > like stack allocation in C++, where a deallocator can run when the scope > exits. > > 2) Even with no scope-based de-allocation, it's common to put smart > pointers into containers like lists and vectors. So you could have a > unique_ptr to a filestream object, and stash the unique_ptr someplace as > a vector element, where the vector itself could be part of some even > more deeply nested structure. At some point, the big structure gets > deleted (maybe through a manually-executed delete statement). When that > happens, if the nested structures are all standard containers full of > unique_ptrs, the top-level finalizer will end up traversing the entire > tree and freeing up the file handles and whatever else might be in > there. > > It occurs to me, maybe #2 above is closer to what the OP is really after > in Python. Yep. C++ smart pointers are a good analogue to python references for purposes of this PEP. > I guess it's doable, but refcounts don't seem like the right > way. Well refcounts are definitely "doable": This is how the reference python implementation, CPython, currently manages to comply with this PEP and can therefore be used for RAII. This PEP is an attempt to _guarantee_ this behaviour and make the elegance of RAII available to all pythonistas that want it. Without this guarantee python is not attractive to applications that must manage non-trivial resources reliably. Aside: I once read somewhere that must have seemed authoritative at the time, that CPython _guarantees_ to continue to behave like this - but now the subject is topical again I can find no trace of this guarantee. From maillist at schwertberger.de Sun Mar 4 08:26:29 2018 From: maillist at schwertberger.de (Dietmar Schwertberger) Date: Sun, 4 Mar 2018 14:26:29 +0100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On 3/4/2018 1:37 PM, Ooomzay wrote: > Not so:- CPython, the reference interpreter, already implements the > required behaviour, as mentioned in the PEP. It does most of the time, but it's not guaranteed. See my previous post. Regards, Dietmar From rosuav at gmail.com Sun Mar 4 08:46:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 00:46:09 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On Mon, Mar 5, 2018 at 12:26 AM, Ooomzay wrote: > Well refcounts are definitely "doable": This is how the reference python implementation, CPython, currently manages to comply with this PEP and can > therefore be used for RAII. > > This PEP is an attempt to _guarantee_ this behaviour and make the elegance > of RAII available to all pythonistas that want it. Without this guarantee > python is not attractive to applications that must manage non-trivial > resources reliably. > > Aside: I once read somewhere that must have seemed authoritative at the > time, that CPython _guarantees_ to continue to behave like this - but now the subject is topical again I can find no trace of this guarantee. That's because there is no such guarantee. In fact, you can probably find places in the docs where it is specifically stated that you cannot depend on __del__ for this. You still haven't said how you're going to cope with reference cycles - or are you expecting every single decref to do a full cycle check? ChrisA From steve+comp.lang.python at pearwood.info Sun Mar 4 08:58:46 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 13:58:46 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sun, 04 Mar 2018 04:37:46 -0800, Ooomzay wrote: > On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: >> [...] >> [This PEP] imposes enormous burdens on the maintainers of at least five >> interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of >> which will need to be re-written to have RAII semantics guaranteed; > > Not so:- CPython, the reference interpreter, already implements the > required behaviour, as mentioned in the PEP. Except that it doesn't. From the docs: "It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits." https://docs.python.org/3/reference/datamodel.html#object.__del__ That limitation of CPython is much reduced now compared to older versions, but there are still circumstances where the __del__ method cannot be called. -- Steve From ooomzay at gmail.com Sun Mar 4 09:11:42 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 06:11:42 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Sunday, 4 March 2018 03:00:13 UTC, Chris Angelico wrote: > This thread is dead. The OP wants to wave a magic wand and say > "__del__ is now guaranteed to be called immediately", No "magic" required: Just one line change in the language reference will do it. > without any explanation The PEP says it all really: To make the very pythonic RAII idiom available in python. > - and, from the look of things, without any understanding > - of what that means for the language What impact on the _language_ (c.f. interpreter) do you think I have not understood? It is is 100% backwards compatible with the language. It breaks nothing. It allows people who want to use the RAII idiom to do so. It allows people who want to use the "with" idiom to continue do so. > and the interpreters. I am well aware of what it will mean for interpreters. For some interpreters it will have zero impact (e.g. CPython) and for some others it would unlikely be economic to make them comply. The decision here is does python want to be more pythonic, and make itself attractive for resource management applications or does it want to be compromised by some implementations? > Everyone else is saying "your magic wand is broken". This is not going to go > anywhere. Well I see a lot of posts that indicate peeps here are more comfortable with the "with" idiom than the RAII idiom but I have not yet seen a single linguistic problem or breakage. As it happens I have used RAII extensively with CPython to manage a debugging environment with complex external resources that need managing very efficiently. (I would use C++ if starting from scratch because it _guarantees_ the required deterministic destruction whereas python does not) From rosuav at gmail.com Sun Mar 4 09:19:06 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 01:19:06 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: On Mon, Mar 5, 2018 at 1:11 AM, Ooomzay wrote: > On Sunday, 4 March 2018 03:00:13 UTC, Chris Angelico wrote: >> This thread is dead. The OP wants to wave a magic wand and say >> "__del__ is now guaranteed to be called immediately", > > No "magic" required: Just one line change in the language reference will do it. Go ahead and actually implement it. It's not just one line in the language reference. >> without any explanation > > The PEP says it all really: To make the very pythonic RAII idiom available in python. > >> - and, from the look of things, without any understanding >> - of what that means for the language > > What impact on the _language_ (c.f. interpreter) do you think I have not understood? > > It is is 100% backwards compatible with the language. It breaks nothing. > > It allows people who want to use the RAII idiom to do so. > > It allows people who want to use the "with" idiom to continue do so. > >> and the interpreters. > > I am well aware of what it will mean for interpreters. For some interpreters it will have zero impact (e.g. CPython) and for some others it would unlikely be economic to make them comply. > I don't even understand you now. First off, you just acknowledged that this WILL impact CPython - you're not simply mandating what CPython already does, you're tightening up the rules significantly. For others - what do you even mean, "unlikely be economic to make them comply"? Are you saying that the Jython project should simply die? Or that it should become non-compliant with the Python spec? Huh? ChrisA From ned at nedbatchelder.com Sun Mar 4 09:37:00 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 4 Mar 2018 09:37:00 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> On 3/4/18 8:26 AM, Ooomzay wrote: > On Sunday, 4 March 2018 03:16:31 UTC, Paul Rubin wrote: >> Chris Angelico writes: >>> Yep, cool. Now do that with all of your smart pointers being on the >>> heap too. You are not allowed to use ANY stack objects. ANY. Got it? >> That's both overconstraining and not even that big a problem the way you >> phrase it. >> >> 1) Python has both the "with" statement and try/finally. Both of these >> run code at the exit from a syntactically defined scope. So they are >> like stack allocation in C++, where a deallocator can run when the scope >> exits. >> >> 2) Even with no scope-based de-allocation, it's common to put smart >> pointers into containers like lists and vectors. So you could have a >> unique_ptr to a filestream object, and stash the unique_ptr someplace as >> a vector element, where the vector itself could be part of some even >> more deeply nested structure. At some point, the big structure gets >> deleted (maybe through a manually-executed delete statement). When that >> happens, if the nested structures are all standard containers full of >> unique_ptrs, the top-level finalizer will end up traversing the entire >> tree and freeing up the file handles and whatever else might be in >> there. >> >> It occurs to me, maybe #2 above is closer to what the OP is really after >> in Python. > Yep. C++ smart pointers are a good analogue to python references for > purposes of this PEP. > >> I guess it's doable, but refcounts don't seem like the right >> way. > Well refcounts are definitely "doable": This is how the reference python implementation, CPython, currently manages to comply with this PEP and can > therefore be used for RAII. > > Are you including cyclic references in your assertion that CPython behaves as you want? --Ned. From ned at nedbatchelder.com Sun Mar 4 09:37:34 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 4 Mar 2018 09:37:34 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: <7c93eb20-e5c6-c21b-83dc-174e4e17694f@nedbatchelder.com> On 3/4/18 7:37 AM, Ooomzay wrote: > On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: >> [...] >> [This PEP] imposes enormous burdens on the maintainers of at least five >> interpreters (CPython, Stackless, Jython, IronPython, PyPy) all of which >> will need to be re-written to have RAII semantics guaranteed; > Not so:- CPython, the reference interpreter, already implements the required behaviour, as mentioned in the PEP. Except for cycles. --Ned. From steve+comp.lang.python at pearwood.info Sun Mar 4 10:21:26 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 15:21:26 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: On Sun, 04 Mar 2018 03:37:38 -0800, Ooomzay wrote: > Please consider the case of a composite resource: You need to implement > __enter__, __exit__ and track the open/closed state at every level in > your component hierarchy - even if some levels hold no resources > directly. I'm sorry, that description is too abstract for me to understand. Can you give a simple example? > This is burdensome, breaks encapsulation, breaks invariance and is error > prone ...very unpythonic. Without a more concrete example, I cannot comment on these claims. [...] > My PEP is about improving the linguistic integrity and fitness for > resource management purpose of the language. So you claim, but *requiring* reference counting semantics does not improve the integrity or fitness of the language. And the required changes to programming styles and practices (no cycles, no globals, put all resources inside their own scope) demonstrate that this is a step backwards. Right now I can write reliable code that uses external resources (such as a database connection or file) and put it in my application's module scope, and still easily manage the resource lifetime. I cannot do that by relying only on RAII. That's a step backwards as far as language fitness. >> In any case, you might not like with statements, but I think they're >> infinitely better than: >> >> def meaningless_function_that_exists_only_to_manage_resource(): >> x = open_resource() >> process(x) > >> def function(): >> meaningless_function_that_exists_only_to_manage_resource() >> sleep(10000) # simulate a long-running function > > Why would you prefer a new construct? I don't prefer a new construct. The "with" statement isn't "new". It goes back to Python 2.5 (`from __future__ import with_statement`) which is more than eleven years old now. That's about half the lifetime of the language! I prefer the with statement because it is *explicit*, simple to use, and clear to read. I can read some code and instantly see that when the with block ends, the resource will be closed, regardless of how many references to the object still exist. I don't have to try to predict (guess!) when the last reference will go out of scope, because that's irrelevant. RAII conflates the lifetime of the object with the lifetime of the resource held by the object. They are not the same, and the object can outlive the resource. Your position is: "RAII makes it really elegant to close the file! All you need to do is make sure that when you want to close the file, you delete all the references to the file, so that it goes out of scope, and the file will be closed." My position is: "If I want to close the file, I'll just close the file. Why should I care that there are zero or one or a million references to it?" >> - the with block is equivalent to a try...finally, and so it is >> guaranteed to close the resource even if an exception occurs; your >> solution isn't. > >> If process(x) creates a non-local reference to x, and then raises an >> exception, and that exception is caught elsewhere, x will not go out of >> scope and won't be closed. >> A regression in the reliability of the code. > > This PEP does not affect existing code. Peeps who are familiar with RAII > understand that creating a global reference to an RAII resource is > explicitly saying "I want this kept open at global scope" and that is > the behaviour that they will be guaranteed. I'm not talking about global scope. Any persistent reference to the object will prevent the resource from being closed. Here's a proof of concept which demonstrates the problem with conflating object scope and resource lifetime. Notice that there are no globals used. def main(): from time import sleep values = [] def process(): f = open('/tmp/foo', 'w') values.append(f) f.write("Hello world!") f.read() # oops! del values[-1] try: process() except IOError: pass # The file should be closed now. But it isn't. sleep(10) # simulate doing a lot of work g = open('/tmp/foo', 'r') assert g.read() == "Hello world!" The assertion fails, because the file hasn't be closed in a timely manner. On the other hand: def main2(): from time import sleep values = [] def process(): with open('/tmp/bar', 'w') as f: values.append(f) f.write("Hello world!") f.read() # oops! del values[-1] try: process() except IOError: pass sleep(10) # simulate doing a lot of work g = open('/tmp/foo', 'r') assert g.read() == "Hello world!" The assertion here passes. Now, these are fairly contrived examples, but in real code the resource owner might be passed into an iterator, or bound to a class attribute, or anything else that holds onto a reference to it. As soon as that happens, and there's another reference to the object anywhere, RAII will be unable to close the resource in a timely manner. -- Steve From breamoreboy at gmail.com Sun Mar 4 11:42:53 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Sun, 4 Mar 2018 16:42:53 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: On 04/03/18 02:28, Ooomzay wrote: > On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote: > [snip] >> def fn(): >> for i in range(10000): >> with open(f"file{i}.txt", "w") as f: >> f.write("Some text") >> >> How would you write this in your RAII style - without leaving 10,000 >> file descriptors open until the end of the function? > > def fn(): > for i in range(10000): > f = RAIIFile(f"file{i}.txt", "w") > f.write("Some text") > Over my dead body. Not that it matters as I can't see this happening in a month of Sundays. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From steve+comp.lang.python at pearwood.info Sun Mar 4 11:46:32 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 16:46:32 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <7c93eb20-e5c6-c21b-83dc-174e4e17694f@nedbatchelder.com> Message-ID: On Sun, 04 Mar 2018 09:37:34 -0500, Ned Batchelder wrote: > On 3/4/18 7:37 AM, Ooomzay wrote: >> On Sunday, 4 March 2018 04:23:07 UTC, Steven D'Aprano wrote: >>> [...] >>> [This PEP] imposes enormous burdens on the maintainers of at least >>> five interpreters (CPython, Stackless, Jython, IronPython, PyPy) all >>> of which will need to be re-written to have RAII semantics guaranteed; >> Not so:- CPython, the reference interpreter, already implements the >> required behaviour, as mentioned in the PEP. > > Except for cycles. And during interpreter shutdown. -- Steve From ned at nedbatchelder.com Sun Mar 4 13:54:32 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 4 Mar 2018 13:54:32 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: <93370a27-d465-a7d0-5f51-1a74ed619a2c@nedbatchelder.com> On 3/4/18 9:11 AM, Ooomzay wrote: > I am well aware of what it will mean for interpreters. For some interpreters it will have zero impact (e.g. CPython) ... There's no point continuing this if you are just going to insist on falsehoods like this.? CPython doesn't currently do what you want, but you are choosing to ignore the cases where it doesn't.? You have to deal with cycles. --Ned. From bc at freeuk.com Sun Mar 4 14:02:29 2018 From: bc at freeuk.com (bartc) Date: Sun, 4 Mar 2018 19:02:29 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> Message-ID: <7pXmC.79036$lm2.51072@fx13.am4> On 04/03/2018 14:11, Ooomzay wrote: > Well I see a lot of posts that indicate peeps here are more comfortable with > the "with" idiom than the RAII idiom but I have not yet seen a single > linguistic problem or breakage. > > As it happens I have used RAII extensively with CPython to manage a debugging environment with complex external resources that need managing very efficiently. I have bit of problem with features normally used for the housekeeping of a language's data structures being roped in to control external resources that it knows nothing about. Which also means that if X is a variable with the sole reference to some external resource, then a mere: X = 0 will close, destroy, or discard that resource. If the resource is non-trivial, then it perhaps deserves a non-trivial piece of code to deal with it when it's no longer needed. It further means that when you did want to discard an expensive resource, then X going out of scope, calling del X or whatever, will not work if a copy of X still exists somewhere. -- bartc From Richard at Damon-Family.org Sun Mar 4 17:09:16 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 4 Mar 2018 17:09:16 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On 3/4/18 8:46 AM, Chris Angelico wrote: > On Mon, Mar 5, 2018 at 12:26 AM, Ooomzay wrote: >> Well refcounts are definitely "doable": This is how the reference python implementation, CPython, currently manages to comply with this PEP and can >> therefore be used for RAII. >> >> This PEP is an attempt to _guarantee_ this behaviour and make the elegance >> of RAII available to all pythonistas that want it. Without this guarantee >> python is not attractive to applications that must manage non-trivial >> resources reliably. >> >> Aside: I once read somewhere that must have seemed authoritative at the >> time, that CPython _guarantees_ to continue to behave like this - but now the subject is topical again I can find no trace of this guarantee. > That's because there is no such guarantee. In fact, you can probably > find places in the docs where it is specifically stated that you > cannot depend on __del__ for this. > > You still haven't said how you're going to cope with reference cycles > - or are you expecting every single decref to do a full cycle check? > > ChrisA My presumption of the proposal is the it does NOT expect that __del__ be called just because an object is no longer reachable but is in a cycle of isolated objects, those would still need to wait for the garbage collector to get them. The request is that when the direct reference count goes to 0, that __del__ gets called. I think that is what CPython promises (but not other versions). I am not positive if __del__ gets called for sure when the object is garbage collected (I thought it did, but I am not positive). I am pretty sure it does NOT get called on object that are still in existence when things terminate (which would be the major difference from a language like C++) What the language does not promise is that in general, __del__ be called 'immediately' on the last reference going away in the general case, because CPythons reference counting is an implementation detail. My understanding of this proposal is to ask that, I would say at least for selected objects, that all implementations perform this reference counting, allowing objects that track 'critical resources' to get disposed of in a timely manner and not wait for garbage collection. And that this is triggered only by the reference count going to zero, and not if the object happens to be in an isolated reference cycle. This does limit what you can do with this sort of object, but that normally isn't a problem. -- Richard Damon From rosuav at gmail.com Sun Mar 4 17:20:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 09:20:24 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On Mon, Mar 5, 2018 at 9:09 AM, Richard Damon wrote: > My presumption of the proposal is the it does NOT expect that __del__ be > called just because an object is no longer reachable but is in a cycle of > isolated objects, those would still need to wait for the garbage collector > to get them. The request is that when the direct reference count goes to 0, > that __del__ gets called. > > I think that is what CPython promises (but not other versions). > > I am not positive if __del__ gets called for sure when the object is garbage > collected (I thought it did, but I am not positive). > > I am pretty sure it does NOT get called on object that are still in > existence when things terminate (which would be the major difference from a > language like C++) > > What the language does not promise is that in general, __del__ be called > 'immediately' on the last reference going away in the general case, because > CPythons reference counting is an implementation detail. > > My understanding of this proposal is to ask that, I would say at least for > selected objects, that all implementations perform this reference counting, > allowing objects that track 'critical resources' to get disposed of in a > timely manner and not wait for garbage collection. And that this is > triggered only by the reference count going to zero, and not if the object > happens to be in an isolated reference cycle. This does limit what you can > do with this sort of object, but that normally isn't a problem. Okay, that sounds reasonable. Let's help things out by creating a special syntax for reference-counted object usage, to allow other implementations to use different forms of garbage collection. When you acquire these kinds of objects, you "mark" them with this special syntax, and Python will call a special method on that object to say "hey, you're now in use". Then when that special syntax is done, Python calls another special method to say "hey, you're not in use any more". Something like this: using some_object: ... ... ... # at the unindent, we're automatically not using it That would call a special method some_object.__now_using__() at the top of the block, and some_object.__not_using__() at the bottom. Or, if the block exits because of an exception, we could call some_object.__not_using(exc) with the exception details. I think this could be a really good feature - it'd allow non-refcounted Python implementations to have a strong guarantee of immediate disposal of a managed resource, and it'd also strengthen the guarantee for refcounted Pythons too. Feel free to bikeshed the exact keywords and function names, of course. ChrisA From ooomzay at gmail.com Sun Mar 4 17:25:39 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 14:25:39 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> Message-ID: <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: > Are you including cyclic references in your assertion that CPython > behaves as you want? Yes. Because the only behaviour required for RAII is to detect and debug such cycles in order to eliminate them. It is a design error/resource leak to create an orphan cycle containing RAII objects. def main(): gc,disable From steve+comp.lang.python at pearwood.info Sun Mar 4 18:42:03 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 23:42:03 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On Mon, 05 Mar 2018 09:20:24 +1100, Chris Angelico wrote: > Okay, that sounds reasonable. Let's help things out by creating a > special syntax for reference-counted object usage [...] > Feel free to bikeshed the exact keywords and function names, of course. I see what you did there. :-) -- Steve From rosuav at gmail.com Sun Mar 4 18:46:45 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 10:46:45 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> Message-ID: On Mon, Mar 5, 2018 at 10:42 AM, Steven D'Aprano wrote: > On Mon, 05 Mar 2018 09:20:24 +1100, Chris Angelico wrote: > >> Okay, that sounds reasonable. Let's help things out by creating a >> special syntax for reference-counted object usage > [...] >> Feel free to bikeshed the exact keywords and function names, of course. > > > I see what you did there. > > :-) > Yeah.... I'm not sure if the OP will see what I did, though... ChrisA From ned at nedbatchelder.com Sun Mar 4 18:55:08 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Sun, 4 Mar 2018 18:55:08 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> Message-ID: <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> On 3/4/18 5:25 PM, Ooomzay wrote: > On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: >> Are you including cyclic references in your assertion that CPython >> behaves as you want? > Yes. Because the only behaviour required for RAII is to detect and debug such cycles in order to eliminate them. It is a design error/resource leak to create an orphan cycle containing RAII objects. > > def main(): > gc,disable > > > This isn't a reasonable position.? Cycles exist, and the gc exists for a reason.? Your proposal isn't going to go anywhere if you just naively ignore cycles. --Ned. From ooomzay at gmail.com Sun Mar 4 19:58:38 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 16:58:38 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> On Sunday, 4 March 2018 15:24:08 UTC, Steven D'Aprano wrote: > On Sun, 04 Mar 2018 03:37:38 -0800, Ooomzay wrote: > > > Please consider the case of a composite resource: You need to implement > > __enter__, __exit__ and track the open/closed state at every level in > > your component hierarchy - even if some levels hold no resources > > directly. > > This is burdensome, breaks encapsulation, breaks invariance and is error > > prone ...very unpythonic. > > Without a more concrete example, I cannot comment on these claims. Here is an example of a composite resource using RAII:- class RAIIFileAccess(): def __init__(self, fname): print("%s Opened" % fname) def __del__(self): print("%s Closed" % fname) class A(): def __init__(self): self.res = RAIIFileAccess("a") class B(): def __init__(self): self.res = RAIIFileAccess("b") class C(): def __init__(self): self.a = A() self.b = B() def main(): c = C() Under this PEP this is all that is needed to guarantee that the files "a" and "b" are closed on exit from main or after any exception has been handled. Also note that if you have a reference to these objects then they are guaranteed to be in a valid/useable/open state (invariant) - no danger or need to worry/check about enter/exit state. Now repeat this example with "with". > [...] > > My PEP is about improving the linguistic integrity and fitness for > > resource management purpose of the language. > > So you claim, but *requiring* reference counting semantics does not > improve the integrity or fitness of the language. We will just have to disagree on that for now. > And the required > changes to programming styles and practices (no cycles, No change required. But if you _choose_ to benefit from RAII you had better not create orphan cycles with RAII objects in them, as that is clearly a resource leak. > no globals, No change required. But if you _choose_ to benefit from RAII you had better take care to delete any RAII resources you choose to hold at global scope in a robust way. (These are exceptional in my experience). > put all resources inside their own scope) No change required. But if you _choose_ to benefit from RAII you can make use of python's existing scopes (functions) or del to restrict resource lifetimes. > >> In any case, you might not like with statements, but I think they're > >> infinitely better than: > >> > >> def meaningless_function_that_exists_only_to_manage_resource(): > >> x = open_resource() > >> process(x) > > > >> def function(): > >> meaningless_function_that_exists_only_to_manage_resource() > >> sleep(10000) # simulate a long-running function > > > > Why would you prefer a new construct? > > I don't prefer a new construct. The "with" statement isn't "new". It goes > back to Python 2.5 (`from __future__ import with_statement`) which is > more than eleven years old now. That's about half the lifetime of the > language! > > I prefer the with statement because it is *explicit*, simple to use, and > clear to read. I can read some code and instantly see that when the with > block ends, the resource will be closed, regardless of how many > references to the object still exist. > > I don't have to try to predict (guess!) when the last reference will go > out of scope, because that's irrelevant. If you don't care about what the other references might be then RAII is not for you. Fine. > RAII conflates the lifetime of the object with the lifetime of the > resource held by the object. This "conflation" is called "invariance" and is usually considered a "very good thing" as you cant have references floating around to half-baked resources. > They are not the same, and the object can > outlive the resource. Not with RAII it can't. Simple. Good. > Your position is: > > "RAII makes it really elegant to close the file! All you need to do is > make sure that when you want to close the file, you delete all the > references to the file, so that it goes out of scope, and the file will > be closed." > > My position is: > > "If I want to close the file, I'll just close the file. Why should I care > that there are zero or one or a million references to it?" Because if you have no idea what references there are you can not assume it is OK to close the file! That would be a truly terrible program design. > >> - the with block is equivalent to a try...finally, and so it is > >> guaranteed to close the resource even if an exception occurs; your > >> solution isn't. It is: RAII will release all resources held, transitively. Try the example above using CPython and put an exception in one of the constructors. > >> If process(x) creates a non-local reference to x, and then raises an > >> exception, and that exception is caught elsewhere, x will not go out of > >> scope and won't be closed. > >> A regression in the reliability of the code. > > This PEP does not affect existing code. Peeps who are familiar with RAII > > understand that creating a global reference to an RAII resource is > > explicitly saying "I want this kept open at global scope" and that is > > the behaviour that they will be guaranteed. > > I'm not talking about global scope. Any persistent reference to the > object will prevent the resource from being closed. > > Here's a proof of concept which demonstrates the problem with conflating > object scope and resource lifetime. Notice that there are no globals used. > > def main(): > from time import sleep > values = [] > def process(): > f = open('/tmp/foo', 'w') > values.append(f) > f.write("Hello world!") > f.read() # oops! > del values[-1] > try: > process() > except IOError: > pass > # The file should be closed now. But it isn't. That's because you have failed to clean up your exception frame after handling it (sys.exc_clear() or whatever depending on version) and it is holding a reference to your resource, keeping it open till end of the function. > sleep(10) # simulate doing a lot of work > g = open('/tmp/foo', 'r') > assert g.read() == "Hello world!" > > > The assertion fails, because the file hasn't be closed in a timely > manner. On the other hand: > > def main2(): > from time import sleep > values = [] > def process(): > with open('/tmp/bar', 'w') as f: > values.append(f) > f.write("Hello world!") > f.read() # oops! > del values[-1] > try: > process() > except IOError: > pass > sleep(10) # simulate doing a lot of work > g = open('/tmp/foo', 'r') > assert g.read() == "Hello world!" > > > The assertion here passes. > > Now, these are fairly contrived examples, but in real code the resource > owner might be passed into an iterator, or bound to a class attribute, or > anything else that holds onto a reference to it. As soon as that happens, > and there's another reference to the object anywhere, RAII will be unable > to close the resource in a timely manner. There is no way you should be closing a resource while other references to it exist. RAII will not forgive such a design. This is a good thing. From ooomzay at gmail.com Sun Mar 4 20:01:32 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 17:01:32 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> Message-ID: <3a1323ac-3029-4d27-a75c-98bcee6aba0b@googlegroups.com> On Sunday, 4 March 2018 23:57:24 UTC, Mark Lawrence wrote: > On 04/03/18 02:28, Ooomzay wrote: > > On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote: > > [snip] > >> def fn(): > >> for i in range(10000): > >> with open(f"file{i}.txt", "w") as f: > >> f.write("Some text") > >> > >> How would you write this in your RAII style - without leaving 10,000 > >> file descriptors open until the end of the function? > > > > def fn(): > > for i in range(10000): > > f = RAIIFile(f"file{i}.txt", "w") > > f.write("Some text") > > > Over my dead body. Care to expand on that? From ooomzay at gmail.com Sun Mar 4 20:07:32 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 17:07:32 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> Message-ID: <804b12a9-ac05-4d96-9809-00415eb17511@googlegroups.com> On Sunday, 4 March 2018 23:55:33 UTC, Ned Batchelder wrote: > On 3/4/18 5:25 PM, Ooomzay wrote: > > On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder wrote: > >> Are you including cyclic references in your assertion that CPython > >> behaves as you want? > > Yes. Because the only behaviour required for RAII is to detect and debug such cycles in order to eliminate them. It is a design error/resource leak to create an orphan cycle containing RAII objects. > > > This isn't a reasonable position.? Cycles exist, and the gc exists for a > reason.? Your proposal isn't going to go anywhere if you just naively > ignore cycles. I am not naively ignoring them. But _if_ you want to use RAII then do not leak them in cycles. Put anything else in there you like and gc them as before. From Richard at Damon-Family.org Sun Mar 4 20:11:24 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 4 Mar 2018 20:11:24 -0500 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> Message-ID: On 3/4/18 6:55 PM, Ned Batchelder wrote: > On 3/4/18 5:25 PM, Ooomzay wrote: >> On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder? wrote: >>> Are you including cyclic references in your assertion that CPython >>> behaves as you want? >> Yes. Because the only behaviour required for RAII is to detect and >> debug such cycles in order to eliminate them. It is a design >> error/resource leak to create an orphan cycle containing RAII objects. >> >> def main(): >> gc,disable >> >> >> > > This isn't a reasonable position.? Cycles exist, and the gc exists for > a reason.? Your proposal isn't going to go anywhere if you just > naively ignore cycles. > > --Ned. While Ooomzay seems to want to say that all cycles are bad, I think it is fair to say that in general in Python they are unavoidable, in part because I can't see any good way to create the equivalent of a 'weak reference' (names aren't objects so don't have properties). The? best I can think of is to create some sort of magical object that can refer to another object, but that reference 'doesn't count'. This seems very unPythonic. What I think can be said is that it should be possible (enforced by the programmer, not the language) to use these RAII objects in ways that don't create cycles (or maybe that the program knows of the cycles and makes the effort to break them when it is important). So perhaps it can be said that cycle that involve major resource RAII objects should exist. -- Richard Damon From ooomzay at gmail.com Sun Mar 4 20:14:12 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 17:14:12 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: Message-ID: <6d1d406a-3a32-42de-890d-858e9934afea@googlegroups.com> On Saturday, 3 March 2018 17:10:53 UTC, Dietmar Schwertberger wrote: > CPython does *not* guarantee destruction when the object reference goes > out of scope, even if there are no other references. > I would very much appreciate such a deterministic behaviour, at least > with CPython. > > I recently had to debug an issue in the matplotlib wx backend (*). Under > certain conditions, the wx device context was not destroyed when the > reference went out of scope. Adding a del to the end of the method or > calling the Destroy method of the context did fix the issue. (There was > also a hidden reference, but avoiding this was not sufficient. The del > was still required.) You say the reference was out of scope but that a del was still required. What were you delling if the reference was out of scope? Could you sketch the code. From ben+python at benfinney.id.au Sun Mar 4 20:17:46 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 05 Mar 2018 12:17:46 +1100 Subject: RFC: Proposal: Deterministic Object Destruction References: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> Message-ID: <85muzn722t.fsf@benfinney.id.au> Richard Damon writes: > [?] I can't see any good way to create the equivalent of a 'weak > reference' (names aren't objects so don't have properties). The? best > I can think of is to create some sort of magical object that can refer > to another object, but that reference 'doesn't count'. This seems very > unPythonic. Do you mean something like the standard library ?weakref? module ? The weakref module allows the Python programmer to create weak references to objects. [?] A weak reference to an object is not enough to keep the object alive: when the only remaining references to a referent are weak references, garbage collection is free to destroy the referent and reuse its memory for something else. However, until the object is actually destroyed the weak reference may return the object even if there are no strong references to it. -- \ ?The cost of education is trivial compared to the cost of | `\ ignorance.? ?Thomas Jefferson | _o__) | Ben Finney From ooomzay at gmail.com Sun Mar 4 20:18:02 2018 From: ooomzay at gmail.com (Ooomzay) Date: Sun, 4 Mar 2018 17:18:02 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <60d93e02-d73d-454c-9400-4c06b73785ff@googlegroups.com> <87a7voa5tr.fsf@nightsong.com> <39691db0-89b2-29e3-d3fa-342742e413bf@nedbatchelder.com> <70799dc3-541b-4f19-b8cf-634ce60d8f3f@googlegroups.com> <08950acb-e22b-ef5b-8b52-88f3c9568105@nedbatchelder.com> Message-ID: <6de7f50f-e03b-45ee-8462-9474323f28bd@googlegroups.com> On Monday, 5 March 2018 01:11:43 UTC, Richard Damon wrote: > On 3/4/18 6:55 PM, Ned Batchelder wrote: > > On 3/4/18 5:25 PM, Ooomzay wrote: > >> On Sunday, 4 March 2018 14:37:30 UTC, Ned Batchelder? wrote: > >>> Are you including cyclic references in your assertion that CPython > >>> behaves as you want? > >> Yes. Because the only behaviour required for RAII is to detect and > >> debug such cycles in order to eliminate them. It is a design > >> error/resource leak to create an orphan cycle containing RAII objects. > >> > >> def main(): > >> gc,disable > >> > >> > >> > > > > This isn't a reasonable position.? Cycles exist, and the gc exists for > > a reason.? Your proposal isn't going to go anywhere if you just > > naively ignore cycles. > > > > --Ned. > > While Ooomzay seems to want to say that all cycles are bad, I only want to say that orphan cycles with RAII objects in them are bad. From breamoreboy at gmail.com Sun Mar 4 22:13:07 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Mon, 5 Mar 2018 03:13:07 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <3a1323ac-3029-4d27-a75c-98bcee6aba0b@googlegroups.com> References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <3a1323ac-3029-4d27-a75c-98bcee6aba0b@googlegroups.com> Message-ID: On 05/03/18 01:01, Ooomzay wrote: > On Sunday, 4 March 2018 23:57:24 UTC, Mark Lawrence wrote: >> On 04/03/18 02:28, Ooomzay wrote: >>> On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote: >>> [snip] >>>> def fn(): >>>> for i in range(10000): >>>> with open(f"file{i}.txt", "w") as f: >>>> f.write("Some text") >>>> >>>> How would you write this in your RAII style - without leaving 10,000 >>>> file descriptors open until the end of the function? >>> >>> def fn(): >>> for i in range(10000): >>> f = RAIIFile(f"file{i}.txt", "w") >>> f.write("Some text") >>> > >> Over my dead body. > > Care to expand on that? > Sure, when you state what you intend doing about reference cycles, which you've been asked about countless times. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From bfaruq at ymail.com Mon Mar 5 03:37:14 2018 From: bfaruq at ymail.com (Faruq Bashir) Date: Mon, 5 Mar 2018 08:37:14 +0000 (UTC) Subject: Bypassing firewall References: <1155209173.10744394.1520239034020.ref@mail.yahoo.com> Message-ID: <1155209173.10744394.1520239034020@mail.yahoo.com> ? ? How will i bypass web application firewall From rosuav at gmail.com Mon Mar 5 03:48:42 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 19:48:42 +1100 Subject: Bypassing firewall In-Reply-To: <1155209173.10744394.1520239034020@mail.yahoo.com> References: <1155209173.10744394.1520239034020.ref@mail.yahoo.com> <1155209173.10744394.1520239034020@mail.yahoo.com> Message-ID: On Mon, Mar 5, 2018 at 7:37 PM, Faruq Bashir via Python-list wrote: > How will i bypass web application firewall Easy! Talk to the person who's in charge of it. Use a $5 wrench if you have to. https://xkcd.com/538/ ChrisA From alister.ware at ntlworld.com Mon Mar 5 04:29:00 2018 From: alister.ware at ntlworld.com (Alister) Date: Mon, 05 Mar 2018 09:29:00 GMT Subject: Bypassing firewall References: <1155209173.10744394.1520239034020.ref@mail.yahoo.com> <1155209173.10744394.1520239034020@mail.yahoo.com> Message-ID: On Mon, 05 Mar 2018 08:37:14 +0000, Faruq Bashir wrote: > How will i bypass web application firewall For what purpose? is this your firewall? From ooomzay at gmail.com Mon Mar 5 06:03:21 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 03:03:21 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <871sh4ps9b.fsf@nightsong.com> <86285db3-43dd-4b17-a0a2-8a50d39747f0@googlegroups.com> <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <3a1323ac-3029-4d27-a75c-98bcee6aba0b@googlegroups.com> Message-ID: <8b1d7b14-2a89-4678-a1a4-8803c1a26c96@googlegroups.com> On Monday, March 5, 2018 at 6:38:49 AM UTC, Mark Lawrence wrote: > On 05/03/18 01:01, Ooomzay wrote: > > On Sunday, 4 March 2018 23:57:24 UTC, Mark Lawrence wrote: > >> On 04/03/18 02:28, Ooomzay wrote: > >>> On Friday, 2 March 2018 15:37:25 UTC, Paul Moore wrote: > >>> [snip] > >>>> def fn(): > >>>> for i in range(10000): > >>>> with open(f"file{i}.txt", "w") as f: > >>>> f.write("Some text") > >>>> > >>>> How would you write this in your RAII style - without leaving 10,000 > >>>> file descriptors open until the end of the function? > >>> > >>> def fn(): > >>> for i in range(10000): > >>> f = RAIIFile(f"file{i}.txt", "w") > >>> f.write("Some text") > >>> > > > >> Over my dead body. > > > > Care to expand on that? > > > > Sure, when you state what you intend doing about reference cycles, which > you've been asked about countless times. Nothing. No change whatever. As I stated in my second post ref cycles are orthogonal to this PEP. If you want to use RAII objects then you will make sure you avoid adding them to orphan cycles by design. If you don't know how to do that then don't write applications that manage critical resources. From ooomzay at gmail.com Mon Mar 5 06:09:33 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 03:09:33 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> Message-ID: <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> On Sunday, 4 March 2018 23:56:09 UTC, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 10:37 PM, Ooomzay wrote: > > Please consider the case of a composite resource: You need to implement > > __enter__, __exit__ and track the open/closed state at every level in > > your component hierarchy - even if some levels hold no resources directly. > > > > This is burdensome, breaks encapsulation, breaks invariance and is error prone > > ...very unpythonic. > > Why do you need to? I don't understand your complaint here - can you > give an example of a composite resource that needs this kind of > special management? Here is an example of a composite resource using RAII:- class RAIIFileAccess(): def __init__(self, fname): print("%s Opened" % fname) def __del__(self): print("%s Closed" % fname) class A(): def __init__(self): self.res = RAIIFileAccess("a") class B(): def __init__(self): self.res = RAIIFileAccess("b") class C(): def __init__(self): self.a = A() self.b = B() def main(): c = C() Under this PEP this is all that is needed to guarantee that the files "a" and "b" are closed on exit from main or after any exception has been handled. Also note that if you have a reference to these objects then they are guaranteed to be in a valid/useable/open state (invariant) - no danger or need to worry/check about enter/exit state. Now repeat this exercise with "with". From rosuav at gmail.com Mon Mar 5 06:24:06 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 22:24:06 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> Message-ID: On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote: > Here is an example of a composite resource using RAII:- > > class RAIIFileAccess(): > def __init__(self, fname): > print("%s Opened" % fname) > def __del__(self): > print("%s Closed" % fname) > > class A(): > def __init__(self): > self.res = RAIIFileAccess("a") > > class B(): > def __init__(self): > self.res = RAIIFileAccess("b") > > class C(): > def __init__(self): > self.a = A() > self.b = B() > > def main(): > c = C() > > Under this PEP this is all that is needed to guarantee that the files "a" > and "b" are closed on exit from main or after any exception has been handled. Okay. And all your PEP needs is for reference count semantics, right? Okay. I'm going to run this in CPython, with reference semantics. You guarantee that those files will be closed after an exception is handled? Right. >>> def main(): ... c = C() ... c.do_stuff() ... >>> main() a Opened b Opened Traceback (most recent call last): File "", line 1, in File "", line 3, in main AttributeError: 'C' object has no attribute 'do_stuff' >>> Uhh.... I'm not seeing any messages about the files getting closed. Maybe exceptions aren't as easy to handle as you think? Or maybe you just haven't tried any of this (which is obvious from the bug in your code - but even if I fix that, there's still a problem with exception handling). You keep insisting that this is an easy thing. We keep pointing out that it isn't. Now you're proving that you haven't even attempted any of this. Do you believe me now? (Probably not.) ChrisA From kirillbalunov at gmail.com Mon Mar 5 07:12:37 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 5 Mar 2018 15:12:37 +0300 Subject: Ways to make a free variable local to a function? Message-ID: Hi, At the moment, in order to slightly speed up the function in Python, free variables are passed as local variables to the function, thereby getting rid of extra look ups. For example, for the following function, I especially do not use list comprehension) and therefore maybe it's not the best example: def func(numb): res = [] for i in range(numb): res.append(int(i) + float(i)) return res You can get rid of additional look ups, in the following ways: # 1. By passing through local variable's default values def func_local_1(numb, _int = int, _float = float, _range = range): res = [] for i in _range(numb): res.append(_int(i) + _float(i)) return res # 2. Through importing them into the function scope def func_local_2(numb): from builtins import int, float, range res = [] for i in range(numb): res.append(int(i) + float(i)) return res # 3. With the help of various types of closures, version 1 def func_closure_1(numb): _int = int _float = float _range = range def inner(numb): res = [] for i in _range(numb): res.append(_int(i) + _float(i)) return res return inner(numb) # 4. With the help of various types of closures, version 2 def func_closure_2(numb): from builtins import int, float, range def inner(numb): res = [] for i in range(numb): res.append(int(i) + float(i)) return res return inner(numb) Option 1 allows you to achieve the maximum result for both small and a large `numb` values. Option 2 yields a significant overhead, when it is required to call function many times with a small number of iterations. For option 3 and 4, notes are the same, but since they are implemented through closures they give additional small overhead. In case of big `numb` (many iterations, many look ups) these options give a gain of ~10%. Option 1 and 3 I do not like because: - syntax highlighting stops working - the signature function is greatly distorted - additional typing (especially with type annotations) I like options 2 and 4, but they yield a significant overhead, for a small number of iterations. Actually, I have the following question: 1. Is there any other way to make the variable local to the function? a. When you compile (define) a function... b. Inject into an already existing function through decorator...(is it possible?) p.s.: I had the following idea, maybe it was already discussed, the possibility of setting _static_ variables for functions, with the following syntax: def func(numb): static int, float, range res = [] for i in range(numb): res.append(int(i) + float(i)) return res Where identifiers for `static` should correspond to free variables for a function, they must be defined at compile time (like for default arguments) and can not be changed inside the function scope. With kind regards, -gdg From tjreedy at udel.edu Mon Mar 5 08:52:40 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Mar 2018 08:52:40 -0500 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: On 3/5/2018 7:12 AM, Kirill Balunov wrote: > Hi, > > At the moment, in order to slightly speed up the function in Python, free > variables are passed as local variables to the function, thereby getting > rid of extra look ups. For example, for the following function, I > especially do not use list comprehension) and therefore maybe it's not the > best example: > > def func(numb): > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > > You can get rid of additional look ups, in the following ways: > > > # 1. By passing through local variable's default values > > def func_local_1(numb, _int = int, _float = float, _range = range): You are not required to mangle the names. def func_local_1(numb, int = int, float = float, range = range): ... > res = [] > for i in _range(numb): > res.append(_int(i) + _float(i)) > return res > > > # 2. Through importing them into the function scope > > def func_local_2(numb): > from builtins import int, float, range > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > > > # 3. With the help of various types of closures, version 1 > > def func_closure_1(numb): > _int = int > _float = float > _range = range > def inner(numb): > res = [] > for i in _range(numb): > res.append(_int(i) + _float(i)) > return res > return inner(numb) > > > # 4. With the help of various types of closures, version 2 > > def func_closure_2(numb): > from builtins import int, float, range > def inner(numb): > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > return inner(numb) > > Option 1 allows you to achieve the maximum result for both small and a > large `numb` values. Option 2 yields a significant overhead, when it is > required to call function many times with a small number of iterations. For > option 3 and 4, notes are the same, but since they are implemented through > closures they give additional small overhead. In case of big `numb` (many > iterations, many look ups) these options give a gain of ~10%. > > Option 1 and 3 I do not like because: > - syntax highlighting stops working Only because you unnecessarily mangle the names. > - the signature function is greatly distorted > - additional typing (especially with type annotations) > > I like options 2 and 4, but they yield a significant overhead, for a small > number of iterations. > > Actually, I have the following question: > > 1. Is there any other way to make the variable local to the function? > a. When you compile (define) a function... > b. Inject into an already existing function through decorator...(is it > possible?) > > p.s.: > > I had the following idea, maybe it was already discussed, the possibility > of setting _static_ variables for functions, with the following syntax: > > def func(numb): > static int, float, range > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > > Where identifiers for `static` should correspond to free variables for a > function, they must be defined at compile time (like for default arguments) > and can not be changed inside the function scope. > > With kind regards, > -gdg > -- Terry Jan Reedy From ooomzay at gmail.com Mon Mar 5 08:58:47 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 05:58:47 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> Message-ID: <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote: > On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote: > > Here is an example of a composite resource using RAII:- > > > > class RAIIFileAccess(): > > def __init__(self, fname): > > print("%s Opened" % fname) > > def __del__(self): > > print("%s Closed" % fname) > > > > class A(): > > def __init__(self): > > self.res = RAIIFileAccess("a") > > > > class B(): > > def __init__(self): > > self.res = RAIIFileAccess("b") > > > > class C(): > > def __init__(self): > > self.a = A() > > self.b = B() > > > > def main(): > > c = C() > > > > Under this PEP this is all that is needed to guarantee that the files "a" > > and "b" are closed on exit from main or after any exception has been handled. > > Okay. And all your PEP needs is for reference count semantics, right? > Okay. I'm going to run this in CPython, with reference semantics. You > guarantee that those files will be closed after an exception is > handled? Right. > > >>> def main(): > ... c = C() > ... c.do_stuff() > ... > >>> main() > a Opened > b Opened > Traceback (most recent call last): > File "", line 1, in > File "", line 3, in main > AttributeError: 'C' object has no attribute 'do_stuff' > >>> > > Uhh.... I'm not seeing any messages about the files getting closed. Then that is indeed a challenge. From CPython back in 2.6 days up to Python36-32 what I see is:- a Opened b Opened Traceback (most recent call last): ... AttributeError: 'C' object has no attribute 'dostuff' a Closed b Closed > Maybe exceptions aren't as easy to handle as you think? Well there is a general issue with exceptions owing to the ease with which one can create cycles that may catch out newbs. But that is not the case here. > Or maybe you > just haven't tried any of this (which is obvious from the bug in your > code Or maybe I just made a typo when simplifying my test case and failed to retest? Here is my fixed case, if someone else could try it in CPython and report back that would be interesting:- class RAIIFileAccess(): def __init__(self, fname): print("%s Opened" % fname) self.fname = fname def __del__(self): print("%s Closed" % self.fname) class A(): def __init__(self): self.res = RAIIFileAccess("a") class B(): def __init__(self): self.res = RAIIFileAccess("b") class C(): def __init__(self): self.a = A() self.b = B() def main(): c = C() c.dostuff() main() > You keep insisting that this is an easy thing. > We keep pointing out > that it isn't. Now you're proving that you haven't even attempted any > of this. Nonsense. But you have got a result I have never seen in many years and I would like to get to the bottom of it. From rosuav at gmail.com Mon Mar 5 09:21:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 01:21:21 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote: > Then that is indeed a challenge. From CPython back in 2.6 days up to Python36-32 what I see is:- > > a Opened > b Opened > Traceback (most recent call last): > ... > AttributeError: 'C' object has no attribute 'dostuff' > a Closed > b Closed > >> Maybe exceptions aren't as easy to handle as you think? > > Well there is a general issue with exceptions owing to the ease > with which one can create cycles that may catch out newbs. But > that is not the case here. > >> Or maybe you >> just haven't tried any of this (which is obvious from the bug in your >> code > > Or maybe I just made a typo when simplifying my test case and failed to retest? > > Here is my fixed case, if someone else could try it in CPython and report back that would be interesting:- > > class RAIIFileAccess(): > def __init__(self, fname): > print("%s Opened" % fname) > self.fname = fname > > def __del__(self): > print("%s Closed" % self.fname) > > class A(): > def __init__(self): > self.res = RAIIFileAccess("a") > > class B(): > def __init__(self): > self.res = RAIIFileAccess("b") > > class C(): > def __init__(self): > self.a = A() > self.b = B() > > def main(): > c = C() > c.dostuff() > > main() Tried this in the interactive interpreter again. >>> main() a Opened b Opened Traceback (most recent call last): File "", line 1, in File "", line 3, in main AttributeError: 'C' object has no attribute 'dostuff' >>> Same problem! If you can't handle this situation, there's something fundamentally wrong with your system. Here's how I'd do it with context managers. from contextlib import contextmanager @contextmanager def file_access(fname): try: print("%s Opened" % fname) yield finally: print("%s Closed" % fname) @contextmanager def c(): try: print("Starting c") with file_access("a") as a, file_access("b") as b: yield finally: print("Cleaning up c") def main(): with c(): dostuff() # NameError >>> main() Starting c a Opened b Opened b Closed a Closed Cleaning up c Traceback (most recent call last): File "", line 1, in File "", line 3, in main NameError: name 'dostuff' is not defined >>> And if you want different semantics, you can lay out c() differently - maybe have the try/finally not contain within the 'with' but be contained within it, or whatever else you like. Exceptions move through the stack exactly the way you'd expect them to. Instead of having an object to represent each resource, you simply have a function with the code needed to allocate and deallocate it. They nest perfectly. ChrisA From ooomzay at gmail.com Mon Mar 5 09:25:18 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 06:25:18 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: <4865ad9c-2207-451b-9b3d-e30985d83fac@googlegroups.com> On Monday, 5 March 2018 13:59:35 UTC, Ooomzay wrote: > On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote: > > On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote: > > > Here is an example of a composite resource using RAII:- > > > > > > class RAIIFileAccess(): > > > def __init__(self, fname): > > > print("%s Opened" % fname) > > > def __del__(self): > > > print("%s Closed" % fname) > > > > > > class A(): > > > def __init__(self): > > > self.res = RAIIFileAccess("a") > > > > > > class B(): > > > def __init__(self): > > > self.res = RAIIFileAccess("b") > > > > > > class C(): > > > def __init__(self): > > > self.a = A() > > > self.b = B() > > > > > > def main(): > > > c = C() > > > > > > Under this PEP this is all that is needed to guarantee that the files "a" > > > and "b" are closed on exit from main or after any exception has been handled. > > > > Okay. And all your PEP needs is for reference count semantics, right? > > Okay. I'm going to run this in CPython, with reference semantics. You > > guarantee that those files will be closed after an exception is > > handled? Right. > > > > >>> def main(): > > ... c = C() > > ... c.do_stuff() > > ... > > >>> main() > > a Opened > > b Opened > > Traceback (most recent call last): > > File "", line 1, in > > File "", line 3, in main > > AttributeError: 'C' object has no attribute 'do_stuff' > > >>> > > > > Uhh.... I'm not seeing any messages about the files getting closed. > > Then that is indeed a challenge. From CPython back in 2.6 days up to Python36-32 what I see is:- > > a Opened > b Opened > Traceback (most recent call last): > ... > AttributeError: 'C' object has no attribute 'dostuff' > a Closed > b Closed > > > Maybe exceptions aren't as easy to handle as you think? > > Well there is a general issue with exceptions owing to the ease > with which one can create cycles that may catch out newbs. But > that is not the case here. > > > Or maybe you > > just haven't tried any of this (which is obvious from the bug in your > > code > > Or maybe I just made a typo when simplifying my test case and failed to retest? > > Here is my fixed case, if someone else could try it in CPython and report back that would be interesting:- > > class RAIIFileAccess(): > def __init__(self, fname): > print("%s Opened" % fname) > self.fname = fname > > def __del__(self): > print("%s Closed" % self.fname) > > class A(): > def __init__(self): > self.res = RAIIFileAccess("a") > > class B(): > def __init__(self): > self.res = RAIIFileAccess("b") > > class C(): > def __init__(self): > self.a = A() > self.b = B() > > def main(): > c = C() > c.dostuff() > > main() > > > You keep insisting that this is an easy thing. > We keep pointing out > > that it isn't. Now you're proving that you haven't even attempted any > > of this. > > Nonsense. But you have got a result I have never seen in many years > and I would like to get to the bottom of it. Ahah... I see now you are running it from a shell so the exception is staying in scope. We just need to include normal exception handling in the example to fix this:- class RAIIFileAccess(): def __init__(self, fname): print("%s Opened" % fname) self.fname = fname def __del__(self): print("%s Closed" % self.fname) class A(): def __init__(self): self.res = RAIIFileAccess("A") class B(): def __init__(self): self.res = RAIIFileAccess("B") class C(): def __init__(self): self.a = A() self.b = B() def main(): try: c = C() c.dostuff() except: print("Boom!") main() From rosuav at gmail.com Mon Mar 5 09:34:38 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 01:34:38 +1100 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: On Tue, Mar 6, 2018 at 12:52 AM, Terry Reedy wrote: > On 3/5/2018 7:12 AM, Kirill Balunov wrote: >> # 1. By passing through local variable's default values >> >> def func_local_1(numb, _int = int, _float = float, _range = range): > > > You are not required to mangle the names. > > def func_local_1(numb, int = int, float = float, range = range): > ... > Even so, this does mess up the function's signature, leaving your callers wondering if they can call it with some sort of range parameter. (Though in this particular instance, range() is only called once, so it's pretty much useless to try to optimize it.) In theory, the CPython bytecode compiler (don't know about other Python implementations) could just add these as constants. They'd then be bound at either compile time or function definition time (by default the former, I think, but the latter would be more useful), and be looked up as quickly as locals. I'm not sure how useful this would be, though. If PEP 572 [1] were to be accepted, you could do something like this: def func(numb): if ((int as int), (float as float)): res = [] for i in range(numb): res.append(int(i) + float(i)) return res Syntactically a bit clunky, but keeps everything inside the function, and DOES create local variables. Not sure it's better than your other options, but it is another option. [1] PEP 572: https://www.python.org/dev/peps/pep-0572/ ChrisA From rosuav at gmail.com Mon Mar 5 09:36:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 01:36:09 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <4865ad9c-2207-451b-9b3d-e30985d83fac@googlegroups.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> <4865ad9c-2207-451b-9b3d-e30985d83fac@googlegroups.com> Message-ID: On Tue, Mar 6, 2018 at 1:25 AM, Ooomzay wrote: > Ahah... I see now you are running it from a shell so the exception is staying in scope. We just need to include normal exception handling in the example to fix this:- > > def main(): > try: > c = C() > c.dostuff() > except: > print("Boom!") > > main() > So RAII depends on absorbing every exception close to where the resource is being managed? You can't permit that exception to bubble up lest the resource get leaked?? ChrisA From ooomzay at gmail.com Mon Mar 5 10:08:01 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 07:08:01 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> <4865ad9c-2207-451b-9b3d-e30985d83fac@googlegroups.com> Message-ID: <72df78f1-37b6-4759-a63a-23950dc8067d@googlegroups.com> On Monday, 5 March 2018 14:36:30 UTC, Chris Angelico wrote: > On Tue, Mar 6, 2018 at 1:25 AM, Ooomzay wrote: > > Ahah... I see now you are running it from a shell so the exception is staying in scope. We just need to include normal exception handling in the example to fix this:- > > > > def main(): > > try: > > c = C() > > c.dostuff() > > except: > > print("Boom!") > > > > main() > > > > So RAII depends on absorbing every exception close to where the > resource is being managed? You can't permit that exception to bubble > up lest the resource get leaked?? The exception can bubble up as many layers as you like but in python the exception leaks out of the handling context and needs its scope limiting. I have previously pointed out that such scoping is still recommended and that a function is pythons scoping construct - so use them for the job. From bc at freeuk.com Mon Mar 5 10:17:01 2018 From: bc at freeuk.com (bartc) Date: Mon, 5 Mar 2018 15:17:01 +0000 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> References: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: On 05/03/2018 13:58, Ooomzay wrote: > On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote: >> On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote: >>> Here is an example of a composite resource using RAII:- >>> >>> class RAIIFileAccess(): >>> def __init__(self, fname): >>> print("%s Opened" % fname) >>> def __del__(self): >>> print("%s Closed" % fname) >>> >>> class A(): >>> def __init__(self): >>> self.res = RAIIFileAccess("a") >>> >>> class B(): >>> def __init__(self): >>> self.res = RAIIFileAccess("b") >>> >>> class C(): >>> def __init__(self): >>> self.a = A() >>> self.b = B() >>> >>> def main(): >>> c = C() >>> >>> Under this PEP this is all that is needed to guarantee that the files "a" >>> and "b" are closed on exit from main or after any exception has been handled. >> >> Okay. And all your PEP needs is for reference count semantics, right? >> Okay. I'm going to run this in CPython, with reference semantics. You >> guarantee that those files will be closed after an exception is >> handled? Right. >> >>>>> def main(): >> ... c = C() >> ... c.do_stuff() >> ... >>>>> main() >> a Opened >> b Opened >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 3, in main >> AttributeError: 'C' object has no attribute 'do_stuff' >>>>> >> >> Uhh.... I'm not seeing any messages about the files getting closed. > > Then that is indeed a challenge. From CPython back in 2.6 days up to Python36-32 what I see is:- > > a Opened > b Opened > Traceback (most recent call last): > ... > AttributeError: 'C' object has no attribute 'dostuff' > a Closed > b Closed > >> Maybe exceptions aren't as easy to handle as you think? > > Well there is a general issue with exceptions owing to the ease > with which one can create cycles that may catch out newbs. But > that is not the case here. > >> Or maybe you >> just haven't tried any of this (which is obvious from the bug in your >> code > > Or maybe I just made a typo when simplifying my test case and failed to retest? > > Here is my fixed case, if someone else could try it in CPython and report back that would be interesting:- > > class RAIIFileAccess(): > def __init__(self, fname): > print("%s Opened" % fname) > self.fname = fname > > def __del__(self): > print("%s Closed" % self.fname) > > class A(): > def __init__(self): > self.res = RAIIFileAccess("a") > > class B(): > def __init__(self): > self.res = RAIIFileAccess("b") > > class C(): > def __init__(self): > self.a = A() > self.b = B() > > def main(): > c = C() > c.dostuff() > > main() I get A and B closed messages when running on CPython 2.7 and 3.6, with the code run from a .py file. (I never use interactive mode.) But not when running on a PyPy version of 2.7 (however that is not CPython). -- bartc From ooomzay at gmail.com Mon Mar 5 10:31:57 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 07:31:57 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: <72b95102-8e92-4d2b-bb8b-f15a81b044ff@googlegroups.com> On Monday, 5 March 2018 15:17:13 UTC, bartc wrote: > On 05/03/2018 13:58, Ooomzay wrote: > > On Monday, 5 March 2018 11:24:37 UTC, Chris Angelico wrote: > >> On Mon, Mar 5, 2018 at 10:09 PM, Ooomzay wrote: > >>> Here is an example of a composite resource using RAII:- > >>> > >>> class RAIIFileAccess(): > >>> def __init__(self, fname): > >>> print("%s Opened" % fname) > >>> def __del__(self): > >>> print("%s Closed" % fname) > >>> > >>> class A(): > >>> def __init__(self): > >>> self.res = RAIIFileAccess("a") > >>> > >>> class B(): > >>> def __init__(self): > >>> self.res = RAIIFileAccess("b") > >>> > >>> class C(): > >>> def __init__(self): > >>> self.a = A() > >>> self.b = B() > >>> > >>> def main(): > >>> c = C() > >>> > >>> Under this PEP this is all that is needed to guarantee that the files "a" > >>> and "b" are closed on exit from main or after any exception has been handled. > >> > >> Okay. And all your PEP needs is for reference count semantics, right? > >> Okay. I'm going to run this in CPython, with reference semantics. You > >> guarantee that those files will be closed after an exception is > >> handled? Right. > >> > >>>>> def main(): > >> ... c = C() > >> ... c.do_stuff() > >> ... > >>>>> main() > >> a Opened > >> b Opened > >> Traceback (most recent call last): > >> File "", line 1, in > >> File "", line 3, in main > >> AttributeError: 'C' object has no attribute 'do_stuff' > >>>>> > >> > >> Uhh.... I'm not seeing any messages about the files getting closed. > > > > Then that is indeed a challenge. From CPython back in 2.6 days up to Python36-32 what I see is:- > > > > a Opened > > b Opened > > Traceback (most recent call last): > > ... > > AttributeError: 'C' object has no attribute 'dostuff' > > a Closed > > b Closed > > > >> Maybe exceptions aren't as easy to handle as you think? > > > > Well there is a general issue with exceptions owing to the ease > > with which one can create cycles that may catch out newbs. But > > that is not the case here. > > > >> Or maybe you > >> just haven't tried any of this (which is obvious from the bug in your > >> code > > > > Or maybe I just made a typo when simplifying my test case and failed to retest? > > > > Here is my fixed case, if someone else could try it in CPython and report back that would be interesting:- > > > > class RAIIFileAccess(): > > def __init__(self, fname): > > print("%s Opened" % fname) > > self.fname = fname > > > > def __del__(self): > > print("%s Closed" % self.fname) > > > > class A(): > > def __init__(self): > > self.res = RAIIFileAccess("a") > > > > class B(): > > def __init__(self): > > self.res = RAIIFileAccess("b") > > > > class C(): > > def __init__(self): > > self.a = A() > > self.b = B() > > > > def main(): > > c = C() > > c.dostuff() > > > > main() > > I get A and B closed messages when running on CPython 2.7 and 3.6, with > the code run from a .py file. (I never use interactive mode.) > > But not when running on a PyPy version of 2.7 (however that is not CPython). Thanks bartc, I have made the example more complete by adding an exception scope - this means it works as designed - in any context. See my reply to Chris. We do not expect this to work in PyPy. From steve+comp.lang.python at pearwood.info Mon Mar 5 11:44:09 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 5 Mar 2018 16:44:09 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> Message-ID: On Sun, 04 Mar 2018 16:58:38 -0800, Ooomzay wrote: > Here is an example of a composite resource using RAII:- > > class RAIIFileAccess(): > def __init__(self, fname): > print("%s Opened" % fname) > def __del__(self): > print("%s Closed" % fname) > > class A(): > def __init__(self): > self.res = RAIIFileAccess("a") > > class B(): > def __init__(self): > self.res = RAIIFileAccess("b") > > class C(): > def __init__(self): > self.a = A() > self.b = B() > > def main(): > c = C() Looking at that code, my major thought is that there is far too much OO design, not enough simplicity. Perhaps I'm missing something, but I have no idea what benefit there is in that style of code over: with open('a') as a: with open('b') as b: process(a, b) So long as you only process a or b inside the nested block, you are guaranteed that they will be open. And unlike your RAII example, they will be closed when you exit, regardless of how many references to them you have, regardless of whether an exception occurs or not, regardless of whether there are cycles or whether they are globals or whether the interpreter is shutting down. I think that at this point, you have convinced me that you want to impose enormous costs on all Python interpreters *and* Python developers, in order to allow you to write C++ code in Python rather than learn Pythonic idioms like the with statement. I don't think this is a good tradeoff. -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 5 11:45:46 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 5 Mar 2018 16:45:46 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> <72b95102-8e92-4d2b-bb8b-f15a81b044ff@googlegroups.com> Message-ID: On Mon, 05 Mar 2018 07:31:57 -0800, Ooomzay wrote: > We do not expect this to work in PyPy. Or Jython, IronPython, possibly not Stackless either, or in the interactive interpreter of (probably) any implementation, or CPython if the object is garbage collected during interpreter shutdown or is in a cycle or has a global reference, or if the object simply doesn't go out of scope as quickly as you would like. -- Steve From ooomzay at gmail.com Mon Mar 5 12:22:33 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 09:22:33 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> Message-ID: <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> On Monday, 5 March 2018 16:47:02 UTC, Steven D'Aprano wrote: > On Sun, 04 Mar 2018 16:58:38 -0800, Ooomzay wrote: > > > Here is an example of a composite resource using RAII:- > > > > class RAIIFileAccess(): > > def __init__(self, fname): > > print("%s Opened" % fname) > > def __del__(self): > > print("%s Closed" % fname) > > > > class A(): > > def __init__(self): > > self.res = RAIIFileAccess("a") > > > > class B(): > > def __init__(self): > > self.res = RAIIFileAccess("b") > > > > class C(): > > def __init__(self): > > self.a = A() > > self.b = B() > > > > def main(): > > c = C() > > Looking at that code, my major thought is that there is far too much OO > design, not enough simplicity. If this is far too much OO for you then RAII will be of no interest to you. This example was specifically in response to a request to illustrate the relative simplicity of RAII in the case of a composite (OO) resource. > Perhaps I'm missing something, but I have no idea what benefit there is > in that style of code over: > > with open('a') as a: > with open('b') as b: > process(a, b) Encapsulation. Your application code is now managing details that should be hidden in the object. This PEP and RAII are unashamedly targeted at OO designs. If you would like to have a shot at coding this without RAII, but preserving the OO design, you will find that it is considerably _simpler_ than the with/context manager approach. > So long as you only process a or b inside the nested block, you are > guaranteed that they will be open. > > And unlike your RAII example, they will be closed when you exit, > regardless of how many references to them you have, regardless of whether > an exception occurs or not, regardless of whether there are cycles or > whether they are globals or whether the interpreter is shutting down. If you choose RAII you will not be cavalier with your references. > I think that at this point, you have convinced me that you want to impose > enormous costs on all Python interpreters *and* Python developers, in > order to allow you to write C++ code in Python rather than learn Pythonic > idioms like the with statement. On interpreters yes. On developers no. You can carry on exactly as you are. From ooomzay at gmail.com Mon Mar 5 12:53:50 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 09:53:50 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico wrote: > On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote: > > Here is my fixed example, if someone else could try it in CPython and report back that would be interesting:- > > > > class RAIIFileAccess(): > > def __init__(self, fname): > > print("%s Opened" % fname) > > self.fname = fname > > > > def __del__(self): > > print("%s Closed" % self.fname) > > > > class A(): > > def __init__(self): > > self.res = RAIIFileAccess("a") > > > > class B(): > > def __init__(self): > > self.res = RAIIFileAccess("b") > > > > class C(): > > def __init__(self): > > self.a = A() > > self.b = B() > > > > def main(): > > c = C() > > c.dostuff() > > > > main() > > Here's how I'd do it with context managers. > > from contextlib import contextmanager > > @contextmanager > def file_access(fname): > try: > print("%s Opened" % fname) > yield > finally: > print("%s Closed" % fname) > > @contextmanager > def c(): > try: > print("Starting c") > with file_access("a") as a, file_access("b") as b: > yield > finally: > print("Cleaning up c") > > def main(): > with c(): > dostuff() # NameError Thank you for having a go... However you have broken the encapsulation of class A and B. These are trivial for the sake of example. I should have used _underscores (i.e. self._res) to make the intent of this example more obvious. Please try again but preserving the integrity/encapsulation of class A & B & C, just as the RAII example does. From rosuav at gmail.com Mon Mar 5 12:58:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 04:58:20 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: On Tue, Mar 6, 2018 at 4:53 AM, Ooomzay wrote: > On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico wrote: >> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote: >> > Here is my fixed example, if someone else could try it in CPython and report back that would be interesting:- >> > >> > class RAIIFileAccess(): >> > def __init__(self, fname): >> > print("%s Opened" % fname) >> > self.fname = fname >> > >> > def __del__(self): >> > print("%s Closed" % self.fname) >> > >> > class A(): >> > def __init__(self): >> > self.res = RAIIFileAccess("a") >> > >> > class B(): >> > def __init__(self): >> > self.res = RAIIFileAccess("b") >> > >> > class C(): >> > def __init__(self): >> > self.a = A() >> > self.b = B() >> > >> > def main(): >> > c = C() >> > c.dostuff() >> > >> > main() >> >> Here's how I'd do it with context managers. >> >> from contextlib import contextmanager >> >> @contextmanager >> def file_access(fname): >> try: >> print("%s Opened" % fname) >> yield >> finally: >> print("%s Closed" % fname) >> >> @contextmanager >> def c(): >> try: >> print("Starting c") >> with file_access("a") as a, file_access("b") as b: >> yield >> finally: >> print("Cleaning up c") >> >> def main(): >> with c(): >> dostuff() # NameError > > > Thank you for having a go... > > However you have broken the encapsulation of class A and B. These > are trivial for the sake of example. I should have used > _underscores (i.e. self._res) to make the intent of > this example more obvious. > > Please try again but preserving the integrity/encapsulation > of class A & B & C, just as the RAII example does. What is B? Is it something that's notionally a resource to be managed? If so, you can trivially add another level to the nesting. ChrisA From tjreedy at udel.edu Mon Mar 5 13:44:17 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 5 Mar 2018 13:44:17 -0500 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: On 3/5/2018 9:34 AM, Chris Angelico wrote: > On Tue, Mar 6, 2018 at 12:52 AM, Terry Reedy wrote: >> On 3/5/2018 7:12 AM, Kirill Balunov wrote: >>> # 1. By passing through local variable's default values >>> >>> def func_local_1(numb, _int = int, _float = float, _range = range): >> >> >> You are not required to mangle the names. >> >> def func_local_1(numb, int = int, float = float, range = range): >> ... >> > > Even so, this does mess up the function's signature, Which I why I only said that using the original names solves the syntax highlighting issue (of marking built-ins as built-ins). > leaving your > callers wondering if they can call it with some sort of range > parameter. (Though in this particular instance, range() is only called > once, so it's pretty much useless to try to optimize it.) > > In theory, the CPython bytecode compiler (don't know about other > Python implementations) could just add these as constants. Yes, what we really want for this sort of thing are unrebindable local constants. A simple syntax change could do it. def func_local_1(numb; int = int, float = float, range = range): The binding after ';' belong in the header because they should be done once. > They'd then > be bound at either compile time or function definition time (by > default the former, I think, but the latter would be more useful), and > be looked up as quickly as locals. I'm not sure how useful this would > be, though. I believe that the occasional practice of re-binding built-in names to locals can be shown to speed up loops run enough times. > If PEP 572 [1] were to be accepted, you could do something like this: > > def func(numb): > if ((int as int), (float as float)): > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > > Syntactically a bit clunky, but keeps everything inside the function, > and DOES create local variables. Not sure it's better than your other > options, but it is another option. Code in the body should be executed everytime the function is called. Those binding should not be as they only need to be done once. > [1] PEP 572: https://www.python.org/dev/peps/pep-0572/ -- Terry Jan Reedy From Sebastien.Boisgerault at gmail.com Mon Mar 5 14:05:17 2018 From: Sebastien.Boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Mon, 5 Mar 2018 11:05:17 -0800 (PST) Subject: Bitstream -- Binary Data for Humans Message-ID: Hi everyone, I have released bitstream, a Python library to manage binary data (at the byte or bit level), hopefully without the pain that this kind of thing usually entails :) If you have struggled with this topic in the past, please take a look at the documentation (http://boisgera.github.io/bitstream/) and tell me what you think. Cheers, S?bastien From ooomzay at gmail.com Mon Mar 5 15:01:21 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 12:01:21 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <23f95047-766c-4a7b-b676-18190fc1bd31@googlegroups.com> <00ea7ffc-5dbf-48a5-a60b-35dc648bacee@googlegroups.com> Message-ID: <67446c1b-beb8-4214-8832-105c9058d137@googlegroups.com> On Monday, 5 March 2018 17:58:40 UTC, Chris Angelico wrote: > On Tue, Mar 6, 2018 at 4:53 AM, Ooomzay wrote: > > On Monday, 5 March 2018 14:21:54 UTC, Chris Angelico wrote: > >> On Tue, Mar 6, 2018 at 12:58 AM, Ooomzay wrote: > >> > Here is my fixed example, if someone else could try it in CPython and report back that would be interesting:- > >> > > >> > class RAIIFileAccess(): > >> > def __init__(self, fname): > >> > print("%s Opened" % fname) > >> > self.fname = fname > >> > > >> > def __del__(self): > >> > print("%s Closed" % self.fname) > >> > > >> > class A(): > >> > def __init__(self): > >> > self.res = RAIIFileAccess("a") > >> > > >> > class B(): > >> > def __init__(self): > >> > self.res = RAIIFileAccess("b") > >> > > >> > class C(): > >> > def __init__(self): > >> > self.a = A() > >> > self.b = B() > >> > > >> > def main(): > >> > c = C() > >> > c.dostuff() > >> > > >> > main() > >> > >> Here's how I'd do it with context managers. > >> > >> from contextlib import contextmanager > >> > >> @contextmanager > >> def file_access(fname): > >> try: > >> print("%s Opened" % fname) > >> yield > >> finally: > >> print("%s Closed" % fname) > >> > >> @contextmanager > >> def c(): > >> try: > >> print("Starting c") > >> with file_access("a") as a, file_access("b") as b: > >> yield > >> finally: > >> print("Cleaning up c") > >> > >> def main(): > >> with c(): > >> dostuff() # NameError > > > > > > Thank you for having a go... > > > > However you have broken the encapsulation of class A and B. These > > are trivial for the sake of example. I should have used > > _underscores (i.e. self._res) to make the intent of > > this example more obvious. > > > > Please try again but preserving the integrity/encapsulation > > of class A & B & C, just as the RAII example does. > > What is B? Is it something that's notionally a resource to be managed? Yes. For example a supply of electrical power controlled via a serial protocol to a programmable power supply - the file is a private detail used for the communications. And lets imagine that this powersupply object needs to keep track of some state such as the voltage - and it has a long lifetime - not just created then destroyed in scope of one function i.e. it is a substantial object. > If so, you can trivially add another level to the nesting. Please illustrate. I really do want to be able to compare like for like. From suabiut at gmail.com Mon Mar 5 15:02:53 2018 From: suabiut at gmail.com (sum abiut) Date: Tue, 6 Mar 2018 07:02:53 +1100 Subject: Bitstream -- Binary Data for Humans In-Reply-To: References: Message-ID: Thanks On 6/03/2018 7:13 AM, "S?bastien Boisg?rault" < Sebastien.Boisgerault at gmail.com> wrote: Hi everyone, I have released bitstream, a Python library to manage binary data (at the byte or bit level), hopefully without the pain that this kind of thing usually entails :) If you have struggled with this topic in the past, please take a look at the documentation (http://boisgera.github.io/bitstream/) and tell me what you think. Cheers, S?bastien -- https://mail.python.org/mailman/listinfo/python-list From ooomzay at gmail.com Mon Mar 5 16:28:03 2018 From: ooomzay at gmail.com (Ooomzay) Date: Mon, 5 Mar 2018 13:28:03 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <87muzmz66s.fsf@nightsong.com> References: <54a9fa91-3997-4d6e-a813-4ee29dc718b9@googlegroups.com> <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <3a1323ac-3029-4d27-a75c-98bcee6aba0b@googlegroups.com> <8b1d7b14-2a89-4678-a1a4-8803c1a26c96@googlegroups.com> <87muzmz66s.fsf@nightsong.com> Message-ID: <5b658b7e-e6b0-4c26-a7d3-1f3b73dfbe38@googlegroups.com> On Monday, 5 March 2018 19:14:05 UTC, Paul Rubin wrote: > Ooomzay writes: > > If you want to use RAII objects then you will make sure you avoid > > adding them to orphan cycles by design. If you don't know how to do > > that then don't write applications that manage critical resources. > > My claim is managing critical resources with refcounts is bug-prone in > the case where the refcounts can become arbitrarily large at runtime. What case is that? Don't go there! You may be jaded because python forgives bad design practices and peeps are inclined to leak resources all over the place for no good reason whatever and then complain when they have trouble cleaning up or exiting! > You say you wrote a program that worked that way, but it sounds > horrendous and I'd like to know how you tested it, maintained it, kept > it maintainable by other programmers, etc. > It's painful to even think about. I too used to suffer pain with python - until I saw that light and worked out how to use it for RAII. So here's the keys for much less pain:- * Use CPython (or C++ ;) * Use RAII for every resource-holding class: Implement __del__ to release any resources acquired in __init__. (This is significantly less effort, and more reliable than adding __enter__ & __exit__ to every class). * Wrap your try-except blocks in functions to prevent exceptions persisting outside the handler because in python they leak. This is typically a very natural factorization. * Take care not to create Cyclic Exceptions in your Exception handling logic. This was the one that had me scratching my head for a couple of hours the first time I inadvertantly created a cycle. * If you have no application-level requirement for persistent orphan cycles, and few, if any, applications do, then disable gc on entry and raise an exception on exit, or any other convenient moment if there is any garbage to be collected. * Immediately plug/bug any leaks that you discover. Do not let them build up or you will drown and loose faith that there can be a better way. From greg.ewing at canterbury.ac.nz Mon Mar 5 17:55:50 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Tue, 06 Mar 2018 11:55:50 +1300 Subject: "built-in" callables? In-Reply-To: References: Message-ID: Stefan Ram wrote: > . So, what does "built-in" in the sense of ?isbuiltin? > actually mean? It means "implemented in C". Yes, this is confusing. The term "built-in" is used in two different ways, and you just have to disambiguate them from context. -- Greg From steve+comp.lang.python at pearwood.info Mon Mar 5 18:04:02 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 5 Mar 2018 23:04:02 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> Message-ID: On Mon, 05 Mar 2018 09:22:33 -0800, Ooomzay wrote: [...] >> Looking at that code, my major thought is that there is far too much >> OO design, not enough simplicity. > > If this is far too much OO for you then RAII will be of no interest to > you. I think this is probably the wisest thing you have said in this entire discussion. OO design is often over-rated and elevated to the position of Holy Writ, full of over-engineered design patterns needed to overcome the flaws of OO design. I see no reason to design my application using OO principles when they aren't needed. You say you have had great success with the RAII pattern in your own Python code. Great! I mean that sincerely. I'm really happy for you. But you now want to force that on *everyone*, even onto implementers where this will be an enormous burden to re-write their interpreter from the ground up to add a reference counter. Why? Do you have even the tiniest interest in running your RAII application under IronPython or Jython? Why do you care that your code, using CPython- only features, must be portable to other implementations? If your PEP is accepted, will you be volunteering your time for the next two or four years to re-write Jython and IronPython? To say nothing of PyPy and possibly Stackless? CPython does what you want. So be satisfied that if you target CPython, you can use the RAII pattern to your heart's desire. [...] >> Perhaps I'm missing something, but I have no idea what benefit there is >> in that style of code over: >> >> with open('a') as a: >> with open('b') as b: >> process(a, b) > > Encapsulation. Your application code is now managing details that should > be hidden in the object. This PEP and RAII are unashamedly targeted at > OO designs. Encapsulation for the sake of encapsulation leads to monstrously over- engineered heights of abstraction. Without a concrete use-case, I have no confidence that this *should* be "hidden in the object". Even if it should, I have no confidence that this specific design for encapsulation and/or data hiding[1] (they aren't the same thing) is the best design. > If you would like to have a shot at coding this without RAII, but > preserving the OO design, you will find that it is considerably > _simpler_ than the with/context manager approach. Preserving the OO design, you say? Okay, since my application apparently isn't allowed to know that it is processing two files, I'll simply delegate that to the object: class C(A, B): def run(self): with open(self.a) as a: with open(self.b) as b: process(a, b) # Later. if __name__ = '__main__': # Enter the Kingdom of Nouns. c = C() c.run() There you go. Encapsulation and OO design. Look, I'm sure that we could go back and forth for weeks trading more and more complicated, convoluted, esoteric or just plain unlikely scenarios leading up to the conclusion you want. I'm even willing to accept for the sake of discussion that in *your specific application's case*, you have come up with the best possible solution. I'm not trying to dissuade you from using RAII in your own applications, if it works for you, great. But I think it is unjustified to try to force that on all implementers unless you have a real need, not just as a matter of principle, to use RAII while still insisting on your code being implementation independent. [...] > If you choose RAII you will not be cavalier with your references. You mean, "if you choose RAII, you cannot afford to be cavalier with your references, because if you fail to meet the rigorous demands of this pattern, your code will be buggy". CPython has a long history of people relying on RAII and ending up with buggy code that doesn't close resources in a timely manner. The reason the with statement was invented was to be a simple and effective alternative to the RAII pattern, which promises the world in theory and fails to deliver in practice when it runs up against the reality that most people *are* cavalier with their references. Not for you, obviously. I'm glad that it works for you. But as a community, we've been there and done that. [1] They aren't the same thing. -- Steve From roel at roelschroeven.net Mon Mar 5 18:28:45 2018 From: roel at roelschroeven.net (Roel Schroeven) Date: Tue, 06 Mar 2018 00:28:45 +0100 Subject: Bitstream -- Binary Data for Humans In-Reply-To: References: Message-ID: S?bastien Boisg?rault schreef op 5/03/2018 20:05: > I have released bitstream, a Python library to manage binary data (at the byte or bit level), > hopefully without the pain that this kind of thing usually entails :) > > If you have struggled with this topic in the past, please take a look at the documentation > (http://boisgera.github.io/bitstream/) and tell me what you think. Hi S?bastien, At work I have some Python code to decode AIS[1] messages, for which I created my own code for handling binary data. It works, but is pretty slow. Not surprising, since handling data at the bit level is not exactly Python's strength. If I find the time, I'll try to replace my code with your bitstream and see if it does what I need it to do, and if it's any faster. If/when I actually get around to it, I'll keep you informed. [1] https://en.wikipedia.org/wiki/Automatic_identification_system Best regards, Roel -- The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. -- Isaac Asimov Roel Schroeven From rosuav at gmail.com Mon Mar 5 18:33:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 10:33:48 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> Message-ID: On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano wrote: > # Later. > if __name__ = '__main__': > # Enter the Kingdom of Nouns. Don't you need a NounKingdomEnterer to do that for you? ChrisA From python at bladeshadow.org Mon Mar 5 18:53:20 2018 From: python at bladeshadow.org (Python) Date: Mon, 5 Mar 2018 17:53:20 -0600 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> Message-ID: <20180305235320.GN10273@bladeshadow.org> On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote: > > Python is often a preferred solution because it is often fantastic for > > rapid implementation and maintainability. The GIL's interference > > with threaded code performance has, for me at least, on several > > occasions been... disappointing (perf costs of removing it aside) > > because it gets in the way of choosing Python for such solutions. > > Jython and IronPython are simply not feasible options for me, for > > multiple reasons that have zero to do with their technical > > suitability. > > Have you actually tried it and run into problems, Yes. It was years ago and I forget the details, but I even posted some sample code here and was told (quite possibly by you) that it was the GIL that was eating my lunch. Someone suggested writing the bits I wanted to thread as a C extension, which largely defeated the purpose of using Python. In at least one case I just used C++, and in another I just ignored the problem until it went away. From drsalists at gmail.com Mon Mar 5 19:09:48 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 5 Mar 2018 16:09:48 -0800 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: <20180305235320.GN10273@bladeshadow.org> References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> <20180305235320.GN10273@bladeshadow.org> Message-ID: On Mon, Mar 5, 2018 at 3:53 PM, Python wrote: > On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote: >> > Python is often a preferred solution because it is often fantastic for >> > rapid implementation and maintainability. The GIL's interference >> > with threaded code performance has, for me at least, on several >> > occasions been... disappointing (perf costs of removing it aside) >> > because it gets in the way of choosing Python for such solutions. >> > Jython and IronPython are simply not feasible options for me, for >> > multiple reasons that have zero to do with their technical >> > suitability. >> >> Have you actually tried it and run into problems, > > Yes. It was years ago and I forget the details, but I even posted > some sample code here and was told (quite possibly by you) that it was > the GIL that was eating my lunch. Someone suggested writing the bits > I wanted to thread as a C extension, which largely defeated the > purpose of using Python. In at least one case I just used C++, and in > another I just ignored the problem until it went away. So how about a little Cython? It has decent GIL control, isn't much different from Python syntactically, and can be used to create C extension modules callable from CPython. It allows you to pretty freely intermix Python data types and C data types - just be careful about implicit conversions from one to the other - they can slow things down. From pengyu.ut at gmail.com Mon Mar 5 19:13:59 2018 From: pengyu.ut at gmail.com (Peng Yu) Date: Mon, 5 Mar 2018 18:13:59 -0600 Subject: How to access the help page of SRE_Pattern? Message-ID: Hi, >>> import re >>> prog=re.compile('[a-f]+') >>> help(prog) I can use the above command to access SRE_Pattern. But this involves the creation of an object of the class. I tried to directly access the class. But it does not work. Does anybody know if there is a way to directly access the class help page? Thanks. >>> help(_sre.SRE_Pattern) Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'SRE_Pattern' >>> help(SRE_Pattern) Traceback (most recent call last): File "", line 1, in NameError: name 'SRE_Pattern' is not defined -- Regards, Peng From steve+comp.lang.python at pearwood.info Mon Mar 5 20:01:42 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 6 Mar 2018 01:01:42 +0000 (UTC) Subject: How to access the help page of SRE_Pattern? References: Message-ID: On Mon, 05 Mar 2018 18:13:59 -0600, Peng Yu wrote: > Hi, > >>>> import re >>>> prog=re.compile('[a-f]+') >>>> help(prog) > > I can use the above command to access SRE_Pattern. But this involves the > creation of an object of the class. If you're using help() interactively, the cost of creating the instance is about a millionth of the cost of actually reading the help text. help(re.compile('')) should be perfectly acceptable, performance-wise, even though it is a tiny bit longer to write than: help(re) If you really want a reference to the SRE_Pattern class, I'm afraid it is not public name. It is technically an implementation detail subject to change without notice. It could change its name, its internal details, its location, so long as it still offers the public regular expression interface. I believe that SRE_Pattern is a built-in class, literally built into the interpreter. The best way to get access to it is to do this once, at the beginning of your code: SRE_Pattern = type(re.compile('')) If you are doing this in the interactive interpreter, you might want to include that in your Python startup file. -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 5 20:11:38 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 6 Mar 2018 01:11:38 +0000 (UTC) Subject: "except" and "subclasscheck" changed between CPython2 and 3 References: Message-ID: On Sat, 03 Mar 2018 04:28:24 +0000, ???? wrote: > Hello, > > I found a difference of behavior about `except` statement between > CPython 2.7 and 3.x . > `except EXC_CLASS:` calls `__subclasscheck__` in 2.7, but does not in > 3.x . Python 3 does not accept virtual subclasses for exception handling. They have to be concrete subclasses of BaseException. There is a feature-request to support that (as Python 2.7 does): https://bugs.python.org/issue12029 but it is stalled. -- Steve From sebastien.boisgerault at gmail.com Tue Mar 6 03:26:10 2018 From: sebastien.boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Tue, 6 Mar 2018 00:26:10 -0800 (PST) Subject: Bitstream -- Binary Data for Humans In-Reply-To: References: Message-ID: <46294d2e-5b09-49d9-a213-f87265444bc5@googlegroups.com> Le mardi 6 mars 2018 00:29:25 UTC+1, Roel Schroeven a ?crit?: > S?bastien Boisg?rault schreef op 5/03/2018 20:05: > > I have released bitstream, a Python library to manage binary data (at the byte or bit level), > > hopefully without the pain that this kind of thing usually entails :) > > > > If you have struggled with this topic in the past, please take a look at the documentation > > (http://boisgera.github.io/bitstream/) and tell me what you think. > > Hi S?bastien, > > At work I have some Python code to decode AIS[1] messages, for which I > created my own code for handling binary data. It works, but is pretty > slow. Not surprising, since handling data at the bit level is not > exactly Python's strength. If I find the time, I'll try to replace my > code with your bitstream and see if it does what I need it to do, and if > it's any faster. > > If/when I actually get around to it, I'll keep you informed. > > [1] https://en.wikipedia.org/wiki/Automatic_identification_system > > > Best regards, > Roel > > -- > The saddest aspect of life right now is that science gathers knowledge > faster than society gathers wisdom. > -- Isaac Asimov > > Roel Schroeven Great, thanks ! From kirillbalunov at gmail.com Tue Mar 6 03:52:22 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 11:52:22 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. Message-ID: This thought occurred to me several times, but I could not decide to write. And since `filter` is a builtin, I think this change should be discussed here, before opening an issue on bug tracker. I propose to delete all references in the `filter` documentation that the first argument can be `None`, with possible depreciation of `None` as the the first argument - FutureWarning in Python 3.8+ and deleting this option in Python 4. Personally, regarding the last point - depreciation, I do not see this as a great necessity, but I do not find that the option with `None` should be offered and suggested through the documentation. Instead, it is better to show an example with using `filter(bool, iterable)` which is absolutely equivalent, more readable, but a little bit slower. %timeit [*filter(None, range(10))] 503 ns ? 0.259 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) %timeit [*filter(bool, range(10))] 512 ns ? 1.09 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) Currently documentation for `None` case uses `identity function is assumed`, what is this `identity` and how it is consistent with truthfulness? In addition, this change makes the perception of `map` and `filter` more consistent,with the rule that first argument must be `callable`. I see only one moment with `None`, since `None` is a keyword, the behavior of `filter(None, iterable)` is alsways guaranteed, but with `bool` it is not. Nevertheless, we are all adults here. With kind regards, -gdg From sebastien.boisgerault at gmail.com Tue Mar 6 03:58:58 2018 From: sebastien.boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Tue, 6 Mar 2018 00:58:58 -0800 (PST) Subject: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited) In-Reply-To: <6010a25a-a3d8-43ee-bcd8-6beebcc5f90a@googlegroups.com> References: <6010a25a-a3d8-43ee-bcd8-6beebcc5f90a@googlegroups.com> Message-ID: <3d19cf80-3614-4a53-9142-a2c6f9c7ad8d@googlegroups.com> Hi Lawrence, Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D?Oliveiro a ?crit?: > On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, S?bastien Boisg?rault wrote: > > I have released bitstream, a Python library to manage binary data > > (at the byte or bit level), hopefully without the pain that this kind > > of thing usually entails :) > > > byte_index = offset / 8 > > This will return a float. The implementation is in Cython, which allows to declare types. The variable byte_index is defined as a size_t. Did I miss something? Do you mean that an intermediate float is used in the generated C code? I guess I should check that. I realize now that I sometimes use the code above to get the bit and bytes index and sometimes divmod ... > > Also I notice you count bit positions from the top of each byte, rather than from the bottom. Any reason for this? I suppose that my mental representation of the bitstream is left-to-right (think sequence of bits) with bits being "big-endian"ish in each byte, therefore new bits enter from the left with big weights. Which should translate to your description. So, no, no specific reason for this I guess. Is one of the representation better that the other (wrt performance for example)?. Cheers, SB From kirillbalunov at gmail.com Tue Mar 6 04:02:09 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 12:02:09 +0300 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: 2018-03-05 17:34 GMT+03:00 Chris Angelico : > In theory, the CPython bytecode compiler (don't know about other > Python implementations) could just add these as constants. They'd then > be bound at either compile time or function definition time (by > default the former, I think, but the latter would be more useful), and > be looked up as quickly as locals. I'm not sure how useful this would > be, though. > With some assumptions, It will be useful for every function call:-) If PEP 572 [1] were to be accepted, you could do something like this: > > def func(numb): > if ((int as int), (float as float)): > res = [] > for i in range(numb): > res.append(int(i) + float(i)) > return res > > Syntactically a bit clunky, but keeps everything inside the function, > and DOES create local variables. Not sure it's better than your other > options, but it is another option. > While I'm +0.5 on yours PEP 572 idea, especially in `while` and `if` statements, this example is an overuse of the proposed syntax ;-) Also it will add an overhead on every function call, and as you said - "Syntactically a bit clunky". With kind regards, -gdg From sebastien.boisgerault at gmail.com Tue Mar 6 04:23:05 2018 From: sebastien.boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Tue, 6 Mar 2018 01:23:05 -0800 (PST) Subject: Bitstream -- Binary Data for Humans In-Reply-To: <46294d2e-5b09-49d9-a213-f87265444bc5@googlegroups.com> References: <46294d2e-5b09-49d9-a213-f87265444bc5@googlegroups.com> Message-ID: <583124ef-26d2-4370-b5dc-571a38ebfcb0@googlegroups.com> Le mardi 6 mars 2018 09:26:50 UTC+1, S?bastien Boisg?rault a ?crit?: > Le mardi 6 mars 2018 00:29:25 UTC+1, Roel Schroeven a ?crit?: > > S?bastien Boisg?rault schreef op 5/03/2018 20:05: > > > I have released bitstream, a Python library to manage binary data (at the byte or bit level), > > > hopefully without the pain that this kind of thing usually entails :) > > > > > > If you have struggled with this topic in the past, please take a look at the documentation > > > (http://boisgera.github.io/bitstream/) and tell me what you think. > > > > Hi S?bastien, > > > > At work I have some Python code to decode AIS[1] messages, for which I > > created my own code for handling binary data. It works, but is pretty > > slow. Not surprising, since handling data at the bit level is not > > exactly Python's strength. If I find the time, I'll try to replace my > > code with your bitstream and see if it does what I need it to do, and if > > it's any faster. > > > > If/when I actually get around to it, I'll keep you informed. > > > > [1] https://en.wikipedia.org/wiki/Automatic_identification_system > > > > > > Best regards, > > Roel > > > > -- > > The saddest aspect of life right now is that science gathers knowledge > > faster than society gathers wisdom. > > -- Isaac Asimov > > > > Roel Schroeven > > Great, thanks ! Hi again Roel, I had a look at the AIS message format from your link (https://en.wikipedia.org/wiki/Automatic_identification_system#Message_format) and this seems to be a nice use case; all the data components seem to be nicely aligned on the byte boundary ... until you see that the payload uses ASCII6(*), which I didn't know about. Thanks again for the info! Cheers, (*) ASCII6 code = ASCII code + 48; (http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_sentence_layer) SB From wolfgang.maier at biologie.uni-freiburg.de Tue Mar 6 04:35:08 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 6 Mar 2018 10:35:08 +0100 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: On 03/05/2018 07:44 PM, Terry Reedy wrote: > On 3/5/2018 9:34 AM, Chris Angelico wrote: >> On Tue, Mar 6, 2018 at 12:52 AM, Terry Reedy wrote: >>> On 3/5/2018 7:12 AM, Kirill Balunov wrote: >>>> # 1. By passing through local variable's default values >>>> >>>> def func_local_1(numb, _int = int, _float = float, _range = range): >>> >>> >>> You are not required to mangle the names. >>> >>> def func_local_1(numb, int = int, float = float, range = range): >>> ... >>> >> >> Even so, this does mess up the function's signature, > > Which I why I only said that using the original names solves the syntax > highlighting issue (of marking built-ins as built-ins). > >> leaving your >> callers wondering if they can call it with some sort of range >> parameter. (Though in this particular instance, range() is only called >> once, so it's pretty much useless to try to optimize it.) >> >> In theory, the CPython bytecode compiler (don't know about other >> Python implementations) could just add these as constants. > > Yes, what we really want for this sort of thing are unrebindable local > constants. A simple syntax change could do it. > > def func_local_1(numb; int = int, float = float, range = range): > > The binding after ';' belong in the header because they should be done once. > Ah, I did not really understand initially what Kirill was trying to achieve by putting the name binding into the function signature. Now I do, but I don't think it is a good idea. Sanctioning this with dedicated syntax would only make Python more static because for any function defined this way, you would lose the ability to alter the behavior of that function through changing the global binding after the function has been called (in the example above, you could no longer mock replace int, float and range on subsequent func_local_1 calls) and I don't think this is something that should be encouraged. Wolfgang From sebastien.boisgerault at gmail.com Tue Mar 6 04:51:05 2018 From: sebastien.boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Tue, 6 Mar 2018 01:51:05 -0800 (PST) Subject: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited) In-Reply-To: <9613f24b-2ed1-450b-af50-51a0ca14fa29@googlegroups.com> References: <6010a25a-a3d8-43ee-bcd8-6beebcc5f90a@googlegroups.com> <3d19cf80-3614-4a53-9142-a2c6f9c7ad8d@googlegroups.com> <9613f24b-2ed1-450b-af50-51a0ca14fa29@googlegroups.com> Message-ID: Le mardi 6 mars 2018 10:23:02 UTC+1, Lawrence D?Oliveiro a ?crit?: > On Tuesday, March 6, 2018 at 9:59:55 PM UTC+13, S?bastien Boisg?rault wrote: > > > > Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D?Oliveiro a ?crit?: > > > >> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, S?bastien Boisg?rault wrote: > >>> > >>> byte_index = offset / 8 > >> > >> This will return a float. > > > > The implementation is in Cython, which allows to declare types. > > The variable byte_index is defined as a size_t. > > Ah, I see. Still it seems unPythonic to use ?/? for integer division. Does it not allow ?//?? Yes, '//' works, see: https://mybinder.org/v2/gh/boisgera/jupyter-cython/master?filepath=Integer%20Division.ipynb Actually, in this Jupyter notebook setting, this is '/' that doesn't work! I should have a new look at this; since performance also matters, I'd also like to have the Cython code with the smallest overhead (even if it is less readable/Pythonic). Cheers, SB From kirillbalunov at gmail.com Tue Mar 6 04:57:49 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 12:57:49 +0300 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: 2018-03-05 21:44 GMT+03:00 Terry Reedy : > Yes, what we really want for this sort of thing are unrebindable local > constants. A simple syntax change could do it. > > def func_local_1(numb; int = int, float = float, range = range): > > The binding after ';' belong in the header because they should be done > once. > > They'd then >> be bound at either compile time or function definition time (by >> default the former, I think, but the latter would be more useful), and >> be looked up as quickly as locals. I'm not sure how useful this would >> be, though. >> > > I believe that the occasional practice of re-binding built-in names to > locals can be shown to speed up loops run enough times. Yes "_unrebindable local constants_" it is what I was thinking about. But I do not agree that they must be passed through arguments, because they should be somewhat static for a function, and could not be changed by any means after function is compiled. Alternative option, more dynamic - to allow injecting local variables into the function via some interface. Currently, there is no such _feature_ in Python, at least I do not know. There was _somewhat_ related discussion about how to change the locals of a frame (https://bugs.python.org/ issue1654367) by making `frame.f_locals` writable, but it seems that it is dead. With kind regards, -gdg From rosuav at gmail.com Tue Mar 6 05:13:39 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 21:13:39 +1100 Subject: Ways to make a free variable local to a function? In-Reply-To: References: Message-ID: On Tue, Mar 6, 2018 at 8:02 PM, Kirill Balunov wrote: > > > 2018-03-05 17:34 GMT+03:00 Chris Angelico : >> >> In theory, the CPython bytecode compiler (don't know about other >> Python implementations) could just add these as constants. They'd then >> be bound at either compile time or function definition time (by >> default the former, I think, but the latter would be more useful), and >> be looked up as quickly as locals. I'm not sure how useful this would >> be, though. > > > With some assumptions, It will be useful for every function call:-) > >> If PEP 572 [1] were to be accepted, you could do something like this: >> >> def func(numb): >> if ((int as int), (float as float)): >> res = [] >> for i in range(numb): >> res.append(int(i) + float(i)) >> return res >> >> Syntactically a bit clunky, but keeps everything inside the function, >> and DOES create local variables. Not sure it's better than your other >> options, but it is another option. > > > While I'm +0.5 on yours PEP 572 idea, especially in `while` and `if` > statements, this example is an overuse of the proposed syntax ;-) Also it > will add an overhead on every function call, and as you said - > "Syntactically a bit clunky". > The run-time overhead should be insignificant; this kind of optimization is done when you're running a tight loop, so it's the run time of the loop body that dominates the function. That's also why I do NOT want this to happen at compile time, even though that would be the easiest. The very latest this should happen is function definition time; it would be extremely surprising otherwise. And if it happens once when the function's called, that's usually not going to be much cost compared to the saving of LOAD_FAST instead of LOAD_GLOBAL in each iteration of the loop. ChrisA From tjreedy at udel.edu Tue Mar 6 05:14:41 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2018 05:14:41 -0500 Subject: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited) In-Reply-To: <3d19cf80-3614-4a53-9142-a2c6f9c7ad8d@googlegroups.com> References: <6010a25a-a3d8-43ee-bcd8-6beebcc5f90a@googlegroups.com> <3d19cf80-3614-4a53-9142-a2c6f9c7ad8d@googlegroups.com> Message-ID: On 3/6/2018 3:58 AM, S?bastien Boisg?rault wrote: > Hi Lawrence, > > Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D?Oliveiro a ?crit?: >> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, S?bastien Boisg?rault wrote: >>> I have released bitstream, a Python library to manage binary data >>> (at the byte or bit level), hopefully without the pain that this kind >>> of thing usually entails :) >> >>> byte_index = offset / 8 >> >> This will return a float. byte_index // 8 will give you the int index directly in both late 2.x and 3.x. -- Terry Jan Redey From rosuav at gmail.com Tue Mar 6 05:18:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 6 Mar 2018 21:18:18 +1100 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Tue, Mar 6, 2018 at 7:52 PM, Kirill Balunov wrote: > This thought occurred to me several times, but I could not decide to write. > And since `filter` is a builtin, I think this change should be discussed > here, before opening an issue on bug tracker. > > I propose to delete all references in the `filter` documentation that the > first argument can be `None`, with possible depreciation of `None` as the > the first argument - FutureWarning in Python 3.8+ and deleting this option > in Python 4. Personally, regarding the last point - depreciation, I do not > see this as a great necessity, but I do not find that the option with `None` > should be offered and suggested through the documentation. Instead, it is > better to show an example with using `filter(bool, iterable)` which is > absolutely > equivalent, more readable, but a little bit slower. > > %timeit [*filter(None, range(10))] > 503 ns ? 0.259 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) > > %timeit [*filter(bool, range(10))] > 512 ns ? 1.09 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) > > Currently documentation for `None` case uses `identity function is > assumed`, what is this `identity` and how it is consistent with > truthfulness? The identity function is: filter(lambda x: x, range(10)) How is it consistent with truthiness? Exactly the same way the underlying object is. There's no requirement for the predicate function to return True or False - it's perfectly acceptable, for instance, to do this: filter(lambda x: x % 3, range(10)) to eliminate all multiples of three. That said, though, any use of filter() that involves a lambda function should probably become list comps or genexps, so filter itself should only be used when there really IS a pre-existing function that does the job. So, for instance, you could strip out every occurrence of the string "0" with: filter(int, list_of_strings) And that still depends on the normal Python rules for boolification. If that's valid, then it should be just as viable to say "filter(identity-function, ...)", which is spelled "filter(None, ...)". ChrisA From sebastien.boisgerault at gmail.com Tue Mar 6 05:37:34 2018 From: sebastien.boisgerault at gmail.com (=?UTF-8?Q?S=C3=A9bastien_Boisg=C3=A9rault?=) Date: Tue, 6 Mar 2018 02:37:34 -0800 (PST) Subject: Bitstream -- Binary Data for Humans (Posting On Python-List Prohibited) In-Reply-To: References: <6010a25a-a3d8-43ee-bcd8-6beebcc5f90a@googlegroups.com> <3d19cf80-3614-4a53-9142-a2c6f9c7ad8d@googlegroups.com> Message-ID: <9edb19cf-0a1f-400f-9ac0-a8b2d9ad3d56@googlegroups.com> Le mardi 6 mars 2018 11:15:15 UTC+1, Terry Reedy a ?crit?: > On 3/6/2018 3:58 AM, S?bastien Boisg?rault wrote: > > Hi Lawrence, > > > > Le mardi 6 mars 2018 01:20:36 UTC+1, Lawrence D?Oliveiro a ?crit?: > >> On Tuesday, March 6, 2018 at 8:06:00 AM UTC+13, S?bastien Boisg?rault wrote: > >>> I have released bitstream, a Python library to manage binary data > >>> (at the byte or bit level), hopefully without the pain that this kind > >>> of thing usually entails :) > >> > >>> byte_index = offset / 8 > >> > >> This will return a float. > > byte_index // 8 > will give you the int index directly in both late 2.x and 3.x. Indeed! And since this is Cython code, I *guess* that I should combine `//` with the 'cdivision' set to True to get the syntax that everyone understands *and* the pure C speed (see http://cython.readthedocs.io/en/latest/src/reference/compilation.html). I need to run some experiments to make sure that this behaves as expected, but this is very likely the way to go. > -- > Terry Jan Redey From steve+comp.lang.python at pearwood.info Tue Mar 6 06:17:44 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 6 Mar 2018 11:17:44 +0000 (UTC) Subject: Do not promote `None` as the first argument to `filter` in documentation. References: Message-ID: On Tue, 06 Mar 2018 11:52:22 +0300, Kirill Balunov wrote: > I propose to delete all references in the `filter` documentation that > the first argument can be `None`, with possible depreciation of `None` > as the the first argument - FutureWarning in Python 3.8+ and deleting > this option in Python 4. Even if we agreed that it is unfortunate that filter accepts None as an argument, since it does (and has done since Python 1.0) there is nothing to be gained by deprecating and removing it. Deprecating and removing it will break code that currently works, for no good reason; removing the documentation is unacceptable, as that makes it too difficult for people to find out what `filter(None, values)` does. > Instead, it is better to show an example with using > `filter(bool, iterable)` which is absolutely > equivalent, more readable, but a little bit slower. So long as `filter(None, ...)` is still documented, I don't mind what example is given. But the idiom `filter(None, ...)` is an old, common idiom, very familiar to many people who have a background in functional programming. It is unfortunate that filter takes the arguments in the order it does. Perhaps it would have been better to write it like this: def filter(iterable, predicate=None): ... Then `filter(values, None)` would be a standard Python idiom, explicitly saying to use the default predicate function. There is no difference to `filter(None, values)` except the order is (sadly) reversed. > Currently documentation for `None` case uses `identity function is > assumed`, what is this `identity` and how it is consistent with > truthfulness? The identity function is a mathematical term for a function that returns its argument unchanged: def identity(x): return x So `filter(func, values)` filters according to func(x); using None instead filters according to x alone, without the expense of calling a do- nothing function: # slow because it has to call the lambda function each time; filter(lambda x: x, values) # fast because filter takes an optimized path filter(None, values) Since filter filters according to the truthy or falsey value of x, it isn't actually necessary to call bool(x). In Python, all values are automatically considered either truthy or falsey. The reason to call bool() is to ensure you have a canonical True/False value, and there's no need for that here. So the identity function should be preferred to bool, for those who understand two things: - the identity function (using None as the predicate function) returns x unchanged; - and that x, like all values, automatically has a truthy value in a boolean context (which includes filter). > In addition, this change makes the perception of `map` and `filter` more > consistent,with the rule that first argument must be `callable`. I consider that a flaw in map. map should also accept None as the identity function, so that map(None, iterable) returns the values of iterable unchanged. def map(function=None, *iterables): if len(iterables) == 0: raise TypeError("map() must have at least two arguments.") if function is None: if len(iterables) > 1: return zip(*iterables) else: assert len(iterables) == 1 return iter(iterables[0]) elif len(iterables) > 1: return (function(*args) for args in zip(*iterables)) else: assert len(iterables) == 1 return (function(arg) for arg in iterables[0]) -- Steve From kirillbalunov at gmail.com Tue Mar 6 08:08:53 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 16:08:53 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 14:17 GMT+03:00 Steven D'Aprano < steve+comp.lang.python at pearwood.info>: > On Tue, 06 Mar 2018 11:52:22 +0300, Kirill Balunov wrote: > > > I propose to delete all references in the `filter` documentation that > > the first argument can be `None`, with possible depreciation of `None` > > as the the first argument - FutureWarning in Python 3.8+ and deleting > > this option in Python 4. > > Even if we agreed that it is unfortunate that filter accepts None as an > argument, since it does (and has done since Python 1.0) there is nothing > to be gained by deprecating and removing it. > > Deprecating and removing it will break code that currently works, for no > good reason; removing the documentation is unacceptable, as that makes it > too difficult for people to find out what `filter(None, values)` does. > As I wrote, __possible depreciation__, I also do not see the point of just breaking someone's code. But I didn't see any benefit to explicitly promote `filter(None, iterable)` form in the documentation as a good style. > > Instead, it is better to show an example with using > > `filter(bool, iterable)` which is absolutely > > equivalent, more readable, but a little bit slower. > > So long as `filter(None, ...)` is still documented, I don't mind what > example is given. > > But the idiom `filter(None, ...)` is an old, common idiom, very familiar > to many people who have a background in functional programming. > While this form familiar and common idiom for those who are familiar with Python from versions < 2.3, before `bool` type was introduced. It looks kinky for newcomers and not obvious at a glance. In functional programming we use a predicate, and `None` does not match predicate definition, while `bool` does! > It is unfortunate that filter takes the arguments in the order it does. > Perhaps it would have been better to write it like this: > > def filter(iterable, predicate=None): > ... > > > Then `filter(values, None)` would be a standard Python idiom, explicitly > saying to use the default predicate function. There is no difference to > `filter(None, values)` except the order is (sadly) reversed. > If such a form was in Python, I probably would agree with you. Although in its present form I like it a lot more and find it more intuitive. > Currently documentation for `None` case uses `identity function is > > assumed`, what is this `identity` and how it is consistent with > > truthfulness? > > The identity function is a mathematical term for a function that returns > its argument unchanged: > > def identity(x): > return x > > So `filter(func, values)` filters according to func(x); using None > instead filters according to x alone, without the expense of calling a do- > nothing function: > > # slow because it has to call the lambda function each time; > filter(lambda x: x, values) > > # fast because filter takes an optimized path > filter(None, values) > > Since filter filters according to the truthy or falsey value of x, it > isn't actually necessary to call bool(x). In Python, all values are > automatically considered either truthy or falsey. The reason to call > bool() is to ensure you have a canonical True/False value, and there's no > need for that here. I went over a bit with the question what is identity function :) But I have a feeling that I perceive all of the above quite the contrary in the context of a `filter` function. And since filter filters according to the truthy or falsey value of x. `None` and `bool` should behave totally equivalent under the hood and I'm 99% sure that it is so. > So the identity function should be preferred to bool, > for those who understand two things: > > - the identity function (using None as the predicate function) > returns x unchanged; > Sorry, but how does the above relates to the `filter` discussion? > > - and that x, like all values, automatically has a truthy value in a > boolean context (which includes filter). > > Yes, and that is why there is no point to `None` since they will do the same thing in context of `filter` function. > > In addition, this change makes the perception of `map` and `filter` more > > consistent,with the rule that first argument must be `callable`. > > I consider that a flaw in map. map should also accept None as the > identity function, so that map(None, iterable) returns the values of > iterable unchanged. > > def map(function=None, *iterables): > if len(iterables) == 0: > raise TypeError("map() must have at least two arguments.") > if function is None: > if len(iterables) > 1: > return zip(*iterables) > else: > assert len(iterables) == 1 > return iter(iterables[0]) > elif len(iterables) > 1: > return (function(*args) for args in zip(*iterables)) > else: > assert len(iterables) == 1 > return (function(arg) for arg in iterables[0]) > And what will be the practical reason to have this? :) With kind regards, -gdg From kirillbalunov at gmail.com Tue Mar 6 08:23:56 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 16:23:56 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 13:18 GMT+03:00 Chris Angelico : > The identity function is: > > filter(lambda x: x, range(10)) > > How is it consistent with truthiness? Exactly the same way the > underlying object is. There's no requirement for the predicate > function to return True or False - it's perfectly acceptable, for > instance, to do this: > > filter(lambda x: x % 3, range(10)) > > to eliminate all multiples of three. > Yes there is no reason to return True and False, but in the case of `None` and `bool` under the hood there will be no difference and the form with `bool` is much more readable. > > That said, though, any use of filter() that involves a lambda function > should probably become list comps or genexps, so filter itself should > only be used when there really IS a pre-existing function that does > the job. Filter is generally faster than list comprehension or generators. %timeit [*filter(lambda x: x % 3, range(1000))] 100 ?s ? 16.4 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) f = lambda x: x % 3 %timeit [*(f(i) for i in range(1000))] 132 ?s ? 73.5 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) %timeit [f(i) for i in range(1000)] 107 ?s ? 179 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > So, for instance, you could strip out every occurrence of the > string "0" with: > > filter(int, list_of_strings) > > And that still depends on the normal Python rules for boolification. > If that's valid, then it should be just as viable to say > "filter(identity-function, ...)", which is spelled "filter(None, > ...)". > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > From cl at isbd.net Tue Mar 6 08:35:45 2018 From: cl at isbd.net (Chris Green) Date: Tue, 6 Mar 2018 13:35:45 +0000 Subject: Do not promote `None` as the first argument to `filter` in documentation. References: Message-ID: Kirill Balunov wrote: > > As I wrote, __possible depreciation__, I also do not see the point of just It's 'deprecation', depreciation is something quite different. People replying have spelt it correctly so you might possibly have noticed I thought/hoped. ... and it does matter a bit because it's not just a mis-spelling, the word you are using has its own meaning and could thus cause confusion. ... and, yes, I know it's a very common and easily made mistake. :-) -- Chris Green ? From __peter__ at web.de Tue Mar 6 08:39:50 2018 From: __peter__ at web.de (Peter Otten) Date: Tue, 06 Mar 2018 14:39:50 +0100 Subject: Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: Message-ID: <2873318.k5KsoXlf1h@palindrom> Chris Angelico wrote: > On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano > wrote: >> # Later. >> if __name__ = '__main__': >> # Enter the Kingdom of Nouns. > > Don't you need a NounKingdomEnterer to do that for you? No, for some extra flexibility there should be a NounKingdomEntererFactory -- which of course has to implement the AbstractNounKingdomEntererFactory interface. Instantiating the right NounKingdomEntererFactory can easily be delegated to a NounKingdomEntererFactoryProducer. It's turtles^Wfactories all the way down... I don't know how this will ever do anything -- my theory is that some good soul managed to sneak a few lines of javascript into the giant xml configuration file ;) From rosuav at gmail.com Tue Mar 6 08:51:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Mar 2018 00:51:41 +1100 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Wed, Mar 7, 2018 at 12:23 AM, Kirill Balunov wrote: > Filter is generally faster than list comprehension or generators. > > %timeit [*filter(lambda x: x % 3, range(1000))] > 100 ?s ? 16.4 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > f = lambda x: x % 3 > > %timeit [*(f(i) for i in range(1000))] > 132 ?s ? 73.5 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > %timeit [f(i) for i in range(1000)] > 107 ?s ? 179 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > These don't do the same thing, though. A more comparable comprehension is: [i for i in range(1000) if i % 3] rosuav at sikorsky:~$ python3 -m timeit '[i for i in range(1000) if i % 3]' 10000 loops, best of 5: 34.5 usec per loop rosuav at sikorsky:~$ python3 -m timeit '[*filter(lambda x: x % 3, range(1000))]' 5000 loops, best of 5: 81.1 usec per loop And my point about comprehensions was that you do NOT use a pointless function for them - you just have inline code. If there is a pre-existing function, sure! Use it. But when you use filter or map with a lambda function, you should probably use a comprehension instead. ChrisA From jsf80238 at gmail.com Tue Mar 6 08:58:59 2018 From: jsf80238 at gmail.com (Jason Friedman) Date: Tue, 6 Mar 2018 06:58:59 -0700 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Tue, Mar 6, 2018 at 1:52 AM, Kirill Balunov wrote: > > I propose to delete all references in the `filter` documentation that the > first argument can be `None`, with possible depreciation of `None` as the > the first argument - FutureWarning in Python 3.8+ and deleting this option > in Python 4. Personally, regarding the last point - depreciation, I do not > see this as a great necessity, but I do not find that the option with > `None` > should be offered and suggested through the documentation. Instead, it is > better to show an example with using `filter(bool, iterable)` which is > absolutely > equivalent, more readable, but a little bit slower. > > Currently documentation for `None` case uses `identity function is > assumed`, what is this `identity` and how it is consistent with > truthfulness? > > In addition, this change makes the perception of `map` and `filter` more > consistent,with the rule that first argument must be `callable`. > > I see only one moment with `None`, since `None` is a keyword, the behavior > of `filter(None, iterable)` is alsways guaranteed, but with `bool` it is > not. Nevertheless, we are all adults here. > I won't pretend I am qualified to debate the technical aspects here, but regarding this snippet: ... Instead, it is better to show an example with using `filter(bool, iterable)` which is absolutely equivalent, more readable ... as a ordinary Python user I'd be interested in improvements to the documentation, including suggestions on real-world usage. For example, Chris Angelico below says in part: ... that said, though, any use of filter() that involves a lambda function should probably become list comps or genexps, so filter itself should only be used when ... Kirill, taking deprecation/removal off the table, what changes would you recommend to the documentation? From kirillbalunov at gmail.com Tue Mar 6 09:03:53 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 17:03:53 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 16:51 GMT+03:00 Chris Angelico : > On Wed, Mar 7, 2018 at 12:23 AM, Kirill Balunov > wrote: > > Filter is generally faster than list comprehension or generators. > > > > %timeit [*filter(lambda x: x % 3, range(1000))] > > 100 ?s ? 16.4 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > > > f = lambda x: x % 3 > > > > %timeit [*(f(i) for i in range(1000))] > > 132 ?s ? 73.5 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > > > %timeit [f(i) for i in range(1000)] > > 107 ?s ? 179 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) > > > > These don't do the same thing, though. A more comparable comprehension is: > > [i for i in range(1000) if i % 3] > > rosuav at sikorsky:~$ python3 -m timeit '[i for i in range(1000) if i % 3]' > 10000 loops, best of 5: 34.5 usec per loop > rosuav at sikorsky:~$ python3 -m timeit '[*filter(lambda x: x % 3, > range(1000))]' > 5000 loops, best of 5: 81.1 usec per loop > > And my point about comprehensions was that you do NOT use a pointless > function for them - you just have inline code. If there is a > pre-existing function, sure! Use it. But when you use filter or map > with a lambda function, you should probably use a comprehension > instead. > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > Thank you, I did not understand you at first, now everything is clear. In this sense of `x % 3`, I fully agree with you. With kind regards, -gdg From kirillbalunov at gmail.com Tue Mar 6 09:07:32 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 17:07:32 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 16:35 GMT+03:00 Chris Green : > It's 'deprecation', depreciation is something quite different. People > replying have spelt it correctly so you might possibly have noticed I > thought/hoped. > > ... and it does matter a bit because it's not just a mis-spelling, the > word you are using has its own meaning and could thus cause confusion. > > ... and, yes, I know it's a very common and easily made mistake. :-) > > I did not ;) Thank you! With kind regards, -gdg From youta.t at gmail.com Tue Mar 6 09:24:52 2018 From: youta.t at gmail.com (Youta TAKAOKA) Date: Tue, 06 Mar 2018 14:24:52 +0000 Subject: "except" and "subclasscheck" changed between CPython2 and 3 In-Reply-To: References: Message-ID: Thank you, Steven. > There is a feature-request to support that (as Python 2.7 does): > > https://bugs.python.org/issue12029 > > but it is stalled. I passed over the ticket. Now, I know that this is a bug, but has not fixed yet. There are (or ware ?) problems about performance and integrity for handling exceptions. It seems that a narrow range of use-cases cause the stalling. Someday we have a type-focused VERY NICE library, (I hope that) it will be resolved. Thanks. -- youta.t at gmail.com From kirillbalunov at gmail.com Tue Mar 6 09:48:19 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 17:48:19 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 16:58 GMT+03:00 Jason Friedman : > > as a ordinary Python user I'd be interested in improvements to the > documentation, including suggestions on real-world usage. > I'm just an ordinary user, just like you :) > Kirill, taking deprecation/removal off the table, what changes would you > recommend to the documentation? > My English is about basic level, so you may need to fix the spelling. But I would write as follows: filter(function, iterable) Construct an iterator from those elements of iterable for which function returns truthy values. iterable may be either a sequence, a container which supports iteration, or an iterator. Note that filter(function, iterable) is equivalent to the generator expression (item for item in iterable if function(item)). In cases when function corresponds to a simple lambda function a generator expression should be preferred. For example, when it is necessary to eliminate all multiples of three (x for x in range(100) if x % 3) should be used, instead of filter(lambda x: x % 3, range(100)) See itertools.filterfalse() for the complementary function that returns elements of iterable for which function returns false. Note: For some historical reasons as the first argument you can use None instead of function, in this case the identity function is assumed. That is, all elements of iterable that are false are removed which is equivalent to (item for item in iterable if item). Currently, for the same purpose the preferred form is `filter(bool, iterable)`. p.s.: maybe _function_ should be changed to _callable_. With kind regards, -gdg From rosuav at gmail.com Tue Mar 6 09:55:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Mar 2018 01:55:09 +1100 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov wrote: > Note: For some historical reasons as the first argument you can use None > instead of function, in this case the identity function is assumed. That > is, all elements of iterable that are false are removed which is equivalent > to (item for item in iterable if item). Currently, for the same purpose the > preferred form is `filter(bool, iterable)`. > I'd prefer to word it something like: If the first argument is None, the identity function is assumed. That is, all elements of the iterable that are false are removed; it is equivalent to (item for item in iterable if item). It is approximately equivalent to (but faster than) filter(bool, iterable). ChrisA From kirillbalunov at gmail.com Tue Mar 6 10:12:21 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 18:12:21 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 17:55 GMT+03:00 Chris Angelico : > On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov > wrote: > > Note: For some historical reasons as the first argument you can use None > > instead of function, in this case the identity function is assumed. That > > is, all elements of iterable that are false are removed which is > equivalent > > to (item for item in iterable if item). Currently, for the same purpose > the > > preferred form is `filter(bool, iterable)`. > > > > I'd prefer to word it something like: > > If the first argument is None, the identity function is assumed. That > is, all elements of the iterable that are false are removed; it is > equivalent to (item for item in iterable if item). It is approximately > equivalent to (but faster than) filter(bool, iterable). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > I do not want to seem rude and stubborn, but how much faster is it to highlight or emphasize it: from random import randint for i in [1, 10, 100, 1000, 10000, 100000]: ls = [randint(0,1) for _ in range(i)] %timeit [*filter(None, ls)] %timeit [*filter(bool, ls)] print() 272 ns ? 0.0346 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 282 ns ? 0.0714 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 283 ns ? 0.0645 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 296 ns ? 0.116 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 1.4 ?s ? 1.32 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 1.41 ?s ? 4.05 ns per loop (mean ? std. dev. of 7 runs, 1000000 loops each) 14.7 ?s ? 40.1 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) 14.7 ?s ? 23.2 ns per loop (mean ? std. dev. of 7 runs, 100000 loops each) 137 ?s ? 186 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) 137 ?s ? 24.7 ns per loop (mean ? std. dev. of 7 runs, 10000 loops each) 1.32 ms ? 285 ns per loop (mean ? std. dev. of 7 runs, 1000 loops each) 1.32 ms ? 908 ns per loop (mean ? std. dev. of 7 runs, 1000 loops each) With kind regards, -gdg From rosuav at gmail.com Tue Mar 6 10:31:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Mar 2018 02:31:19 +1100 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Wed, Mar 7, 2018 at 2:12 AM, Kirill Balunov wrote: > > > 2018-03-06 17:55 GMT+03:00 Chris Angelico : >> >> On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov >> wrote: >> > Note: For some historical reasons as the first argument you can use None >> > instead of function, in this case the identity function is assumed. That >> > is, all elements of iterable that are false are removed which is >> > equivalent >> > to (item for item in iterable if item). Currently, for the same purpose >> > the >> > preferred form is `filter(bool, iterable)`. >> > >> >> I'd prefer to word it something like: >> >> If the first argument is None, the identity function is assumed. That >> is, all elements of the iterable that are false are removed; it is >> equivalent to (item for item in iterable if item). It is approximately >> equivalent to (but faster than) filter(bool, iterable). >> >> ChrisA >> -- >> https://mail.python.org/mailman/listinfo/python-list > > > I do not want to seem rude and stubborn, but how much faster is it to > highlight or emphasize it: > Timings mean little. Why do we write: if lst: instead of: if bool(lst): ? Because it's unnecessary and pointless to call bool() on something before using it in a boolean context. If that concept causes you problems, it's not the fault of the filter function; filter simply uses something in a boolean context. So "the identity function" is more correct than "the bool() function". ChrisA From kirillbalunov at gmail.com Tue Mar 6 10:33:29 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 6 Mar 2018 18:33:29 +0300 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: 2018-03-06 17:55 GMT+03:00 Chris Angelico : > If the first argument is None, the identity function is assumed. That > is, all elements of the iterable that are false are removed; it is > equivalent to (item for item in iterable if item). It is approximately > equivalent to (but faster than) filter(bool, iterable). > > ChrisA > -- > https://mail.python.org/mailman/listinfo/python-list > If you look in C source for `filter_next` https://github.com/python/cpython/blob/5d92647102fac9e116b98ab8bbc632eeed501c34/Python/bltinmodule.c#L593, there is a line: int checktrue = lz->func == Py_None || lz->func == (PyObject *)&PyBool_Type; So the only difference between `filter(None, ls`) and `filter(bool, ls)` is LOAD_NAME vs LOAD_CONST and that `None` is checked before than `bool`. With kind regards, -gdg From rosuav at gmail.com Tue Mar 6 10:37:01 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Mar 2018 02:37:01 +1100 Subject: Do not promote `None` as the first argument to `filter` in documentation. In-Reply-To: References: Message-ID: On Wed, Mar 7, 2018 at 2:33 AM, Kirill Balunov wrote: > > > 2018-03-06 17:55 GMT+03:00 Chris Angelico : >> >> If the first argument is None, the identity function is assumed. That >> is, all elements of the iterable that are false are removed; it is >> equivalent to (item for item in iterable if item). It is approximately >> equivalent to (but faster than) filter(bool, iterable). >> >> ChrisA >> -- >> https://mail.python.org/mailman/listinfo/python-list > > > If you look in C source for `filter_next` > https://github.com/python/cpython/blob/5d92647102fac9e116b98ab8bbc632eeed501c34/Python/bltinmodule.c#L593, > there is a line: > > int checktrue = lz->func == Py_None || lz->func == (PyObject *)&PyBool_Type; > > So the only difference between `filter(None, ls`) and `filter(bool, ls)` is > LOAD_NAME vs LOAD_CONST and that `None` is checked before than `bool`. > Assuming that nobody's shadowed the name 'bool' anywhere, which has to be checked for at run time. (Which is the job of LOAD_NAME.) ChrisA From __peter__ at web.de Tue Mar 6 10:41:38 2018 From: __peter__ at web.de (Peter Otten) Date: Tue, 06 Mar 2018 16:41:38 +0100 Subject: Do not promote `None` as the first argument to `filter` in documentation. References: Message-ID: Chris Angelico wrote: > On Wed, Mar 7, 2018 at 2:12 AM, Kirill Balunov > wrote: >> >> >> 2018-03-06 17:55 GMT+03:00 Chris Angelico : >>> >>> On Wed, Mar 7, 2018 at 1:48 AM, Kirill Balunov >>> wrote: >>> > Note: For some historical reasons as the first argument you can use >>> > None instead of function, in this case the identity function is >>> > assumed. That is, all elements of iterable that are false are removed >>> > which is equivalent >>> > to (item for item in iterable if item). Currently, for the same >>> > purpose the >>> > preferred form is `filter(bool, iterable)`. >>> > >>> >>> I'd prefer to word it something like: >>> >>> If the first argument is None, the identity function is assumed. That >>> is, all elements of the iterable that are false are removed; it is >>> equivalent to (item for item in iterable if item). It is approximately >>> equivalent to (but faster than) filter(bool, iterable). >>> >>> ChrisA >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >> >> >> I do not want to seem rude and stubborn, but how much faster is it to >> highlight or emphasize it: >> > > Timings mean little. Why do we write: > > if lst: > > instead of: > > if bool(lst): > > ? Because it's unnecessary and pointless to call bool() on something > before using it in a boolean context. If that concept causes you > problems, it's not the fault of the filter function; filter simply > uses something in a boolean context. > > So "the identity function" is more correct than "the bool() function". Fun fact: CPython handles filter(bool) and filter(None) the same way, it sets the checktrue flag and chooses the fast path: static PyObject * filter_next(filterobject *lz) { PyObject *item; PyObject *it = lz->it; long ok; PyObject *(*iternext)(PyObject *); int checktrue = lz->func == Py_None || lz->func == (PyObject *)&PyBool_Type; iternext = *Py_TYPE(it)->tp_iternext; for (;;) { item = iternext(it); if (item == NULL) return NULL; if (checktrue) { ok = PyObject_IsTrue(item); } else { PyObject *good; good = PyObject_CallFunctionObjArgs(lz->func, item, NULL); if (good == NULL) { Py_DECREF(item); return NULL; } ok = PyObject_IsTrue(good); Py_DECREF(good); } if (ok > 0) return item; Py_DECREF(item); if (ok < 0) return NULL; } } If there were a built-in identity() function it could be treated the same way. From ooomzay at gmail.com Tue Mar 6 17:09:53 2018 From: ooomzay at gmail.com (Ooomzay) Date: Tue, 6 Mar 2018 14:09:53 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> Message-ID: <37f63e45-ba64-47aa-aa30-aeff8f48188b@googlegroups.com> On Monday, 5 March 2018 23:06:53 UTC, Steven D'Aprano wrote: > On Mon, 05 Mar 2018 09:22:33 -0800, Ooomzay wrote: > [...] > > If you would like to have a shot at coding this without RAII, but > > preserving the OO design, you will find that it is considerably > > simpler than the with/context manager approach. > > Preserving the OO design, you say? Okay, since my application apparently > isn't allowed to know that it is processing two files, I'll simply > delegate that to the object: > > class C(A, B): > def run(self): > with open(self.a) as a: > with open(self.b) as b: > process(a, b) > > # Later. > if __name__ = '__main__': > # Enter the Kingdom of Nouns. > c = C() > c.run() > > There you go. Encapsulation and OO design. 1. This does not execute. It is only when you actually flesh out these deliberately minimal classes A, B & C with PEP232 paraphernalia that we all be able to see clearly how much cleaner or messier things are with PEP232. 2. Your Class C breaks A & B's encapsulation by inheriting rather than composing them. Apart from increasing coupling and preventing substitution, C contained A & B in this example to illustrate that there is no RAII related burden whatever_on this intermediate class as it manages no external resources directly. It does not even need to implement __del__. 3. You have assumed a single threaded application. Please imagine that A and B are classes managing some remote valves and maintain their own threads as well as a serial port for comms. And C is there to coordinate them towards some greater purpose. If you would like to try and add PEP232 support to classes A,B & C to the point that you can create and destroy c = C() in an exception-safe way we may all learn something. > I'm not trying to dissuade you from using RAII in your own applications, > if it works for you, great. Unfortunately, despite having conquered it, without a _guarantee_ of this behaviour from the language, or at least one mainstream implementation, I will not invest in python again. Nor recommend any one else with a serious real world resource management application to do so. This was the original motive for my PEP. From roel at roelschroeven.net Tue Mar 6 17:15:04 2018 From: roel at roelschroeven.net (Roel Schroeven) Date: Tue, 06 Mar 2018 23:15:04 +0100 Subject: Bitstream -- Binary Data for Humans In-Reply-To: <583124ef-26d2-4370-b5dc-571a38ebfcb0@googlegroups.com> References: <46294d2e-5b09-49d9-a213-f87265444bc5@googlegroups.com> <583124ef-26d2-4370-b5dc-571a38ebfcb0@googlegroups.com> Message-ID: S?bastien Boisg?rault schreef op 6/03/2018 10:23: > I had a look at the AIS message format from your link(https://en.wikipedia.org/wiki/Automatic_identification_system#Message_format) and this seems to be a nice use case; all the data components seem to be nicely aligned on the byte boundary ... until you see that the payload uses ASCII6(*), which I didn't know about. > > Thanks again for the info! ASCII6 is only the first layer where your Bitstream could be useful. It's the easy part :). The main work comes after that: decoding ASCII6 results in binary data which we need to decode into AIS messages. You can see the most important ones at https://navcen.uscg.gov/?pageName=AISMessagesA and https://navcen.uscg.gov/?pageName=AISMessagesAStatic but there are many more. So for each message we first extract the first 6 bits, which tells us the message identifier. What we need to do afterwards depends on that message identifier. The links above describe how to deal with messages 1, 2 and 3 (they are identical except for the message identifier; I don't know why there are 3 instead of just one) and 5. Basically it's things like 2 bits for this, next 30 bits for that, etc. That's where I'm hoping your Bitstream can really shine. Also note, but now I'm getting off topic, that we don't have to deal with the low-level data as is described in your link: that is handled by the AIS transponder or AIS receiver, from which we receive it using simple NMEA 0183 serial sentences: see https://en.wikipedia.org/w/index.php?title=NMEA_0183&oldid=828034316#Message_structure -- The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. -- Isaac Asimov Roel Schroeven From rosuav at gmail.com Tue Mar 6 17:21:10 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 7 Mar 2018 09:21:10 +1100 Subject: RFC: Proposal: Deterministic Object Destruction In-Reply-To: <37f63e45-ba64-47aa-aa30-aeff8f48188b@googlegroups.com> References: <8dac4860-00ed-4b09-9c42-b9a3b59cbb86@googlegroups.com> <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> <37f63e45-ba64-47aa-aa30-aeff8f48188b@googlegroups.com> Message-ID: On Wed, Mar 7, 2018 at 9:09 AM, Ooomzay wrote: >> I'm not trying to dissuade you from using RAII in your own applications, >> if it works for you, great. > > Unfortunately, despite having conquered it, without a _guarantee_ of this > behaviour from the language, or at least one mainstream implementation, > I will not invest in python again. Nor recommend any one else with a serious > real world resource management application to do so. This was the original > motive for my PEP. What a pity Python will have to lose someone whose sole goal was to write C++ code. Had you but chosen to write Python code instead, you could have saved us all a hundred emails or so and just used the 'with' statement, same as the rest of us do. Considering that "real world resource management" is *exactly* the purpose of the 'with' statement, I honestly don't see what the problem is. ChrisA From steve+comp.lang.python at pearwood.info Tue Mar 6 17:32:57 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 6 Mar 2018 22:32:57 +0000 (UTC) Subject: RFC: Proposal: Deterministic Object Destruction References: <19497ea2-8074-7e09-4734-4bcb08b28110@gmail.com> <10c0bc43-35cc-4912-a4b3-57eace26fdcd@googlegroups.com> <3e6b5531-ef90-4ad0-abb3-9443df75c07c@googlegroups.com> <95ec40d7-5b8f-4263-a310-40f0768c092e@googlegroups.com> <37f63e45-ba64-47aa-aa30-aeff8f48188b@googlegroups.com> Message-ID: On Tue, 06 Mar 2018 14:09:53 -0800, Ooomzay wrote: > Unfortunately, despite having conquered it, without a _guarantee_ of > this behaviour from the language, or at least one mainstream > implementation, I will not invest in python again. Oh well, so sad. See you later. -- Steve From c00298344 at louisiana.edu Tue Mar 6 18:00:49 2018 From: c00298344 at louisiana.edu (Jeremy Jamar St. Julien) Date: Tue, 6 Mar 2018 17:00:49 -0600 (CST) Subject: Subprocess Startup Error Message-ID: <2093169867.2417129.1520377249944.JavaMail.zimbra@louisiana.edu> Whenever I try to open the python shell it says IDLE?s subprocess didn?t make a connection. Everything worked fine yesterday and I haven?t done anything I think would cause this problem. Any way to fix this? I?ve tried repairing and redownloading From tjreedy at udel.edu Tue Mar 6 21:12:04 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 6 Mar 2018 21:12:04 -0500 Subject: Subprocess Startup Error In-Reply-To: <2093169867.2417129.1520377249944.JavaMail.zimbra@louisiana.edu> References: <2093169867.2417129.1520377249944.JavaMail.zimbra@louisiana.edu> Message-ID: On 3/6/2018 6:00 PM, Jeremy Jamar St. Julien wrote: > Whenever I try to open the python shell it says IDLE?s subprocess didn?t make a connection. You must be referring to IDLE's GUI Shell, not Python's normal console text (TUI?) shell or REPL. IDLE normally runs its GUI in one process and your code in another process, connected by a socket. The message means that the socket connection failed. Everything worked fine yesterday and I haven?t done anything I think would cause this problem. Something has done something somewhere. Any way to fix this? I?ve tried repairing and redownloading Start IDLE in a console with python -m idlelib (idlelib.idle on 2.7) and you *may* get an exception or other error message that explains the situation. There are 5 to 10 known possible reasons. What python binary are you running IDLE with? (It is displayed on the first line of output when you start.) Start normally and look, then start in the console and look, to make sure you are running the same Python both times. -- Terry Jan Reedy From formisc at gmail.com Tue Mar 6 23:03:15 2018 From: formisc at gmail.com (Andrew Z) Date: Tue, 6 Mar 2018 23:03:15 -0500 Subject: LXML: can't register namespace Message-ID: Hello, with 3.6 and latest greatest lxml: from lxml import etree tree = etree.parse('Sample.xml') etree.register_namespace('','http://www.example.com') causes: Traceback (most recent call last): File "/home/az/Work/flask/tutorial_1/src/xml_oper.py", line 16, in etree.register_namespace('','http://www.example.com') File "src/lxml/etree.pyx", line 203, in lxml.etree.register_namespace (src/lxml/etree.c:11705) File "src/lxml/apihelpers.pxi", line 1631, in lxml.etree._tagValidOrRaise (src/lxml/etree.c:35382) ValueError: Invalid tag name '' partial Sample.xml: it seems to not be happy with the empty tag . But i'm not sure why and how to go about it. thank you AZ From steve+comp.lang.python at pearwood.info Wed Mar 7 00:35:17 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 7 Mar 2018 05:35:17 +0000 (UTC) Subject: LXML: can't register namespace References: Message-ID: On Tue, 06 Mar 2018 23:03:15 -0500, Andrew Z wrote: > Hello, > with 3.6 and latest greatest lxml: > > from lxml import etree > > tree = etree.parse('Sample.xml') > etree.register_namespace('','http://www.example.com') > it seems to not be happy with the empty tag . But i'm not sure why and > how to go about it. Have you tried using something other than the empty string? In the interactive interpreter, what does help(etree.register_namespace) say? -- Steve From ooomzay at gmail.com Wed Mar 7 01:42:56 2018 From: ooomzay at gmail.com (Ooomzay) Date: Tue, 6 Mar 2018 22:42:56 -0800 (PST) Subject: Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <2873318.k5KsoXlf1h@palindrom> Message-ID: On Tuesday, 6 March 2018 14:12:38 UTC, Peter Otten wrote: > Chris Angelico wrote: > > > On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano > > wrote: > >> # Later. > >> if __name__ = '__main__': > >> # Enter the Kingdom of Nouns. > > > > Don't you need a NounKingdomEnterer to do that for you? > > No, for some extra flexibility there should be a NounKingdomEntererFactory For example open(). > -- which of course has to implement the AbstractNounKingdomEntererFactory > interface. For example a PEP 343 "Context Manager". In the swamplands of the pythonistas the one-eyed man is BDFL! Amen. From ooomzay at gmail.com Wed Mar 7 01:56:57 2018 From: ooomzay at gmail.com (Ooomzay) Date: Tue, 6 Mar 2018 22:56:57 -0800 (PST) Subject: Layers of abstraction, was Re: RFC: Proposal: Deterministic Object Destruction In-Reply-To: References: <2873318.k5KsoXlf1h@palindrom> Message-ID: On Wednesday, 7 March 2018 06:43:10 UTC, Ooomzay wrote: > On Tuesday, 6 March 2018 14:12:38 UTC, Peter Otten wrote: > > Chris Angelico wrote: > > > > > On Tue, Mar 6, 2018 at 10:04 AM, Steven D'Aprano > > > wrote: > > >> # Later. > > >> if __name__ = '__main__': > > >> # Enter the Kingdom of Nouns. > > > > > > Don't you need a NounKingdomEnterer to do that for you? > > > > No, for some extra flexibility there should be a NounKingdomEntererFactory > > For example open(). > > > -- which of course has to implement the AbstractNounKingdomEntererFactory > > interface. > > For example a PEP 343 "Context Manager". > > In the swamplands of the pythonistas the one-eyed man is BDFL! > > Amen. Damn. That should have read:- In the swamplands of the pythonistas the man with a badly leaking boat is BDFL. From andrewm at object-craft.com.au Wed Mar 7 02:59:26 2018 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Wed, 07 Mar 2018 18:59:26 +1100 Subject: csv module and NULL data byte In-Reply-To: <20180228205259.39594497@bigbox.christie.dr> References: <20180228205259.39594497@bigbox.christie.dr> Message-ID: <20180307075926.D9E96280179@robusta.syd.object-craft.com.au> >> Last time I read the documentation, it was recommended that >> the file be opened in BINARY mode ("rb"). > >It recommends binary mode, but seems to largely work fine with >text/ascii mode or even arbitrary iterables. I've not seen the >rationale behind the binary recommendation, but in 10+ years of using >the csv module, I've not found any issues in using text/ascii mode >that were solved by switching to using binary mode. The CSV module was originally written by Dave Cole. I subsequently made changes necessary to get it included a standard part of Python. I also implemented the "dialect" logic, and I put a lot of time into making the parser and generator produce the same results as Excel. That particular recommendation is necessary because Excel has some particular behaviours around CR and LF and quoting. For the parser to produce the same result as Excel, it must see the raw bytes with no re-ordering or suppression of CRs. Unfortunately, I haven't had time to be involved in the module for a few years. I wasn't involved with the Unicode changes necessary in Python 3, and I have not verified that it is still compatible with recent versions of Excel. -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From eng.laio at voepassaredo.com.br Wed Mar 7 08:45:10 2018 From: eng.laio at voepassaredo.com.br (eng.laio at voepassaredo.com.br) Date: Wed, 7 Mar 2018 05:45:10 -0800 (PST) Subject: Python installer hangs in Windows 7 In-Reply-To: References: <765703485.1601081.1486357381135.ref@mail.yahoo.com> <765703485.1601081.1486357381135@mail.yahoo.com> Message-ID: <9c25eddc-9dcb-4ca4-96db-6ba086bd4cc3@googlegroups.com> Hi! Exact same problem here! Stucked at "initializing setup", windows 7, pyhton 3.6.4, etc, etc, etc.... However, while looking for solutions on the internet.. guess who's decided to come to party??? Setup finally started! I guess we just have to be pacient and give its time... From formisc at gmail.com Wed Mar 7 08:55:05 2018 From: formisc at gmail.com (Andrew Z) Date: Wed, 7 Mar 2018 08:55:05 -0500 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: Yes, if i give it any non empty tag - all goes well. All im trying to do is to extract a namespace ( i try to keep simple here. Just first one for now) and register it so i can save xml later on. On Mar 7, 2018 00:38, "Steven D'Aprano" < steve+comp.lang.python at pearwood.info> wrote: > On Tue, 06 Mar 2018 23:03:15 -0500, Andrew Z wrote: > > > Hello, > > with 3.6 and latest greatest lxml: > > > > from lxml import etree > > > > tree = etree.parse('Sample.xml') > > etree.register_namespace('','http://www.example.com') > > > it seems to not be happy with the empty tag . But i'm not sure why and > > how to go about it. > > Have you tried using something other than the empty string? > > In the interactive interpreter, what does > > help(etree.register_namespace) > > say? > > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list > From c00298344 at louisiana.edu Wed Mar 7 09:07:09 2018 From: c00298344 at louisiana.edu (Jeremy Jamar St. Julien) Date: Wed, 7 Mar 2018 08:07:09 -0600 (CST) Subject: Python 3.6 In-Reply-To: <1666831057.2581540.1520431623138.JavaMail.zimbra@louisiana.edu> Message-ID: <2108114722.2581564.1520431629774.JavaMail.zimbra@louisiana.edu> How do i open python 3.6 in a console and how do i see the binary its running with From rhodri at kynesim.co.uk Wed Mar 7 09:29:44 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 7 Mar 2018 14:29:44 +0000 Subject: Python 3.6 In-Reply-To: <2108114722.2581564.1520431629774.JavaMail.zimbra@louisiana.edu> References: <2108114722.2581564.1520431629774.JavaMail.zimbra@louisiana.edu> Message-ID: <3499ac56-9122-d91a-ee90-8ca8f123138d@kynesim.co.uk> On 07/03/18 14:07, Jeremy Jamar St. Julien wrote: > How do i open python 3.6 in a console and how do i see the binary its running with Can you give us a little more information? What operating system are you using? When you say "console", do you mean whatever passes for a terminal window on your OS or the IDLE program? What exactly do you mean by "the binary its [sic] running with"? -- Rhodri James *-* Kynesim Ltd From c00298344 at louisiana.edu Wed Mar 7 09:41:11 2018 From: c00298344 at louisiana.edu (Jeremy Jamar St. Julien) Date: Wed, 7 Mar 2018 08:41:11 -0600 (CST) Subject: Console In-Reply-To: <419493174.2595004.1520433533883.JavaMail.zimbra@louisiana.edu> Message-ID: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> I had an problem when trying to start the python GUI. It said there was a subprocess startup error. I was told to start IDLE in a console with idlelib and see what python binary i was runnning IDLE with. Im using windows 10 and i guess console refers to the terminal window and i have no idea what they meant by "the binary its running with" From rhodri at kynesim.co.uk Wed Mar 7 09:55:30 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 7 Mar 2018 14:55:30 +0000 Subject: Console In-Reply-To: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> References: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: On 07/03/18 14:41, Jeremy Jamar St. Julien wrote: > I had an problem when trying to start the python GUI. It said there was a subprocess startup error. I was told to start IDLE in a console with idlelib and see what python binary i was runnning IDLE with. Im using windows 10 and i guess console refers to the terminal window and i have no idea what they meant by "the binary its running with" > I'm afraid I can't help you with with the Console program. Windows does like to bury Console deep in its menus, and I don't have a copy of Windows 10 to find it in. Once you have a console running, all you should need to do then is type "idle" at the prompt. That should open a window, which amongst other things will tell you the version of Python it is using. If that fails, try "idle -n" instead; it may not make any difference, but it's worth a go. -- Rhodri James *-* Kynesim Ltd From wolfgang.maier at biologie.uni-freiburg.de Wed Mar 7 10:34:47 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Wed, 7 Mar 2018 16:34:47 +0100 Subject: Console In-Reply-To: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> References: <419493174.2595004.1520433533883.JavaMail.zimbra@louisiana.edu> <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: On 03/07/2018 03:41 PM, Jeremy Jamar St. Julien wrote: > I had an problem when trying to start the python GUI. It said there was a subprocess startup error. I was told to start IDLE in a console with idlelib and see what python binary i was runnning IDLE with. Im using windows 10 and i guess console refers to the terminal window and i have no idea what they meant by "the binary its running with" > The simplest way to open the console on Windows is to press Win+R (the Windows key together with the r key that is). In the run dialog that appears then, type cmd and the console window should appear. To start IDLE from there type: py -m idlelib Good luck, Wolfgang From eryksun at gmail.com Wed Mar 7 10:47:27 2018 From: eryksun at gmail.com (eryk sun) Date: Wed, 7 Mar 2018 15:47:27 +0000 Subject: Console In-Reply-To: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> References: <419493174.2595004.1520433533883.JavaMail.zimbra@louisiana.edu> <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: On Wed, Mar 7, 2018 at 2:41 PM, Jeremy Jamar St. Julien wrote: > I had an problem when trying to start the python GUI. It said there was a subprocess startup > error. I was told to start IDLE in a console with idlelib and see what python binary i was > runnning IDLE with. Im using windows 10 and i guess console refers to the terminal window > and i have no idea what they meant by "the binary its running with" The shortcuts installed for IDLE run idle.pyw explicitly via the installed version of pythonw.exe (no console version, as opposed to python.exe). There shouldn't be any mixup there. But if it's failing for some other reason, then using the python.exe console app will help to diagnose it. Run cmd.exe or powershell.exe. (FYI, these are console-based shells. They *attach* to a console that they either inherit or allocate, just like python.exe. Generally a shell is not a console or terminal.) Then try `python.exe -m idlelib`. If that command isn't found, try `py.exe -m idlelib`. This will inherit CMD's console, and return control to CMD either after you exit IDLE, or after Python dies on an unhandled exception. From c00298344 at louisiana.edu Wed Mar 7 12:13:42 2018 From: c00298344 at louisiana.edu (Jeremy Jamar St. Julien) Date: Wed, 7 Mar 2018 11:13:42 -0600 (CST) Subject: Console In-Reply-To: References: <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: <965786290.2671368.1520442822682.JavaMail.zimbra@louisiana.edu> I think i fixed the problem for now. When i tried to run programs it was giving me errors in certain prewritten python files so i wiped them all and redownloaded them from the website ----- Original Message ----- From: "Rhodri James" To: python-list at python.org Sent: Wednesday, March 7, 2018 8:55:30 AM Subject: Re: Console On 07/03/18 14:41, Jeremy Jamar St. Julien wrote: > I had an problem when trying to start the python GUI. It said there was a subprocess startup error. I was told to start IDLE in a console with idlelib and see what python binary i was runnning IDLE with. Im using windows 10 and i guess console refers to the terminal window and i have no idea what they meant by "the binary its running with" > I'm afraid I can't help you with with the Console program. Windows does like to bury Console deep in its menus, and I don't have a copy of Windows 10 to find it in. Once you have a console running, all you should need to do then is type "idle" at the prompt. That should open a window, which amongst other things will tell you the version of Python it is using. If that fails, try "idle -n" instead; it may not make any difference, but it's worth a go. -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list From ooomzay at gmail.com Wed Mar 7 13:17:42 2018 From: ooomzay at gmail.com (Ooomzay) Date: Wed, 7 Mar 2018 10:17:42 -0800 (PST) Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> Message-ID: <097dd512-592e-47a0-b24c-6692f23a57e9@googlegroups.com> On Thursday, 1 March 2018 22:44:59 UTC, Rob Gaddi wrote: > On 03/01/2018 02:24 PM, Lawrence D?Oliveiro wrote: > > On Thursday, March 1, 2018 at 6:44:39 PM UTC+13, Paul Rubin wrote: > >> DOM trees are a classic example (see the various DOM modules in the > >> Python stdlib). Non-leaf nodes have a list of child nodes, child nodes > >> have pointers back upwards to their parent, and each child node has > >> pointers to its left and right siblings if they exist. It's all very > >> circular. > > > > This is why you have weak refs. > > > But making sure that the failure to use weakrefs doesn't memory leak the > program into the pavement is why you have background garbage collection > with intelligent cyclical reference breaking. That is the worst justification for gc I have seen yet, and also the most truthful. gc.set_debug(gc.DEBUG_LEAK) is your friend my friends. From rosuav at gmail.com Wed Mar 7 13:23:55 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Mar 2018 05:23:55 +1100 Subject: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited) In-Reply-To: <097dd512-592e-47a0-b24c-6692f23a57e9@googlegroups.com> References: <20c727d6-e61a-42b1-90c7-c529fe6f67c5@googlegroups.com> <87r2p4z6xv.fsf@nightsong.com> <0cb89904-5a2b-4efc-8d6c-cfb9554e4533@googlegroups.com> <097dd512-592e-47a0-b24c-6692f23a57e9@googlegroups.com> Message-ID: On Thu, Mar 8, 2018 at 5:17 AM, Ooomzay wrote: > On Thursday, 1 March 2018 22:44:59 UTC, Rob Gaddi wrote: >> On 03/01/2018 02:24 PM, Lawrence D?Oliveiro wrote: >> > On Thursday, March 1, 2018 at 6:44:39 PM UTC+13, Paul Rubin wrote: >> >> DOM trees are a classic example (see the various DOM modules in the >> >> Python stdlib). Non-leaf nodes have a list of child nodes, child nodes >> >> have pointers back upwards to their parent, and each child node has >> >> pointers to its left and right siblings if they exist. It's all very >> >> circular. >> > >> > This is why you have weak refs. >> > >> But making sure that the failure to use weakrefs doesn't memory leak the >> program into the pavement is why you have background garbage collection >> with intelligent cyclical reference breaking. > > That is the worst justification for gc I have seen yet, and also the most truthful. gc.set_debug(gc.DEBUG_LEAK) is your friend my friends. Look. You obviously want to use C++, so go use it, and stop trying to make *us* use it. We've chosen to use Python. *plonk* ChrisA From bc at freeuk.com Wed Mar 7 14:59:31 2018 From: bc at freeuk.com (bartc) Date: Wed, 7 Mar 2018 19:59:31 +0000 Subject: Console In-Reply-To: References: <419493174.2595004.1520433533883.JavaMail.zimbra@louisiana.edu> <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: On 07/03/2018 15:34, Wolfgang Maier wrote: > On 03/07/2018 03:41 PM, Jeremy Jamar St. Julien wrote: >> I had an problem when trying to start the python GUI. It said there >> was a subprocess startup error. I was told to start IDLE in a console >> with idlelib and see what python binary i was runnning IDLE with. Im >> using windows 10 and i guess console refers to the terminal window and >> i have no idea what they meant by "the binary its running with" >> > > The simplest way to open the console on Windows is to press Win+R (the > Windows key together with the r key that is). In the run dialog that > appears then, type cmd and the console window should appear. With Windows 10 all the useful apps that are now hard to find are listed by right-clicking the Start button. The console one is called Command Prompt. -- bartc From eryksun at gmail.com Wed Mar 7 15:21:27 2018 From: eryksun at gmail.com (eryk sun) Date: Wed, 7 Mar 2018 20:21:27 +0000 Subject: Console In-Reply-To: References: <419493174.2595004.1520433533883.JavaMail.zimbra@louisiana.edu> <1337018180.2596424.1520433671391.JavaMail.zimbra@louisiana.edu> Message-ID: On Wed, Mar 7, 2018 at 7:59 PM, bartc wrote: > On 07/03/2018 15:34, Wolfgang Maier wrote: >> >> On 03/07/2018 03:41 PM, Jeremy Jamar St. Julien wrote: >>> >>> I had an problem when trying to start the python GUI. It said there was a >>> subprocess startup error. I was told to start IDLE in a console with idlelib >>> and see what python binary i was runnning IDLE with. Im using windows 10 and >>> i guess console refers to the terminal window and i have no idea what they >>> meant by "the binary its running with" >>> >> >> The simplest way to open the console on Windows is to press Win+R (the >> Windows key together with the r key that is). In the run dialog that appears >> then, type cmd and the console window should appear. > > > With Windows 10 all the useful apps that are now hard to find are listed by > right-clicking the Start button. > > The console one is called Command Prompt. The default shell is now "Windows PowerShell", which I think was pushed in the Creators Update. PowerShell is a console-based shell as well, but it's different from a classic shell. If you're uncomfortable with it, you can switch back to CMD (Command Prompt) under Settings => Personalization => Taskbar. From python at bladeshadow.org Wed Mar 7 16:36:43 2018 From: python at bladeshadow.org (Python) Date: Wed, 7 Mar 2018 15:36:43 -0600 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> <20180305235320.GN10273@bladeshadow.org> Message-ID: <20180307213643.GO10273@bladeshadow.org> On Mon, Mar 05, 2018 at 04:09:48PM -0800, Dan Stromberg wrote: > On Mon, Mar 5, 2018 at 3:53 PM, Python wrote: > > On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote: > >> > Python is often a preferred solution because it is often fantastic for > >> > rapid implementation and maintainability. The GIL's interference > >> > with threaded code performance has, for me at least, on several > >> > occasions been... disappointing (perf costs of removing it aside) > >> > because it gets in the way of choosing Python for such solutions. > >> > Jython and IronPython are simply not feasible options for me, for > >> > multiple reasons that have zero to do with their technical > >> > suitability. > >> > >> Have you actually tried it and run into problems, > > > > Yes. It was years ago and I forget the details, but I even posted > > some sample code here and was told (quite possibly by you) that it was > > the GIL that was eating my lunch. Someone suggested writing the bits > > I wanted to thread as a C extension, which largely defeated the > > purpose of using Python. In at least one case I just used C++, and in > > another I just ignored the problem until it went away. > > So how about a little Cython? Nope. As a practical matter, the only Python option I have is Python proper (i.e. CPython). From rosuav at gmail.com Wed Mar 7 16:44:16 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 8 Mar 2018 08:44:16 +1100 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: <20180307213643.GO10273@bladeshadow.org> References: <65185193-a1e5-437d-8679-fd9fb0fca0ba@googlegroups.com> <20180302204555.GL10273@bladeshadow.org> <20180305235320.GN10273@bladeshadow.org> <20180307213643.GO10273@bladeshadow.org> Message-ID: On Thu, Mar 8, 2018 at 8:36 AM, Python wrote: > On Mon, Mar 05, 2018 at 04:09:48PM -0800, Dan Stromberg wrote: >> On Mon, Mar 5, 2018 at 3:53 PM, Python wrote: >> > On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote: >> >> > Python is often a preferred solution because it is often fantastic for >> >> > rapid implementation and maintainability. The GIL's interference >> >> > with threaded code performance has, for me at least, on several >> >> > occasions been... disappointing (perf costs of removing it aside) >> >> > because it gets in the way of choosing Python for such solutions. >> >> > Jython and IronPython are simply not feasible options for me, for >> >> > multiple reasons that have zero to do with their technical >> >> > suitability. >> >> >> >> Have you actually tried it and run into problems, >> > >> > Yes. It was years ago and I forget the details, but I even posted >> > some sample code here and was told (quite possibly by you) that it was >> > the GIL that was eating my lunch. Someone suggested writing the bits >> > I wanted to thread as a C extension, which largely defeated the >> > purpose of using Python. In at least one case I just used C++, and in >> > another I just ignored the problem until it went away. >> >> So how about a little Cython? > > Nope. As a practical matter, the only Python option I have is > Python proper (i.e. CPython). So you can use C++, but only if you write the entire program in it, instead of using Cython and only transforming the part that's actually slow? ChrisA From tmrsg11 at gmail.com Wed Mar 7 16:57:51 2018 From: tmrsg11 at gmail.com (C W) Date: Wed, 7 Mar 2018 16:57:51 -0500 Subject: Which part of the loop is it going through in this class frame? Message-ID: Hello, I am new to OOP. I'm a bit confused about the following code. class Clock(object): def __init__(self, time): self.time = time def print_time(self): time = '6:30' print(self.time) clock = Clock('5:30') clock.print_time() 5:30 I set time to 6:30, but it's coming out to 5:30. I guess it's because I passed in 5:30, so, it's replaced? How does line-by-line execution run inside a frame? How does __init__ work? I understand you must have __init__. Is it run before print_time(), if so, why don't I just set self.time = '6:30' instead of self.time = time? Thanks! From jladasky at itu.edu Wed Mar 7 17:22:08 2018 From: jladasky at itu.edu (jladasky at itu.edu) Date: Wed, 7 Mar 2018 14:22:08 -0800 (PST) Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: On Wednesday, March 7, 2018 at 1:58:33 PM UTC-8, C W wrote: > Hello, > > I am new to OOP. There are (at least) two purposes for classes: 1) To group together data and functions in a meaningful way. Functions which are defined inside a class are called methods. 2) To allow the preservation of state when you exit a method. >I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time Here you are binding SELF.time to the value you passed in using the name time. The value of self.time is preserved when you leave the __init__ method, and that value is accessible again later when you call other methods in your Clock object. The local variable called time in __init__() will disappear when you leave __init__(). > def print_time(self): > time = '6:30' > print(self.time) > > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? No, time and self.time are two different variables. Your program is printing self.time, which you defined when you created your Clock object. And honestly, that's the more intuitive behavior for an object. Why create an object and then not use its state? > How does line-by-line execution run inside a frame? How does __init__ work? > I understand you must have __init__. Is it run before print_time(), Yes, __init__() is the first method that gets called when you create an instance of a class. You don't name __init__() explicitly, because Python has to construct the reference to self. You perform all of the initial setup of your object in __init__(). > if so, > why don't I just set self.time = '6:30' instead of self.time = time? If you don't want to pass a time to a Clock when you create one, you could hard-code self.time = "6:30" in __init__(). But again, your instincts are correct and intuitive to me. You want a Clock object that you can set to a user-defined time, when you create it. > Thanks! From youta.t at gmail.com Wed Mar 7 17:22:11 2018 From: youta.t at gmail.com (Youta TAKAOKA) Date: Wed, 07 Mar 2018 22:22:11 +0000 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: When we want to set attributes of `self`, we should do it explicitly. Not like `this.` in java or C#, `self.` is needed in python. So, the method > def print_time(self): > time = '6:30' > print(self.time) does 1. set a local variable `time` as '6:30' 2. print `time` which is an attribute of `self` Two `time` s are different. It runs line-by-line. `__init__` works at the insantiation. -- youta.t at gmail.com From python at bladeshadow.org Wed Mar 7 17:39:10 2018 From: python at bladeshadow.org (Python) Date: Wed, 7 Mar 2018 16:39:10 -0600 Subject: How to make Python run as fast (or faster) than Julia In-Reply-To: References: <20180302204555.GL10273@bladeshadow.org> <20180305235320.GN10273@bladeshadow.org> <20180307213643.GO10273@bladeshadow.org> Message-ID: <20180307223910.GP10273@bladeshadow.org> On Thu, Mar 08, 2018 at 08:44:16AM +1100, Chris Angelico wrote: > On Thu, Mar 8, 2018 at 8:36 AM, Python wrote: > > On Mon, Mar 05, 2018 at 04:09:48PM -0800, Dan Stromberg wrote: > >> On Mon, Mar 5, 2018 at 3:53 PM, Python wrote: > >> > On Sat, Mar 03, 2018 at 08:18:03AM +1100, Chris Angelico wrote: > >> >> > Python is often a preferred solution because it is often fantastic for > >> >> > rapid implementation and maintainability. The GIL's interference > >> >> > with threaded code performance has, for me at least, on several > >> >> > occasions been... disappointing (perf costs of removing it aside) > >> >> > because it gets in the way of choosing Python for such solutions. > >> >> > Jython and IronPython are simply not feasible options for me, for > >> >> > multiple reasons that have zero to do with their technical > >> >> > suitability. > >> >> > >> >> Have you actually tried it and run into problems, > >> > > >> > Yes. It was years ago and I forget the details, but I even posted > >> > some sample code here and was told (quite possibly by you) that it was > >> > the GIL that was eating my lunch. Someone suggested writing the bits > >> > I wanted to thread as a C extension, which largely defeated the > >> > purpose of using Python. In at least one case I just used C++, and in > >> > another I just ignored the problem until it went away. > >> > >> So how about a little Cython? > > > > Nope. As a practical matter, the only Python option I have is > > Python proper (i.e. CPython). > > So you can use C++, but only if you write the entire program in it, > instead of using Cython and only transforming the part that's actually > slow? That describes the situation closely enough to correctly that I don't feel the need to elaborate, yes. I will say that /in theory/ I can do whatever I want; in practice there are significant barriers to integrate new tools into the build system I'm required to use that make it prohibitive to do so. There are complexities there which I don't care to describe, but the point remains, effectively, the only Python option I have available is CPython. From johnpote at jptechnical.co.uk Wed Mar 7 17:43:34 2018 From: johnpote at jptechnical.co.uk (John Pote) Date: Wed, 7 Mar 2018 22:43:34 +0000 Subject: csv module and NULL data byte In-Reply-To: <20180307075926.D9E96280179@robusta.syd.object-craft.com.au> References: <20180228205259.39594497@bigbox.christie.dr> <20180307075926.D9E96280179@robusta.syd.object-craft.com.au> Message-ID: <20b871bd-0f52-f4f6-e35f-f1bbce00dc38@jptechnical.co.uk> On 07/03/2018 07:59, Andrew McNamara wrote: >>> Last time I read the documentation, it was recommended that >>> the file be opened in BINARY mode ("rb"). >> It recommends binary mode, but seems to largely work fine with >> text/ascii mode or even arbitrary iterables. I've not seen the >> rationale behind the binary recommendation, but in 10+ years of using >> the csv module, I've not found any issues in using text/ascii mode >> that were solved by switching to using binary mode. > The CSV module was originally written by Dave Cole. I subsequently > made changes necessary to get it included a standard part of Python. > I also implemented the "dialect" logic, and I put a lot of time into > making the parser and generator produce the same results as Excel. > > That particular recommendation is necessary because Excel has some > particular behaviours around CR and LF and quoting. For the parser to > produce the same result as Excel, it must see the raw bytes with no > re-ordering or suppression of CRs. > > Unfortunately, I haven't had time to be involved in the module for a few > years. I wasn't involved with the Unicode changes necessary in Python 3, > and I have not verified that it is still compatible with recent versions > of Excel. Any idea why it might throw an exception on encountering a NULL in the input stream? It accepts all other 255 byte values. Was this behaviour intended? Perhaps a comment should be added to the docs. Thanks for your work on the module anyway. John From cs at cskk.id.au Wed Mar 7 17:44:02 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Thu, 8 Mar 2018 09:44:02 +1100 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: <20180307224402.GA91820@cskk.homeip.net> On 07Mar2018 16:57, C W wrote: >I am new to OOP. I'm a bit confused about the following code. > >class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > >clock = Clock('5:30') >clock.print_time() >5:30 > >I set time to 6:30, but it's coming out to 5:30. I guess it's because I >passed in 5:30, so, it's replaced? No, it is because the scope isn't what you think. In the print_time() method, the variable "time" is just a local variable. It is unrelated the the instance ("self")'s "self.time" value. So you're printing self.time, which is unchanged from when the object was initialised. Try adding: print(time) to the method and compare the values. >How does line-by-line execution run inside a frame? How does __init__ work? >I understand you must have __init__. Is it run before print_time(), if so, >why don't I just set self.time = '6:30' instead of self.time = time? A "frame" is normally what we refer to as a "stack frame": if it a little record which the language implementation uses to keep track of nested function calls. When you call a function, a new stack frame is pushed, which amongst other things contains a reference to the local variables alive during the call. When you return from a function the frame is popped and discarded. The __init__ method is called as the initial setup of an object, just after it is created. When you program went: clock = Clock('5:30') a new "Clock" instance was made. Then that instance's __init__ method was called with the '5:30' which you passed to the class name. That is how the string '5:30' gets stored as the .time attribute of the instance. The reason you say "self.time = time" is that you want to store the value passed to the constructor. It might be any time. clock1 = Clock('5:30') clock2 = Clock('5:45') Now you have to distinct Clock instances, one holding the time '5:30' and one holding the time '5:45'. Your print_time method is a little odd. Normally such a function would not take an argument, just print the internal value from the Clock instance. So typically it would look like this: def print_time(self): print(self.time) See that is doesn't take a time parameter? Regarding setting self.time directly to '6:30', that depends what you want to do with it. A "Clock" might have different purposes, and you should decide what the purpose of yours actually is. As an example, Python's "datetime" module contains classes for dates and times and "datetimes" containing a date and a clocklike time. Normally these are made once and NOT CHANGED. So they're like a special kind of number. They have various methods for computing, for you, things like the day of the week or writing the date or time out in particular formats, or comparing two dates. But a sometimes you don't want a fixed timestamp. You might be modelling a clock, whose value changes as time passes. In that case you may well want to adjust the values inside it. You're print_time method seems odd because it does 2 things: it prints a time, but it also seems to want to change the internal time. It is usually good to separate such things. Finally, setting the time attrubute directly is a reasonable thing to do, but it varies with (a) your needs and (b) the OOP paradigm you're following and how strict you're being. In a "pure" OOP environment one never sets the internal state of an object directly, so one never goes: clock1.time = '5:55' Instead you would always go: clock1.set_time('5:45') and be _unaware_ of how the object may be storing that vaue internally. Some languages will enforce this, and provide no way to modify an object except via "setter" methods. This has some advantages: You can switch out objects implementations provided they provide the same setter methods - your calling code doesn't know or care how the clock manages the internal values. In principle you can make objects "remote" i.e. not local im-memory things at all, if that is sensible. This is why some OOP descriptions treat methods as messages: you send a "set_time" request to the clock with a new time, or you send a "get_time" request to a clock the retrive the current time. Also, because everything is done through setter or getter methods, you can't accidentally damage the object internal state by setting to nonsense. The setter methods should sanity check the incoming values, and the getters will always return sensible values. If the caller expects to directly modify the internals then they might set the minutes to negative values, or values >= 60, or some other nonsensical value. By using methods all the responsibility is inside the class where it is easy to keep correct. Python lets you use that paradigm, but it doesn't prevent you directly accessing or changing the internals. The philosophy is that you should exercise restraint. So you _can_ directly set the clock.time value; whether that is a good idea depends on the clock implementation and your program. In Python there is a naming convention: values which may be directy accessed are given ordinary names (like ".time") and values which are part of the internals, subject to change, or tricky to keep correct are given names starting with underscores (like "._time"). If you're accessing a _name that is a clue that you're probably doing the wrong thing (if your outside the class - obviously the class itself must work on these values). Cheers, Cameron Simpson (formerly cs at zip.com.au) From ben+python at benfinney.id.au Wed Mar 7 18:18:08 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 08 Mar 2018 10:18:08 +1100 Subject: Which part of the loop is it going through in this class frame? References: Message-ID: <85muzj5vbj.fsf@benfinney.id.au> C W writes: > I am new to OOP. Welcome, and congratulations on learning Python. > I'm a bit confused about the following code. > > def print_time(self): Begins a function definition. The function will receive one argument (the class instance), and bind the name ?self? to that. > time = '6:30' Creates a new text string, and binds the name ?time? to that. Nothing else ever uses that local name. > print(self.time) Gets the value of ?self.time? ? which means, get the object referred to by ?self?, look up an attribute named ?time?, and get the object that name is bound to ? then pass that object as an argument to call ?print?. The ?print? function will get a text representation of the object, and emit that text to output. Nothing else happens in the function; so, the local name ?time? falls out of scope and is never used. The function then returns ?None?. > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30 No, you bound *a* name ?time? locally inside the ?print_time? function; but you never used that afterward. The local name ?time? is a different reference from the ?self.time? reference. > How does line-by-line execution run inside a frame? How does __init__ > work? After Python creates an instance of a class (using that class's ?__new__? method as the constructor), it then tells the instance to initialise itself; it calls the object's ?__init__? method as the initialiser. So the initialiser, named ?__init__?, is called once the object exists, but before the caller gets to use that object. The initialiser's job is to initialise the state of the object; your class does this by setting the per-instance ?time? attribute. So, by the time your statement binds the name ?clock? to the new Clock instance, that instance already has an attribute ?clock.time? with the value ?'5:30'?. That attribute is then available when something else uses that object; for example, the ?print_time? method accesses that attribute and prints it out. -- \ ?As the most participatory form of mass speech yet developed, | `\ the Internet deserves the highest protection from governmental | _o__) intrusion.? ?U.S. District Court Judge Dalzell | Ben Finney From tjreedy at udel.edu Wed Mar 7 18:30:23 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 7 Mar 2018 18:30:23 -0500 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: On 3/7/2018 4:57 PM, C W wrote: > Hello, > > I am new to OOP. I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) Local name 'time' is bound to '6:30'. Attribute 'time' of the 'self' instance is bound to '5:30' because of the call and initization. You asked to print the latter. If you write 'print(time)' you will see '6:30'. > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? > > How does line-by-line execution run inside a frame? How does __init__ work? > I understand you must have __init__. Is it run before print_time(), if so, > why don't I just set self.time = '6:30' instead of self.time = time? > > Thanks! > -- Terry Jan Reedy From subramanianaug at gmail.com Wed Mar 7 18:31:14 2018 From: subramanianaug at gmail.com (Subramanian P V) Date: Wed, 7 Mar 2018 15:31:14 -0800 (PST) Subject: what is the best qr package Message-ID: what is the best qr package From drsalists at gmail.com Wed Mar 7 20:07:35 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 7 Mar 2018 17:07:35 -0800 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: On Wed, Mar 7, 2018 at 1:57 PM, C W wrote: > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? time and self.time are 2 different things. > How does line-by-line execution run inside a frame To quickly come to grips with execution order, try a graphical or curses debugger like winpdb (it's cross platform, despite the name) or pudb. For this purpose don't do something primitive like pdb - it won't help as much. From dan at tombstonezero.net Wed Mar 7 20:28:31 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Thu, 8 Mar 2018 01:28:31 +0000 (UTC) Subject: Which part of the loop is it going through in this class frame? References: Message-ID: On Wed, 07 Mar 2018 16:57:51 -0500, C W wrote: > Hello, > > I am new to OOP. I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? Here's a hint: within a method, time and self.time are not the same (at least not in Python; other languages may (and do) work differently). Dan From dan at tombstonezero.net Wed Mar 7 21:10:19 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Thu, 8 Mar 2018 02:10:19 +0000 (UTC) Subject: Which part of the loop is it going through in this class frame? References: Message-ID: On Wed, 07 Mar 2018 16:57:51 -0500, C W wrote: > class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? Hint: in Python, self.time and time are not the same (unless you've played some sort of weird trick somewhere). Dan From skip.montanaro at gmail.com Wed Mar 7 21:34:58 2018 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Wed, 7 Mar 2018 20:34:58 -0600 Subject: Futurize maps StringIO.StringIO to io.StringIO Message-ID: I had some deja vu recently, cuz of this old thread: https://mail.python.org/pipermail/python-list/2013-May/648245.html Today the context is different, but the problem remains the same. One of the futurize refactoring tools converts usage of StringIO.StringIO to io.StringIO, which, given that the latter accepts just about anything you through at it, is a bit surprising. Perhaps there is no true corresponding class in the io module? Skip From formisc at gmail.com Wed Mar 7 23:16:19 2018 From: formisc at gmail.com (Andrew Z) Date: Wed, 7 Mar 2018 23:16:19 -0500 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: help(etree.register_namespace) Help on cython_function_or_method in module lxml.etree: register_namespace(prefix, uri) Registers a namespace prefix that newly created Elements in that namespace will use. The registry is global, and any existing mapping for either the given prefix or the namespace URI will be removed. On Wed, Mar 7, 2018 at 8:55 AM, Andrew Z wrote: > Yes, if i give it any non empty tag - all goes well. > > All im trying to do is to extract a namespace ( i try to keep simple here. > Just first one for now) and register it so i can save xml later on. > > > On Mar 7, 2018 00:38, "Steven D'Aprano" pearwood.info> wrote: > >> On Tue, 06 Mar 2018 23:03:15 -0500, Andrew Z wrote: >> >> > Hello, >> > with 3.6 and latest greatest lxml: >> > >> > from lxml import etree >> > >> > tree = etree.parse('Sample.xml') >> > etree.register_namespace('','http://www.example.com') >> >> > it seems to not be happy with the empty tag . But i'm not sure why and >> > how to go about it. >> >> Have you tried using something other than the empty string? >> >> In the interactive interpreter, what does >> >> help(etree.register_namespace) >> >> say? >> >> >> >> -- >> Steve >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > From frank at chagford.com Wed Mar 7 23:48:20 2018 From: frank at chagford.com (Frank Millman) Date: Thu, 8 Mar 2018 06:48:20 +0200 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: "C W" wrote in message news:CAE2FW2nuDJcmvuKAvZH01TrKqEENtkDXDPBAWcPHHsgx8jVLVA at mail.gmail.com... > > Hello, > > I am new to OOP. I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? > In print_time(), the variable 'time' is not the same as the variable 'self.time'. 'time', a local variable, is set to '6:30'. 'self.time', an instance variable, is set to '5:30'. You printed self.time, hence the result. Frank Millman From arj.python at gmail.com Thu Mar 8 01:12:26 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Thu, 8 Mar 2018 10:12:26 +0400 Subject: Module Issue Message-ID: i have a project at https://github.com/Abdur-rahmaanJ/honeybot see https://github.com/Abdur-rahmaanJ/honeybot/tree/master/honeybot my question is : how to include a util file / module that can be imported in both core_plugins and user_plugins? if that is too difficult, let us take only core_plugins make a module not found error not appear when running main.py -- Abdur-Rahmaan Janhangeer https://github.com/abdur-rahmaanj Mauritius Garanti sans virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From dieter at handshake.de Thu Mar 8 01:45:30 2018 From: dieter at handshake.de (dieter) Date: Thu, 08 Mar 2018 07:45:30 +0100 Subject: Which part of the loop is it going through in this class frame? References: Message-ID: <87woynvzed.fsf@handshake.de> C W writes: > I am new to OOP. I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > > clock = Clock('5:30') > clock.print_time() > 5:30 > > I set time to 6:30, but it's coming out to 5:30. I guess it's because I > passed in 5:30, so, it's replaced? Inside a method, you must explicitely use the instance (conventionally (and in your case) names "self") to access an object attribute. In your example, this means, you must use "self.time = '6:30' not "time" alone (you did this in the "__init__" method). If you are using "time" (without the "self." prefix), you are defining a new local variable. In this regard, Python differs from many object oriented languages. From harirammanohar159 at gmail.com Thu Mar 8 02:50:38 2018 From: harirammanohar159 at gmail.com (harirammanohar159 at gmail.com) Date: Wed, 7 Mar 2018 23:50:38 -0800 (PST) Subject: chatterbot -ML Message-ID: Hi, Entire chatterbot we feed the bot with data and it responds accordingly. But why they say its a machine learning dialogue conversation. i dont see any machine learning in that. Atleast if bot doesnt have data what user speaks, if it learns and speaks next time then we can accept machine learning is there. can any one enlighten me? From antoon.pardon at vub.be Thu Mar 8 06:09:29 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Thu, 8 Mar 2018 12:09:29 +0100 Subject: socket: Did I find a bug? Message-ID: <4f307ac2-83ff-e3b3-fb38-70e46196e10e@vub.be> This is on a debian 9 box python 2.7.13 My interpretation is that a timeout exception is thrown and that the args attribute of such an exception is an empty tuple which then causes an IndexError in line 482 of module /usr/lib/python2.7/socket.py. Does that soundplausible? Here is the traceback: Traceback (most recent call last): File "/usr/local/lib/python-apps/rmtdump/pymain.py", line 101, in main Exit_Nr = process(sys.argv) File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 249, in program process(path) File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 184, in process of = rmt.open(rpj(rmt5lcl(rt), archive), "wb") File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 211, in open return ftpfile(ftp, fn, mode, True) File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 70, in __init__ self.cnct = self.ftp.transfercmd("%s %s" % (cmd, rfn)) File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd return self.ntransfercmd(cmd, rest)[0] File "/usr/lib/python2.7/ftplib.py", line 710, in ntransfercmd conn, size = FTP.ntransfercmd(self, cmd, rest) File "/usr/lib/python2.7/ftplib.py", line 339, in ntransfercmd resp = self.sendcmd(cmd) File "/usr/lib/python2.7/ftplib.py", line 249, in sendcmd return self.getresp() File "/usr/lib/python2.7/ftplib.py", line 215, in getresp resp = self.getmultiline() File "/usr/lib/python2.7/ftplib.py", line 201, in getmultiline line = self.getline() File "/usr/lib/python2.7/ftplib.py", line 186, in getline line = self.file.readline(self.maxline + 1) File "/usr/lib/python2.7/socket.py", line 482, in readline if e.args[0] == EINTR: IndexError: tuple index out of range Locals by frame, innermost last Frame main in /usr/local/lib/python-apps/rmtdump/pymain.py at line 119 backtrace = fun = pn = 'rmtdump' process = Frame program in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 281 ErrInfo = IndexError('tuple index out of range',) aborted = True argv = ['/usr/local/sbin/rmtdump'] path = '/home/antoon' Frame process in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 208 ErrInfo = IndexError('tuple index out of range',) b_rt = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be' backupflag = 'rwx' backupstamp = 1520450190 checkstamp = 1519846075 cleanstamp = 1 dirs = ['Archives-1.sbd', 'INBOX.sbd'] entries = ['@TODO.msf', 'Archives-1.msf', 'Archives-1.sbd', 'Archives.msf', 'Drafts-1.msf', 'Drafts.msf', 'INBOX', ... entry = 'msgFilterRules.dat' file_info = posix.stat_result(st_mode=33188, st_ino=10365097, st_dev=2058, st_nlink=1, st_uid=1000, st_gid=1000, st_ ... fileinfo = posix.stat_result(st_mode=33152, st_ino=10621705, st_dev=2058, st_nlink=1, st_uid=1000, st_gid=1000, st_ ... files = ['Junk.msf', 'Drafts.msf', 'filterlog.html', 'Drafts-1.msf', 'Archives.msf', 'junklog.html', 'msgFilterR ... filestamp = 1463826952.531162 fqpn = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/msgFilterRules.dat' islink = ismount = last = 1520450604.318512 ls_stamp = ['+1520450190', '1519846075', '1', '0', '0'] newfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!2018-03-07 at 20_16_30-tbz' now = 1520450549 nr_of_tries = 0 of = None oldfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' path = '/home/antoon' rt = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be' st_stamp = '+1520450190:1519846075:1' stamp = '1' start = 1520450604 tf = timefn = '!2018-03-07 at 20_16_30-tbz' tryal = 0 update = True Frame open in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 211 fn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' ftp = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) mode = 'wb' self = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) Frame __init__ in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 70 bound = True ch = 'b' cmd = 'STOR' cn = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) count = 1 mode = 'wb' rfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' self = None Frame transfercmd in /usr/lib/python2.7/ftplib.py at line 376 cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' rest = None self = Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 710 cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' rest = None self = Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 352 cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' conn = host = '192.110.0.56' port = 62332 rest = None self = size = None Frame sendcmd in /usr/lib/python2.7/ftplib.py at line 249 cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' self = Frame getresp in /usr/lib/python2.7/ftplib.py at line 215 self = Frame getmultiline in /usr/lib/python2.7/ftplib.py at line 201 self = Frame getline in /usr/lib/python2.7/ftplib.py at line 186 self = Frame readline in /usr/lib/python2.7/socket.py at line 482 buf = buf_len = 0 e = timeout() self = size = 8193 From songofacandy at gmail.com Thu Mar 8 06:16:47 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 8 Mar 2018 20:16:47 +0900 Subject: socket: Did I find a bug? In-Reply-To: <4f307ac2-83ff-e3b3-fb38-70e46196e10e@vub.be> References: <4f307ac2-83ff-e3b3-fb38-70e46196e10e@vub.be> Message-ID: https://mail.python.org/pipermail/python-list/2016-February/703154.html On Thu, Mar 8, 2018 at 8:09 PM, Antoon Pardon wrote: > This is on a debian 9 box python 2.7.13 > > My interpretation is that a timeout exception is thrown and that the > args attribute of such an exception is an empty tuple which then causes > an IndexError in line 482 of module /usr/lib/python2.7/socket.py. Does > that soundplausible? > > Here is the traceback: > > Traceback (most recent call last): > File "/usr/local/lib/python-apps/rmtdump/pymain.py", line 101, in main > Exit_Nr = process(sys.argv) > File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 249, in program > process(path) > File "/usr/local/lib/python-apps/rmtdump/rmtdump.py", line 184, in process > of = rmt.open(rpj(rmt5lcl(rt), archive), "wb") > File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 211, in open > return ftpfile(ftp, fn, mode, True) > File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 70, in __init__ > self.cnct = self.ftp.transfercmd("%s %s" % (cmd, rfn)) > File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd > return self.ntransfercmd(cmd, rest)[0] > File "/usr/lib/python2.7/ftplib.py", line 710, in ntransfercmd > conn, size = FTP.ntransfercmd(self, cmd, rest) > File "/usr/lib/python2.7/ftplib.py", line 339, in ntransfercmd > resp = self.sendcmd(cmd) > File "/usr/lib/python2.7/ftplib.py", line 249, in sendcmd > return self.getresp() > File "/usr/lib/python2.7/ftplib.py", line 215, in getresp > resp = self.getmultiline() > File "/usr/lib/python2.7/ftplib.py", line 201, in getmultiline > line = self.getline() > File "/usr/lib/python2.7/ftplib.py", line 186, in getline > line = self.file.readline(self.maxline + 1) > File "/usr/lib/python2.7/socket.py", line 482, in readline > if e.args[0] == EINTR: > IndexError: tuple index out of range > Locals by frame, innermost last > > Frame main in /usr/local/lib/python-apps/rmtdump/pymain.py at line 119 > backtrace = > fun = > pn = 'rmtdump' > process = > > Frame program in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 281 > ErrInfo = IndexError('tuple index out of range',) > aborted = True > argv = ['/usr/local/sbin/rmtdump'] > path = '/home/antoon' > > Frame process in /usr/local/lib/python-apps/rmtdump/rmtdump.py at line 208 > ErrInfo = IndexError('tuple index out of range',) > b_rt = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be' > backupflag = 'rwx' > backupstamp = 1520450190 > checkstamp = 1519846075 > cleanstamp = 1 > dirs = ['Archives-1.sbd', 'INBOX.sbd'] > entries = ['@TODO.msf', 'Archives-1.msf', 'Archives-1.sbd', 'Archives.msf', 'Drafts-1.msf', 'Drafts.msf', 'INBOX', ... > entry = 'msgFilterRules.dat' > file_info = posix.stat_result(st_mode=33188, st_ino=10365097, st_dev=2058, st_nlink=1, st_uid=1000, st_gid=1000, st_ ... > fileinfo = posix.stat_result(st_mode=33152, st_ino=10621705, st_dev=2058, st_nlink=1, st_uid=1000, st_gid=1000, st_ ... > files = ['Junk.msf', 'Drafts.msf', 'filterlog.html', 'Drafts-1.msf', 'Archives.msf', 'junklog.html', 'msgFilterR ... > filestamp = 1463826952.531162 > fqpn = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/msgFilterRules.dat' > islink = > ismount = > last = 1520450604.318512 > ls_stamp = ['+1520450190', '1519846075', '1', '0', '0'] > newfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!2018-03-07 at 20_16_30-tbz' > now = 1520450549 > nr_of_tries = 0 > of = None > oldfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > path = '/home/antoon' > rt = '/home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be' > st_stamp = '+1520450190:1519846075:1' > stamp = '1' > start = 1520450604 > tf = > timefn = '!2018-03-07 at 20_16_30-tbz' > tryal = 0 > update = True > > Frame open in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 211 > fn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > ftp = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) > mode = 'wb' > self = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) > > Frame __init__ in /usr/local/lib/python-apps/rmtdump/ftputil.py at line 70 > bound = True > ch = 'b' > cmd = 'STOR' > cn = FTPS(antoon at pardon-sleeuwaegen.be@ftp.adrive.com) > count = 1 > mode = 'wb' > rfn = 'home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > self = None > > Frame transfercmd in /usr/lib/python2.7/ftplib.py at line 376 > cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > rest = None > self = > > Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 710 > cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > rest = None > self = > > Frame ntransfercmd in /usr/lib/python2.7/ftplib.py at line 352 > cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > conn = > host = '192.110.0.56' > port = 62332 > rest = None > self = > size = None > > Frame sendcmd in /usr/lib/python2.7/ftplib.py at line 249 > cmd = 'STOR home/antoon/.icedove/clam9zaw.default/ImapMail/mail.priorweb.be/!ENTRY-tbz' > self = > > Frame getresp in /usr/lib/python2.7/ftplib.py at line 215 > self = > > Frame getmultiline in /usr/lib/python2.7/ftplib.py at line 201 > self = > > Frame getline in /usr/lib/python2.7/ftplib.py at line 186 > self = > > Frame readline in /usr/lib/python2.7/socket.py at line 482 > buf = > buf_len = 0 > e = timeout() > self = > size = 8193 > > -- > https://mail.python.org/mailman/listinfo/python-list -- INADA Naoki From steve+comp.lang.python at pearwood.info Thu Mar 8 06:30:36 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 8 Mar 2018 11:30:36 +0000 (UTC) Subject: Which part of the loop is it going through in this class frame? References: Message-ID: On Wed, 07 Mar 2018 16:57:51 -0500, C W wrote: > Hello, > > I am new to OOP. I'm a bit confused about the following code. > > class Clock(object): > def __init__(self, time): > self.time = time Here you set the instance attribute "self.time". > def print_time(self): > time = '6:30' > print(self.time) Here you set the local variable "time", which is completely unrelated to the attribute "self.time". If you are used to languages where "foo" inside a method is a short-cut for "self.foo" or "this.foo", Python does not do that. Local variables and instance attributes are distinct concepts, and Python keeps them distinct. > How does line-by-line execution run inside a frame? There isn't actually line-by-line execution as such, although it can be very similar. Before the interpreter runs Python code, it compiles it to byte-code, and then runs the byte-code. A single line of source code could result in any number of lines of byte code, from zero to an unlimited number. > How does __init__ > work? I understand you must have __init__. You understand wrongly then :-) It is normal and common to have an __init__ method, but it is not compulsory. If your class doesn't need one, you don't need to write it. The __init__ method is the initialiser. Think of it as very similar to the constructor in some other languages, and for now the differences aren't important. The usual purpose of the __init__ method is to initialise the instance and set any attributes needed. > Is it run before print_time(), The __init__ method is called once, when the instance is first created. So the short answer is, yes, it will run before print_time(). But only once. > if so, why don't I just set self.time = '6:30' instead of > self.time = time? Because then every instance will be set to 6:30, instead of letting you set each instance to a different time. -- Steve From mal at europython.eu Thu Mar 8 07:12:11 2018 From: mal at europython.eu (M.-A. Lemburg) Date: Thu, 8 Mar 2018 13:12:11 +0100 Subject: EuroPython 2018: Hotel tips and general update Message-ID: <7c9b9348-2e0f-42a6-a007-cca5a79fb034@europython.eu> As you may know, the Edinburgh Fringe Festival is starting one week after EuroPython 2018 in Edinburgh. Since this typically attracts many thousands of people and artists, the hotels are filling up quickly in Edinburgh. If you?re planning to attend EuroPython, please book your hotel early. Many booking sites offer free cancellations, so there?s no risk in making reservations now, even if you decide not to come in the end. Room allocation for EuroPython 2018 ----------------------------------- To help with this, we have partnered with the Edinburgh Convention Bureau to set aside a number of hotel rooms which are reserved for EuroPython attendees. These rooms will be held reserved until a few weeks before the conference and are also available with a free cancellation option. The Edinburgh Convention Bureau has setup the following website for booking rooms from this allocation: * EuroPython 2018 Hotel Booking Website * https://cabsedinburgh.eventsair.com/QuickEventWebsitePortal/europython-2018/home-page (run by the Edinburgh Convention Bureau) Update on EuroPython 2018 ------------------------- Meanwhile, we wanted to give you an update of where we are with the conference organization: We are still working on getting everything setup for launching the website, opening ticket sales and the Call for Proposals (CFP). This year the EuroPython Society (EPS) will be running the ticket sales, rather than a local organization and we are facing some challenges related to VAT taxes, which are taking longer to sort out than expected. This is the main reason for the delay you are seeing, but we?re getting there. Enjoy, -- EuroPython 2018 Team https://ep2018.europython.eu/ https://www.europython-society.org/ PS: Please forward or retweet to help us reach all interested parties: https://twitter.com/europython/status/971716086445027328 Thanks. From mal at europython.eu Thu Mar 8 07:16:01 2018 From: mal at europython.eu (M.-A. Lemburg) Date: Thu, 8 Mar 2018 13:16:01 +0100 Subject: EuroPython 2018: Our conference logo Message-ID: After intense work with our designer, we are happy to present the logo for the EuroPython 2018 conference: https://blog.europython.eu/post/171656128512/europython-2018-our-conference-logo The colors and patterns are a play on Scottish kilts, referring to the location, and a theatre curtain, representing the many presentations we?ll have at the conference. We hope you like it as much as we do. Enjoy, -- EuroPython 2018 Team https://ep2018.europython.eu/ https://www.europython-society.org/ PS: Please forward or retweet to help us reach all interested parties: https://twitter.com/europython/status/971716219232575488 Thanks. From tkadm30 at yandex.com Thu Mar 8 07:38:12 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Thu, 8 Mar 2018 07:38:12 -0500 Subject: psychocomputational analysis toolkit in Python? Message-ID: Hi i want to develop a psychocomputational analysis toolkit in python for studying real-time neuroimaging data from focused ultrasound/brain-to-brain interfaces. Any suggestions where to look for cool projects in github? :) Best regards, Etienne -- Etienne Robillard tkadm30 at yandex.com https://www.isotopesoftware.ca/ https://www.isotoperesearch.ca/ From renesd at gmail.com Thu Mar 8 07:41:28 2018 From: renesd at gmail.com (=?UTF-8?Q?Ren=C3=A9_Dudfield?=) Date: Thu, 8 Mar 2018 13:41:28 +0100 Subject: ANN: pygameweb 0.0.4 - Northern Rockhopper Penguin Message-ID: G'day. pygameweb is the source code for the pygame website at https://www.pygame.org/ [image: 6954251406_95ebe75f55_m] - #52 Adding ability to deploy through travisci. And docs. - #46 Wiki readability. Wiki tools on left out of reading line. - Twitter card for releases. For nicer view when posting to twitter. - Wiki table of contents improvements. - builds docs for pygame, when something lands on pygame github master branch. - project release feeds bugfix. - Only show one recent release for each project. Some projects release a few times in a row. - Wiki, nicer pre code handling. - Wiki code uses inline colors, so copy/paste works with colors too. - use https for login and register buttons, even when on http (see Why can't we turn off HTTPS? ). - Ask more bad robot web crawlers not to be bad. See a full diff of changes since last release . Majestic photo of a Northern Rockhopper Penguin by Brian Gratwicke From christopherrmullins at gmail.com Thu Mar 8 09:44:45 2018 From: christopherrmullins at gmail.com (Christopher Mullins) Date: Thu, 8 Mar 2018 09:44:45 -0500 Subject: psychocomputational analysis toolkit in Python? In-Reply-To: References: Message-ID: You might find MNE useful, and if what you're doing happens to fit somewhere in their package you could contribute to it -- they're a good group to work with. https://www.martinos.org/mne/stable/index.html https://github.com/mne-tools/mne-python From tkadm30 at yandex.com Thu Mar 8 10:06:16 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Thu, 8 Mar 2018 10:06:16 -0500 Subject: psychocomputational analysis toolkit in Python? In-Reply-To: References: Message-ID: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> Thanks for sharing this link. MNE looks pretty cool. :-) I'm mostly interested in mapping specific cortical pathways to brain-to-brain interfaces implicated in the ultrasonic neuromodulation of behavior. Would it be possible to emulate a minimally functional brain-to-brain coupling system entirely in Python? Best regards, Etienne Le 2018-03-08 ? 09:44, Christopher Mullins a ?crit?: > You might find MNE useful, and if what you're doing happens to fit > somewhere in their package you could contribute to it -- they're a > good group to work with. > > https://www.martinos.org/mne/stable/index.html > https://github.com/mne-tools/mne-python -- Etienne Robillard tkadm30 at yandex.com https://www.isotoperesearch.ca/ From christopherrmullins at gmail.com Thu Mar 8 10:31:56 2018 From: christopherrmullins at gmail.com (Christopher Mullins) Date: Thu, 8 Mar 2018 10:31:56 -0500 Subject: psychocomputational analysis toolkit in Python? In-Reply-To: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> References: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> Message-ID: > Would it be possible to emulate a minimally functional brain-to-brain coupling system entirely in Python? I don't know what that would entail, but the links I shared have a mailing list and a very responsive gitter, both of which would be great places to ask about that! (You're welcome to ask here of course, but this being the general python mailing list, it's a shot in the dark.) Good luck! Chris From bgailer at gmail.com Thu Mar 8 10:33:48 2018 From: bgailer at gmail.com (Bob Gailer) Date: Thu, 8 Mar 2018 10:33:48 -0500 Subject: psychocomputational analysis toolkit in Python? In-Reply-To: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> References: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> Message-ID: On Mar 8, 2018 10:08 AM, "Etienne Robillard" wrote: > > Thanks for sharing this link. MNE looks pretty cool. :-) > > I'm mostly interested in mapping specific cortical pathways to brain-to-brain interfaces implicated in the ultrasonic neuromodulation of behavior. > > Would it be possible to emulate a minimally functional brain-to-brain coupling system entirely in Python? Since python is a general purpose programming language, I'd say yes. I would add the use of a database management system, since you need some way of storing the data. sqlite is an excellent database management system with an excellent python interface. Unlike other database systems, sqlite is single-user system. You can always migrate later to a multi-user system. The sqlite3 module is included in the python distribution. From breamoreboy at gmail.com Thu Mar 8 13:03:30 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Thu, 8 Mar 2018 18:03:30 +0000 Subject: Futurize maps StringIO.StringIO to io.StringIO In-Reply-To: References: Message-ID: On 08/03/18 02:34, Skip Montanaro wrote: > I had some deja vu recently, cuz of this old thread: > > https://mail.python.org/pipermail/python-list/2013-May/648245.html > > Today the context is different, but the problem remains the same. One > of the futurize refactoring tools converts usage of StringIO.StringIO > to io.StringIO, which, given that the latter accepts just about > anything you through at it, is a bit surprising. Perhaps there is no > true corresponding class in the io module? > > Skip > Maybe you need io.BytesIO? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From tkadm30 at yandex.com Thu Mar 8 13:20:06 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Thu, 8 Mar 2018 13:20:06 -0500 Subject: psychocomputational analysis toolkit in Python? In-Reply-To: References: <945e4245-fca7-806f-86a8-925cdd9660fe@yandex.com> Message-ID: <9f217222-6dc8-5462-d6b9-26208c41842e@yandex.com> OK I got it, thank you! :) Essentially, I want to develop a psychocomputational analysis toolkit in Python for studying real-time neuroimaging data (of cortical activity) from low-intensity, focused ultrasounds (LIFU) and possibly emulate a functional brain-to-brain coupling system on my 32-bit computer as a proof-of-concept to demonstrate the vulnerability of the human cochlear pathway to become a target of sophisticated ultrasonic side-channel attacks from mobile devices. Etienne Le 2018-03-08 ? 10:31, Christopher Mullins a ?crit?: > > Would it be possible to emulate a minimally functional > brain-to-brain coupling system entirely in Python? > > I don't know what that would entail, but the links I shared have a > mailing list and a very responsive gitter, both of which would be > great places to ask about that!? (You're welcome to ask here of > course, but this being the general python mailing list, it's a shot in > the dark.)? Good luck! > > Chris -- Etienne Robillard tkadm30 at yandex.com https://www.isotoperesearch.ca/ From skip.montanaro at gmail.com Thu Mar 8 15:57:40 2018 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Thu, 8 Mar 2018 14:57:40 -0600 Subject: Unnoticed traceback in a thread Message-ID: I have a program which is almost always running in a single thread. It uses a daemon thread (via threading.Timer) to periodically checkpoint some state. The program runs for days at a time. Over the past couple days, two instances of the subthread croaked with tracebacks because while they were iterating over the checkpointable data, the main thread modified the data. Blammo! It was an easy fix (threading.Lock) and everything is back on an even keel. After the bug was tickled, the program continued to do its thing. It just stopped checkpointing. Given the way the program is managed, the traceback wound up in an obscure log file, and it was a couple days before I noticed it. I've seen this sort of thread-dies-but-program-doesn't-crash situation before. I can look at ways to more actively monitor the contents of the obscure log file, but is there some non-hackish way that the demise of the daemon thread can take down the entire program? (Sometimes it's good that it doesn't crash. Other times it would at least be handy if it did.) Thx, Skip From arequipeno at gmail.com Thu Mar 8 18:23:16 2018 From: arequipeno at gmail.com (Ian Pilcher) Date: Thu, 8 Mar 2018 17:23:16 -0600 Subject: Spot the invalid syntax Message-ID: (Because I certainly can't.) > def _san_dnsname_ips(dnsname, dnsname_is_cname=False): > """ > Returns a set of IP addresses, managed by this IPa instance, > that correspond to the DNS name (from the subjectAltName). > > """ > fqdn = dnsutil.DNSName(dnsname).make_absolute() > if fqdn.__len__() < 4: > logger.debug("Skipping IPs for %s: hostname too short" % dnsname) > return () > zone = dnsutil.DNSName(resolver.zone_for_name(fqdn)) > name = fqdn.relativize(zone) > try: > result = api.Command['dnsrecord_show'](zone, name)['result'] > except errors.NotFound as nf: > logger.debug("Skipping IPs for %s: %s" % (dnsname, nf)) > return () > ips = set() > for ip in itertools.chain(result.get('arecord', ()), > result.get('aaaarecord', ())): > if _ip_rdns_ok(ip, fqdn): > ips.add(ip) > cnames = result.get('cnamerecord', ()) > if cnames: > if dnsname_is_cname: > logger.debug("Skipping IPs for %s: chained CNAME" % dnsname) > else: > for cname in cnames: > if not cname.endswith('.'): > cname = u'%s.%s' % (cname, zone) > ips.update(_san_dnsname_ips(cname, True) > return ips 2.7 and 3.6 are both giving me: File "/tmp/test.py", line 32 return ips ^ SyntaxError: invalid syntax I've checked for tabs and mismatched parentheses. Aargh! -- ======================================================================== Ian Pilcher arequipeno at gmail.com -------- "I grew up before Mark Zuckerberg invented friendship" -------- ======================================================================== From rosuav at gmail.com Thu Mar 8 18:26:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Mar 2018 10:26:17 +1100 Subject: Spot the invalid syntax In-Reply-To: References: Message-ID: On Fri, Mar 9, 2018 at 10:23 AM, Ian Pilcher wrote: > (Because I certainly can't.) > >> ips.update(_san_dnsname_ips(cname, True) >> return ips > > > 2.7 and 3.6 are both giving me: > > File "/tmp/test.py", line 32 > return ips > ^ > SyntaxError: invalid syntax > > I've checked for tabs and mismatched parentheses. Check the immediately preceding line (the one I quoted). (Sometimes, you just need another pair of eyes.) ChrisA From ben+python at benfinney.id.au Thu Mar 8 18:30:51 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 10:30:51 +1100 Subject: Spot the invalid syntax References: Message-ID: <85ina65ems.fsf@benfinney.id.au> Ian Pilcher writes: > > ips.update(_san_dnsname_ips(cname, True) > > return ips > > 2.7 and 3.6 are both giving me: > > File "/tmp/test.py", line 32 > return ips > ^ > SyntaxError: invalid syntax > > I've checked for tabs and mismatched parentheses. Not sufficiently, it seems. Check the line preceding the ?return? statement. Then, switch to using a programmer's text editor (I prefer Emacs) that can spot these syntax errors while you type. -- \ ?Nothing worth saying is inoffensive to everyone. Nothing worth | `\ saying will fail to make you enemies. And nothing worth saying | _o__) will not produce a confrontation.? ?Johann Hari, 2011 | Ben Finney From ben+python at benfinney.id.au Thu Mar 8 18:33:05 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 10:33:05 +1100 Subject: Spot the invalid syntax References: Message-ID: <85efku5ej2.fsf@benfinney.id.au> Chris Angelico writes: > (Sometimes, you just need another pair of eyes.) In many cases, those eyes can be virtual and non-human. That's what syntax highlighting, and tools even more impressive (e.g. linting tools that run continually), offer in a programmer's text editor: a pair of eyes looking for mistakes while you type. -- \ ?Giving every man a vote has no more made men wise and free | `\ than Christianity has made them good.? ?Henry L. Mencken | _o__) | Ben Finney From rosuav at gmail.com Thu Mar 8 18:42:35 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Mar 2018 10:42:35 +1100 Subject: Spot the invalid syntax In-Reply-To: <85efku5ej2.fsf@benfinney.id.au> References: <85efku5ej2.fsf@benfinney.id.au> Message-ID: On Fri, Mar 9, 2018 at 10:33 AM, Ben Finney wrote: > Chris Angelico writes: > >> (Sometimes, you just need another pair of eyes.) > > In many cases, those eyes can be virtual and non-human. > > That's what syntax highlighting, and tools even more impressive (e.g. > linting tools that run continually), offer in a programmer's text > editor: a pair of eyes looking for mistakes while you type. > Often true, but not always. What would often be useful, but I've not usually (ever?) seen, is a linter rule that continuation lines MUST be indented further than the initial line. That would catch this problem, because the "return" line is being misparsed as a continuation line, but it's indented far less. ChrisA From bob at mellowood.ca Thu Mar 8 18:44:00 2018 From: bob at mellowood.ca (Bob van der Poel) Date: Thu, 8 Mar 2018 16:44:00 -0700 Subject: Spot the invalid syntax In-Reply-To: References: Message-ID: On Thu, Mar 8, 2018 at 4:23 PM, Ian Pilcher wrote: > (Because I certainly can't.) > > def _san_dnsname_ips(dnsname, dnsname_is_cname=False): >> """ >> Returns a set of IP addresses, managed by this IPa instance, >> that correspond to the DNS name (from the subjectAltName). >> >> """ >> fqdn = dnsutil.DNSName(dnsname).make_absolute() >> if fqdn.__len__() < 4: >> logger.debug("Skipping IPs for %s: hostname too short" % dnsname) >> return () >> zone = dnsutil.DNSName(resolver.zone_for_name(fqdn)) >> name = fqdn.relativize(zone) >> try: >> result = api.Command['dnsrecord_show'](zone, name)['result'] >> except errors.NotFound as nf: >> logger.debug("Skipping IPs for %s: %s" % (dnsname, nf)) >> return () >> ips = set() >> for ip in itertools.chain(result.get('arecord', ()), >> result.get('aaaarecord', ())): >> if _ip_rdns_ok(ip, fqdn): >> ips.add(ip) >> cnames = result.get('cnamerecord', ()) >> if cnames: >> if dnsname_is_cname: >> logger.debug("Skipping IPs for %s: chained CNAME" % dnsname) >> else: >> for cname in cnames: >> if not cname.endswith('.'): >> cname = u'%s.%s' % (cname, zone) >> ips.update(_san_dnsname_ips(cname, True) >> return ips >> > > 2.7 and 3.6 are both giving me: > > File "/tmp/test.py", line 32 > return ips > ^ > SyntaxError: invalid syntax > > I've checked for tabs and mismatched parentheses. > > Aargh! > > Check the line before the error. Seems there may be a ')' missing :) > -- > ======================================================================== > Ian Pilcher arequipeno at gmail.com > -------- "I grew up before Mark Zuckerberg invented friendship" -------- > ======================================================================== > > -- > https://mail.python.org/mailman/listinfo/python-list > -- **** Listen to my FREE CD at http://www.mellowood.ca/music/cedars **** Bob van der Poel ** Wynndel, British Columbia, CANADA ** EMAIL: bob at mellowood.ca WWW: http://www.mellowood.ca From ben+python at benfinney.id.au Thu Mar 8 18:57:27 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 10:57:27 +1100 Subject: Spot the invalid syntax References: <85efku5ej2.fsf@benfinney.id.au> Message-ID: <85a7vi5deg.fsf@benfinney.id.au> Chris Angelico writes: > On Fri, Mar 9, 2018 at 10:33 AM, Ben Finney wrote: > > In many cases, those eyes can be virtual and non-human. > > > > That's what syntax highlighting, and tools even more impressive > > (e.g. linting tools that run continually), offer in a programmer's > > text editor: a pair of eyes looking for mistakes while you type. > > Often true, but not always. You mean the tool is not always looking for mistakes while you type? If you mean that the tool doesn't catch all mistakes: of course not, and I didn't imply it would. Are you saying that's a reason against using such automated tools? (If not, I don't really understand what objection you're making.) Certainly it'd be good to always have a *perfect* overseer checking for mistakes while one types. Until that happy day, though, let's use the tools available to improve our code before even attempting to run it. -- \ ?The fact that I have no remedy for all the sorrows of the | `\ world is no reason for my accepting yours. It simply supports | _o__) the strong probability that yours is a fake.? ?Henry L. Mencken | Ben Finney From rosuav at gmail.com Thu Mar 8 19:02:39 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 9 Mar 2018 11:02:39 +1100 Subject: Spot the invalid syntax In-Reply-To: <85a7vi5deg.fsf@benfinney.id.au> References: <85efku5ej2.fsf@benfinney.id.au> <85a7vi5deg.fsf@benfinney.id.au> Message-ID: On Fri, Mar 9, 2018 at 10:57 AM, Ben Finney wrote: > Chris Angelico writes: > >> On Fri, Mar 9, 2018 at 10:33 AM, Ben Finney wrote: >> > In many cases, those eyes can be virtual and non-human. >> > >> > That's what syntax highlighting, and tools even more impressive >> > (e.g. linting tools that run continually), offer in a programmer's >> > text editor: a pair of eyes looking for mistakes while you type. >> >> Often true, but not always. > > You mean the tool is not always looking for mistakes while you type? > > If you mean that the tool doesn't catch all mistakes: of course not, and > I didn't imply it would. Are you saying that's a reason against using > such automated tools? (If not, I don't really understand what objection > you're making.) LOL! Sorry, I derped a bit on the grammar there. Yes, I meant that you can often catch those mistakes with your editor, but not always. There are some errors that cannot possibly be caught by an editor, of course; but there are others that theoretically could be, but I almost never see them. That's the point I was raising - not really an objection, but a segue into my next comment. ChrisA From python at mrabarnett.plus.com Thu Mar 8 19:04:50 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 9 Mar 2018 00:04:50 +0000 Subject: Spot the invalid syntax In-Reply-To: References: Message-ID: <0439799f-4497-9b77-cce9-ebcbc7c36968@mrabarnett.plus.com> On 2018-03-08 23:23, Ian Pilcher wrote: > (Because I certainly can't.) > >> def _san_dnsname_ips(dnsname, dnsname_is_cname=False): >> """ >> Returns a set of IP addresses, managed by this IPa instance, >> that correspond to the DNS name (from the subjectAltName). >> >> """ >> fqdn = dnsutil.DNSName(dnsname).make_absolute() >> if fqdn.__len__() < 4: >> logger.debug("Skipping IPs for %s: hostname too short" % dnsname) >> return () >> zone = dnsutil.DNSName(resolver.zone_for_name(fqdn)) >> name = fqdn.relativize(zone) >> try: >> result = api.Command['dnsrecord_show'](zone, name)['result'] >> except errors.NotFound as nf: >> logger.debug("Skipping IPs for %s: %s" % (dnsname, nf)) >> return () >> ips = set() >> for ip in itertools.chain(result.get('arecord', ()), >> result.get('aaaarecord', ())): >> if _ip_rdns_ok(ip, fqdn): >> ips.add(ip) >> cnames = result.get('cnamerecord', ()) >> if cnames: >> if dnsname_is_cname: >> logger.debug("Skipping IPs for %s: chained CNAME" % dnsname) >> else: >> for cname in cnames: >> if not cname.endswith('.'): >> cname = u'%s.%s' % (cname, zone) >> ips.update(_san_dnsname_ips(cname, True) >> return ips > > 2.7 and 3.6 are both giving me: > > File "/tmp/test.py", line 32 > return ips > ^ > SyntaxError: invalid syntax > > I've checked for tabs and mismatched parentheses. > > Aargh! > Look closely at the preceding line. Count the parentheses. From clear0755 at naver.com Thu Mar 8 19:07:40 2018 From: clear0755 at naver.com (=?utf-8?B?64W47Jew7IiY?=) Date: Fri, 09 Mar 2018 09:07:40 +0900 Subject: =?utf-8?B?SSBhbSBhIHN0dWRlbnQgc3R1ZHlpbmcgUHl0aG9uIGluIEtvcmVhLiBJIGZvdW5kIHN0cg==?= =?utf-8?B?YW5nZSB0aGluZyB3aGlsZSBzdHVkeWluZyB0aHJvdWdoIGlkbGU=?= Message-ID: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> If you type print (" hello\ rpython ") into the python 3.7.0.b2, only the python is printed and i learned it's a crystal. However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 idle, it is output as hellopython. I wonder why it prints like this. I would appreciate your answer. I have attached the file so I would appreciate your reference. From tjreedy at udel.edu Thu Mar 8 19:15:45 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 8 Mar 2018 19:15:45 -0500 Subject: Spot the invalid syntax In-Reply-To: <85ina65ems.fsf@benfinney.id.au> References: <85ina65ems.fsf@benfinney.id.au> Message-ID: On 3/8/2018 6:30 PM, Ben Finney wrote: > Ian Pilcher writes: > >>> ips.update(_san_dnsname_ips(cname, True) | <= auto-indent If you type this code line in IDLE or other decent Python-aware code editor, the smart indent would put curser where I put the '|', thus indicating that there is no ')' matching the function call '(' >>> return ips If you delete the helpful indent signal to type return here, ... >> >> 2.7 and 3.6 are both giving me: >> >> File "/tmp/test.py", line 32 >> return ips >> ^ >> SyntaxError: invalid syntax >> >> I've checked for tabs and mismatched parentheses. IDLE, etc, flashes the ([{ opener for every )]} closer typed. Not seeing an expected flash and seeing an unexpected indent catches most mismatches for me. > Not sufficiently, it seems. Check the line preceding the ?return? > statement. > > Then, switch to using a programmer's text editor (I prefer Emacs) that > can spot these syntax errors while you type. I have never used Emacs, but I understand it has a decent Python-aware mode. -- Terry Jan Reedy From arequipeno at gmail.com Thu Mar 8 19:40:51 2018 From: arequipeno at gmail.com (Ian Pilcher) Date: Thu, 8 Mar 2018 18:40:51 -0600 Subject: Spot the invalid syntax In-Reply-To: References: Message-ID: On 03/08/2018 05:26 PM, Chris Angelico wrote: > On Fri, Mar 9, 2018 at 10:23 AM, Ian Pilcher wrote: >> (Because I certainly can't.) >> >>> ips.update(_san_dnsname_ips(cname, True) >>> return ips >> I've checked for tabs and mismatched parentheses. > > Check the immediately preceding line (the one I quoted). > > (Sometimes, you just need another pair of eyes.) > D'oh! Clearly, my checking needs work. Thanks mucho! -- ======================================================================== Ian Pilcher arequipeno at gmail.com -------- "I grew up before Mark Zuckerberg invented friendship" -------- ======================================================================== From arequipeno at gmail.com Thu Mar 8 19:43:21 2018 From: arequipeno at gmail.com (Ian Pilcher) Date: Thu, 8 Mar 2018 18:43:21 -0600 Subject: Spot the invalid syntax In-Reply-To: <85ina65ems.fsf@benfinney.id.au> References: <85ina65ems.fsf@benfinney.id.au> Message-ID: On 03/08/2018 05:30 PM, Ben Finney wrote: > Not sufficiently, it seems. Check the line preceding the ?return? > statement. Indeed. :-/ > Then, switch to using a programmer's text editor (I prefer Emacs) that > can spot these syntax errors while you type. The sad thing is that I am. Just too bone-headed to use properly, apparently. -- ======================================================================== Ian Pilcher arequipeno at gmail.com -------- "I grew up before Mark Zuckerberg invented friendship" -------- ======================================================================== From python at mrabarnett.plus.com Thu Mar 8 19:47:21 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 9 Mar 2018 00:47:21 +0000 Subject: Spot the invalid syntax In-Reply-To: <85a7vi5deg.fsf@benfinney.id.au> References: <85efku5ej2.fsf@benfinney.id.au> <85a7vi5deg.fsf@benfinney.id.au> Message-ID: <37c22ed8-0728-ec3e-a15d-b22afeb16186@mrabarnett.plus.com> On 2018-03-08 23:57, Ben Finney wrote: > Chris Angelico writes: > >> On Fri, Mar 9, 2018 at 10:33 AM, Ben Finney wrote: >> > In many cases, those eyes can be virtual and non-human. >> > >> > That's what syntax highlighting, and tools even more impressive >> > (e.g. linting tools that run continually), offer in a programmer's >> > text editor: a pair of eyes looking for mistakes while you type. >> >> Often true, but not always. > > You mean the tool is not always looking for mistakes while you type? > > If you mean that the tool doesn't catch all mistakes: of course not, and > I didn't imply it would. Are you saying that's a reason against using > such automated tools? (If not, I don't really understand what objection > you're making.) > > Certainly it'd be good to always have a *perfect* overseer checking for > mistakes . Until that happy day, though, let's use the > tools available to improve our code before even attempting to run it. > "... while one types"? I'm not so sure. In the past I've used an IDE (not for Python) that, for example, underlined any identifiers that hadn't been defined, but found it annoying because it was indicating errors that were due to my not having finished writing yet! Type, pause for thought, syntax error... yes, I know it's a syntax error at the moment, but I'm not finished! From rgaddi at highlandtechnology.invalid Thu Mar 8 20:18:18 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Thu, 8 Mar 2018 17:18:18 -0800 Subject: Unnoticed traceback in a thread (Posting On Python-List Prohibited) In-Reply-To: <03ac0f9b-1f06-44b0-bff0-35a8ce7ba9a8@googlegroups.com> References: <87o9jyi3yw.fsf@nightsong.com> <03ac0f9b-1f06-44b0-bff0-35a8ce7ba9a8@googlegroups.com> Message-ID: On 03/08/2018 05:06 PM, Lawrence D?Oliveiro wrote: > On Friday, March 9, 2018 at 11:43:00 AM UTC+13, Paul Rubin wrote: >> That you're using threading.Lock for something like this is a borderline >> code smell. > > Any use of multithreading in non-CPU-intensive code is a code smell. > But CPU intensive code is exactly the circumstance where Python threading lets you down. It really shines when you're I/O-bound. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From tmrsg11 at gmail.com Thu Mar 8 20:25:42 2018 From: tmrsg11 at gmail.com (C W) Date: Thu, 8 Mar 2018 20:25:42 -0500 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: Thank you guys, lots of great answers, very helpful. I got it! A follow-up question: How did the value of "object" get passed to "time"? Obviously, they have different names. How did Python make that connection? Code is below for convenience. class Clock(object): def __init__(self, time): self.time = time def print_time(self): time = '6:30' print(self.time) clock = Clock('5:30') clock.print_time() 5:30 Thank you! On Thu, Mar 8, 2018 at 6:30 AM, Steven D'Aprano < steve+comp.lang.python at pearwood.info> wrote: > On Wed, 07 Mar 2018 16:57:51 -0500, C W wrote: > > > Hello, > > > > I am new to OOP. I'm a bit confused about the following code. > > > > class Clock(object): > > def __init__(self, time): > > self.time = time > > Here you set the instance attribute "self.time". > > > def print_time(self): > > time = '6:30' > > print(self.time) > > Here you set the local variable "time", which is completely unrelated to > the attribute "self.time". > > If you are used to languages where "foo" inside a method is a short-cut > for "self.foo" or "this.foo", Python does not do that. Local variables > and instance attributes are distinct concepts, and Python keeps them > distinct. > > > > How does line-by-line execution run inside a frame? > > There isn't actually line-by-line execution as such, although it can be > very similar. Before the interpreter runs Python code, it compiles it to > byte-code, and then runs the byte-code. A single line of source code > could result in any number of lines of byte code, from zero to an > unlimited number. > > > How does __init__ > > work? I understand you must have __init__. > > You understand wrongly then :-) > > It is normal and common to have an __init__ method, but it is not > compulsory. If your class doesn't need one, you don't need to write it. > > The __init__ method is the initialiser. Think of it as very similar to > the constructor in some other languages, and for now the differences > aren't important. The usual purpose of the __init__ method is to > initialise the instance and set any attributes needed. > > > Is it run before print_time(), > > The __init__ method is called once, when the instance is first created. > So the short answer is, yes, it will run before print_time(). But only > once. > > > if so, why don't I just set self.time = '6:30' instead of > > self.time = time? > > Because then every instance will be set to 6:30, instead of letting you > set each instance to a different time. > > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list > From ben+python at benfinney.id.au Thu Mar 8 20:47:18 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 12:47:18 +1100 Subject: Relationship between =?utf-8?B?4oCYb2JqZWN04oCZ?= and the name of an object (was: Which part of the loop is it going through in this class frame?) References: Message-ID: <85606658bd.fsf_-_@benfinney.id.au> C W writes: > A follow-up question: (When you want to switch the discussion to a different subject, please indicate that by changing the Subject field in your message. I am not really clear on what you're asking, so I have guessed and made a subject that I think summarises what you mean.) > How did the value of "object" get passed to "time"? Obviously, they > have different names. How did Python make that connection? In the context of your original question, I think this one reveals that you are missing some fundamental knowledge of how Python treats objects and references. Every object has a type. Types are themselves objects; the most basic type, in Python, is named ?object?. Yes, already this makes it somewhat difficult to discuss these concepts :-) You will probably benefit greatly by reading and watching this presentation by Ned Batchelder, that walks you through some common minuderstandings, and replaces them with correct concepts for how Python treats objects and references. Facts and Myths about Names and Values -- \ ?If you ever fall off the Sears Tower, just go real limp, | `\ because maybe you'll look like a dummy and people will try to | _o__) catch you because, hey, free dummy.? ?Jack Handey | Ben Finney From ben+python at benfinney.id.au Thu Mar 8 20:59:52 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 12:59:52 +1100 Subject: I found strange thing while studying through idle References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> Message-ID: <851sgu57qf.fsf@benfinney.id.au> Welcome, and congratulations on beginning with Python. ??? writes: > If you type print (" hello\ rpython ") into the python 3.7.0.b2 I am not using Python 3.7 (it isn't released yet); I recommend staying with the latest Python release. Today, that is version 3.6. That difference in version should not make a difference to the ?print? behaviour, so I will assume your code behaves the same in Python 3.6. When I use Python 3.6 and run the code example you give, it behaves this way:: >>> print (" hello\ rpython ") hello\ rpython > only the python is printed and i learned it's a crystal. (I don't understand that last clause; what crystal?) The output above is what I expect. What do you expect? > However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 > idle, it is output as hellopython. That's not what I see; the output in Idle version 3.6 is exactly the same:: >>> print (" hello\ rpython ") hello\ rpython > I wonder why it prints like this. So do I. Please take the time to cut and paste *exactly* what program code is being run, and *exactly* what text output you see. Don't attempt to re-type it manually, because often that makes it different in detail. One of the tricky things to learn in programming is that small details can make a big difference! > I have attached the file so I would appreciate your reference. File attachments are unlikely to survive to most readers of this forum (they get dropped, to reduce traffic size and spam and dangerous attachments). Instead, keep the program code short and simple; then, copy and paste it exactly into the text body of your message. That's much more likely to let us see exactly what you're seeing. -- \ ?Better not take a dog on the space shuttle, because if he | `\ sticks his head out when you're coming home his face might burn | _o__) up.? ?Jack Handey | Ben Finney From skip.montanaro at gmail.com Thu Mar 8 21:27:47 2018 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Fri, 09 Mar 2018 02:27:47 +0000 Subject: Futurize maps StringIO.StringIO to io.StringIO In-Reply-To: References: Message-ID: Thanks. In one instance I used six.StringIO, which seems to work. The question I had was more why futurize makes a transformation which is an obvious semantic change. Skip On Thu, Mar 8, 2018, 7:55 PM Mark Lawrence wrote: > On 08/03/18 02:34, Skip Montanaro wrote: > > I had some deja vu recently, cuz of this old thread: > > > > https://mail.python.org/pipermail/python-list/2013-May/648245.html > > > > Today the context is different, but the problem remains the same. One > > of the futurize refactoring tools converts usage of StringIO.StringIO > > to io.StringIO, which, given that the latter accepts just about > > anything you through at it, is a bit surprising. Perhaps there is no > > true corresponding class in the io module? > > > > Skip > > > > Maybe you need io.BytesIO? > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > -- > https://mail.python.org/mailman/listinfo/python-list > From steve+comp.lang.python at pearwood.info Thu Mar 8 22:57:53 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 9 Mar 2018 03:57:53 +0000 (UTC) Subject: I found strange thing while studying through idle References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: I'm afraid the original post by ??? has not come through to me, so I will have to reply to Ben's reply. On Fri, 09 Mar 2018 12:59:52 +1100, Ben Finney wrote: > I am not using Python 3.7 (it isn't released yet); I recommend staying > with the latest Python release. Today, that is version 3.6. 3.6 is the latest *stable* release; but there are unstable releases, such as 3.7.0.b2. We should be glad that there are people willing to run unstable and beta versions, otherwise they would not get any real-world testing, bugs would not be fixed, and the "stable" X.Y.0 release would be a de-facto unstable, untried beta version. > When I use Python 3.6 and run the code example you give, it behaves this > way:: > > >>> print (" hello\ rpython ") > hello\ rpython My guess is that the Original Poster carelessly re-typed his code instead of copying and pasting it. When he said he tried this: print (" hello\ rpython ") what he *actually* did was probably something like this: print ("hello\rpython") When I try that, I get this: py> print ("hello \rpython") python which seems to match the OP's description. Printing strings containing \r (carriage returns) may be dependent on the console or terminal you use, and even on *where* the carriage return is in the string, but in general I expect that printing \r will instruct the terminal to return to the beginning of the CURRENT line. The rest of the string will then overwrite the beginning of the line. For example: py> print ("hello world\rpython") pythonworld >> However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 >> idle, it is output as hellopython. Again, my prediction is that the OP has carelessly retyped his code, and what he means is print ("hello \rpython") which in IDLE *does not* return to the start of the line. In IDLE 3.5 on Linux, I get this: >>> print('hello\rpython') hello\rpython Curiously, that's not a backslash r, it's actually a carriage return: when I copy and paste it in this text, the editor treated it as a new line character: # direct copy and paste becomes this in my editor >>> print('hello\rpython') hello python But it is possible that due to differences between platforms, the OP's version of IDLE doesn't display a carriage return as \r but rather as an invisible zero-width space. -- Steve From python at mrabarnett.plus.com Thu Mar 8 23:11:25 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 9 Mar 2018 04:11:25 +0000 Subject: I found strange thing while studying through idle In-Reply-To: <851sgu57qf.fsf@benfinney.id.au> References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: <12c8aed3-3511-7f75-29c8-d24abfcda7b7@mrabarnett.plus.com> On 2018-03-09 01:59, Ben Finney wrote: > Welcome, and congratulations on beginning with Python. > > ??? writes: > >> If you type print (" hello\ rpython ") into the python 3.7.0.b2 > > I am not using Python 3.7 (it isn't released yet); I recommend staying > with the latest Python release. Today, that is version 3.6. > > That difference in version should not make a difference to the ?print? > behaviour, so I will assume your code behaves the same in Python 3.6. > > When I use Python 3.6 and run the code example you give, it behaves this > way:: > > >>> print (" hello\ rpython ") > hello\ rpython > >> only the python is printed and i learned it's a crystal. > > (I don't understand that last clause; what crystal?) > > The output above is what I expect. What do you expect? > >> However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 >> idle, it is output as hellopython. > > That's not what I see; the output in Idle version 3.6 is exactly the > same:: > > >>> print (" hello\ rpython ") > hello\ rpython > >> I wonder why it prints like this. > > So do I. > > Please take the time to cut and paste *exactly* what program code is > being run, and *exactly* what text output you see. > > Don't attempt to re-type it manually, because often that makes it > different in detail. One of the tricky things to learn in programming is > that small details can make a big difference! > >> I have attached the file so I would appreciate your reference. > > File attachments are unlikely to survive to most readers of this forum > (they get dropped, to reduce traffic size and spam and dangerous > attachments). > > Instead, keep the program code short and simple; then, copy and paste it > exactly into the text body of your message. That's much more likely to > let us see exactly what you're seeing. > I think the line was actually: print (" hello\rpython ") and the question is why you get: python in a command prompt window, but: hellopython in IDLE. (I also think that "crystal" is a translation error! Is it meant to be "control code"?) From python at mrabarnett.plus.com Thu Mar 8 23:16:47 2018 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 9 Mar 2018 04:16:47 +0000 Subject: I found strange thing while studying through idle In-Reply-To: References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: <6ab71842-f08b-bd82-012e-0351ff592c80@mrabarnett.plus.com> On 2018-03-09 03:57, Steven D'Aprano wrote: [snip] > In IDLE 3.5 on Linux, I get this: > >>>> print('hello\rpython') > hello\rpython > > Curiously, that's not a backslash r, it's actually a carriage return: > when I copy and paste it in this text, the editor treated it as a new > line character: > > # direct copy and paste becomes this in my editor >>>> print('hello\rpython') > hello > python > > > But it is possible that due to differences between platforms, the OP's > version of IDLE doesn't display a carriage return as \r but rather as an > invisible zero-width space. > IDLE 3.6 on Windows, it looks like this: hellopython but when I copy and paste it into an editor I get: hello python From steve+comp.lang.python at pearwood.info Thu Mar 8 23:18:45 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 9 Mar 2018 04:18:45 +0000 (UTC) Subject: Spot the invalid syntax References: <85efku5ej2.fsf@benfinney.id.au> <85a7vi5deg.fsf@benfinney.id.au> <37c22ed8-0728-ec3e-a15d-b22afeb16186@mrabarnett.plus.com> Message-ID: On Fri, 09 Mar 2018 00:47:21 +0000, MRAB wrote: > On 2018-03-08 23:57, Ben Finney wrote: >> You mean the tool is not always looking for mistakes while you type? [...] >> Certainly it'd be good to always have a *perfect* overseer checking for >> mistakes . Until that happy day, though, let's use the tools available >> to improve our code before even attempting to run it. >> > "... while one types"? I'm not so sure. > > In the past I've used an IDE (not for Python) that, for example, > underlined any identifiers that hadn't been defined, but found it > annoying because it was indicating errors that were due to my not having > finished writing yet! Aye, I find that error-checking while you type is a PITA. I tried using the Spyder IDE for a week or so, it was just so painful to use I couldn't keep going. I'm not the world's fastest typest, so the fact that Spyder couldn't keep up with me was a serious black mark against it. There was visible lag between me typing and the characters showing up in the editor: I could type something like "for x in" before the "f" would appear. Errors were highlighted in (I think?) red, and by displaying an icon next to the offending line. So as I would type, the screen would be constantly updated, slowly: # me trying to type "print(mylist)" p pr # highlight in red and show the error icon pri # highlight in red and show the error icon prin # highlight in red and show the error icon print # remove the highlighting and icon print( # display an error icon for missing parenthesis print() # okay, I'll type the paren first to keep the checker silent print(m) print(my) # highlight the my in red and show the error icon [skip a few more keystrokes] print(mylist) # finally remove the error highlighting And because of the lag, it took about 15-20 seconds for the screen to update what took me 4-6 seconds to type. This experience was so painful and unpleasant that it has permanently turned me off Spyder. I'd rather use Windows 95 Notepad. > Type, pause for thought, syntax error... yes, I know it's a syntax error > at the moment, but I'm not finished! Not even "pause for thought". I can *maybe* see the sense in checking each physical line once it is completed, but that's it. Or at least don't check words for mispellings until you have completed the token and started a new token, like your spell checker probably does. (The spell checker in this editor doesn't check for spelling errors until I hit the spacebar or Enter key, or use punctuation.) -- Steve From steve+comp.lang.python at pearwood.info Thu Mar 8 23:28:48 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 9 Mar 2018 04:28:48 +0000 (UTC) Subject: Which part of the loop is it going through in this class frame? References: Message-ID: On Thu, 08 Mar 2018 20:25:42 -0500, C W wrote: > Thank you guys, lots of great answers, very helpful. I got it! > > A follow-up question: > > How did the value of "object" get passed to "time"? Obviously, they have > different names. How did Python make that connection? It didn't. You have misunderstood what is happening. Let's go though it bit by bit: > Code is below for convenience. > > class Clock(object): The "class" statement declares a new class, called "Clock", which inherits from the built-in class "object". [Aside: some people describe "object" as a "type" rather than a class. There are some differences in meaning between type/class in computer science, and in Python 2 they are slightly different things, but in Python 3 you can consider class and type to be synonyms.] So object is not a parameter, it is a superclass. This tells Python that your Clock class is a subclass of object. > def __init__(self, time): > self.time = time The initialiser __init__ method is special, because it defines the signature for calling the class. So when you say: clock = Clock("5:30") Python creates a new Clock instance, and calls __init__ and passes "5:30" as the *time* parameter. (I have glossed over some technical details and complexities.) The thing to remember is that when you create a new instance by calling the class Clock(...), your arguments have to match the __init__ method. -- Steve From ben+python at benfinney.id.au Fri Mar 9 00:33:16 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 09 Mar 2018 16:33:16 +1100 Subject: I found strange thing while studying through idle References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> <12c8aed3-3511-7f75-29c8-d24abfcda7b7@mrabarnett.plus.com> Message-ID: <85sh994xur.fsf@benfinney.id.au> MRAB writes: > I think the line was actually [?] > (I also think that "crystal" is [?] Let's not get into the warrens of responding to what the OP *didn't* write. we're in no hurry. I'd like to wait for clarification from the original poster, and not guess what they meant. -- \ ?The Vatican is not a state.? a state must have people. There | `\ are no Vaticanians.? No-one gets born in the Vatican except by | _o__) an unfortunate accident.? ?Geoffrey Robertson, 2010-09-18 | Ben Finney From cs at cskk.id.au Fri Mar 9 01:33:08 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Fri, 9 Mar 2018 17:33:08 +1100 Subject: Which part of the loop is it going through in this class frame? In-Reply-To: References: Message-ID: <20180309063308.GA17484@cskk.homeip.net> On 08Mar2018 20:25, C W wrote: >Thank you guys, lots of great answers, very helpful. I got it! > >A follow-up question: > >How did the value of "object" get passed to "time"? Obviously, they have >different names. How did Python make that connection? > >Code is below for convenience. > >class Clock(object): > def __init__(self, time): > self.time = time > def print_time(self): > time = '6:30' > print(self.time) > >clock = Clock('5:30') >clock.print_time() >5:30 "object" isn't really passed anywhere, except to your class construction of "Clock". It hasn't any relation to the name "time" at all. In Python, all classes are subclasses (directly or not) of the base class called "object", and your class definition makes that explicit here: class Clock(object): This defines your class as just like the "object" class, but with the following extra features, which you go on the define below it. With "new style classes" (a Python notion, not something special about OOP in general) the "object" base class is implicit, so you could just as well have written: class Clock: to achieve the same purpose. Being a subclass of "object" gets you an assortment of methods immediately, such as __str__ and __repr__ (used to present a class as a string). You even get an __init__ for free, but the default does nothing and that is usually not what you need. Cheers, Cameron Simpson (formerly cs at zip.com.au) From dieter at handshake.de Fri Mar 9 01:36:39 2018 From: dieter at handshake.de (dieter) Date: Fri, 09 Mar 2018 07:36:39 +0100 Subject: Module Issue References: Message-ID: <87k1ulzrew.fsf@handshake.de> Abdur-Rahmaan Janhangeer writes: > i have a project at > > https://github.com/Abdur-rahmaanJ/honeybot > > see https://github.com/Abdur-rahmaanJ/honeybot/tree/master/honeybot > > my question is : > > how to include a util file / module that can be imported in both > core_plugins and > user_plugins? > > if that is too difficult, let us take only core_plugins > > make a module not found error not appear when running main.py Have a look at so called "namespace package"s. A "namespace package" is in some sense a "virtual package" consisting of an extensible set of subpackages (real and potentially other "namespace" packages). The subpackages may declare dependencies on other subpackages of the "namespace package" and thereby ensure, that the subset they depend on are installed if they are installed. "dm.zope.rpc" is an example for such a "namespace package". It contains common (= cross protocol) RPC (= "Remote Procedure Call") infrastructure (and some protocol plugins); "dm.zope.rpc.wsdl_suds" is a subpackage providing support for the SOAP/WSDL protocol (using "suds" as basic implementation component). From dieter at handshake.de Fri Mar 9 01:45:54 2018 From: dieter at handshake.de (dieter) Date: Fri, 09 Mar 2018 07:45:54 +0100 Subject: socket: Did I find a bug? References: <4f307ac2-83ff-e3b3-fb38-70e46196e10e@vub.be> Message-ID: <87fu59zqzh.fsf@handshake.de> Antoon Pardon writes: > This is on a debian 9 box python 2.7.13 > > My interpretation is that a timeout exception is thrown and that the > args attribute of such an exception is an empty tuple which then causes > an IndexError in line 482 of module /usr/lib/python2.7/socket.py. Does > that soundplausible? Yes -- however, usually the "args" tuple of an exception is not empty. I am using the FTP class in Python 2.7 myself and I am quite (though not completely) sure that I already have had timeouts in my FTP interaction -- without seeing your problem. Nevertheless, you might see a bug: there are two kinds of timeouts: connection and read timeout. You are facing a read timeout (if you see a timeout at all); while I likely saw connection timeouts. However, the resulting timeout exception should in both cases be identical with respect to the "args" tuple. > Here is the traceback: > > Traceback (most recent call last): > ... > File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 211, in open > return ftpfile(ftp, fn, mode, True) > File "/usr/local/lib/python-apps/rmtdump/ftputil.py", line 70, in __init__ > self.cnct = self.ftp.transfercmd("%s %s" % (cmd, rfn)) > File "/usr/lib/python2.7/ftplib.py", line 376, in transfercmd > return self.ntransfercmd(cmd, rest)[0] > File "/usr/lib/python2.7/ftplib.py", line 710, in ntransfercmd > conn, size = FTP.ntransfercmd(self, cmd, rest) > File "/usr/lib/python2.7/ftplib.py", line 339, in ntransfercmd > resp = self.sendcmd(cmd) > File "/usr/lib/python2.7/ftplib.py", line 249, in sendcmd > return self.getresp() > File "/usr/lib/python2.7/ftplib.py", line 215, in getresp > resp = self.getmultiline() > File "/usr/lib/python2.7/ftplib.py", line 201, in getmultiline > line = self.getline() > File "/usr/lib/python2.7/ftplib.py", line 186, in getline > line = self.file.readline(self.maxline + 1) > File "/usr/lib/python2.7/socket.py", line 482, in readline > if e.args[0] == EINTR: > IndexError: tuple index out of range From dieter at handshake.de Fri Mar 9 01:55:14 2018 From: dieter at handshake.de (dieter) Date: Fri, 09 Mar 2018 07:55:14 +0100 Subject: Unnoticed traceback in a thread References: Message-ID: <87bmfxzqjx.fsf@handshake.de> Skip Montanaro writes: > I have a program which is almost always running in a single thread. It > uses a daemon thread (via threading.Timer) to periodically checkpoint > some state. The program runs for days at a time. > > Over the past couple days, two instances of the subthread croaked with > tracebacks because while they were iterating over the checkpointable > data, the main thread modified the data. Blammo! It was an easy fix > (threading.Lock) and everything is back on an even keel. > > After the bug was tickled, the program continued to do its thing. It > just stopped checkpointing. Given the way the program is managed, the > traceback wound up in an obscure log file, and it was a couple days > before I noticed it. I've seen this sort of > thread-dies-but-program-doesn't-crash situation before. I can look at > ways to more actively monitor the contents of the obscure log file, > but is there some non-hackish way that the demise of the daemon thread > can take down the entire program? (Sometimes it's good that it doesn't > crash. Other times it would at least be handy if it did.) I approach situations like this by running the thread function inside a "try: ... except: ..." block. In the "except" handler, I can then so whatever is necessary if the thread function has died unexpectedly -- e.g. kill the complete process. Concretely: instead of "start_new_thread(my_thread_function, ...)", I use def wrapped_thread_function(*args, **kw): try: my_thread_function(*args, **kw) except: ... do whatever is necessary should "my_thread_function" fails ... start_new_thread(wrapped_thread_function, ...) Similar, should you use the "Thread" class (instead of "start_new_thread"). From stefan_ml at behnel.de Fri Mar 9 04:22:23 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 9 Mar 2018 10:22:23 +0100 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: Andrew Z schrieb am 07.03.2018 um 05:03: > Hello, > with 3.6 and latest greatest lxml: > > from lxml import etree > > tree = etree.parse('Sample.xml') > etree.register_namespace('','http://www.example.com') The default namespace prefix is spelled None (because there is no prefix for it) and not the empty string. > causes: > Traceback (most recent call last): > File "/home/az/Work/flask/tutorial_1/src/xml_oper.py", line 16, in > > etree.register_namespace('','http://www.example.com') > File "src/lxml/etree.pyx", line 203, in lxml.etree.register_namespace > (src/lxml/etree.c:11705) > File "src/lxml/apihelpers.pxi", line 1631, in lxml.etree._tagValidOrRaise > (src/lxml/etree.c:35382) > ValueError: Invalid tag name '' > > partial Sample.xml: > > > > > md_status_nonpro="true" type="INDIVIDUAL" prefix="jadoe"> > > > > it seems to not be happy with the empty tag . > But i'm not sure why and how to go about it. Could you explain why you want to do that? Stefan From steve+comp.lang.python at pearwood.info Fri Mar 9 06:41:01 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 9 Mar 2018 11:41:01 +0000 (UTC) Subject: LXML: can't register namespace References: Message-ID: On Fri, 09 Mar 2018 10:22:23 +0100, Stefan Behnel wrote: > Andrew Z schrieb am 07.03.2018 um 05:03: >> Hello, >> with 3.6 and latest greatest lxml: >> >> from lxml import etree >> >> tree = etree.parse('Sample.xml') >> etree.register_namespace('','http://www.example.com') > > The default namespace prefix is spelled None (because there is no prefix > for it) and not the empty string. Is that documented somewhere? Is there a good reason not to support "" as the empty prefix? -- Steve From stefan_ml at behnel.de Fri Mar 9 07:08:10 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 9 Mar 2018 13:08:10 +0100 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: Steven D'Aprano schrieb am 09.03.2018 um 12:41: > On Fri, 09 Mar 2018 10:22:23 +0100, Stefan Behnel wrote: > >> Andrew Z schrieb am 07.03.2018 um 05:03: >>> Hello, >>> with 3.6 and latest greatest lxml: >>> >>> from lxml import etree >>> >>> tree = etree.parse('Sample.xml') >>> etree.register_namespace('','http://www.example.com') >> >> The default namespace prefix is spelled None (because there is no prefix >> for it) and not the empty string. > > Is that documented somewhere? http://lxml.de/tutorial.html#namespaces > Is there a good reason not to support "" as the empty prefix? Well, the "empty prefix" is not an "empty" prefix, it's *no* prefix. The result is not ":tag" instead of "prefix:tag", the result is "tag". But even ignoring that difference, why should the API support two ways of spelling the same thing, and thus encourage users to write diverging code? Stefan From steve+comp.lang.python at pearwood.info Fri Mar 9 07:59:27 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 9 Mar 2018 12:59:27 +0000 (UTC) Subject: LXML: can't register namespace References: Message-ID: On Fri, 09 Mar 2018 13:08:10 +0100, Stefan Behnel wrote: >> Is there a good reason not to support "" as the empty prefix? > > Well, the "empty prefix" is not an "empty" prefix, it's *no* prefix. The > result is not ":tag" instead of "prefix:tag", the result is "tag". That makes sense, thanks. -- Steve From __peter__ at web.de Fri Mar 9 08:11:41 2018 From: __peter__ at web.de (Peter Otten) Date: Fri, 09 Mar 2018 14:11:41 +0100 Subject: LXML: can't register namespace References: Message-ID: Stefan Behnel wrote: > Andrew Z schrieb am 07.03.2018 um 05:03: >> Hello, >> with 3.6 and latest greatest lxml: >> >> from lxml import etree >> >> tree = etree.parse('Sample.xml') >> etree.register_namespace('','http://www.example.com') > > The default namespace prefix is spelled None (because there is no prefix > for it) and not the empty string. Does that mean the OP shouldn't use register_namespace() at all or that he's supposed to replace "" with None? If the latter -- it looks like None accepted either: (lxml_again)$ python Python 3.4.3 (default, Nov 28 2017, 16:41:13) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from lxml import etree >>> etree.register_namespace(None, "http://www.example.com") Traceback (most recent call last): File "", line 1, in File "src/lxml/etree.pyx", line 200, in lxml.etree.register_namespace (src/lxml/etree.c:11612) File "src/lxml/apihelpers.pxi", line 1442, in lxml.etree._utf8 (src/lxml/etree.c:32933) TypeError: Argument must be bytes or unicode, got 'NoneType' >>> etree.__version__ '4.1.1' From stefan_ml at behnel.de Fri Mar 9 09:46:06 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 9 Mar 2018 15:46:06 +0100 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: Peter Otten schrieb am 09.03.2018 um 14:11: > Stefan Behnel wrote: > >> Andrew Z schrieb am 07.03.2018 um 05:03: >>> Hello, >>> with 3.6 and latest greatest lxml: >>> >>> from lxml import etree >>> >>> tree = etree.parse('Sample.xml') >>> etree.register_namespace('','http://www.example.com') >> >> The default namespace prefix is spelled None (because there is no prefix >> for it) and not the empty string. > > Does that mean the OP shouldn't use register_namespace() at all or that he's > supposed to replace "" with None? It meant neither of the two, but now that you ask, I would recommend the first. ;) An application global setup for the default namespace is never a good idea, thus my question regarding the actual intention of the OP. Depending on the context, the right thing to do might be be to either not care at all, or to not use the default namespace but a normally prefixed one instead, or to define a (default) namespace mapping for a newly created tree, as shown in the namespace tutorial. http://lxml.de/tutorial.html#namespaces Usually, not caring about namespace prefixes is the best approach. Parsers, serialisers and compressors can deal with them perfectly and safely, humans should just ignore the clutter, pitfalls and complexity that they introduce. Stefan From rgaddi at highlandtechnology.invalid Fri Mar 9 12:46:02 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Fri, 9 Mar 2018 09:46:02 -0800 Subject: I found strange thing while studying through idle In-Reply-To: References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: On 03/08/2018 07:57 PM, Steven D'Aprano wrote: > [snip] > > But it is possible that due to differences between platforms, the > OP's version of IDLE doesn't display a carriage return as \r but > rather as an invisible zero-width space. > Just to derail this conversation a bit, does anyone have a use case in the modern (Py3) age for '\r'? I use b'\r' fairly regularly when talking to serial port devices. But the string version? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From p.f.moore at gmail.com Fri Mar 9 13:10:57 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 9 Mar 2018 18:10:57 +0000 Subject: I found strange thing while studying through idle In-Reply-To: References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: On 9 March 2018 at 17:46, Rob Gaddi wrote: > On 03/08/2018 07:57 PM, Steven D'Aprano wrote: >> >> [snip] >> >> But it is possible that due to differences between platforms, the >> OP's version of IDLE doesn't display a carriage return as \r but >> rather as an invisible zero-width space. >> > > Just to derail this conversation a bit, does anyone have a use case in the > modern (Py3) age for '\r'? I use b'\r' fairly regularly when talking to > serial port devices. But the string version? It's fairly common for backing up and rewriting a progress line in simple console programs: for i in range(100): print(f"\r \rCompleted: {i}%", end='') sleep(0.5) Paul From rosuav at gmail.com Fri Mar 9 13:18:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Mar 2018 05:18:11 +1100 Subject: I found strange thing while studying through idle In-Reply-To: References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> <851sgu57qf.fsf@benfinney.id.au> Message-ID: On Sat, Mar 10, 2018 at 5:10 AM, Paul Moore wrote: > On 9 March 2018 at 17:46, Rob Gaddi wrote: >> On 03/08/2018 07:57 PM, Steven D'Aprano wrote: >>> >>> [snip] >>> >>> But it is possible that due to differences between platforms, the >>> OP's version of IDLE doesn't display a carriage return as \r but >>> rather as an invisible zero-width space. >>> >> >> Just to derail this conversation a bit, does anyone have a use case in the >> modern (Py3) age for '\r'? I use b'\r' fairly regularly when talking to >> serial port devices. But the string version? > > It's fairly common for backing up and rewriting a progress line in > simple console programs: > > for i in range(100): > print(f"\r \rCompleted: {i}%", end='') > sleep(0.5) > Yep, though I prefer end="\r" (or end="\33[K\r" if you can use ANSI codes) rather than putting the \r at the beginning. It looks nicer to have the cursor at the end of the line (which your way does, but mine doesn't), but on the other hand, leaving the cursor at the beginning means that any output lines (ending with a normal \n) will more cleanly overwrite the completion percentage line. It's tradeoffs all around, as always. But yes, that's one VERY common use of \r. ChrisA From kwpolska at gmail.com Fri Mar 9 14:49:07 2018 From: kwpolska at gmail.com (Chris Warrick) Date: Fri, 9 Mar 2018 20:49:07 +0100 Subject: I am a student studying Python in Korea. I found strange thing while studying through idle In-Reply-To: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> Message-ID: On 9 March 2018 at 01:07, ??? wrote: > If you type print (" hello\ rpython ") into the python 3.7.0.b2, only the python is printed and i learned it's a crystal. However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 idle, it is output as hellopython. I wonder why it prints like this. I would appreciate your answer. > > I have attached the file so I would appreciate your reference. > -- > https://mail.python.org/mailman/listinfo/python-list In both cases, 'hellopython' is printed, only the behavior of the cursor changes. The `\r` means ?move cursor to the start of the line? in some places, including Terminal/Command Prompt. But not everyone processes the backspace character ? IDLE ignores it, as do many other text editors. Another important thing to note about \r is this: >>> print("python\rhi") hithon (PS. it?s better to use a stable version, especially when you?re learning. PPS. file attachments do not work on this list.) -- Chris Warrick PGP: 5EAAEA16

On 9 March 2018 at 01:07, ??? <clear0755 at naver.com> wrote:
If you type print (" hello\ rpython ") into the python 3.7.0.b2, only the python is printed and i learned it's a crystal. However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 idle, it is output as hellopython. I wonder why it prints like this. I would appreciate your answer.

I have attached the file so I would appreciate your reference.
--
https://mail.python.org/mailman/listinfo/python-list



--
Chris Warrick <https://chriswarrick.com/>
PGP: 5EAAEA16
From tjreedy at udel.edu Fri Mar 9 16:03:48 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 9 Mar 2018 16:03:48 -0500 Subject: I am a student studying Python in Korea. I found strange thing while studying through idle In-Reply-To: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> References: <5fde72403488bf1016bffb5a4b5c12a7@cweb08.nm.nhnsystem.com> Message-ID: On 3/8/2018 7:07 PM, ??? wrote: > If you type print (" hello\ rpython ") into the python 3.7.0.b2, only the python is printed and i learned it's a crystal. '\r' is a control character than means 'return to the beginning of the line'. When you execute "print('hello\rpython')" in Python running in an operating system console, the effect of '\r' depends on the console. If the console tries to emulate terminals which moved the cursor or printing head in response to '\r', then that is what happens. The effect of various control codes depends on the terminal emulation and can be different on different systems. > However, if you type print (" hello\ rpython ") in the python 3.7.0.b2 idle, it is output as hellopython. IDLE is a cross-platform development environment and its shell is not a terminal emulator. Requests to imitate particular terminals/consoles have been rejected. In the case of \r and backspace, it was felt that it is better to not erase what has been printed. If one wants to know how a program will operate in a particular console, one should test it there. > I have attached the file. Don't do this. Copy and paste relevant snippets of code into the message. Attachments are ignored and not sent to subscribers or mirrors. - Terry Jan Reedy From steve+comp.lang.python at pearwood.info Fri Mar 9 20:13:04 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 10 Mar 2018 01:13:04 +0000 (UTC) Subject: Enumerating all 3-tuples Message-ID: I am trying to enumerate all the three-tuples (x, y, z) where each of x, y, z can range from 1 to ? (infinity). This is clearly unhelpful: for x in itertools.count(1): for y in itertools.count(1): for z in itertools.count(1): print(x, y, z) as it never advances beyond x=1, y=1 since the innermost loop never finishes. Georg Cantor to the rescue! (Well, almost...) https://en.wikipedia.org/wiki/Pairing_function The Russian mathematician Cantor came up with a *pairing function* that encodes a pair of integers into a single one. For example, he maps the coordinate pairs to integers as follows: 1,1 -> 1 2,1 -> 2 1,2 -> 3 3,1 -> 4 2,2 -> 5 and so forth. He does this by writing out the coordinates in a grid: 1,1 1,2 1,3 1,4 ... 2,1 2,2 2,3 2,4 ... 3,1 3,2 3,3 3,4 ... 4,1 4,2 4,3 4,4 ... ... and then iterating over them along the diagonals, starting from the top corner. That's just what I'm after, and I have this function that works for 2-tuples: def cantor(start=0): """Yield coordinate pairs using Cantor's Pairing Function. Yields coordinate pairs in (Z*,Z*) over the diagonals: >>> it = cantor() >>> [next(it) for _ in range(10)] [(0,0), (1,0), (0,1), (2,0), (1,1), (0,2), (3,0), (2,1), (1,2), (0,3)] If ``start`` is given, it is used as the first x- and y-coordinate. """ i = start while True: for j in range(start, i+1): yield (i-j+start, j) i += 1 But I've stared at this for an hour and I can't see how to extend the result to three coordinates. I can lay out a grid in the order I want: 1,1,1 1,1,2 1,1,3 1,1,4 ... 2,1,1 2,1,2 2,1,3 2,1,4 ... 1,2,1 1,2,2 1,2,3 1,2,4 ... 3,1,1 3,1,2 3,1,3 3,1,4 ... 2,2,1 2,2,2 2,2,3 2,2,4 ... ... and applying Cantor's diagonal order will give me what I want, but damned if I can see how to do it in code. Clearly I'm having a "cannot brain today" moment. Can anyone help me out here? -- Steve From bc at freeuk.com Fri Mar 9 21:00:48 2018 From: bc at freeuk.com (bartc) Date: Sat, 10 Mar 2018 02:00:48 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: Message-ID: <3%GoC.115966$856.31339@fx42.am4> On 10/03/2018 01:13, Steven D'Aprano wrote: > I am trying to enumerate all the three-tuples (x, y, z) where each of x, > y, z can range from 1 to ? (infinity). > > This is clearly unhelpful: > > for x in itertools.count(1): > for y in itertools.count(1): > for z in itertools.count(1): > print(x, y, z) > > as it never advances beyond x=1, y=1 since the innermost loop never > finishes. > > Georg Cantor to the rescue! (Well, almost...) > > https://en.wikipedia.org/wiki/Pairing_function > > The Russian mathematician Cantor came up with a *pairing function* that > encodes a pair of integers into a single one. For example, he maps the > coordinate pairs to integers as follows: > > 1,1 -> 1 > 2,1 -> 2 > 1,2 -> 3 > 3,1 -> 4 > 2,2 -> 5 > > and so forth. He does this by writing out the coordinates in a grid: > > 1,1 1,2 1,3 1,4 ... > 2,1 2,2 2,3 2,4 ... > 3,1 3,2 3,3 3,4 ... > 4,1 4,2 4,3 4,4 ... > ... ... > But I've stared at this for an hour and I can't see how to extend the > result to three coordinates. I can lay out a grid in the order I want: > > 1,1,1 1,1,2 1,1,3 1,1,4 ... > 2,1,1 2,1,2 2,1,3 2,1,4 ... > 1,2,1 1,2,2 1,2,3 1,2,4 ... > 3,1,1 3,1,2 3,1,3 3,1,4 ... > 2,2,1 2,2,2 2,2,3 2,2,4 ... > ... > I can't see the patterns here that I can see in the 2-D grid (where the first number in each pair in the n'th row is n, and the second number in the n'th column is n). Maybe it needs to be 3-D? (Eg if the 3rd number in the triple is the Plane number, then plane 1 looks like: 1,1,1 1,2,1 1,3,1 2,1,1 2,2,1 2,3,1 3,1,1 3,2,1 3,3,1 ... ... But whether that has an equivalent traversal path like the diagonals of the 2-D, I don't know. I'm just guessing.) -- bartc From rosuav at gmail.com Fri Mar 9 21:14:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 10 Mar 2018 13:14:19 +1100 Subject: Enumerating all 3-tuples In-Reply-To: References: Message-ID: On Sat, Mar 10, 2018 at 12:13 PM, Steven D'Aprano wrote: > The Russian mathematician Cantor came up with a *pairing function* that > encodes a pair of integers into a single one. For example, he maps the > coordinate pairs to integers as follows: > > 1,1 -> 1 > 2,1 -> 2 > 1,2 -> 3 > 3,1 -> 4 > 2,2 -> 5 > > and so forth. He does this by writing out the coordinates in a grid: > > 1,1 1,2 1,3 1,4 ... > 2,1 2,2 2,3 2,4 ... > 3,1 3,2 3,3 3,4 ... > 4,1 4,2 4,3 4,4 ... > ... > > and then iterating over them along the diagonals, starting from the top > corner. The diagonals all have a constant sum. You catch the item with a sum of 2, and then the two with a sum of 3, then the three with a sum of 4, etc. Can that pattern be extended to three dimensions? (1, 1, 1) (1, 1, 2) (1, 2, 1) (2, 1, 1) (1, 1, 3) (1, 2, 2) (1, 3, 1) (2, 1, 2) (2, 2, 1) (3, 1, 1) etc Not sure if that helps or not. ChrisA From python at mrabarnett.plus.com Fri Mar 9 21:18:41 2018 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 10 Mar 2018 02:18:41 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: Message-ID: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> On 2018-03-10 01:13, Steven D'Aprano wrote: > I am trying to enumerate all the three-tuples (x, y, z) where each of x, > y, z can range from 1 to ? (infinity). > > This is clearly unhelpful: > > for x in itertools.count(1): > for y in itertools.count(1): > for z in itertools.count(1): > print(x, y, z) > > as it never advances beyond x=1, y=1 since the innermost loop never > finishes. > > Georg Cantor to the rescue! (Well, almost...) > > https://en.wikipedia.org/wiki/Pairing_function > > The Russian mathematician Cantor came up with a *pairing function* that > encodes a pair of integers into a single one. For example, he maps the > coordinate pairs to integers as follows: > > 1,1 -> 1 > 2,1 -> 2 > 1,2 -> 3 > 3,1 -> 4 > 2,2 -> 5 > > and so forth. He does this by writing out the coordinates in a grid: > > 1,1 1,2 1,3 1,4 ... > 2,1 2,2 2,3 2,4 ... > 3,1 3,2 3,3 3,4 ... > 4,1 4,2 4,3 4,4 ... > ... > > and then iterating over them along the diagonals, starting from the top > corner. That's just what I'm after, and I have this function that works > for 2-tuples: > > def cantor(start=0): > """Yield coordinate pairs using Cantor's Pairing Function. > > Yields coordinate pairs in (Z*,Z*) over the diagonals: > > >>> it = cantor() > >>> [next(it) for _ in range(10)] > [(0,0), (1,0), (0,1), (2,0), (1,1), (0,2), (3,0), (2,1), (1,2), (0,3)] > > If ``start`` is given, it is used as the first x- and y-coordinate. > """ > i = start > while True: > for j in range(start, i+1): > yield (i-j+start, j) > i += 1 > > > But I've stared at this for an hour and I can't see how to extend the > result to three coordinates. I can lay out a grid in the order I want: > > 1,1,1 1,1,2 1,1,3 1,1,4 ... > 2,1,1 2,1,2 2,1,3 2,1,4 ... > 1,2,1 1,2,2 1,2,3 1,2,4 ... > 3,1,1 3,1,2 3,1,3 3,1,4 ... > 2,2,1 2,2,2 2,2,3 2,2,4 ... > ... > > and applying Cantor's diagonal order will give me what I want, but damned > if I can see how to do it in code. > > Clearly I'm having a "cannot brain today" moment. > > Can anyone help me out here? > Think about the totals of each triple. You'll get totals of 3, then totals of 4, etc. This might help, although the order they come out might not be what you want: def triples(): for total in itertools.count(1): for i in range(1, total): for j in range(1, total - i): yield i, j, total - (i + j) From formisc at gmail.com Fri Mar 9 21:37:23 2018 From: formisc at gmail.com (Andrew Z) Date: Fri, 9 Mar 2018 21:37:23 -0500 Subject: LXML: can't register namespace In-Reply-To: References: Message-ID: Stefan, thank you for the link. That explains the line of thinking of the package designer(s). I also looked@ beautifulsoup and found it to work better with my old brains. On Fri, Mar 9, 2018 at 9:46 AM, Stefan Behnel wrote: > Peter Otten schrieb am 09.03.2018 um 14:11: > > Stefan Behnel wrote: > > > >> Andrew Z schrieb am 07.03.2018 um 05:03: > >>> Hello, > >>> with 3.6 and latest greatest lxml: > >>> > >>> from lxml import etree > >>> > >>> tree = etree.parse('Sample.xml') > >>> etree.register_namespace('','http://www.example.com') > >> > >> The default namespace prefix is spelled None (because there is no prefix > >> for it) and not the empty string. > > > > Does that mean the OP shouldn't use register_namespace() at all or that > he's > > supposed to replace "" with None? > > It meant neither of the two, but now that you ask, I would recommend the > first. ;) > > An application global setup for the default namespace is never a good idea, > thus my question regarding the actual intention of the OP. Depending on the > context, the right thing to do might be be to either not care at all, or to > not use the default namespace but a normally prefixed one instead, or to > define a (default) namespace mapping for a newly created tree, as shown in > the namespace tutorial. > > http://lxml.de/tutorial.html#namespaces > > Usually, not caring about namespace prefixes is the best approach. Parsers, > serialisers and compressors can deal with them perfectly and safely, humans > should just ignore the clutter, pitfalls and complexity that they > introduce. > > Stefan > > -- > https://mail.python.org/mailman/listinfo/python-list > From ben.usenet at bsb.me.uk Fri Mar 9 21:44:24 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sat, 10 Mar 2018 02:44:24 +0000 Subject: Enumerating all 3-tuples References: Message-ID: <878tb0zm2f.fsf@bsb.me.uk> Steven D'Aprano writes: > I am trying to enumerate all the three-tuples (x, y, z) where each of x, > y, z can range from 1 to ? (infinity). > > This is clearly unhelpful: > > for x in itertools.count(1): > for y in itertools.count(1): > for z in itertools.count(1): > print(x, y, z) > > as it never advances beyond x=1, y=1 since the innermost loop never > finishes. > > Georg Cantor to the rescue! (Well, almost...) > > https://en.wikipedia.org/wiki/Pairing_function > > The Russian mathematician Cantor came up with a *pairing function* that > encodes a pair of integers into a single one. For example, he maps the > coordinate pairs to integers as follows: > > 1,1 -> 1 > 2,1 -> 2 > 1,2 -> 3 > 3,1 -> 4 > 2,2 -> 5 > > and so forth. He does this by writing out the coordinates in a grid: > > 1,1 1,2 1,3 1,4 ... > 2,1 2,2 2,3 2,4 ... > 3,1 3,2 3,3 3,4 ... > 4,1 4,2 4,3 4,4 ... > ... > > and then iterating over them along the diagonals, starting from the top > corner. That's just what I'm after, and I have this function that works > for 2-tuples: > > def cantor(start=0): > """Yield coordinate pairs using Cantor's Pairing Function. > > Yields coordinate pairs in (Z*,Z*) over the diagonals: > > >>> it = cantor() > >>> [next(it) for _ in range(10)] > [(0,0), (1,0), (0,1), (2,0), (1,1), (0,2), (3,0), (2,1), (1,2), (0,3)] > > If ``start`` is given, it is used as the first x- and y-coordinate. > """ > i = start > while True: > for j in range(start, i+1): > yield (i-j+start, j) > i += 1 > > > But I've stared at this for an hour and I can't see how to extend the > result to three coordinates. I can lay out a grid in the order I want: > > 1,1,1 1,1,2 1,1,3 1,1,4 ... > 2,1,1 2,1,2 2,1,3 2,1,4 ... > 1,2,1 1,2,2 1,2,3 1,2,4 ... > 3,1,1 3,1,2 3,1,3 3,1,4 ... > 2,2,1 2,2,2 2,2,3 2,2,4 ... > ... > > and applying Cantor's diagonal order will give me what I want, but damned > if I can see how to do it in code. Rather than a grid you would need a cube, and the diagonals become planes. But I think that is an easier way (no code yet though!) unless you are set on one particular enumeration: consider the triple as a pair one element of which runs over the enumeration of pairs you already have. Start with 1,1 <-> 1 2,1 <-> 2 1,2 <-> 3 3,1 <-> 4 2,2 <-> 5 1,3 <-> 6 ... but replace the first element by the numbered pair from this same list: (1,1),1 (2,1),1 (1,1),2 (1,2),1 (2,1),2 (1,1),3 ... If it were a sane time here I'd try to code this. Looks like fun but it must wait... -- Ben. From p.f.moore at gmail.com Sat Mar 10 06:15:49 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 10 Mar 2018 11:15:49 +0000 Subject: Enumerating all 3-tuples In-Reply-To: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> References: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> Message-ID: On 10 March 2018 at 02:18, MRAB wrote: > On 2018-03-10 01:13, Steven D'Aprano wrote: >> >> I am trying to enumerate all the three-tuples (x, y, z) where each of x, >> y, z can range from 1 to ? (infinity). >> >> This is clearly unhelpful: >> >> for x in itertools.count(1): >> for y in itertools.count(1): >> for z in itertools.count(1): >> print(x, y, z) >> >> as it never advances beyond x=1, y=1 since the innermost loop never >> finishes. >> >> Georg Cantor to the rescue! (Well, almost...) [...] >> Can anyone help me out here? >> > Think about the totals of each triple. You'll get totals of 3, then totals > of 4, etc. > > This might help, although the order they come out might not be what you > want: > > def triples(): > for total in itertools.count(1): > for i in range(1, total): > for j in range(1, total - i): > yield i, j, total - (i + j) Mathematically, that's the usual generalisation of Cantor's diagonal argument. Paul From themaelstorm at gmail.com Sat Mar 10 06:56:40 2018 From: themaelstorm at gmail.com (themaelstorm at gmail.com) Date: Sat, 10 Mar 2018 03:56:40 -0800 (PST) Subject: Python installer hangs in Windows 7 In-Reply-To: <9c25eddc-9dcb-4ca4-96db-6ba086bd4cc3@googlegroups.com> References: <765703485.1601081.1486357381135.ref@mail.yahoo.com> <765703485.1601081.1486357381135@mail.yahoo.com> <9c25eddc-9dcb-4ca4-96db-6ba086bd4cc3@googlegroups.com> Message-ID: On Wednesday, 7 March 2018 14:45:23 UTC+1, eng.... at voepassaredo.com.br wrote: > Hi! Exact same problem here! Stucked at "initializing setup", windows 7, pyhton 3.6.4, etc, etc, etc.... > > However, while looking for solutions on the internet.. guess who's decided to come to party??? Setup finally started! I guess we just have to be pacient and give its time... exactly what happened to me on win10. got stuck that seemed forever. restarted. stuck again. while reading these comments i noticed it was moving forward From ben.usenet at bsb.me.uk Sat Mar 10 09:23:49 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sat, 10 Mar 2018 14:23:49 +0000 Subject: Enumerating all 3-tuples References: <878tb0zm2f.fsf@bsb.me.uk> Message-ID: <871sgsypoq.fsf@bsb.me.uk> Ben Bacarisse writes: > Steven D'Aprano writes: > >> I am trying to enumerate all the three-tuples (x, y, z) where each of x, >> y, z can range from 1 to ? (infinity). > ... But I think that is an easier way (no code yet though!) unless > you are set on one particular enumeration: consider the triple as a pair > one element of which runs over the enumeration of pairs you already > have. > > Start with > > 1,1 <-> 1 > 2,1 <-> 2 > 1,2 <-> 3 > 3,1 <-> 4 > 2,2 <-> 5 > 1,3 <-> 6 > ... > > but replace the first element by the numbered pair from this same list: > > (1,1),1 > (2,1),1 > (1,1),2 > (1,2),1 > (2,1),2 > (1,1),3 > ... > > If it were a sane time here I'd try to code this. Looks like fun but it > must wait... One advantage of this method is that is does rely on any numeric properties of the base list. Any sequence-like object can be used to make an list of n-tuples. Unfortunately my Python is not up to using iterators well enough to avoid a "maximum recursion depth exceeded" error. Basically everything must be kept "lazy" and I'm not sure how to do that in Python. It will help my Python to learn so I will have a go. Off topic: I knocked up this Haskell version as a proof-of-concept: import Data.List pn n l = pn' n (map (:[]) l) where pn' n lists | n == 1 = lists | otherwise = diag (pn' (n-1) lists) lists diag l1 l2 = zipWith (++) (concat (inits l1)) (concat (map reverse (inits l2))) Notes: map (:[]) l turns [1, 2, 3, ...] into [[1], [2], [3], ...] inits gives the list of initial segments of l. I.e. (inits "abc") is ["", "a", "ab", "abc"]. concat joins a list of lists into one list. zipWith (++) l1 l2 makes a list by pair-wise appending the elements of l1 and l2. -- Ben. From harshbh1998 at gmail.com Sat Mar 10 10:41:15 2018 From: harshbh1998 at gmail.com (Harsh Bhardwaj) Date: Sat, 10 Mar 2018 21:11:15 +0530 Subject: Initializing error Message-ID: On Mar 5, 2018 11:16 AM, "Harsh Bhardwaj" wrote: Plz fix this problem for me. I tried every way to fix this. Windows 7(32-bit) From harshbh1998 at gmail.com Sat Mar 10 10:43:08 2018 From: harshbh1998 at gmail.com (Harsh Bhardwaj) Date: Sat, 10 Mar 2018 21:13:08 +0530 Subject: Initializing error Message-ID: Plz fix this problem for me. I tried every way to fix this. Windows 7(32-bit) From gisdudester at gmail.com Sat Mar 10 12:23:10 2018 From: gisdudester at gmail.com (GISDude) Date: Sat, 10 Mar 2018 09:23:10 -0800 (PST) Subject: pip and command line Message-ID: <075f7ae5-9535-42be-9f32-7f364b484d34@googlegroups.com> Hi all, I'm hoping someone could help a wannabe python coder out. I'm an aspiring Data/AI/ML coder/programmer/datafiend - that might help with my situation. In my various fits of python downloads, I've managed to download Anaconda, Pyscripter, IDLE(3.6/32 AND 64bit), IDLE 2.7. I'm thinking of PyCharm next... but that's another story. I'm having a problem using pip: 1. Does one have to use pip in the commandline? Why can't I use pip while in IDLE or in my Anaconda distro? 2. For some unknown reason, I have "defaulted" to use IDLE python 3.6, 32 bit version. I would like to use some modules that are in the 64 bit(pyiso is one...don't ask it is a module for using DOE EIA data off their website) and the requests module is not in the 64 bit version, NOT my 32 bit version. A) So...should I maybe uninstall my anaconda, pyscripter? Just use IDLE? B) Maybe change the envirionment variables to go between 3.6 32 bit and 3.6 64 bit. Seems kinda lame to myself... Thanks all and appreciate the feedback R~ From tjreedy at udel.edu Sat Mar 10 13:27:37 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 10 Mar 2018 13:27:37 -0500 Subject: pip and command line In-Reply-To: <075f7ae5-9535-42be-9f32-7f364b484d34@googlegroups.com> References: <075f7ae5-9535-42be-9f32-7f364b484d34@googlegroups.com> Message-ID: On 3/10/2018 12:23 PM, GISDude wrote: > Hi all, > I'm hoping someone could help a wannabe python coder out. I'm an aspiring Data/AI/ML coder/programmer/datafiend - that might help with my situation. > > In my various fits of python downloads, I've managed to download Anaconda, Anaconda is a cpython distribution that comes with various 3rd party modules pre-installed. > Pyscripter, I am not familiar with this. > IDLE(3.6/32 AND 64bit), IDLE 2.7. If you are on Windows, you did not download 'IDLE'. You downloaded various versions of CPython from PSF that include the corresponding versions of IDLE, with its integrated shell and editor, but not the 3rd party modules included with Anaconda. If you are just starting with Python, and do not work somewhere that uses 2.7, I would ignore 2.7. Unless you absolutely need the 32 bit version, I delete it. > I'm having a problem using pip: > 1. Does one have to use pip in the commandline? Yes, use command line programs in a command line console/window. > Why can't I use pip while in IDLE Would you expect to use pip in MS Excel? If you type 'python' at a command line, and get the '>>>' REPL prompt (google REPL), then your input goes to the python interpreter, which only accepts python statements. The same is true in the IDLE Shell, and when you run python code in the editor. [Some 3rd party IDEs have a front end for PIP. I wanted to add this for IDLE but the project was vetoed when partly done.] > 2. For some unknown reason, I have "defaulted" to use IDLE python 3.6, 32 bit version. The order of installation, choices you made during installation, and how you start python. (With just one python installed, there would be no problem.) Read the first chapter of https://docs.python.org/3/using/index.html and the chapter for your OS (which you should have specified ;-) -- Terry Jan Reedy From ben.usenet at bsb.me.uk Sat Mar 10 13:38:27 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sat, 10 Mar 2018 18:38:27 +0000 Subject: Enumerating all 3-tuples References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> Message-ID: <87po4bwzbw.fsf@bsb.me.uk> Sorry for following up to myself again... Ben Bacarisse writes: >> ... But I think that is an easier way (no code yet though!) unless >> you are set on one particular enumeration: consider the triple as a pair >> one element of which runs over the enumeration of pairs you already >> have. This algorithm has the disadvantage that the sequence is not any of the usual "zig-zag" patterns. Using numbers, for example the sums of the triples are not monotonic: *Main> take 30 $ map sum (pn 2 [1..]) [3,4,4,5,5,4,6,6,5,5,7,7,6,6,5,8,8,7,7,6,5,9,9,8,8,7,6,6,10,10] That probably makes it a non-starter for you. -- Ben. From bc at freeuk.com Sat Mar 10 13:44:20 2018 From: bc at freeuk.com (bartc) Date: Sat, 10 Mar 2018 18:44:20 +0000 Subject: Enumerating all 3-tuples In-Reply-To: <871sgsypoq.fsf@bsb.me.uk> References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> Message-ID: [repost as original seems to have gone to email; my newsreader has somehow acquired a 'Reply' button where 'Followup' normally goes.] On 10/03/2018 14:23, Ben Bacarisse wrote: > Ben Bacarisse writes: > Off topic: I knocked up this Haskell version as a proof-of-concept: > > import Data.List > > pn n l = pn' n (map (:[]) l) > where pn' n lists | n == 1 = lists > | otherwise = diag (pn' (n-1) lists) lists > diag l1 l2 = zipWith (++) (concat (inits l1)) > (concat (map reverse (inits l2))) > > Notes: > > map (:[]) l turns [1, 2, 3, ...] into [[1], [2], [3], ...] > > inits gives the list of initial segments of l. I.e. (inits "abc") is > ["", "a", "ab", "abc"]. > > concat joins a list of lists into one list. > > zipWith (++) l1 l2 makes a list by pair-wise appending the elements of > l1 and l2. > What's the output? (And what's the input; how do you invoke pn, if that's how it's done?) -- Bartc From ben.usenet at bsb.me.uk Sat Mar 10 15:06:20 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sat, 10 Mar 2018 20:06:20 +0000 Subject: Enumerating all 3-tuples References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> Message-ID: <87k1ujwv9f.fsf@bsb.me.uk> bartc writes: > [repost as original seems to have gone to email; my newsreader has > somehow acquired a 'Reply' button where 'Followup' normally goes.] [I thought it was intended but my reply bounced.] > On 10/03/2018 14:23, Ben Bacarisse wrote: >> Ben Bacarisse writes: > >> Off topic: I knocked up this Haskell version as a proof-of-concept: >> >> import Data.List >> >> pn n l = pn' n (map (:[]) l) >> where pn' n lists | n == 1 = lists >> | otherwise = diag (pn' (n-1) lists) lists >> diag l1 l2 = zipWith (++) (concat (inits l1)) >> (concat (map reverse (inits l2))) >> > What's the output? (And what's the input; how do you invoke pn, if > that's how it's done?) You pass a number and a list which should probably be infinite like [1..]. You'd better take only a few of the resulting elements then: *Main> let triples = pn 3 [1..] *Main> take 20 triples [[1,1,1],[1,1,2],[1,2,1],[1,1,3],[1,2,2],[2,1,1],[1,1,4],[1,2,3],[2,1,2],[1,3,1],[1,1,5],[1,2,4],[2,1,3],[1,3,2],[2,2,1],[1,1,6],[1,2,5],[2,1,4],[1,3,3],[2,2,2]] or you can index the list to look at particular elements: *Main> triples !! 10000000 [70,6,1628] but, as I've said, the order of the results is not the usual one (except for pairs). -- Ben. From bc at freeuk.com Sat Mar 10 16:20:05 2018 From: bc at freeuk.com (bartc) Date: Sat, 10 Mar 2018 21:20:05 +0000 Subject: Enumerating all 3-tuples In-Reply-To: <87k1ujwv9f.fsf@bsb.me.uk> References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> Message-ID: On 10/03/2018 20:06, Ben Bacarisse wrote: > bartc writes: > >> [repost as original seems to have gone to email; my newsreader has >> somehow acquired a 'Reply' button where 'Followup' normally goes.] > > [I thought it was intended but my reply bounced.] > >> On 10/03/2018 14:23, Ben Bacarisse wrote: >>> Ben Bacarisse writes: >> >>> Off topic: I knocked up this Haskell version as a proof-of-concept: >>> >>> import Data.List >>> >>> pn n l = pn' n (map (:[]) l) >>> where pn' n lists | n == 1 = lists >>> | otherwise = diag (pn' (n-1) lists) lists >>> diag l1 l2 = zipWith (++) (concat (inits l1)) >>> (concat (map reverse (inits l2))) >>> > >> What's the output? (And what's the input; how do you invoke pn, if >> that's how it's done?) > > You pass a number and a list which should probably be infinite like > [1..]. You'd better take only a few of the resulting elements then: > > *Main> let triples = pn 3 [1..] > *Main> take 20 triples > [[1,1,1],[1,1,2],[1,2,1],[1,1,3],[1,2,2],[2,1,1],[1,1,4],[1,2,3],[2,1,2],[1,3,1],[1,1,5],[1,2,4],[2,1,3],[1,3,2],[2,2,1],[1,1,6],[1,2,5],[2,1,4],[1,3,3],[2,2,2]] > > or you can index the list to look at particular elements: > > *Main> triples !! 10000000 > [70,6,1628] > > but, as I've said, the order of the results is not the usual one (except > for pairs). OK. I ran it like this: main = print (take 20 (pn 3 [1..])) But I'm trying to understand the patterns in the sequence. If I use: main = print (take 50 (pn 2 [1..])) then group the results into sets of 1, 2, 3, etc pairs, showing each group on a new line, then this gives sequences which are equivalent to the diagonals of the OP's 2D grid. (Except they don't alternate in direction; is that what you mean?) I'll have to study the pn 3 version some more. (pn 10 gives an interesting view of it too.) -- bartc From ben.usenet at bsb.me.uk Sat Mar 10 16:53:25 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sat, 10 Mar 2018 21:53:25 +0000 Subject: Enumerating all 3-tuples References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> Message-ID: <87efkrwqay.fsf@bsb.me.uk> bartc writes: > On 10/03/2018 20:06, Ben Bacarisse wrote: >>> On 10/03/2018 14:23, Ben Bacarisse wrote: >>>> Off topic: I knocked up this Haskell version as a proof-of-concept: >>>> >>>> import Data.List >>>> >>>> pn n l = pn' n (map (:[]) l) >>>> where pn' n lists | n == 1 = lists >>>> | otherwise = diag (pn' (n-1) lists) lists >>>> diag l1 l2 = zipWith (++) (concat (inits l1)) >>>> (concat (map reverse (inits l2))) >>>> >> but, as I've said, the order of the results is not the usual one (except >> for pairs). > OK. I ran it like this: > > main = print (take 20 (pn 3 [1..])) > > But I'm trying to understand the patterns in the sequence. If I use: > > main = print (take 50 (pn 2 [1..])) > > then group the results into sets of 1, 2, 3, etc pairs, showing each > group on a new line, then this gives sequences which are equivalent to > the diagonals of the OP's 2D grid. (Except they don't alternate in > direction; is that what you mean?) Yes, the pattern for pairs (pn 2 [1..]) is the normal one (one of them anyway). In the 2D grid the diagonals are lines of equal sum (for a numeric grid, of course) and all the pairs that sum to x appear before those that sum to x+1: *Main> take 28 $ map sum $ pn 2 [1..] [2,3,3,4,4,4,5,5,5,5,6,6,6,6,6,7,7,7,7,7,7,8,8,8,8,8,8,8] The recursive step breaks this rule. I'm sure there's a better rule, but it's getting more and more off-topic... -- Ben. From ben.usenet at bsb.me.uk Sat Mar 10 20:40:01 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Sun, 11 Mar 2018 01:40:01 +0000 Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> Message-ID: <878tazwfta.fsf@bsb.me.uk> Lawrence D?Oliveiro writes: > On Sunday, March 11, 2018 at 3:24:05 AM UTC+13, Ben Bacarisse wrote: >> Unfortunately my Python is not up to using iterators well enough to >> avoid a "maximum recursion depth exceeded" error. > > My solution only needed recursion up to a depth equal to the length of > the tuples. The ordering is done based on the maximum value of any > element of the tuple. I'm sure deep recursion is not needed, it's just tricky translating from a lazy language when one is not familiar with all the iterator facilities in Python. For example, I couldn't find an append operation that returns an iterable. It would be nice to avoid relying on any value-based ordering. There is inevitably an order that comes from the iterable whose elements are being paired, trippled or whatever, but that's all that's really required for a solution. -- Ben. From formisc at gmail.com Sat Mar 10 22:43:07 2018 From: formisc at gmail.com (Andrew Z) Date: Sat, 10 Mar 2018 22:43:07 -0500 Subject: Flask: request vs Request Message-ID: hello, i'm playing with Michael Grinberg's "Restful API with python and flask" tutorial . And i'm very much confused whats the difference between request and Request. the "request" is introduced @ the point he talks about "POST". Here is the sample from the tutorial: from flask import request @app.route('/todo/api/v1.0/tasks', methods=['POST'])def create_task(): if not request.json or not 'title' in request.json: abort(400) task = { 'id': tasks[-1]['id'] + 1, 'title': request.json['title'], 'description': request.json.get('description', ""), 'done': False } tasks.append(task) return jsonify({'task': task}), 201 the help system talks about somekind of LocalProxy class and then im getting lost .... The immediate trouble is that i don't see any methods for "request". ( Or, at least, PyCharm can't find anything) and not sure what the "*r*equest" is for and what i can do with it. Or should i use the "*R*equest" Appreciate some guidance on the matter. From christopherrmullins at gmail.com Sat Mar 10 23:09:56 2018 From: christopherrmullins at gmail.com (Christopher Mullins) Date: Sat, 10 Mar 2018 23:09:56 -0500 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: In the code you linked, I don't see where the *R*equest is used. The request variable is setup by flask when you annotate the function with the resource endpoint and POST method. It contains the content of the request which can be converted to json if that content type was specified. From breamoreboy at gmail.com Sat Mar 10 23:30:05 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Sun, 11 Mar 2018 04:30:05 +0000 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: On 11/03/18 04:09, Christopher Mullins wrote: > In the code you linked, I don't see where the *R*equest is used. The > request variable is setup by flask when you annotate the function with the > resource endpoint and POST method. It contains the content of the request > which can be converted to json if that content type was specified. > Could you please give some context when you reply, TIA. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From formisc at gmail.com Sat Mar 10 23:51:48 2018 From: formisc at gmail.com (Andrew Z) Date: Sat, 10 Mar 2018 23:51:48 -0500 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: that's right there is no Request used. So what for the Request is used for then? Here it is explained that request is for current request... that's kinda of clears the air a bit. but where do i find it's methods? On Sat, Mar 10, 2018 at 11:09 PM, Christopher Mullins < christopherrmullins at gmail.com> wrote: > In the code you linked, I don't see where the *R*equest is used. The > request variable is setup by flask when you annotate the function with the > resource endpoint and POST method. It contains the content of the request > which can be converted to json if that content type was specified. > > From Irv at furrypants.com Sun Mar 11 00:18:50 2018 From: Irv at furrypants.com (Irv Kalb) Date: Sat, 10 Mar 2018 21:18:50 -0800 Subject: Stock quote API ? Message-ID: Hi, I teach courses on beginning Python (Python3). In one of my topics, I explain how we can write simple programs that reach out to the internet and download data (request/response). I show a number of examples using: urllib.request.urlopen( ) to get things like weather data, currency exchange rates, etc. I just tried my examples again, and they are all working fine, except for one. I had an example where I used the call above to get simple (American) stock quotes from Yahoo. However, with this example, now I get a bunch errors. In tracking it down, I found that Yahoo has shut down this public API, discontinued this service. So ... I am looking for a replacement. I have done quite a bit of searching, but I have not been able to find a simple way to get a stock quote (no need for historical data - most recent price is fine). I have found many examples where people have built custom packages for doing this type of thing. However, I am in a college environment, and I cannot install any new packages on the computers there. I've also seen examples of people building SQL-style queries to get this type of information, but that's beyond what I am trying to teach. Wondering if anyone has any example of an API where I could just make a call using Python Standard Library interfaces to get stock quotes? Thanks in advance, Irv From rosuav at gmail.com Sun Mar 11 00:26:08 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 11 Mar 2018 16:26:08 +1100 Subject: Stock quote API ? In-Reply-To: References: Message-ID: On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb wrote: > Hi, > > I teach courses on beginning Python (Python3). In one of my topics, I explain how we can write simple programs that reach out to the internet and download data (request/response). > > I show a number of examples using: urllib.request.urlopen( ) to get things like weather data, currency exchange rates, etc. > > I just tried my examples again, and they are all working fine, except for one. I had an example where I used the call above to get simple (American) stock quotes from Yahoo. However, with this example, now I get a bunch errors. In tracking it down, I found that Yahoo has shut down this public API, discontinued this service. > > So ... I am looking for a replacement. I have done quite a bit of searching, but I have not been able to find a simple way to get a stock quote (no need for historical data - most recent price is fine). I have found many examples where people have built custom packages for doing this type of thing. However, I am in a college environment, and I cannot install any new packages on the computers there. I've also seen examples of people building SQL-style queries to get this type of information, but that's beyond what I am trying to teach. > > Wondering if anyone has any example of an API where I could just make a call using Python Standard Library interfaces to get stock quotes? > Check out https://www.alphavantage.co/ for something you can query for free. Extensive and amazingly useful. One of my students did some second-tier analysis on the data they provide as a capstone project on stock trading analysis. You may want to consider, though, modifying the "no new packages" rule. The 'requests' library is WAY better for teaching Python and web APIs than the raw urllib. Get just a small handful of pip-installable packages whitelisted and your life will be better. ChrisA From miki.tebeka at gmail.com Sun Mar 11 06:54:40 2018 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Sun, 11 Mar 2018 03:54:40 -0700 (PDT) Subject: PyCon Israel 2018 CFP is Open Message-ID: Hi, PyCon Israel 2018 call for papers is open, submit a talk today, another three tomorrow :) See more at http://il.pycon.org/2018/ All the best, -- Miki From skip.montanaro at gmail.com Sun Mar 11 09:44:37 2018 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Sun, 11 Mar 2018 08:44:37 -0500 Subject: Unnoticed traceback in a thread In-Reply-To: <87bmfxzqjx.fsf@handshake.de> References: <87bmfxzqjx.fsf@handshake.de> Message-ID: > Concretely: instead of "start_new_thread(my_thread_function, ...)", > I use > > def wrapped_thread_function(*args, **kw): > try: > my_thread_function(*args, **kw) > except: > ... do whatever is necessary should "my_thread_function" fails ... > > start_new_thread(wrapped_thread_function, ...) > > Similar, should you use the "Thread" class (instead of "start_new_thread"). Thanks, that looks like an excellent suggestion. Skip From formisc at gmail.com Sun Mar 11 10:02:56 2018 From: formisc at gmail.com (Andrew Z) Date: Sun, 11 Mar 2018 10:02:56 -0400 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: Sorry guys. i realized i was "mumbling" . What is the difference between Request and request? - RTFM , Andrew Questions: a. when do i use Request? b. why can't I see any properties of the request? For example, in Michael's tutorial, there is a call for request.json. PyCharm brings no suggestions when i type in "request. " On Sat, Mar 10, 2018 at 11:30 PM, Mark Lawrence wrote: > On 11/03/18 04:09, Christopher Mullins wrote: > >> In the code you linked, I don't see where the *R*equest is used. The >> request variable is setup by flask when you annotate the function with the >> resource endpoint and POST method. It contains the content of the request >> which can be converted to json if that content type was specified. >> >> > Could you please give some context when you reply, TIA. > > -- > My fellow Pythonistas, ask not what our language can do for you, ask > what you can do for our language. > > Mark Lawrence > > -- > https://mail.python.org/mailman/listinfo/python-list > From gisdudester at gmail.com Sun Mar 11 14:08:20 2018 From: gisdudester at gmail.com (GISDude) Date: Sun, 11 Mar 2018 11:08:20 -0700 (PDT) Subject: pip and command line In-Reply-To: References: <075f7ae5-9535-42be-9f32-7f364b484d34@googlegroups.com> Message-ID: On Saturday, March 10, 2018 at 10:28:00 AM UTC-8, Terry Reedy wrote: > On 3/10/2018 12:23 PM, GISDude wrote: > > Hi all, > > I'm hoping someone could help a wannabe python coder out. I'm an aspiring Data/AI/ML coder/programmer/datafiend - that might help with my situation. > > > > In my various fits of python downloads, I've managed to download Anaconda, > > Anaconda is a cpython distribution that comes with various 3rd party > modules pre-installed. > > > Pyscripter, > > I am not familiar with this. > > > IDLE(3.6/32 AND 64bit), IDLE 2.7. > > If you are on Windows, you did not download 'IDLE'. You downloaded > various versions of CPython from PSF that include the corresponding > versions of IDLE, with its integrated shell and editor, but not the 3rd > party modules included with Anaconda. > > If you are just starting with Python, and do not work somewhere that > uses 2.7, I would ignore 2.7. Unless you absolutely need the 32 bit > version, I delete it. > > > I'm having a problem using pip: > > 1. Does one have to use pip in the commandline? > > Yes, use command line programs in a command line console/window. > > > Why can't I use pip while in IDLE > > Would you expect to use pip in MS Excel? If you type 'python' at a > command line, and get the '>>>' REPL prompt (google REPL), then your > input goes to the python interpreter, which only accepts python > statements. The same is true in the IDLE Shell, and when you run python > code in the editor. > > [Some 3rd party IDEs have a front end for PIP. I wanted to add this for > IDLE but the project was vetoed when partly done.] > > > 2. For some unknown reason, I have "defaulted" to use IDLE python 3.6, 32 bit version. > > The order of installation, choices you made during installation, and how > you start python. (With just one python installed, there would be no > problem.) Read the first chapter of > https://docs.python.org/3/using/index.html and the chapter for your OS > (which you should have specified ;-) > > -- > Terry Jan Reedy Many thanks Terry. I'm back to the proverbial drawing board. On a side note, I'm uninstalling ANACONDA and re-installing. Seems like the whold 400mb of the win executable is missing a bunch of stuff. From tjreedy at udel.edu Sun Mar 11 15:35:31 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Mar 2018 15:35:31 -0400 Subject: pip and command line In-Reply-To: References: <075f7ae5-9535-42be-9f32-7f364b484d34@googlegroups.com> Message-ID: On 3/11/2018 2:08 PM, GISDude wrote: > On Saturday, March 10, 2018 at 10:28:00 AM UTC-8, Terry Reedy wrote: >>> Why can't I use pip while in IDLE >> >> Would you expect to use pip in MS Excel? If you type 'python' at a >> command line, and get the '>>>' REPL prompt (google REPL), then your >> input goes to the python interpreter, which only accepts python >> statements. The same is true in the IDLE Shell, and when you run python >> code in the editor. >> >> [Some 3rd party IDEs have a front end for PIP. I wanted to add this for >> IDLE but the project was vetoed when partly done.] What one can do in any Python REPL or program is to use subprocess to call pip. I used something like the following, recently, to prove it could be done, to install numpy from IDLE shell on Windows. >>> import subprocess as sub >>> sub.run("pip install numpy", stdout=sub.PIPE, stderr=sub.STDOUT).stdout But since I have Command Prompt pinned to the taskbar and often open, there is no point to this extra typing for me. -- Terry Jan Reedy From formisc at gmail.com Sun Mar 11 18:57:00 2018 From: formisc at gmail.com (Andrew Z) Date: Sun, 11 Mar 2018 18:57:00 -0400 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: after reading the docs and stackoverflow i got some rudimentary understanding of Request vs request. i still don't understand why i PyCharm won't show properties, but it is tool centric question that hardly belongs to this group. On Sun, Mar 11, 2018 at 10:02 AM, Andrew Z wrote: > Sorry guys. i realized i was "mumbling" . > > What is the difference between Request and request? - RTFM > , Andrew > > Questions: > a. when do i use Request? > b. why can't I see any properties of the request? For example, in > Michael's tutorial, there is a call for request.json. PyCharm brings no > suggestions when i type in "request. " > > > > > On Sat, Mar 10, 2018 at 11:30 PM, Mark Lawrence > wrote: > >> On 11/03/18 04:09, Christopher Mullins wrote: >> >>> In the code you linked, I don't see where the *R*equest is used. The >>> request variable is setup by flask when you annotate the function with >>> the >>> resource endpoint and POST method. It contains the content of the request >>> which can be converted to json if that content type was specified. >>> >>> >> Could you please give some context when you reply, TIA. >> >> -- >> My fellow Pythonistas, ask not what our language can do for you, ask >> what you can do for our language. >> >> Mark Lawrence >> >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > From andrewm at object-craft.com.au Mon Mar 12 00:05:57 2018 From: andrewm at object-craft.com.au (Andrew McNamara) Date: Mon, 12 Mar 2018 15:05:57 +1100 Subject: csv module and NULL data byte In-Reply-To: <20b871bd-0f52-f4f6-e35f-f1bbce00dc38@jptechnical.co.uk> References: <20180228205259.39594497@bigbox.christie.dr> <20180307075926.D9E96280179@robusta.syd.object-craft.com.au> <20b871bd-0f52-f4f6-e35f-f1bbce00dc38@jptechnical.co.uk> Message-ID: <20180312040557.231BD28017A@robusta.syd.object-craft.com.au> >Any idea why it might throw an exception on encountering a NULL in the >input stream? It accepts all other 255 byte values. Was this behaviour >intended? Perhaps a comment should be added to the docs. >Thanks for your work on the module anyway. The original module was like this - it comes about through the C convention of null-terminated strings (the module is coded in C). It would have been fairly involved to removed the restriction, and I would have run the risk of introducing subtle breakage for a feature nobody had asked for in the years we were maintaining the module outside the Python core... 8-) -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From Irv at furrypants.com Mon Mar 12 00:19:55 2018 From: Irv at furrypants.com (Irv Kalb) Date: Sun, 11 Mar 2018 21:19:55 -0700 Subject: Stock quote API ? In-Reply-To: References: Message-ID: > On Mar 10, 2018, at 9:26 PM, Chris Angelico wrote: > > On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb wrote: >> Hi, >> >> I teach courses on beginning Python (Python3). In one of my topics, I explain how we can write simple programs that reach out to the internet and download data (request/response). >> >> I show a number of examples using: urllib.request.urlopen( ) to get things like weather data, currency exchange rates, etc. >> >> I just tried my examples again, and they are all working fine, except for one. I had an example where I used the call above to get simple (American) stock quotes from Yahoo. However, with this example, now I get a bunch errors. In tracking it down, I found that Yahoo has shut down this public API, discontinued this service. >> >> So ... I am looking for a replacement. I have done quite a bit of searching, but I have not been able to find a simple way to get a stock quote (no need for historical data - most recent price is fine). I have found many examples where people have built custom packages for doing this type of thing. However, I am in a college environment, and I cannot install any new packages on the computers there. I've also seen examples of people building SQL-style queries to get this type of information, but that's beyond what I am trying to teach. >> >> Wondering if anyone has any example of an API where I could just make a call using Python Standard Library interfaces to get stock quotes? >> > > Check out https://www.alphavantage.co/ for something you can query for > free. Extensive and amazingly useful. One of my students did some > second-tier analysis on the data they provide as a capstone project on > stock trading analysis. > > You may want to consider, though, modifying the "no new packages" > rule. The 'requests' library is WAY better for teaching Python and web > APIs than the raw urllib. Get just a small handful of pip-installable > packages whitelisted and your life will be better. > > ChrisA > Hi Chris, Thank you very much for this. It is very close to what I am looking for. I had seen this early in my searches but I didn't go into it in detail because it looked like it was designed to give way more information than I was looking for - for example, the first example is about time series data. I did look into it today, and I got a free API key to check it out. It does have the ability to give just a stock quote for a symbol, but it looks like the minimum I can get back is a csv: symbol,price,volume,timestamp MSFT,96.1800,--,2018-03-09 16:01:30 which is easy enough for me to break apart. I just wish there was a way to eliminate the header line so I wouldn't have to go through an explanation about that. Thanks very much. If I can't find another one that just give back a price, I'll probably use this one. Irv PS: The "no new packages" rule is not mine. It's the rule imposed by the college. They are the administrators of the computers and I don't have an admin password. From rosuav at gmail.com Mon Mar 12 00:34:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 12 Mar 2018 15:34:04 +1100 Subject: Stock quote API ? In-Reply-To: References: Message-ID: On Mon, Mar 12, 2018 at 3:19 PM, Irv Kalb wrote: > >> On Mar 10, 2018, at 9:26 PM, Chris Angelico wrote: >> >> On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb wrote: >>> Hi, >>> >>> I teach courses on beginning Python (Python3). In one of my topics, I explain how we can write simple programs that reach out to the internet and download data (request/response). >>> >>> I show a number of examples using: urllib.request.urlopen( ) to get things like weather data, currency exchange rates, etc. >>> >>> I just tried my examples again, and they are all working fine, except for one. I had an example where I used the call above to get simple (American) stock quotes from Yahoo. However, with this example, now I get a bunch errors. In tracking it down, I found that Yahoo has shut down this public API, discontinued this service. >>> >>> So ... I am looking for a replacement. I have done quite a bit of searching, but I have not been able to find a simple way to get a stock quote (no need for historical data - most recent price is fine). I have found many examples where people have built custom packages for doing this type of thing. However, I am in a college environment, and I cannot install any new packages on the computers there. I've also seen examples of people building SQL-style queries to get this type of information, but that's beyond what I am trying to teach. >>> >>> Wondering if anyone has any example of an API where I could just make a call using Python Standard Library interfaces to get stock quotes? >>> >> >> Check out https://www.alphavantage.co/ for something you can query for >> free. Extensive and amazingly useful. One of my students did some >> second-tier analysis on the data they provide as a capstone project on >> stock trading analysis. >> >> You may want to consider, though, modifying the "no new packages" >> rule. The 'requests' library is WAY better for teaching Python and web >> APIs than the raw urllib. Get just a small handful of pip-installable >> packages whitelisted and your life will be better. >> >> ChrisA >> > > Hi Chris, > > Thank you very much for this. It is very close to what I am looking for. I had seen this early in my searches but I didn't go into it in detail because it looked like it was designed to give way more information than I was looking for - for example, the first example is about time series data. > > I did look into it today, and I got a free API key to check it out. It does have the ability to give just a stock quote for a symbol, but it looks like the minimum I can get back is a csv: > > symbol,price,volume,timestamp > MSFT,96.1800,--,2018-03-09 16:01:30 > > which is easy enough for me to break apart. I just wish there was a way to eliminate the header line so I wouldn't have to go through an explanation about that. > > Thanks very much. If I can't find another one that just give back a price, I'll probably use this one. It's usually easier to find a decent API that gives too much info than to find something that gives exactly what you want - because "what you want" usually isn't the same as "what someone else wants", and the easiest way to give both is to give lots of info. > PS: The "no new packages" rule is not mine. It's the rule imposed by the college. They are the administrators of the computers and I don't have an admin password. > Oh, I'm sure. But talk to the college about getting a small number of packages whitelisted. You don't have to ask for arbitrary package installation, just for a tiny handful of really REALLY useful packages. If you were teaching data science with Python, would you restrict yourself to the standard library, or would you use numpy, pandas, etc? Web development is the same - while you CAN do everything with just the stdlib, it's way better to pick up a few well-known packages that tie in well with that. ChrisA From mail at timgolden.me.uk Mon Mar 12 04:51:32 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 12 Mar 2018 08:51:32 +0000 Subject: Can mock.mock_open.read return different values? Message-ID: <84c81c3a-91b6-bb4e-ac14-b41357039c91@timgolden.me.uk> I'm contributing to a codebase which makes heavy use of mock in the test suite, a technique which I'm aware of but have used only rarely. In one situation it uses mock.mock_open(read_data="...") and then asserts again mock_open.return_value.read.call_count. A code change I've made results in an increase in the call count but also the open() I've introduced opens the file in binary mode and does something with the resulting data. Hugely simplified, the new code and unchanged test looks like this: import os, sys import unittest from unittest import mock def read_file(filename): # # This section is new # with open(filename, "rb") as f: text = f.read() if text.startswith(b"#"): pass with open(filename) as f: text = f.read() if text.startswith("#"): pass return text class TestS(unittest.TestCase): def test_read_file(self): mock_open = mock.mock_open(read_data="abc") with mock.patch('builtins.open', mock_open): data = read_file("abc") assert mock_open.return_value.read.call_count == 1 if __name__ == '__main__': unittest.main() I would expect the test to fail because of the call_count change. But in fact it errors out because the newly-added "if test.startswith()" receives a string, not bytes, from the Mock's read_data functionality. Ignore for the moment any question of changing the read_file implementation to assist testing. And leave aside the question of whether a mock_open is really a good test approach here. Is there any way in which I can have the mock_open object return bytes for the first open and a string for the second? I've looked at setting a side_effect function against the mock_open.return_value.read Mock, but I can't see a way of having the function know whether it's supposed to be returning bytes or string. TJG From arek.bulski at gmail.com Mon Mar 12 05:12:15 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Mon, 12 Mar 2018 09:12:15 +0000 Subject: __import__ fails randomly due to missing file Message-ID: Hi, For the record, I am the developer of Construct library. And the question relates to it. https://construct.readthedocs.io/en/latest/# I have an issue of Travis failing randomly at test cases. The cause is always __import__ failing to import a dynamically generated code. Library generates a string, saves it to a file (closes the file too), then __import__ it. I dont think there is anything wrong with the (generated imported) code itself. It seems like its some sort of issue with the Python runtime (or filesystem) not detecting the file because it was created only a moment ago. I tried to solve it by adding os.sync() between writing the file and importing it, but it seems to have only decreased the error rate, not solved it. Frankly, this is a heisenbug. Re-running the Travis job usually fixes it. Its random and happens very rarely. I havent pinpointed it to any particular version of CPython or Pypy (although I dont remember it ever happening on Pypy). An example: https://travis-ci.org/construct/construct/jobs/352213556 The code in question: https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L402-L414 https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L179-L180 -- ~ Arkadiusz Bulski ~ tel 503 357 111 lub WhatsApp/Telegram From p.f.moore at gmail.com Mon Mar 12 06:00:52 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Mar 2018 10:00:52 +0000 Subject: __import__ fails randomly due to missing file In-Reply-To: References: Message-ID: On 12 March 2018 at 09:12, Arkadiusz Bulski wrote: > Hi, > > For the record, I am the developer of Construct library. And the question > relates to it. > https://construct.readthedocs.io/en/latest/# > > I have an issue of Travis failing randomly at test cases. The cause is > always __import__ failing to import a dynamically generated code. Library > generates a string, saves it to a file (closes the file too), then > __import__ it. I dont think there is anything wrong with the (generated > imported) code itself. It seems like its some sort of issue with the Python > runtime (or filesystem) not detecting the file because it was created only > a moment ago. I tried to solve it by adding os.sync() between writing the > file and importing it, but it seems to have only decreased the error rate, > not solved it. Frankly, this is a heisenbug. Re-running the Travis job > usually fixes it. Its random and happens very rarely. I havent pinpointed > it to any particular version of CPython or Pypy (although I dont remember > it ever happening on Pypy). > > An example: > https://travis-ci.org/construct/construct/jobs/352213556 > > The code in question: > https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L402-L414 > https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L179-L180 The import system caches os.stat information, so it's possible that if you're creating the file and then relatively quickly doing the import, there may be some cache effect going on (granularity of file modification timestamps?) I don't know the details, but it might be worth looking in that area... Paul From arek.bulski at gmail.com Mon Mar 12 06:42:34 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Mon, 12 Mar 2018 10:42:34 +0000 Subject: __import__ fails randomly due to missing file In-Reply-To: References: Message-ID: I inserted importlib.invalidate_caches() instead of os.sync() and at a glance, it seems to have fixed it. Big thanks Paul! pon., 12 mar 2018 o 11:00 u?ytkownik Paul Moore napisa?: > On 12 March 2018 at 09:12, Arkadiusz Bulski wrote: > > Hi, > > > > For the record, I am the developer of Construct library. And the question > > relates to it. > > https://construct.readthedocs.io/en/latest/# > > > > I have an issue of Travis failing randomly at test cases. The cause is > > always __import__ failing to import a dynamically generated code. Library > > generates a string, saves it to a file (closes the file too), then > > __import__ it. I dont think there is anything wrong with the (generated > > imported) code itself. It seems like its some sort of issue with the > Python > > runtime (or filesystem) not detecting the file because it was created > only > > a moment ago. I tried to solve it by adding os.sync() between writing the > > file and importing it, but it seems to have only decreased the error > rate, > > not solved it. Frankly, this is a heisenbug. Re-running the Travis job > > usually fixes it. Its random and happens very rarely. I havent pinpointed > > it to any particular version of CPython or Pypy (although I dont remember > > it ever happening on Pypy). > > > > An example: > > https://travis-ci.org/construct/construct/jobs/352213556 > > > > The code in question: > > > https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L402-L414 > > > https://github.com/construct/construct/blob/d9c645ed0a75b5a2b64318113cf69cac822fd8f5/construct/core.py#L179-L180 > > The import system caches os.stat information, so it's possible that if > you're creating the file and then relatively quickly doing the import, > there may be some cache effect going on (granularity of file > modification timestamps?) I don't know the details, but it might be > worth looking in that area... > > Paul > -- ~ Arkadiusz Bulski ~ tel 503 357 111 lub WhatsApp/Telegram From christopherrmullins at gmail.com Mon Mar 12 09:10:32 2018 From: christopherrmullins at gmail.com (Christopher Mullins) Date: Mon, 12 Mar 2018 09:10:32 -0400 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: > > Could you please give some context when you reply, TIA > Whoops, thanks for the reminder Mark. So what for the Request is used for then? In general when you see that something in Python starts with a capital letter, that indicates a class. Check out the relevant section of the PEP8 coding style [1], this is a good reference to have on hand. (This generalization doesn't apply to builtin types which follow a separate convention, also outlined in [1] -- and of course there are a numerous exceptions.) This holds for your variable in question "request" which is an instance of the Request class. Check out [2] for more information on this. PyCharm tries to do its best but I've had this problem too. You can always open up a python REPL and look at the object yourself: >>> import flask >>> r = flask.Request >>> r >>> dir(r) and so on. When I'm working in python I like to keep one of these open for this purpose. HTH, Chris [1] https://www.python.org/dev/peps/pep-0008/#class-names [2] http://flask.pocoo.org/docs/0.12/api/#incoming-request-data From robin at reportlab.com Mon Mar 12 09:17:15 2018 From: robin at reportlab.com (Robin Becker) Date: Mon, 12 Mar 2018 13:17:15 +0000 Subject: Enumerating all 3-tuples In-Reply-To: <87efkrwqay.fsf@bsb.me.uk> References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> Message-ID: <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> It's possible to generalize the cantor pairing function to triples, but that may not give you what you want. Effectively you can generate an arbitrary number of triples using an iterative method. My sample code looked like this import math def cantor_pair(k1,k2): return (((k1+k2)*(k1+k2+1))>>1) + k2 def inverse_cantor_pair(z): w = int((math.sqrt(8*z+1)-1)/2.0) t = (w*(w+1))>>1 j = z - t i = w - j return i, j def cantor_triple(i,j,k): return cantor_pair(cantor_pair(i,j),k) def inverse_cantor_triple(z): j,k = inverse_cantor_pair(z) i,j = inverse_cantor_pair(j) return i,j,k if __name__=='__main__': for z in xrange(100): i,j,k = inverse_cantor_triple(z) print z, repr((i,j,k)) or changing the construction import math def cantor_pair(k1,k2): return (((k1+k2)*(k1+k2+1))>>1) + k2 def inverse_cantor_pair(z): w = int((math.sqrt(8*z+1)-1)/2.0) t = (w*(w+1))>>1 j = z - t i = w - j return i, j def cantor_triple(i,j,k): return cantor_pair(i,cantor_pair(j,k)) def inverse_cantor_triple(z): i,k = inverse_cantor_pair(z) j,k = inverse_cantor_pair(k) return i,j,k if __name__=='__main__': for z in xrange(100): i,j,k = inverse_cantor_triple(z) print z, repr((i,j,k)), cantor_triple(i,j,k)==z this give different outcomes, but both appear to be a correct mapping of non-negative integers to triplets. -- Robin Becker From formisc at gmail.com Mon Mar 12 09:27:00 2018 From: formisc at gmail.com (Andrew Z) Date: Mon, 12 Mar 2018 09:27:00 -0400 Subject: Flask: request vs Request In-Reply-To: References: Message-ID: Thank you Christopher. On Mar 12, 2018 09:10, "Christopher Mullins" wrote: > Could you please give some context when you reply, TIA >> > > Whoops, thanks for the reminder Mark. > > So what for the Request is used for then? > > > In general when you see that something in Python starts with a capital > letter, that indicates a class. Check out the relevant section of the PEP8 > coding style [1], this is a good reference to have on hand. (This > generalization doesn't apply to builtin types which follow a separate > convention, also outlined in [1] -- and of course there are a numerous > exceptions.) > > This holds for your variable in question "request" which is an instance of > the Request class. Check out [2] for more information on this. PyCharm > tries to do its best but I've had this problem too. You can always open up > a python REPL and look at the object yourself: > > >>> import flask > >>> r = flask.Request > >>> r > > >>> dir(r) > > and so on. When I'm working in python I like to keep one of these open > for this purpose. > > HTH, > Chris > > [1] https://www.python.org/dev/peps/pep-0008/#class-names > [2] http://flask.pocoo.org/docs/0.12/api/#incoming-request-data > > > From nospam at yrl.co.uk Mon Mar 12 11:31:30 2018 From: nospam at yrl.co.uk (Elliott Roper) Date: Mon, 12 Mar 2018 15:31:30 +0000 Subject: Enumerating all 3-tuples References: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> Message-ID: <0001HW.2056D4D204FCC9EA7000013B12CF@news.giganews.com> On 10 Mar 2018, Paul Moore wrote (in article): > On 10 March 2018 at 02:18, MRAB wrote: > > On 2018-03-10 01:13, Steven D'Aprano wrote: > > > > > > I am trying to enumerate all the three-tuples (x, y, z) where each of x, > > > y, z can range from 1 to ? (infinity). > > > > > > This is clearly unhelpful: > > > > > > for x in itertools.count(1): > > > for y in itertools.count(1): > > > for z in itertools.count(1): > > > print(x, y, z) > > > > > > as it never advances beyond x=1, y=1 since the innermost loop never > > > finishes. > > > > > > Georg Cantor to the rescue! (Well, almost...) > [...] > > > Can anyone help me out here? > > Think about the totals of each triple. You'll get totals of 3, then totals > > of 4, etc. > > > > This might help, although the order they come out might not be what you > > want: > > > > def triples(): > > for total in itertools.count(1): > > for i in range(1, total): > > for j in range(1, total - i): > > yield i, j, total - (i + j) > > Mathematically, that's the usual generalisation of Cantor's diagonal argument. > > Paul Would a multi-dimensional Hilbert curve do the job? See the Wikipedia article for starters -- To de-mung my e-mail address:- fsnospam$elliott$$ PGP Fingerprint: 1A96 3CF7 637F 896B C810 E199 7E5C A9E4 8E59 E248 From michael.stemper at gmail.com Mon Mar 12 11:53:51 2018 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Mon, 12 Mar 2018 10:53:51 -0500 Subject: Initializing error In-Reply-To: References: Message-ID: On 2018-03-10 09:41, Harsh Bhardwaj wrote: > On Mar 5, 2018 11:16 AM, "Harsh Bhardwaj" wrote: > > Plz fix this problem for me. I tried every way to fix this. > Windows 7(32-bit) That's a pretty bad problem. Fortunately, there is a fix: -- Michael F. Stemper Soglin for governor. From robin at reportlab.com Mon Mar 12 11:54:05 2018 From: robin at reportlab.com (Robin Becker) Date: Mon, 12 Mar 2018 15:54:05 +0000 Subject: Enumerating all 3-tuples In-Reply-To: <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On 12/03/2018 13:17, Robin Becker wrote: > It's possible to generalize the cantor pairing function to triples, but that may not give you what you want. Effectively you can > generate an arbitrary number of triples using an iterative method. My sample code looked like this ....ct mapping of non-negative integers to triplets. An alternative approach gives more orderly sequences using a variable base number construction class Tupilator(object): def __init__(self,degree=3): self.i = 0 self.n = 2 self.degree = degree def next(self): x = self.i v = [] a =v.append n = self.n if not x: a(0) while x>0: x, d = divmod(x,n) a(d) if sum(v)==self.degree*(n-1): self.n += 1 pad = self.degree - len(v) if pad>0: v += pad*[0] self.i += 1 return tuple(v) if __name__=='__main__': t = Tupilator() for z in xrange(100): print z, t.next() > 0 (0, 0, 0) > 1 (1, 0, 0) > 2 (0, 1, 0) > 3 (1, 1, 0) > 4 (0, 0, 1) > 5 (1, 0, 1) > 6 (0, 1, 1) > 7 (1, 1, 1) > 8 (2, 2, 0) > 9 (0, 0, 1) > 10 (1, 0, 1) > 11 (2, 0, 1) > 12 (0, 1, 1) > 13 (1, 1, 1) > 14 (2, 1, 1) > 15 (0, 2, 1) > 16 (1, 2, 1) > 17 (2, 2, 1) > 18 (0, 0, 2) > 19 (1, 0, 2) > 20 (2, 0, 2) > 21 (0, 1, 2) > 22 (1, 1, 2) > 23 (2, 1, 2) > 24 (0, 2, 2) > 25 (1, 2, 2) > 26 (2, 2, 2) > 27 (3, 2, 1) > 28 (0, 3, 1) > 29 (1, 3, 1) > 30 (2, 3, 1) > 31 (3, 3, 1) > 32 (0, 0, 2) > 33 (1, 0, 2) > 34 (2, 0, 2) > 35 (3, 0, 2) > 36 (0, 1, 2) > 37 (1, 1, 2) > 38 (2, 1, 2) > 39 (3, 1, 2) > 40 (0, 2, 2) ........ > 80 (0, 1, 3) > 81 (1, 1, 3) > 82 (2, 1, 3) > 83 (3, 1, 3) > 84 (4, 1, 3) > 85 (0, 2, 3) > 86 (1, 2, 3) > 87 (2, 2, 3) > 88 (3, 2, 3) > 89 (4, 2, 3) > 90 (0, 3, 3) > 91 (1, 3, 3) > 92 (2, 3, 3) > 93 (3, 3, 3) > 94 (4, 3, 3) > 95 (0, 4, 3) > 96 (1, 4, 3) > 97 (2, 4, 3) > 98 (3, 4, 3) > 99 (4, 4, 3) -- Robin Becker From rosuav at gmail.com Mon Mar 12 14:05:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 05:05:20 +1100 Subject: Enumerating all 3-tuples In-Reply-To: References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On Tue, Mar 13, 2018 at 2:54 AM, Robin Becker wrote: > On 12/03/2018 13:17, Robin Becker wrote: > An alternative approach gives more orderly sequences using a variable base > number construction > >> 4 (0, 0, 1) >> 9 (0, 0, 1) >> 18 (0, 0, 2) >> 32 (0, 0, 2) I spy duplicates. ChrisA From rosuav at gmail.com Mon Mar 12 14:07:01 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 05:07:01 +1100 Subject: Initializing error In-Reply-To: References: Message-ID: On Tue, Mar 13, 2018 at 2:53 AM, Michael F. Stemper wrote: > On 2018-03-10 09:41, Harsh Bhardwaj wrote: >> >> On Mar 5, 2018 11:16 AM, "Harsh Bhardwaj" wrote: >> >> Plz fix this problem for me. I tried every way to fix this. >> Windows 7(32-bit) > > > That's a pretty bad problem. Fortunately, there is a fix: > > And if that fix isn't appropriate to you, Bhardwaj, please notice that the quoted text above is everything that we got. You cannot attach images to your posts, and we cannot read your mind (not legally, anyway). You'll have to ask your question with enough information for us to answer it. ChrisA From skip.montanaro at gmail.com Mon Mar 12 14:42:41 2018 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 12 Mar 2018 13:42:41 -0500 Subject: Enumerating all 3-tuples In-Reply-To: References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: >>> 4 (0, 0, 1) >>> 9 (0, 0, 1) >>> 18 (0, 0, 2) >>> 32 (0, 0, 2) > > I spy duplicates. I didn't realize we'd started playing "I Spy ." ? ? Skip From ziyaajiang at gmail.com Mon Mar 12 14:43:01 2018 From: ziyaajiang at gmail.com (Yuan Xue) Date: Mon, 12 Mar 2018 13:43:01 -0500 Subject: test Message-ID: <87muzdceyy.fsf@gmail.com> test From rosuav at gmail.com Mon Mar 12 14:44:40 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 05:44:40 +1100 Subject: Enumerating all 3-tuples In-Reply-To: References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On Tue, Mar 13, 2018 at 5:42 AM, Skip Montanaro wrote: >>>> 4 (0, 0, 1) >>>> 9 (0, 0, 1) >>>> 18 (0, 0, 2) >>>> 32 (0, 0, 2) >> >> I spy duplicates. > > I didn't realize we'd started playing "I Spy." > We're actually playing Calvinball, but don't tell the others. ChrisA From alister.ware at ntlworld.com Mon Mar 12 15:26:45 2018 From: alister.ware at ntlworld.com (Alister) Date: Mon, 12 Mar 2018 19:26:45 GMT Subject: test References: <87muzdceyy.fsf@gmail.com> Message-ID: On Mon, 12 Mar 2018 13:43:01 -0500, Yuan Xue wrote: > test failed -- Mollison's Bureaucracy Hypothesis: If an idea can survive a bureaucratic review and be implemented it wasn't worth doing. From codewizard at gmail.com Mon Mar 12 15:47:41 2018 From: codewizard at gmail.com (codewizard at gmail.com) Date: Mon, 12 Mar 2018 12:47:41 -0700 (PDT) Subject: Can mock.mock_open.read return different values? In-Reply-To: References: <84c81c3a-91b6-bb4e-ac14-b41357039c91@timgolden.me.uk> Message-ID: <6dee695a-e6d6-4f74-aa44-c284437e7510@googlegroups.com> On Monday, March 12, 2018 at 4:51:53 AM UTC-4, Tim Golden wrote: > I'm contributing to a codebase which makes heavy use of mock in the test > suite, a technique which I'm aware of but have used only rarely. In one > situation it uses mock.mock_open(read_data="...") and then asserts again > mock_open.return_value.read.call_count. > > A code change I've made results in an increase in the call count but > also the open() I've introduced opens the file in binary mode and does > something with the resulting data. > > Hugely simplified, the new code and unchanged test looks like this: > > import os, sys > import unittest > from unittest import mock > > def read_file(filename): > > # > # This section is new > # > with open(filename, "rb") as f: > text = f.read() > if text.startswith(b"#"): > pass > > with open(filename) as f: > text = f.read() > if text.startswith("#"): > pass > > return text > > class TestS(unittest.TestCase): > > def test_read_file(self): > mock_open = mock.mock_open(read_data="abc") > with mock.patch('builtins.open', mock_open): > data = read_file("abc") > assert mock_open.return_value.read.call_count == 1 > > if __name__ == '__main__': > unittest.main() > > > I would expect the test to fail because of the call_count change. But in > fact it errors out because the newly-added "if test.startswith()" > receives a string, not bytes, from the Mock's read_data functionality. > > Ignore for the moment any question of changing the read_file > implementation to assist testing. And leave aside the question of > whether a mock_open is really a good test approach here. > > Is there any way in which I can have the mock_open object return bytes > for the first open and a string for the second? I've looked at setting a > side_effect function against the mock_open.return_value.read Mock, but I > can't see a way of having the function know whether it's supposed to be > returning bytes or string. > > > TJG 2 ways come to mind: 1. Have the side effect function check something that differentiates those 2 calls. In this case, checking mode argument value should work. 2. Rely on the order of calls with a local side effect function closing over a call number. Something like this (untested): def test_read_file(self): call_number = 0 def open_side_effect(filename, mode='r'): call_number += 1 if call_number == 1: return b'some bytes' if call_number == 2: return 'some string' with mock.patch(..., side_effect=open_side_effect): # test Regards, Igor. From cs at cskk.id.au Mon Mar 12 17:36:42 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Tue, 13 Mar 2018 08:36:42 +1100 Subject: test In-Reply-To: References: Message-ID: <20180312213642.GA34988@cskk.homeip.net> On 17Mar2014 11:42, Mark H Harris wrote: >On 3/16/14 5:07 AM, Chris ?Kwpolska? Warrick wrote: >>Why not use the mailing list instead? It?s a much easier way to >>access this place. > >I prefer to 'pull' rather than receive the 'push'. > >The newsreader idea is better because threading works better, and >because the interface is simpler. I don't know, just preference >I guess. I use mutt for email. Great threading support. My usenetness has mostly lapsed, but I used to funnel groups into my mail filer, read with mutt (same tool for news and mail!), and have a script for sendmail which intercepted group.name at usenet and send that via NNTP (obviously my puller inserted those addresses in the pull). >What are most active pythoniacs doing with this these days? I use the mailing lists and mutt for reading. Cheers, Cameron Simpson (formerly cs at zip.com.au) From ziyaajiang at gmail.com Mon Mar 12 19:07:08 2018 From: ziyaajiang at gmail.com (Yuan Xue) Date: Mon, 12 Mar 2018 18:07:08 -0500 Subject: test References: <87muzdceyy.fsf@gmail.com> Message-ID: <87d1087v1f.fsf@gmail.com> Alister writes: > On Mon, 12 Mar 2018 13:43:01 -0500, Yuan Xue wrote: > >> test > > failed Funny, I am wodering what type of persons are still using this clumsy system, and their purposes of using it. From ben.usenet at bsb.me.uk Mon Mar 12 20:58:36 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Tue, 13 Mar 2018 00:58:36 +0000 Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <878tazwfta.fsf@bsb.me.uk> Message-ID: <87muzcvlj7.fsf@bsb.me.uk> Lawrence D?Oliveiro writes: > On Sunday, March 11, 2018 at 2:40:16 PM UTC+13, Ben Bacarisse wrote: >> It would be nice to avoid relying on any value-based ordering. > > I don?t see why. The set of elements has to have the same cardinality > as the set of natural numbers, after all. Why not take advantage of > that? Maybe we are saying the same thing? The elements from which the tuple members are drawn need only be a sequence defined by a successor function. The values themselves need have no ordering nor, indeed, have any of the arithmetic properties of N. Of course you can always generate n-tuples of N and then map these to n-tuples of the intended sequence but that seems inelegant. -- Ben. From dvl at psu.edu Mon Mar 12 23:01:15 2018 From: dvl at psu.edu (ROGER GRAYDON CHRISTMAN) Date: Mon, 12 Mar 2018 23:01:15 -0400 Subject: Stock quote AP In-Reply-To: mailman.30.1520870403.24345.python-list@python.org References: Message-ID: <1520910075l.12451962l.0l@psu.edu> On Mon, Mar 12, 2018 12:00 PM, Irv Kalb wrote: > > On Mar 10, 2018, at 9:26 PM, Chris Angelico wrote: >> >> On Sun, Mar 11, 2018 at 4:18 PM, Irv Kalb wrote: >> Hi, >> >> I teach courses on beginning Python (Python3). In one of my >topics, I explain how we can write simple programs that reach out to the >internet and download data (request/response). >> >> I show a number of examples using: urllib.request.urlopen( > ) to get things like weather data, currency >exchange rates, etc. >> >> I just tried my examples again, and they are all working fine, except for >one. I had an example where I used the call above to get simple >(American) stock quotes from Yahoo. However, with this example, now >I get a bunch errors. In tracking it down, I found that Yahoo has shut down >this public API, discontinued this service. >> >> So ... I am looking for a replacement. I have done quite a bit of >searching, but I have not been able to find a simple way to get a stock quote >(no need for historical data - most recent price is fine). I have >found many examples where people have built custom packages for doing this type >of thing. However, I am in a college environment, and I cannot install any new >packages on the computers there. I've also seen examples of people building >SQL-style queries to get this type of information, but that's beyond what I am >trying to teach. >> >> Wondering if anyone has any example of an API where I could just make a >call using Python Standard Library interfaces to get stock quotes? >> >> >> Check out https://www.alphavantage.co/ for something you can query for >> free. Extensive and amazingly useful. One of my students did some >> second-tier analysis on the data they provide as a capstone project on >> stock trading analysis. >> >> You may want to consider, though, modifying the "no new packages" >> rule. The 'requests' library is WAY better for teaching Python and web >> APIs than the raw urllib. Get just a small handful of pip-installable >> packages whitelisted and your life will be better. >> >> ChrisA >> > >Hi Chris, > >Thank you very much for this. It is very close to what I am looking for. I >had seen this early in my searches but I didn't go into it in detail because it >looked like it was designed to give way more information than I was looking for >- for example, the first example is about time series data. > >I did look into it today, and I got a free API key to check it out. It does >have the ability to give just a stock quote for a symbol, but it looks like the >minimum I can get back is a csv: > >symbol,price,volume,timestamp >MSFT,96.1800,--,2018-03-09 16:01:30 > >which is easy enough for me to break apart. I just wish there was a way to >eliminate the header line so I wouldn't have to go through an explanation about >that. > >Thanks very much. If I can't find another one that just give back a price, >I'll probably use this one. > >Irv > > > Have you introduced the concepts of dictionaries in your Python course yet? In my introductory course, I present dictionaries even before if statements and for loops, since they have so many very useful features for data manipulation. That first line in the csv file is perfect for use with a dictionary. You would just need to get a dictionary Reader from the csv module, import urllib.request import io, csv file = urllib.request.urlopen(site + filename) data = io.TextIOWrapper(file, newline="", encoding="utf-8") reader = csv.DictReader(data) for row in reader: At this point, you could get your data from row['symbol'], row['price'] If you want to pretend that you don't actually know what the headings are, you could simply enumerate that row dictionary value, at it will tell you both the heading the value. Each line in the file would be a new dictionary. Roger Christman Pennsylvania State University From robin at reportlab.com Tue Mar 13 05:21:48 2018 From: robin at reportlab.com (Robin Becker) Date: Tue, 13 Mar 2018 09:21:48 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On 12/03/2018 18:05, Chris Angelico wrote: > On Tue, Mar 13, 2018 at 2:54 AM, Robin Becker wrote: >> On 12/03/2018 13:17, Robin Becker wrote: >> An alternative approach gives more orderly sequences using a variable base >> number construction >> >>> 4 (0, 0, 1) >>> 9 (0, 0, 1) >>> 18 (0, 0, 2) >>> 32 (0, 0, 2) > > I spy duplicates. > > ChrisA > oh well there goes that idea :( -- Robin Becker From steve+comp.lang.python at pearwood.info Tue Mar 13 07:01:09 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 11:01:09 +0000 (UTC) Subject: Enumerating all 3-tuples References: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> Message-ID: On Sat, 10 Mar 2018 11:15:49 +0000, Paul Moore wrote: > On 10 March 2018 at 02:18, MRAB wrote: [...] >> This might help, although the order they come out might not be what you >> want: >> >> def triples(): >> for total in itertools.count(1): >> for i in range(1, total): >> for j in range(1, total - i): >> yield i, j, total - (i + j) > > Mathematically, that's the usual generalisation of Cantor's diagonal > argument. Thanks MRAB. Paul, do you have a reference for that? Wolfram Mathworld is not very helpful regarding generalising Cantor's diagonalisation pairing function. (It's not the diagonal argument, that's something else :-) -- Steve From steve+comp.lang.python at pearwood.info Tue Mar 13 07:10:40 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 11:10:40 +0000 (UTC) Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <878tazwfta.fsf@bsb.me.uk> Message-ID: On Sun, 11 Mar 2018 01:40:01 +0000, Ben Bacarisse wrote: > I'm sure deep recursion is not needed, it's just tricky translating from > a lazy language when one is not familiar with all the iterator > facilities in Python. For example, I couldn't find an append operation > that returns an iterable. The Python iterator protocol is intentionally lightweight, so there's no "append" operation and + is not supported. But you can call it = itertools.chain(it, iterator2) It's not often that I think Ruby's ability to monkey-patch arbitrary objects including built-ins is a good idea, but the ability to allow iterator+iterator is *almost* one of those times. -- Steve From steve+comp.lang.python at pearwood.info Tue Mar 13 07:14:29 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 11:14:29 +0000 (UTC) Subject: Enumerating all 3-tuples References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On Mon, 12 Mar 2018 13:17:15 +0000, Robin Becker wrote: > It's possible to generalize the cantor pairing function to triples, but > that may not give you what you want. Effectively you can generate an > arbitrary number of triples using an iterative method. My sample code > looked like this > > > import math > > def cantor_pair(k1,k2): > return (((k1+k2)*(k1+k2+1))>>1) + k2 > > def inverse_cantor_pair(z): > w = int((math.sqrt(8*z+1)-1)/2.0) > t = (w*(w+1))>>1 > j = z - t > i = w - j > return i, j I definitely want to avoid any round trips through float in order to use sqrt. But thanks for the code, I'll certainly steal it, er I mean study it for future reference :-) -- Steve From steve+comp.lang.python at pearwood.info Tue Mar 13 07:16:59 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 11:16:59 +0000 (UTC) Subject: How do Fractions convert to int? Message-ID: How do Fractions convert to ints when they have no __int__ method? py> from fractions import Fraction py> x = Fraction(99, 2) py> int(x) # works fine 49 py> x.__int__ Traceback (most recent call last): File "", line 1, in AttributeError: 'Fraction' object has no attribute '__int__' -- Steve From p.f.moore at gmail.com Tue Mar 13 07:19:00 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Mar 2018 11:19:00 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: <6f54bbd1-bf75-2d26-5f9a-b11a8ad169ff@mrabarnett.plus.com> Message-ID: On 13 March 2018 at 11:01, Steven D'Aprano wrote: > On Sat, 10 Mar 2018 11:15:49 +0000, Paul Moore wrote: > >> On 10 March 2018 at 02:18, MRAB wrote: > [...] >>> This might help, although the order they come out might not be what you >>> want: >>> >>> def triples(): >>> for total in itertools.count(1): >>> for i in range(1, total): >>> for j in range(1, total - i): >>> yield i, j, total - (i + j) >> >> Mathematically, that's the usual generalisation of Cantor's diagonal >> argument. > > Thanks MRAB. Paul, do you have a reference for that? Wolfram Mathworld is > not very helpful regarding generalising Cantor's diagonalisation pairing > function. > > (It's not the diagonal argument, that's something else :-) Sadly, other than my memories of my maths degree from 30 years ago, no I don't. I'll see if I can dig something out. Paul From robin.koch at t-online.de Tue Mar 13 07:40:26 2018 From: robin.koch at t-online.de (Robin Koch) Date: Tue, 13 Mar 2018 12:40:26 +0100 Subject: How do Fractions convert to int? In-Reply-To: References: Message-ID: Am 13.03.2018 um 12:16 schrieb Steven D'Aprano: > How do Fractions convert to ints when they have no __int__ method? It uses __trunc__: | In general, the int() conversion should try __int__() first and if | it is not found, try __trunc__().[https://www.python.org/dev/peps/pep-3141/] -- Robin Koch From steve+comp.lang.python at pearwood.info Tue Mar 13 08:06:23 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 12:06:23 +0000 (UTC) Subject: How do Fractions convert to int? References: Message-ID: On Tue, 13 Mar 2018 12:40:26 +0100, Robin Koch wrote: > Am 13.03.2018 um 12:16 schrieb Steven D'Aprano: > >> How do Fractions convert to ints when they have no __int__ method? > > It uses __trunc__: Ah, thanks! -- Steve From antoon.pardon at vub.be Tue Mar 13 08:15:59 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Tue, 13 Mar 2018 13:15:59 +0100 Subject: Enumerating all 3-tuples In-Reply-To: References: Message-ID: On 10-03-18 02:13, Steven D'Aprano wrote: > I am trying to enumerate all the three-tuples (x, y, z) where each of x, > y, z can range from 1 to ? (infinity). > > This is clearly unhelpful: > > for x in itertools.count(1): > for y in itertools.count(1): > for z in itertools.count(1): > print(x, y, z) > > as it never advances beyond x=1, y=1 since the innermost loop never > finishes. > > Georg Cantor to the rescue! (Well, almost...) > > https://en.wikipedia.org/wiki/Pairing_function I came up with the following generalisation: from itertools import count def summation2(total): for n in range(1, total): yield n, total - n def summation(total, n): if n == 2: yield from summation2(total) else: for i in range(1, total - n + 2): for tail in summation(total - i, n - 1): yield (i,) + tail def cantor(n): for total in count(n): yield from summation(total, n) From robin at reportlab.com Tue Mar 13 09:35:49 2018 From: robin at reportlab.com (Robin Becker) Date: Tue, 13 Mar 2018 13:35:49 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <87k1ujwv9f.fsf@bsb.me.uk> <87efkrwqay.fsf@bsb.me.uk> <1edf960c-8306-ec9c-c380-da1c0e01f52e@chamonix.reportlab.co.uk> Message-ID: On 13/03/2018 11:14, Steven D'Aprano wrote: > On Mon, 12 Mar 2018 13:17:15 +0000, Robin Becker wrote: > >> It's possible to generalize the cantor pairing function to triples, but >> that may not give you what you want. Effectively you can generate an >> arbitrary number of triples using an iterative method. My sample code >> looked like this >> >> >> import math >> >> def cantor_pair(k1,k2): >> return (((k1+k2)*(k1+k2+1))>>1) + k2 >> >> def inverse_cantor_pair(z): >> w = int((math.sqrt(8*z+1)-1)/2.0) >> t = (w*(w+1))>>1 >> j = z - t >> i = w - j >> return i, j > > I definitely want to avoid any round trips through float in order to use > sqrt. > > > But thanks for the code, I'll certainly steal it, er I mean study it for > future reference :-) > > well I guess Cantor didn't worry about rounding errors :) For high z there's an integer square root function which seems to work pretty well here http://code.activestate.com/recipes/577821-integer-square-root-function/ I'm not sure if there are any other sensible invertible pairing functions on non-negative integers; this page mentions a couple implemented in matlab https://uk.mathworks.com/matlabcentral/fileexchange/44253-three-different-bijections-or-pairing-functions-between-n-and-n%5E2--including-cantor-polynomials- and this describes the elegant pairing more http://szudzik.com/ElegantPairing.pdf It seems reasonable that a mapping N x N --> N should require a square root in the inverse. -- Robin Becker From tjreedy at udel.edu Tue Mar 13 10:58:42 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 13 Mar 2018 10:58:42 -0400 Subject: 2.7 EOL = 2020 January 1 Message-ID: On March 10, on thread "Python 2.7 -- bugfix or security before EOL?", Guido van Russum wrote "The way I see the situation for 2.7 is that EOL is January 1st, 2020, and there will be no updates, not even source-only security patches, after that date. Support (from the core devs, the PSF, and python.org) stops completely on that date. If you want support for 2.7 beyond that day you will have to pay a commercial vendor. Of course it's open source so people are also welcome to fork it. But the core devs have toiled long enough, and the 2020 EOL date (an extension from the originally annouced 2015 EOL!) was announced with sufficient lead time and fanfare that I don't feel bad about stopping to support it at all." Two days later, Benjamin Peterson, the 2.7 release manager, replied "Sounds good to me. I've updated the PEP to say 2.7 is completely dead on Jan 1 2020." adding "The final release may not literally be on January 1st". https://www.python.org/dev/peps/pep-0373/ now says "2.7 will receive bugfix support until January 1, 2020." -- Terry Jan Reedy From python.list at tim.thechases.com Tue Mar 13 14:30:55 2018 From: python.list at tim.thechases.com (Tim Chase) Date: Tue, 13 Mar 2018 13:30:55 -0500 Subject: 2.7 EOL = 2020 January 1 In-Reply-To: References: Message-ID: <20180313133055.34e75b65@bigbox.christie.dr> On 2018-03-13 10:58, Terry Reedy wrote: > Two days later, Benjamin Peterson, the 2.7 release manager, replied > "Sounds good to me. I've updated the PEP to say 2.7 is completely > dead on Jan 1 2020." adding "The final release may not literally be > on January 1st". Am I the only one saddened by this announcement? I mean, it could have been something like """ "VOOM"?!? Mate, 2.7 wouldn't "voom" if you put four million volts through it! It's bleedin' demised! It's not pinin'! It's passed on! This 2.x series is no more! It has ceased to be! It's expired and gone to meet its maker! It's a stiff! Bereft of life, it rests in peace! If you hadn't nailed it to your dependencies it'd be pushing up the daisies! Its metabolic processes are now 'istory! It's off the twig! It's kicked the bucket, it's shuffled off its mortal coil, run down the curtain and joined the bleedin' choir invisible!! THIS IS AN EX-VERSION!! """ Pythonically-yers, -tkc From rosuav at gmail.com Tue Mar 13 16:30:23 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Mar 2018 07:30:23 +1100 Subject: 2.7 EOL = 2020 January 1 In-Reply-To: <20180313133055.34e75b65@bigbox.christie.dr> References: <20180313133055.34e75b65@bigbox.christie.dr> Message-ID: On Wed, Mar 14, 2018 at 5:30 AM, Tim Chase wrote: > On 2018-03-13 10:58, Terry Reedy wrote: >> Two days later, Benjamin Peterson, the 2.7 release manager, replied >> "Sounds good to me. I've updated the PEP to say 2.7 is completely >> dead on Jan 1 2020." adding "The final release may not literally be >> on January 1st". > > Am I the only one saddened by this announcement? I mean, it could > have been something like > > """ > "VOOM"?!? Mate, 2.7 wouldn't "voom" if you put four million volts > through it! It's bleedin' demised! It's not pinin'! It's passed on! > This 2.x series is no more! It has ceased to be! It's expired and > gone to meet its maker! It's a stiff! Bereft of life, it rests in > peace! If you hadn't nailed it to your dependencies it'd be pushing > up the daisies! Its metabolic processes are now 'istory! It's off > the twig! It's kicked the bucket, it's shuffled off its mortal coil, > run down the curtain and joined the bleedin' choir invisible!! > THIS IS AN EX-VERSION!! > """ > Wait until Jan 2nd 2020. Then post that. ChrisA From breamoreboy at gmail.com Tue Mar 13 16:36:30 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Tue, 13 Mar 2018 20:36:30 +0000 Subject: 2.7 EOL = 2020 January 1 In-Reply-To: <20180313133055.34e75b65@bigbox.christie.dr> References: <20180313133055.34e75b65@bigbox.christie.dr> Message-ID: On 13/03/18 18:30, Tim Chase wrote: > On 2018-03-13 10:58, Terry Reedy wrote: >> Two days later, Benjamin Peterson, the 2.7 release manager, replied >> "Sounds good to me. I've updated the PEP to say 2.7 is completely >> dead on Jan 1 2020." adding "The final release may not literally be >> on January 1st". > > Am I the only one saddened by this announcement? I mean, it could > have been something like > > """ > "VOOM"?!? Mate, 2.7 wouldn't "voom" if you put four million volts > through it! It's bleedin' demised! It's not pinin'! It's passed on! > This 2.x series is no more! It has ceased to be! It's expired and > gone to meet its maker! It's a stiff! Bereft of life, it rests in > peace! If you hadn't nailed it to your dependencies it'd be pushing > up the daisies! Its metabolic processes are now 'istory! It's off > the twig! It's kicked the bucket, it's shuffled off its mortal coil, > run down the curtain and joined the bleedin' choir invisible!! > THIS IS AN EX-VERSION!! > """ > > Pythonically-yers, > > -tkc > > I'd still like to know what the core developers have ever done for us. Apart from . -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From fabianosidler at swissonline.ch Tue Mar 13 17:37:16 2018 From: fabianosidler at swissonline.ch (Fabiano Sidler) Date: Tue, 13 Mar 2018 22:37:16 +0100 Subject: TLSServer: certificate one request behind... Message-ID: <20180313213716.GA50383@mailboxes.bgd.fipscode.net> Hi folks! I have written a TLSServer for testing purposes that generates self-signed certificates upon request. This works pretty well except that the certificates are always supplied one request too late: # gets no cert, but a handshake failure instead $ openssl s_client -connect localhost:1234 -servername server1 # gets the cert for server1 $ openssl s_client -connect localhost:1234 -servername server2 # gets the cert for server2 $ openssl s_client -connect localhost:1234 -servername server3 What's the reason for this? Please find attached my TLSServer. Best wishes, Fabiano From dkasak at termina.org.uk Tue Mar 13 18:56:37 2018 From: dkasak at termina.org.uk (Denis Kasak) Date: Tue, 13 Mar 2018 23:56:37 +0100 Subject: Enumerating all 3-tuples In-Reply-To: References: Message-ID: <765fc7e7a1b8aeb716d251fd15cc2919@termina.org.uk> On 2018-03-10 02:13, Steven D'Aprano wrote: > > But I've stared at this for an hour and I can't see how to extend the > result to three coordinates. I can lay out a grid in the order I want: > > 1,1,1 1,1,2 1,1,3 1,1,4 ... > 2,1,1 2,1,2 2,1,3 2,1,4 ... > 1,2,1 1,2,2 1,2,3 1,2,4 ... > 3,1,1 3,1,2 3,1,3 3,1,4 ... > 2,2,1 2,2,2 2,2,3 2,2,4 ... > ... > > and applying Cantor's diagonal order will give me what I want, but > damned > if I can see how to do it in code. If you want this exact order, it can be reproduced in the following way. The triples can be viewed as a pair of a pair and a natural number: (1,1),1 (1,1),2 (1,1),3 ... (2,1),1 (2,1),2 (2,1),3 ... (1,2),1 (1,2),2 (1,2),3 ... . . . . . . . . . Decomposing this by the diagonals yields 1: (1,1),1 2: (2,1),1 (1,1),2 3: (1,2),1 (2,1),2 (1,1),3 . . . Notice that the first elements of each pair (i.e. the inner pairs) are given by the (inverse of the) original Cantor pairing function, in decreasing order. The second elements are just the natural numbers. Naming the inverse c, we can rewrite the above like this: 1: c(1),1 2: c(2),1 c(1),2 3: c(3),1 c(2),2 c(1),3 . . . Rewriting this to spell out the mapping between the natural numbers and the pairs, we get 1 -> c(1),1 2 -> c(2),1 3 -> c(1),2 4 -> c(3),1 5 -> c(2),2 6 -> c(1),3 . . . Squinting a bit, this might seem familiar. This is exactly the same as the original pairing function, except for an additional application of c to the first element of the pair! 1 -> 1,1 2 -> 2,1 3 -> 1,2 4 -> 3,1 5 -> 2,2 6 -> 1,3 This leads fairly naturally to the implementation. from itertools import accumulate, count def c(i): """ Inverse of the Cantor pairing function, mapping N ? N?N. """ assert i >= 1 # partial sums of the series 1 + 2 + 3 + ... sums = accumulate(count(1)) n = 0 while True: m = next(sums) if m < i: n += 1 else: r = m - i break return r + 1, n - r + 1 def c3(i): """ Inverse of the Cantor pairing function generalization to triples, mapping N ? N?N?N. """ n, m = c(i) return c(n) + (m,) Applying c3 to the natural numbers gives the sequence you wanted: >>> s = map(c3, count(1)) >>> pprint([next(s) for _ in range(10)]) [(1, 1, 1), (2, 1, 1), (1, 1, 2), (1, 2, 1), (2, 1, 2), (1, 1, 3), (3, 1, 1), (1, 2, 2), (2, 1, 3), (1, 1, 4)] -- Denis Kasak From steve+comp.lang.python at pearwood.info Tue Mar 13 19:45:34 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 23:45:34 +0000 (UTC) Subject: 2.7 EOL = 2020 January 1 References: Message-ID: On Tue, 13 Mar 2018 10:58:42 -0400, Terry Reedy wrote: > On March 10, on thread "Python 2.7 -- bugfix or security before EOL?", > Guido van Russum wrote > > "The way I see the situation for 2.7 is that EOL is January 1st, 2020, > and there will be no updates, not even source-only security patches, > after that date. Support (from the core devs, the PSF, and python.org) > stops completely on that date. If you want support for 2.7 beyond that > day you will have to pay a commercial vendor. [...]" *dances a happy dance* Thanks Terry for posting the message, thanks Guido for drawing a line in the sand and sticking with it, and thanks to all the core devs who have worked so hard to keep the transition from 2 -> 3 as gentle as possible. Like Python 1 before it, Python 2 has been a wonderful language to work with, but it is long past time to move on to Python 3. Or pay for the support. We can only ask volunteers for so much. -- Steve From dhilip04211 at gmail.com Tue Mar 13 20:23:46 2018 From: dhilip04211 at gmail.com (Dhileepan Kumar) Date: Tue, 13 Mar 2018 17:23:46 -0700 (PDT) Subject: serial communication error for loop Message-ID: <9ca85b13-6c91-4c67-8621-ec5faceffbdb@googlegroups.com> #!/usr/bin/python import serial import MySQLdb while True: #establish connection to MySQL. You'll have to change this for your database. dbConn = MySQLdb.connect("localhost","root","1234","ifet") or die ("could not connect to database") #open a cursor to the database cursor = dbConn.cursor() device = 'com3' #this will have to be changed to the serial port you are using arduino = serial.Serial(device, 9600) try: data = arduino.readline() #read the data from the arduino if not data.strip(): continue pieces = data.split("\t") #split the data by the tab #Here we are going to insert the data into the Database try: cursor.execute("INSERT INTO answer1 (voltage,current) VALUES (%s,%s)", (pieces[0],pieces[1])) dbConn.commit() #commit the insert cursor.close() #close the cursor except MySQLdb.IntegrityError: print "failed to insert data" finally: cursor.close() #close just incase it failed except: print "Failed to get data from Arduino!" ................................................................................ it is my program.if do not use while true loop successfully run but can't get a continues data from arduino so i have decide to using while loop but if using while loop shows some error (project) C:\Users\DHILEEPAN\project\project>python mysql.py Traceback (most recent call last): File "mysql.py", line 13, in arduino = serial.Serial(device, 9600) File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 31, in __init_ super(Serial, self).__init__(*args, **kwargs) File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialutil.py", line 240, in __init_ self.open() File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 62, in open raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError())) serial.serialutil.SerialException: could not open port 'com3': WindowsError(5, 'Access is denied.') please any one fix this From rosuav at gmail.com Tue Mar 13 21:13:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Mar 2018 12:13:18 +1100 Subject: serial communication error for loop In-Reply-To: <9ca85b13-6c91-4c67-8621-ec5faceffbdb@googlegroups.com> References: <9ca85b13-6c91-4c67-8621-ec5faceffbdb@googlegroups.com> Message-ID: On Wed, Mar 14, 2018 at 11:23 AM, Dhileepan Kumar wrote: > > #!/usr/bin/python > > import serial > import MySQLdb > while True: > #establish connection to MySQL. You'll have to change this for your database. > dbConn = MySQLdb.connect("localhost","root","1234","ifet") or die ("could not connect to database") Ah, you've come from Perl I see. > #open a cursor to the database > cursor = dbConn.cursor() > > device = 'com3' #this will have to be changed to the serial port you are using > arduino = serial.Serial(device, 9600) You're opening the database AND the serial port every time you go through the loop. Is that intentional? > try: > data = arduino.readline() #read the data from the arduino > if not data.strip(): > continue > pieces = data.split("\t") #split the data by the tab > #Here we are going to insert the data into the Database > try: > cursor.execute("INSERT INTO answer1 (voltage,current) VALUES (%s,%s)", (pieces[0],pieces[1])) > dbConn.commit() #commit the insert > cursor.close() #close the cursor > except MySQLdb.IntegrityError: > print "failed to insert data" > finally: > cursor.close() #close just incase it failed > except: > print "Failed to get data from Arduino!" Ouch. Any exception, you just print out a generic message and move on? You're blinding yourself. Let the exception get properly displayed. It might help you. > ................................................................................ > it is my program.if do not use while true loop successfully run but can't get a continues data from arduino so i have decide to using while loop but if using while loop shows some error > > > (project) C:\Users\DHILEEPAN\project\project>python mysql.py > Traceback (most recent call last): > File "mysql.py", line 13, in > arduino = serial.Serial(device, 9600) > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 31, in __init_ > super(Serial, self).__init__(*args, **kwargs) > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialutil.py", line 240, in __init_ > self.open() > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 62, in open > raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError())) > serial.serialutil.SerialException: could not open port 'com3': WindowsError(5, 'Access is denied.') > > please any one fix this > This looks like possibly the device is already in use. Try opening the device _just once_, or if you have to reopen it every time through the loop, make sure you close it. ChrisA From python at mrabarnett.plus.com Tue Mar 13 21:15:44 2018 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 14 Mar 2018 01:15:44 +0000 Subject: serial communication error for loop In-Reply-To: <9ca85b13-6c91-4c67-8621-ec5faceffbdb@googlegroups.com> References: <9ca85b13-6c91-4c67-8621-ec5faceffbdb@googlegroups.com> Message-ID: <2d69f05c-6960-bb6c-8dbf-c59d5b06b8a1@mrabarnett.plus.com> On 2018-03-14 00:23, Dhileepan Kumar wrote: > > #!/usr/bin/python > > import serial > import MySQLdb > while True: > #establish connection to MySQL. You'll have to change this for your database. > dbConn = MySQLdb.connect("localhost","root","1234","ifet") or die ("could not connect to database") > #open a cursor to the database > cursor = dbConn.cursor() > > device = 'com3' #this will have to be changed to the serial port you are using > arduino = serial.Serial(device, 9600) > > > try: > data = arduino.readline() #read the data from the arduino > if not data.strip(): > continue > pieces = data.split("\t") #split the data by the tab > #Here we are going to insert the data into the Database > try: > cursor.execute("INSERT INTO answer1 (voltage,current) VALUES (%s,%s)", (pieces[0],pieces[1])) > dbConn.commit() #commit the insert > cursor.close() #close the cursor > except MySQLdb.IntegrityError: > print "failed to insert data" > finally: > cursor.close() #close just incase it failed > except: > print "Failed to get data from Arduino!" > > ................................................................................ > it is my program.if do not use while true loop successfully run but can't get a continues data from arduino so i have decide to using while loop but if using while loop shows some error > > > (project) C:\Users\DHILEEPAN\project\project>python mysql.py > Traceback (most recent call last): > File "mysql.py", line 13, in > arduino = serial.Serial(device, 9600) > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 31, in __init_ > super(Serial, self).__init__(*args, **kwargs) > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialutil.py", line 240, in __init_ > self.open() > File "C:\Users\DHILEEPAN\Envs\project\lib\site-packages\serial\serialwin32.py", line 62, in open > raise SerialException("could not open port {!r}: {!r}".format(self.portstr, ctypes.WinError())) > serial.serialutil.SerialException: could not open port 'com3': WindowsError(5, 'Access is denied.') > > please any one fix this > It cannot open the port the second time because the port is already open from the first time. You have put too much inside the loop. It should be more like: # open database and serial while True: # do read and insertion # close database and serial Even better, use the 'with' statement, which is neater: with : with : while True: # do read and insertion From ben.usenet at bsb.me.uk Tue Mar 13 21:18:11 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Wed, 14 Mar 2018 01:18:11 +0000 Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <878tazwfta.fsf@bsb.me.uk> <87muzcvlj7.fsf@bsb.me.uk> Message-ID: <87zi3btpyk.fsf@bsb.me.uk> Lawrence D?Oliveiro writes: > On Tuesday, March 13, 2018 at 1:58:48 PM UTC+13, Ben Bacarisse wrote: >> Of course you can always generate n-tuples of N and then map these to >> n-tuples of the intended sequence but that seems inelegant. > > This whole discussion seems to be going off on esoteric, irrelevant > tangents. The problem was not that hard to solve. Sure, but that's one thing that makes Usenet (and email lists) interesting. The original problem -- triples of natural numbers -- is not particularly hard, but the general problem -- enumerating n-tuples of some sequence -- is more interesting because it is a bit harder. -- Ben. From steve+comp.lang.python at pearwood.info Wed Mar 14 00:08:30 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 14 Mar 2018 04:08:30 +0000 (UTC) Subject: Python gotcha of the day Message-ID: Explain the difference between these two triple-quoted strings: Here is a triple-quoted string containing spaces and a triple-quote: py> """ \""" """ ' """ ' But remove the spaces, and two of the quotation marks disappear: py> """\"""""" '"' If nobody gets the answer, I shall reveal all later. (Hint: it is not a bug.) -- Steve From cs at cskk.id.au Wed Mar 14 00:40:55 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 14 Mar 2018 15:40:55 +1100 Subject: Python gotcha of the day In-Reply-To: References: Message-ID: <20180314044055.GA68052@cskk.homeip.net> On 14Mar2018 04:08, Steven D'Aprano wrote: >Explain the difference between these two triple-quoted strings: > >Here is a triple-quoted string containing spaces and a triple-quote: > >py> """ \""" """ >' """ ' > > >But remove the spaces, and two of the quotation marks disappear: > >py> """\"""""" >'"' """\"""" "" Implicit string concatenation of an empty string. I confess I try to have my quote delimiters distinct from any embedded quotes to avoid wanting backslashes. It is conincidental that my habit tends to sidestep the above issue. Cheers, Cameron Simpson (formerly cs at zip.com.au) From dan at tombstonezero.net Wed Mar 14 00:46:59 2018 From: dan at tombstonezero.net (Dan Sommers) Date: Wed, 14 Mar 2018 04:46:59 +0000 (UTC) Subject: Python gotcha of the day References: Message-ID: On Wed, 14 Mar 2018 04:08:30 +0000, Steven D'Aprano wrote: > Explain the difference between these two triple-quoted strings: > But remove the spaces, and two of the quotation marks disappear: > > py> """\"""""" > '"' That's (a) a triple quoted string containing a single escaped quote, followed by (b) a quoted empty string. The two consecutive strings are concatenated into a single string. (Too many years of shell programming to miss this one!) Dan From nad at python.org Wed Mar 14 00:52:32 2018 From: nad at python.org (Ned Deily) Date: Wed, 14 Mar 2018 00:52:32 -0400 Subject: [RELEASE] Python 3.6.5rc1 is now available for testing Message-ID: Announcing the immediate availability of Python 3.6.5 release candidate 1! Python 3.6.5rc1 is the first release candidate for Python 3.6.5, the next maintenance release of Python 3.6. While 3.6.5rc1 is a preview release and, thus, not intended for production environments, we encourage you to explore it and provide feedback via the Python bug tracker (https://bugs.python.org). 3.6.5 is planned for final release on 2018-03-26 with the next maintenance release expected to follow in about 3 months. You can find Python 3.6.5rc1 and more information here: https://www.python.org/downloads/release/python-365rc1/ Attention macOS users: as of 3.6.5rc1, there is a new additional installer variant for macOS 10.9+ that includes a built-in version of Tcl/Tk 8.6. This variant is expected to become the default variant in future releases. Check it out! -- Ned Deily nad at python.org -- [] From WhoKnows at newsguy.com Wed Mar 14 01:16:20 2018 From: WhoKnows at newsguy.com (Bill) Date: Wed, 14 Mar 2018 01:16:20 -0400 Subject: Python gotcha of the day In-Reply-To: References: Message-ID: Dan Sommers wrote: > On Wed, 14 Mar 2018 04:08:30 +0000, Steven D'Aprano wrote: > >> Explain the difference between these two triple-quoted strings: > >> But remove the spaces, and two of the quotation marks disappear: >> >> py> """\"""""" >> '"' > > That's (a) a triple quoted string containing a single escaped quote, > followed by (b) a quoted empty string. The two consecutive strings are > concatenated into a single string. The last part stumped me! : ) > > (Too many years of shell programming to miss this one!) > > Dan > From tjol at tjol.eu Wed Mar 14 05:40:38 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Wed, 14 Mar 2018 10:40:38 +0100 Subject: Python gotcha of the day In-Reply-To: References: Message-ID: <9d0e0c8b-5f19-357f-83a0-5f348ec3cab7@tjol.eu> On 2018-03-14 05:08, Steven D'Aprano wrote: > Explain the difference between these two triple-quoted strings: > > Here is a triple-quoted string containing spaces and a triple-quote: > > py> """ \""" """ > ' """ ' > > > But remove the spaces, and two of the quotation marks disappear: > > py> """\"""""" > '"' > > > If nobody gets the answer, I shall reveal all later. > > (Hint: it is not a bug.) > Ah, subtle! Initially I thought the first one was being interpreted as ''' """ ''' and the second one as "" '"' "" "" which left me rather puzzled as to why the first wasn't being interpreted as "" ' "' " " "" but of course that's not what's going on at all. The second one is '''"''' "" As to WHY - in both your examples, the literal can be interpreted as a triple-quoted string, so it is (rather than some combination of single-quoted strings). And, in both cases, the SHORTEST possible reading as a triple-quoted string is used. There, now I can go back to work. - Thomas From brian.j.oney at googlemail.com Wed Mar 14 13:02:09 2018 From: brian.j.oney at googlemail.com (Brian Oney) Date: Wed, 14 Mar 2018 18:02:09 +0100 Subject: Python gotcha of the day In-Reply-To: <9d0e0c8b-5f19-357f-83a0-5f348ec3cab7@tjol.eu> References: <9d0e0c8b-5f19-357f-83a0-5f348ec3cab7@tjol.eu> Message-ID: explicit is better than implicit. That gives me an idea for a module with the following debugging command line functionality. import sass >>> "" ":p" Traceback: Are you telling me that ' ' is supposed to an operator? (Rock thrown) On March 14, 2018 10:40:38 AM GMT+01:00, Thomas Jollans wrote: >On 2018-03-14 05:08, Steven D'Aprano wrote: >> Explain the difference between these two triple-quoted strings: >> >> Here is a triple-quoted string containing spaces and a triple-quote: >> >> py> """ \""" """ >> ' """ ' >> >> >> But remove the spaces, and two of the quotation marks disappear: >> >> py> """\"""""" >> '"' >> >> >> If nobody gets the answer, I shall reveal all later. >> >> (Hint: it is not a bug.) >> > >Ah, subtle! > >Initially I thought the first one was being interpreted as > >''' """ ''' > >and the second one as > >"" '"' "" "" > >which left me rather puzzled as to why the first wasn't being >interpreted as > >"" ' "' " " "" > >but of course that's not what's going on at all. The second one is > >'''"''' "" > >As to WHY - in both your examples, the literal can be interpreted as a >triple-quoted string, so it is (rather than some combination of >single-quoted strings). And, in both cases, the SHORTEST possible >reading as a triple-quoted string is used. > >There, now I can go back to work. > >- Thomas > >-- >https://mail.python.org/mailman/listinfo/python-list From tedo.vrbanec at gmail.com Wed Mar 14 14:30:15 2018 From: tedo.vrbanec at gmail.com (tedo.vrbanec at gmail.com) Date: Wed, 14 Mar 2018 11:30:15 -0700 (PDT) Subject: matplotlib icon Message-ID: <1e8bad6f-87ac-4d42-9e45-57d7d8aba624@googlegroups.com> I am getting this logging.INFO notice: Could not load matplotlib icon: bad option "foobar": must be aspect, attributes, client, colormapwindows, command, deiconify, focusmodel, forget, frame, geometry, grid, group, iconbitmap, iconify, iconmask, iconname, iconphoto, iconposition, iconwindow, manage, maxsize, minsize, overrideredirect, positionfrom, protocol, resizable, sizefrom, stackorder, state, title, transient, or withdraw I even set icon, but this notice remains. There's nothing on web or in matplotlib documentation. Python 2.7, Debian From tjreedy at udel.edu Wed Mar 14 15:23:39 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2018 15:23:39 -0400 Subject: matplotlib icon In-Reply-To: <1e8bad6f-87ac-4d42-9e45-57d7d8aba624@googlegroups.com> References: <1e8bad6f-87ac-4d42-9e45-57d7d8aba624@googlegroups.com> Message-ID: On 3/14/2018 2:30 PM, tedo.vrbanec at gmail.com wrote: > I am getting this logging.INFO notice: > Could not load matplotlib icon: bad option "foobar": must be aspect, attributes, client, colormapwindows, command, deiconify, focusmodel, forget, frame, geometry, grid, group, iconbitmap, iconify, iconmask, iconname, iconphoto, iconposition, iconwindow, manage, maxsize, minsize, overrideredirect, positionfrom, protocol, resizable, sizefrom, stackorder, state, title, transient, or withdraw > I even set icon, but this notice remains. > There's nothing on web or in matplotlib documentation. > Python 2.7, Debian We need more context, such as the code that emits the warning, to be sure of anything. However, some code apparently tries to modify non-existent option 'foobar' on, I suspect, a GUI widget. I have seen similar messages from tkinter/tk. The option mistake prevents the icon load. -- Terry Jan Reedy From fabianosidler at swissonline.ch Wed Mar 14 16:12:57 2018 From: fabianosidler at swissonline.ch (Fabiano Sidler) Date: Wed, 14 Mar 2018 21:12:57 +0100 Subject: TLSServer: certificate one request behind... In-Reply-To: <20180313213716.GA50383@mailboxes.bgd.fipscode.net> References: <20180313213716.GA50383@mailboxes.bgd.fipscode.net> Message-ID: <20180314201257.GA55874@mailboxes.bgd.fipscode.net> Thus wrote Fabiano Sidler: > What's the reason for this? Please find attached my TLSServer. Oh, sorry...! Apparently, the attachment has been stripped. Here inline: === tlsserver.py === from socketserver import ThreadingTCPServer,StreamRequestHandler import ssl class TLSServer(ThreadingTCPServer): def __init__(self, *args, **kwargs): super(TLSServer, self).__init__(*args, **kwargs) ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) ctx.set_servername_callback(self.servername_callback) ctx.check_hostname = False self._ctx = ctx def get_request(self): s,a = super(TLSServer, self).get_request() s = self._ctx.wrap_socket(s, server_side=True) return s,a def servername_callback(self, sock, req_hostname, cb_context): return ssl.ALERT_DESCRIPTION_INTERNAL_ERROR from OpenSSL import crypto as x509 from tempfile import NamedTemporaryFile class SelfSigningServer(TLSServer): def servername_callback(self, sock, req_hostname, cb_context): key = x509.PKey() key.generate_key(x509.TYPE_RSA, 2048) cert = x509.X509() subj = cert.get_subject() subj.C = 'CH' subj.ST = 'ZH' subj.L = 'Zurich' subj.O = 'ACME Inc.' subj.OU = 'IT dept.' subj.CN = req_hostname cert.set_version(0x02) cert.set_serial_number(1000) cert.gmtime_adj_notBefore(0) cert.gmtime_adj_notAfter(10*365*24*60*60) cert.set_issuer(subj) cert.set_pubkey(key) cert.sign(key, 'sha256') certfile = NamedTemporaryFile() keyfile = NamedTemporaryFile() certfile.write(x509.dump_certificate(x509.FILETYPE_PEM, cert)) keyfile.write(x509.dump_privatekey(x509.FILETYPE_PEM, key)) certfile.seek(0) keyfile.seek(0) cb_context.load_cert_chain(certfile=certfile.name, keyfile=keyfile.name) cb_context.set_servername_callback(self.servername_callback) sock.context = cb_context certfile.close() keyfile.close() class SelfSigningHandler(StreamRequestHandler): def handle(self): self.wfile.write(b'Hello World!\r\n') server = SelfSigningServer(('localhost',1234), SelfSigningHandler) server.serve_forever() === tlsserver.py === Thanks again! From lists at mostrom.pp.se Wed Mar 14 16:31:43 2018 From: lists at mostrom.pp.se (Jan Erik =?utf-8?q?Mostr=C3=B6m?=) Date: Wed, 14 Mar 2018 21:31:43 +0100 Subject: macOS specific - reading calendar information Message-ID: I've been trying to find some example of how to read calendar info on macOS but I haven't found anything ... I'm probably just bad at searching !! What I want to do is to read calendar info for a date range. Does anyone know of an example of how to do this? = jem From larry.martell at gmail.com Wed Mar 14 16:40:05 2018 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 14 Mar 2018 16:40:05 -0400 Subject: macOS specific - reading calendar information In-Reply-To: References: Message-ID: On Wed, Mar 14, 2018 at 4:31 PM, Jan Erik Mostr?m wrote: > I've been trying to find some example of how to read calendar info on macOS > but I haven't found anything ... I'm probably just bad at searching !! > > What I want to do is to read calendar info for a date range. Does anyone > know of an example of how to do this? What does 'read calendar info' mean? What exactly are you trying to read from? From Irv at furrypants.com Wed Mar 14 18:04:21 2018 From: Irv at furrypants.com (Irv Kalb) Date: Wed, 14 Mar 2018 15:04:21 -0700 Subject: urllib.request.urlopen fails with https (was: Re: Stock quote APi) In-Reply-To: <1520910075l.12451962l.0l@psu.edu> References: <1520910075l.12451962l.0l@psu.edu> Message-ID: Thanks to Chris A and Roger C for their information. Very helpful and I am working on both of their suggestions. But now I've run into a new related problem. I still am trying to get stock information using Standard Library calls for now (although I understand that the requests module makes this kind of thing very easy). My general approach (Python 3) is to build up a URL by concatenating an parameters, and make a call to url.request.urlopen. I then do a 'read' on what was returned. For example, in my class, I use the following code to get weather data from weatherman.org: URL = 'http://api.openweathermap.org/data/2.5/weather?q=' + city + '&mode=xml'+ '&APPID=' + API_KEY response = urllib.request.urlopen(URL) responseString = str(response.read()) I then have some code to parse out the pieces of the returned information. This works great and an excellent example of the power of using API's with Python. In looking for a stock data API, Chris suggested alpha vantage.co (and Roger made a nice suggestion as to how to parse it). But when I use their API, I get an error: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) I have tried a number of different API's and found that if the URL starts with 'https', it always fails. Even a simple example that I found here: https://pythonspot.com/urllib-tutorial-python-3/ fails the same way. Here's what I see when I try that example: >>> import urllib.request >>> html = urllib.request.urlopen('https://arstechnica.com').read() Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1318, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1239, in request self._send_request(method, url, body, headers, encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1285, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1234, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1026, in _send_output self.send(msg) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 964, in send self.connect() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1400, in connect server_hostname=server_hostname) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket _context=self, _session=session) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__ self.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 1, in html = urllib.request.urlopen('https://arstechnica.com').read() File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 223, in urlopen return opener.open(url, data, timeout) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 526, in open response = self._open(req, data) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 544, in _open '_open', req) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(*args) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1361, in https_open context=self._context, check_hostname=self._check_hostname) File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open raise URLError(err) urllib.error.URLError: >>> All my tests return the same tracebacks. Am I doing something wrong? Is there another way (besides using the requests module which DOES work for me) to get data from an https URL? Any help would be appreciated. Thanks, Irv From rosuav at gmail.com Wed Mar 14 18:31:27 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 09:31:27 +1100 Subject: urllib.request.urlopen fails with https (was: Re: Stock quote APi) In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: On Thu, Mar 15, 2018 at 9:04 AM, Irv Kalb wrote: > ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) > > Am I doing something wrong? Is there another way (besides using the requests module which DOES work for me) to get data from an https URL? So.... my reading of this is that you have a major issue with root certificates. Bleh. The normal response is "use pip to install something that has your certs", but if you really absolutely cannot do that, you may have to just disable certificate verification. This is NOT a good thing to do, and you should have a big noisy comment explaining why you have to make your code insecure. Put that RIGHT next to the code that disables verification, so that if anyone copies and pastes it, they'll have a good chance of seeing it. (Basically, what you're doing is downgrading the protection of HTTPS to something nearer plain HTTP. That's fine for what you're doing, but any code you give to students is likely to be copied and pasted into their production code.) See if you can tie in with your OS's cert store first. If you can't, look at "import ssl" and creating a custom SSL context that doesn't verify. ChrisA From tim at akwebsoft.com Wed Mar 14 19:20:26 2018 From: tim at akwebsoft.com (Tim Johnson) Date: Wed, 14 Mar 2018 15:20:26 -0800 Subject: Restore via pip to new OS Message-ID: <20180314232026.GD4384@mail.akwebsoft.com> I'm currently running both python and python3 on ubuntu 14.04. Plan is to do a complete re-install of ubuntu 16.04 on a fresh hard drive. I've accumulated a list of pip-install packages via pip list > piplist.txt. Can I duplicate the same packages on the new OS by pip -r piplist.txt? thanks -- Tim http://www.akwebsoft.com, http://www.tj49.com From greg.ewing at canterbury.ac.nz Wed Mar 14 19:27:48 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 15 Mar 2018 12:27:48 +1300 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: Chris Angelico wrote: > (Basically, > what you're doing is downgrading the protection of HTTPS to something > nearer plain HTTP. That's fine for what you're doing, but any code you > give to students is likely to be copied and pasted into their > production code.) > > See if you can tie in with your OS's cert store first. If you can't, > look at "import ssl" and creating a custom SSL context that doesn't > verify. That's likely to distract and confuse students as well. I would suggest looking for a different example that doesn't require interacting with an https site. -- Greg From rosuav at gmail.com Wed Mar 14 19:33:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 10:33:59 +1100 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: On Thu, Mar 15, 2018 at 10:27 AM, Gregory Ewing wrote: > Chris Angelico wrote: >> >> (Basically, >> what you're doing is downgrading the protection of HTTPS to something >> nearer plain HTTP. That's fine for what you're doing, but any code you >> give to students is likely to be copied and pasted into their >> production code.) >> >> See if you can tie in with your OS's cert store first. If you can't, >> look at "import ssl" and creating a custom SSL context that doesn't >> verify. > > > That's likely to distract and confuse students as well. > > I would suggest looking for a different example that > doesn't require interacting with an https site. That means going back to the original problem: "how do we get a usable stock price API?". My preferred solution is to just install 'requests', of course. ChrisA From p.f.moore at gmail.com Wed Mar 14 19:40:06 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Mar 2018 23:40:06 +0000 Subject: Restore via pip to new OS In-Reply-To: <20180314232026.GD4384@mail.akwebsoft.com> References: <20180314232026.GD4384@mail.akwebsoft.com> Message-ID: Use pip freeze rather than pip list. That will give you the information in "requirements file" format that pip install -r can read. Paul On 14 March 2018 at 23:20, Tim Johnson wrote: > I'm currently running both python and python3 on ubuntu 14.04. > Plan is to do a complete re-install of ubuntu 16.04 on a fresh > hard drive. > > I've accumulated a list of pip-install packages via > pip list > piplist.txt. > > Can I duplicate the same packages on the new OS by > pip -r piplist.txt? > > thanks > -- > Tim > http://www.akwebsoft.com, http://www.tj49.com > -- > https://mail.python.org/mailman/listinfo/python-list From tim at akwebsoft.com Wed Mar 14 19:55:34 2018 From: tim at akwebsoft.com (Tim Johnson) Date: Wed, 14 Mar 2018 15:55:34 -0800 Subject: Restore via pip to new OS In-Reply-To: References: <20180314232026.GD4384@mail.akwebsoft.com> Message-ID: <20180314235534.GE4384@mail.akwebsoft.com> * Paul Moore [180314 15:42]: > Use pip freeze rather than pip list. That will give you the > information in "requirements file" format that pip install -r can > read. > > On 14 March 2018 at 23:20, Tim Johnson wrote: <...> > > Can I duplicate the same packages on the new OS by > > pip -r piplist.txt? Thank you Paul. Cheers -- Tim http://www.akwebsoft.com, http://www.tj49.com From ben.usenet at bsb.me.uk Wed Mar 14 21:56:08 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Thu, 15 Mar 2018 01:56:08 +0000 Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <878tazwfta.fsf@bsb.me.uk> <87muzcvlj7.fsf@bsb.me.uk> <87zi3btpyk.fsf@bsb.me.uk> <2a7a6d8a-c202-4ac0-8e16-5a6cda30468b@googlegroups.com> Message-ID: <87vadygkzr.fsf@bsb.me.uk> Lawrence D?Oliveiro writes: > On Wednesday, March 14, 2018 at 2:18:24 PM UTC+13, Ben Bacarisse wrote: >> Lawrence D?Oliveiro writes: >> >> The original problem -- triples of natural numbers -- is >> not particularly hard, but the general problem -- enumerating n-tuples >> of some sequence -- is more interesting because it is a bit harder. > > It?s not harder--it?s exactly the same difficulty. You iterate the > naturals, then interpose a function mapping them onto the set that you > really want to use. Yes, I said exactly that a couple of messages ago. -- Ben. From greg.ewing at canterbury.ac.nz Thu Mar 15 00:54:15 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Thu, 15 Mar 2018 17:54:15 +1300 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: Chris Angelico wrote: > That means going back to the original problem: "how do we get a usable > stock price API?". Does it have to be stock prices in particular? Or just some simple piece of data that demonstrates the principles of fetching a url and parsing the result? -- Greg From rosuav at gmail.com Thu Mar 15 01:26:34 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 16:26:34 +1100 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: On Thu, Mar 15, 2018 at 3:54 PM, Gregory Ewing wrote: > Chris Angelico wrote: >> >> That means going back to the original problem: "how do we get a usable >> stock price API?". > > > Does it have to be stock prices in particular? > Or just some simple piece of data that demonstrates > the principles of fetching a url and parsing the > result? > You'd have to ask the OP, but if you make too big a change to the API used, the rest of the course might have to change too. If the next thing done with the downloaded data is to graph it, for instance, then you'll need some other API that gives graphable data. Etcetera, etcetera, etcetera. ChrisA From ben+python at benfinney.id.au Thu Mar 15 02:11:43 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 15 Mar 2018 17:11:43 +1100 Subject: Python gotcha of the day References: Message-ID: <85k1ud50m8.fsf@benfinney.id.au> Steven D'Aprano writes: > py> """\"""""" > '"' That's an empty string delimited by ?"?; followed by a double-quote character, escaped, delimited by ?"?; followed by two more empty strings. They concatenate to a single one-character string. Equivalent to "" + "\"" + "" + "" without the ?+? operations. > If nobody gets the answer, I shall reveal all later. My interpretation may not be how the language defines it; but it does match the result! -- \ ?I have yet to see any problem, however complicated, which, | `\ when you looked at it in the right way, did not become still | _o__) more complicated.? ?Paul Anderson | Ben Finney From lists at mostrom.pp.se Thu Mar 15 03:32:35 2018 From: lists at mostrom.pp.se (Jan Erik =?utf-8?q?Mostr=C3=B6m?=) Date: Thu, 15 Mar 2018 08:32:35 +0100 Subject: macOS specific - reading calendar information In-Reply-To: References: Message-ID: On 14 Mar 2018, at 21:40, Larry Martell wrote: >> I've been trying to find some example of how to read calendar info on >> macOS >> but I haven't found anything ... I'm probably just bad at searching >> !! >> >> What I want to do is to read calendar info for a date range. Does >> anyone >> know of an example of how to do this? > > > What does 'read calendar info' mean? What exactly are you trying to > read from? I would like to read what calendar events I have on a range of days. I would like to get the data from whatever storage Calendar use, in my personal case I sync to iCloud. An example would be something along these lines x = getcalendarevents('2018-03-15', '2018-03-25') or even better x = getcalendarevents('2018-03-15', '2018-03-25',[calendar1, calendar2, ...]) and x would contain some data structure that gives me the events that occurs from 2018-03-15 up to and including 2018-03-25, each event containing at least start/end time, title, place, calendar to which the event belong. = jem From auriocus at gmx.de Thu Mar 15 03:58:13 2018 From: auriocus at gmx.de (Christian Gollwitzer) Date: Thu, 15 Mar 2018 08:58:13 +0100 Subject: macOS specific - reading calendar information In-Reply-To: References: Message-ID: Am 15.03.18 um 08:32 schrieb Jan Erik Mostr?m: > I would like to read what calendar events I have on a range of days. I > would like to get the data from whatever storage Calendar use, in my > personal case I sync to iCloud. > > An example would be something along these lines > > x = getcalendarevents('2018-03-15', '2018-03-25') > > or even better > > x = getcalendarevents('2018-03-15', '2018-03-25',[calendar1, calendar2, > ...]) > > > and x would contain some data structure that gives me the events that > occurs from 2018-03-15 up to and including 2018-03-25, each event > containing at least start/end time, title, place, calendar to which the > event belong. I assume you mean the calendar app included within OSX. It can be scripted, start from here: https://developer.apple.com/library/content/documentation/AppleApplications/Conceptual/CalendarScriptingGuide/index.html The native script language is Apple Script or JavaScript. If you want to control it from Python, you need a bridge. I have no experience with it, but a quick Google search reveals https://pypi.python.org/pypi/py-applescript and https://docs.python.org/2/library/macosa.html Christian From tjol at tjol.eu Thu Mar 15 05:21:24 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Thu, 15 Mar 2018 10:21:24 +0100 Subject: Python gotcha of the day In-Reply-To: <85k1ud50m8.fsf@benfinney.id.au> References: <85k1ud50m8.fsf@benfinney.id.au> Message-ID: On 2018-03-15 07:11, Ben Finney wrote: > Steven D'Aprano writes: > >> py> """\"""""" >> '"' > > That's an empty string delimited by ?"?; followed by a double-quote > character, escaped, delimited by ?"?; followed by two more empty > strings. They concatenate to a single one-character string. > > Equivalent to > > "" + "\"" + "" + "" > > without the ?+? operations. > >> If nobody gets the answer, I shall reveal all later. > > My interpretation may not be how the language defines it; but it does > match the result! > Then riddle me this: if """\"""""" is equivalent to "" + "\"" + "" + "", then why isn't """ \""" """ equivalent to "" + " \"" + " " + ""? As I said earlier: I initially thought the way you're thinking, but got stuck at that question and had to back-track my reasoning :-) T From songofacandy at gmail.com Thu Mar 15 07:08:16 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 15 Mar 2018 20:08:16 +0900 Subject: Homebrew changed default Python to python 3! In-Reply-To: References: Message-ID: FYI, they reverted python->python3 symlink. python command is now Python 2 again. https://discourse.brew.sh/t/python-and-pep-394/1813 Even though this revert, it is significant step: * many formulas dropped `depends_on "python"`. Python 2 was installed often by dependency before but it's rare for now. * Some formulas (including vim) keep `depends_on "python". It means they are depending on Python 3 instead of 2. It makes vim plugin authors can consider about dropping Python 2 support. (vim is linked with Python 3 on recent Ubuntu already) I hope Apple will provide "/usr/bin/python2" at least in next macOS, to allow people using "python2" shebang for scripts support only Python 2. (And I hope Python 3.6 (or 3.7) is pre-installed in next macOS)... From ben.usenet at bsb.me.uk Thu Mar 15 07:20:30 2018 From: ben.usenet at bsb.me.uk (Ben Bacarisse) Date: Thu, 15 Mar 2018 11:20:30 +0000 Subject: Enumerating all 3-tuples (Posting On Python-List Prohibited) References: <878tb0zm2f.fsf@bsb.me.uk> <871sgsypoq.fsf@bsb.me.uk> <878tazwfta.fsf@bsb.me.uk> <87muzcvlj7.fsf@bsb.me.uk> <87zi3btpyk.fsf@bsb.me.uk> <2a7a6d8a-c202-4ac0-8e16-5a6cda30468b@googlegroups.com> <87vadygkzr.fsf@bsb.me.uk> <937d14be-5d2e-400c-ab26-78126b60875b@googlegroups.com> Message-ID: <87lgetfuv5.fsf@bsb.me.uk> Lawrence D?Oliveiro writes: > On Thursday, March 15, 2018 at 2:56:24 PM UTC+13, Ben Bacarisse wrote: >> >> Lawrence D?Oliveiro writes: >> >>> On Wednesday, March 14, 2018 at 2:18:24 PM UTC+13, Ben Bacarisse wrote: >>> >>>> Lawrence D?Oliveiro writes: >>>> >>>> The original problem -- triples of natural numbers -- is >>>> not particularly hard, but the general problem -- enumerating n-tuples >>>> of some sequence -- is more interesting because it is a bit harder. >>> >>> It?s not harder--it?s exactly the same difficulty. You iterate the >>> naturals, then interpose a function mapping them onto the set that you >>> really want to use. >> >> Yes, I said exactly that a couple of messages ago. > > I see. So you said it is a bit harder, and you also said it?s not > harder at all. Got it. Yes, I was not clear. It was the point about mapping from a solution that uses the natural numbers that's I mentioned before. The "bit harder" comes from generalising to n-tuples. (I get the feeling I should apologise for something because I think I have annoyed or irritated you. I'm sure I don't always get the right tone but I really hope I've not been rude. I'm sorry if I have been.) -- Ben. From arek.bulski at gmail.com Thu Mar 15 07:54:36 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Thu, 15 Mar 2018 11:54:36 +0000 Subject: List slicing on Python 2.7 Message-ID: I have a custom class (a lazy list-like container) that needs to support slicing. The __getitem__ checks if index is of slice type, and does a list comprehension over individual integer indexes. The code works fine on Python 3 but fails on 2.7, both CPython and PyPy. The print inside __getitem__ doesnt even get executed when its a slice. Does 2.7 have different object model, where slices are handled by a different method than __getitem__? The implementation and error log https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 https://travis-ci.org/construct/construct/jobs/353782126#L887 -- ~ Arkadiusz Bulski From dkasak at termina.org.uk Thu Mar 15 09:09:16 2018 From: dkasak at termina.org.uk (Denis Kasak) Date: Thu, 15 Mar 2018 14:09:16 +0100 Subject: Enumerating all 3-tuples In-Reply-To: <765fc7e7a1b8aeb716d251fd15cc2919@termina.org.uk> References: <765fc7e7a1b8aeb716d251fd15cc2919@termina.org.uk> Message-ID: On 2018-03-13 23:56, Denis Kasak wrote: > On 2018-03-10 02:13, Steven D'Aprano wrote: >> >> But I've stared at this for an hour and I can't see how to extend the >> result to three coordinates. I can lay out a grid in the order I want: >> >> 1,1,1 1,1,2 1,1,3 1,1,4 ... >> 2,1,1 2,1,2 2,1,3 2,1,4 ... >> 1,2,1 1,2,2 1,2,3 1,2,4 ... >> 3,1,1 3,1,2 3,1,3 3,1,4 ... >> 2,2,1 2,2,2 2,2,3 2,2,4 ... >> ... >> >> and applying Cantor's diagonal order will give me what I want, but >> damned >> if I can see how to do it in code. > [snip] > > The triples can be viewed as a pair of a pair and a natural number: > > (1,1),1 (1,1),2 (1,1),3 ... > (2,1),1 (2,1),2 (2,1),3 ... > (1,2),1 (1,2),2 (1,2),3 ... > . . . > . . . > . . . [snip] > def c3(i): > """ > Inverse of the Cantor pairing function generalization to > triples, > mapping N ? N?N?N. > """ > n, m = c(i) > return c(n) + (m,) In fact, extending this idea further, we can view the grid as a Cartesian product of two sets: the natural numbers and an arbitrary enumerable set. In your intended grid layout, the natural numbers were put (in their usual ordering) on the horizontal axis and the 2-tuples given by the pairing function on the vertical axis. However, diagonalizing this grid allows us to enumerate the Cartesian product itself, which means we can repeat the process by using this enumeration as the new vertical axis. Therefore, this process generalizes naturally to an arbitrary number of dimensions: def cr(i, d=2): """ Inverse of the Cantor pairing function generalization to d-tuples, mapping N ? N^d. """ if d == 1: return i elif d == 2: return c(i) else: n, m = c(i) return cr(n, d-1) + (m,) >>> def first_ten(d): ... return [cr(i, d) for i in range(1, 11)] >>> for d in range(2, 5): ... pprint(first_ten(d)) [(1, 1), (2, 1), (1, 2), (3, 1), (2, 2), (1, 3), (4, 1), (3, 2), (2, 3), (1, 4)] [(1, 1, 1), (2, 1, 1), (1, 1, 2), (1, 2, 1), (2, 1, 2), (1, 1, 3), (3, 1, 1), (1, 2, 2), (2, 1, 3), (1, 1, 4)] [(1, 1, 1, 1), (2, 1, 1, 1), (1, 1, 1, 2), (1, 1, 2, 1), (2, 1, 1, 2), (1, 1, 1, 3), (1, 2, 1, 1), (1, 1, 2, 2), (2, 1, 1, 3), (1, 1, 1, 4)] -- Denis Kasak From michealmancini at gmail.com Thu Mar 15 09:28:36 2018 From: michealmancini at gmail.com (michealmancini at gmail.com) Date: Thu, 15 Mar 2018 06:28:36 -0700 (PDT) Subject: number of loops Message-ID: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> I would like to have this offer 6 guesses, but instead it gives one guess and prints the if statement/output 6 times....any ideas where I went wrong? import random number_of_guesses = 0 print ('Hello! what is your name?') my_name=input() print (my_name + " " + 'sounds like the name of my next opponent') print ('Shall we play?') yes_or_no = input() if yes_or_no == 'yes': print ('ok! guess my number') else: print ('log off and go home') my_number=random.randint(1,100) your_guess=input() your_guess=int(your_guess) for number in range(6): if your_guess > my_number: print ('nope, too high') if your_guess < my_number: print ('nope, too low') if your_guess == my_number: break if your_guess == my_number: number_of_guesses=str(number_of_guesses) print ('good job, you guessed correctly') else: print('nope, you lose') From vlastimil.brom at gmail.com Thu Mar 15 09:57:20 2018 From: vlastimil.brom at gmail.com (Vlastimil Brom) Date: Thu, 15 Mar 2018 14:57:20 +0100 Subject: List slicing on Python 2.7 In-Reply-To: References: Message-ID: 2018-03-15 12:54 GMT+01:00 Arkadiusz Bulski : > I have a custom class (a lazy list-like container) that needs to support > slicing. The __getitem__ checks if index is of slice type, and does a list > comprehension over individual integer indexes. The code works fine on > Python 3 but fails on 2.7, both CPython and PyPy. The print inside > __getitem__ doesnt even get executed when its a slice. Does 2.7 have > different object model, where slices are handled by a different method than > __getitem__? > > The implementation and error log > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 > https://travis-ci.org/construct/construct/jobs/353782126#L887 > > -- > ~ Arkadiusz Bulski > -- > https://mail.python.org/mailman/listinfo/python-list Hi, it looks like, the older method __getslice__ is still used in python 2.7 cf.: https://docs.python.org/2/reference/datamodel.html#object.__getslice__ You might need to implement this method in your class as well for compatibility with python 2. regards vbr From ned at nedbatchelder.com Thu Mar 15 11:24:22 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Mar 2018 11:24:22 -0400 Subject: List slicing on Python 2.7 In-Reply-To: References: Message-ID: <41ff03e4-7cf0-7fb2-778c-ea66d70ab1a4@nedbatchelder.com> On 3/15/18 9:57 AM, Vlastimil Brom wrote: > 2018-03-15 12:54 GMT+01:00 Arkadiusz Bulski : >> I have a custom class (a lazy list-like container) that needs to support >> slicing. The __getitem__ checks if index is of slice type, and does a list >> comprehension over individual integer indexes. The code works fine on >> Python 3 but fails on 2.7, both CPython and PyPy. The print inside >> __getitem__ doesnt even get executed when its a slice. Does 2.7 have >> different object model, where slices are handled by a different method than >> __getitem__? >> >> The implementation and error log >> https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 >> https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 >> https://travis-ci.org/construct/construct/jobs/353782126#L887 >> >> -- >> ~ Arkadiusz Bulski >> -- >> https://mail.python.org/mailman/listinfo/python-list > Hi, > it looks like, the older method __getslice__ is still used in python 2.7 > cf.: > https://docs.python.org/2/reference/datamodel.html#object.__getslice__ > > You might need to implement this method in your class as well for > compatibility with python 2. > > Python 2 will use __getitem__ for slices: ??? $ python2.7 ??? Python 2.7.10 (default, May 30 2015, 12:06:13) ??? [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin ??? Type "help", "copyright", "credits" or "license" for more information. ??? >>> class Sliceable(object): ??? ...???? def __getitem__(self, thing): ??? ...???????? print type(thing) ??? ...???????? print thing ??? ... ??? >>> Sliceable()[1:5] ??? ??? slice(1, 5, None) ??? >>> Sliceable()[:] ??? ??? slice(None, None, None) ??? >>> --Ned. From bgailer at gmail.com Thu Mar 15 11:25:37 2018 From: bgailer at gmail.com (Bob Gailer) Date: Thu, 15 Mar 2018 11:25:37 -0400 Subject: number of loops In-Reply-To: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> References: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> Message-ID: On Mar 15, 2018 9:30 AM, wrote: > > I would like to have this offer 6 guesses, but instead it gives one guess and prints the if statement/output 6 times....any ideas where I went wrong? I suggest you conduct a walk-through. That means pointing using a pencil or a mouse pointer at each statement and noticing what it does. pay attention to what happens when the loop repeats. This will do more for you than if I just tell you the answer. > > import random > > number_of_guesses = 0 > > print ('Hello! what is your name?') > my_name=input() > print (my_name + " " + 'sounds like the name of my next opponent') > print ('Shall we play?') > > yes_or_no = input() > > if yes_or_no == 'yes': > print ('ok! guess my number') > else: > print ('log off and go home') > > my_number=random.randint(1,100) > > your_guess=input() > your_guess=int(your_guess) > > for number in range(6): > if your_guess > my_number: > print ('nope, too high') > if your_guess < my_number: > print ('nope, too low') > if your_guess == my_number: > break > > if your_guess == my_number: > number_of_guesses=str(number_of_guesses) > print ('good job, you guessed correctly') > else: > print('nope, you lose') > > -- > https://mail.python.org/mailman/listinfo/python-list From Irv at furrypants.com Thu Mar 15 11:43:55 2018 From: Irv at furrypants.com (Irv Kalb) Date: Thu, 15 Mar 2018 08:43:55 -0700 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: <11B4D9D6-43BE-4378-B9E8-D0A09FB64BC5@furrypants.com> On Mar 14, 2018, at 9:54 PM, Gregory Ewing wrote: > > Chris Angelico wrote: >> That means going back to the original problem: "how do we get a usable >> stock price API?". > > Does it have to be stock prices in particular? > Or just some simple piece of data that demonstrates > the principles of fetching a url and parsing the > result? > > -- > Greg > -- I am the OP. No, it does not have to be stock prices. I used stock prices because it made for a very clear example in my curriculum. In my class, I demo how we can write a small Python program to "pretend to be a browser" and get a full page of html, then I talk about how you can use an API with a small Python program to get just the data you really want. I found a Yahoo API that did exactly what I wanted, and that has worked for years - but now Yahoo has cancelled that service. I thought it would be simple to find another simple API to get a stock price but I haven't found one. All the ones that do work seem to require https, which doesn't work easily with url lib.request.urlopen. I do use other API's like openweathermap.org to get weather data and api.fixer.io to get currency exchange rates. I get the data back and work through parsing the results. I am still very interested in getting a stock quote example but If you have other examples that have a URL that is http based, I am interested. Irv From arek.bulski at gmail.com Thu Mar 15 12:07:14 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Thu, 15 Mar 2018 16:07:14 +0000 Subject: List slicing on Python 2.7 In-Reply-To: References: Message-ID: Found the answer on stack overflow. Some types on some runtimes (builtins and on Python 2) use another method __getslice__ instead of __getitem__. https://docs.python.org/2/reference/datamodel.html#object.__getslice__ czw., 15 mar 2018 o 12:54 u?ytkownik Arkadiusz Bulski napisa?: > I have a custom class (a lazy list-like container) that needs to support > slicing. The __getitem__ checks if index is of slice type, and does a list > comprehension over individual integer indexes. The code works fine on > Python 3 but fails on 2.7, both CPython and PyPy. The print inside > __getitem__ doesnt even get executed when its a slice. Does 2.7 have > different object model, where slices are handled by a different method than > __getitem__? > > The implementation and error log > > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 > > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 > https://travis-ci.org/construct/construct/jobs/353782126#L887 > > -- > ~ Arkadiusz Bulski > -- ~ Arkadiusz Bulski From __peter__ at web.de Thu Mar 15 12:35:46 2018 From: __peter__ at web.de (Peter Otten) Date: Thu, 15 Mar 2018 17:35:46 +0100 Subject: List slicing on Python 2.7 References: <41ff03e4-7cf0-7fb2-778c-ea66d70ab1a4@nedbatchelder.com> Message-ID: Ned Batchelder wrote: > On 3/15/18 9:57 AM, Vlastimil Brom wrote: >> 2018-03-15 12:54 GMT+01:00 Arkadiusz Bulski : >>> I have a custom class (a lazy list-like container) that needs to support >>> slicing. The __getitem__ checks if index is of slice type, and does a >>> list comprehension over individual integer indexes. The code works fine >>> on Python 3 but fails on 2.7, both CPython and PyPy. The print inside >>> __getitem__ doesnt even get executed when its a slice. Does 2.7 have >>> different object model, where slices are handled by a different method >>> than __getitem__? >>> >>> The implementation and error log >>> https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 >>> https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 >>> https://travis-ci.org/construct/construct/jobs/353782126#L887 >>> >>> -- >>> ~ Arkadiusz Bulski >>> -- >>> https://mail.python.org/mailman/listinfo/python-list >> Hi, >> it looks like, the older method __getslice__ is still used in python 2.7 >> cf.: >> https://docs.python.org/2/reference/datamodel.html#object.__getslice__ >> >> You might need to implement this method in your class as well for >> compatibility with python 2. >> >> > > Python 2 will use __getitem__ for slices: > > $ python2.7 > Python 2.7.10 (default, May 30 2015, 12:06:13) > [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> class Sliceable(object): > ... def __getitem__(self, thing): > ... print type(thing) > ... print thing > ... > >>> Sliceable()[1:5] > > slice(1, 5, None) > >>> Sliceable()[:] > > slice(None, None, None) > >>> __getslice__() takes precedence, and the OP subclasses list: $ python Python 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> class A(list): ... def __getitem__(self, index): return index ... >>> a = A() >>> a[:] [] >>> a[::] slice(None, None, None) >>> A.__getslice__ = lambda self, *args: args >>> a[:] (0, 9223372036854775807) From jcasale at activenetwerx.com Thu Mar 15 14:17:31 2018 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Thu, 15 Mar 2018 18:17:31 +0000 Subject: Context manager on method call from class Message-ID: <377f3351b8dc4bf2a4183df968d3434f@activenetwerx.com> I have a class which implements a context manager, its __init__ has a signature and the __enter__ returns an instance of the class. Along with several methods which implement functionality on the instance, I have one method which itself must open a context manager against a call on an instance attribute. This context manager does not return an instance of itself, it merely opens a context. I am not thrilled about is the implementation I have used and I am wondering if there is a better way. It's been a long time since I have worked in Python and I am probably overlooking something obvious. Syntactically I achieve what I want in use, but it looks awkward in its implementation. How can I clean up the first class and decorator to implement the functionality on a method in the Foo class? class MethodContextManager: def __init__(self, obj, key, value): self.obj = obj self.key = key self.value = value def __enter__(self): self.obj.some_method(key, value) def __exit__(self, exc_type, exc_val, exc_tb): self.obj.another_method() def method_context_manager(func): def wrapper(self, key, value): return MethodContextManager(self.instance, key, value) return wrapper class Foo: def __init__(self, bar): self.bar = bar self.instance = None def __enter__(self): self.instance = SomeFunc(this.bar) return self def __exit__(self, exc_type, exc_val, exc_tb): # Cleanup... @method_context_manager def baz(self, key, value): pass with Foo(bar) as foo: foo.other_call() with foo.baz(42, "a"): foo.other_call() with foo.baz(420, None): foo.other_call() foo.other_call() foo.other_call() foo.other_call() From rgaddi at highlandtechnology.invalid Thu Mar 15 14:47:26 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Thu, 15 Mar 2018 11:47:26 -0700 Subject: Context manager on method call from class In-Reply-To: References: <377f3351b8dc4bf2a4183df968d3434f@activenetwerx.com> Message-ID: On 03/15/2018 11:17 AM, Joseph L. Casale wrote: > I have a class which implements a context manager, its __init__ > has a signature and the __enter__ returns an instance of the > class. > > Along with several methods which implement functionality on > the instance, I have one method which itself must open a context > manager against a call on an instance attribute. This context manager > does not return an instance of itself, it merely opens a context. > > I am not thrilled about is the implementation I have used and I > am wondering if there is a better way. It's been a long time since > I have worked in Python and I am probably overlooking something > obvious. Syntactically I achieve what I want in use, but it looks awkward > in its implementation. How can I clean up the first class and decorator > to implement the functionality on a method in the Foo class? > > [snip] from contextlib import contextmanager. Then you just use the @contextmanager decorator on a function, have it set up, yield the context you want, and clean up after. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From ben+python at benfinney.id.au Thu Mar 15 14:59:49 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 16 Mar 2018 05:59:49 +1100 Subject: Python gotcha of the day References: <85k1ud50m8.fsf@benfinney.id.au> Message-ID: <85fu514122.fsf@benfinney.id.au> Thomas Jollans writes: > Then riddle me this: > > if """\"""""" is equivalent to "" + "\"" + "" + "", then why isn't > """ \""" """ equivalent to "" + " \"" + " " + ""? Who can say? I was only asked for an explanation, not a consistent one :-) Perhaps it's a micro-optimisation, to create more empty strings that can be discarded quickly. Perhaps not. > As I said earlier: I initially thought the way you're thinking, but got > stuck at that question and had to back-track my reasoning :-) I was not hindered by the burden to make my explanation work for everything, just for the one example presented. This allowed me to quickly get back to feeding my goldfish. -- \ ?What do religious fundamentalists and big media corporations | `\ have in common? They believe that they own culture, they are so | _o__) self-righteous about it ?? ?Nina Paley, 2011 | Ben Finney From jcasale at activenetwerx.com Thu Mar 15 15:15:50 2018 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Thu, 15 Mar 2018 19:15:50 +0000 Subject: Context manager on method call from class In-Reply-To: References: <377f3351b8dc4bf2a4183df968d3434f@activenetwerx.com> , Message-ID: <59b66fb4b1b84bcb86e90e4571cc16d0@activenetwerx.com> From: Python-list on behalf of Rob Gaddi Sent: Thursday, March 15, 2018 12:47 PM To: python-list at python.org Subject: Re: Context manager on method call from class ? > from contextlib import contextmanager. > > Then you just use the @contextmanager decorator on a function, have it > set up, yield the context you want, and clean up after. Nice, all of what I wrote replaced with three lines:) Thank you, jlc From listes at salort.eu Thu Mar 15 15:35:37 2018 From: listes at salort.eu (Julien Salort) Date: Thu, 15 Mar 2018 20:35:37 +0100 Subject: Make synchronous generator from an asynchronous generator Message-ID: <599ef329-95b4-10ea-98c1-1ca1669486e5@salort.eu> Hello, I have recently got the need to convert a synchronous function to asynchronous function because I needed to run three of them concurrently. Since I am using Python 3.6, I used async def and asyncio and that has worked out great. Thank you guys for making this possible. Because I wanted to keep the synchronous function for scripts which used it, without unnecessarily duplicating the code, I built also a synchronous function from this new asynchronous one, like that: def acquire_to_files(self, *args, **kwargs): ??? loop = asyncio.get_event_loop() ??? loop.run_until_complete(self.acquire_to_files_async(*args, **kwargs)) ??? loop.close() So far, it works fine. I am quite happy. I can await acquire_to_files_async in situations where I need concurrency, and call acquire_to_files in situations where I don't, and no code is duplicated. Now I wish to do the same game with a generator (which happens to be called by the acquire_to_files_async function). So I made it into an asynchronous generator, and it works as expected. My problem: how do I proceed if I wish to build a synchronous generator from this asynchronous generator ? (for situations where concurrency is not needed) i.e. what is the equivalent of the above snippet for generators ? Thanks, Julien PS: I was surprised however that enumerate does not work when applied to an asynchronous generator, i.e.: async for ii, img in enumerate(async_gen()): ??? pass raises TypeError: 'async_gen' object is not iterable. From ned at nedbatchelder.com Thu Mar 15 16:28:41 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 15 Mar 2018 16:28:41 -0400 Subject: List slicing on Python 2.7 In-Reply-To: References: <41ff03e4-7cf0-7fb2-778c-ea66d70ab1a4@nedbatchelder.com> Message-ID: <61934013-a7dc-9454-ae5b-92b92f5970cd@nedbatchelder.com> On 3/15/18 12:35 PM, Peter Otten wrote: > Ned Batchelder wrote: > >> On 3/15/18 9:57 AM, Vlastimil Brom wrote: >>> 2018-03-15 12:54 GMT+01:00 Arkadiusz Bulski : >>>> I have a custom class (a lazy list-like container) that needs to support >>>> slicing. The __getitem__ checks if index is of slice type, and does a >>>> list comprehension over individual integer indexes. The code works fine >>>> on Python 3 but fails on 2.7, both CPython and PyPy. The print inside >>>> __getitem__ doesnt even get executed when its a slice. Does 2.7 have >>>> different object model, where slices are handled by a different method >>>> than __getitem__? >>>> >>>> The implementation and error log >>>> > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/construct/core.py#L4785-L4796 > https://github.com/construct/construct/blob/8839aac2b68c9e8240e9d9c041a196b0a7aa7d9b/tests/test_core.py#L1148 >>>> https://travis-ci.org/construct/construct/jobs/353782126#L887 >>>> >>>> -- >>>> ~ Arkadiusz Bulski >>>> -- >>>> https://mail.python.org/mailman/listinfo/python-list >>> Hi, >>> it looks like, the older method __getslice__ is still used in python 2.7 >>> cf.: >>> https://docs.python.org/2/reference/datamodel.html#object.__getslice__ >>> >>> You might need to implement this method in your class as well for >>> compatibility with python 2. >>> >>> >> Python 2 will use __getitem__ for slices: >> >> $ python2.7 >> Python 2.7.10 (default, May 30 2015, 12:06:13) >> [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>> class Sliceable(object): >> ... def __getitem__(self, thing): >> ... print type(thing) >> ... print thing >> ... >> >>> Sliceable()[1:5] >> >> slice(1, 5, None) >> >>> Sliceable()[:] >> >> slice(None, None, None) >> >>> > __getslice__() takes precedence, and the OP subclasses list: > > $ python > Python 2.7.6 (default, Nov 23 2017, 15:49:48) > [GCC 4.8.4] on linux2 > Type "help", "copyright", "credits" or "license" for more information. >>>> class A(list): > ... def __getitem__(self, index): return index > ... >>>> a = A() >>>> a[:] > [] >>>> a[::] > slice(None, None, None) >>>> A.__getslice__ = lambda self, *args: args >>>> a[:] > (0, 9223372036854775807) > > Another good reason not to subclass list. --Ned. From simon.wonng at gmail.com Thu Mar 15 16:40:51 2018 From: simon.wonng at gmail.com (simon.wonng at gmail.com) Date: Thu, 15 Mar 2018 13:40:51 -0700 (PDT) Subject: Python installer hangs in Windows 7 In-Reply-To: References: <765703485.1601081.1486357381135.ref@mail.yahoo.com> <765703485.1601081.1486357381135@mail.yahoo.com> Message-ID: On Monday, December 4, 2017 at 12:44:48 PM UTC-8, christian... at gmail.com wrote: > Same with me, except that I tried to install Python 3.6.3. Unchecking "Install launcher for all users" helped, however. This worked for me, thanks! From drsalists at gmail.com Thu Mar 15 16:46:59 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 15 Mar 2018 13:46:59 -0700 Subject: number of loops In-Reply-To: References: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> Message-ID: On Thu, Mar 15, 2018 at 8:25 AM, Bob Gailer wrote: > On Mar 15, 2018 9:30 AM, wrote: >> >> I would like to have this offer 6 guesses, but instead it gives one guess > and prints the if statement/output 6 times....any ideas where I went wrong? > > I suggest you conduct a walk-through. That means pointing using a pencil or > a mouse pointer at each statement and noticing what it does. pay attention > to what happens when the loop repeats. I'm guessing a GUI or curses debugger would help quite a bit. EG's: winpdb, pudb. winpdb is cross-platform, despite the name. pudb might not work on Microsoft Windows, though perhaps it would in Cygwin. These, and many other debuggers allow you to single step through your code and see what the execution order is, one statement at a time. It's a pretty valuable learning experience early on, IMO. From python at bladeshadow.org Thu Mar 15 18:58:46 2018 From: python at bladeshadow.org (Python) Date: Thu, 15 Mar 2018 17:58:46 -0500 Subject: Python gotcha of the day In-Reply-To: References: <85k1ud50m8.fsf@benfinney.id.au> Message-ID: <20180315225846.GU10273@bladeshadow.org> On Thu, Mar 15, 2018 at 10:21:24AM +0100, Thomas Jollans wrote: > On 2018-03-15 07:11, Ben Finney wrote: > > Steven D'Aprano writes: > > > >> py> """\"""""" [...] > Then riddle me this: > > if """\"""""" is equivalent to "" + "\"" + "" + "", then why isn't > """ \""" """ equivalent to "" + " \"" + " " + ""? I've been away on vacation and have deleted a bunch of messages, so it's likely someone else has already answered this, but if not the "secret sauce" is that three consecutive quotes are one token, making the string (or part of it) a triple-quoted string. That makes the result of both strings consistent, i.e.: """\"""""" => '"' - The first three quotes start the triple-quoted string. - The quote after the backslash is escaped, and is part of the string - The next three quotes end the triple-quoted string - the remaining two quotes yeild the empty string The version with spaces is completely consistent: """ \""" """ => ' """ ' - The first three quotes start the triple-quoted string. - They're followed by a space, part of the TQS. - The quote after the backslash is escaped, and is part of the string - The next TWO quotes do not terminate the TQS, because there are not three of them consecutively, thus are part of the TQS. - the space is clearly part of the TQS. - the remaining three consecutive quotes terminate the TQS. But the real answer, I think, is: Just Say No to this in real code... It's extraordinarily unlikely that you'll have a real-world case that requires it, and it's extremely difficult to read even in you know well the relevant rules. If you found yourself tempted to do this, there's almost certainly a clearer way to write it, either by reducing the thing to its result, or by doing string addtion, or what have you. From nad at python.org Thu Mar 15 22:39:38 2018 From: nad at python.org (Ned Deily) Date: Thu, 15 Mar 2018 22:39:38 -0400 Subject: urllib.request.urlopen fails with https In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: On 2018-03-14 18:04, Irv Kalb wrote: > File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open > raise URLError(err) > urllib.error.URLError: If you are using Python 3.6 for macOS from a python.org installer, did you follow the instructions displayed in the installer ReadMe and also saved at: /Applications/Python 3.6/ReadMe.rtf to run the "Install Certificates.command" ? Either double-click on it in the Finder or, from a shell command line, type: open "/Applications/Python 3.6/Install Certificates.command" Certificate verification and OpenSSL **NEW** This variant of Python 3.6 now includes its own private copy of OpenSSL 1.0.2. Unlike previous releases, the deprecated Apple-supplied OpenSSL libraries are no longer used. This also means that the trust certificates in system and user keychains managed by the Keychain Access application and the security command line utility are no longer used as defaults by the Python ssl module. For 3.6.0, a sample command script is included in /Applications/Python 3.6 to install a curated bundle of default root certificates from the third-party certifi package (https://pypi.python.org/pypi/certifi). If you choose to use certifi, you should consider subscribing to the project's email update service to be notified when the certificate bundle is updated. From nad at python.org Thu Mar 15 22:48:11 2018 From: nad at python.org (Ned Deily) Date: Thu, 15 Mar 2018 22:48:11 -0400 Subject: macOS specific - reading calendar information In-Reply-To: References: Message-ID: On 2018-03-15 03:58, Christian Gollwitzer wrote: > Am 15.03.18 um 08:32 schrieb Jan Erik Mostr?m: >> I would like to read what calendar events I have on a range of days. I >> would like to get the data from whatever storage Calendar use, in my >> personal case I sync to iCloud. [...] > The native script language is Apple Script or JavaScript. If you want to > control it from Python, you need a bridge. I have no experience with it, > but a quick Google search reveals > https://pypi.python.org/pypi/py-applescript and > https://docs.python.org/2/library/macosa.html py-appscript was a great tool but is no longer maintained by its developer because Apple deprecated much of the interfaces it was built on. It still might be useful in this case. At a lower level, the PyObjC project provides bridging between Python and Objective C and provides Python wrappers for many macOS system frameworks, include the CalendarStore framework. https://pypi.python.org/pypi/pyobjc From ganesh1pal at gmail.com Fri Mar 16 01:34:22 2018 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Fri, 16 Mar 2018 11:04:22 +0530 Subject: how do I retry a command only for a specific exception / error Message-ID: import time from functools import wraps from qa.utils.easy_popen import run def retry(ExceptionToCheck, tries=4, delay=5, backoff=2, logger=None): """ Retry calling the decorated function """ def deco_retry(f): @wraps(f) def f_retry(*args, **kwargs): mtries, mdelay = tries, delay while mtries > 1: try: return f(*args, **kwargs) except ExceptionToCheck, e: msg = "%s, Retrying in %d seconds..." % (str(e), mdelay) if logger: logger.warning(msg) else: print msg time.sleep(mdelay) mtries -= 1 mdelay *= backoff return f(*args, **kwargs) return f_retry # true decorator return deco_retry @retry(Exception, tries=4, delay=2) def test_network(text): try: cmd = "ifconfig -a" stdout, stderr, exit_code = run(cmd, timeout=300) print stdout, stderr, exit_code if exit_code != 0: raise RuntimeError("ERROR (exit_code %d): " "\nstdout: %s\nstderr: %s\n" % (exit_code, stdout, stderr)) except Exception as e: print str(e) raise Exception("Failed") print "Success: ", text test_network("it works!") All that I am trying to do here is write a generic function that will re-retry the command few more times before failing the test Case 1: + ve case ( say cmd= ifconfig ?a ) gpal-zkj2wrc-1# python file5.py vmx0: flags=1008843 metric 0 mtu 1500 options=403bb ether 00:50:56:90:ef:3d inet 1.224.39.1 netmask 0xffff0000 broadcast 1.224.255.255 zone 1 inet6 fe80::250:56ff:fe90:ef3d%vmx0 prefixlen 64 scopeid 0x1 zone 1 inet6 fdfe:9042:c53d:0:250:56ff:fe90:ef3d prefixlen 64 zone 1 nd6 options=21 media: Ethernet 10Gbase-T status: active vmx1: flags=8843 metric 0 mtu 1500 options=403bb ether 00:50:56:90:62:44 inet 10.224.39.1 netmask 0xfffffc00 broadcast 10.224.39.255 zone 1 nd6 options=29 media: Ethernet 10Gbase-T status: active Case 1 : Say I have invalid input command ( say ifconfig ?a ) it will work gpal-zkj2wrc-1# python file5.py ifconfig: illegal option -- z usage: ifconfig [-f type:format] [-L] [-C] [-g groupname] interface address_family [address [dest_address]] [parameters] ifconfig interface create ifconfig -a [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] [address_family] ifconfig -l [-d] [-u] [address_family] ifconfig [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] 1 ERROR (exit_code 1): stdout: stderr: ifconfig: illegal option -- z usage: ifconfig [-f type:format] [-L] [-C] [-g groupname] interface address_family [address [dest_address]] [parameters] ifconfig interface create ifconfig -a [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] [address_family] ifconfig -l [-d] [-u] [address_family] ifconfig [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] Failed, Retrying in 2 seconds... ifconfig: illegal option -- z usage: ifconfig [-f type:format] [-L] [-C] [-g groupname] interface address_family [address [dest_address]] [parameters] ifconfig interface create ifconfig -a [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] [address_family] ifconfig -l [-d] [-u] [address_family] ifconfig [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] 1 ERROR (exit_code 1): stdout: stderr: ifconfig: illegal option -- z usage: ifconfig [-f type:format] [-L] [-C] [-g groupname] interface address_family [address [dest_address]] [parameters] ifconfig interface create ifconfig -a [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] [address_family] ifconfig -l [-d] [-u] [address_family] ifconfig [-L] [-C] [-g groupname] [-d] [-m] [-u] [-v] Failed, Retrying in 4 seconds... Case 3: Assuming my command threw an exception say OSError , how do I retry a command only for a specific exception / error I am on Python 2.7 and Linux Regards, Ganesh Pal From steve+comp.lang.python at pearwood.info Fri Mar 16 01:51:34 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 16 Mar 2018 05:51:34 +0000 (UTC) Subject: how do I retry a command only for a specific exception / error References: Message-ID: On Fri, 16 Mar 2018 11:04:22 +0530, Ganesh Pal wrote: > All that I am trying to do here is write a generic function that will > re-retry > the command few more times before failing the test Something like this should do it. It gives up immediately on fatal errors and tries again on temporary ones. (You have to specify what you consider fatal or temporary, of course.) It uses exponential backup to wait longer and longer each time you fail, before eventually giving up. delay = 2 for attempts in range(max_attempts): try: command() except PermanentFailureError: raise except TemporaryFailureError: sleep(delay) delay *= 2 else: break else: raise RuntimeError("too many attempts") -- Steve From ian.g.kelly at gmail.com Fri Mar 16 11:55:36 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Fri, 16 Mar 2018 09:55:36 -0600 Subject: Make synchronous generator from an asynchronous generator In-Reply-To: <599ef329-95b4-10ea-98c1-1ca1669486e5@salort.eu> References: <599ef329-95b4-10ea-98c1-1ca1669486e5@salort.eu> Message-ID: On Thu, Mar 15, 2018 at 1:35 PM, Julien Salort wrote: > Because I wanted to keep the synchronous function for scripts which used it, without unnecessarily duplicating the code, I built also a synchronous function from this new asynchronous one, like that: > > def acquire_to_files(self, *args, **kwargs): > loop = asyncio.get_event_loop() > loop.run_until_complete(self.acquire_to_files_async(*args, **kwargs)) > loop.close() Note that this function can't be called more than once, because it closes the event loop at the end. Next time you call it it will get the closed event loop and try to schedule on it and then raise an exception because it's closed. > So far, it works fine. I am quite happy. > > I can await acquire_to_files_async in situations where I need concurrency, and call acquire_to_files in situations where I don't, and no code is duplicated. > > > Now I wish to do the same game with a generator (which happens to be called by the acquire_to_files_async function). So I made it into an asynchronous generator, and it works as expected. > > My problem: how do I proceed if I wish to build a synchronous generator from this asynchronous generator ? > > (for situations where concurrency is not needed) > > i.e. what is the equivalent of the above snippet for generators ? Well, it's not pretty, but this seems to work: async def agen(): yield 1 yield 2 yield 3 def gen(): loop = asyncio.get_event_loop() ait = agen().__aiter__() try: while True: yield loop.run_until_complete(ait.__anext__()) except StopAsyncIteration: return py> list(gen()) [1, 2, 3] It would take some additional code if you wanted to wire up gen().send() and gen().throw() to work as well, but it should be doable. From listes at salort.eu Fri Mar 16 13:43:42 2018 From: listes at salort.eu (Julien Salort) Date: Fri, 16 Mar 2018 18:43:42 +0100 Subject: Make synchronous generator from an asynchronous generator In-Reply-To: References: <599ef329-95b4-10ea-98c1-1ca1669486e5@salort.eu> Message-ID: <62b0dcc7-4365-c5f7-c951-5001f538fb0f@salort.eu> Le 16/03/2018 ? 16:55, Ian Kelly a ?crit?: > Note that this function can't be called more than once, because it > closes the event loop at the end. Next time you call it it will get > the closed event loop and try to schedule on it and then raise an > exception because it's closed. Ah ok. So, if I replace get_event_loop() with new_event_loop(), then it's fine ? > Well, it's not pretty, but this seems to work: [...] Thanks ! Julien From Irv at furrypants.com Fri Mar 16 18:59:10 2018 From: Irv at furrypants.com (Irv Kalb) Date: Fri, 16 Mar 2018 15:59:10 -0700 Subject: urllib.request.urlopen fails with https - SOLVED In-Reply-To: References: <1520910075l.12451962l.0l@psu.edu> Message-ID: <138D78B8-BDB1-45B5-9E37-0F9C5253EF10@furrypants.com> Thank you, thank you, thank you. That fixed it (at least on my computer, I'll see if I can do that at my school). Irv > On Mar 15, 2018, at 7:39 PM, Ned Deily wrote: > > On 2018-03-14 18:04, Irv Kalb wrote: >> File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/urllib/request.py", line 1320, in do_open >> raise URLError(err) >> urllib.error.URLError: > > If you are using Python 3.6 for macOS from a python.org installer, did > you follow the instructions displayed in the installer ReadMe and also > saved at: > > /Applications/Python 3.6/ReadMe.rtf > > to run the "Install Certificates.command" ? > > Either double-click on it in the Finder or, from a shell command line, type: > > open "/Applications/Python 3.6/Install Certificates.command" > > > Certificate verification and OpenSSL > > **NEW** This variant of Python 3.6 now includes its own private copy of > OpenSSL 1.0.2. Unlike previous releases, the deprecated Apple-supplied > OpenSSL libraries are no longer used. This also means that the trust > certificates in system and user keychains managed by the Keychain Access > application and the security command line utility are no longer used as > defaults by the Python ssl module. For 3.6.0, a sample command script > is included in /Applications/Python 3.6 to install a curated bundle of > default root certificates from the third-party certifi package > (https://pypi.python.org/pypi/certifi). If you choose to use certifi, > you should consider subscribing to the project's email update service to > be notified when the certificate bundle is updated. > > > -- > https://mail.python.org/mailman/listinfo/python-list > From steve+comp.lang.python at pearwood.info Fri Mar 16 19:16:48 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 16 Mar 2018 23:16:48 +0000 (UTC) Subject: Treatment of NANs in the statistics module Message-ID: The bug tracker currently has a discussion of a bug in the median(), median_low() and median_high() functions that they wrongly compute the medians in the face of NANs in the data: https://bugs.python.org/issue33084 I would like to ask people how they would prefer to handle this issue: (1) Put the responsibility on the caller to strip NANs from their data. If there is a NAN in your data, the result of calling median() is implementation-defined. This is the current behaviour, and is likely to be the fastest. (2) Return a NAN. (3) Raise an exception. (4) median() should strip out NANs. (5) All of the above, selected by the caller. (In which case, which would you prefer as the default?) Thank you. -- Steve From tjreedy at udel.edu Fri Mar 16 22:08:42 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 16 Mar 2018 22:08:42 -0400 Subject: Treatment of NANs in the statistics module In-Reply-To: References: Message-ID: On 3/16/2018 7:16 PM, Steven D'Aprano wrote: > The bug tracker currently has a discussion of a bug in the median(), > median_low() and median_high() functions that they wrongly compute the > medians in the face of NANs in the data: > > https://bugs.python.org/issue33084 > > I would like to ask people how they would prefer to handle this issue: > > (1) Put the responsibility on the caller to strip NANs from their data. 1 to 3 all put responsibility on the caller to strip NANs to get a sane answer. The question is what to do if the caller does not (1) > If there is a NAN in your data, the result of calling median() is > implementation-defined. This is the current behaviour, and is likely to > be the fastest. I hate implementation-defined behavior. > (2) Return a NAN. I don't like NANs as implemented and used, or unused. > (3) Raise an exception. That leave this. > (4) median() should strip out NANs. and then proceed in a deterministic fashion to give an answer. > (5) All of the above, selected by the caller. (In which case, which would > you prefer as the default?) I would frame this as an alternative: 'ignore_nan=False (3) or =True (4). Or nan='ignore' versus 'raise' (or 'strict') These are like the choices encoding. What do statistics.mean() and other functions do? The proposed quantile() will have the same issue. BMDP and other packages had and have general options for dealing with missing values, and that is what NAN is. -- Terry Jan Reedy From steve+comp.lang.python at pearwood.info Fri Mar 16 23:13:34 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 17 Mar 2018 03:13:34 +0000 (UTC) Subject: Treatment of NANs in the statistics module References: Message-ID: On Fri, 16 Mar 2018 22:08:42 -0400, Terry Reedy wrote: > On 3/16/2018 7:16 PM, Steven D'Aprano wrote: >> The bug tracker currently has a discussion of a bug in the median(), >> median_low() and median_high() functions that they wrongly compute the >> medians in the face of NANs in the data: [...] >> (4) median() should strip out NANs. > > and then proceed in a deterministic fashion to give an answer. Indeed. >> (5) All of the above, selected by the caller. (In which case, which >> would you prefer as the default?) > > I would frame this as an alternative: 'ignore_nan=False (3) or =True > (4). Or nan='ignore' versus 'raise' (or 'strict') These are like the > choices encoding. That's what I'm thinking. But which would you have as default? I'm guessing "raise". > What do statistics.mean() and other functions do? Because they do actual arithmetic on the data points, the presence of a float NAN will propagate through to the end of the calculation. A Decimal NAN will behave in three different ways: - a signalling NAN will raise when any operation is performed on it; - a quiet NAN will raise if the current Decimal context is set to raise on invalid operations, and propagate otherwise. It seems reasonable for median() to handle NANs better than it currently does, in which case I'd expect the rest of the statistics module to do the same. > The proposed quantile() will have the same issue. > > BMDP and other packages had and have general options for dealing with > missing values, and that is what NAN is. I don't wish to get into an argument about whether NANs are missing values or could be missing values, but R supports both NANs and a dedicated NA ("not available") missing value. By default, either will cause median to return NA, but there is an option to ignore NANs: > median(c(1, 2, 3, 4, NaN)) [1] NA > median(c(1, 2, 3, 4, NaN), na.rm=TRUE) [1] 2.5 -- Steve From formisc at gmail.com Sat Mar 17 01:36:07 2018 From: formisc at gmail.com (Andrew Z) Date: Sat, 17 Mar 2018 01:36:07 -0400 Subject: A better solution Message-ID: Hello, im not entirely happy with my solution and would love to hear your suggestions on how to improve the solution. I simplified the task while keeping the code working. Task: financial accounts are described by XML documents. I want to architecture the code to be easily extendible ( easy to add more accounts) and with minimum repetitive code in each class. All XML documents have a few common (SSN, first/last, sex, etc.) and unique fields. Both common and unique fields may or may not be present in the passed parameters. I came up with the following solution: one class as a common "account" with children that have specific functionality for their specific fields. What i _really_ don't like is the a repetitive code in each _init_ and "update" method. Code: -----start ---- from flask import Flask, jsonify, request import xml_oper app = Flask(__name__) @app.route('/') def index(): return "Hello, World!" @app.route('/INT/API/1.0/Accounts/IRA_Simple', methods =['POST']) def post_IRA_Simple(): json = request.get_json() account = xml_oper.account_type['IRA'] account.update_xml(output_file_name='IRA.xml',json=json) return jsonify( 'all good ), 201 if __name__ == '__main__': app.run(debug=True) xml_oper,py from lxml import etree from bs4 import BeautifulSoup as Soup class Account(): ''' Parent for all account types ''' _file_path_source = 'XXX/XML_Docs/Sample_ApplicationXML/' _file_path_dest = 'XXXXX/src/XML/' def __init__(self): pass def _open_xml(self, file_name): self._soup = Soup(open(Account._file_path_source + file_name, 'r'), 'lxml') def _write_xml(self, file_name): f = open(Account._file_path_dest + file_name, 'w') f.write(self._soup.prettify()) f.close() def _update(self, json): # update only common fields self._soup.application.customer['email'] = 'bla at email.com' class IRA(Account): def __init__(self, source_file_name): super().__init__() super()._open_xml(file_name=source_file_name) def update_xml(self, output_file_name, json): super()._update(json= json) # update specific to IRA account fields # values may or may not exit super()._write_xml(file_name=output_file_name) class Checking(Account): def __init__(self, source_file_name): super().__init__() super()._open_xml(file_name=source_file_name) def update_xml(self, output_file_name, json): super()._update(json= json) # update specific to CHK account fields # values may or may not exit super()._write_xml(file_name=output_file_name) account_type = { 'IRA' : IRA('IRA_Sample.xml'), 'Checking' : Checking('Checking_Sample.xml') } --- end---- Thank you. From ben+python at benfinney.id.au Sat Mar 17 01:41:01 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 17 Mar 2018 16:41:01 +1100 Subject: Treatment of NANs in the statistics module References: Message-ID: <85r2oj2r9u.fsf@benfinney.id.au> Steven D'Aprano writes: > I would like to ask people how they would prefer to handle [the > computation of median when the data set contains NaN]: > > (1) Put the responsibility on the caller to strip NANs from their > data. If there is a NAN in your data, the result of calling median() > is implementation-defined. This is the least Pythonic; there is no good reason IMO for specifying a behaviour in the implementation. > (2) Return a NAN. I think this makes the most sense. A result of ?NaN? communicates effectively that there is no meaningful number which is the median of the data set. It is then up to the caller to decide, based on that unambiguous result, what to do about it. > (3) Raise an exception. To raise an exception might be justifiable if there were no better option; but it creates the problem that a program which has not been tested for data containing NaN will catastrophically fail, instead of continuing. To raise an exception also forecloses the decision of whether ?NaN in the data set? is a problem. That is, IMO, a decision for the caller to make; it could well be that the caller is happy to use ?NaN? as the result. That should IMO be up to the caller to decide. > (4) median() should strip out NANs. Too much magic. This needlessly conflates different inputs as though they are the same. > (5) All of the above, selected by the caller. (In which case, which > would you prefer as the default?) Please, no. The function should have one well-defined behaviour, and all the options (raise an exception, strip the NaN from the input, etc.) can be done in a wrapper function by those who want it done. My vote, among those options, goes for ?when the input contains ?NaN?, return ?NaN??. -- \ ?But Marge, what if we chose the wrong religion? Each week we | `\ just make God madder and madder.? ?Homer, _The Simpsons_ | _o__) | Ben Finney From ben+python at benfinney.id.au Sat Mar 17 02:08:02 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 17 Mar 2018 17:08:02 +1100 Subject: Treatment of NANs in the statistics module References: <85r2oj2r9u.fsf@benfinney.id.au> Message-ID: <85bmfn2q0t.fsf@benfinney.id.au> Ben Finney writes: > Steven D'Aprano writes: > > > (1) [?] If there is a NAN in your data, the result of calling > > median() is implementation-defined. > > This is the least Pythonic; there is no good reason IMO for specifying a > behaviour in the implementation. That's a confused statement. I mean to say: there's no good reason IMO to leave the behaviour undefined in the function specification. -- \ ?I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.? ?Douglas Adams | Ben Finney From steve+comp.lang.python at pearwood.info Sat Mar 17 04:12:48 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 17 Mar 2018 08:12:48 +0000 (UTC) Subject: Treatment of NANs in the statistics module References: <85r2oj2r9u.fsf@benfinney.id.au> Message-ID: On Sat, 17 Mar 2018 16:41:01 +1100, Ben Finney wrote: >> (4) median() should strip out NANs. > > Too much magic. Statistically, ignoring NANs is equivalent to taking them as missing values. That is, for the purposes of calculating some statistic (let's say, median, although it applies to others as well), the sample data: 21, 37, 41, NAN, 65, 72 is equivalent to: 21, 37, 41, 65, 72 That's probably the most mathematically correct thing to do, *if* you interpret NANs as missing values. Thanks for your feedback. -- Steve From leo at superlel.me Sat Mar 17 05:52:11 2018 From: leo at superlel.me (=?UTF-8?Q?L=c3=a9o_El_Amri?=) Date: Sat, 17 Mar 2018 10:52:11 +0100 Subject: Treatment of NANs in the statistics module In-Reply-To: References: Message-ID: On 17/03/2018 00:16, Steven D'Aprano wrote: > The bug tracker currently has a discussion of a bug in the median(), > median_low() and median_high() functions that they wrongly compute the > medians in the face of NANs in the data: > > https://bugs.python.org/issue33084 > > I would like to ask people how they would prefer to handle this issue: TL;DR: I choose (5) I'm agree with Terry Reedy for his proposal for the (5), however, I want to define precisely what we mean with "ignore". In my opinion "ignoring" should be more like "stripping". In the case the number of data points is odd, we can return a NAN without any concerns. But in the case the number of data points is even, and at least one of the two middle values is a NAN, we're probably going to have an exception raised. In this case, to not over-complicate things, I think we should go with this meaning for "ignore": "Removing" NAN before actual data points processing. In this case, we should have two possible options for the keyword argument "nan": 'strip' (Which does what I just described) and 'raise' (Which raises an exception if there is a NAN in the data points). We should still consider adding an "ignore" option in a later time. This option would blindly ignore NAN values. If an exception is encountered during the actual processing (Let's say we have an even number of data points, and a NAN in one of the two values), it is raised up to the caller. >From my point of view, I prefer the (5). With a default of 'strip'. Your argument with (1) being the fastest (I believe, in terms of running-time, tell me if I'm wrong) can be achieved with the 'ignore' option. Going with (1) would force Python developers to write implementation specific code (Oh rather "implementation-defined-prone" code). In this case (5) goes easy with Python-side code. Options from (2) to (4) force Python developers to adopt a behavior. It's not necessarily a bad thing, but since (5) allows flexibility at no cost I don't see why we shouldn't go with it. -- L?o El Amri From michael.stemper at gmail.com Sat Mar 17 10:44:33 2018 From: michael.stemper at gmail.com (Michael F. Stemper) Date: Sat, 17 Mar 2018 09:44:33 -0500 Subject: number of loops In-Reply-To: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> References: <8c592561-db27-495e-962d-28ed741ec771@googlegroups.com> Message-ID: On 2018-03-15 08:28, michealmancini at gmail.com wrote: > I would like to have this offer 6 guesses, but instead it gives one guess and prints the if statement/output 6 times....any ideas where I went wrong? It sounds as if you'd like to have it making the offer within the loop rather than before the loop. Some other thoughts: > number_of_guesses = 0 Further on down, you convert this to a string. It might be simpler to just say: number_of_guesses = '0' This will eliminate the need to convert it to a string later on. Right here is where you offer a guess: > if yes_or_no == 'yes': > print ('ok! guess my number') > else: > print ('log off and go home') > > my_number=random.randint(1,100) Here is where you read the guess: > your_guess=input() > your_guess=int(your_guess) You could combine these two statements into: your_guess=int(input()) Here's the loop: > for number in range(6): > if your_guess > my_number: > print ('nope, too high') > if your_guess < my_number: > print ('nope, too low') > if your_guess == my_number: > break Notice how the loop doesn't include anything to offer or read a guess? Here's where you convert the integer 0 from the beginning of the program into a string. As I mentioned above, it'd be simpler to just initialize to '0' instead of to 0. > if your_guess == my_number: > number_of_guesses=str(number_of_guesses) > print ('good job, you guessed correctly') > else: > print('nope, you lose') Of course, since you never use this variable for anything, you could also drop it altogether. -- Michael F. Stemper Nostalgia just ain't what it used to be. From duncan at invalid.invalid Sat Mar 17 13:04:31 2018 From: duncan at invalid.invalid (duncan smith) Date: Sat, 17 Mar 2018 17:04:31 +0000 Subject: Treatment of NANs in the statistics module In-Reply-To: References: Message-ID: On 16/03/18 23:16, Steven D'Aprano wrote: > The bug tracker currently has a discussion of a bug in the median(), > median_low() and median_high() functions that they wrongly compute the > medians in the face of NANs in the data: > > https://bugs.python.org/issue33084 > > I would like to ask people how they would prefer to handle this issue: > > (1) Put the responsibility on the caller to strip NANs from their data. > If there is a NAN in your data, the result of calling median() is > implementation-defined. This is the current behaviour, and is likely to > be the fastest. > > (2) Return a NAN. > > (3) Raise an exception. > > (4) median() should strip out NANs. > > (5) All of the above, selected by the caller. (In which case, which would > you prefer as the default?) > > > Thank you. > > > > (2). A user can check for a returned NaN if necessary (so no real need for (3)). Silently stripping out NaNs strikes me as a terrible idea. The user should decide how NaNs should be dealt with. Optional arguments to govern the handling of NaNs - OK as long as the default behaviour is to return a NaN. There is no sensible default for handling NaNs (or missing values). Cheers. Duncan From vitaly.krug at gmail.com Sat Mar 17 22:39:15 2018 From: vitaly.krug at gmail.com (Vitaly Krug) Date: Sat, 17 Mar 2018 19:39:15 -0700 Subject: Is it possible to do something similar to gnome glib's GSource in asyncio? Message-ID: I am looking for a way to get a callback each time before asyncio event loop goes back to waiting on I/O (select, epoll, etc.). Documentation and googling for the answer hasn't helped yet. I am looking for something similar to gnome glib's GSource interface ( https://developer.gnome.org/glib/stable/glib-The-Main-Event-Loop.html#GSource ). Here is my use case: Sometimes it's necessary to wrap an asynchronous implementation in a synchronous API. This is how Pika AMQP client's BlockingConnection adapter is built on top of Pika's asynchronous SelectConnection adapter without incurring the overhead of multi-threading. The blocking wrapper makes calls into the asynchronous layer and runs the event loop (pika's proprietary select/epoll/kqueue event loop) while waiting for the asynchronous request to complete. For no-reply (i.e., no-ack) type of requests, the blocking wrapper simply waits for the write buffers to empty out before returning - for this, it checks the size of the write buffer each time before event loop goes back to waiting for I/O on select/epoll/kqueue. And if the blocking layer finds that the write buffer has emptied, it stops the event loop, so control can return to the user of the blocking interface. So, I need to be able to do something similar in asyncio to facilitate rebasing BlockingConnection on asyncio in Python3. Thank you in advance. Vitaly From rustompmody at gmail.com Sun Mar 18 09:55:39 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 18 Mar 2018 06:55:39 -0700 (PDT) Subject: Treatment of NANs in the statistics module In-Reply-To: References: Message-ID: <58f04983-60b0-43e7-8b3d-40eb12cc5423@googlegroups.com> On Saturday, March 17, 2018 at 3:22:46 PM UTC+5:30, L?o El Amri wrote: > On 17/03/2018 00:16, Steven D'Aprano wrote: > > The bug tracker currently has a discussion of a bug in the median(), > > median_low() and median_high() functions that they wrongly compute the > > medians in the face of NANs in the data: > > > > https://bugs.python.org/issue33084 > > > > I would like to ask people how they would prefer to handle this issue: > > TL;DR: I choose (5) Just like to point out that 5 is really 5a and 5b 5a One can give the option at the function call point 5b One can set a module level flag: See how pandas sets 'pandas.options. ...' for similar choices https://pandas.pydata.org/pandas-docs/stable/missing_data.html I guess I'd go for 5b even though it makes the code less 'functional' in the sense of FP ? ie the same (looking) function call can have different effects From vitaly.krug at gmail.com Sun Mar 18 12:26:23 2018 From: vitaly.krug at gmail.com (Vitaly Krug) Date: Sun, 18 Mar 2018 09:26:23 -0700 Subject: Is it possible to do something similar to gnome glib's GSource in asyncio? In-Reply-To: References: Message-ID: Mea culpa! It sounds like I should be able to create a specialized future to test for my loop exit condition and then use that future with asyncio loop's `run_until_complete()` method. On Sat, Mar 17, 2018 at 7:39 PM, Vitaly Krug wrote: > I am looking for a way to get a callback each time before asyncio event > loop goes back to waiting on I/O (select, epoll, etc.). Documentation and > googling for the answer hasn't helped yet. I am looking for something > similar to gnome glib's GSource interface (https://developer.gnome.org/ > glib/stable/glib-The-Main-Event-Loop.html#GSource). > > Here is my use case: > > Sometimes it's necessary to wrap an asynchronous implementation in a > synchronous API. This is how Pika AMQP client's BlockingConnection adapter > is built on top of Pika's asynchronous SelectConnection adapter without > incurring the overhead of multi-threading. > > The blocking wrapper makes calls into the asynchronous layer and runs the > event loop (pika's proprietary select/epoll/kqueue event loop) while > waiting for the asynchronous request to complete. > > For no-reply (i.e., no-ack) type of requests, the blocking wrapper simply > waits for the write buffers to empty out before returning - for this, it > checks the size of the write buffer each time before event loop goes back > to waiting for I/O on select/epoll/kqueue. And if the blocking layer finds > that the write buffer has emptied, it stops the event loop, so control can > return to the user of the blocking interface. > > So, I need to be able to do something similar in asyncio to facilitate > rebasing BlockingConnection on asyncio in Python3. > > Thank you in advance. > > Vitaly > > > > From fabianosidler at swissonline.ch Sun Mar 18 15:01:01 2018 From: fabianosidler at swissonline.ch (Fabiano Sidler) Date: Sun, 18 Mar 2018 20:01:01 +0100 Subject: TLSServer: certificate one request behind... In-Reply-To: <20180314201257.GA55874@mailboxes.bgd.fipscode.net> References: <20180313213716.GA50383@mailboxes.bgd.fipscode.net> <20180314201257.GA55874@mailboxes.bgd.fipscode.net> Message-ID: <20180318190100.GA78655@mailboxes.bgd.fipscode.net> Hello? Rfd, anyone? Thus wrote Fabiano Sidler: > Thus wrote Fabiano Sidler: > > What's the reason for this? Please find attached my TLSServer. > > Oh, sorry...! Apparently, the attachment has been stripped. Here inline: > > === tlsserver.py === > from socketserver import ThreadingTCPServer,StreamRequestHandler > import ssl > > class TLSServer(ThreadingTCPServer): > def __init__(self, *args, **kwargs): > super(TLSServer, self).__init__(*args, **kwargs) > ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) > ctx.set_servername_callback(self.servername_callback) > ctx.check_hostname = False > self._ctx = ctx > def get_request(self): > s,a = super(TLSServer, self).get_request() > s = self._ctx.wrap_socket(s, server_side=True) > return s,a > def servername_callback(self, sock, req_hostname, cb_context): > return ssl.ALERT_DESCRIPTION_INTERNAL_ERROR > > > from OpenSSL import crypto as x509 > from tempfile import NamedTemporaryFile > > class SelfSigningServer(TLSServer): > def servername_callback(self, sock, req_hostname, cb_context): > key = x509.PKey() > key.generate_key(x509.TYPE_RSA, 2048) > cert = x509.X509() > subj = cert.get_subject() > subj.C = 'CH' > subj.ST = 'ZH' > subj.L = 'Zurich' > subj.O = 'ACME Inc.' > subj.OU = 'IT dept.' > subj.CN = req_hostname > cert.set_version(0x02) > cert.set_serial_number(1000) > cert.gmtime_adj_notBefore(0) > cert.gmtime_adj_notAfter(10*365*24*60*60) > cert.set_issuer(subj) > cert.set_pubkey(key) > cert.sign(key, 'sha256') > certfile = NamedTemporaryFile() > keyfile = NamedTemporaryFile() > certfile.write(x509.dump_certificate(x509.FILETYPE_PEM, cert)) > keyfile.write(x509.dump_privatekey(x509.FILETYPE_PEM, key)) > certfile.seek(0) > keyfile.seek(0) > cb_context.load_cert_chain(certfile=certfile.name, keyfile=keyfile.name) > cb_context.set_servername_callback(self.servername_callback) > sock.context = cb_context > certfile.close() > keyfile.close() > > class SelfSigningHandler(StreamRequestHandler): > def handle(self): > self.wfile.write(b'Hello World!\r\n') > > server = SelfSigningServer(('localhost',1234), SelfSigningHandler) > server.serve_forever() > === tlsserver.py === > > Thanks again! > -- > https://mail.python.org/mailman/listinfo/python-list From formisc at gmail.com Sun Mar 18 23:32:02 2018 From: formisc at gmail.com (Andrew Z) Date: Sun, 18 Mar 2018 23:32:02 -0400 Subject: Keys in dict and keys not in dict Message-ID: hello, i'd like to check if a function parameter (json) has all the keys I expect it to have and if it doesn't - point out the one that is missing. What's the good way of doing that? "good way" - something concise... i'd like to avoid using : if key in json: #pass else print(" Oops, i did it again ...") thank you AZ From rosuav at gmail.com Sun Mar 18 23:41:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Mar 2018 14:41:05 +1100 Subject: Keys in dict and keys not in dict In-Reply-To: References: Message-ID: On Mon, Mar 19, 2018 at 2:32 PM, Andrew Z wrote: > hello, > > i'd like to check if a function parameter (json) has all the keys I expect > it to have and if it doesn't - point out the one that is missing. > > What's the good way of doing that? > > "good way" - something concise... i'd like to avoid using : > if key in json: > #pass > else > print(" Oops, i did it again ...") Sounds like a set operation to me. expected = {"foo", "bar", "spam"} missing = expected - set(json) if missing: print("Missing keys:", missing) ChrisA From ben+python at benfinney.id.au Mon Mar 19 00:18:52 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 19 Mar 2018 15:18:52 +1100 Subject: Keys in dict and keys not in dict References: Message-ID: <85y3io1yvn.fsf@benfinney.id.au> Chris Angelico writes: > Sounds like a set operation to me. > > expected = {"foo", "bar", "spam"} > missing = expected - set(json) That works (because iterating a dict returns its keys). But it is less immediately understandable, IMO, than this:: expected = {"foo", "bar", "spam"} missing = expected - set(json.keys()) -- \ ?The greater the artist, the greater the doubt; perfect | `\ confidence is granted to the less talented as a consolation | _o__) prize.? ?Robert Hughes | Ben Finney From rosuav at gmail.com Mon Mar 19 00:25:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 19 Mar 2018 15:25:48 +1100 Subject: Keys in dict and keys not in dict In-Reply-To: <85y3io1yvn.fsf@benfinney.id.au> References: <85y3io1yvn.fsf@benfinney.id.au> Message-ID: On Mon, Mar 19, 2018 at 3:18 PM, Ben Finney wrote: > Chris Angelico writes: > >> Sounds like a set operation to me. >> >> expected = {"foo", "bar", "spam"} >> missing = expected - set(json) > > That works (because iterating a dict returns its keys). But it is less > immediately understandable, IMO, than this:: > > expected = {"foo", "bar", "spam"} > missing = expected - set(json.keys()) > Sure, whichever way you want to do it. Either way, it's a set difference operation that gives the OP's desired information. ChrisA From songofacandy at gmail.com Mon Mar 19 03:09:01 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 19 Mar 2018 16:09:01 +0900 Subject: Keys in dict and keys not in dict In-Reply-To: <85y3io1yvn.fsf@benfinney.id.au> References: <85y3io1yvn.fsf@benfinney.id.au> Message-ID: > expected = {"foo", "bar", "spam"} > missing = expected - set(json.keys()) > dict.keys() returns set-like object. So `missing = expected - json.keys()` works fine, and it's more efficient. -- INADA Naoki From __peter__ at web.de Mon Mar 19 03:09:39 2018 From: __peter__ at web.de (Peter Otten) Date: Mon, 19 Mar 2018 08:09:39 +0100 Subject: Keys in dict and keys not in dict References: <85y3io1yvn.fsf@benfinney.id.au> Message-ID: Ben Finney wrote: > Chris Angelico writes: > >> Sounds like a set operation to me. >> >> expected = {"foo", "bar", "spam"} >> missing = expected - set(json) > > That works (because iterating a dict returns its keys). But it is less > immediately understandable, IMO, than this:: > > expected = {"foo", "bar", "spam"} > missing = expected - set(json.keys()) There's no need to materialize the set of keys: >>> expected = {"foo", "bar", "ham"} >>> json = dict(foo=1, bar=2, spam=3) >>> expected - json.keys() {'ham'} In Python 2 use json.viewkeys() instead of keys(). From formisc at gmail.com Mon Mar 19 09:17:47 2018 From: formisc at gmail.com (Andrew Z) Date: Mon, 19 Mar 2018 09:17:47 -0400 Subject: Keys in dict and keys not in dict In-Reply-To: References: <85y3io1yvn.fsf@benfinney.id.au> Message-ID: Beautiful. Thank you Chris, Ben, Peter and Inada. On Mar 19, 2018 3:14 AM, "INADA Naoki" wrote: > > expected = {"foo", "bar", "spam"} > > missing = expected - set(json.keys()) > > > > dict.keys() returns set-like object. > So `missing = expected - json.keys()` works fine, and it's more efficient. > > -- > INADA Naoki > -- > https://mail.python.org/mailman/listinfo/python-list > From tkadm30 at yandex.com Mon Mar 19 12:08:39 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Mon, 19 Mar 2018 12:08:39 -0400 Subject: Thank you Python community! Message-ID: You guys just made me realize something very obvious. :-) I'm in the process right now of watching the excellent documentary named "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned about the major opioid epidemic in the US. I would like to thank you guys sincerely for helping a lot of people to stay clean, and focus on programming high-level stuff in Python instead of doing some really nasty drugs. I'm also wondering, could we exploit this strategy even further to help people willing to stop doing drugs by teaching them some stuff in Python? Murdering random drug dealers won't help out, i'm afraid. Neither do putting some of them in jail is sufficient to prevent people from trying out cocaine and heroin it seems. So, I would like to take this opportunity to advocate for this precise form of rehabilitation using open source software programming as a vector to make people use their brains in positive ways. The trick I think is to let people use programming as a way to socialize more in order to stimulate or distract their minds whiling keeping them away from drugs. It's time we regroup and combine our forces to help people who really needs our help. Sincerely, Etienne -- Etienne Robillard tkadm30 at yandex.com https://www.isotopesoftware.ca/ From ganesh1pal at gmail.com Mon Mar 19 12:38:09 2018 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Mon, 19 Mar 2018 22:08:09 +0530 Subject: how do I retry a command only for a specific exception / error In-Reply-To: References: Message-ID: On Fri, Mar 16, 2018 at 11:21 AM, Steven D'Aprano < steve+comp.lang.python at pearwood.info> wrote: > On Fri, 16 Mar 2018 11:04:22 +0530, Ganesh Pal wrote: > > > All that I am trying to do here is write a generic function that will > > re-retry > > the command few more times before failing the test > > > Something like this should do it. It gives up immediately on fatal errors > and tries again on temporary ones. (You have to specify what you consider > fatal or temporary, of course.) It uses exponential backup to wait longer > and longer each time you fail, before eventually giving up. > > > This is a good suggestion , I like the way this is written , but what I > have failed to figure out is how to translate the possible TemporaryFailureErrors > to a different exception class/type and retry . > > In my case , every command is executed using a run() function that > calls out to subprocess.Popen(). Which will return stdout, stderr, > exit_code and we would need to retry only for a specific > TemporaryFailureError . > > > > Example : Say , If we are not able to SSH to the host , and I get > ?connection refused? error I would want to retry only for this specific case > > > > # Sample modified code > > > > delay = 2 > > max_attempts =4 > for attempts in range(max_attempts): > try: > cmd = "ssh root at localhost.com" > > stdout, stderr, exit_code = run(cmd, timeout=300) > > print stdout, stderr, exit_code > > if exit_code != 0: > > raise RuntimeError("ERROR (exit_code %d): " > > "\nstdout: %s\nstderr: %s\n" % (exit_code, > stdout, stderr)) > > except Exeception as e : > raise > > # if we have ?connection refused? error then retry after some time > except TemporaryFailureError: > sleep(delay) > delay *= 2 > else: > break > else: > raise RuntimeError("too many attempts") > > > > Regards, Ganesh From Irv at furrypants.com Mon Mar 19 13:04:53 2018 From: Irv at furrypants.com (Irv Kalb) Date: Mon, 19 Mar 2018 10:04:53 -0700 Subject: Style Q: Instance variables defined outside of __init__ Message-ID: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> I am building some classes for use in future curriculum. I am using PyCharm for my development. On the right hand edge of the PyCharm editor window, you get some little bars indicating warnings or errors in your code. I like this feature and try to clean up as many of those as I can. However, there is one warning that I am seeing often, and I'm not sure about how to handle it. The warning I see is: "Instance attribute defined outside of __init__ ... " The following is a simple example. I am creating a card class that I am using to build card games with PyGame. class Card(): BACK_OF_CARD_IMAGE = pygame.image.load('images/backOfCard.png') def __init__(self, window, name, suit, value): self.window = window self.suit = suit self.cardName = name + ' of ' + suit self.value = value fileName = 'images/' + self.cardName + '.png' self.image = pygame.image.load(fileName) self.backOfCardImage = Card.BACK_OF_CARD_IMAGE self.conceal() def conceal(self): self.faceUp = False def reveal(self): self.faceUp = True In this class, I get warnings on the single lines of the conceal and reveal methods. Yes, the warnings are correct, the variable "self.faceUp" is not defined inside the __init__ method. My __init__ method calls self.conceal which is a more logical place to initialize this variable. My question is, what is the Pythonic way to handle this? In the research that I have done, I see split results. Some people say that this type of thing is fine and these warnings should just be ignored. While others say that all instance variables should be defined in the __init__ method. I like that idea (and have done so in other languages), but if I define this variable there, what value should I give it? Do I redundantly set it to the proper starting value (in case False), or do as others have suggested, set it to None (which seems a little odd for something that I will use as a Boolean). I have many more similar cases. For example, in many small game programs, at the end of my __init__ method, I call a "reset" method in the same class which initializes a bunch of instance variables for playing a game. When the game is over, if the user wants to play again, I call the same reset method. Very clean, and works very well, but all the instance variables defined in that reset method gets the same warning messages. Thanks, Irv From ethan at stoneleaf.us Mon Mar 19 13:25:10 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 19 Mar 2018 10:25:10 -0700 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: <5AAFF276.7060508@stoneleaf.us> On 03/19/2018 10:04 AM, Irv Kalb wrote: > I am building some classes for use in future curriculum. I am using PyCharm for my development. On the right hand edge of the PyCharm editor window, you get some little bars indicating warnings or errors in your code. I like this feature and try to clean up as many of those as I can. However, there is one warning that I am seeing often, and I'm not sure about how to handle it. The warning I see is: > > "Instance attribute defined outside of __init__ ..." > > The following is a simple example. I am creating a card class that I am using to build card games with PyGame. > class Card(): > > BACK_OF_CARD_IMAGE = pygame.image.load('images/backOfCard.png') > > def __init__(self, window, name, suit, value): > self.window = window > self.suit = suit > self.cardName = name + ' of ' + suit > self.value = value > fileName = 'images/' + self.cardName + '.png' > self.image = pygame.image.load(fileName) > self.backOfCardImage = Card.BACK_OF_CARD_IMAGE > > self.conceal() > > def conceal(self): > self.faceUp = False > > def reveal(self): > self.faceUp = True > > > In this class, I get warnings on the single lines of the conceal and reveal methods. Yes, the warnings are correct, the variable "self.faceUp" is not defined inside the __init__ method. My __init__ method calls self.conceal which is a more logical place to initialize this variable. > My question is, what is the Pythonic way to handle this? In the research that I have done, I see split results. Some people say that this type of thing is fine and these warnings should just be ignored. While others say that all instance variables should be defined in the __init__ method. I like that idea (and have done so in other languages), but if I define this variable there, what value should I give it? Do I redundantly set it to the proper starting value (in case False), or do as others have suggested, set it to None (which seems a little odd for something that I will use as a Boolean). > > I have many more similar cases. For example, in many small game programs, at the end of my __init__ method, I call a "reset" method in the same class which initializes a bunch of instance variables for playing a game. When the game is over, if the user wants to play again, I call the same reset method. Very clean, and works very well, but all the instance variables defined in that reset method gets the same warning messages. Define them in __init__; otherwise, you get an error if reveal() is called before conceal() is. Also, a card is either revealed or concealed, it can't be both and it can't be neither -- so set it in __init__. As a side note: I find card_name easier to read than cardName. :) -- ~Ethan~ From ned at nedbatchelder.com Mon Mar 19 13:27:52 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 19 Mar 2018 13:27:52 -0400 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: On 3/19/18 1:04 PM, Irv Kalb wrote: > I am building some classes for use in future curriculum. I am using PyCharm for my development. On the right hand edge of the PyCharm editor window, you get some little bars indicating warnings or errors in your code. I like this feature and try to clean up as many of those as I can. However, there is one warning that I am seeing often, and I'm not sure about how to handle it. The warning I see is: > > "Instance attribute defined outside of __init__ ..." > > The following is a simple example. I am creating a card class that I am using to build card games with PyGame. > class Card(): > > BACK_OF_CARD_IMAGE = pygame.image.load('images/backOfCard.png') > > def __init__(self, window, name, suit, value): > self.window = window > self.suit = suit > self.cardName = name + ' of ' + suit > self.value = value > fileName = 'images/' + self.cardName + '.png' > self.image = pygame.image.load(fileName) > self.backOfCardImage = Card.BACK_OF_CARD_IMAGE > > self.conceal() > > def conceal(self): > self.faceUp = False > > def reveal(self): > self.faceUp = True > > > In this class, I get warnings on the single lines of the conceal and reveal methods. Yes, the warnings are correct, the variable "self.faceUp" is not defined inside the __init__ method. My __init__ method calls self.conceal which is a more logical place to initialize this variable. > My question is, what is the Pythonic way to handle this? In the research that I have done, I see split results. Some people say that this type of thing is fine and these warnings should just be ignored. While others say that all instance variables should be defined in the __init__ method. I like that idea (and have done so in other languages), but if I define this variable there, what value should I give it? Do I redundantly set it to the proper starting value (in case False), or do as others have suggested, set it to None (which seems a little odd for something that I will use as a Boolean). I understand this tension: it's nice to assign only meaningful values, and to do it in as few places as possible.? But it's also nice to have all of your attributes in one place.? This is a by-product of Python having no declarations, only definitions (assignments).? So to mention the attributes initially, you have to choose a value for them.? If I were to add faceUp to __init__, I would assign False to it. > > I have many more similar cases. For example, in many small game programs, at the end of my __init__ method, I call a "reset" method in the same class which initializes a bunch of instance variables for playing a game. When the game is over, if the user wants to play again, I call the same reset method. Very clean, and works very well, but all the instance variables defined in that reset method gets the same warning messages. You didn't ask about this, so perhaps you already know it, but you can disable this particular Pylint warning, either line-by-line, file-by-file, or for the entire project. --Ned. From rosuav at gmail.com Mon Mar 19 13:34:16 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Mar 2018 04:34:16 +1100 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <5AAFF276.7060508@stoneleaf.us> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <5AAFF276.7060508@stoneleaf.us> Message-ID: On Tue, Mar 20, 2018 at 4:25 AM, Ethan Furman wrote: > Define them in __init__; otherwise, you get an error if reveal() is called > before conceal() is. Also, a card is either revealed or concealed, it can't > be both and it can't be neither -- so set it in __init__. __init__ calls conceal. ChrisA From bc at freeuk.com Mon Mar 19 15:05:18 2018 From: bc at freeuk.com (bartc) Date: Mon, 19 Mar 2018 19:05:18 +0000 Subject: Thank you Python community! In-Reply-To: References: Message-ID: On 19/03/2018 16:08, Etienne Robillard wrote: > You guys just made me realize something very obvious. :-) > > I'm in the process right now of watching the excellent documentary named > "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned > about the major opioid epidemic in the US. > > I would like to thank you guys sincerely for helping a lot of people to > stay clean, and focus on programming high-level stuff in Python instead > of doing some really nasty drugs. > > I'm also wondering, could we exploit this strategy even further to help > people willing to stop doing drugs by teaching them some stuff in Python? You either code in Python, or you're forced to take drugs? Got it. > The trick I think is to let people use programming as a way to socialize > more in order to stimulate or distract their minds whiling keeping them > away from drugs. I've often wondered what the guys who invented C (around 1970) must have been smoking to have come up with some of those ideas. -- bartc From larry.martell at gmail.com Mon Mar 19 15:21:26 2018 From: larry.martell at gmail.com (Larry Martell) Date: Mon, 19 Mar 2018 15:21:26 -0400 Subject: Thank you Python community! In-Reply-To: References: Message-ID: On Mon, Mar 19, 2018 at 12:08 PM, Etienne Robillard wrote: > You guys just made me realize something very obvious. :-) > > I'm in the process right now of watching the excellent documentary named > "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned about > the major opioid epidemic in the US. Have no clue what this has to do with python, but the opioid epidemic was created by collision between big pharma and the government. From adrian.ordona at gmail.com Mon Mar 19 15:33:50 2018 From: adrian.ordona at gmail.com (Adrian Ordona) Date: Mon, 19 Mar 2018 12:33:50 -0700 Subject: Missing python36.dll Message-ID: <4A3250CC-669A-4714-A986-C340E2D8A11C@gmail.com> Hi - just downloaded the latest version of Python (3.6.4) but keep getting an error because of missing .dll file. i tried uninstalling and reinstalling it but still getting the same error. Search online for the file but to no avail. Can someone direct me as to where to download the missing dll file or help me with the issue. thanks, Adrian From tkadm30 at yandex.com Mon Mar 19 15:37:42 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Mon, 19 Mar 2018 15:37:42 -0400 Subject: Thank you Python community! In-Reply-To: References: Message-ID: <4336457a-2a07-018f-93e2-aceebbb3d8c2@yandex.com> Le 2018-03-19 ? 15:21, Larry Martell a ?crit?: > On Mon, Mar 19, 2018 at 12:08 PM, Etienne Robillard wrote: >> You guys just made me realize something very obvious. :-) >> >> I'm in the process right now of watching the excellent documentary named >> "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned about >> the major opioid epidemic in the US. > Have no clue what this has to do with python, but the opioid epidemic > was created by collision between big pharma and the government. Quite simply, coding stuff in Python (or any other high-level programming language) is probably a very effective (and underestimated) way to help people in need finding a real goal in their life and the feeling of accomplishment. Etienne -- Etienne Robillard tkadm30 at yandex.com https://www.isotopesoftware.ca/ From info at tundraware.com Mon Mar 19 15:40:22 2018 From: info at tundraware.com (Tim Daneliuk) Date: Mon, 19 Mar 2018 14:40:22 -0500 Subject: Thank you Python community! In-Reply-To: References: Message-ID: <6ag5oe-s602.ln1@oceanview.tundraware.com> On 03/19/2018 02:05 PM, bartc wrote: > I've often wondered what the guys who invented C (around 1970) must have been?smoking?to?have?come?up?with?some?of?those?ideas. I dunno, but I do know that - if they were smoking something - it was rolled in greenbar paper ... From rgaddi at highlandtechnology.invalid Mon Mar 19 16:18:40 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 19 Mar 2018 13:18:40 -0700 Subject: Thank you Python community! In-Reply-To: <6ag5oe-s602.ln1@oceanview.tundraware.com> References: <6ag5oe-s602.ln1@oceanview.tundraware.com> Message-ID: On 03/19/2018 12:40 PM, Tim Daneliuk wrote: > On 03/19/2018 02:05 PM, bartc wrote: >> I've often wondered what the guys who invented C (around 1970) must have been?smoking?to?have?come?up?with?some?of?those?ideas. > > I dunno, but I do know that - if they were smoking something - it was > rolled in greenbar paper ... > "Look, you can make a filter out of the pinfeed!" -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From python at mrabarnett.plus.com Mon Mar 19 17:07:02 2018 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 19 Mar 2018 21:07:02 +0000 Subject: Missing python36.dll In-Reply-To: <4A3250CC-669A-4714-A986-C340E2D8A11C@gmail.com> References: <4A3250CC-669A-4714-A986-C340E2D8A11C@gmail.com> Message-ID: <0ea93e82-7fab-a4cf-7cc1-52145738fd8d@mrabarnett.plus.com> On 2018-03-19 19:33, Adrian Ordona wrote: > Hi - just downloaded the latest version of Python (3.6.4) but keep getting an error > because of missing .dll file. i tried uninstalling and reinstalling it but still getting the same error. Search online for the file but to no avail. Can someone direct me as to where to download the missing dll file or help me with the issue. > You didn't say which installer you used. I've just updated from Python 3.6.3 to Python 3.6.4 using the "executable installer" (https://www.python.org/downloads/release/python-364/). I didn't have a problem; the DLL was also updated (it's in the Python folder next to "python.exe"). From malaclypse2 at gmail.com Mon Mar 19 17:19:06 2018 From: malaclypse2 at gmail.com (Jerry Hill) Date: Mon, 19 Mar 2018 17:19:06 -0400 Subject: Missing python36.dll In-Reply-To: <0ea93e82-7fab-a4cf-7cc1-52145738fd8d@mrabarnett.plus.com> References: <4A3250CC-669A-4714-A986-C340E2D8A11C@gmail.com> <0ea93e82-7fab-a4cf-7cc1-52145738fd8d@mrabarnett.plus.com> Message-ID: On Mon, Mar 19, 2018 at 5:07 PM, MRAB wrote: > You didn't say which installer you used. > ?It might also be helpful to know: Did you install python for "Just Me" or "All Users" in the installer? Does the user you're logged in as have enough authority to install for All Users if that's what you picked? What OS ?are you using? Since you're asking about DLLs, it's obviously some flavor of Windows, but you don't say which one. Did you notice any errors messages or log files generated by the installer? From ben+python at benfinney.id.au Mon Mar 19 17:58:40 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 20 Mar 2018 08:58:40 +1100 Subject: Thank you Python community! References: Message-ID: <85r2of20dr.fsf@benfinney.id.au> Etienne Robillard writes: > I would like to thank you guys sincerely for helping a lot of people > to stay clean, and focus on programming high-level stuff in Python > instead of doing some really nasty drugs. Thank you for the kind words. I'd love to believe the Python community has the significant benefit you describe. That just raises the importance of being skeptical (because I *want* it to be true, means more rigour is needed to avoid the bias to believe it; read about ?confirmation bias? to see the problem). So, in the interest of having our beliefs match the facts: What is the verifiable, statistically sound evidence that this effect actually occurs as a result of the Python community's help with programming? Where is it pbulished so we can see it? What is the size of the measured effect? > It's time we regroup and combine our forces to help people who really > needs our help. Combining forces to help the needy is a sentiment I can fully endorse, for sure. -- \ "Success is going from one failure to the next without a loss | `\ of enthusiasm." -- Winston Churchill | _o__) | Ben Finney From ben+python at benfinney.id.au Mon Mar 19 18:04:27 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 20 Mar 2018 09:04:27 +1100 Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: <85muz32044.fsf@benfinney.id.au> Irv Kalb writes: > In this class, I get warnings on the single lines of the conceal and > reveal methods. This is good! It prompts the question: Why are those methods defined as they are? If those methods are only ever intended to set the value of an attribute: Why not just set that attribute directly? (Remember that Python properties allow you to postpone the decision later of whether that attribute is a property.) If those methods are intended to later expand to have side effects: Why are you calling those methods from the initialiser? The initialiser should instead just set the initial state of the attributes, avoiding whatever side-effects are in ?conceal? etc. So, it seems that the warning is correct. The initialiser's job is to initialise the instance, it should not accidentally invoke a bunch of side effects. This means the initialiser should tend to just setting attribute values, not calling a bunch of other methods. -- \ ?It is undesirable to believe a proposition when there is no | `\ ground whatever for supposing it true.? ?Bertrand Russell, _The | _o__) Value of Scepticism_, 1928 | Ben Finney From tkadm30 at yandex.com Mon Mar 19 19:23:05 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Mon, 19 Mar 2018 19:23:05 -0400 Subject: Thank you Python community! In-Reply-To: <85r2of20dr.fsf@benfinney.id.au> References: <85r2of20dr.fsf@benfinney.id.au> Message-ID: <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> Hi Ben, Thank you for your reply. I would like to make such an experimental research/investigation on the effects of Python software programming on opioid addiction. :-) Probably studying the learning of Python in people with cocaine and heroin addiction would be significant and interesting. Kind regards, Etienne Le 2018-03-19 ? 17:58, Ben Finney a ?crit?: > Etienne Robillard writes: > >> I would like to thank you guys sincerely for helping a lot of people >> to stay clean, and focus on programming high-level stuff in Python >> instead of doing some really nasty drugs. > Thank you for the kind words. > > I'd love to believe the Python community has the significant benefit you > describe. That just raises the importance of being skeptical (because I > *want* it to be true, means more rigour is needed to avoid the bias to > believe it; read about ?confirmation bias? to see the problem). > > So, in the interest of having our beliefs match the facts: What is the > verifiable, statistically sound evidence that this effect actually > occurs as a result of the Python community's help with programming? > Where is it pbulished so we can see it? What is the size of the measured > effect? > >> It's time we regroup and combine our forces to help people who really >> needs our help. > Combining forces to help the needy is a sentiment I can fully endorse, > for sure. > -- Etienne Robillard tkadm30 at yandex.com https://www.isotoperesearch.ca/ From steve+comp.lang.python at pearwood.info Mon Mar 19 19:36:31 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 19 Mar 2018 23:36:31 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: On Mon, 19 Mar 2018 10:04:53 -0700, Irv Kalb wrote: > Some people say > that this type of thing is fine and these warnings should just be > ignored. While others say that all instance variables should be defined > in the __init__ method. Whenever anyone says "you should do this", the question to ask is, "what bad thing will happen if I don't?" You are getting a warning that self.faceUp is not defined in __init__. That means that there must be something bad that will happen if you don't define faceUp in __init__. Any idea what? If there is no such bad thing that will happen, then PyCharm is crying wolf: it is wasting your time with a warning for a non-problem. > I like that idea (and have done so in other > languages), but if I define this variable there, what value should I > give it? Do I redundantly set it to the proper starting value (in case > False), Why is it redundant? Since you're going to call conceal() anyway, why not just set the value and not bother with conceal? Especially since conceal appears to do nothing but set the flag. Maybe it is *conceal* which is redundant and needs to be removed. On the other hand, instance.conceal() instance.reveal() is much more descriptive than "faceUp = True". > or do as others have suggested, set it to None (which seems a > little odd for something that I will use as a Boolean). Ah yes, the good old "rubbish advice from the internet" strike again. Unless your faceUp is intended to represent a *three state flag* (say, face up, face down, and standing on its side...) then why distinguish between None and False? What possible benefit to this is it? My guess is that they're thinking that setting it to None indicates that it is uninitialised. Except that None is a perfectly good falsey value, so unless you specifically and carefully change *all* your code that currently says: if self.faceUp: ... into: if self.faceUp is None: raise SomeError elif self.faceUp: ... then it makes no difference at all. Just use False and be done with it. > I have many more similar cases. For example, in many small game > programs, at the end of my __init__ method, I call a "reset" method in > the same class which initializes a bunch of instance variables for > playing a game. When the game is over, if the user wants to play again, > I call the same reset method. Very clean, and works very well, but all > the instance variables defined in that reset method gets the same > warning messages. Indeed. Linters and code checkers often waste your time warning you against things that aren't wrong. -- Steve From ben+python at benfinney.id.au Mon Mar 19 19:36:53 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 20 Mar 2018 10:36:53 +1100 Subject: Thank you Python community! References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> Message-ID: <858tan1vu2.fsf@benfinney.id.au> Etienne Robillard writes: > I would like to make such an experimental research/investigation on > the effects of Python software programming on opioid addiction. :-) Okay. The wording of your message implied that you know this already happens now, though. How did you come to know this? -- \ ?Experience is that marvelous thing that enables you to | `\ recognize a mistake when you make it again.? ?Franklin P. Jones | _o__) | Ben Finney From jfong at ms4.hinet.net Mon Mar 19 19:46:36 2018 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Mon, 19 Mar 2018 16:46:36 -0700 (PDT) Subject: {:2X} didn't output what I expected Message-ID: <1f899b8f-534d-43fb-a95a-acd9fee796a2@googlegroups.com> D:\Temp>py Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> '{:02X}'.format(256) '100' >>> What I expected is '00'. Am I wrong? Best Regards, Jach Fong From tkadm30 at yandex.com Mon Mar 19 19:54:53 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Mon, 19 Mar 2018 19:54:53 -0400 Subject: Thank you Python community! In-Reply-To: <858tan1vu2.fsf@benfinney.id.au> References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> <858tan1vu2.fsf@benfinney.id.au> Message-ID: <47e53d9e-4697-7646-7fad-500957a8a505@yandex.com> Le 2018-03-19 ? 19:36, Ben Finney a ?crit?: > Etienne Robillard writes: > >> I would like to make such an experimental research/investigation on >> the effects of Python software programming on opioid addiction. :-) > Okay. The wording of your message implied that you know this already > happens now, though. How did you come to know this? > I'm not really sure what you mean here. Anyways, i found "Drugs Inc." a highly informative and educational documentary. I just thought that doing some Python coding should be a very effective way to stimulate your brain in positive ways. Etienne -- Etienne Robillard tkadm30 at yandex.com https://www.isotopesoftware.ca/ From rosuav at gmail.com Mon Mar 19 20:05:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Mar 2018 11:05:49 +1100 Subject: {:2X} didn't output what I expected In-Reply-To: <1f899b8f-534d-43fb-a95a-acd9fee796a2@googlegroups.com> References: <1f899b8f-534d-43fb-a95a-acd9fee796a2@googlegroups.com> Message-ID: On Tue, Mar 20, 2018 at 10:46 AM, wrote: > D:\Temp>py > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. >>>> '{:02X}'.format(256) > '100' >>>> > What I expected is '00'. Am I wrong? > Python avoids losing data. If you really want to enforce that this is two characters long, you can either restrict the data first (maybe with "% 256"), or trim the resulting string: >>> '{:02X}'.format(256)[-2:] '00' ChrisA From jfong at ms4.hinet.net Mon Mar 19 20:32:00 2018 From: jfong at ms4.hinet.net (jfong at ms4.hinet.net) Date: Mon, 19 Mar 2018 17:32:00 -0700 (PDT) Subject: {:2X} didn't output what I expected In-Reply-To: References: <1f899b8f-534d-43fb-a95a-acd9fee796a2@googlegroups.com> Message-ID: <61624624-c71d-469b-bcb0-7244dfdd7e3d@googlegroups.com> Chris Angelico? 2018?3?20???? UTC+8??8?06?05???? > On Tue, Mar 20, 2018 at 10:46 AM, wrote: > > D:\Temp>py > > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > >>>> '{:02X}'.format(256) > > '100' > >>>> > > What I expected is '00'. Am I wrong? > > > > Python avoids losing data. If you really want to enforce that this is > two characters long, you can either restrict the data first (maybe > with "% 256"), or trim the resulting string: > > >>> '{:02X}'.format(256)[-2:] > '00' > > ChrisA I had overlooked the document there it says "width ... defining the minimum field width...'. I was wrong, it's not a demand. Thank you, ChrisA. --Jach From rosuav at gmail.com Mon Mar 19 20:58:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 20 Mar 2018 11:58:21 +1100 Subject: {:2X} didn't output what I expected In-Reply-To: <61624624-c71d-469b-bcb0-7244dfdd7e3d@googlegroups.com> References: <1f899b8f-534d-43fb-a95a-acd9fee796a2@googlegroups.com> <61624624-c71d-469b-bcb0-7244dfdd7e3d@googlegroups.com> Message-ID: On Tue, Mar 20, 2018 at 11:32 AM, wrote: > Chris Angelico? 2018?3?20???? UTC+8??8?06?05???? >> On Tue, Mar 20, 2018 at 10:46 AM, wrote: >> > D:\Temp>py >> > Python 3.4.4 (v3.4.4:737efcadf5a6, Dec 20 2015, 19:28:18) [MSC v.1600 32 bit (Intel)] on win32 >> > Type "help", "copyright", "credits" or "license" for more information. >> >>>> '{:02X}'.format(256) >> > '100' >> >>>> >> > What I expected is '00'. Am I wrong? >> > >> >> Python avoids losing data. If you really want to enforce that this is >> two characters long, you can either restrict the data first (maybe >> with "% 256"), or trim the resulting string: >> >> >>> '{:02X}'.format(256)[-2:] >> '00' >> >> ChrisA > > I had overlooked the document there it says "width ... defining the minimum field width...'. I was wrong, it's not a demand. Thank you, ChrisA. Yep. It's a policy that goes back a long way. These kinds of width markers are often used for columnar data; but if something doesn't fit, it's far better to push the column out a bit (ugly) than to chop off a digit (loss of data, especially bad if it trims from the front). So if you actually DO want that, you need to specifically request it. ChrisA From zhuzxsandy at gmail.com Mon Mar 19 20:59:05 2018 From: zhuzxsandy at gmail.com (Nathan Zhu) Date: Tue, 20 Mar 2018 08:59:05 +0800 Subject: how to obtain the text for BeautifulSoup object Message-ID: Hi Team, could anyone help me? for webpage having source code like this: ... number name I only can use below sentence, since there are a lot of tag em and tag a in other area. output = bs4.BeautifulSoup(res.content,'lxml').findAll("span",{"class":"xst thread-name"}) how can I get the text in tag em and tag a under tag span? thank you for your support! Nathan From rantingrickjohnson at gmail.com Mon Mar 19 22:21:04 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Mon, 19 Mar 2018 19:21:04 -0700 (PDT) Subject: Thank you Python community! In-Reply-To: References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> <858tan1vu2.fsf@benfinney.id.au> Message-ID: <78521e33-0739-4706-a9ac-973eef7bcfdf@googlegroups.com> On Monday, March 19, 2018 at 6:37:21 PM UTC-5, Ben Finney wrote: > -- > \ "Success is going from one failure to the next without a loss | > `\ of enthusiasm." -- Winston Churchill | > _o__) | > Ben Finney > -- > \ ?Experience is that marvelous thing that enables you to | > `\ recognize a mistake when you make it again.? ?Franklin P. Jones | > _o__) | > Ben Finney Okay Ben, i believe it's time to come clean and finally admit to this fine community, that these quotes of yours are not merely chosen at random, but are in fact, delectable morsels of laser-focused sarcasm. From tjreedy at udel.edu Tue Mar 20 02:43:13 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 20 Mar 2018 02:43:13 -0400 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: On 3/19/2018 1:04 PM, Irv Kalb wrote: > However, there is one warning that I am seeing often, an > I'm not sure about how to handle it. The warning I see is: > > "Instance attribute defined outside of __init__ ..." Style checkers are notorious for sometimes giving bad advice and being overly opinionated. I think a claim that in all programs all attributes should be set *in* __init__, as opposed to *during* initialization, is wrong. All attribute setting is side-effect from a functional view (and usually 'bad' to a functionalist). There is no reason to not delegate some of it to sub-init functions when it makes sense to do do. There is good reason to do so when it makes the code easier to understand *and test*. Example: the IDLE's options setting dialog. Until last summer, it was one class* with at perhaps a thousand lines of initialization code (with no automated test). To me, the dictate that they should have all been in the __init__ function is absurd. Fortunately, there were about 9 subsidiary functions. * We split off a class for each tab, but keep the separate functions to create the tab page widgets and load them with current values. > The following is a simple example. I am creating a card class that I am using to build card games with PyGame. > class Card(): > > BACK_OF_CARD_IMAGE = pygame.image.load('images/backOfCard.png') > > def __init__(self, window, name, suit, value): > self.window = window > self.suit = suit > self.cardName = name + ' of ' + suit > self.value = value > fileName = 'images/' + self.cardName + '.png' > self.image = pygame.image.load(fileName) > self.backOfCardImage = Card.BACK_OF_CARD_IMAGE > > self.conceal() > > def conceal(self): > self.faceUp = False > > def reveal(self): > self.faceUp = True If the single line is all these functions do, I *might* suggest getting rid of them. But this is really a separate issue. -- Terry Jan Reedy From __peter__ at web.de Tue Mar 20 03:39:58 2018 From: __peter__ at web.de (Peter Otten) Date: Tue, 20 Mar 2018 08:39:58 +0100 Subject: how to obtain the text for BeautifulSoup object References: Message-ID: Nathan Zhu wrote: > Hi Team, > > could anyone help me? > > for webpage having source code like this: > ... > > number > name > > > I only can use below sentence, since there are a lot of tag em and tag a > in other area. > output = > bs4.BeautifulSoup(res.content,'lxml').findAll("span",{"class":"xst > thread-name"}) > > how can I get the text in tag em and tag a under tag span? for thread in output: print("em:", thread.em.text) print("a:", thread.a.text) print() From tkadm30 at yandex.com Tue Mar 20 03:57:05 2018 From: tkadm30 at yandex.com (Etienne Robillard) Date: Tue, 20 Mar 2018 03:57:05 -0400 Subject: Thank you Python community! In-Reply-To: <78521e33-0739-4706-a9ac-973eef7bcfdf@googlegroups.com> References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> <858tan1vu2.fsf@benfinney.id.au> <78521e33-0739-4706-a9ac-973eef7bcfdf@googlegroups.com> Message-ID: Le 2018-03-19 ? 22:21, Rick Johnson a ?crit?: > On Monday, March 19, 2018 at 6:37:21 PM UTC-5, Ben Finney wrote: > >> -- >> \ "Success is going from one failure to the next without a loss | >> `\ of enthusiasm." -- Winston Churchill | >> _o__) | >> Ben Finney > >> -- >> \ ?Experience is that marvelous thing that enables you to | >> `\ recognize a mistake when you make it again.? ?Franklin P. Jones | >> _o__) | >> Ben Finney > Okay Ben, i believe it's time to come clean and finally admit to this fine > community, that these quotes of yours are not merely chosen at random, but are > in fact, delectable morsels of laser-focused sarcasm. Here's another one I have in mind theses days: "The only thing necessary for the triumph of evil is for good men to do nothing." -Edmund Burke Etienne -- Etienne Robillard tkadm30 at yandex.com https://www.isotoperesearch.ca/ From steve+comp.lang.python at pearwood.info Tue Mar 20 04:52:29 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2018 08:52:29 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: On Tue, 20 Mar 2018 02:43:13 -0400, Terry Reedy wrote: > I think a claim that in all programs all attributes should be set *in* > __init__, as opposed to *during* initialization, is wrong. All > attribute setting is side-effect from a functional view (and usually > 'bad' to a functionalist). There is no reason to not delegate some of > it to sub-init functions when it makes sense to do do. There is good > reason to do so when it makes the code easier to understand *and test*. That is really well said Terry, thank you for articulating what I was thinking but couldn't find the words for. -- Steve From renting at astron.nl Tue Mar 20 06:19:59 2018 From: renting at astron.nl (Adriaan Renting) Date: Tue, 20 Mar 2018 11:19:59 +0100 Subject: [OT] Re: Thank you Python community! References: <5AB0D2860200001B0001EDE7@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> Message-ID: <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> That sounds more like a conspiracy theory than a real analysis of the problem. Looking at it from here in Europe, most of the analysis I've been able to read and watch about it, points to a different cause: A lack of security: People flee to drugs (alcohol, tobacco, coffee and other (illegal) drugs) to escape stessful environments. - A lack of job security - A lack of proper healthcare - Broken and single parent families - Poor and/or expensive education - Mental illness What I learned from visiting the USA many times is: - Americans value the individual, not the collective. They have been trained to distrust anything collective, even if it is the only way to solve an issue. - Money has too much influence at all levels, especially in politics. The combination of an election system that is 2 centuries behind the times, with modern compute power, data gathering and media technology is corrupting the system. - The USA always depended a lot on immigration and never had to keep it's own working population well educated and healthy to run the economy. This is changing and one of the big cultural wars raging currently. The broken healthcare system is a symptom of these issues. There are other reasons people get into drugs*, but from what I've read, my impression is that these are major causes to the current problems in the USA. Successive US governments have been waging unwinnable "wars", like the "War on Drugs" for decades, so entire industries can keep profiting. Until you fix the problems in society the demand will not go away and the problems will stay. Cheers. *) Drug use also correlates with boredom and low amounts of sunlight and an amount of predisposition. Correlation is not always causation, but can be. >>> Larry Martell 19-3-2018 20:21 >>> On Mon, Mar 19, 2018 at 12:08 PM, Etienne Robillard wrote: > You guys just made me realize something very obvious. :-) > > I'm in the process right now of watching the excellent documentary named > "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned about > the major opioid epidemic in the US. Have no clue what this has to do with python, but the opioid epidemic was created by collision between big pharma and the government. -- https://mail.python.org/mailman/listinfo/python-list Adriaan Renting | Email: renting at astron.nl Software Engineer Radio Observatory ASTRON | Phone: +31 521 595 100 (797 direct) P.O. Box 2 | GSM: +31 6 24 25 17 28 NL-7990 AA Dwingeloo | FAX: +31 521 595 101 The Netherlands | Web: http://www.astron.nl/~renting/ From alister.ware at ntlworld.com Tue Mar 20 07:14:13 2018 From: alister.ware at ntlworld.com (Alister) Date: Tue, 20 Mar 2018 11:14:13 GMT Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: <926sC.19683$lm7.13822@fx12.am4> On Tue, 20 Mar 2018 08:52:29 +0000, Steven D'Aprano wrote: > On Tue, 20 Mar 2018 02:43:13 -0400, Terry Reedy wrote: > >> I think a claim that in all programs all attributes should be set *in* >> __init__, as opposed to *during* initialization, is wrong. All >> attribute setting is side-effect from a functional view (and usually >> 'bad' to a functionalist). There is no reason to not delegate some of >> it to sub-init functions when it makes sense to do do. There is good >> reason to do so when it makes the code easier to understand *and test*. > > That is really well said Terry, thank you for articulating what I was > thinking but couldn't find the words for. but why would a functional programmer be programming an OOP class? -- No one becomes depraved in a moment. -- Decimus Junius Juvenalis From alister.ware at ntlworld.com Tue Mar 20 07:18:12 2018 From: alister.ware at ntlworld.com (Alister) Date: Tue, 20 Mar 2018 11:18:12 GMT Subject: Thank you Python community! References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> <858tan1vu2.fsf@benfinney.id.au> <78521e33-0739-4706-a9ac-973eef7bcfdf@googlegroups.com> Message-ID: On Mon, 19 Mar 2018 19:21:04 -0700, Rick Johnson wrote: > On Monday, March 19, 2018 at 6:37:21 PM UTC-5, Ben Finney wrote: > >> -- >> \ "Success is going from one failure to the next without a loss >> | >> `\ of enthusiasm." -- Winston Churchill >> | >> _o__) >> | >> Ben Finney > > >> -- >> \ ?Experience is that marvelous thing that enables you to >> | >> `\ recognize a mistake when you make it again.? ?Franklin P. Jones >> | >> _o__) >> | >> Ben Finney > > Okay Ben, i believe it's time to come clean and finally admit to this > fine community, that these quotes of yours are not merely chosen at > random, but are in fact, delectable morsels of laser-focused sarcasm. maybe not - I use fortune to generate mine & it can be supprisingly apt at times -- The primary function of the design engineer is to make things difficult for the fabricator and impossible for the serviceman. From larry.martell at gmail.com Tue Mar 20 08:37:13 2018 From: larry.martell at gmail.com (Larry Martell) Date: Tue, 20 Mar 2018 08:37:13 -0400 Subject: [OT] Re: Thank you Python community! In-Reply-To: <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> References: <5AB0D2860200001B0001EDE7@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> Message-ID: On Tue, Mar 20, 2018 at 6:19 AM, Adriaan Renting wrote: > > That sounds more like a conspiracy theory than a real analysis of the > problem. Follow the money. > > > Looking at it from here in Europe, most of the analysis I've been able to > read and watch about it, points to a different cause: > > > A lack of security: People flee to drugs (alcohol, tobacco, coffee and other > (illegal) drugs) to escape stessful environments. > > > - A lack of job security > > - A lack of proper healthcare > > - Broken and single parent families > > - Poor and/or expensive education > > - Mental illness > > > What I learned from visiting the USA many times is: > > - Americans value the individual, not the collective. They have been trained > to distrust anything collective, even if it is the only way to solve an > issue. > > - Money has too much influence at all levels, especially in politics. The > combination of an election system that is 2 centuries behind the times, with > modern compute power, data gathering and media technology is corrupting the > system. > > - The USA always depended a lot on immigration and never had to keep it's > own working population well educated and healthy to run the economy. This is > changing and one of the big cultural wars raging currently. > > > The broken healthcare system is a symptom of these issues. > > > There are other reasons people get into drugs*, but from what I've read, my > impression is that these are major causes to the current problems in the > USA. > > > Successive US governments have been waging unwinnable "wars", like the "War > on Drugs" for decades, so entire industries can keep profiting. Until you > fix the problems in society the demand will not go away and the problems > will stay. > > > Cheers. > > > *) Drug use also correlates with boredom and low amounts of sunlight and an > amount of predisposition. Correlation is not always causation, but can be. > > >>>> Larry Martell 19-3-2018 20:21 >>> > On Mon, Mar 19, 2018 at 12:08 PM, Etienne Robillard > wrote: >> You guys just made me realize something very obvious. :-) >> >> I'm in the process right now of watching the excellent documentary named >> "Drugs Inc." on Netflix and I'm basically stunned and deeply concerned >> about >> the major opioid epidemic in the US. > > Have no clue what this has to do with python, but the opioid epidemic > was created by collision between big pharma and the government. From grant.b.edwards at gmail.com Tue Mar 20 10:14:56 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 20 Mar 2018 14:14:56 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> Message-ID: On 2018-03-20, Alister via Python-list wrote: > but why would a functional programmer be programming an OOP class? Part of a 12-step recovery plan? ;) -- Grant Edwards grant.b.edwards Yow! Hello. I know at the divorce rate among gmail.com unmarried Catholic Alaskan females!! From robin at reportlab.com Tue Mar 20 10:21:02 2018 From: robin at reportlab.com (Robin Becker) Date: Tue, 20 Mar 2018 14:21:02 +0000 Subject: curious asymmetry in range limiting Message-ID: <38e315cd-6b23-5073-c922-2f46cbb08b0e@chamonix.reportlab.co.uk> I don't know how I never came across this before, but there's a curious asymmetry in the way ranges are limited Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> s = '0123456789' >>> print(repr(s[-5:5])) '' >>> print(repr(s[5:15])) '56789' >>> why is the underflow start index treated so differently from the limit index overflow? I suppose there must be some reason, but it eludes me. -- Robin Becker From robin at reportlab.com Tue Mar 20 10:36:46 2018 From: robin at reportlab.com (Robin Becker) Date: Tue, 20 Mar 2018 14:36:46 +0000 Subject: curious asymmetry in range limiting In-Reply-To: <38e315cd-6b23-5073-c922-2f46cbb08b0e@chamonix.reportlab.co.uk> References: <38e315cd-6b23-5073-c922-2f46cbb08b0e@chamonix.reportlab.co.uk> Message-ID: So eventually I tried this > C:\code\hg-repos\reportlab>\python36\python -c"s='0123456789';print(repr(s[-5:15]))" > '56789' > > C:\code\hg-repos\reportlab>\python36\python -c"s='0123456789';print(repr(s[-6:15]))" > '456789' and the light dawned :) seems the negative indexes rules apply to both On 20/03/2018 14:21, Robin Becker wrote: > I don't know how I never came across this before, but there's a curious asymmetry in the way ranges are limited > > Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> s = '0123456789' > >>> print(repr(s[-5:5])) > '' > >>> print(repr(s[5:15])) > '56789' > >>> > > why is the underflow start index treated so differently from the limit index overflow? I suppose there must be some reason, but it > eludes me. -- Robin Becker From wolfgang.maier at biologie.uni-freiburg.de Tue Mar 20 10:42:27 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Tue, 20 Mar 2018 15:42:27 +0100 Subject: curious asymmetry in range limiting In-Reply-To: <38e315cd-6b23-5073-c922-2f46cbb08b0e@chamonix.reportlab.co.uk> References: <38e315cd-6b23-5073-c922-2f46cbb08b0e@chamonix.reportlab.co.uk> Message-ID: On 03/20/2018 03:21 PM, Robin Becker wrote: > I don't know how I never came across this before, but there's a curious asymmetry in the way ranges are limited > > Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> s = '0123456789' > >>> print(repr(s[-5:5])) > '' > >>> print(repr(s[5:15])) > '56789' > >>> > > why is the underflow start index treated so differently from the limit index overflow? I suppose there must be some reason, but it > eludes me. > It's because in the slice [-5:5] the -5 is not trying to access an element at an index < 0, but indicating the fifth-last element in the sequence. From rantingrickjohnson at gmail.com Tue Mar 20 11:10:15 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Tue, 20 Mar 2018 08:10:15 -0700 (PDT) Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: <359cd818-74ae-45ba-a0a1-c1379d3461fe@googlegroups.com> On Tuesday, March 20, 2018 at 1:43:39 AM UTC-5, Terry Reedy wrote: [...] > > class Card(): > > > > BACK_OF_CARD_IMAGE = pygame.image.load('images/backOfCard.png') > > > > def __init__(self, window, name, suit, value): > > self.window = window > > self.suit = suit > > self.cardName = name + ' of ' + suit > > self.value = value > > fileName = 'images/' + self.cardName + '.png' > > self.image = pygame.image.load(fileName) > > self.backOfCardImage = Card.BACK_OF_CARD_IMAGE > > > > self.conceal() > > > > def conceal(self): > > self.faceUp = False > > > > def reveal(self): > > self.faceUp = True > > If the single line is all these functions do, I *might* > suggest getting rid of them. But this is really a separate > issue. While i normally agree with your advice Terry, in this case, i must protest as i am a firm believer that callers should _never_ be fiddling with the internals of an object unless doing so is the only available option -- and such cases of last resort indicate a code-smell. Whether the practice is considered Pythonic or not (i don't care), i would strongly suggest to the OP and anyone within ear shot, that clear interfaces must be _strictly_ utilized. Sure, in this case a named function may be overkill, however, even in the superfluous cases, not providing an interface encourage callers to engage in sloppy behavior and reinforces sloppy behavior in authors. And both parties will be doomed to maintenance nightmares when `reveal` and/or `conceal` grow more functional behavior. Laziness is not always a bad thing -- i mean, let's face it folks, python is designed to be a lazy language -- but experience has proven that when an author refuses to provide clear interfaces to his/her objects, the author and the caller will both pay a heavy price in the future. IOWs, laziness should never be allowed to wax into an orthodoxy. From rantingrickjohnson at gmail.com Tue Mar 20 11:14:15 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Tue, 20 Mar 2018 08:14:15 -0700 (PDT) Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <926sC.19683$lm7.13822@fx12.am4> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> Message-ID: <4d0d6f0c-0ce3-46ec-8c20-471b27ded6e0@googlegroups.com> On Tuesday, March 20, 2018 at 6:14:34 AM UTC-5, Alister wrote: > On Tue, 20 Mar 2018 08:52:29 +0000, Steven D'Aprano wrote: > > On Tue, 20 Mar 2018 02:43:13 -0400, Terry Reedy wrote: > > > > > I think a claim that in all programs all attributes > > > should be set *in* __init__, as opposed to *during* > > > initialization, is wrong. All attribute setting is side- > > > effect from a functional view (and usually 'bad' to a > > > functionalist). There is no reason to not delegate some > > > of it to sub-init functions when it makes sense to do do. > > > There is good reason to do so when it makes the code > > > easier to understand *and test*. > > > > That is really well said Terry, thank you for articulating > > what I was thinking but couldn't find the words for. > > but why would a functional programmer be programming an OOP > class? The question does indeed beg, but we're not allowed to question mandates form "on high", we are simply required to follow them. IOWs, "Do as they _say_, not as logic dictates" From steve+comp.lang.python at pearwood.info Tue Mar 20 11:15:34 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2018 15:15:34 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> Message-ID: On Tue, 20 Mar 2018 11:14:13 +0000, Alister via Python-list wrote: > but why would a functional programmer be programming an OOP class? I read a really good blog post a while back, which I sadly can no longer find (I had it bookmarked until Firefox ate my bookmarks) that discussed exactly that question. What they came up with is that its really hard to do useful work for large applications with 100% pure functional programming styles. As you add functional techniques to your code, you find your code quality increasing. Things like pure functions with no hidden state and no side effects can have a massive positive effect on code quality -- up to a point. As you approach closer and closer to 100% pure functional code, you get fewer improvements and your code gets harder to write and maintain. (Think about things like dealing with user preferences, and having to pass them around from function to function. Think about functions that don't need a parameter themselves, but need to call a function that does. Think about data processing without any mutable data structures. Sometimes a modicum of state is exactly what we need.) He maintains that the sweet spot is about 80% functional, and 20% OOP. That seems to be about right to me: write classes with just the bare minimum state you need, and have your methods work using a nearly-pure functional style: they should take arguments, they should return results, they should rarely operate by side-effects, etc. Most of Python's design is like that. Apart from a few mutable data structures, Python's built-ins tend to be immutable. We don't do things like: finder = SubstringFinder(start=1, end=55) finder.set_string(string_to_search) finder.set_target(substring) position = finder.find() rather we use the semi-functional style: position = string_to_search.find(substring, start=1, end=55) Don't be fooled by the dot method call syntax: that's exactly equivalent to the purely functional style find(string_to_search, substring, start=1, end=55) -- Steve From steve+comp.lang.python at pearwood.info Tue Mar 20 11:17:55 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2018 15:17:55 +0000 (UTC) Subject: how do I retry a command only for a specific exception / error References: Message-ID: On Mon, 19 Mar 2018 22:08:09 +0530, Ganesh Pal wrote: I'm sorry Ganesh, you have appeared to have just quoted my post without writing anything new. (I haven't taken the time to read your post in fine detail.) Apart from "Regards, Ganesh" at the very end, everything is quoted text (starting with a > sign). Please ensure quoted text is quoted, and new text you write is unquoted. That way you are more likely to get useful replies. -- Steve From rosuav at gmail.com Tue Mar 20 11:44:32 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Mar 2018 02:44:32 +1100 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> Message-ID: On Wed, Mar 21, 2018 at 2:15 AM, Steven D'Aprano wrote: > On Tue, 20 Mar 2018 11:14:13 +0000, Alister via Python-list wrote: > >> but why would a functional programmer be programming an OOP class? > > I read a really good blog post a while back, which I sadly can no longer > find (I had it bookmarked until Firefox ate my bookmarks) that discussed > exactly that question. > > What they came up with is that its really hard to do useful work for > large applications with 100% pure functional programming styles. As you > add functional techniques to your code, you find your code quality > increasing. Things like pure functions with no hidden state and no side > effects can have a massive positive effect on code quality -- up to a > point. > > As you approach closer and closer to 100% pure functional code, you get > fewer improvements and your code gets harder to write and maintain. > > He maintains that the sweet spot is about 80% functional, and 20% OOP. Yep, I remember that article. Sadly, I can't find it back now either; maybe someone can craft a regular expression to find it? https://xkcd.com/208/ This week I dived into the source code for an application that's built out of 20 Python files in four package directories plus a top-level "app.py" as its entry point. You'd think that that means I can write my own Python script that imports from those packages and calls on the same code, right? Sadly, no; because this app is 100% OOP and 0% functional. (Not to be confused with a *non-functioning* app; it functions perfectly well. It just doesn't follow *functional programming* idioms.) Deep inside the imported modules, a class's __init__ will check command-line arguments to control its behaviour, and in the event of a problem (eg JSON data lacking vital information), will print a failure message and exit the program. Were this class written in a pure functional style, it would use the construction parameters, and raise an exception if something's wrong. It's entirely possible for a class to follow functional programming style, at least partly. (In theory, it should be possible to use a class in a 100% functional style. A class is just a callable function that then happens to return a thing of a type that has the same name as the thing you called, so there's no reason the two styles can't completely overlap.) ChrisA From rantingrickjohnson at gmail.com Tue Mar 20 12:08:57 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Tue, 20 Mar 2018 09:08:57 -0700 (PDT) Subject: [OT] Re: Thank you Python community! In-Reply-To: References: <5AB0D2860200001B0001EDE7@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> Message-ID: <165fde7c-8675-419b-8d86-5a21896bda17@googlegroups.com> On Tuesday, March 20, 2018 at 7:03:11 AM UTC-5, Adriaan Renting wrote: (on the subject of the opioid epidemic) > That sounds more like a conspiracy theory than a real > analysis of the problem. Looking at it from here in > Europe, most of the analysis I've been able to read and > watch about it, points to a different cause: A lack of > security: People flee to drugs (alcohol, tobacco, coffee > and other (illegal) drugs) to escape st[r]essful > environments. Personally, I drink coffee simply so i can function in the morning. If i'm feeling anything, it's unconsciousness. As far as drugs, alcohol and tobacco are concerned; those days are long behind me. > - A lack of job security > - A lack of proper healthcare > - Broken and single parent families > - Poor and/or expensive education > - Mental illness And don't forget boredom! ;-) > What I learned from visiting the USA many times is: - > Americans value the individual, not the collective. Yes. Individualism is a natural part of rejecting your birth county; packing up your belonging; sailing across an ocean; and starting over from scratch. To the "true americans" (not the fake ones who come here looking for handouts) there is nothing more important than freedom. A true american would never sacrifice freedom for security. A true american spits in the face of so-called security, because they know it's a bald faced lie, as there is no such thing as "security" in such a hostile universe as our. "Security" is simply snake oil sold to ignoramuses. It plays on the emotions of infantile humans. > They have been trained to distrust anything collective, True americans are born in every country of the world. They come from all walks of life and every ethnic and racial background. But of all their differences, they share a common respect for individual liberty. And that's why most outsiders cannot understand how america could possibly become so strong in a paltry few hundered years and survive for as long as she has. > even if it is the only way to solve an issue. - Money has > too much influence at all levels, especially in politics. Money has been ruining politics for thousands of years... where have your been? > The combination of an election system that is 2 centuries > behind the times, with modern compute power, data gathering > and media technology is corrupting the system. The media is to blame for much of the ills of modern society, bathing our subconscious in gratuitous violence, and stoking our emotions over the most frivolous and petty differences between us. > - The USA always depended a lot on immigration and never > had to keep it's own working population well educated and > healthy to run the economy. This is changing and one of the > big cultural wars raging currently. Again, another failure of government who is in bed with greedy corporation. Nationalism is feared and branded as pure evil these days due to one most atrocious usage of it in recent history, but nationalism (the non-religios kind!) may be the _only_ force that has prevented thermo nuclear war up to this point. Even that murderous jerk Stalin would not have wanted to witness his beloved USSR smoulder to ashes. But with the rabid stigmatization of nationalism these days, we are entering a time in which the masses no longer care about their fellow country(wo)men. And in a world of Mutually Assured Destruction, a lack of social comradery leads to national suicide. And in this universe, there is no choice or action that is devoid of consequence. All forces possess varying degrees applicability. > The broken healthcare system is a symptom of these issues. "Broken healthcare" is a buzz word that animates emotional weaklings. The state of health care has absolutely nothing to do with the social turmoil that plagues us. "Healthcare" is only a recent phenomenon. Hardly a few hundred years ago doctors were few and far between. And to believe that "fixing the healthcare system" (whatever that means!) will suddenly usher in an "epoch of Universal Utopianism", is the bone-headed height of naivete. Don't be a fool. Healthcare has nothing to do with our overall happiness, it is merely another divisive tool utilized by the manipulators to animate the selfish and jealous idiots into a mad frenzy. An in such a discombobulated emotional state, humans are more willing to sacrifice freedom for paltry priviledge. Rights cannot be taken away. Whereas priviledge _can_. > There are other reasons people get into drugs*, but from > what I've read, my impression is that these are major > causes to the current problems in the USA. BRAVO! YES! THAT'S RIGHT! Blame all the world's problems on america. At least that way, you won't have to actually _think_! @_@ > Successive US governments have been waging unwinnable > "wars", like the "War on Drugs" for decades, so entire > industries can keep profiting. Until you fix the problems > in society the demand will not go away and the problems > will stay. The war on drugs has been the most ignorant of all wars ever fought in history of human existence. It elevates the status of criminals and enrichs them; encourages gang violence; kills and harms innocent people on a daily basis; contributes to governmental corruption. And it does all of this while endowing the busibodies among us (the most putrid and destructive of all human phenotypes) with a sense of self worth. From tevans.uk at googlemail.com Tue Mar 20 12:22:12 2018 From: tevans.uk at googlemail.com (Tom Evans) Date: Tue, 20 Mar 2018 16:22:12 +0000 Subject: Writing a C extension - borrowed references Message-ID: Hi all I'm writing my first C extension for Python here, and all is going well. However, I was reading [1], and the author there is advocating Py_INCREF 'ing *every* borrowed reference. Now, I get that if I do something to mutate and perhaps invalidate the PyObject that was borrowed I can get unpredictable results - eg, by getting a borrowed reference to a value from a dictionary, clearing the dictionary and then expecting to use the borrowed reference. However, if my code does not do things like that, is there a chance of a borrowed reference being invalidated that is not related to my use of the thing I borrowed it from? Is this cargo cult advice (sometimes the gods need this structure, and so to please the gods it must be everywhere), sensible belt and braces or needless overkill? Cheers Tom [1] http://pythonextensionpatterns.readthedocs.io/en/latest/refcount.html#borrowed-references From p.f.moore at gmail.com Tue Mar 20 12:33:15 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 20 Mar 2018 16:33:15 +0000 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On 20 March 2018 at 16:22, Tom Evans via Python-list wrote: > Hi all > > I'm writing my first C extension for Python here, and all is going > well. However, I was reading [1], and the author there is advocating > Py_INCREF 'ing *every* borrowed reference. > > Now, I get that if I do something to mutate and perhaps invalidate the > PyObject that was borrowed I can get unpredictable results - eg, by > getting a borrowed reference to a value from a dictionary, clearing > the dictionary and then expecting to use the borrowed reference. > > However, if my code does not do things like that, is there a chance of > a borrowed reference being invalidated that is not related to my use > of the thing I borrowed it from? Is this cargo cult advice (sometimes > the gods need this structure, and so to please the gods it must be > everywhere), sensible belt and braces or needless overkill? You *can* safely use a borrowed reference, but knowing when it's safe to do so is definitely not easy. For example, in the presence of threads, your borrowed reference could be invalidated on another thread, so you have to be careful how the GIL is managed. It's often very difficult to be sure that code doesn't call back into arbitrary Python code, and you need to take care of that if you want to use the borrowed reference. Conversely, the benefit of not doing an INCREF is generally a pretty tiny performance improvement, which is only worthwhile if you're in an extremely tight loop. So while it's OK to use a borrowed reference, the reality is that if you don't fully understand why you want to, and all the trade-offs, it's probably not worth the risk. Or, to put it another way, "if you need to ask, you can't afford to". Paul From rosuav at gmail.com Tue Mar 20 12:37:25 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Mar 2018 03:37:25 +1100 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On Wed, Mar 21, 2018 at 3:22 AM, Tom Evans via Python-list wrote: > Hi all > > I'm writing my first C extension for Python here, and all is going > well. However, I was reading [1], and the author there is advocating > Py_INCREF 'ing *every* borrowed reference. > > Now, I get that if I do something to mutate and perhaps invalidate the > PyObject that was borrowed I can get unpredictable results - eg, by > getting a borrowed reference to a value from a dictionary, clearing > the dictionary and then expecting to use the borrowed reference. > > However, if my code does not do things like that, is there a chance of > a borrowed reference being invalidated that is not related to my use > of the thing I borrowed it from? Is this cargo cult advice (sometimes > the gods need this structure, and so to please the gods it must be > everywhere), sensible belt and braces or needless overkill? Yep, that's cargo cult advice. If it were good advice, the Python API would have been built to never return borrowed references, so you always have to decref everything. You should incref those sorts of borrowed references if you're going to do something that could call arbitrary Python code and then continue using the reference; but most of the time, that borrowed ref is going to be used immediately and then finished with, and in those very common situations, it's pointless to incref and decref. It's also perfectly safe to use borrowed refs with immutable objects that you have refs to; for instance, you can inspect the elements of a tuple that way. The tuple MUST be holding references to all its members, and the tuple itself can't be disposed of while you have a reference to it, so you have a guarantee that those borrowed refs are valid. """The analogy in C is having two pointers to the same memory location: so who is responsible for freeing the memory and what happens if the other pointer tries to access that free?d memory?""" That ignores the entire point of reference counting. By using a borrowed reference, you're clearly declaring that the OTHER guy is responsible for freeing it. >From footnote 3: """Of course we never remove items in a list we merely decrement their reference count (and if that hits zero then they are deleted).""" This is conflating two separate concepts. You most assuredly CAN remove items from a list, and the code in the footnote does exactly that. What it won't necessarily do is destroy those objects. """An important takeaway here is that incrementing and decrementing reference counts is a cheap operation but the consequences of getting it wrong can be expensive. A precautionary approach in your code might be to always increment borrowed references when they are instantiated and then always decrement them before they go out of scope. That way you incur two cheap operations but eliminate a vastly more expensive one.""" Actually, managing ref counts isn't always cheap. So instead of just always incrementing and always decrementing, how about this: If you are going to call arbitrary code that could do anything at all, guard around THAT CODE and nothing else, by protecting any borrowed references. People talk a lot about calling functions that could do literally anything, but honestly, how often does that even happen? Not often enough to justify cargo cult advice. ChrisA From rosuav at gmail.com Tue Mar 20 12:38:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Mar 2018 03:38:24 +1100 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On Wed, Mar 21, 2018 at 3:22 AM, Tom Evans via Python-list wrote: > Hi all > > I'm writing my first C extension for Python here, and all is going > well. However, I was reading [1], and the author there is advocating > Py_INCREF 'ing *every* borrowed reference. BTW, have you looked into Cython? It's smart enough to take care of a lot of this sort of thing for you. ChrisA From ned at nedbatchelder.com Tue Mar 20 12:55:10 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 20 Mar 2018 12:55:10 -0400 Subject: [OT] Re: Thank you Python community! In-Reply-To: <165fde7c-8675-419b-8d86-5a21896bda17@googlegroups.com> References: <5AB0D2860200001B0001EDE7@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> <5AB0E04F0200001B0001EDF5@smtp1.astron.nl> <165fde7c-8675-419b-8d86-5a21896bda17@googlegroups.com> Message-ID: On 3/20/18 12:08 PM, Rick Johnson wrote: > On Tuesday, March 20, 2018 at 7:03:11 AM UTC-5, Adriaan Renting wrote: > > (on the subject of the opioid epidemic) The [OT] in the subject line is right: let's not get off on a political tangent. --Ned. From tevans.uk at googlemail.com Tue Mar 20 13:03:53 2018 From: tevans.uk at googlemail.com (Tom Evans) Date: Tue, 20 Mar 2018 17:03:53 +0000 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On Tue, Mar 20, 2018 at 4:38 PM, Chris Angelico wrote: > BTW, have you looked into Cython? It's smart enough to take care of a > lot of this sort of thing for you. I did a bit; this work is to replace our old python 2 SAML client, which used python-lasso and python-libxml2, both packages that are built as part of building the C library and thus an utter PITA to package for different versions of python (something our Infra team were extremely reluctant to do). When the latest (PITA to build) version of python-lasso started interfering with python-xmlsec (validating an invalid signature was causing segfaults), I got fed up of fighting it. I actually also maintain a C version of the same code, using the same libraries, so porting those few segments of code to Python/C seemed more expedient than rewriting in Cython. I'm not writing an API to these libraries, just a few functions. Cheers Tom From Irv at furrypants.com Tue Mar 20 13:07:36 2018 From: Irv at furrypants.com (Irv Kalb) Date: Tue, 20 Mar 2018 10:07:36 -0700 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> Message-ID: <60D9F5B5-2FDE-42E1-834B-F7DD021E79EB@furrypants.com> > On Mar 19, 2018, at 10:27 AM, Ned Batchelder wrote: > > On 3/19/18 1:04 PM, Irv Kalb wrote: >> I am building some classes for use in future curriculum. I am using PyCharm for my development. On the right hand edge of the PyCharm editor window, you get some little bars indicating warnings or errors in your code. I like this feature and try to clean up as many of those as I can. However, there is one warning that I am seeing often, and I'm not sure about how to handle it. The warning I see is: >> >> "Instance attribute defined outside of __init__ ..." >> >> > > I understand this tension: it's nice to assign only meaningful values, and to do it in as few places as possible. But it's also nice to have all of your attributes in one place. This is a by-product of Python having no declarations, only definitions (assignments). So to mention the attributes initially, you have to choose a value for them. If I were to add faceUp to __init__, I would assign False to it. >> Thanks to everyone who responded to my question - good discussion. Special thanks to Ned who accurately described the "tension" that I am trying to resolve. I am aware of all the issues involved. My example code was an attempt to demonstrate the clearest, simplest case possible. My question is not about this small case. In classes designed for full games, I often have a "reset" method that resets many instance variables (and potentially other display fields and graphics) for a new round of playing a game. Grouping this into a method called something like "reset" makes logical sense to me. You tell the game object to reset itself, and it does whatever it needs to do to reset for a new round. My __init__ method calls reset to initialize the first round of the game. This ensures that every play of the game goes through the same initialization. With the approach suggested (initializing instance variables in my __init__ and in method like reset), I would wind up duplicating all such code. I know myself, and I am worried that this has the potential for introducing an error in my coding (specifically, add an initialization of an instance variable in my __init__ but then forget to add the same line in my "reset" method). So the tension that I am feeling is in the trade off of the "Don't Repeat Yourself" (DRY) rule vs the nicety of defining all instance variables in the __init__ method, and with that a resolution to the lint warnings. After further thought (and considering by the discussion here), I think that the DRY rule outweighs the other. I will keep my separate initialization methods, and call them at the end of my __init__ methods. Again, the real reason I'm asking for the "best practices" (even though I hate that term), is because I want to explain this concept to students who will be learning the basics of OOP. Thanks, Irv From python at bladeshadow.org Tue Mar 20 13:14:03 2018 From: python at bladeshadow.org (Python) Date: Tue, 20 Mar 2018 12:14:03 -0500 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: <20180320171403.GV10273@bladeshadow.org> On Tue, Mar 20, 2018 at 04:33:15PM +0000, Paul Moore wrote: > Or, to put it another way, "if you need to ask, you can't afford to". If you don't know, you need to ask so that you can learn whether or not you can afford to. :( From neilc at norwich.edu Tue Mar 20 13:18:33 2018 From: neilc at norwich.edu (Neil Cerutti) Date: Tue, 20 Mar 2018 17:18:33 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> <4d0d6f0c-0ce3-46ec-8c20-471b27ded6e0@googlegroups.com> Message-ID: On 2018-03-20, Rick Johnson wrote: > On Tuesday, March 20, 2018 at 6:14:34 AM UTC-5, Alister wrote: >> On Tue, 20 Mar 2018 08:52:29 +0000, Steven D'Aprano wrote: >> > On Tue, 20 Mar 2018 02:43:13 -0400, Terry Reedy wrote: >> > >> > > I think a claim that in all programs all attributes >> > > should be set *in* __init__, as opposed to *during* >> > > initialization, is wrong. All attribute setting is side- >> > > effect from a functional view (and usually 'bad' to a >> > > functionalist). There is no reason to not delegate some >> > > of it to sub-init functions when it makes sense to do do. >> > > There is good reason to do so when it makes the code >> > > easier to understand *and test*. >> > >> > That is really well said Terry, thank you for articulating >> > what I was thinking but couldn't find the words for. >> >> but why would a functional programmer be programming an OOP >> class? > > The question does indeed beg, but we're not allowed to > question mandates form "on high", we are simply required to > follow them. > > IOWs, "Do as they _say_, not as logic dictates" The Introduction to Computer Science class I'm taking divided program design into two categories: Top Down Design, and Object Oriented Design. It's good, because it reminded me of the wisdom of dividing memory into Random Access and Read Only. My automotive course will probaly divide cars into Automatic Transmission, and Front Wheel Drive. -- Neil Cerutti From grant.b.edwards at gmail.com Tue Mar 20 13:25:48 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 20 Mar 2018 17:25:48 +0000 (UTC) Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> <4d0d6f0c-0ce3-46ec-8c20-471b27ded6e0@googlegroups.com> Message-ID: On 2018-03-20, Neil Cerutti wrote: > My automotive course will probaly divide cars into Automatic > Transmission, and Front Wheel Drive. I get your point: the two characteristics are, in theory, orthogonal. But, in the US, the two appear to be correlated. ISTM that cars with manual transmissions are much more likely to be RWD than are automatics. -- Grant Edwards grant.b.edwards Yow! Inside, I'm already at SOBBING! gmail.com From rgaddi at highlandtechnology.invalid Tue Mar 20 13:36:44 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Tue, 20 Mar 2018 10:36:44 -0700 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On 03/20/2018 10:03 AM, Tom Evans wrote: > On Tue, Mar 20, 2018 at 4:38 PM, Chris Angelico wrote: >> BTW, have you looked into Cython? It's smart enough to take care of a >> lot of this sort of thing for you. > > I did a bit; this work is to replace our old python 2 SAML client, > which used python-lasso and python-libxml2, both packages that are > built as part of building the C library and thus an utter PITA to > package for different versions of python (something our Infra team > were extremely reluctant to do). When the latest (PITA to build) > version of python-lasso started interfering with python-xmlsec > (validating an invalid signature was causing segfaults), I got fed up > of fighting it. > > I actually also maintain a C version of the same code, using the same > libraries, so porting those few segments of code to Python/C seemed > more expedient than rewriting in Cython. I'm not writing an API to > these libraries, just a few functions. > > Cheers > > Tom > If all you're doing is a thin-wrapper around a C library, have you thought about just using ctypes? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From rosuav at gmail.com Tue Mar 20 13:45:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Mar 2018 04:45:50 +1100 Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> <4d0d6f0c-0ce3-46ec-8c20-471b27ded6e0@googlegroups.com> Message-ID: On Wed, Mar 21, 2018 at 4:18 AM, Neil Cerutti wrote: > The Introduction to Computer Science class I'm taking divided > program design into two categories: Top Down Design, and Object > Oriented Design. It's good, because it reminded me of the wisdom > of dividing memory into Random Access and Read Only. > > My automotive course will probaly divide cars into Automatic > Transmission, and Front Wheel Drive. Game reviews on Steam can be rated as Helpful, Unhelpful, or Funny, thus giving a ternary division of similar quality. ChrisA From tevans.uk at googlemail.com Tue Mar 20 13:46:34 2018 From: tevans.uk at googlemail.com (Tom Evans) Date: Tue, 20 Mar 2018 17:46:34 +0000 Subject: [OT] Re: Style Q: Instance variables defined outside of __init__ Message-ID: On Tue, Mar 20, 2018 at 5:25 PM, Grant Edwards wrote: > On 2018-03-20, Neil Cerutti wrote: > >> My automotive course will probaly divide cars into Automatic >> Transmission, and Front Wheel Drive. > > I get your point: the two characteristics are, in theory, orthogonal. > But, in the US, the two appear to be correlated. ISTM that cars with > manual transmissions are much more likely to be RWD than are > automatics. I find that slightly strange, I guess in the US, manual transmission correlates to the non default option(?), along with RWD. In Europe, RWD and automatic transmission are more expensive options than FWD and manual transmissions, and most cars are FWD and manual. At least most cars I can afford.. Cheers Tom From tevans.uk at googlemail.com Tue Mar 20 14:00:35 2018 From: tevans.uk at googlemail.com (Tom Evans) Date: Tue, 20 Mar 2018 18:00:35 +0000 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: On Tue, Mar 20, 2018 at 5:36 PM, Rob Gaddi wrote: > If all you're doing is a thin-wrapper around a C library, have you thought > about just using ctypes? Yep; the C library whose API I'm using uses macros to cast things to the right structure, and (similar to Cython), as I already _have_ the code, I wasn't particularly interested in working out how to convert things like: LASSO_SAMLP2_NAME_ID_POLICY(LASSO_SAMLP2_AUTHN_REQUEST( LASSO_PROFILE(login)->request)->NameIDPolicy )->Format = strdup(LASSO_SAML2_NAME_IDENTIFIER_FORMAT_PERSISTENT); into ctypes compatible syntax, when I can simply adapt the working C code to Python. :) Plus, there is the library static initialisation to manage, the issues of distributing the C libraries if I do a C wrapper to call from ctypes. This way, it can be distributed from our devpi very easily. Cheers Tom From john_ladasky at sbcglobal.net Tue Mar 20 14:02:18 2018 From: john_ladasky at sbcglobal.net (John Ladasky) Date: Tue, 20 Mar 2018 11:02:18 -0700 (PDT) Subject: Thank you Python community! In-Reply-To: References: Message-ID: <87a267c4-dd67-47c7-a9f5-2e9e71b3a8b5@googlegroups.com> On Monday, March 19, 2018 at 9:28:09 AM UTC-7, Etienne Robillard wrote: > I would like to thank you guys sincerely for helping a lot of people to > stay clean, and focus on programming high-level stuff in Python instead > of doing some really nasty drugs. Yeah, it's either Python or that horrifying street drug PHP. I know which one I'm choosing. From grant.b.edwards at gmail.com Tue Mar 20 14:41:08 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Tue, 20 Mar 2018 18:41:08 +0000 (UTC) Subject: [OT] Re: Style Q: Instance variables defined outside of __init__ References: Message-ID: On 2018-03-20, Tom Evans via Python-list wrote: > On Tue, Mar 20, 2018 at 5:25 PM, Grant Edwards > wrote: >> On 2018-03-20, Neil Cerutti wrote: >> >>> My automotive course will probaly divide cars into Automatic >>> Transmission, and Front Wheel Drive. >> >> I get your point: the two characteristics are, in theory, orthogonal. >> But, in the US, the two appear to be correlated. ISTM that cars with >> manual transmissions are much more likely to be RWD than are >> automatics. > > I find that slightly strange, I guess in the US, manual transmission > correlates to the non default option(?), along with RWD. In the US, manual transmissions are only offered on "sport" models/packges which tend to be sold to people who know/care more about cars that the average driver. Those models are also much more likely to be RWD because those people are much more likely to prefer the handling of RWD over that of FWD. > In Europe, RWD and automatic transmission are more expensive options > than FWD and manual transmissions, and most cars are FWD and manual. In the US, cars with manual transmission tend to be high priced because they are higher-performance, sports/enthusiast cars. They're either traditional US V8 muscle cars like Ford Mustangs and Dodge Callengers, or they're models trying to compete with BMW 3 and 5 series. They may be cheaper than similar performing cars with automatics (which are now often high-tech dual-clutch 8-speed mechanical wonders), but more expensive than the _average_ automatic transmission car. -- Grant Edwards grant.b.edwards Yow! I'm having a BIG BANG at THEORY!! gmail.com From codewizard at gmail.com Tue Mar 20 15:29:17 2018 From: codewizard at gmail.com (codewizard at gmail.com) Date: Tue, 20 Mar 2018 12:29:17 -0700 (PDT) Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <60D9F5B5-2FDE-42E1-834B-F7DD021E79EB@furrypants.com> Message-ID: On Tuesday, March 20, 2018 at 1:10:19 PM UTC-4, Irv Kalb wrote: > I am aware of all the issues involved. My example code was an attempt to demonstrate the clearest, simplest case possible. My question is not about this small case. In classes designed for full games, I often have a "reset" method that resets many instance variables (and potentially other display fields and graphics) for a new round of playing a game. Grouping this into a method called something like "reset" makes logical sense to me. You tell the game object to reset itself, and it does whatever it needs to do to reset for a new round. My __init__ method calls reset to initialize the first round of the game. This ensures that every play of the game goes through the same initialization. Calling reset() from __init__() and duplicating resetting logic during initialization aren't the only choices. My personal preference in such situations is to put all initialization code in __init__. Then write a separate factory function / method to create the new object for the next round. Optionally, this factory can be parametrized by the previous round object. Regards, Igor. From subramanianaug at gmail.com Tue Mar 20 15:49:07 2018 From: subramanianaug at gmail.com (Subramanian P V) Date: Tue, 20 Mar 2018 12:49:07 -0700 (PDT) Subject: in multiprocessing pool map how to stop one process from executing ahead Message-ID: <4ca5dcd6-905e-4193-ab11-333317196dbb@googlegroups.com> I am excecting custom commands like shell on multiple linux hosts. and if in one host one of the commands fail. I want that process not to proceed. If the remote command throws an error i am logging it .. but the process goes to next command . but if i terminate the command, the process will terminate all other processes on other linux machines as well. From ben+python at benfinney.id.au Tue Mar 20 18:03:11 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 21 Mar 2018 09:03:11 +1100 Subject: Celestial Emporium of Benevolent Knowledge (was: Style Q: Instance variables defined outside of __init__) References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> <4d0d6f0c-0ce3-46ec-8c20-471b27ded6e0@googlegroups.com> Message-ID: <85zi32z9pc.fsf_-_@benfinney.id.au> Neil Cerutti writes: > The Introduction to Computer Science class I'm taking divided program > design into two categories: Top Down Design, and Object Oriented > Design. It's good, because it reminded me of the wisdom of dividing > memory into Random Access and Read Only. > > My automotive course will probaly divide cars into Automatic > Transmission, and Front Wheel Drive. Look to the classics. Jorge Luis Borges, 1942: Let us consider the eighth category [of those proposed by John Wilkins for a universal language], the category of stones. Wilkins divides them into common (silica, gravel, schist), modics (marble, amber, coral), precious (pearl, opal), transparent (amethyst, sapphire) and insolubles (chalk, arsenic). Almost as surprising as the eighth, is the ninth category. This one reveals to us that metals can be imperfect (cinnabar, mercury), artificial (bronze, brass), recremental (filings, rust) and natural (gold, tin, copper). Beauty belongs to the sixteenth category; it is a living brood fish, an oblong one. These ambiguities, redundancies and deficiencies remind us of those which doctor Franz Kuhn attributes to a certain Chinese encyclopaedia entitled 'Celestial Empire of Benevolent Knowledge'. In its remote pages it is written that the animals are divided into: (a) belonging to the emperor, (b) embalmed, (c) tame, (d) sucking pigs, (e) sirens, (f) fabulous, (g) stray dogs, (h) included in the present classification, (i) frenzied, (j) innumerable, (k) drawn with a very fine camelhair brush, (l) et cetera, (m) having just broken the water pitcher, (n) that from a long way off look like flies. Those who want the source material for the Celestial Emporium of Benevolent Knowledge are recommended to first read the Wikipedia article . A relevant 2003 work gives an illustration of a similar system . -- \ ?I'm not a bad guy! I work hard, and I love my kids. So why | `\ should I spend half my Sunday hearing about how I'm going to | _o__) Hell?? ?Homer Simpson | Ben Finney From ben+python at benfinney.id.au Tue Mar 20 18:09:53 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 21 Mar 2018 09:09:53 +1100 Subject: Style Q: Instance variables defined outside of __init__ References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> Message-ID: <85vadqz9e6.fsf@benfinney.id.au> Steven D'Aprano writes: > What they came up with is that its really hard to do useful work for > large applications with 100% pure functional programming styles. As you > add functional techniques to your code, you find your code quality > increasing. Things like pure functions with no hidden state and no side > effects can have a massive positive effect on code quality -- up to a > point. > > As you approach closer and closer to 100% pure functional code, you get > fewer improvements and your code gets harder to write and maintain. Right. I hope no-one advocates a *purely* functional approach to any program that needs to do work on anything external. Functional programming is a paradigm that is an essential tool to *minimise* the side-effects of code, with a corresponding increase in the testability and therefore the confidence one can place in that code. Any program which needs to interact with systems outside itself ? which is to say, any program which performs useful work, ultimately ? must have side effects. So it's absurd to advocate removing *all* side effects. So, those who sneer that Python isn't purely functional are, in my view, paying Python a compliment: that it doesn't get in the way of doing useful work on things people care about getting done. Python supports a highly functional style, while not going IMO too far down the road of purity at the expense of usefulness. -- \ ?The reason we come up with new versions is not to fix bugs. | `\ It's absolutely not.? ?Bill Gates, 1995-10-23 | _o__) | Ben Finney From ben+python at benfinney.id.au Tue Mar 20 18:17:52 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 21 Mar 2018 09:17:52 +1100 Subject: Sig monster agency (was: Thank you Python community!) References: <85r2of20dr.fsf@benfinney.id.au> <08abe918-203a-aee7-8490-c365d31fbbb5@yandex.com> <858tan1vu2.fsf@benfinney.id.au> <78521e33-0739-4706-a9ac-973eef7bcfdf@googlegroups.com> Message-ID: <85r2oez90v.fsf_-_@benfinney.id.au> Alister via Python-list writes: > maybe [Ben's signatures are not sarcasm] - I use fortune to generate > mine & it can be supprisingly apt at times Yes. They are randomly chosen by my sig monster when I compose a message; I have some option to ask for another, but no direct input in the selection. When my sig monster pulls a particularly apt aphorism from my curated database, I occasionally end the message with praise for my sig monster and offer it a biscuit, as is customary (and whimsical, because the sig monster typically doesn't eat them and shows no sign that it alters its behaviour in response to the offer). But too often when I include such praise for my sig monster, people get confused and think I'm offering *them* a cookie, or calling them a monster, or some such. Explaining what's going on gets strange :-) -- \ ?Do unto others twenty-five percent better than you expect them | `\ to do unto you. (The twenty-five percent is [to correct] for | _o__) error.)? ?Linus Pauling's Golden Rule | Ben Finney From greg.ewing at canterbury.ac.nz Tue Mar 20 21:44:22 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Wed, 21 Mar 2018 14:44:22 +1300 Subject: Thank you Python community! In-Reply-To: <87a267c4-dd67-47c7-a9f5-2e9e71b3a8b5@googlegroups.com> References: <87a267c4-dd67-47c7-a9f5-2e9e71b3a8b5@googlegroups.com> Message-ID: John Ladasky wrote: > Yeah, it's either Python or that horrifying street drug PHP. I know which one I'm choosing. Python is definitely the best language for getting high on: https://xkcd.com/353/ -- Greg From larry.martell at gmail.com Tue Mar 20 22:20:16 2018 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 21 Mar 2018 02:20:16 +0000 Subject: Multiprocessing on a remote host Message-ID: Is there a way to use the multiprocessing lib to run a job on a remote host? From steve+comp.lang.python at pearwood.info Tue Mar 20 23:15:31 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 21 Mar 2018 03:15:31 +0000 (UTC) Subject: Multiprocessing on a remote host References: Message-ID: On Wed, 21 Mar 2018 02:20:16 +0000, Larry Martell wrote: > Is there a way to use the multiprocessing lib to run a job on a remote > host? Don't try to re-invent the wheel. This is a solved problem. https://stackoverflow.com/questions/1879971/what-is-the-current-choice-for-doing-rpc-in-python I've used both rpyc and pyro, they work fine and are easy to learn. -- Steve From steve+comp.lang.python at pearwood.info Wed Mar 21 01:14:30 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 21 Mar 2018 05:14:30 +0000 (UTC) Subject: Enumerating all 3-tuples References: <765fc7e7a1b8aeb716d251fd15cc2919@termina.org.uk> Message-ID: On Tue, 13 Mar 2018 23:56:37 +0100, Denis Kasak wrote: [...] > The triples can be viewed as a pair of a pair and a natural number: > > (1,1),1 (1,1),2 (1,1),3 ... > (2,1),1 (2,1),2 (2,1),3 ... > (1,2),1 (1,2),2 (1,2),3 ... [...] > This leads fairly naturally to the implementation. > > from itertools import accumulate, count > > def c(i): > """ > Inverse of the Cantor pairing function, mapping N ? N?N. """ > assert i >= 1 > > # partial sums of the series 1 + 2 + 3 + ... sums = > accumulate(count(1)) > n = 0 > > while True: > m = next(sums) > if m < i: > n += 1 > else: > r = m - i > break > > return r + 1, n - r + 1 > > > def c3(i): > """ > Inverse of the Cantor pairing function generalization to > triples, > mapping N ? N?N?N. > """ > n, m = c(i) > return c(n) + (m,) > > Applying c3 to the natural numbers gives the sequence you wanted: Thanks! That looks interesting, I haven't had a chance to play with it in a lot of detail yet, but it looks to me like every time you generate a triple, it starts counting up from 1. So iterating over c3(1), c3(2), c3(4), ... c3(something big) is going to have O(N**2) performance. It's like the old joke about the Polish painter: http://global.joelonsoftware.com/English/Articles/BacktoBasics.html Since that's exactly what I need to do, that might be a problem. On the other hand, I doubt I'll need to generate more than a few thousand of these, so it might be fast enough. I guess I have to run some benchmarks to find out. But however they turn out, I appreciate your time and code. Thanks heaps! -- Steve From robin at reportlab.com Wed Mar 21 05:22:01 2018 From: robin at reportlab.com (Robin Becker) Date: Wed, 21 Mar 2018 09:22:01 +0000 Subject: Enumerating all 3-tuples In-Reply-To: References: <765fc7e7a1b8aeb716d251fd15cc2919@termina.org.uk> Message-ID: On 21/03/2018 05:14, Steven D'Aprano wrote: > On Tue, 13 Mar 2018 23:56:37 +0100, Denis Kasak wrote: > > [...] >> The triples can be viewed as a pair of a pair and a natural number: >> >> (1,1),1 (1,1),2 (1,1),3 ... >> (2,1),1 (2,1),2 (2,1),3 ... >> (1,2),1 (1,2),2 (1,2),3 ... > > [...] >> This leads fairly naturally to the implementation. >> >> from itertools import accumulate, count >> >> def c(i): >> """ ........... > That looks interesting, I haven't had a chance to play with it in a lot > of detail yet, but it looks to me like every time you generate a triple, > it starts counting up from 1. > > So iterating over c3(1), c3(2), c3(4), ... c3(something big) is going to > have O(N**2) performance. It's like the old joke about the Polish painter: > This cantor inverse is much faster; it uses the integer sqrt from here http://code.activestate.com/recipes/577821-integer-square-root-function/ so far as I can tell it works for the cantor below def cantor(k1,k2): return (((k1+k2)*(k1+k2+1))>>1) + k2 def isqrt(x): if x < 0: raise ValueError('square root not defined for negative numbers') n = int(x) if n == 0: return 0 a, b = divmod(n.bit_length(), 2) x = 2**(a+b) while True: y = (x + n//x)//2 if y >= x: return x x = y def inverse_cantor(z): w = int((isqrt(8*z+1)-1)//2) t = (w*(w+1))>>1 j = z - t i = w - j return i, j but it doesn't agree with Denis' inverse I guess because this is defined on non-negative integers, but his functions are defined on positive integers. > http://global.joelonsoftware.com/English/Articles/BacktoBasics.html > > Since that's exactly what I need to do, that might be a problem. > > On the other hand, I doubt I'll need to generate more than a few thousand > of these, so it might be fast enough. I guess I have to run some > benchmarks to find out. > > But however they turn out, I appreciate your time and code. Thanks heaps! > > > -- Robin Becker From gurpreetsinghluky at gmail.com Wed Mar 21 06:15:30 2018 From: gurpreetsinghluky at gmail.com (gurpreetsinghluky at gmail.com) Date: Wed, 21 Mar 2018 03:15:30 -0700 (PDT) Subject: Want to convert the msg file in html file so that i can read the tables emebedded in msg file using python Message-ID: Please help me From asphjt at laposte.net Wed Mar 21 06:25:04 2018 From: asphjt at laposte.net (asphjt at laposte.net) Date: Wed, 21 Mar 2018 03:25:04 -0700 (PDT) Subject: lire du son en format natif avec python Message-ID: <084ecf71-fbdb-42a7-8b02-1ec9a9e1d39b@googlegroups.com> Bonjour ? tous, En utilisant le module pyaudio pour enregistrer du son, j'ai une chaine de caract?res de la forme b'\x01\x00\n\x00\x04\x00\xfe\xff\x04\x00\x0b\x00\n\x00\x07\x00'b'\x01\x00\xff\xff\x00\x00\xff\xff\x01\x00\n\x00\n\x00\n\x00 qui correspond aux valeurs hexad?cimales (je pense) du son brut, et j'aimerai pouvoir le lire sans passer par un encodage ou format de compression comme mp3, wav,... J'ai lu que l'on pouvait le faire avec ossaudiodev, mais je n'arrive pas ? le trouver, ni ? l'installer depuis pyzo. Si quelqu'un peut m'aider, merci ? lui d'avance. From sankarramanv at gmail.com Wed Mar 21 06:44:48 2018 From: sankarramanv at gmail.com (sankarramanv at gmail.com) Date: Wed, 21 Mar 2018 03:44:48 -0700 (PDT) Subject: Reg python regexp Message-ID: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> Hi, I have a requirement. cmd="cat |grep -c 'if [ -t 1 ]; then mesg n 2>/dev/null; fi'" I need to escape only the square brackets in above variable since its not grepping without escaping the brackets. Please help. Thanks. From vincent.vande.vyvre at telenet.be Wed Mar 21 06:53:13 2018 From: vincent.vande.vyvre at telenet.be (Vincent Vande Vyvre) Date: Wed, 21 Mar 2018 11:53:13 +0100 Subject: lire du son en format natif avec python In-Reply-To: <084ecf71-fbdb-42a7-8b02-1ec9a9e1d39b@googlegroups.com> References: <084ecf71-fbdb-42a7-8b02-1ec9a9e1d39b@googlegroups.com> Message-ID: <25d3bb89-ad89-03b5-7e0a-cd8326b4ffbd@telenet.be> Le 21/03/18 ? 11:25, asphjt--- via Python-list a ?crit?: > Bonjour ? tous, > En utilisant le module pyaudio pour enregistrer du son, j'ai une chaine de caract?res de la forme b'\x01\x00\n\x00\x04\x00\xfe\xff\x04\x00\x0b\x00\n\x00\x07\x00'b'\x01\x00\xff\xff\x00\x00\xff\xff\x01\x00\n\x00\n\x00\n\x00 qui correspond aux valeurs hexad?cimales (je pense) du son brut, et j'aimerai pouvoir le lire sans passer par un encodage ou format de compression comme mp3, wav,... > J'ai lu que l'on pouvait le faire avec ossaudiodev, mais je n'arrive pas ? le trouver, ni ? l'installer depuis pyzo. > Si quelqu'un peut m'aider, merci ? lui d'avance. Hi, This is an English mailing list. If you prefer talk in French, I recommend you this forum: https://www.developpez.net/forums/f96/autres-langages/python-zope/ Best Vincent From rhodri at kynesim.co.uk Wed Mar 21 06:55:49 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 21 Mar 2018 10:55:49 +0000 Subject: Reg python regexp In-Reply-To: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> References: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> Message-ID: <73449c4d-561f-da08-6fdd-9bd93f82e0ee@kynesim.co.uk> On 21/03/18 10:44, sankarramanv at gmail.com wrote: > Hi, > > I have a requirement. > > cmd="cat |grep -c 'if [ -t 1 ]; then mesg n 2>/dev/null; fi'" > > I need to escape only the square brackets in above variable since its not grepping without escaping the brackets. You need to escape the square brackets as you normally would for your shell, with backslashes I presume. Then you need to escape the backslashes so they aren't interpreted specially by Python, with more backslashes. -- Rhodri James *-* Kynesim Ltd From p.f.moore at gmail.com Wed Mar 21 06:56:06 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 21 Mar 2018 10:56:06 +0000 Subject: Reg python regexp In-Reply-To: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> References: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> Message-ID: Hi, You don't need a regexp for this, the "replace" method on a string will do what you want: >>> s = 'this is a [string' >>> print(s.replace('[', '\\[')) this is a \[string Paul On 21 March 2018 at 10:44, wrote: > Hi, > > I have a requirement. > > cmd="cat |grep -c 'if [ -t 1 ]; then mesg n 2>/dev/null; fi'" > > I need to escape only the square brackets in above variable since its not grepping without escaping the brackets. > > Please help. > > Thanks. > -- > https://mail.python.org/mailman/listinfo/python-list From rosuav at gmail.com Wed Mar 21 06:56:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 21 Mar 2018 21:56:43 +1100 Subject: Reg python regexp In-Reply-To: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> References: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> Message-ID: On Wed, Mar 21, 2018 at 9:44 PM, wrote: > Hi, > > I have a requirement. > > cmd="cat |grep -c 'if [ -t 1 ]; then mesg n 2>/dev/null; fi'" > > I need to escape only the square brackets in above variable since its not grepping without escaping the brackets. > > Please help. You're putting this into a Python script. Why not use Python to search the file instead of grep? That'd also eliminate the superfluous "cat file |" at the start. Python is not a shell language. You don't have to, and shouldn't, write everything by invoking other programs. ChrisA From steve+comp.lang.python at pearwood.info Wed Mar 21 07:05:36 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Wed, 21 Mar 2018 11:05:36 +0000 (UTC) Subject: Want to convert the msg file in html file so that i can read the tables emebedded in msg file using python References: Message-ID: On Wed, 21 Mar 2018 03:15:30 -0700, gurpreetsinghluky wrote: > Please help me We'd love to help if only we knew what you wanted. Can you explain what you want? Give an example of the data you are working with and the results you expect? -- Steve From tjol at tjol.eu Wed Mar 21 08:00:37 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Wed, 21 Mar 2018 13:00:37 +0100 Subject: lire du son en format natif avec python In-Reply-To: <084ecf71-fbdb-42a7-8b02-1ec9a9e1d39b@googlegroups.com> References: <084ecf71-fbdb-42a7-8b02-1ec9a9e1d39b@googlegroups.com> Message-ID: <6e911cd2-541e-1abd-cd4f-e6447f167c69@tjol.eu> On 2018-03-21 11:25, asphjt--- via Python-list wrote: > Bonjour ? tous, > En utilisant le module pyaudio pour enregistrer du son, j'ai une chaine de caract?res de la forme b'\x01\x00\n\x00\x04\x00\xfe\xff\x04\x00\x0b\x00\n\x00\x07\x00'b'\x01\x00\xff\xff\x00\x00\xff\xff\x01\x00\n\x00\n\x00\n\x00 qui correspond aux valeurs hexad?cimales (je pense) du son brut, et j'aimerai pouvoir le lire sans passer par un encodage ou format de compression comme mp3, wav,... > J'ai lu que l'on pouvait le faire avec ossaudiodev, mais je n'arrive pas ? le trouver, ni ? l'installer depuis pyzo. > Si quelqu'un peut m'aider, merci ? lui d'avance. > Bonjour, dans cette groupe nous ne parlons qu'anglais. Il y a des forums Pythons francophones [1], par exemples les forums des l'association francophone Python (afpy) [2]. Ici, ?crivez en anglais la prochaine fois s.v.p. [1] https://wiki.python.org/moin/FrenchLanguage [2] https://www.afpy.org/forums Au sujet de votre question: vous pouvez aussi lire et passer ce que vous avez enregistr? avec PyAudio. Je pense qu'il faut le passer directement ? la m?thode write() du 'stream' PyAudio. -- Thomas From larry.martell at gmail.com Wed Mar 21 09:27:37 2018 From: larry.martell at gmail.com (Larry Martell) Date: Wed, 21 Mar 2018 09:27:37 -0400 Subject: Multiprocessing on a remote host In-Reply-To: References: Message-ID: On Tue, Mar 20, 2018 at 11:15 PM, Steven D'Aprano wrote: > On Wed, 21 Mar 2018 02:20:16 +0000, Larry Martell wrote: > >> Is there a way to use the multiprocessing lib to run a job on a remote >> host? > > Don't try to re-invent the wheel. This is a solved problem. > > https://stackoverflow.com/questions/1879971/what-is-the-current-choice-for-doing-rpc-in-python > > I've used both rpyc and pyro, they work fine and are easy to learn. Yeah, I saw that and I wasn't trying to reinvent the wheel. On this page https://docs.python.org/2/library/multiprocessing.html it says this: The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads. Due to this, the multiprocessing module allows the programmer to fully leverage multiple processors on a given machine. I took 'remote concurrency' to mean I could use it run a process on another host. But I don't see how to do that, and I was asking if it was possible or am I misinterpreting the docs. From ganesh1pal at gmail.com Wed Mar 21 09:35:37 2018 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Wed, 21 Mar 2018 13:35:37 +0000 Subject: how do I retry a command only for a specific exception / error In-Reply-To: References: Message-ID: Please ensure quoted text is quoted, and new text you write is unquoted. > That way you are more likely to get useful > Sorry , Steve I didn't realize but thanks for pointing out I will take care I was on a mobile phone and messed the quoted text >Something like this should do it. It gives up immediately on fatal >errors >and tries again on temporary ones. (You have to specify what you >consider >fatal or temporary, of course. This is a good suggestion , I like the way the code is written , but what I have failed to understand is how to translate the possible TemporaryFailureErrors to a different exception class/type and retry . In my case , every command is executed using a run() function that calls out to subprocess.Popen(). Which will return stdout, stderr, exit_code and we would need to retry only for a specific TemporaryFailureError . Example : Say , If we are not able to SSH to the host , and I get ?connection refused? error I would want to retry only for this specific case From youta.t at gmail.com Wed Mar 21 09:49:37 2018 From: youta.t at gmail.com (Youta TAKAOKA) Date: Wed, 21 Mar 2018 13:49:37 +0000 Subject: Reg python regexp In-Reply-To: <73449c4d-561f-da08-6fdd-9bd93f82e0ee@kynesim.co.uk> References: <47884682-a208-414f-99ae-97f762834962@googlegroups.com> <73449c4d-561f-da08-6fdd-9bd93f82e0ee@kynesim.co.uk> Message-ID: sankarramanv, It seems for me that this task does not need both python AND shell. Only python does it, as well as only shell. Of course, there can be some restrictions let you use both. (the real world is filled up with such troublesome matters !) If you *really* need to use `lgrep`, try `-f` option. `lgrep -f` uses pattern as just a fixed text, not regexp. 2018?3?21?(?) 20:35 Rhodri James : > On 21/03/18 10:44, sankarramanv at gmail.com wrote: > > Hi, > > > > I have a requirement. > > > > cmd="cat |grep -c 'if [ -t 1 ]; then mesg n 2>/dev/null; fi'" > > > > I need to escape only the square brackets in above variable since its > not grepping without escaping the brackets. > > You need to escape the square brackets as you normally would for your > shell, with backslashes I presume. Then you need to escape the > backslashes so they aren't interpreted specially by Python, with more > backslashes. > > -- > Rhodri James *-* Kynesim Ltd > -- > https://mail.python.org/mailman/listinfo/python-list > From fabiofz at gmail.com Wed Mar 21 13:52:06 2018 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Wed, 21 Mar 2018 14:52:06 -0300 Subject: PyDev 6.3.2 released Message-ID: PyDev 6.3.2 Release Highlights PyDev changes: - Type inference - PyDev can now uses information on .pyi files (when along the typed .py file) for type inference. - Fixed issue opening code completion preferences page. About PyDev PyDev is an open-source Python IDE on top of Eclipse for Python, Jython and IronPython development, now also available for Python on Visual Studio Code. It comes with goodies such as code completion, syntax highlighting, syntax analysis, code analysis, refactor, debug, interactive console, etc. It is also available as a standalone through LiClipse with goodies such as multiple cursors, theming and support for many other languages, such as Django Templates, Jinja2, Html, JavaScript, etc. Links: PyDev: http://pydev.org PyDev Blog: http://pydev.blogspot.com PyDev on VSCode: http://pydev.org/vscode LiClipse: http://www.liclipse.com PyVmMonitor - Python Profiler: http://www.pyvmmonitor.com/ Cheers, Fabio Zadrozny From jaqo59 at gmail.com Wed Mar 21 18:59:27 2018 From: jaqo59 at gmail.com (Jacques Bikoundou) Date: Wed, 21 Mar 2018 18:59:27 -0400 Subject: Modules Message-ID: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> Hi,?I am writing because I use a Python 3 kernel on my notebook that I access through the Microsoft Azure ML. I experience error messages when using modules such as 'speedml' or 'xgboost'. The error messagessay: no such module. How can I solve this?Thanks,?Jacques? Sent from Samsung tablet From rantingrickjohnson at gmail.com Wed Mar 21 19:50:07 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 21 Mar 2018 16:50:07 -0700 (PDT) Subject: Modules In-Reply-To: References: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> Message-ID: Hmm, let's try a little interactive session, shall we? Did your error message look something like this? >>> import spam Traceback (most recent call last): File "", line 1, in import spam ImportError: No module named spam >>> import eggs Traceback (most recent call last): File "", line 1, in import eggs ImportError: No module named eggs >>> from green.eggs import ham Traceback (most recent call last): File "", line 1, in from green.eggs import ham ImportError: No module named green.eggs Hmm. It seems that python cannot find a module, then there is no way to import it. What to do? Ah ha! We check the search path! >>> import sys >>> sys.path [...List of directories...] Q: Are your modules inside any of the listed directories? From rantingrickjohnson at gmail.com Wed Mar 21 20:03:20 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 21 Mar 2018 17:03:20 -0700 (PDT) Subject: Want to convert the msg file in html file so that i can read the tables emebedded in msg file using python In-Reply-To: References: Message-ID: <6ce1ea5a-7ef3-4491-97bc-c4e50efe0b7d@googlegroups.com> On Wednesday, March 21, 2018 at 5:15:47 AM UTC-5, gurpreet... at gmail.com wrote: > TITLE: "Want to convert the msg file in html file so that i > can read the tables emebedded in msg file using python" My guess that the OP meant to say: "msg file *INTO* html file" -- where "msg file" is a file holding email or text message data in MBOX format, and the OP wants to display this data using HTML table[s]. But that's only a guess. *scratches head* > Please help me Ditto! From jaqo59 at gmail.com Wed Mar 21 20:49:07 2018 From: jaqo59 at gmail.com (Jacques Bikoundou) Date: Thu, 22 Mar 2018 00:49:07 +0000 Subject: Modules In-Reply-To: References: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> Message-ID: It said: ImportError: no module named 'speedml' On Wed, Mar 21, 2018, 19:57 Rick Johnson wrote: > Hmm, let's try a little interactive session, shall we? Did > your error message look something like this? > > >>> import spam > > Traceback (most recent call last): > File "", line 1, in > import spam > ImportError: No module named spam > > >>> import eggs > > Traceback (most recent call last): > File "", line 1, in > import eggs > ImportError: No module named eggs > > >>> from green.eggs import ham > > Traceback (most recent call last): > File "", line 1, in > from green.eggs import ham > ImportError: No module named green.eggs > > Hmm. It seems that python cannot find a module, then there > is no way to import it. What to do? Ah ha! We check the > search path! > > >>> import sys > >>> sys.path > [...List of directories...] > > Q: Are your modules inside any of the listed directories? > -- > https://mail.python.org/mailman/listinfo/python-list > From rantingrickjohnson at gmail.com Wed Mar 21 22:04:39 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Wed, 21 Mar 2018 19:04:39 -0700 (PDT) Subject: Modules In-Reply-To: References: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> Message-ID: On Wednesday, March 21, 2018 at 7:49:42 PM UTC-5, Jacques Bikoundou wrote: > It said: ImportError: no module named 'speedml' I see. And did you check the search path[1] to ensure that the modules you want to import are indeed located in a directory which python normally searches? As an academic excercise, and to better understand the the Python import mechanism, let us imagine that Python is house keeping robot. And we shall dub her "Rosie5000". And you, being the master programmer of "Rosie5000, the housekeeping robot", you have (wisely) programed Rosie to search in specific locations of your home for items that you request (aka: "import"). And being that you knew ahead of time that you would be asking for food and drinks on a regular basis, the first place you programmed Rosie to search is in the kitchen (aka: "stdlib"). But then one day, an old college roomie drops by for an unexpected visit, and of course, his car is running low on oil (big surprise, right?). So you ask Rosie to fetch a can of oil: "import motoroil" you say, and immediately -- being that Rosie is a mindless bucket of bolts; and being that she has only been programmed to search in the kitchen; and simultaneously, being, that she is oblivious to the fact that kitchens are hardly ever the proper storage location for cans of nasty motor oil -- Rosie goes rummaging through the cupboards, systematically searching for a can of "dead-dino-goo", only to come back empty handed ("Bad Rosie!" you say, "Bad!"). But then compassion smacks you, and you say: "Oops, i forgot to program Rosie to search in the garage, my Bad!" Yes, "You bad" is correct! The moral of our little story is that Python, much like your beloved Rosie5000, will only look where you tell it to look. Now, the Python gods have been kind enough to preprogram some default search locations, but not all scripts are automatically installed in these default locations (nor should they be!). So if your script exists *OUTSIDE* of the Python search path, you have two options (and possibly more) to choose from: (1) Move the script into a directory that is _already_ included in the search path (see: sys.path) (2) Add the script's containing directory to the Python search path manually (but don't do it by appending to sys.path, that's bad juju!) The first case is just a matter of cutting and pasting the script in your file browser (aka: points and clicks). But the second is a little more complicated and can be achieved in many ways, some of which are OS specific (requiring voodoo on the command line!). See the following links for more detail (not exhaustive). (NOTE: These links point to Python2.x documentation, but if you're using a different version, be sure to choose your version from the little drop- down box at the top of the page so that you see the most relevent info) https://docs.python.org/2/library/sys.html#sys.path https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH https://docs.python.org/2/library/site.html?highlight=pth -- [1] import sys; print(sys.path) From python at mrabarnett.plus.com Wed Mar 21 22:20:54 2018 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 22 Mar 2018 02:20:54 +0000 Subject: Modules In-Reply-To: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> References: <5ab2e3cf.54bbca0a.fa121.1d54@mx.google.com> Message-ID: On 2018-03-21 22:59, Jacques Bikoundou wrote: > > Hi,?I am writing because I use a Python 3 kernel on my notebook that I access through the Microsoft Azure ML. I experience error messages when using modules such as 'speedml' or 'xgboost'. The error messagessay: no such module. How can I solve this?Thanks,?Jacques > > Sent from Samsung tablet > A silly question, but have you installed those modules? From rustompmody at gmail.com Thu Mar 22 00:12:55 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Wed, 21 Mar 2018 21:12:55 -0700 (PDT) Subject: Style Q: Instance variables defined outside of __init__ In-Reply-To: <87h8pauw8s.fsf@nightsong.com> References: <9D1AA5F1-1366-43D9-B3C2-A3DC01ED6AE4@furrypants.com> <926sC.19683$lm7.13822@fx12.am4> <85vadqz9e6.fsf@benfinney.id.au> <87h8pauw8s.fsf@nightsong.com> Message-ID: <11134ce0-09af-4cb6-bc83-7d5c6a73bf06@googlegroups.com> On Wednesday, March 21, 2018 at 5:37:48 AM UTC+5:30, Paul Rubin wrote: > Ben Finney writes: > > Any program which needs to interact with systems outside itself ? which > > is to say, any program which performs useful work, ultimately ? must > > have side effects. So it's absurd to advocate removing *all* side > > effects. > > The way it (conceptually) works in Haskell is you write a purely > functional program that returns a pure value of type "I/O action". Saw this coincidentally adjacent to this thread https://youtu.be/ROor6_NGIWU Its a view 'from the other side': a big name in functional programming (not haskell) musing on how the world of IO, more generally world interaction, could be cleaned up From stefan_ml at behnel.de Thu Mar 22 02:17:27 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 22 Mar 2018 07:17:27 +0100 Subject: Writing a C extension - borrowed references In-Reply-To: References: Message-ID: Tom Evans via Python-list schrieb am 20.03.2018 um 18:03: > On Tue, Mar 20, 2018 at 4:38 PM, Chris Angelico wrote: >> BTW, have you looked into Cython? It's smart enough to take care of a >> lot of this sort of thing for you. > > I did a bit; this work is to replace our old python 2 SAML client, > which used python-lasso and python-libxml2, both packages that are > built as part of building the C library and thus an utter PITA to > package for different versions of python (something our Infra team > were extremely reluctant to do). When the latest (PITA to build) > version of python-lasso started interfering with python-xmlsec > (validating an invalid signature was causing segfaults), I got fed up > of fighting it. If rewriting is an option (just mentioning it ;) ), something like this would probably be based on lxml these days. It also comes with a C/Cython-level API, in case you need to a) do some kind of low-level tree processing or b) integrate with some external library. > I actually also maintain a C version of the same code, using the same > libraries, so porting those few segments of code to Python/C seemed > more expedient than rewriting in Cython. I'm not writing an API to > these libraries, just a few functions. Happy if that works for you, but let me still add a quick note that, from my experience, rewriting C-API code in Cython tends to be a surprisingly quick thing to do (it's basically just reverse engineering the Python code from the C-API code), but it's a one-time investment that can reduce the long-term maintenance cost a lot, often improves the performance, and usually also leads to nicer Python APIs, as what you get in the end is more obvious from the code, so you can focus on the handling more easily. Stefan From damjanstojanovski454 at gmail.com Thu Mar 22 14:30:08 2018 From: damjanstojanovski454 at gmail.com (Damjan Stojanovski) Date: Thu, 22 Mar 2018 11:30:08 -0700 (PDT) Subject: I keep getting this error message when i try to run a program in Python Message-ID: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> Dear all I am totally new in learning Python and i am learning from a beginner's book called automate the boring stuff with Python and i downloaded the Python 64 bit version for Windows and i started writing some examples from the book. At the end whenever i try to run the program i keep getting an error message and it shows that the error is in the beginning in the version of Python. (See a pic bellow) Can someone help me and tell me what to do and why is this error showing up??? Thanks Dan From damjanstojanovski454 at gmail.com Thu Mar 22 14:37:00 2018 From: damjanstojanovski454 at gmail.com (Damjan Stojanovski) Date: Thu, 22 Mar 2018 11:37:00 -0700 (PDT) Subject: I keep getting this error message when i try to run a program in Python In-Reply-To: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> References: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> Message-ID: i am sorry but i can not find a way to attach an image here so you can see what i mean. Please someone tell me how to add an image here. Thanks Dan From larry.martell at gmail.com Thu Mar 22 14:44:38 2018 From: larry.martell at gmail.com (Larry Martell) Date: Thu, 22 Mar 2018 14:44:38 -0400 Subject: I keep getting this error message when i try to run a program in Python In-Reply-To: References: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> Message-ID: On Thu, Mar 22, 2018 at 2:37 PM, Damjan Stojanovski wrote: > i am sorry but i can not find a way to attach an image here so you can see what i mean. Please someone tell me how to add an image here. You can't. Copy/paste the error. And please include the post you are replying to. From tjreedy at udel.edu Thu Mar 22 14:46:10 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 22 Mar 2018 14:46:10 -0400 Subject: I keep getting this error message when i try to run a program in Python In-Reply-To: References: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> Message-ID: On 3/22/2018 2:37 PM, Damjan Stojanovski wrote: > i am sorry but i can not find a way to attach an image here so you can see what i mean. Please someone tell me how to add an image here. You cannot and there is no need. List is text only, no attachments. Reduce code to the minimum needed to get error. Copy and paste both code and trackback. -- Terry Jan Reedy From tjol at tjol.eu Thu Mar 22 14:47:39 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Thu, 22 Mar 2018 19:47:39 +0100 Subject: I keep getting this error message when i try to run a program in Python In-Reply-To: References: <2e8b5c1d-da14-4013-9d46-9e05035c369c@googlegroups.com> Message-ID: <33b53fe7-fdb6-3775-edbb-bdf7ab91e0e2@tjol.eu> On 2018-03-22 19:37, Damjan Stojanovski wrote: > i am sorry but i can not find a way to attach an image here so you can see what i mean. Please someone tell me how to add an image here. > This is a text only list. From toby at tobiah.org Thu Mar 22 15:46:26 2018 From: toby at tobiah.org (Tobiah) Date: Thu, 22 Mar 2018 12:46:26 -0700 Subject: Putting Unicode characters in JSON Message-ID: I have some mailing information in a Mysql database that has characters from various other countries. The table says that it's using latin-1 encoding. I want to send this data out as JSON. So I'm just taking each datum and doing 'name'.decode('latin-1') and adding the resulting Unicode value right into my JSON structure before doing .dumps() on it. This seems to work, and I can consume the JSON with another program and when I print values, they look nice with the special characters and all. I was reading though, that JSON files must be encoded with UTF-8. So should I be doing string.decode('latin-1').encode('utf-8')? Or does the json module do that for me when I give it a unicode object? Thanks From rosuav at gmail.com Thu Mar 22 16:09:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 07:09:50 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 6:46 AM, Tobiah wrote: > I have some mailing information in a Mysql database that has > characters from various other countries. The table says that > it's using latin-1 encoding. I want to send this data out > as JSON. > > So I'm just taking each datum and doing 'name'.decode('latin-1') > and adding the resulting Unicode value right into my JSON structure > before doing .dumps() on it. This seems to work, and I can consume > the JSON with another program and when I print values, they look nice > with the special characters and all. > > I was reading though, that JSON files must be encoded with UTF-8. So > should I be doing string.decode('latin-1').encode('utf-8')? Or does > the json module do that for me when I give it a unicode object? Reconfigure your MySQL database to use UTF-8. There is no reason to use Latin-1 in the database. If that isn't an option, make sure your JSON files are pure ASCII, which is the common subset of UTF-8 and Latin-1. ChrisA From toby at tobiah.org Thu Mar 22 16:15:34 2018 From: toby at tobiah.org (Tobiah) Date: Thu, 22 Mar 2018 13:15:34 -0700 Subject: Putting Unicode characters in JSON References: Message-ID: On 03/22/2018 01:09 PM, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 6:46 AM, Tobiah wrote: >> I have some mailing information in a Mysql database that has >> characters from various other countries. The table says that >> it's using latin-1 encoding. I want to send this data out >> as JSON. >> >> So I'm just taking each datum and doing 'name'.decode('latin-1') >> and adding the resulting Unicode value right into my JSON structure >> before doing .dumps() on it. This seems to work, and I can consume >> the JSON with another program and when I print values, they look nice >> with the special characters and all. >> >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do that for me when I give it a unicode object? > > Reconfigure your MySQL database to use UTF-8. There is no reason to > use Latin-1 in the database. > > If that isn't an option, make sure your JSON files are pure ASCII, > which is the common subset of UTF-8 and Latin-1. > > ChrisA > It works the way I'm doing it. I checked and it turns out that whether I do datum.decode('latin-1') or datum.decode('latin-1').encode('utf8') I get identical JSON files after .dumps(). From saxri89 at gmail.com Thu Mar 22 16:38:14 2018 From: saxri89 at gmail.com (Xristos Xristoou) Date: Thu, 22 Mar 2018 13:38:14 -0700 (PDT) Subject: PyQt4 QWebView cant load google maps markers Message-ID: <8f05e680-8a1e-47a2-b39a-6b789f30e189@googlegroups.com> I want to create a simple python app using pyqt,QWebView and google maps with markers. The problem is that,the markers does not load inside the QWebView, as you can see they load just fine in the browser. here the simple code : import sys from PyQt4.QtCore import * from PyQt4.QtGui import * from PyQt4.QtWebKit import * import os app = QApplication(sys.argv) web_view= dialog.findChild(QWebView,"webView") google='https://www.google.gr/maps/place/Granite+Mountain+Hotshots+Memorial+State+Park/@34.1749572,-112.850308,12.78z/data=!4m13!1m7!3m6!1s0x872b08ebcb4c186b:0x423927b17fc1cd71!2zzpHPgc65zrbPjM69zrEsIM6Xzr3Pic68zq3Ovc61z4IgzqDOv867zrnPhM61zq_Otc-C!3b1!8m2!3d34.0489281!4d-111.0937311!3m4!1s0x80d330577faee965:0x66d75aef24890ae1!8m2!3d34.2032843!4d-112.7746582?hl=el' web_view2.load(QUrl(google)) web_view2.show() sys.exit(app.exec_()) error message : right click and select Inspect, then go to console and observe the messages console I take this error : TypeError: 'undefined' is not a function (evaluating 'this.D.bind(this)') marker.js:58 Any idea how to can fix that ? I need to add more settings to QWebView to work google maps with markers ? From steve+comp.lang.python at pearwood.info Thu Mar 22 19:47:57 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Thu, 22 Mar 2018 23:47:57 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do that for me when I give it a unicode object? > > Reconfigure your MySQL database to use UTF-8. There is no reason to use > Latin-1 in the database. You don't know that. You don't know what technical, compatibility, policy or historical constraints are on the database. > If that isn't an option, make sure your JSON files are pure ASCII, which > is the common subset of UTF-8 and Latin-1. And that's utterly unnecessary, since any character which can be stored in the Latin-1 MySQL database can be stored in the Unicode JSON. -- Steven From rosuav at gmail.com Thu Mar 22 20:08:56 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 11:08:56 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: > >>> I was reading though, that JSON files must be encoded with UTF-8. So >>> should I be doing string.decode('latin-1').encode('utf-8')? Or does >>> the json module do that for me when I give it a unicode object? >> >> Reconfigure your MySQL database to use UTF-8. There is no reason to use >> Latin-1 in the database. > > You don't know that. You don't know what technical, compatibility, policy > or historical constraints are on the database. Okay. Give me a good reason for the database itself to be locked to Latin-1. Make sure you explain how potentially saving the occasional byte of storage (compared to UTF-8) justifies limiting the available character set to the ones that happen to be in Latin-1, yet it's essential to NOT limit the character set to ASCII. >> If that isn't an option, make sure your JSON files are pure ASCII, which >> is the common subset of UTF-8 and Latin-1. > > And that's utterly unnecessary, since any character which can be stored > in the Latin-1 MySQL database can be stored in the Unicode JSON. > Irrelevant; if you fetch eight-bit data out of the database, it isn't going to be a valid JSON file unless (1) it's really ASCII, like I suggest; (2) you re-encode it to UTF-8; or (3) it was actually UTF-8 all along, despite being declared as Latin-1. Restricting JSON to ASCII is a very easy and common thing to do. It just means that every non-ASCII character gets represented as a \u escape sequence. In Python's JSON encoder, that's the ensure_ascii parameter. Utterly unnecessary? How about standards-compliant and entirely effective, unlike the re-encoding that means that the database-stored blob is invalid JSON and must be re-encoded again on the way out? ChrisA From ben+python at benfinney.id.au Thu Mar 22 20:25:10 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 23 Mar 2018 11:25:10 +1100 Subject: Putting Unicode characters in JSON References: Message-ID: <857eq3zli1.fsf@benfinney.id.au> Chris Angelico writes: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: > > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: > >> Reconfigure your MySQL database to use UTF-8. There is no reason to > >> use Latin-1 in the database. > > > > You don't know that. You don't know what technical, compatibility, > > policy or historical constraints are on the database. > > Okay. Give me a good reason for the database itself to be locked to > Latin-1. That's a loaded question. If the database maintainers have a technical, compatibility, policy, or historical reason that constrains an existing database to a specific character set, then that is a good reason for them. Is it a good reason to create a new, independent database using Latin-1 character set? No. But that's implicit in the fact it's an existing constraint, because it's an existing database. Is it a reason likely to convince you? Or convince me? No. But that's irrelevant because they don't have any need to convince us of the goodness of their reason to keep that encoding :-) -- \ ?Instead of having ?answers? on a math test, they should just | `\ call them ?impressions?, and if you got a different | _o__) ?impression?, so what, can't we all be brothers?? ?Jack Handey | Ben Finney From tjol at tjol.eu Thu Mar 22 20:27:46 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Fri, 23 Mar 2018 01:27:46 +0100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: <0e263b68-138f-98b8-4302-dce0605ec4ce@tjol.eu> On 22/03/18 20:46, Tobiah wrote: > I was reading though, that JSON files must be encoded with UTF-8.? So > should I be doing string.decode('latin-1').encode('utf-8')?? Or does > the json module do that for me when I give it a unicode object? Definitely not. In fact, that won't even work. >>> import json >>> s = 'd?j? vu'.encode('latin1') >>> s b'd\xe9j\xe0 vu' >>> json.dumps(s.decode('latin1').encode('utf8')) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/lib/python3.6/json/encoder.py", line 199, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode return _iterencode(o, 0) File "/usr/lib/python3.6/json/encoder.py", line 180, in default o.__class__.__name__) TypeError: Object of type 'bytes' is not JSON serializable >>> You should make sure that either the file you're writing to is opened as UTF-8 text, or the ensure_ascii parameter of dumps() or dump() is set to True (the default) ? and then write the data in ASCII or any ASCII-compatible encoding (e.g. UTF-8). Basically, the default behaviour of the json module means you don't really have to worry about encodings at all once your original data is in unicode strings. -- Thomas From rosuav at gmail.com Thu Mar 22 20:33:25 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 11:33:25 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: <857eq3zli1.fsf@benfinney.id.au> References: <857eq3zli1.fsf@benfinney.id.au> Message-ID: On Fri, Mar 23, 2018 at 11:25 AM, Ben Finney wrote: > Chris Angelico writes: > >> On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano >> wrote: >> > On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >> Reconfigure your MySQL database to use UTF-8. There is no reason to >> >> use Latin-1 in the database. >> > >> > You don't know that. You don't know what technical, compatibility, >> > policy or historical constraints are on the database. >> >> Okay. Give me a good reason for the database itself to be locked to >> Latin-1. > > That's a loaded question. If the database maintainers have a technical, > compatibility, policy, or historical reason that constrains an existing > database to a specific character set, then that is a good reason for > them. > > Is it a good reason to create a new, independent database using Latin-1 > character set? No. But that's implicit in the fact it's an existing > constraint, because it's an existing database. > > Is it a reason likely to convince you? Or convince me? No. But that's > irrelevant because they don't have any need to convince us of the > goodness of their reason to keep that encoding :-) I happen to know that many deployments of MySQL have Latin-1 as the default character set. Is *that* a good enough reason for you? Because it isn't for me. There is NOT always a good reason for a suboptimal configuration. Thus I stand by my recommendation to reconfigure the database as first option, with other options *following* it in my original email. ChrisA From ben+python at benfinney.id.au Thu Mar 22 20:39:21 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 23 Mar 2018 11:39:21 +1100 Subject: Putting Unicode characters in JSON References: <857eq3zli1.fsf@benfinney.id.au> Message-ID: <85370rzkue.fsf@benfinney.id.au> Chris Angelico writes: > There is NOT always a good reason for a suboptimal configuration. True. Did anyone claim otherwise? What I saw Steven responding to was your claim that there is *never* a good reason to do it. To refute that, it's sufficient to show that good reason can exist in some cases. That's entirely compatible with a good reason not existing in other cases. -- \ ?Education is learning what you didn't even know you didn't | `\ know.? ?Daniel J. Boorstin, historian, 1914?2004 | _o__) | Ben Finney From steve+comp.lang.python at pearwood.info Thu Mar 22 20:39:37 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 23 Mar 2018 00:39:37 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Fri, 23 Mar 2018 11:08:56 +1100, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: >> On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >>>> I was reading though, that JSON files must be encoded with UTF-8. So >>>> should I be doing string.decode('latin-1').encode('utf-8')? Or does >>>> the json module do that for me when I give it a unicode object? >>> >>> Reconfigure your MySQL database to use UTF-8. There is no reason to >>> use Latin-1 in the database. >> >> You don't know that. You don't know what technical, compatibility, >> policy or historical constraints are on the database. > > Okay. Give me a good reason for the database itself to be locked to > Latin-1. Make sure you explain how potentially saving the occasional > byte of storage (compared to UTF-8) justifies limiting the available > character set to the ones that happen to be in Latin-1, yet it's > essential to NOT limit the character set to ASCII. I'll better than that, I'll give multiple good reasons to use Latin-1. It's company policy to only use Latin-1, because the CEO was once employed by the Unicode Consortium, and fired in disgrace after embezzling funds, and ever since then he has refused to use Unicode. Compatibility with other databases, systems or tools that require Latin-1. The database has to send information to embedded devices that don't include a full Unicode implementation, but do support Latin-1. The data doesn't actually represent text, but Python 2 style byte- strings, and Latin-1 is just a convenient, easy way to get that that ensures ASCII bytes look like ASCII characters. >>> If that isn't an option, make sure your JSON files are pure ASCII, >>> which is the common subset of UTF-8 and Latin-1. >> >> And that's utterly unnecessary, since any character which can be stored >> in the Latin-1 MySQL database can be stored in the Unicode JSON. >> >> > Irrelevant; if you fetch eight-bit data out of the database, it isn't > going to be a valid JSON file unless (1) it's really ASCII, like I > suggest; (2) you re-encode it to UTF-8; or (3) it was actually UTF-8 all > along, despite being declared as Latin-1. As Tobiah pointed out in his question, he's fetching the data from the database, calling decode('latin-1'), and placing the resulting Unicode string into the JSON. There's no need to explicitly encode the Unicode string to a UTF-8 byte string, in fact it is the wrong thing to do since JSON doesn't support it: # The right way is to use a Unicode string. py> json.dumps("Hello ?") '"Hello \\u00fc"' # The wrong way is to encode to UTF-8 first. py> json.dumps("Hello ?".encode('utf-8')) Traceback (most recent call last): ... TypeError: b'Hello \xc3\xbc' is not JSON serializable (Results in Python 3 -- Python 2 may be doing something shonky.) -- Steve From rosuav at gmail.com Thu Mar 22 20:44:25 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 11:44:25 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: <85370rzkue.fsf@benfinney.id.au> References: <857eq3zli1.fsf@benfinney.id.au> <85370rzkue.fsf@benfinney.id.au> Message-ID: On Fri, Mar 23, 2018 at 11:39 AM, Ben Finney wrote: > Chris Angelico writes: > >> There is NOT always a good reason for a suboptimal configuration. > > True. Did anyone claim otherwise? > > What I saw Steven responding to was your claim that there is *never* a > good reason to do it. > > To refute that, it's sufficient to show that good reason can exist in > some cases. That's entirely compatible with a good reason not existing > in other cases. > I'll concede that sometimes it's a lot of effort to fix misconfigured databases. However, it's a form of technical debt, and when you start investigating problems ("this isn't working because X is incompatible with Y"), that's an indication that it's starting to cost you to *maintain* the misconfiguration. Hence it's more likely to be worth just paying off your technical debt and moving forward. ChrisA From rosuav at gmail.com Thu Mar 22 21:05:34 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 12:05:34 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 11:39 AM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 11:08:56 +1100, Chris Angelico wrote: >> Okay. Give me a good reason for the database itself to be locked to >> Latin-1. Make sure you explain how potentially saving the occasional >> byte of storage (compared to UTF-8) justifies limiting the available >> character set to the ones that happen to be in Latin-1, yet it's >> essential to NOT limit the character set to ASCII. > > I'll better than that, I'll give multiple good reasons to use Latin-1. > > It's company policy to only use Latin-1, because the CEO was once > employed by the Unicode Consortium, and fired in disgrace after > embezzling funds, and ever since then he has refused to use Unicode. You clearly can't afford to quit your job, so I won't mention that possibility. (Oops too late.) But a CEO is not God, and you *can* either dispute or subvert stupid orders. I don't consider this a *good* reason. Maybe a reason, but not a good one. > Compatibility with other databases, systems or tools that require Latin-1. Change them one at a time. When you have to pass data to something that has to receive Latin-1, you encode it to Latin-1. The database can still store UTF-8. Leaving it at Latin-1 is not "good reason for using Latin-1", so much as "we haven't gotten around to changing it yet". > The database has to send information to embedded devices that don't > include a full Unicode implementation, but do support Latin-1. Okay, that's a valid reason, if an incredibly rare one. You have to specifically WANT an encoding error if you try to store something that, later on, will cause problems. It's like asking for a 32-bit signed integer type in Python, because you're using it for something where it's eventually going to be sent to something that can't use larger numbers. Not something that wants a core feature, usually. > The data doesn't actually represent text, but Python 2 style byte- > strings, and Latin-1 is just a convenient, easy way to get that that > ensures ASCII bytes look like ASCII characters. The OP is talking about JSON. Reason makes no sense in that context. And if it really is a byte string, why store it as a Latin-1 string? Store it as the type BLOB instead. Latin-1 is not "arbitrary bytes". It is a very specific encoding that cannot decode every possible byte value. Using Latin-1 to store arbitrary bytes is just as wrong as using ASCII to store eight-bit data. So, you've given me one possible reason that is EXTREMELY situational and, even there, could be handled differently. And it's only valid when you're working with something that supports more than ASCII and no more than Latin-1, and moreover, you have the need for non-ASCII characters. (Otherwise, just use ASCII, which you can declare as UTF-8 if you wish.) ChrisA From saxri89 at gmail.com Fri Mar 23 00:03:42 2018 From: saxri89 at gmail.com (Xristos Xristoou) Date: Thu, 22 Mar 2018 21:03:42 -0700 (PDT) Subject: PyQt4 QWebView cant load google maps markers Message-ID: <6d0d272c-efe4-4994-b848-3b86dc380183@googlegroups.com> I want to create a simple python app using pyqt,QWebView and google maps with markers. The problem is that,the markers does not load inside the QWebView, as you can see they load just fine in the browser. here the simple code : import sys from PyQt4.QtCore import * from PyQt4.QtGui import * from PyQt4.QtWebKit import * import os app = QApplication(sys.argv) web_view= dialog.findChild(QWebView,"webView") google='https://www.google.gr/maps/place/Granite+Mountain+Hotshots+Memorial+State+Park/@34.1749572,-112.850308,12.78z/data=!4m13!1m7!3m6!1s0x872b08ebcb4c186b:0x423927b17fc1cd71!2zzpHPgc65zrbPjM69zrEsIM6Xzr3Pic68zq3Ovc61z4IgzqDOv867zrnPhM61zq_Otc-C!3b1!8m2!3d34.0489281!4d-111.0937311!3m4!1s0x80d330577faee965:0x66d75aef24890ae1!8m2!3d34.2032843!4d-112.7746582?hl=el' web_view2.load(QUrl(google)) web_view2.show() sys.exit(app.exec_()) error message : right click and select Inspect, then go to console and observe the messages console I take this error : TypeError: 'undefined' is not a function (evaluating 'this.D.bind(this)') marker.js:58 Any idea how to can fix that ? I need to add more settings to QWebView to work google maps with markers ? From steve+comp.lang.python at pearwood.info Fri Mar 23 01:35:31 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 23 Mar 2018 05:35:31 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Fri, 23 Mar 2018 12:05:34 +1100, Chris Angelico wrote: > Latin-1 is not "arbitrary bytes". It is a very specific encoding that > cannot decode every possible byte value. Yes it can. py> blob = bytes(range(256)) py> len(blob) 256 py> blob[45:55] b'-./0123456' py> s = blob.decode('latin1') py> len(s) 256 py> s[45:55] '-./0123456' -- Steve From rosuav at gmail.com Fri Mar 23 03:35:20 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 18:35:20 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 4:35 PM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 12:05:34 +1100, Chris Angelico wrote: > >> Latin-1 is not "arbitrary bytes". It is a very specific encoding that >> cannot decode every possible byte value. > > Yes it can. > > py> blob = bytes(range(256)) > py> len(blob) > 256 > py> blob[45:55] > b'-./0123456' > py> s = blob.decode('latin1') > py> len(s) > 256 > py> s[45:55] > '-./0123456' > That doesn't seem to be a strictly-correct Latin-1 decoder, then. There are a number of unassigned byte values in ISO-8859-1. ChrisA From python at bdurham.com Fri Mar 23 04:30:49 2018 From: python at bdurham.com (Malcolm Greene) Date: Fri, 23 Mar 2018 02:30:49 -0600 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement Message-ID: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> Looking for advice on how to expand f-string literal strings whose values I'm reading from a configuration file vs hard coding into my script as statements. I'm using f-strings as a very simple template language. I'm currently using the following technique to expand these f-strings. Is there a better way? Bonus if anyone has suggestions on how my expand() function can access the locals() value of the caller so this parameter doesn't have to be passed in explicitly. *def *expand(expression, values): """Expand expression using passed in locals()""" triple_quote = *"'" ** 3 expression = dedent(expression) *return *eval(*f'f{triple_quote}{expression}{triple_quote}'*, *None*, values) product_name = 'Bike ABC' product_sku = '123456' product_price = 299.95 discount = 0.85 # read in product description template # product_description_template might look like: {product_sku} : # {product_name}: ${product_price * discount}product_description_template = config('product_description_template') # expand the {expressions} in product_description_template using my # locals()product_description = expand(product_description_template, locals()) From python at bdurham.com Fri Mar 23 04:37:18 2018 From: python at bdurham.com (Malcolm Greene) Date: Fri, 23 Mar 2018 02:37:18 -0600 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement In-Reply-To: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> Message-ID: <1521794238.1623038.1313296248.1F50124B@webmail.messagingengine.com> My original post reformatted for text mode: Looking for advice on how to expand f-string literal strings whose values I'm reading from a configuration file vs hard coding into my script as statements. I'm using f-strings as a very simple template language. I'm currently using the following technique to expand these f-strings. Is there a better way? Bonus if anyone has suggestions on how my expand() function can access the locals() value of the caller so this parameter doesn't have to be passed in explicitly. def expand(expression, values): """Expand expression using passed in locals()""" triple_quote = "'" * 3 expression = dedent(expression) return eval(f'f{triple_quote}{expression}{triple_quote}', None, values) product_name = 'Bike ABC' product_sku = '123456' product_price = 299.95 discount = 0.85 # read in product description template # product_description_template might look like: {product_sku} : {product_name}: ${product_price * discount} product_description_template = config('product_description_template') # expand the {expressions} in product_description_template using my locals() product_description = expand(product_description_template, locals()) From rosuav at gmail.com Fri Mar 23 04:50:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 19:50:24 +1100 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement In-Reply-To: <1521794238.1623038.1313296248.1F50124B@webmail.messagingengine.com> References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> <1521794238.1623038.1313296248.1F50124B@webmail.messagingengine.com> Message-ID: On Fri, Mar 23, 2018 at 7:37 PM, Malcolm Greene wrote: > My original post reformatted for text mode: > > Looking for advice on how to expand f-string literal strings whose values I'm reading from a configuration file vs hard coding into > my script as statements. I'm using f-strings as a very simple template language. > > I'm currently using the following technique to expand these f-strings. Is there a better way? They're not a simple template language; they're a special form of expression. If you want something you can read from a config file, look instead at str.format(). # Just an ordinary string at this point, and can be read from a file or anything product_description_template = "{product_sku} : {product_name}: ${discounted_price}" # Need to pre-calculate any compound values discounted_price = product_price * discount print(product_description_template.format(**locals())) You can't execute arbitrary code this way, but that's generally a _good_ thing. If you really want a templating language that can execute arbitrary code, the best way is to actually make your templating language BE code; for instance, you can make your config file be a Python script that creates a bunch of functions, or use exec/eval directly (though I still wouldn't recommend the latter if you call this in any sort of tight loop; eval and exec are not cheap, so it's a lot more efficient to compile once). ChrisA From p.f.moore at gmail.com Fri Mar 23 05:46:49 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2018 09:46:49 +0000 Subject: Putting Unicode characters in JSON In-Reply-To: <0e263b68-138f-98b8-4302-dce0605ec4ce@tjol.eu> References: <0e263b68-138f-98b8-4302-dce0605ec4ce@tjol.eu> Message-ID: On 23 March 2018 at 00:27, Thomas Jollans wrote: > On 22/03/18 20:46, Tobiah wrote: >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do that for me when I give it a unicode object? > > Definitely not. In fact, that won't even work. > >>>> import json >>>> s = 'd?j? vu'.encode('latin1') >>>> s > b'd\xe9j\xe0 vu' >>>> json.dumps(s.decode('latin1').encode('utf8')) > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.6/json/__init__.py", line 231, in dumps > return _default_encoder.encode(obj) > File "/usr/lib/python3.6/json/encoder.py", line 199, in encode > chunks = self.iterencode(o, _one_shot=True) > File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode > return _iterencode(o, 0) > File "/usr/lib/python3.6/json/encoder.py", line 180, in default > o.__class__.__name__) > TypeError: Object of type 'bytes' is not JSON serializable >>>> > > You should make sure that either the file you're writing to is opened as > UTF-8 text, or the ensure_ascii parameter of dumps() or dump() is set to > True (the default) ? and then write the data in ASCII or any > ASCII-compatible encoding (e.g. UTF-8). > > Basically, the default behaviour of the json module means you don't > really have to worry about encodings at all once your original data is > in unicode strings. >From my analysis of the OP's comments, I suspect he's using Python 2, which muddles the distinction between bytes and (Unicode) text, and that's why he is seeing such strange results. Getting this right in Python 2 is going to involve having a clear understanding of how text and bytes differ, and carefully tracking which values are conceptually text and which are conceptually bytes. In my view one of the easiest ways of doing this is to try writing the code you want in Python 3, and watch how it breaks (as you've demonstrated above, it will!) Then, if you need your code to work in Python 2, apply the knowledge you've gained to the Python 2 codebase. Unfortunately, that may not be practical (people can be locked on Python 2 for all sorts of reasons). If that's the case, then I can't offer much help to the OP beyond "learn how Unicode works" - which isn't much help, as that's basically what he asked in the first place... Paul From steve+comp.lang.python at pearwood.info Fri Mar 23 06:29:24 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 23 Mar 2018 10:29:24 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote: > That doesn't seem to be a strictly-correct Latin-1 decoder, then. There > are a number of unassigned byte values in ISO-8859-1. That's incorrect, but I don't blame you for getting it wrong. Who thought that it was a good idea to distinguish between "ISO 8859-1" and "ISO-8859-1" as two related but distinct encodings? https://en.wikipedia.org/wiki/ISO/IEC_8859-1 The old ISO 8859-1 standard, the one with undefined values, is mostly of historical interest. For the last twenty years or so, anyone talking about either Latin-1 or ISO-8859-1 (with or without dashes) is almost meaning the 1992 IANA superset version which defines all 256 characters: "In 1992, the IANA registered the character map ISO_8859-1:1987, more commonly known by its preferred MIME name of ISO-8859-1 (note the extra hyphen over ISO 8859-1), a superset of ISO 8859-1, for use on the Internet. This map assigns the C0 and C1 control characters to the unassigned code values thus provides for 256 characters via every possible 8-bit value." Either that, or they actually mean Windows-1252, but let's not go there. -- Steve From rosuav at gmail.com Fri Mar 23 06:35:34 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 21:35:34 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 9:29 PM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote: > >> That doesn't seem to be a strictly-correct Latin-1 decoder, then. There >> are a number of unassigned byte values in ISO-8859-1. > > That's incorrect, but I don't blame you for getting it wrong. Who thought > that it was a good idea to distinguish between "ISO 8859-1" and > "ISO-8859-1" as two related but distinct encodings? > > https://en.wikipedia.org/wiki/ISO/IEC_8859-1 > > The old ISO 8859-1 standard, the one with undefined values, is mostly of > historical interest. For the last twenty years or so, anyone talking > about either Latin-1 or ISO-8859-1 (with or without dashes) is almost > meaning the 1992 IANA superset version which defines all 256 characters: > > "In 1992, the IANA registered the character map ISO_8859-1:1987, > more commonly known by its preferred MIME name of ISO-8859-1 > (note the extra hyphen over ISO 8859-1), a superset of ISO > 8859-1, for use on the Internet. This map assigns the C0 and C1 > control characters to the unassigned code values thus provides > for 256 characters via every possible 8-bit value." > > > Either that, or they actually mean Windows-1252, but let's not go there. > Wait, whaaa....... Though in my own defense, MySQL itself seems to have a bit of a problem with encoding names. Its "utf8" is actually "UTF-8 with a maximum of three bytes per character", in contrast to "utf8mb4" which is, well, UTF-8. In any case, abusing "Latin-1" to store binary data is still wrong. That's what BLOB is for. ChrisA From none at gmail.com Fri Mar 23 08:16:32 2018 From: none at gmail.com (ast) Date: Fri, 23 Mar 2018 13:16:32 +0100 Subject: Entering a very large number Message-ID: <5ab4f023$0$12260$426a74cc@news.free.fr> Hi I found this way to put a large number in a variable. C = int( "28871482380507712126714295971303939919776094592797" "22700926516024197432303799152733116328983144639225" "94197780311092934965557841894944174093380561511397" "99994215424169339729054237110027510420801349667317" "55152859226962916775325475044445856101949404200039" "90443211677661994962953925045269871932907037356403" "22737012784538991261203092448414947289768854060249" "76768122077071687938121709811322297802059565867") It works but is it not optimal since there is a string to int conversion. I was not able to put an integer directly because character '\' for line cut doesnt work inside an integer C = \ 28871482380507712126714295971303939919776094592797\ 22700926516024197432303799152733116328983144639225\ ... 76768122077071687938121709811322297802059565867 doesn't work Do you have a better idea ? Thx From wolfgang.maier at biologie.uni-freiburg.de Fri Mar 23 08:30:11 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 23 Mar 2018 13:30:11 +0100 Subject: Entering a very large number In-Reply-To: <5ab4f023$0$12260$426a74cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: On 03/23/2018 01:16 PM, ast wrote: > Hi > > I found this way to put a large number in > a variable. > > C = int( > "28871482380507712126714295971303939919776094592797" > "22700926516024197432303799152733116328983144639225" > "94197780311092934965557841894944174093380561511397" > "99994215424169339729054237110027510420801349667317" > "55152859226962916775325475044445856101949404200039" > "90443211677661994962953925045269871932907037356403" > "22737012784538991261203092448414947289768854060249" > "76768122077071687938121709811322297802059565867") > A very simple improvement would be to use a single triple-quoted string. Assuming you are copy/pasting the number from somewhere that will save a lot of your time. > It works but is it not optimal since there is a > string to int conversion. > > I was not able to put an integer directly because > character '\' for line cut doesnt work inside an > integer > > C = \ > 28871482380507712126714295971303939919776094592797\ > 22700926516024197432303799152733116328983144639225\ > ... > 76768122077071687938121709811322297802059565867 > > doesn't work > > Do you have a better idea ? > > Thx > > From rustompmody at gmail.com Fri Mar 23 08:43:37 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Fri, 23 Mar 2018 05:43:37 -0700 (PDT) Subject: Entering a very large number In-Reply-To: <5ab4f023$0$12260$426a74cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: > Hi > > I found this way to put a large number in > a variable. What stops you from entering the number on one single (v long) line? In case there is a religious commitment to PEP 8 dicta, the recommended meditation is this line (also from PEP8): "However, know when to be inconsistent -- sometimes style guide recommendations just aren't applicable" From wolfgang.maier at biologie.uni-freiburg.de Fri Mar 23 08:55:28 2018 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 23 Mar 2018 13:55:28 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: On 03/23/2018 01:30 PM, Wolfgang Maier wrote: > On 03/23/2018 01:16 PM, ast wrote: >> Hi >> >> I found this way to put a large number in >> a variable. >> >> C = int( >> "28871482380507712126714295971303939919776094592797" >> "22700926516024197432303799152733116328983144639225" >> "94197780311092934965557841894944174093380561511397" >> "99994215424169339729054237110027510420801349667317" >> "55152859226962916775325475044445856101949404200039" >> "90443211677661994962953925045269871932907037356403" >> "22737012784538991261203092448414947289768854060249" >> "76768122077071687938121709811322297802059565867") >> > > A very simple improvement would be to use a single > triple-quoted string. Assuming you are copy/pasting > the number from somewhere that will save a lot of your > time. Like this, for example: n = int( ''.join(""" 37107287533902102798797998220837590246510135740250 46376937677490009712648124896970078050417018260538 74324986199524741059474233309513058123726617309629 91942213363574161572522430563301811072406154908250 23067588207539346171171980310421047513778063246676 89261670696623633820136378418383684178734361726757 28112879812849979408065481931592621691275889832738 44274228917432520321923589422876796487670272189318 47451445736001306439091167216856844588711603153276 70386486105843025439939619828917593665686757934951 62176457141856560629502157223196586755079324193331 64906352462741904929101432445813822663347944758178 92575867718337217661963751590579239728245598838407 58203565325359399008402633568948830189458628227828 80181199384826282014278194139940567587151170094390 35398664372827112653829987240784473053190104293586 86515506006295864861532075273371959191420517255829 71693888707715466499115593487603532921714970056938 54370070576826684624621495650076471787294438377604 53282654108756828443191190634694037855217779295145 36123272525000296071075082563815656710885258350721 45876576172410976447339110607218265236877223636045 17423706905851860660448207621209813287860733969412 """.split()) ) From none at gmail.com Fri Mar 23 09:01:03 2018 From: none at gmail.com (ast) Date: Fri, 23 Mar 2018 14:01:03 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: <5ab4fa92$0$7586$426a34cc@news.free.fr> Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: > On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: >> Hi >> >> I found this way to put a large number in >> a variable. > > What stops you from entering the number on one single (v long) line? It is not beautiful and not very readable. It is better to have a fixed number of digits per line (eg 50) import this Beautiful is better than ugly. Readability counts. > > In case there is a religious commitment to PEP 8 dicta, the recommended > meditation is this line (also from PEP8): > > "However, know when to be inconsistent -- sometimes style guide recommendations just aren't applicable" > Yes I am using pylint which flags too long lines (80 characters) From none at gmail.com Fri Mar 23 09:07:21 2018 From: none at gmail.com (ast) Date: Fri, 23 Mar 2018 14:07:21 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: <5ab4fc0c$0$7202$426a34cc@news.free.fr> Le 23/03/2018 ? 13:30, Wolfgang Maier a ?crit?: > On 03/23/2018 01:16 PM, ast wrote: > A very simple improvement would be to use a single > triple-quoted string. Assuming you are copy/pasting > the number from somewhere that will save a lot of your > time. no, it seems that sone \n are inserted inside the number >>> C = int(""" 1234 5678""") Traceback (most recent call last): File "", line 3, in 5678""") ValueError: invalid literal for int() with base 10: '\n1234\n5678' From tjol at tjol.eu Fri Mar 23 09:11:41 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Fri, 23 Mar 2018 14:11:41 +0100 Subject: Entering a very large number In-Reply-To: <5ab4fa92$0$7586$426a34cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> Message-ID: <97c8b319-65b4-3d61-a0b2-7b1e6133adef@tjol.eu> On 2018-03-23 14:01, ast wrote: > Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: >> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: >>> Hi >>> >>> I found this way to put a large number in >>> a variable. >> >> What stops you from entering the number on one single (v long) line? > > > It is not beautiful and not very readable. It is better to > have a fixed number of digits per line (eg 50) > > import this > > Beautiful is better than ugly. > Readability counts. I would tend to read this as a reason not to have extremely large numbers in your code in the first place if you can avoid it. > > >> >> In case there is a religious commitment to PEP 8 dicta, the recommended >> meditation is this line (also from PEP8): >> >> "However, know when to be inconsistent -- sometimes style guide >> recommendations just aren't applicable" >> > > Yes I am using pylint which flags too long lines (80 characters) From dfnsonfsduifb at gmx.de Fri Mar 23 09:11:54 2018 From: dfnsonfsduifb at gmx.de (Johannes Bauer) Date: Fri, 23 Mar 2018 14:11:54 +0100 Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> Message-ID: On 23.03.2018 14:01, ast wrote: > It is not beautiful and not very readable. It is better to > have a fixed number of digits per line (eg 50) Oh yes, because clearly a 400-digit number becomes VERY beautiful and readable once you add line breaks to it. Cheers, Joe -- >> Wo hattest Du das Beben nochmal GENAU vorhergesagt? > Zumindest nicht ?ffentlich! Ah, der neueste und bis heute genialste Streich unsere gro?en Kosmologen: Die Geheim-Vorhersage. - Karl Kaos ?ber R?diger Thomas in dsa From none at gmail.com Fri Mar 23 09:12:27 2018 From: none at gmail.com (ast) Date: Fri, 23 Mar 2018 14:12:27 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: <5ab4fd3f$0$9289$426a34cc@news.free.fr> Le 23/03/2018 ? 13:55, Wolfgang Maier a ?crit?: > On 03/23/2018 01:30 PM, Wolfgang Maier wrote: >> On 03/23/2018 01:16 PM, ast wrote: > > n = int( > ??? ''.join(""" > 37107287533902102798797998220837590246510135740250 > 46376937677490009712648124896970078050417018260538 > 74324986199524741059474233309513058123726617309629 ... > 45876576172410976447339110607218265236877223636045 > 17423706905851860660448207621209813287860733969412 > """.split()) > ) > yes, good idea From antoon.pardon at vub.be Fri Mar 23 09:16:19 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Fri, 23 Mar 2018 14:16:19 +0100 Subject: Entering a very large number In-Reply-To: <5ab4fa92$0$7586$426a34cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> Message-ID: <00a3eef2-0725-7b6d-c851-e0cff5d629bb@vub.be> On 23-03-18 14:01, ast wrote: > Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: >> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: >>> Hi >>> >>> I found this way to put a large number in >>> a variable. >> >> What stops you from entering the number on one single (v long) line? > > > It is not beautiful and not very readable. It is better to > have a fixed number of digits per line (eg 50) Numbers that large are not readable, no matter how you put then in your code. Such a blob of digits just doesn't carry much meaning to humans. What meaningful information from number can you easily retrieve from representing the number in some kind of table form that you can't from just writing the number on one line? -- Antoon. From none at gmail.com Fri Mar 23 09:35:30 2018 From: none at gmail.com (ast) Date: Fri, 23 Mar 2018 14:35:30 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> <00a3eef2-0725-7b6d-c851-e0cff5d629bb@vub.be> Message-ID: <5ab502a5$0$5220$426a34cc@news.free.fr> Le 23/03/2018 ? 14:16, Antoon Pardon a ?crit?: > On 23-03-18 14:01, ast wrote: >> Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: >>> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: > What meaningful information from number can you easily retrieve from > representing the number in some kind of table form that you can't from > just writing the number on one line? > digit n? 103 is 1 digit n? 150 is 7 ... From Richard at Damon-Family.org Fri Mar 23 09:55:03 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Fri, 23 Mar 2018 09:55:03 -0400 Subject: Entering a very large number In-Reply-To: <5ab502a5$0$5220$426a34cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> <00a3eef2-0725-7b6d-c851-e0cff5d629bb@vub.be> <5ab502a5$0$5220$426a34cc@news.free.fr> Message-ID: On 3/23/18 9:35 AM, ast wrote: > Le 23/03/2018 ? 14:16, Antoon Pardon a ?crit?: >> On 23-03-18 14:01, ast wrote: >>> Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: >>>> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: > > >> What meaningful information from number can you easily retrieve from >> representing the number in some kind of table form that you can't from >> just writing the number on one line? >> > > digit n? 103 is 1 > digit n? 150 is 7 > ... > If the value of the specific digits is meaningful, they you likely don't really have a number, but a digital representation, for which a string is probably the better way to define it, and take the cost of the conversion as part of the cost to be pretty. -- Richard Damon From antoon.pardon at vub.be Fri Mar 23 10:04:39 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Fri, 23 Mar 2018 15:04:39 +0100 Subject: Entering a very large number In-Reply-To: <5ab502a5$0$5220$426a34cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fa92$0$7586$426a34cc@news.free.fr> <00a3eef2-0725-7b6d-c851-e0cff5d629bb@vub.be> <5ab502a5$0$5220$426a34cc@news.free.fr> Message-ID: <94c1ad99-4e1c-ce56-6628-6c5c7116b393@vub.be> On 23-03-18 14:35, ast wrote: > Le 23/03/2018 ? 14:16, Antoon Pardon a ?crit?: >> On 23-03-18 14:01, ast wrote: >>> Le 23/03/2018 ? 13:43, Rustom Mody a ?crit?: >>>> On Friday, March 23, 2018 at 5:46:56 PM UTC+5:30, ast wrote: > > >> What meaningful information from number can you easily retrieve from >> representing the number in some kind of table form that you can't from >> just writing the number on one line? >> > > digit n? 103 is 1 > digit n? 150 is 7 > ... > In what way is that meaningful information? How will that information help you in any way with writing your program? -- Antoon. From Richard at Damon-Family.org Fri Mar 23 10:12:03 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Fri, 23 Mar 2018 10:12:03 -0400 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: <82686147-5af5-6092-e0dc-105356a624ad@Damon-Family.org> On 3/23/18 6:35 AM, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 9:29 PM, Steven D'Aprano > wrote: >> On Fri, 23 Mar 2018 18:35:20 +1100, Chris Angelico wrote: >> >>> That doesn't seem to be a strictly-correct Latin-1 decoder, then. There >>> are a number of unassigned byte values in ISO-8859-1. >> That's incorrect, but I don't blame you for getting it wrong. Who thought >> that it was a good idea to distinguish between "ISO 8859-1" and >> "ISO-8859-1" as two related but distinct encodings? >> >> https://en.wikipedia.org/wiki/ISO/IEC_8859-1 >> >> The old ISO 8859-1 standard, the one with undefined values, is mostly of >> historical interest. For the last twenty years or so, anyone talking >> about either Latin-1 or ISO-8859-1 (with or without dashes) is almost >> meaning the 1992 IANA superset version which defines all 256 characters: >> >> "In 1992, the IANA registered the character map ISO_8859-1:1987, >> more commonly known by its preferred MIME name of ISO-8859-1 >> (note the extra hyphen over ISO 8859-1), a superset of ISO >> 8859-1, for use on the Internet. This map assigns the C0 and C1 >> control characters to the unassigned code values thus provides >> for 256 characters via every possible 8-bit value." >> >> >> Either that, or they actually mean Windows-1252, but let's not go there. >> > Wait, whaaa....... > > Though in my own defense, MySQL itself seems to have a bit of a > problem with encoding names. Its "utf8" is actually "UTF-8 with a > maximum of three bytes per character", in contrast to "utf8mb4" which > is, well, UTF-8. > > In any case, abusing "Latin-1" to store binary data is still wrong. > That's what BLOB is for. > > ChrisA One comment on this whole argument, the original poster asked how to get data from a database that WAS using Latin-1 encoding into JSON (which wants UTF-8 encoding) and was asking if something needed to be done beyond using .decode('Latin-1'), and in particular if they need to use a .encode('UTF-8'). The answer should be a simple Yes or No. Instead, someone took the opportunity to advocate that a wholesale change to the database was the only reasonable course of action. First comment, when someone is proposing a change, it is generally put on them the burden of proof that the change is warranted. This is especially true when they are telling someone else they need to make such a change. Absolute statements are very hard to prove (but the difficulty of proof doesn't relieve the need to provide it), and in fact are fairly easy to disprove (one counter example disproves an absolute statement). Counter examples to the absolute statement have been provided. When dealing with a code base, backwards compatibility IS important, and casually changing something that fundamental isn't the first thing that someone should be thinking about, We weren't given any details about the overall system this was part of, and they easily could be other code using the database that such a change would break. One easy Python example is to look back at the change from Python 2 to Python 3, how many years has that gone on, and how many more will people continue to deal with it? This was over a similar issue, that at least for today, Unicode is the best solution for storing arbitrary text, and forcing that change down to the fundamental level. -- Richard Damon From ned at nedbatchelder.com Fri Mar 23 10:33:02 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 23 Mar 2018 10:33:02 -0400 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement In-Reply-To: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> Message-ID: <92cc25a0-94ac-593c-687c-d5163e865363@nedbatchelder.com> On 3/23/18 4:30 AM, Malcolm Greene wrote: > Looking for advice on how to expand f-string literal strings whose > values I'm reading from a configuration file vs hard coding into > my script as statements. I'm using f-strings as a very simple > template language. > I'm currently using the following technique to expand these f-strings. > Is there a better way? > Bonus if anyone has suggestions on how my expand() function can access > the locals() value of the caller so this parameter doesn't have to be > passed in explicitly. > *def *expand(expression, values): > > """Expand expression using passed in locals()""" > > triple_quote = *"'" ** 3 > expression = dedent(expression) > > *return *eval(*f'f{triple_quote}{expression}{triple_quote}'*, > *None*, values) > > product_name = 'Bike ABC' > > product_sku = '123456' > > product_price = 299.95 > > discount = 0.85 > > > > # read in product description template > > # product_description_template might look like: {product_sku} : > # {product_name}: ${product_price * discount}product_description_template = config('product_description_template') > > > > # expand the {expressions} in product_description_template using my > # locals()product_description = expand(product_description_template, locals()) > > Perhaps it doesn't need to be said, but just to be sure: don't use eval if you don't trust the people writing the configuration file. They can do nearly unlimited damage to your environment.? They are writing code that you are running. --Ned. From toby at tobiah.org Fri Mar 23 10:46:16 2018 From: toby at tobiah.org (Tobiah) Date: Fri, 23 Mar 2018 07:46:16 -0700 Subject: Putting Unicode characters in JSON References: Message-ID: On 03/22/2018 12:46 PM, Tobiah wrote: > I have some mailing information in a Mysql database that has > characters from various other countries.? The table says that > it's using latin-1 encoding.? I want to send this data out > as JSON. > > So I'm just taking each datum and doing 'name'.decode('latin-1') > and adding the resulting Unicode value right into my JSON structure > before doing .dumps() on it.? This seems to work, and I can consume > the JSON with another program and when I print values, they look nice > with the special characters and all. > > I was reading though, that JSON files must be encoded with UTF-8.? So > should I be doing string.decode('latin-1').encode('utf-8')?? Or does > the json module do that for me when I give it a unicode object? Thanks for all the discussion. A little more about our setup: We have used a LAMP stack system for almost 20 years to deploy hundreds of websites. The database tables are latin-1 only because at the time we didn't know how or care to change it. More and more, 'special' characters caused a problem. They would not come out correctly in a .csv file or wouldn't print correctly. Lately, I noticed that a JSON file I was sending out was delivering unreadable characters. That's when I started to look into Unicode a bit more. From the discussion, and my own guesses, it looks as though all have to do is string.decode('latin-1'), and stuff that Unicode object right into my structure that gets handed to json.dumps(). If I changed my database tables to all be UTF-8 would this work cleanly without any decoding? Whatever people are doing to get these characters in, whether it's foreign keyboards, or fancy escape sequences in the web forms, would their intended characters still go into the UTF-8 database as the proper characters? Or now do I have to do a conversion on the way in to the database? We also get import data that often comes in .xlsx format. What encoding do I get when I dump a .csv from that? Do I have to ask the sender? I already know that they don't know. Toby From as at sci.fi Fri Mar 23 10:48:18 2018 From: as at sci.fi (Anssi Saari) Date: Fri, 23 Mar 2018 16:48:18 +0200 Subject: PyQt4 QWebView cant load google maps markers References: <8f05e680-8a1e-47a2-b39a-6b789f30e189@googlegroups.com> Message-ID: Xristos Xristoou writes: > I want to create a simple python app using pyqt,QWebView and google maps with markers. > > The problem is that,the markers does not load inside the QWebView, as > you can see they load just fine in the browser. Well, since you got a javascript error, maybe Qt4 doesn't support the version of javascript Google uses today? Qt4 was released over a decade ago. I tried with PyQt5 and the page loads and a marker shows up. It then disappears as Google puts up a couple of popups. There's a complaint about browser capabilities and also a privacy note from Google. From grant.b.edwards at gmail.com Fri Mar 23 11:04:43 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 23 Mar 2018 15:04:43 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On 2018-03-23, Chris Angelico wrote: > On Fri, Mar 23, 2018 at 10:47 AM, Steven D'Aprano > wrote: >> On Fri, 23 Mar 2018 07:09:50 +1100, Chris Angelico wrote: >> >>>> I was reading though, that JSON files must be encoded with UTF-8. So >>>> should I be doing string.decode('latin-1').encode('utf-8')? Or does >>>> the json module do that for me when I give it a unicode object? >>> >>> Reconfigure your MySQL database to use UTF-8. There is no reason to use >>> Latin-1 in the database. >> >> You don't know that. You don't know what technical, compatibility, policy >> or historical constraints are on the database. > > Okay. Give me a good reason for the database itself to be locked to > Latin-1. Because the DB administrator wont change things without orders from his boss, who won't order changes without because there's no budget for that. OK, perhaps it's not a _good_ reason by your metrics, but reasons like that are what you find in the real world. -- Grant Edwards grant.b.edwards Yow! ... My pants just went at on a wild rampage through a gmail.com Long Island Bowling Alley!! From grant.b.edwards at gmail.com Fri Mar 23 11:13:52 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Fri, 23 Mar 2018 15:13:52 +0000 (UTC) Subject: Putting Unicode characters in JSON References: <82686147-5af5-6092-e0dc-105356a624ad@Damon-Family.org> Message-ID: On 2018-03-23, Richard Damon wrote: > One comment on this whole argument, the original poster asked how to get > data from a database that WAS using Latin-1 encoding into JSON (which > wants UTF-8 encoding) and was asking if something needed to be done > beyond using .decode('Latin-1'), and in particular if they need to use a > .encode('UTF-8'). The answer should be a simple Yes or No. > > Instead, someone took the opportunity to advocate that a wholesale > change to the database was the only reasonable course of action. Well, this is (somewhat) Usenet, where the most common answer to "How do I do X?" is usually "X is stupid. You don't want to do X. You should do Y." This is generally followed by somebody proposing Z instead of Y and a long debate about the relative merits of Y and Z. One learns to either justify X or just ingore the subthreads about Y and Z. Except sometimes the answer _is_ that you really don't want to do X, and probably should do Y or Z. -- Grant Edwards grant.b.edwards Yow! Oh my GOD -- the at SUN just fell into YANKEE gmail.com STADIUM!! From rosuav at gmail.com Fri Mar 23 11:28:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 02:28:31 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Sat, Mar 24, 2018 at 1:46 AM, Tobiah wrote: > On 03/22/2018 12:46 PM, Tobiah wrote: >> >> I have some mailing information in a Mysql database that has >> characters from various other countries. The table says that >> it's using latin-1 encoding. I want to send this data out >> as JSON. >> >> So I'm just taking each datum and doing 'name'.decode('latin-1') >> and adding the resulting Unicode value right into my JSON structure >> before doing .dumps() on it. This seems to work, and I can consume >> the JSON with another program and when I print values, they look nice >> with the special characters and all. >> >> I was reading though, that JSON files must be encoded with UTF-8. So >> should I be doing string.decode('latin-1').encode('utf-8')? Or does >> the json module do that for me when I give it a unicode object? > > > > Thanks for all the discussion. A little more about our setup: > We have used a LAMP stack system for almost 20 years to deploy > hundreds of websites. The database tables are latin-1 only because > at the time we didn't know how or care to change it. > > More and more, 'special' characters caused a problem. They would > not come out correctly in a .csv file or wouldn't print correctly. > Lately, I noticed that a JSON file I was sending out was delivering > unreadable characters. That's when I started to look into Unicode > a bit more. From the discussion, and my own guesses, it looks > as though all have to do is string.decode('latin-1'), and stuff > that Unicode object right into my structure that gets handed to > json.dumps(). Yep, this is sounding more and more like you need to go UTF-8 everywhere. > If I changed my database tables to all be UTF-8 would this > work cleanly without any decoding? Whatever people are doing > to get these characters in, whether it's foreign keyboards, > or fancy escape sequences in the web forms, would their intended > characters still go into the UTF-8 database as the proper characters? > Or now do I have to do a conversion on the way in to the database? The best way to do things is to let your Python-MySQL bridge do the decoding for you; you'll simply store and get back Unicode strings. That's how things happen by default in Python 3 (I believe; been a while since I used MySQL, but it's like that with PostgreSQL). My recommendation is to give it a try; most likely, things will just work. > We also get import data that often comes in .xlsx format. What > encoding do I get when I dump a .csv from that? Do I have to > ask the sender? I already know that they don't know. Ah, now, that's a potential problem. A CSV file can't tell you what encoding it's in. Fortunately, UTF-8 is designed to be fairly dependable: if you attempt to decode something as UTF-8 and it works, you can be confident that it really is UTF-8. But ultimately, you have to just ask the person who exports it: "please export it in UTF-8". Generally, things should "just work" as long as you're consistent with encodings, and the easiest way to be consistent is to use UTF-8 everywhere. It's a simple rule that everyone can follow. (Hopefully. :) ) ChrisA From python at bdurham.com Fri Mar 23 12:37:53 2018 From: python at bdurham.com (Malcolm Greene) Date: Fri, 23 Mar 2018 10:37:53 -0600 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons Message-ID: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> Looking for your suggestions on best practice techniques for managing secrets used by Python daemon scripts. Use case is Windows scripts running as NT Services, but interested in Linux options as well. Here's what we're considering 1. Storing secrets in environment vars 2. Storing secrets in config file only in subfolder with access limited to daemon account only3. Using a 3rd party vault product Thanks From python at bdurham.com Fri Mar 23 12:39:05 2018 From: python at bdurham.com (Malcolm Greene) Date: Fri, 23 Mar 2018 10:39:05 -0600 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement In-Reply-To: <92cc25a0-94ac-593c-687c-d5163e865363@nedbatchelder.com> References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> <92cc25a0-94ac-593c-687c-d5163e865363@nedbatchelder.com> Message-ID: <1521823145.2872029.1313788232.6DCC70A2@webmail.messagingengine.com> > Perhaps it doesn't need to be said, but just to be sure: don't use eval if you don't trust the people writing the configuration file. They can do nearly unlimited damage to your environment.? They are writing code that you are running. Of course! Script and config file are running in a private subnet and both are maintained by a single developer. From ned at nedbatchelder.com Fri Mar 23 13:07:04 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 23 Mar 2018 13:07:04 -0400 Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement In-Reply-To: <1521823145.2872029.1313788232.6DCC70A2@webmail.messagingengine.com> References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> <92cc25a0-94ac-593c-687c-d5163e865363@nedbatchelder.com> <1521823145.2872029.1313788232.6DCC70A2@webmail.messagingengine.com> Message-ID: <25049535-148a-9fb7-32be-d5624e5233b3@nedbatchelder.com> On 3/23/18 12:39 PM, Malcolm Greene wrote: >> Perhaps it doesn't need to be said, but just to be sure: don't use eval if you don't trust the people writing the configuration file. They can do nearly unlimited damage to your environment.? They are writing code that you are running. > Of course! Script and config file are running in a private subnet and both are maintained by a single developer. Then why make your life difficult?? Put the "configuration" in a .py file. --Ned. From saxri89 at gmail.com Fri Mar 23 13:17:35 2018 From: saxri89 at gmail.com (Xristos Xristoou) Date: Fri, 23 Mar 2018 10:17:35 -0700 (PDT) Subject: PyQt4 QWebView cant load google maps markers In-Reply-To: <6d0d272c-efe4-4994-b848-3b86dc380183@googlegroups.com> References: <6d0d272c-efe4-4994-b848-3b86dc380183@googlegroups.com> Message-ID: <1347482c-dcc6-4d70-a91f-011348955724@googlegroups.com> tell some solution ?yes you are correct for some seconds show me the mark but only for some seconds.can i update my pyqt4 to show me the mark ? From steve+comp.lang.python at pearwood.info Fri Mar 23 14:19:22 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Fri, 23 Mar 2018 18:19:22 +0000 (UTC) Subject: Python 3.6: How to expand f-string literals read from a file vs inline statement References: <1521793849.1620487.1313282032.3B9F05CC@webmail.messagingengine.com> <92cc25a0-94ac-593c-687c-d5163e865363@nedbatchelder.com> <1521823145.2872029.1313788232.6DCC70A2@webmail.messagingengine.com> Message-ID: On Fri, 23 Mar 2018 10:39:05 -0600, Malcolm Greene wrote: >> Perhaps it doesn't need to be said, but just to be sure: don't use eval >> if you don't trust the people writing the configuration file. They can >> do nearly unlimited damage to your environment.? They are writing code >> that you are running. > > Of course! Script and config file are running in a private subnet Okay. So only users who have access to the private subnet can inject code into your application. That covers a *lot* of ground: "The private subnet is used by me and my wife, and we both have root on the system and trust each other implicitly." "The private subnet is used by five thousand really smart and technically savvy but emotionally immature teens who are constantly trying to escalate privileges and take over the system." I always find it amusing when techies imagine that hackers on the internet are the only security threat. http://www.zdnet.com/article/the-top-five-internal-security-threats/ https://blog.trendmicro.com/most-data-security-threats-are-internal- forrester-says/ > and both are maintained by a single developer. And this is relevant to the security risk in what way? -- Steve From drsalists at gmail.com Fri Mar 23 14:50:52 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 23 Mar 2018 11:50:52 -0700 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> Message-ID: I'd put them in a file with access to the daemon.. Putting credentials in an environment variable is insecure on Linux, because ps auxwwe lists environment variables. On Fri, Mar 23, 2018 at 9:37 AM, Malcolm Greene wrote: > Looking for your suggestions on best practice techniques for managing > secrets used by Python daemon scripts. Use case is Windows scripts > running as NT Services, but interested in Linux options as well. > Here's what we're considering > > 1. Storing secrets in environment vars > 2. Storing secrets in config file only in subfolder with access limited > to daemon account only3. Using a 3rd party vault product > > Thanks > -- > https://mail.python.org/mailman/listinfo/python-list From info at wingware.com Fri Mar 23 16:16:10 2018 From: info at wingware.com (Wingware) Date: Fri, 23 Mar 2018 16:16:10 -0400 Subject: ANN: Wing Python IDEs version 6.0.11 released Message-ID: <5AB5608A.60607@wingware.com> Hi, We've just released Wing 6.0.11 , which implements auto-save and restore for remote files, adds a Russian translation of the UI (thanks to Alexandr Dragukin), improves remote development error reporting and recovery after network breaks, correctly terminates SSH tunnels when switching projects or quitting, fixes severe network slowdown seen on High Sierra, auto-reactivates expired annual licenses without restarting Wing, and makes about 20 other improvements. For details, see https://wingware.com/pub/wingide/6.0.11/CHANGELOG.txt Download Now About Wing Wing is a family of cross-platform Python IDEs with powerful integrated editing, debugging, unit testing, and project management features. Wing runs on Windows, Linux, and OS X, and can be used to develop any kind of Python code for web, desktop, embedded scripting, and other applications. Wing 101 and Wing Personal omit some features and are free to download and use without a license. Wing Pro requires purchasing or upgrading a license, or obtaining a 30-day trial at startup. Version 6 introduces many new features, including improved multi-selection, much easier remote development , debugging from the Python Shell, recursive debugging, PEP 484 and 526 type hinting, support for Python 3.6 and 3.7, Vagrant , Jupyter , and Django 1.10+, easier Raspberry Pi development, optimized debugger, OS X full screen mode, One Dark color palette, Russian localization (thanks to Alexandr Dragukin), expanded free product line, and much more. For details, see What's New in Wing Version 6 . Wing 6 works with Python versions 2.5 through 2.7 and 3.2 through 3.7, including also Anaconda, ActivePython, EPD, Stackless, and others derived from the CPython implementation. For more product information, please visit wingware.com Upgrading You can try Wing 6 without removing older versions. Wing 6 will read and convert your old preferences, settings, and projects. Projects should be saved to a new name since previous versions of Wing cannot read Wing 6 projects. See also Migrating from Older Versions and Upgrading . Links Release notice: https://wingware.com/news/2018-03-21 Downloads and Free Trial: https://wingware.com/downloads Buy: https://wingware.com/store/purchase Upgrade: https://wingware.com/store/upgrade Questions? Don't hesitate to email us at support at wingware.com. Thanks, -- Stephan Deibel Wingware | Python IDE The Intelligent Development Environment for Python Programmers wingware.com From steve+comp.lang.python at pearwood.info Fri Mar 23 20:11:45 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 00:11:45 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote: > If I changed my database tables to all be UTF-8 would this work cleanly > without any decoding? Not reliably or safely. It will appear to work so long as you have only pure ASCII strings from the database, and then crash when you don't: py> text_from_database = u"hello w?rld".encode('latin1') py> print text_from_database hello w?rld py> json.dumps(text_from_database) Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/json/__init__.py", line 231, in dumps return _default_encoder.encode(obj) File "/usr/local/lib/python2.7/json/encoder.py", line 195, in encode return encode_basestring_ascii(o) UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 7: invalid start byte > Whatever people are doing to get these characters > in, whether it's foreign keyboards, or fancy escape sequences in the web > forms, would their intended characters still go into the UTF-8 database > as the proper characters? Or now do I have to do a conversion on the way > in to the database? There is no way to answer that, because it depends on how you are getting the characters, what you are doing to them, and how you put them in the database. In the best possible scenario, your process is: - user input comes in as UTF-8; - you store it in the database; - which converts it to Latin-1 (sometimes losing data: see below) in which case, changing the database field to utf8mb4 (NOT plain utf8, thanks to a ludicrously idiotic design flaw in MySQL utf8 is not actually utf8) should work nicely. I mentioned losing data: if your user enters, let's say the Greek letters '???' (or emojis, or any of about a million other characters) then Latin1 cannot represent them. Presumably your database is throwing them away: py> s = '???' # what the user wanted py> db = s.encode('latin-1', errors='replace') # what the database recorded py> json.dumps(db.decode('latin-1')) # what you end up with '"???"' Or, worse, you're getting moji-bake: py> s = '???' # what the user wanted py> json.dumps(s.encode('utf-8').decode('latin-1')) '"\\u00ce\\u00b1\\u00ce\\u00b2\\u00ce\\u00b3"' > We also get import data that often comes in .xlsx format. What encoding > do I get when I dump a .csv from that? Do I have to ask the sender? I > already know that they don't know. They never do :-( In Python 2, I believe the CSV module will assume ASCII-plus-random-crap, and it will work fine so long as it actually is ASCII. Otherwise you'll get random-crap: possibly an exception, possibly moji-bake. The sad truth is that as soon as you leave the nice, clean world of pure Unicode, and start dealing with legacy encodings, everything turns to quicksand. If you haven't already done so, you really should start by reading Joel Spolsky's introduction to Unicode: http://global.joelonsoftware.com/English/Articles/Unicode.html and Ned Batchelder's post on dealing with Unicode and Python: https://nedbatchelder.com/text/unipain.html -- Steve From rosuav at gmail.com Fri Mar 23 20:21:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 11:21:09 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: On Sat, Mar 24, 2018 at 11:11 AM, Steven D'Aprano wrote: > On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote: > >> If I changed my database tables to all be UTF-8 would this work cleanly >> without any decoding? > > Not reliably or safely. It will appear to work so long as you have only > pure ASCII strings from the database, and then crash when you don't: > > py> text_from_database = u"hello w?rld".encode('latin1') > py> print text_from_database > hello w?rld > py> json.dumps(text_from_database) > Traceback (most recent call last): > File "", line 1, in > File "/usr/local/lib/python2.7/json/__init__.py", line 231, in dumps > return _default_encoder.encode(obj) > File "/usr/local/lib/python2.7/json/encoder.py", line 195, in encode > return encode_basestring_ascii(o) > UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 7: > invalid start byte > If the database has been configured to use UTF-8 (as mentioned, that's "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get back valid UTF-8. At least, if ever you don't, that's a MySQL bug, and not your fault. So yes, it WILL work cleanly. Reliably and safely. ChrisA From arek.bulski at gmail.com Sat Mar 24 00:34:51 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Sat, 24 Mar 2018 04:34:51 +0000 Subject: Since when builtin dict preserve key order? Message-ID: I already asked on PYPY and they confirmed that any version of pypy, including 2.7, has dict preserving insertion order. I am familiar with ordered **kw which was introduced in 3.6 but I also heard that builtin dict preserves order since 3.5. Is that true? -- ~ Arkadiusz Bulski From rosuav at gmail.com Sat Mar 24 00:39:54 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 15:39:54 +1100 Subject: Since when builtin dict preserve key order? In-Reply-To: References: Message-ID: On Sat, Mar 24, 2018 at 3:34 PM, Arkadiusz Bulski wrote: > I already asked on PYPY and they confirmed that any version of pypy, > including 2.7, has dict preserving insertion order. I am familiar with > ordered **kw which was introduced in 3.6 but I also heard that builtin dict > preserves order since 3.5. Is that true? > I don't think 3.5 had it. According to the tracker, it landed in 3.6: https://bugs.python.org/issue27350 But yes, current versions of CPython preserve insertion order. ChrisA From steve+comp.lang.python at pearwood.info Sat Mar 24 02:10:47 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 06:10:47 +0000 (UTC) Subject: Putting Unicode characters in JSON References: Message-ID: On Sat, 24 Mar 2018 11:21:09 +1100, Chris Angelico wrote: >>> If I changed my database tables to all be UTF-8 would this work >>> cleanly without any decoding? >> >> Not reliably or safely. It will appear to work so long as you have only >> pure ASCII strings from the database, and then crash when you don't: [...] > If the database has been configured to use UTF-8 (as mentioned, that's > "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get > back valid UTF-8. Of course you will -- sorry I got confused. -- Steve From marko at pacujo.net Sat Mar 24 04:48:35 2018 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 24 Mar 2018 10:48:35 +0200 Subject: Since when builtin dict preserve key order? References: Message-ID: <87o9jd97vg.fsf@elektro.pacujo.net> Chris Angelico : > On Sat, Mar 24, 2018 at 3:34 PM, Arkadiusz Bulski > wrote: >> I already asked on PYPY and they confirmed that any version of pypy, >> including 2.7, has dict preserving insertion order. I am familiar >> with ordered **kw which was introduced in 3.6 but I also heard that >> builtin dict preserves order since 3.5. Is that true? >> > > I don't think 3.5 had it. According to the tracker, it landed in 3.6: > > https://bugs.python.org/issue27350 > > But yes, current versions of CPython preserve insertion order. Is that part of the Python Language Specification? If not, it shouldn't be exploited in application programs. Marko From rosuav at gmail.com Sat Mar 24 05:14:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 20:14:02 +1100 Subject: Since when builtin dict preserve key order? In-Reply-To: <87o9jd97vg.fsf@elektro.pacujo.net> References: <87o9jd97vg.fsf@elektro.pacujo.net> Message-ID: On Sat, Mar 24, 2018 at 7:48 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Sat, Mar 24, 2018 at 3:34 PM, Arkadiusz Bulski >> wrote: >>> I already asked on PYPY and they confirmed that any version of pypy, >>> including 2.7, has dict preserving insertion order. I am familiar >>> with ordered **kw which was introduced in 3.6 but I also heard that >>> builtin dict preserves order since 3.5. Is that true? >>> >> >> I don't think 3.5 had it. According to the tracker, it landed in 3.6: >> >> https://bugs.python.org/issue27350 >> >> But yes, current versions of CPython preserve insertion order. > > Is that part of the Python Language Specification? If not, it shouldn't > be exploited in application programs. > Yes, it is; but the language spec wasn't locked in quite as soon as the functionality was. So you may find that CPython 3.6 preserves order more than the language requires. Starting with 3.7 (I believe), the language spec is significantly tighter. ChrisA From storchaka at gmail.com Sat Mar 24 05:37:20 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 24 Mar 2018 11:37:20 +0200 Subject: Since when builtin dict preserve key order? In-Reply-To: <87o9jd97vg.fsf@elektro.pacujo.net> References: <87o9jd97vg.fsf@elektro.pacujo.net> Message-ID: 24.03.18 10:48, Marko Rauhamaa ????: > Chris Angelico : >> On Sat, Mar 24, 2018 at 3:34 PM, Arkadiusz Bulski >> wrote: >>> I already asked on PYPY and they confirmed that any version of pypy, >>> including 2.7, has dict preserving insertion order. I am familiar >>> with ordered **kw which was introduced in 3.6 but I also heard that >>> builtin dict preserves order since 3.5. Is that true? >>> >> >> I don't think 3.5 had it. According to the tracker, it landed in 3.6: >> >> https://bugs.python.org/issue27350 >> >> But yes, current versions of CPython preserve insertion order. > > Is that part of the Python Language Specification? If not, it shouldn't > be exploited in application programs. Only in the future version 3.7. Surprisingly it isn't mentioned in "What?s New In Python 3.7" yet. From marko at pacujo.net Sat Mar 24 07:48:43 2018 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 24 Mar 2018 13:48:43 +0200 Subject: Since when builtin dict preserve key order? References: <87o9jd97vg.fsf@elektro.pacujo.net> Message-ID: <87k1u18zj8.fsf@elektro.pacujo.net> Chris Angelico : > On Sat, Mar 24, 2018 at 7:48 PM, Marko Rauhamaa wrote: >> Is that part of the Python Language Specification? If not, it >> shouldn't be exploited in application programs. > > Yes, it is; but the language spec wasn't locked in quite as soon as > the functionality was. So you may find that CPython 3.6 preserves > order more than the language requires. Starting with 3.7 (I believe), > the language spec is significantly tighter. I take it, then, that the Language Specification follows CPython's version numbering. I wonder how other Python implementations declare their standards compliance. Marko From rosuav at gmail.com Sat Mar 24 07:53:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 22:53:04 +1100 Subject: Since when builtin dict preserve key order? In-Reply-To: <87k1u18zj8.fsf@elektro.pacujo.net> References: <87o9jd97vg.fsf@elektro.pacujo.net> <87k1u18zj8.fsf@elektro.pacujo.net> Message-ID: On Sat, Mar 24, 2018 at 10:48 PM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Sat, Mar 24, 2018 at 7:48 PM, Marko Rauhamaa wrote: >>> Is that part of the Python Language Specification? If not, it >>> shouldn't be exploited in application programs. >> >> Yes, it is; but the language spec wasn't locked in quite as soon as >> the functionality was. So you may find that CPython 3.6 preserves >> order more than the language requires. Starting with 3.7 (I believe), >> the language spec is significantly tighter. > > I take it, then, that the Language Specification follows CPython's > version numbering. I wonder how other Python implementations declare > their standards compliance. > Have you thought to look? $ pypy Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30) [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2 Tada. ChrisA From marko at pacujo.net Sat Mar 24 08:47:03 2018 From: marko at pacujo.net (Marko Rauhamaa) Date: Sat, 24 Mar 2018 14:47:03 +0200 Subject: Since when builtin dict preserve key order? References: <87o9jd97vg.fsf@elektro.pacujo.net> <87k1u18zj8.fsf@elektro.pacujo.net> Message-ID: <87fu4p8wu0.fsf@elektro.pacujo.net> Chris Angelico : > On Sat, Mar 24, 2018 at 10:48 PM, Marko Rauhamaa wrote: >> I take it, then, that the Language Specification follows CPython's >> version numbering. I wonder how other Python implementations declare >> their standards compliance. > > Have you thought to look? > > $ pypy > Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30) > [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2 > > Tada. FTFY: Yes, that's how it works. For example PyPy prints out this initial prompt: $ pypy Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30) [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2 Marko From tim at akwebsoft.com Sat Mar 24 11:40:17 2018 From: tim at akwebsoft.com (Tim Johnson) Date: Sat, 24 Mar 2018 07:40:17 -0800 Subject: Pip Version Confusion Message-ID: <20180324154017.GA15995@mail.akwebsoft.com> I'm on Ubuntu 16.04. I'm getting the following message from pip: You are using pip version 8.1.1, however version 9.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command. # But then I get this : tim at linus:~/Downloads$ which pip /home/tim/.local/bin/pip # and this: tim at linus:~/Downloads$ pip --version pip 9.0.3 from /home/tim/.local/lib/python2.7/site-packages (python 2.7) Not sure how to resolve - please advise, thanks -- Tim From steve+comp.lang.python at pearwood.info Sat Mar 24 12:16:55 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 16:16:55 +0000 (UTC) Subject: Pip Version Confusion References: <20180324154017.GA15995@mail.akwebsoft.com> Message-ID: On Sat, 24 Mar 2018 07:40:17 -0800, Tim Johnson wrote: > I'm on Ubuntu 16.04. > > I'm getting the following message from pip: > > You are using pip version 8.1.1, however version 9.0.3 is available. You > should consider upgrading via the 'pip install --upgrade pip' command. [...] > Not sure how to resolve - please advise, thanks Well, what happens when you try running pip install --upgrade pip as it says? I expect that you've probably got two (or more!) pips installed, and somehow sometimes you get one and sometimes the other. The problem is, it seems to me, that we can get pip installed through at least three mechanisms: - installing pip by hand, via source or some other mechanism; - using ensure-pip in a sufficiently recent Python; - using your OS's package manager. And of course each version of Python can have its own pip. To diagnose this, I would try: Query your package manager, what version of pip does it say is installed? How many versions of Python do you have installed? Which one are you running? Precisely what command line did you give to get the message above? If you cd out of whatever directory you are in, does the message change? If you run pip as a different user, what happens? If you run "locate pip", how many pip installations does it find, and what are they? -- Steve From hjp-python at hjp.at Sat Mar 24 12:35:50 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 24 Mar 2018 17:35:50 +0100 Subject: Putting Unicode characters in JSON In-Reply-To: References: Message-ID: <20180324163550.ibu2kl2hw4fvjiwd@hjp.at> On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 11:11 AM, Steven D'Aprano > wrote: > > On Fri, 23 Mar 2018 07:46:16 -0700, Tobiah wrote: > >> If I changed my database tables to all be UTF-8 would this work cleanly > >> without any decoding? > > > > Not reliably or safely. It will appear to work so long as you have only > > pure ASCII strings from the database, and then crash when you don't: > > > > py> text_from_database = u"hello w?rld".encode('latin1') > > py> print text_from_database > > hello w?rld [...] > > If the database has been configured to use UTF-8 (as mentioned, that's > "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get > back valid UTF-8. Actually (with python3 and mysql.connector), you'll get back str values, not byte values encoded in utf-8 or latin-1. You don't have to decode them because the driver already did it. So as a Python programmer, you don't care what character set the database uses internally, as this is almost completely hidden from you (The one aspect that isn't hidden is of course the set of characters that you can store in a character field: Obviously, you can't store Chinese characters in a latin1 field). If you are using Python2, manual encoding and decoding may be necessary. (AFAICS the OP still hasn't stated which Python version he uses) hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sat Mar 24 13:24:56 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 24 Mar 2018 18:24:56 +0100 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> Message-ID: <20180324172456.t6qxwkwqmxutbcsb@hjp.at> On 2018-03-23 11:50:52 -0700, Dan Stromberg wrote: > I'd put them in a file with access to the daemon.. > > Putting credentials in an environment variable is insecure on Linux, > because ps auxwwe lists environment variables. But only those of your own processes. So both methods are about equally secure: If you can become the daemon user (or root), then you can read the secret. (Historically, many unixes allowed all users to read the environment variables of all processes. I don't know if this is still the case for e.g. Solaris or AIX - or macOS) hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From darcy at VybeNetworks.com Sat Mar 24 14:14:01 2018 From: darcy at VybeNetworks.com (D'Arcy Cain) Date: Sat, 24 Mar 2018 13:14:01 -0500 Subject: Accessing parent objects Message-ID: I'm not even sure how to describe what I am trying to do which perhaps indicates that what I am trying to do is the wrong solution to my problem in the first place but let me give it a shot. Look at the following code. class C1(dict): class C2(object): def f(self): return X['field'] O1 = C1() O1['field'] = 1 O2 = O1.C2() print(O2.f()) I am trying to figure out what "X" should be. I know how to access the parent class but in this case I am trying to access the parent object. I tried various forms of super() but that didn't seem to work. -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From tim at akwebsoft.com Sat Mar 24 14:21:09 2018 From: tim at akwebsoft.com (Tim Johnson) Date: Sat, 24 Mar 2018 10:21:09 -0800 Subject: Pip Version Confusion In-Reply-To: References: <20180324154017.GA15995@mail.akwebsoft.com> Message-ID: <20180324182109.GB15995@mail.akwebsoft.com> * Steven D'Aprano [180324 08:29]: > On Sat, 24 Mar 2018 07:40:17 -0800, Tim Johnson wrote: > > > I'm on Ubuntu 16.04. > > > > I'm getting the following message from pip: > > > > You are using pip version 8.1.1, however version 9.0.3 is available. You > > should consider upgrading via the 'pip install --upgrade pip' command. > [...] > > Not sure how to resolve - please advise, thanks > > > Well, what happens when you try running > > pip install --upgrade pip tim at linus:~/Downloads$ pip install --upgrade pip Requirement already up-to-date: pip in /home/tim/.local/lib/python2.7/site-packages > > I expect that you've probably got two (or more!) pips installed, and > somehow sometimes you get one and sometimes the other. The problem is, it > seems to me, that we can get pip installed through at least three > mechanisms: pips all over the place ... pips, pips2, pips2.7 all but /usr/bin/pip2 show version as 9.0.3 /usr/bin/pip2 is version 8.1.1 # code follows #!/usr/bin/python # EASY-INSTALL-ENTRY-SCRIPT: 'pip==8.1.1','console_scripts','pip2' __requires__ = 'pip==8.1.1' import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.exit( load_entry_point('pip==8.1.1', 'console_scripts', 'pip2')() ) Looks like I should not have install pip via the ubuntu package manager. > - installing pip by hand, via source or some other mechanism; > > - using ensure-pip in a sufficiently recent Python; > > - using your OS's package manager. > > And of course each version of Python can have its own pip. > > > To diagnose this, I would try: > > Query your package manager, what version of pip does it say is installed? > > How many versions of Python do you have installed? Which one are you > running? > > Precisely what command line did you give to get the message above? > > If you cd out of whatever directory you are in, does the message change? > > If you run pip as a different user, what happens? > > If you run "locate pip", how many pip installations does it find, and > what are they? > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list -- Tim Johnson http://www.akwebsoft.com, http://www.tj49.com From tim at akwebsoft.com Sat Mar 24 14:35:00 2018 From: tim at akwebsoft.com (Tim Johnson) Date: Sat, 24 Mar 2018 10:35:00 -0800 Subject: Pip Version Confusion In-Reply-To: <20180324182109.GB15995@mail.akwebsoft.com> References: <20180324154017.GA15995@mail.akwebsoft.com> <20180324182109.GB15995@mail.akwebsoft.com> Message-ID: <20180324183500.GC15995@mail.akwebsoft.com> * Tim Johnson [180324 10:32]: > * Steven D'Aprano [180324 08:29]: > > On Sat, 24 Mar 2018 07:40:17 -0800, Tim Johnson wrote: > > > > > I'm on Ubuntu 16.04. > > > > > > I'm getting the following message from pip: > > > > > > You are using pip version 8.1.1, however version 9.0.3 is available. You > > > should consider upgrading via the 'pip install --upgrade pip' command. > > [...] > > > Not sure how to resolve - please advise, thanks > > > > > > Well, what happens when you try running > > > > pip install --upgrade pip > tim at linus:~/Downloads$ pip install --upgrade pip > Requirement already up-to-date: pip in > /home/tim/.local/lib/python2.7/site-packages > > > > I expect that you've probably got two (or more!) pips installed, and > > somehow sometimes you get one and sometimes the other. The problem is, it > > seems to me, that we can get pip installed through at least three > > mechanisms: > > pips all over the place ... pips, pips2, pips2.7 > all but /usr/bin/pip2 show version as 9.0.3 > /usr/bin/pip2 is version 8.1.1 > # code follows > #!/usr/bin/python > # EASY-INSTALL-ENTRY-SCRIPT: 'pip==8.1.1','console_scripts','pip2' > __requires__ = 'pip==8.1.1' > import sys > from pkg_resources import load_entry_point > > if __name__ == '__main__': > sys.exit( > load_entry_point('pip==8.1.1', 'console_scripts', 'pip2')() > ) > > Looks like I should not have install pip via the ubuntu package manager. From the package manager: python-pip/xenial-updates,xenial-updates,now 8.1.1-2ubuntu0.4 all [installed] python-pip-whl/xenial-updates,xenial-updates,now 8.1.1-2ubuntu0.4 all [installed,automatic] Think I've found it... I'll trying 'deinstalling' that package and see if the nonsense goes away. Thank you Steven! -- Tim Johnson http://www.akwebsoft.com, http://www.tj49.com From __peter__ at web.de Sat Mar 24 14:53:03 2018 From: __peter__ at web.de (Peter Otten) Date: Sat, 24 Mar 2018 19:53:03 +0100 Subject: Accessing parent objects References: Message-ID: D'Arcy Cain wrote: > I'm not even sure how to describe what I am trying to do which perhaps > indicates that what I am trying to do is the wrong solution to my > problem in the first place but let me give it a shot. Look at the > following code. > > class C1(dict): > class C2(object): > def f(self): > return X['field'] > > O1 = C1() > O1['field'] = 1 > O2 = O1.C2() > print(O2.f()) > > I am trying to figure out what "X" should be. I know how to access the > parent class but in this case I am trying to access the parent object. > I tried various forms of super() but that didn't seem to work. You can make an arbitrary number of C1 instances and an arbitrary number of C2 instances; there cannot be an implicit connection between any pair. You have to be explicit: class C1(dict): def C2(self): return C2(self) class C2(object): def __init__(self, parent): self.parent = parent def f(self): return self.parent["field"] O1 = C1() O1['field'] = 1 O2 = O1.C2() print(O2.f()) From rosuav at gmail.com Sat Mar 24 15:30:54 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 06:30:54 +1100 Subject: Putting Unicode characters in JSON In-Reply-To: <20180324163550.ibu2kl2hw4fvjiwd@hjp.at> References: <20180324163550.ibu2kl2hw4fvjiwd@hjp.at> Message-ID: On Sun, Mar 25, 2018 at 3:35 AM, Peter J. Holzer wrote: > On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: >> If the database has been configured to use UTF-8 (as mentioned, that's >> "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get >> back valid UTF-8. > > Actually (with python3 and mysql.connector), you'll get back str values, > not byte values encoded in utf-8 or latin-1. You don't have to decode > them because the driver already did it. > > So as a Python programmer, you don't care what character set the > database uses internally, as this is almost completely hidden from you > (The one aspect that isn't hidden is of course the set of characters > that you can store in a character field: Obviously, you can't store > Chinese characters in a latin1 field). Good. I mentioned earlier that that's how it is with PostgreSQL and psycopg2, but wasn't sure about the MySQL interface modules. Glad to know that it is. ChrisA From rosuav at gmail.com Sat Mar 24 15:41:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 06:41:21 +1100 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: <20180324172456.t6qxwkwqmxutbcsb@hjp.at> References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> <20180324172456.t6qxwkwqmxutbcsb@hjp.at> Message-ID: On Sun, Mar 25, 2018 at 4:24 AM, Peter J. Holzer wrote: > On 2018-03-23 11:50:52 -0700, Dan Stromberg wrote: >> I'd put them in a file with access to the daemon.. >> >> Putting credentials in an environment variable is insecure on Linux, >> because ps auxwwe lists environment variables. > > But only those of your own processes. So both methods are about equally > secure: If you can become the daemon user (or root), then you can read > the secret. If you can become the daemon user, you can do whatever the daemon user can. (Do people ever write HTTP crawler scripts that run as user "spiderman"? Yeah, probably.) So, pretty much _by definition_, both methods are insecure there. (And obviously if you can become root, everything's wiiiiiide open.) Normally, I would consider that to be fine; proper use of Unix user isolation is usually sufficient. But if you are absolutely crazily paranoid, you CAN try other methods; just be aware that whatever you do is going to move the problem around, not solve it. First thing that comes to my mind is storing the secret on a completely separate computer, and the daemon establishes a socket connection to that computer to request the secret. If another process attempts to get the secret, the secret-manager first pings its existing socket, saying "hey, you okay there?", and then between you, you decide what to do. Could be fun to mess around with, but normally I would just depend on file and process ownership. ChrisA From hjp-python at hjp.at Sat Mar 24 15:59:58 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sat, 24 Mar 2018 20:59:58 +0100 Subject: Putting Unicode characters in JSON In-Reply-To: References: <20180324163550.ibu2kl2hw4fvjiwd@hjp.at> Message-ID: <20180324195958.t65a4tqjinl3u75z@hjp.at> On 2018-03-25 06:30:54 +1100, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 3:35 AM, Peter J. Holzer wrote: > > On 2018-03-24 11:21:09 +1100, Chris Angelico wrote: > >> If the database has been configured to use UTF-8 (as mentioned, that's > >> "utf8mb4" in MySQL), you won't get that byte sequence back. You'll get > >> back valid UTF-8. > > > > Actually (with python3 and mysql.connector), you'll get back str values, > > not byte values encoded in utf-8 or latin-1. You don't have to decode > > them because the driver already did it. > > > > So as a Python programmer, you don't care what character set the > > database uses internally, as this is almost completely hidden from you > > (The one aspect that isn't hidden is of course the set of characters > > that you can store in a character field: Obviously, you can't store > > Chinese characters in a latin1 field). > > Good. I mentioned earlier that that's how it is with PostgreSQL and > psycopg2, but wasn't sure about the MySQL interface modules. Glad to > know that it is. I'm surprised that PEP 249 doesn't specify this. It seems worth standardizing (OTOH it's the "one obvious way to do it", so maybe it doesn't need to be specified explicitely). The interfaces for the 4 databases I use (psycopg2 for PostgreSQL, cx_Oracle, mysql.connector and sqlite3) all behave the same for varchar fields, which makes me reasonably confident that this is also the case for other databases. hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rosuav at gmail.com Sat Mar 24 16:01:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 07:01:02 +1100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: On Sun, Mar 25, 2018 at 5:14 AM, D'Arcy Cain wrote: > I'm not even sure how to describe what I am trying to do which perhaps > indicates that what I am trying to do is the wrong solution to my > problem in the first place but let me give it a shot. Look at the > following code. > > class C1(dict): > class C2(object): > def f(self): > return X['field'] > > O1 = C1() > O1['field'] = 1 > O2 = O1.C2() > print(O2.f()) > > I am trying to figure out what "X" should be. I know how to access the > parent class but in this case I am trying to access the parent object. > I tried various forms of super() but that didn't seem to work. > What you'd need here is for O1.C2 to be an instance method, instead of simply a nested class. That way, it can retain a reference to O1. Something like: class C1(dict): class _C2(object): def __init__(self, parent): self.parent = parent def f(self): return self.parent["field"] def C2(self): return self._C2(self) It may also be possible to do this through __new__, but I haven't confirmed it. Might need some careful shenanigans if you want to have O1.C2 actually be the class. ChrisA From tjol at tjol.eu Sat Mar 24 16:23:01 2018 From: tjol at tjol.eu (Thomas Jollans) Date: Sat, 24 Mar 2018 21:23:01 +0100 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> <20180324172456.t6qxwkwqmxutbcsb@hjp.at> Message-ID: On 24/03/18 20:41, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 4:24 AM, Peter J. Holzer wrote: >> On 2018-03-23 11:50:52 -0700, Dan Stromberg wrote: >>> I'd put them in a file with access to the daemon.. >>> >>> Putting credentials in an environment variable is insecure on Linux, >>> because ps auxwwe lists environment variables. >> >> But only those of your own processes. So both methods are about equally >> secure: If you can become the daemon user (or root), then you can read >> the secret. > > If you can become the daemon user, you can do whatever the daemon user > can. If you're using something like SELinux, I don't think that's *necessarily* true (but I really don't know much about SELinux). Normally, though, I should think that protecting the secret with user isolation (e.g. by putting it into a file with the right permissions) should be fine. Environment variables should be fine too, but really this just moves the problem up one level: where does the parent process get the secret when it sets up the environment? From rosuav at gmail.com Sat Mar 24 16:31:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 07:31:02 +1100 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> <20180324172456.t6qxwkwqmxutbcsb@hjp.at> Message-ID: On Sun, Mar 25, 2018 at 7:23 AM, Thomas Jollans wrote: > On 24/03/18 20:41, Chris Angelico wrote: >> On Sun, Mar 25, 2018 at 4:24 AM, Peter J. Holzer wrote: >>> On 2018-03-23 11:50:52 -0700, Dan Stromberg wrote: >>>> I'd put them in a file with access to the daemon.. >>>> >>>> Putting credentials in an environment variable is insecure on Linux, >>>> because ps auxwwe lists environment variables. >>> >>> But only those of your own processes. So both methods are about equally >>> secure: If you can become the daemon user (or root), then you can read >>> the secret. >> >> If you can become the daemon user, you can do whatever the daemon user >> can. > > If you're using something like SELinux, I don't think that's > *necessarily* true (but I really don't know much about SELinux). Me neither. Anyone here happen to know? > Normally, though, I should think that protecting the secret with user > isolation (e.g. by putting it into a file with the right permissions) > should be fine. > > Environment variables should be fine too, but really this just moves the > problem up one level: where does the parent process get the secret when > it sets up the environment? A common situation is that the secret is owned by *root*, not the daemon process's user. So the parent process, running as root, reads the secret into memory, then forks, drops privileges, and exec's to the daemon with the secret in the environment. ChrisA From rantingrickjohnson at gmail.com Sat Mar 24 16:31:13 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 24 Mar 2018 13:31:13 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: Message-ID: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> On Saturday, March 24, 2018 at 1:20:24 PM UTC-5, D'Arcy Cain wrote: > I'm not even sure how to describe what I am trying to do > which perhaps indicates that what I am trying to do is the > wrong solution to my problem in the first place but let me > give it a shot. Look at the following code. > > class C1(dict): > class C2(object): > def f(self): > return X['field'] > > O1 = C1() > O1['field'] = 1 > O2 = O1.C2() > print(O2.f()) I hope this dynamic setting of instance attributes is done for a good reason, and not because you don't know how to do it any other way? Also, i hope this nested class structure is also done for a very good reason, because i consider such to be a code smell. > I am trying to figure out what "X" should be. I know how > to access the parent class but in this case I am trying to > access the parent object. Your poor utilization of OOP terminology is defeating you. In every OOP object there can be zero or more "class level variables" (aka: class attributes) and there can be zero or more "instance level variables" (aka: instance attributes). A class attribute can be accessed from anywhere (if you know the class symbol and the attribute name, that is). But an instance attribute can only be accessed in one of two manners: from within an instance (via `self.attrname`) or from outside an instance (using a symbol bound to the instance, such as: `O1.field` or O1['field']). And while there might be a way to walk down the "proverbial node tree" and arrive at the specific instance you wish to target (a la DHTML), such would be considered black magic, as it does not follow the standard protocols of OOP attribute access. My suspicion is that your attempting to solve this problem using the wrong methodology. If you would be so kind as to explain exactly what your are doing, perhaps someone here could offer a better solution. Perhaps OOP is not the best solution. Or perhaps it is? But we cannot be sure until we know more about the general problem. > I tried various forms of super() but that didn't seem to > work. Python's super (at least <= 2.x) is notoriously blinkered. I avoid it like the plague. And i'd advise you to do the same. PS: I am pleased to see you old fellers are finally warming up to the wonderful OOP paradigm, but i gotta say, what is most entertaining to me is watching you seasoned pros struggle with the simplist OOP concepts. And don't get angry, because i'm laughing at you, not with you. (or is it the other way 'round??? o_O) From drsalists at gmail.com Sat Mar 24 16:33:27 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Sat, 24 Mar 2018 13:33:27 -0700 Subject: Since when builtin dict preserve key order? In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 9:34 PM, Arkadiusz Bulski wrote: > I already asked on PYPY and they confirmed that any version of pypy, > including 2.7, has dict preserving insertion order. I am familiar with > ordered **kw which was introduced in 3.6 but I also heard that builtin dict > preserves order since 3.5. Is that true? Empirically speaking: $ cat dict-order below cmd output started 2018 Sat Mar 24 01:28:15 PM PDT #!/usr/local/cpython-2.7/bin/python """Check if a dictionary appears to have insertion-order preserved.""" n = 10 expected_list = list(range(n - 1, -1, -1)) dict_ = {} for i in expected_list: dict_[i] = 2*i actual_list = list(dict_) if actual_list != expected_list: raise AssertionError('%s != %s' % (actual_list, expected_list)) above cmd output done 2018 Sat Mar 24 01:28:15 PM PDT dstromberg at zareason2:~/src/python-tests x86_64-unknown-linux-gnu 15808 $ make below cmd output started 2018 Sat Mar 24 01:28:18 PM PDT pythons --file dict-order /usr/local/cpython-1.0/bin/python (1.0.1) bad File "dict-order", line 3 """Check if a dictionary appears to have insertion-order preserved.""" ^ SyntaxError: invalid syntax /usr/local/cpython-1.1/bin/python (1.1) bad Traceback (innermost last): File "dict-order", line 7, in ? expected_list = list(range(n - 1, -1, -1)) NameError: list /usr/local/cpython-1.2/bin/python (1.2) bad Traceback (innermost last): File "dict-order", line 7, in ? expected_list = list(range(n - 1, -1, -1)) NameError: list /usr/local/cpython-1.3/bin/python (1.3) bad Traceback (innermost last): File "dict-order", line 7, in ? expected_list = list(range(n - 1, -1, -1)) NameError: list /usr/local/cpython-1.4/bin/python (1.4) bad Traceback (innermost last): File "dict-order", line 13, in ? actual_list = list(dict_) TypeError: list() argument must be a sequence /usr/local/cpython-1.5/bin/python (1.5.2) bad Traceback (innermost last): File "dict-order", line 13, in ? actual_list = list(dict_) TypeError: list() argument must be a sequence /usr/local/cpython-1.6/bin/python (1.6.1) bad Traceback (most recent call last): File "dict-order", line 13, in ? actual_list = list(dict_) TypeError: list() argument must be a sequence /usr/local/cpython-2.0/bin/python (2.0.1) bad Traceback (most recent call last): File "dict-order", line 13, in ? actual_list = list(dict_) TypeError: list() argument must be a sequence /usr/local/cpython-2.1/bin/python (2.1.0) bad Traceback (most recent call last): File "dict-order", line 13, in ? actual_list = list(dict_) TypeError: list() argument must be a sequence /usr/local/cpython-2.2/bin/python (2.2.0) bad Traceback (most recent call last): File "dict-order", line 16, in ? raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-2.3/bin/python (2.3.0) bad Traceback (most recent call last): File "dict-order", line 16, in ? raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-2.4/bin/python (2.4.0) bad Traceback (most recent call last): File "dict-order", line 16, in ? raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-2.5/bin/python (2.5.6) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-2.6/bin/python (2.6.9) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-2.7/bin/python (2.7.13) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.0/bin/python (3.0.1) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.1/bin/python (3.1.5) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.2/bin/python (3.2.5) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.3/bin/python (3.3.3) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.4/bin/python (3.4.2) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.5/bin/python (3.5.0) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/cpython-3.6/bin/python (3.6.0) good /usr/local/cpython-3.7/bin/python (3.7.0b2) good /usr/local/jython-2.7/bin/jython (2.7.0) bad Traceback (most recent call last): File "dict-order", line 16, in raise AssertionError('%s != %s' % (actual_list, expected_list)) AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] /usr/local/pypy-5.10.0/bin/pypy (2.7.13) good /usr/local/pypy-5.3.1/bin/pypy (2.7.10) good /usr/local/pypy-5.9.0/bin/pypy (2.7.13) good /usr/local/pypy3-5.10.0/bin/pypy3 (3.5.3) good /usr/local/pypy3-5.5.0/bin/pypy3 (3.3.5) good /usr/local/pypy3-5.8.0-with-lzma-fixes/bin/pypy3 (3.5.3) good /usr/local/pypy3-5.8.0/bin/pypy3 (3.5.3) good /usr/local/pypy3-5.9.0/bin/pypy3 (3.5.3) good /usr/local/micropython-git-2017-06-16/bin/micropython (3.4.0) bad Traceback (most recent call last): File "dict-order", line 16, in AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, 4, 3, 2, 1, 0] But of course, this doesn't show the fact that order preservation is present but not guaranteed in 3.6. From arek.bulski at gmail.com Sat Mar 24 17:52:51 2018 From: arek.bulski at gmail.com (Arkadiusz Bulski) Date: Sat, 24 Mar 2018 21:52:51 +0000 Subject: Since when builtin dict preserve key order? In-Reply-To: References: Message-ID: What exactly do you mean its present but not guaranteed? Do you mean its a property of CPython 3.6 implementation but not Python as a standard? sob., 24 mar 2018 o 21:33 u?ytkownik Dan Stromberg napisa?: > On Fri, Mar 23, 2018 at 9:34 PM, Arkadiusz Bulski > wrote: > > I already asked on PYPY and they confirmed that any version of pypy, > > including 2.7, has dict preserving insertion order. I am familiar with > > ordered **kw which was introduced in 3.6 but I also heard that builtin > dict > > preserves order since 3.5. Is that true? > > Empirically speaking: > $ cat dict-order > below cmd output started 2018 Sat Mar 24 01:28:15 PM PDT > #!/usr/local/cpython-2.7/bin/python > > """Check if a dictionary appears to have insertion-order preserved.""" > > n = 10 > > expected_list = list(range(n - 1, -1, -1)) > > dict_ = {} > for i in expected_list: > dict_[i] = 2*i > > actual_list = list(dict_) > > if actual_list != expected_list: > raise AssertionError('%s != %s' % (actual_list, expected_list)) > above cmd output done 2018 Sat Mar 24 01:28:15 PM PDT > dstromberg at zareason2:~/src/python-tests x86_64-unknown-linux-gnu 15808 > > $ make > below cmd output started 2018 Sat Mar 24 01:28:18 PM PDT > pythons --file dict-order > /usr/local/cpython-1.0/bin/python (1.0.1) bad > File "dict-order", line 3 > """Check if a dictionary appears to have insertion-order > preserved.""" > > ^ > SyntaxError: invalid syntax > /usr/local/cpython-1.1/bin/python (1.1) bad > Traceback (innermost last): > File "dict-order", line 7, in ? > expected_list = list(range(n - 1, -1, -1)) > NameError: list > /usr/local/cpython-1.2/bin/python (1.2) bad > Traceback (innermost last): > File "dict-order", line 7, in ? > expected_list = list(range(n - 1, -1, -1)) > NameError: list > /usr/local/cpython-1.3/bin/python (1.3) bad > Traceback (innermost last): > File "dict-order", line 7, in ? > expected_list = list(range(n - 1, -1, -1)) > NameError: list > /usr/local/cpython-1.4/bin/python (1.4) bad > Traceback (innermost last): > File "dict-order", line 13, in ? > actual_list = list(dict_) > TypeError: list() argument must be a sequence > /usr/local/cpython-1.5/bin/python (1.5.2) bad > Traceback (innermost last): > File "dict-order", line 13, in ? > actual_list = list(dict_) > TypeError: list() argument must be a sequence > /usr/local/cpython-1.6/bin/python (1.6.1) bad > Traceback (most recent call last): > File "dict-order", line 13, in ? > actual_list = list(dict_) > TypeError: list() argument must be a sequence > /usr/local/cpython-2.0/bin/python (2.0.1) bad > Traceback (most recent call last): > File "dict-order", line 13, in ? > actual_list = list(dict_) > TypeError: list() argument must be a sequence > /usr/local/cpython-2.1/bin/python (2.1.0) bad > Traceback (most recent call last): > File "dict-order", line 13, in ? > actual_list = list(dict_) > TypeError: list() argument must be a sequence > /usr/local/cpython-2.2/bin/python (2.2.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in ? > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-2.3/bin/python (2.3.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in ? > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-2.4/bin/python (2.4.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in ? > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-2.5/bin/python (2.5.6) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-2.6/bin/python (2.6.9) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-2.7/bin/python (2.7.13) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.0/bin/python (3.0.1) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.1/bin/python (3.1.5) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.2/bin/python (3.2.5) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.3/bin/python (3.3.3) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.4/bin/python (3.4.2) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.5/bin/python (3.5.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/cpython-3.6/bin/python (3.6.0) good > /usr/local/cpython-3.7/bin/python (3.7.0b2) good > /usr/local/jython-2.7/bin/jython (2.7.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in > raise AssertionError('%s != %s' % (actual_list, expected_list)) > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > /usr/local/pypy-5.10.0/bin/pypy (2.7.13) good > /usr/local/pypy-5.3.1/bin/pypy (2.7.10) good > /usr/local/pypy-5.9.0/bin/pypy (2.7.13) good > /usr/local/pypy3-5.10.0/bin/pypy3 (3.5.3) good > /usr/local/pypy3-5.5.0/bin/pypy3 (3.3.5) good > /usr/local/pypy3-5.8.0-with-lzma-fixes/bin/pypy3 (3.5.3) good > /usr/local/pypy3-5.8.0/bin/pypy3 (3.5.3) good > /usr/local/pypy3-5.9.0/bin/pypy3 (3.5.3) good > /usr/local/micropython-git-2017-06-16/bin/micropython (3.4.0) bad > Traceback (most recent call last): > File "dict-order", line 16, in > AssertionError: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] != [9, 8, 7, 6, 5, > 4, 3, 2, 1, 0] > > But of course, this doesn't show the fact that order preservation is > present but not guaranteed in 3.6. > -- ~ Arkadiusz Bulski From rosuav at gmail.com Sat Mar 24 18:00:27 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 09:00:27 +1100 Subject: Since when builtin dict preserve key order? In-Reply-To: References: Message-ID: On Sun, Mar 25, 2018 at 8:52 AM, Arkadiusz Bulski wrote: > What exactly do you mean its present but not guaranteed? Do you mean its a > property of CPython 3.6 implementation but not Python as a standard? Correct. The implementation was done in CPython (basically lifting it from PyPy, AIUI), and then a lengthy discussion eventually settled on making it a language requirement. ChrisA From hjp-python at hjp.at Sat Mar 24 19:05:56 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 25 Mar 2018 00:05:56 +0100 Subject: Entering a very large number In-Reply-To: <5ab4fd3f$0$9289$426a34cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> Message-ID: <20180324230556.klcvdfdzcotlztav@hjp.at> On 2018-03-23 14:12:27 +0100, ast wrote: > Le 23/03/2018 ? 13:55, Wolfgang Maier a ?crit?: > > On 03/23/2018 01:30 PM, Wolfgang Maier wrote: > > > On 03/23/2018 01:16 PM, ast wrote: [quoted from the first mail in this thread:] > > > > It works but is it not optimal since there is a > > > > string to int conversion. > > n = int( > > ??? ''.join(""" > > 37107287533902102798797998220837590246510135740250 > > 46376937677490009712648124896970078050417018260538 > > 74324986199524741059474233309513058123726617309629 > ... > > > 45876576172410976447339110607218265236877223636045 > > 17423706905851860660448207621209813287860733969412 > > """.split()) > > ) > > > > yes, good idea Not if you want to avoid that string to int conversion (as you stated). That is still there, but in addition you now split the string into a list and then join the list into a different string. hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From steve+comp.lang.python at pearwood.info Sat Mar 24 19:54:56 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 23:54:56 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> Message-ID: On Sat, 24 Mar 2018 13:31:13 -0700, Rick Johnson wrote: > On Saturday, March 24, 2018 at 1:20:24 PM UTC-5, D'Arcy Cain wrote: [...] >> I tried various forms of super() but that didn't seem to work. Define "doesn't see to work". > Python's super (at least <= 2.x) is notoriously blinkered. I avoid it > like the plague. And i'd advise you to do the same. There's nothing wrong with super() in Python 2. You just have to understand what you're doing. It's still the right solution for doing inheritance the right way. (Of course it doesn't exist in Python 1, so if Rick is still using Python 1.5, that's a good excuse not to use super().) The trick is to use new-style classes that inherit from object, and avoid the old-style classes that don't: # Good class Spam(object): ... # Not so good class Spam: ... > PS: I am pleased to see you old fellers are finally warming up to the > wonderful OOP paradigm, but i gotta say, what is most entertaining to me > is watching you seasoned pros struggle with the simplist OOP concepts. Ah yes, simple concepts like being unable to get super() working in Python 2. Good one Rick. -- Steve From steve+comp.lang.python at pearwood.info Sat Mar 24 21:47:01 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 01:47:01 +0000 (UTC) Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On Sun, 25 Mar 2018 00:05:56 +0100, Peter J. Holzer wrote: [...] >> yes, good idea > > Not if you want to avoid that string to int conversion (as you stated). > > That is still there, but in addition you now split the string into a > list and then join the list into a different string. I'm glad I wasn't the only one who spotted that. There's something very curious about somebody worried about efficiency choosing a *less* efficient solution than what they started with. To quote W.A. Wulf: "More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason ? including blind stupidity." As Donald Knuth observed: "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." The Original Poster (OP) is concerned about saving, what, a tenth of a microsecond in total? Hardly seems worth the effort, especially if you're going to end up with something even slower. -- Steve From rantingrickjohnson at gmail.com Sat Mar 24 22:20:19 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 24 Mar 2018 19:20:19 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> Message-ID: <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> On Saturday, March 24, 2018 at 6:57:29 PM UTC-5, Steven D'Aprano wrote: > There's nothing wrong with super() in Python 2. You just > have to understand what you're doing. It's still the right > solution for doing inheritance the right way. The problem is, Python's super is not intuitive. And i'd like to use it. I really would! Because, yes, you are correct when you say: "super is the proper way to do inherence". But for some reason, i just cannot get it to work reliably. And reading the docs doesn't help. Seriously, i find it difficult to believe no one else has been stumped by Python's super. I have used super in other languages with no issues -- everything is intuitive -- but in Python, my favorite language, forget about it. So now, i've got all this Python code with explicit inheritance, and it works, but it just feels ugly. From rosuav at gmail.com Sat Mar 24 22:28:40 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 13:28:40 +1100 Subject: Accessing parent objects In-Reply-To: <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> Message-ID: On Sun, Mar 25, 2018 at 1:20 PM, Rick Johnson wrote: > On Saturday, March 24, 2018 at 6:57:29 PM UTC-5, Steven D'Aprano wrote: >> There's nothing wrong with super() in Python 2. You just >> have to understand what you're doing. It's still the right >> solution for doing inheritance the right way. > > The problem is, Python's super is not intuitive. > > And i'd like to use it. I really would! Because, yes, you > are correct when you say: "super is the proper way to do > inherence". But for some reason, i just cannot get it to > work reliably. > > And reading the docs doesn't help. > > Seriously, i find it difficult to believe no one else has > been stumped by Python's super. I have used super in other > languages with no issues -- everything is intuitive -- but > in Python, my favorite language, forget about it. > > So now, i've got all this Python code with explicit > inheritance, and it works, but it just feels ugly. So tell me, how do these other (beautifully intuitive) languages handle multiple inheritance? I'm sure it's really easy to make super() work when there's exactly one superclass that you can lock in at compile time. ChrisA From rantingrickjohnson at gmail.com Sat Mar 24 23:08:47 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sat, 24 Mar 2018 20:08:47 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> Message-ID: On Saturday, March 24, 2018 at 9:29:02 PM UTC-5, Chris Angelico wrote: > So tell me, how do these other (beautifully intuitive) > languages handle multiple inheritance? I'm sure it's really > easy to make super() work when there's exactly one > superclass that you can lock in at compile time. After sending my response to Steven, i began to ponder why i had rejected Python's super (because, after all, it's been a few years now, and frankly, i forgot), and i realized it was more a matter of consistency. You see, Tkinter (the Py2 version) uses old style classes (though i beleve that design flaw has been rectified in Py3), and being that i rely heavily on Tkinter for a large portion of my GUI apps (not because i necessarily prefer Tkinter over the other offerings, mind you, but simply because it is in the stdlib) the inconsistency of using super _outside_ of Tkinter code whilst simultaneously using explicit inheritance _inside_ Tkinter code was quite frankly, disturbing to me. Thus, i chose to abandon super altogether. But when i make the switch to Python4000, i will probably adopt super at that time, along with all the other fully matured goodies. In light of this revelation, I still don't believe Python's super() function is as intuitive, as say, Ruby's super statement is, but it is the correct way to do things, nonetheless. So yeah, i'll have to cross that rickety old bridge at some point in the future. Just not right now. From steve+comp.lang.python at pearwood.info Sun Mar 25 00:29:08 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 04:29:08 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> Message-ID: On Sat, 24 Mar 2018 20:08:47 -0700, Rick Johnson wrote: > After sending my response to Steven, i began to ponder why i had > rejected Python's super (because, after all, it's been a few years now, > and frankly, i forgot), and i realized it was more a matter of > consistency. > > You see, Tkinter (the Py2 version) uses old style classes That would be an inconvenience and annoyance. > (though i beleve that design flaw has been rectified in Py3), Indeed. Not so much a design flaw as such, just an annoyance due to historical reasons. What was a reasonable design back in 1994 didn't age well, but backwards-compatibility prevented fixing it until Python 3. [...] > the inconsistency of using super > _outside_ of Tkinter code whilst simultaneously using explicit > inheritance _inside_ Tkinter code was quite frankly, disturbing to me. ?A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines.? https://en.wikiquote.org/wiki/Consistency > Thus, i chose to abandon super altogether. Baby, bathwater. > But when i make the switch to Python4000, i will probably adopt super at > that time, along with all the other fully matured goodies. So, having avoided the first unstable compatibility-breaking version, Python 3000, you're going to wait until the next hypothetical unstable, compatibility-breaking version before upgrading? Python 3 is now six point releases in (and very soon to have a seventh, 3.7 being in beta as we speak). It is stable, feature-rich, and a joy to work in. As well as a heap of great new features, there have been a metric tonne of performance improvements, making Python 3 faster than 2.7 for many tasks, e.g. https://mail.python.org/pipermail/python-dev/2017-July/148641.html Python 4 is not going to be a massively compatibility-breaking change like Python 3 was, and version 4 is likely about 5 or 6 years away. Some hypothetical "Python 4000", or more likely "5000", is at least a decade away, if it ever occurs again. (Which it probably won't.) In the meantime, Python 2.7 will fall out of extended support in less than two years. Time to get with the programme, Rick, and stop living in the past :-) > In light of this revelation, I still don't believe Python's super() > function is as intuitive, as say, Ruby's super statement is, but it is > the correct way to do things, nonetheless. For comparison, here's how Python's super works in 3.x: def method(self, spam, eggs, cheese): result = super().method(spam, eggs, cheese) In other words, you must explicitly state the method you are calling, and give the arguments you want to pass on. Of course you can manipulate or remove those arguments as needed, or add new ones: result = super().method(spam+1, eggs*2, keyword='aardvark') because it's just an ordinary method call. In fact, once you have got the super proxy, you can even call another method: result = super().something_different(arg) if you need to for some reason. Remember: super() in Python gives you a proxy to the parent classes themselves, not just the parent's method of the same name. Ruby's super reduces typing, at the cost of needing more magic and less flexibility: http://rubylearning.com/satishtalim/ruby_overriding_methods.html Ruby's super doesn't return a superclass proxy, it automagically calls the same method as the current method, so you can't delegate to a different method if required. In addition, Ruby doesn't support multiple inheritance, which is exactly when super() is most important. Ruby's super calls the method in two ways: - with no arguments, using the parenthesis-free syntax, Ruby automagically forwards the same arguments to the (single) parent; - with parentheses, you have to manually pass the arguments you want to pass. So there's weird magic going on where `super` and `super()` both call the method but with different arguments. Ewww. But yes, Ruby's super requires less typing for the common case of "call the parent's method and pass the exact same arguments". > So yeah, i'll have to cross that rickety old bridge I see what you did there. -- Steve From arsh840 at gmail.com Sun Mar 25 04:01:11 2018 From: arsh840 at gmail.com (Arshpreet Singh) Date: Sun, 25 Mar 2018 01:01:11 -0700 (PDT) Subject: Question regarding objects in __call__() methods Message-ID: I am debugging a set of code which is something like this: http://dpaste.com/1JXTCF0 I am not able to understand that what role internet object is playing and how I can use/call it. As debugging the code I got at line 10. I am sending a request to particular API and returning a request_object . further deep down it generates the "response_object" as from my requirements that should be JSON object but I am only getting Python-Object in hexa form, is there any way I can get to know how to use internet_object so I can get to know how to use that internet_object with response. I am also able to call __call__ method something like this: def hierarchy_attach(env, parent): svc = IMS(env) svc(parent) From jugurtha.hadjar at gmail.com Sun Mar 25 05:37:28 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 10:37:28 +0100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: On 03/24/2018 07:14 PM, D'Arcy Cain wrote: > class C1(dict): > class C2(object): > def f(self): > return X['field'] > > O1 = C1() > O1['field'] = 1 > O2 = O1.C2() > print(O2.f()) I prefer to *feed* the child to the parent or vice versa. Simplifies things like testing. Something like this: <----------------------------------------------------------------------> class C1(object): ??? def __init__(self): ??????? self.child = None ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) ??? def adopt(self, child=None): ??????? self.child = child ??????? s = ("I am {self.__class__.__name__} and I adopted " "{self.child.__class__.__name__}".format(self=self)) ??????? print(s) class C2(object): ??? def __init__(self, parent=None): ??????? self.parent = parent ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) ??????? self.parent.foo() ??? def adoptme(self, parent=None): ??????? parent = parent or self.parent ??????? if parent is None: ??????????? print("No parent yet") ??????? else: ??????????? self.parent = parent ??????????? parent.adopt(self) c2 = C2() c2.adoptme()??? # No parent yet c1 = C1() c2.adoptme(c1)? # I am C1 and I adopted C2 c2.foo()??????? # I am C2 foo ??????????????? # I am C1 foo c1.adopt(c2)??? # I am C1 and I adopted C2 <--------------------------------------------------------------------> -- ~ Jugurtha Hadjar, From jugurtha.hadjar at gmail.com Sun Mar 25 06:10:55 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 11:10:55 +0100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: There was a typo in my original reply: <--------------------------------------------------------------------> class C1(object): ??? def __init__(self): ??????? self.child = None ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) ??? def adopt(self, child=None): ??????? self.child = child ??????? s = ("I am {self.__class__.__name__} and I adopted " "{self.child.__class__.__name__}".format(self=self)) ??????? print(s) class C2(object): ??? def __init__(self, parent=None): ??????? self.parent = parent ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) ??????? self.parent.foo() ??? def adoptme(self, parent=None): ??????? s = "I am {self.__class__.__name__} looking for adoption" ??????? print(s.format(self=self)) ??????? parent = parent or self.parent ??????? if parent is None: ??????????? print("No parent yet") ??????? else: ??????????? self.parent = parent ??????????? parent.adopt(self) c2 = C2() c2.adoptme()??? # No parent yet c1 = C1() c2.adoptme(c1)? # I am C2 looking for adoption ??????????????? # I am C1 and I adopted C2 c2.foo()??????? # I am C2 foo ??????????????? # I am C1 foo c1.adopt(c2)??? # I am C1 and I adopted C2 <------------------------------------------------------------------------> -- ~ Jugurtha Hadjar, From rosuav at gmail.com Sun Mar 25 06:17:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 21:17:59 +1100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: On Sun, Mar 25, 2018 at 8:37 PM, Jugurtha Hadjar wrote: > On 03/24/2018 07:14 PM, D'Arcy Cain wrote: >> >> class C1(dict): >> class C2(object): >> def f(self): >> return X['field'] >> >> O1 = C1() >> O1['field'] = 1 >> O2 = O1.C2() >> print(O2.f()) > > > I prefer to *feed* the child to the parent or vice versa. Congrats, this ranks on my list of "creative people who sound like psycho murderers". Digital artists and cooks tend to rank fairly highly on that list. > class C1(object): > def __init__(self): > self.child = None > class C2(object): > def __init__(self, parent=None): > self.parent = parent The trouble with this is that there's fully-constructed objects with no parent-child relationships. Why should you have a two-step construction process? It makes a LOT more sense to simply require a parent on construction, rather than feeding one to the other in a post-construction assignment. And if you remove the default here, your suggestion isn't materially different from what's already been posted. ChrisA From darcy at VybeNetworks.com Sun Mar 25 06:57:40 2018 From: darcy at VybeNetworks.com (D'Arcy Cain) Date: Sun, 25 Mar 2018 05:57:40 -0500 Subject: Accessing parent objects In-Reply-To: References: Message-ID: On 03/25/2018 04:37 AM, Jugurtha Hadjar wrote: > On 03/24/2018 07:14 PM, D'Arcy Cain wrote: >> class C1(dict): >> ?? class C2(object): >> ???? def f(self): >> ?????? return X['field'] >> >> O1 = C1() >> O1['field'] = 1 >> O2 = O1.C2() >> print(O2.f()) > > I prefer to *feed* the child to the parent or vice versa. Simplifies > things like testing. That was my original solution but it seems clumsy. O2 = O1.C2(O1) IOW passing the parent object to the child class. It just seems like there should be some way to access the parent object in C2. > Something like this: > > class C1(object): ... > class C2(object): > ??? def __init__(self, parent=None): > self.parent = parent Perhaps your email client is collapsing leading spaces but that isn't what I wrote. The C2 class is supposed to be a member of C1. That's why I called it as O1.C2() instead of just C2(). I was hoping that by doing so that the data in O1 would somehow be available without having to explicitly pass it as an argument. However, I would have made the parameter required. I don't see the point of calling another method to add it in later. Of course, if I do that there's no point in making C2 part of C1. What you wrote would be just fine and obvious. -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From darcy at VybeNetworks.com Sun Mar 25 07:07:55 2018 From: darcy at VybeNetworks.com (D'Arcy Cain) Date: Sun, 25 Mar 2018 06:07:55 -0500 Subject: Accessing parent objects In-Reply-To: References: Message-ID: On 03/25/2018 05:10 AM, Jugurtha Hadjar wrote: > ??????? print("I am {self.__class__.__name__} foo".format(self=self)) Unrelated to the original issue but why not one of the following? print("I am {0.__class__.__name__} foo".format(self)) print(f"I am {self.__class__.__name__} foo") -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From darcy at vex.net Sun Mar 25 07:11:53 2018 From: darcy at vex.net (D'Arcy Cain) Date: Sun, 25 Mar 2018 06:11:53 -0500 Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> Message-ID: <3c89b0c9-290c-5779-52e3-1c982b0560a3@vex.net> On 03/24/2018 06:54 PM, Steven D'Aprano wrote: >> On Saturday, March 24, 2018 at 1:20:24 PM UTC-5, D'Arcy Cain wrote: >>> I tried various forms of super() but that didn't seem to work. > > Define "doesn't see to work". It accesses the parent class. I want to access the parent object. -- D'Arcy J.M. Cain System Administrator, Vex.Net http://www.Vex.Net/ IM:darcy at Vex.Net VoIP: sip:darcy at Vex.Net From steve+comp.lang.python at pearwood.info Sun Mar 25 07:42:41 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 11:42:41 +0000 (UTC) Subject: Accessing parent objects References: Message-ID: On Sun, 25 Mar 2018 21:17:59 +1100, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 8:37 PM, Jugurtha Hadjar > wrote: [...] >> I prefer to *feed* the child to the parent or vice versa. > > Congrats, this ranks on my list of "creative people who sound like > psycho murderers". Digital artists and cooks tend to rank fairly highly > on that list. :-) >> class C1(object): >> def __init__(self): >> self.child = None >> class C2(object): >> def __init__(self, parent=None): >> self.parent = parent > > The trouble with this is that there's fully-constructed objects with no > parent-child relationships. Why should you have a two-step construction > process? Such a two-step construction process is normally called "dependency injection", and it's really useful. https://martinfowler.com/articles/injection.html Instead of class C2 controlling what its parent is: class C2: def __init__(self): self.parent = C1() the caller decides what parent to use. Provided the parent provides the same interface as C1, you can use any duck-type you like. The thing is, we use dependency injection in Python all the time. We just don't call it that! We normally write things like: class Spam: def __init__(self, queue=None): if queue is None: queue = Queue() self.queue = queue and so we get the best of both worlds: the ability to inject the queue we want when we need to, and the freedom to not bother when we don't. This is such second nature that we don't even think that what we're doing has a fancy OOP design pattern name that Java programmers stress over. And because we duck-type, we don't care if queue is a real queue, or just something that has all the queue methods we require. > It makes a LOT more sense to simply require a parent on > construction, rather than feeding one to the other in a > post-construction assignment. I tend to agree... but not everyone in the OOP world does. Quoting from Wikipedia: There are at least three ways an object can receive a reference to an external module: - constructor injection: the dependencies are provided through a class constructor. - setter injection: the client exposes a setter method that the injector uses to inject the dependency. - interface injection: the dependency provides an injector method that will inject the dependency into any client passed to it. https://en.wikipedia.org/wiki/Dependency_injection -- Steve From jugurtha.hadjar at gmail.com Sun Mar 25 07:42:45 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 12:42:45 +0100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> On 03/25/2018 11:17 AM, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 8:37 PM, Jugurtha Hadjar > wrote: >> On 03/24/2018 07:14 PM, D'Arcy Cain wrote: >>> class C1(dict): >>> class C2(object): >>> def f(self): >>> return X['field'] >>> >>> O1 = C1() >>> O1['field'] = 1 >>> O2 = O1.C2() >>> print(O2.f()) >> >> I prefer to *feed* the child to the parent or vice versa. > Congrats, this ranks on my list of "creative people who sound like > psycho murderers". Digital artists and cooks tend to rank fairly > highly on that list. Given that without prior knowledge, Python may suggest a satirical show or a huge snake, I'm right at home as either creative or a psycho. >> class C1(object): >> def __init__(self): >> self.child = None >> class C2(object): >> def __init__(self, parent=None): >> self.parent = parent > The trouble with this is that there's fully-constructed objects with > no parent-child relationships. Why should you have a two-step > construction process? It makes a LOT more sense to simply require a > parent on construction, rather than feeding one to the other in a > post-construction assignment. And if you remove the default here, your > suggestion isn't materially different from what's already been posted. > Right, I see.. What do you think about something like this: <-----------------------------------------------------------------------> class C2(object): ??? def __init__(self, parent=None): ??????? self.parent = parent ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) ??????? self.parent.foo() class C1(object): ??? def __init__(self, child_class=None): ??????? self.child = child_class(parent=self) ??? def foo(self): ??????? print("I am {self.__class__.__name__} foo".format(self=self)) c1 = C1(child_class=C2) c1.child.foo()? # I am C2 foo ??????????????? # I am C1 foo <------------------------------------------------------------------------> Check my logic: The reason I'm going through these contortions is to allow the user to choose which child class to use instead of hard coding C2 inside C1, hence the psycho feeding/injecting of C2 in C1. I can delay making a choice and I can even postpone it to run-time (say I have a web-page with a menu that asks which child class I want to use, I can select C2, or C1 and I'll have the instance created with my choice). Does this make sense and is there a more succinct way to do it this way? -- ~ Jugurtha Hadjar, From steve+comp.lang.python at pearwood.info Sun Mar 25 07:46:53 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 11:46:53 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <3c89b0c9-290c-5779-52e3-1c982b0560a3@vex.net> Message-ID: On Sun, 25 Mar 2018 06:11:53 -0500, D'Arcy Cain wrote: > On 03/24/2018 06:54 PM, Steven D'Aprano wrote: >>> On Saturday, March 24, 2018 at 1:20:24 PM UTC-5, D'Arcy Cain wrote: >>>> I tried various forms of super() but that didn't seem to work. >> >> Define "doesn't see to work". > > It accesses the parent class. I want to access the parent object. Ah. Well, no wonder it doesn't work: you're confusing the OO inheritance concept of "parent" (a superclass) with whatever relationship you have between one instance and another instance. Just because they share the same name doesn't make them the same concept. -- Steve From jugurtha.hadjar at gmail.com Sun Mar 25 07:46:55 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 12:46:55 +0100 Subject: Accessing parent objects In-Reply-To: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: <3dfbb96b-13f0-df34-6111-4b2c129fae8c@gmail.com> Erratum: "I can select C2, or C3 and I'll have the instance created with my choice" -- ~ Jugurtha Hadjar, From rantingrickjohnson at gmail.com Sun Mar 25 07:49:21 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 25 Mar 2018 04:49:21 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> Message-ID: <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> On Saturday, March 24, 2018 at 11:31:38 PM UTC-5, Steven D'Aprano wrote: > On Sat, 24 Mar 2018 20:08:47 -0700, Rick Johnson wrote: [...] > > > > the inconsistency of using super _outside_ of Tkinter code > > whilst simultaneously using explicit inheritance _inside_ > > Tkinter code was quite frankly, disturbing to me. > > "A foolish consistency is the hobgoblin of little minds, > adored by little statesmen and philosophers and divines." > > > > Thus, i chose to abandon super altogether. > > Baby, bathwater. I'm afraid quote-mining won't help your argument. There is nothing wrong about choosing consistency over inconsistency. But what i find more troubling, is that you seem to misunderstand the significance of Emerson's famous "foolish consistency" quip; which became the spearhead of a more broad treatise warning us of the dangers inherent in orthodoxy. And while Emerson's warning can be applied widely to personal philosophy, the warning applies only very narrowly to the realm of programming; which forces us to use certain structures and forms that are beyond our control. For example, a Python programmer can no more define a class with the keyword "object" than he/she could define a function with the keyword "function", or import a module from a package using colons, exclamation-marks and ampersands: >>> object Foo(): SyntaxError: invalid syntax >>> function bar(): SyntaxError: invalid syntax >>> from foolish:consistency! import &platitudes SyntaxError: invalid syntax Python dictates that we follow certain well-defined rules. But following these rules is no evidence of a foolish consistency, much less an orthodoxy. For example, here is a quote from PEP8, which underscores my position and explains the significance of the "foolish consistency" quip: """ A Foolish Consistency is the Hobgoblin of Little Minds --------------------------------------------------------- One of Guido's key insights is that code is read much more often than it is written. The guidelines provided here are intended to improve the readability of code and make it consistent across the wide spectrum of Python code. As PEP 20 says, "Readability counts". A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important. """ Okay. So if there is one lesson to take away from this quote, the lesson is that readability is important, and futhermore, that readability is highly dependant on maintaining a consistent presentation/form of source code (aka: consistency). Thus, a sporadic implementation of super() in my source code would violate the spirit of PEP8. And although i freely admit that super() is technically the correct way to go, it's not my fault that Python2's Tkinter is incompatible with super, and so my decision to maintain consistency -- and thus readability -- perfectly aligns with the standards of PEP8. Thus, it is Pythonic. So yes, I chose the path of _consistency_, for which the natural side effect is _readability_, and Steven, there is absolutely nothing "foolish" about that! > > But when i make the switch to Python4000, i will probably > > adopt super at that time, along with all the other fully > > matured goodies. > > So, having avoided the first unstable compatibility- > breaking version, Python 3000, you're going to wait until > the next hypothetical unstable, compatibility-breaking > version before upgrading? Yes. But there's more "method to my madness" than meets your eye. And although you don't know it yet, you're about to provide the justification for my delay using your very own words (so stay tuned, this is going to be fun!) (well, perhaps not for you!) > Python 3 is now six point releases in (and very soon to > have a seventh, 3.7 being in beta as we speak). It is > stable, feature-rich, and a joy to work in. As well as a > heap of great new features, there have been a metric tonne > of performance improvements, making Python 3 faster than > 2.7 for many tasks, e.g. > > [snip: link] > > Python 4 is not going to be a massively compatibility- > breaking change like Python 3 was, Exactly! Thus the reason i am avoiding Python3 like the plague. > and version 4 is likely about 5 or 6 years away. That's fine. I'm in no hurry. > Some hypothetical "Python 4000", or more likely "5000", is > at least a decade away, if it ever occurs again. (Which it > probably won't.) Which is it? "5 or 6 years" or "a decade"? Make up your mind, would ya? Even if an official version number "4" is not offered, and Python3 is the last version of Python, at some point the churn of Python3 will slow, and thus, in sprit it will become the stable equivalent of Python4. So sorry to disappoint you Steven, but either way, i win. O:-) > In the meantime, Python 2.7 will fall out of extended > support in less than two years. What? Me worry? o_O > For comparison, here's how Python's super works in 3.x: > > [...] > > Ruby's super reduces typing, at the cost of needing more > magic and less flexibility: > > [snip: link] > > Ruby's super doesn't return a superclass proxy, it > automagically calls the same method as the current method, > so you can't delegate to a different method if required. In > addition, Ruby doesn't support multiple inheritance, Ruby is fundamentally _opposed_ to the idea of multiple inheritance -- as MI is rife with issues (technical, practical, and intuitive in nature) and thus, not a wise solution -- but you would have known that Steven, had you actually read the official tutorial and had any _real_ experience with the language. Mores the pity! > which is exactly when super() is most important. Your assertion presupposes that MI is the only way to solve the problem that MI proposes to solve. And it is not only the Ruby folks who find MI to be woefully inadequate. The impracticality of MI has been the subject of much debate. > Ruby's super calls the method in two ways: > > - with no arguments, using the parenthesis-free syntax, > Ruby automagically forwards the same arguments to the > (single) parent; Which is merely a natural result of Ruby's function/method call syntax. Not allowing a parenthesis-free super call would not only be inconsistent, it would be foolishly inconsistent. > - with parentheses, you have to manually pass the arguments > you want to pass. Well, duh! Welcome to Ruby land. And please, keep your arms and legs in the vehicle at all times, and leave your preconceived notions where they belong -- where the sun don't shine. > So there's weird magic going on where `super` and `super()` > both call the method but with different arguments. Ewww. It's only weird because you are judging through a Python lens. Ruby is not Python. And Python is not Ruby. > But yes, Ruby's super requires less typing for the common > case of "call the parent's method and pass the exact same > arguments". Finally a concession! Gheez, and my only regret is that i had to pull-on my hip waders to get here. From steve+comp.lang.python at pearwood.info Sun Mar 25 08:30:44 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 12:30:44 +0000 (UTC) Subject: Accessing parent objects References: Message-ID: On Sun, 25 Mar 2018 05:57:40 -0500, D'Arcy Cain wrote: > That was my original solution but it seems clumsy. > > O2 = O1.C2(O1) Are you intentionally trying to melt my brain with horribly obfuscated, meaningless names? If so, you've succeeded admirably millennium hand and shrimp buggarit. :-) > IOW passing the parent object to the child class. It just seems like > there should be some way to access the parent object in C2. Of course there's a way, it's just tricky. There's just no *automatic* way. Classes are no different to any other object: objects only know what they hold a reference to, not what holds a reference to them. Given: class Spam: eggs = "Wibble" the class Spam knows about the string "Wibble", but the string has no way of knowing about Spam. The same applies to nested classes: class Spam: class Eggs: pass There's no difference here, except that unlike strings, classes can hold references to other objects. So we can inject a reference to Spam to Eggs, but unfortunately we can't do from inside Spam, since it doesn't exist as yet! class Spam: class Eggs: pass Eggs.owner = Spam # NameError But we can do it from the outside once the class is built: Spam.Eggs.owner = Spam Some possible solutions: - use a class decorator or a metaclass - do the injection in the Spam __init__ method: class Spam: class Eggs: pass def __init__(self): self.egg = egg = type(self).Eggs() egg.owner = self Possibly use a Weak Reference instead of a regular reference. -- Steve From bc at freeuk.com Sun Mar 25 08:32:39 2018 From: bc at freeuk.com (bartc) Date: Sun, 25 Mar 2018 13:32:39 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On 25/03/2018 02:47, Steven D'Aprano wrote: > On Sun, 25 Mar 2018 00:05:56 +0100, Peter J. Holzer wrote: > > [...] >>> yes, good idea >> >> Not if you want to avoid that string to int conversion (as you stated). >> >> That is still there, but in addition you now split the string into a >> list and then join the list into a different string. > > I'm glad I wasn't the only one who spotted that. > > There's something very curious about somebody worried about efficiency > choosing a *less* efficient solution than what they started with. To > quote W.A. Wulf: > > "More computing sins are committed in the name of efficiency (without > necessarily achieving it) than for any other single reason ? including > blind stupidity." > > As Donald Knuth observed: > > "We should forget about small efficiencies, say about 97% of the time: > premature optimization is the root of all evil." > > The Original Poster (OP) is concerned about saving, what, a tenth of a > microsecond in total? Hardly seems worth the effort, especially if you're > going to end up with something even slower. Using CPython on my machine, doing a string to int conversion that specific number took 200 times as long as doing a normal assignment. That conversion took 4 microseconds. Not significant if it's only done once. But it might be executed a million times. -- bartc From adrienoyono at gmail.com Sun Mar 25 08:40:04 2018 From: adrienoyono at gmail.com (adrien oyono) Date: Sun, 25 Mar 2018 14:40:04 +0200 Subject: A question related to the PYTHONPATH Message-ID: Hello everyone, This is my first email to the python list, I'll try my best to do it well. TL;DR I have recently read the documentation about how imports work on python, and I was wondering why, when you execute a python file, the current directory is not added by default to the PYTHONPATH ? Extended: Let's assume we have this directory structure for a python package: $ tree A A ??? B1 ? ??? b1.py ? ??? C1 ? ? ??? c1_1.py ? ? ??? c1.py ? ? ??? __init__.py ? ??? __init__.py ??? B2 ? ??? b2.py ? ??? C2 ? ? ??? c2.py ? ? ??? __init__.py ? ??? __init__.py ??? __init__.py ??? toto.py 4 directories, 11 files with the file A/B2/C2/c2.py content being: import sys print(sys.path) import A from A.B2.b2 import PLOP from A.B1.C1.c1 import TOTO print(TOTO) If I do, from the top level directory containing the package A: $ python A/B2/C2/c2.py I get an : ['/tmp/A/B2/C2', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/cris/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist- packages/gtk-2.0'] Traceback (most recent call last): File "A/B2/C2/c2.py", line 9, in import A ImportError: No module named A Which is normal according to the documentation of Python 2 (https://docs.python.org/2/tutorial/modules.html#the-module-search-path): When a module named spam is imported, the interpreter first searches for a > built-in module with that name. If not found, it then searches for a file > named spam.py in a list of directories given by the variable sys.path > . sys.path > is initialized from > these locations: > > - the directory containing the input script (or the current directory). > - PYTHONPATH > (a > list of directory names, with the same syntax as the shell variable > PATH). > - the installation-dependent default. > > After initialization, Python programs can modify sys.path > . The directory > containing the script being run is placed at the beginning of the search > path, ahead of the standard library path. This means that scripts in that > directory will be loaded instead of modules of the same name in the library > directory. This is an error unless the replacement is intended. See section Standard > Modules > for > more information. > Meanwhile when I do: $ python -m A.B2.C2.c2 I get the script running correctly: ['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/cris/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0'] 42 An empty string is now pre-pended to the value of sys.path, which means, if I correctly interpret the documentation of Python 3, that the current directory is added to the PYTHONPATH (https://docs.python.org/3/ reference/import.html#path-entry-finders): The current working directory ? denoted by an empty string ? is handled > slightly differently from other entries on sys.path > . First, if the > current working directory is found to not exist, no value is stored in > sys.path_importer_cache > . > Second, the value for the current working directory is looked up fresh for > each module lookup. Third, the path used for sys.path_importer_cache > and > returned by importlib.machinery.PathFinder.find_spec() > > will be the actual current working directory and not the empty string. > What I am interested in is what is the reason of this difference ? P.S.: - My english may suck a bit, please blame it on me being a French native speaker - Sorry if the colours hurt ? *Adrien OYONO* From auriocus at gmx.de Sun Mar 25 10:01:28 2018 From: auriocus at gmx.de (Christian Gollwitzer) Date: Sun, 25 Mar 2018 16:01:28 +0200 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: Am 25.03.18 um 14:32 schrieb bartc: > Using CPython on my machine, doing a string to int conversion that > specific number took 200 times as long as doing a normal assignment. > > That conversion took 4 microseconds. > > Not significant if it's only done once. But it might be executed a > million times. Honestly, why should it be executed a million times? Do you have a million different 400 digit numbers as constants in your code? If so, I suggest to store them in a database file accompanied with the code. If there are few different only, then don't do the conversion a million times. Convert them at module initialization and assign them to a global variable. Christian From ian.g.kelly at gmail.com Sun Mar 25 10:09:02 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 25 Mar 2018 08:09:02 -0600 Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <3c89b0c9-290c-5779-52e3-1c982b0560a3@vex.net> Message-ID: On Sun, Mar 25, 2018 at 5:46 AM, Steven D'Aprano wrote: > On Sun, 25 Mar 2018 06:11:53 -0500, D'Arcy Cain wrote: >> It accesses the parent class. I want to access the parent object. > > Ah. Well, no wonder it doesn't work: you're confusing the OO inheritance > concept of "parent" (a superclass) with whatever relationship you have > between one instance and another instance. Just because they share the > same name doesn't make them the same concept. No, he's merely confusing the concept of nested classes in Python with nested classes in Java, which actually supports this magic. To the OP: Python isn't Java. Nested classes in Python work like "static" nested classes in Java. From steve+comp.lang.python at pearwood.info Sun Mar 25 10:09:03 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 14:09:03 +0000 (UTC) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: >> - with no arguments, using the parenthesis-free syntax, >> Ruby automagically forwards the same arguments to the (single) >> parent; > > Which is merely a natural result of Ruby's function/method call syntax. > Not allowing a parenthesis-free super call would not only be > inconsistent, it would be foolishly inconsistent. I never said anything about not allowing it. But since you've gone on the defence about parens-free function calls, how is this for "consistency" in Ruby? [steve at ando ruby]$ ruby ws-example.rb a + b => 7 a+b => 7 a+ b => 7 a +b => 3 Here's the source code: # --- cut --- def a(x=4) x+2 end b = 1 print "a + b => ", (a + b), "\n" print "a+b => ", (a+b), "\n" print "a+ b => ", (a+ b), "\n" print "a +b => ", (a +b), "\n" # --- cut --- >> So there's weird magic going on where `super` and `super()` both call >> the method but with different arguments. Ewww. > > It's only weird because you are judging through a Python lens. Ruby is > not Python. And Python is not Ruby. No, its weird because in *both cases* there are no arguments given, but in one case there are no arguments passed, but in the other case, some unknown number of invisible arguments are passed. Consider a bunch of Ruby function calls: f() # calls f with no arguments f # calls f with no arguments foo() # calls foo with no arguments foo # calls foo with no arguments bar() # calls bar with no arguments bar # calls bar with no arguments super() # calls the superclass method with no arguments super # MAGIC HAPPENS! calls the superclass method with some unknown number of arguments! If you want to argue that's a useful feature, okay, I'll give you the benefit of the doubt. But useful or not, it's still weird and surprising. And deeply, fundamentally inconsistent. -- Steve From tjreedy at udel.edu Sun Mar 25 10:25:45 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2018 10:25:45 -0400 Subject: Accessing parent objects In-Reply-To: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On 3/25/2018 7:42 AM, Jugurtha Hadjar wrote: > class C2(object): > ??? def __init__(self, parent=None): > ??????? self.parent = parent Since parent is required, it should not be optional. > ??? def foo(self): > ??????? print("I am {self.__class__.__name__} foo".format(self=self)) > ??????? self.parent.foo() None.foo will raise AttributeError. > class C1(object): > ??? def __init__(self, child_class=None): > ??????? self.child = child_class(parent=self) Ditto. None() will raise TypeError If your intent is to force passing parent/child_class by name rather than by position, use *, as in def __init__(self, *, child_class): > ??? def foo(self): > ??????? print("I am {self.__class__.__name__} foo".format(self=self)) > > c1 = C1(child_class=C2) > c1.child.foo()? # I am C2 foo > ??????????????? # I am C1 foo -- Terry Jan Reedy From steve+comp.lang.python at pearwood.info Sun Mar 25 10:36:08 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 14:36:08 +0000 (UTC) Subject: Multiple Inheritance [was Re: Accessing parent objects] References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: [...] > Ruby is fundamentally _opposed_ to the idea of multiple inheritance -- > as MI is rife with issues (technical, practical, and intuitive in > nature) and thus, not a wise solution -- but you would have known that > Steven I know that many people consider multiple inheritance (MI) to be a bad idea. I would even agree with them that using MI is often over-kill, difficult to get right, and doesn't really scale well to large class hierarchies. (But then, what does?) But sometimes MI is the easiest way to get the job done. Michele Simionato has spent a *lot* of time writing about the limitations of MI, and alternatives such as mixins and traits. See his blog on Artima, a good place to start is here: https://www.artima.com/weblogs/viewpost.jsp?thread=246488 In Ruby, mixins are a special kind of thing; in Python, mixins are merely implemented as ordinary classes, using multiple inheritance. There are pros and cons to each approach. Actually, in Ruby, mixins are implemented using inheritance too, despite using distinct keywords. https://andrewbrookins.com/technology/mixins-in-python-and-ruby-compared/ [...] >> which is exactly when super() is most important. > > Your assertion presupposes that MI is the only way to solve the problem > that MI proposes to solve. I don't understand your reasoning. All I'm saying is that if your langage doesn't support MI, then super doesn't gain you much except to save a few keystrokes. Where super is really necessary is when your language supports MI, and specifically MI where the class hierarchy can form a diamond shape. > And it is not only the Ruby folks who find MI > to be woefully inadequate. The impracticality of MI has been the subject > of much debate. Indeed. And the correct word to use is *debate*. -- Steve From steve+comp.lang.python at pearwood.info Sun Mar 25 10:40:04 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 14:40:04 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <3c89b0c9-290c-5779-52e3-1c982b0560a3@vex.net> Message-ID: On Sun, 25 Mar 2018 08:09:02 -0600, Ian Kelly wrote: > On Sun, Mar 25, 2018 at 5:46 AM, Steven D'Aprano > wrote: >> On Sun, 25 Mar 2018 06:11:53 -0500, D'Arcy Cain wrote: >>> It accesses the parent class. I want to access the parent object. >> >> Ah. Well, no wonder it doesn't work: you're confusing the OO >> inheritance concept of "parent" (a superclass) with whatever >> relationship you have between one instance and another instance. Just >> because they share the same name doesn't make them the same concept. > > No, he's merely confusing the concept of nested classes in Python with > nested classes in Java, which actually supports this magic. > > To the OP: Python isn't Java. Nested classes in Python work like > "static" nested classes in Java. TIL, thanks. (I'm not sure exactly what I've learned, but I learned *something* about nested classes in Java.) -- Steve From steve+comp.lang.python at pearwood.info Sun Mar 25 10:49:44 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 14:49:44 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: > On Saturday, March 24, 2018 at 11:31:38 PM UTC-5, Steven D'Aprano wrote: [...] >> "A foolish consistency is the hobgoblin of little minds, adored by >> little statesmen and philosophers and divines." >> > >> > Thus, i chose to abandon super altogether. >> >> Baby, bathwater. > > I'm afraid quote-mining won't help your argument. There is nothing wrong > about choosing consistency over inconsistency. Depends on whether it is a foolish consistency or a sensible consistency. [...] > Python dictates that we follow certain well-defined rules. Indeed. And one of those "rules" is that super is the right way to handle inheritance in a class. Unfortunately, for historical reasons, old-style Python 1 classes don't support super. So I sympathise with your plight in regards to tkinter classes. (However that sympathy is heavily dissipated by the knowledge that this is a self-inflicted wound, due to your stubborn refusal to use Python 3.) But refusing to use super in modern, new-style classes that don't have anything to do with tkinter is precisely the sort of *foolish* consistency that Emerson was warning about. Your position is equivalent to noting that tuples don't have a sort method, and so "for the sake of consistency" refusing to use list.sort() even when you *know* you have a list. Or, and perhaps even more relevant, because *some* functions don't support keyword arguments: len(obj=arg) # TypeError for the sake of consistency you must never use keyword arguments even though your code would be improved by it: function_with_many_optional_arguments( None, None, None, None, None, None, None, None, None, None, arg ) [...] > Thus, a sporadic implementation of super() in my source code would > violate the spirit of PEP8. That's the sort of dubious reading of PEP 8 that Guido was warning against in his quote from Emerson. Just because for technical reasons class Foo cannot use super, does not mean that you should never use super in any other class for the sake of "consistency". Consistency within a single class is more important than a foolish consistency across all classes. Especially since by your own admission, you are *giving up correctness* in order to buy "consistency": > And although i freely admit that super() is > technically the correct way to go [...] >> So, having avoided the first unstable compatibility- breaking version, >> Python 3000, you're going to wait until the next hypothetical unstable, >> compatibility-breaking version before upgrading? > > Yes. But there's more "method to my madness" than meets your eye. And > although you don't know it yet, you're about to provide the > justification for my delay using your very own words [...] >> Python 4 is not going to be a massively compatibility-breaking change >> like Python 3 was, > > Exactly! Thus the reason i am avoiding Python3 like the plague. Er... you do realise that Python 4.x is going to keep compatibility with Python 3.x? Whatever reasons you think you have for avoiding Python 3, those exact same reasons will apply to Python 4. In fact, as Python continues to add more features, the distance between 2.7 and the latest version is just going to continue growing. If you're holding out under some fantasy that Python 4 is going to undo all the new features and become a de facto version 2.8, you are going to be very disappointed. >> and version 4 is likely about 5 or 6 years away. > > That's fine. I'm in no hurry. > >> Some hypothetical "Python 4000", or more likely "5000", is at least a >> decade away, if it ever occurs again. (Which it probably won't.) > > Which is it? "5 or 6 years" or "a decade"? Make up your mind, would ya? The moniker "Python X-thousand" (Python 3000, 4000, 5000...) is usually understood to mean a backwards-compatibility breaking version. Since Python 4 will *not* be such a version, what you are calling 4000 is better called "5000". If it even exists at all, it will be in the distant future, long after 4.0 and 4.1 are memories: my estimate is that nobody in the Python community will even consider another such major compatibility break until 4.7 or 4.9. And even that is a very slim chance. > Even if an official version number "4" is not offered, and Python3 is > the last version of Python, at some point the churn of Python3 will > slow, and thus, in sprit it will become the stable equivalent of > Python4. So sorry to disappoint you Steven, but either way, i win. O:-) Do you know what they call programming languages that stop evolving? "Dead." -- Steve From pfeiffer at cs.nmsu.edu Sun Mar 25 10:53:54 2018 From: pfeiffer at cs.nmsu.edu (Joe Pfeiffer) Date: Sun, 25 Mar 2018 08:53:54 -0600 Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> Message-ID: <1b605kury5.fsf@pfeifferfamily.net> ast writes: > Hi > > I found this way to put a large number in > a variable. > > C = int( > "28871482380507712126714295971303939919776094592797" > "22700926516024197432303799152733116328983144639225" > "94197780311092934965557841894944174093380561511397" > "99994215424169339729054237110027510420801349667317" > "55152859226962916775325475044445856101949404200039" > "90443211677661994962953925045269871932907037356403" > "22737012784538991261203092448414947289768854060249" > "76768122077071687938121709811322297802059565867") > > It works but is it not optimal since there is a > string to int conversion. > > I was not able to put an integer directly because > character '\' for line cut doesnt work inside an > integer > > C = \ > 28871482380507712126714295971303939919776094592797\ > 22700926516024197432303799152733116328983144639225\ > ... > 76768122077071687938121709811322297802059565867 After following the thread for a while... you will, of course, simply have to do a string to int conversion no matter what approach you take to writing it. The number is a string of digits; it has to be converted to the internal representation. Even if you write C = 2887148238050771212671429597130393991977609459279722700926516024197432303799152733116328983144639225941977803110929349655578418949441740933805615113979999421542416933972905423711002751042080134966731755152859226962916775325475044445856101949404200039904432116776619949629539250452698719329070373564032273701278453899126120309244841494728976885406024976768122077071687938121709811322297802059565867 the conversion happens. Of all the variations people have proposed to try to make the number more readable, the one you started with above seems clearest to me. From bc at freeuk.com Sun Mar 25 11:00:21 2018 From: bc at freeuk.com (bartc) Date: Sun, 25 Mar 2018 16:00:21 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: <8QOtC.408418$fK4.340865@fx17.am4> On 25/03/2018 15:01, Christian Gollwitzer wrote: > Am 25.03.18 um 14:32 schrieb bartc: >> Using CPython on my machine, doing a string to int conversion that >> specific number took 200 times as long as doing a normal assignment. >> >> That conversion took 4 microseconds. >> >> Not significant if it's only done once. But it might be executed a >> million times. > > Honestly, why should it be executed a million times? Because it's inside a function that is called a million times? Do you have a > million different 400 digit numbers as constants in your code? If so, I > suggest to store them in a database file accompanied with the code. > > If there are few different only, then don't do the conversion a million > times. Convert them at module initialization and assign them to a global > variable. That's just another workaround. You don't really want the global namespace polluted with names that belong inside functions. And you might not want to do all that initialisation on behalf of hundreds of functions that may or may not be called. Neither do you want the code at module level to be cluttered with all these giant constants. The real problem is in writing a very long constant that doesn't comfortably fit on one screen line, and where the editor used doesn't offer any help (in displaying on multiple lines) and neither does the language. -- bartc From jugurtha.hadjar at gmail.com Sun Mar 25 11:01:57 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 16:01:57 +0100 Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On 03/25/2018 03:25 PM, Terry Reedy wrote: > On 3/25/2018 7:42 AM, Jugurtha Hadjar wrote: > >> class C2(object): >> ???? def __init__(self, parent=None): >> ???????? self.parent = parent > > Since parent is required, it should not be optional. > You can still call it the way you'd call it if it were a positional argument. i.e: def foo(keyword_argument=None): ??? print(keyword_argument) foo(3) foo(keyword_argument=3) Just because it's a keyword argument doesn't mean it's not required. I could have provided a default using `self.parent = parent or C2`, but I didn't want to assume C2 was defined in that namespace and wanted to give as generic a code as I could. Furthermore, the only case I'd use a positional argument is if I were 100% certain the code will not change, which I'm not. Plus when the API will change (and it will), it will be harder to deal with it that if it were a keyword argument. If you change the order you'd have to change the whole codebase where the class/function is instantiated/called, whereas if you use a keyword argument, you could just catch the old way call, issue a deprecation warning, then route the call for the new way call. What benefit does the positional argument provide? >> ???? def foo(self): >> ???????? print("I am {self.__class__.__name__} foo".format(self=self)) >> ???????? self.parent.foo() > > None.foo will raise AttributeError. > Right.. As I said, I tried to assume as little as possible about OP's code and namespace. Didn't want to include C1 in __init__ signature because I wasn't sure it was available in the namespace. It is easy to address, though: Example A: class C2(object): ???? def __init__(self, parent=C2): ???????? self.parent = parent Example B: class C2(object): ??? def __init__(self, parent=None): ??????? self.parent = parent or C2 Furthermore, having a positional argument will not save us. We can still break the code if we do the following: class C2(object): ??? def __init__(self, parent): ??????? self.parent = parent ??? def foo(self): ??????? self.parent.foo() c1 = C1() c2 = C2(None) c2.foo() Making it positional didn't fix our wickedness. >> class C1(object): >> ???? def __init__(self, child_class=None): >> ???????? self.child = child_class(parent=self) > > Ditto.? None() will raise TypeError > Ditto, so would it if your C2 instance is None for whatever reason. > If your intent is to force passing parent/child_class by name rather > than by position, use *, as in > ??? def __init__(self, *, child_class): > > >> ???? def foo(self): >> ???????? print("I am {self.__class__.__name__} foo".format(self=self)) >> >> c1 = C1(child_class=C2) >> c1.child.foo()? # I am C2 foo >> ???????????????? # I am C1 foo > > I don't want to force using named params vs keyword params, or I would have used the asterisk in the first place. I write that code, as I said, because I have zero to gain from using a positional argument (which using keyword arguments behaves the same way as far as I'm concerned if I call the params in order), and a lot to lose: when I change the API, I'd have to change the whole parts where I instantiate the class or call the method. Suppose I have a class like this: class Foo(object): ??? def __init__(self, name, height, address): ??????? self.name = name ??????? self.height = height ??????? self.address = address Suppose I then want to change it to: class Foo(object): ??? def __init__(self, status, name, address, height): ??????? self.status = status ??????? self.name = name ??????? self.height = height ??????? self.address = address This would totally screw up other parts of the code where the class is used. I tend to use positional arguments when it's *way* unlikely the code will ever change, and even then I probably wouldn't. -- ~ Jugurtha Hadjar, From jugurtha.hadjar at gmail.com Sun Mar 25 11:03:03 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 16:03:03 +0100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: <721af2a8-d8ab-0ffb-d44d-7de4fbc20523@gmail.com> On 03/25/2018 11:57 AM, D'Arcy Cain wrote: > >> Something like this: >> >> class C1(object): > ... >> class C2(object): >> ??? def __init__(self, parent=None): >> self.parent = parent > Perhaps your email client is collapsing leading spaces but that isn't > what I wrote. The C2 class is supposed to be a member of C1. That's > why I called it as O1.C2() instead of just C2(). I was hoping that by > doing so that the data in O1 would somehow be available without having > to explicitly pass it as an argument. It's not my mail client, this was my proposed code in which C2 is defined *outside* of C1, then you inject one or the other (can't decide for you as it depends on what you're trying to do). In other words, it's not an indentation error. > However, I would have made the parameter required. I don't see the > point of calling another method to add it in later. Of course, if I do > that there's no point in making C2 part of C1. What you wrote would be > just fine and obvious. > I tend to use keyword arguments a lot because it gives me a lot of freedom. Just a perk: I can do things like change the API and I'd only have to catch the call, issue a deprecation warning, and then use the new API instead of having the code be rigid. -- ~ Jugurtha Hadjar, From flitterio at gmail.com Sun Mar 25 11:05:17 2018 From: flitterio at gmail.com (Fran Litterio) Date: Sun, 25 Mar 2018 11:05:17 -0400 Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <1b605kury5.fsf@pfeifferfamily.net> Message-ID: On 3/25/2018 10:53 AM, Joe Pfeiffer wrote: > After following the thread for a while... you will, of course, simply > have to do a string to int conversion no matter what approach you take > to writing it. The number is a string of digits; it has to be converted > to the internal representation. Even if you write > > C = 2887148238050771212671429597130393991977609459279722700926516024197432303799152733116328983144639225941977803110929349655578418949441740933805615113979999421542416933972905423711002751042080134966731755152859226962916775325475044445856101949404200039904432116776619949629539250452698719329070373564032273701278453899126120309244841494728976885406024976768122077071687938121709811322297802059565867 > > the conversion happens. But doesn't that string-to-int conversion happen at compile-time not run-time? -- Fran From bc at freeuk.com Sun Mar 25 11:06:51 2018 From: bc at freeuk.com (bartc) Date: Sun, 25 Mar 2018 16:06:51 +0100 Subject: Entering a very large number In-Reply-To: <1b605kury5.fsf@pfeifferfamily.net> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <1b605kury5.fsf@pfeifferfamily.net> Message-ID: On 25/03/2018 15:53, Joe Pfeiffer wrote: > ast writes: >> C = int( >> "28871482380507712126714295971303939919776094592797" >> "22700926516024197432303799152733116328983144639225" >> "94197780311092934965557841894944174093380561511397" >> "99994215424169339729054237110027510420801349667317" >> "55152859226962916775325475044445856101949404200039" >> "90443211677661994962953925045269871932907037356403" >> "22737012784538991261203092448414947289768854060249" >> "76768122077071687938121709811322297802059565867") > After following the thread for a while... you will, of course, simply > have to do a string to int conversion no matter what approach you take > to writing it. What, even with you write this: C = 12 ? > The number is a string of digits; it has to be converted > to the internal representation. Which is usually done by a compiler, and it will only do it once not each time the line is encountered in a running program. (Although a compiler will also do the conversion when the line is never executed!) > Even if you write > > C = 2887148238050771212671429597130393991977609459279722700926516024197432303799152733116328983144639225941977803110929349655578418949441740933805615113979999421542416933972905423711002751042080134966731755152859226962916775325475044445856101949404200039904432116776619949629539250452698719329070373564032273701278453899126120309244841494728976885406024976768122077071687938121709811322297802059565867 > > the conversion happens. But not at runtime. -- bartc From ian.g.kelly at gmail.com Sun Mar 25 11:16:19 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 25 Mar 2018 09:16:19 -0600 Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On Sun, Mar 25, 2018 at 9:01 AM, Jugurtha Hadjar wrote: > > On 03/25/2018 03:25 PM, Terry Reedy wrote: >> >> On 3/25/2018 7:42 AM, Jugurtha Hadjar wrote: >> >>> class C2(object): >>> def __init__(self, parent=None): >>> self.parent = parent >> >> >> Since parent is required, it should not be optional. >> > > You can still call it the way you'd call it if it were a positional > argument. > > i.e: > > def foo(keyword_argument=None): > print(keyword_argument) > > foo(3) > foo(keyword_argument=3) > > > Just because it's a keyword argument doesn't mean it's not required. I could > have provided a default using `self.parent = parent or C2`, but I didn't > want to assume C2 was defined in that namespace and wanted to give as > generic a code as I could. You misunderstand what keyword arguments are. Giving the argument a default does not make it a keyword argument. Passing the argument with keyword syntax is what makes it a keyword argument. It happens at the call site, not when it's defined (with the exception of keyword-only arguments, which this is not). This function takes an argument x either by position or by keyword: def foo(x): pass This function also takes an argument x either by position or by keyword: def foo(x=None): pass The only difference is that the latter makes the argument optional by assigning a default, whereas the former requires it. The point is, if the argument is effectively required and doesn't have a sensible default, then don't make it optional. You can still pass it by keyword if that's what you want. From ian.g.kelly at gmail.com Sun Mar 25 11:31:06 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Sun, 25 Mar 2018 09:31:06 -0600 Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On Sun, Mar 25, 2018 at 9:01 AM, Jugurtha Hadjar wrote: > > On 03/25/2018 03:25 PM, Terry Reedy wrote: >> None.foo will raise AttributeError. >> > > Right.. As I said, I tried to assume as little as possible about OP's code > and namespace. Didn't want to include C1 in __init__ signature because I > wasn't sure it was available in the namespace. > > It is easy to address, though: > > Example A: > > class C2(object): > def __init__(self, parent=C2): > self.parent = parent This doesn't work as written (try it!) > Furthermore, having a positional argument will not save us. We can still > break the code if we do the following: > > class C2(object): > def __init__(self, parent): > self.parent = parent > def foo(self): > self.parent.foo() > > c1 = C1() > c2 = C2(None) > c2.foo() > > Making it positional didn't fix our wickedness. The difference is that in this case the fault is on the caller for passing a nonsense value. In the original the fault is on the class author for providing a useless default and implying that the argument is not required. From tjreedy at udel.edu Sun Mar 25 11:33:59 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 25 Mar 2018 11:33:59 -0400 Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On 3/25/2018 11:01 AM, Jugurtha Hadjar wrote: > > On 03/25/2018 03:25 PM, Terry Reedy wrote: >> On 3/25/2018 7:42 AM, Jugurtha Hadjar wrote: >> >>> class C2(object): >>> ???? def __init__(self, parent=None): >>> ???????? self.parent = parent >> >> Since parent is required, it should not be optional. >> > > You can still call it the way you'd call it if it were a positional > argument. You totally missed my point. Given the definition above, 'parent' can be passed either way, but self.parent must be not be None and therefore 'parent' should not have the unreplaced default value None. It should either have a legal default value or no default value. -- Terry Jan Reedy From grant.b.edwards at gmail.com Sun Mar 25 11:47:22 2018 From: grant.b.edwards at gmail.com (Grant Edwards) Date: Sun, 25 Mar 2018 15:47:22 +0000 (UTC) Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On 2018-03-25, bartc wrote: > On 25/03/2018 02:47, Steven D'Aprano wrote: > >> The Original Poster (OP) is concerned about saving, what, a tenth of a >> microsecond in total? Hardly seems worth the effort, especially if you're >> going to end up with something even slower. > > Using CPython on my machine, doing a string to int conversion that > specific number took 200 times as long as doing a normal assignment. > > That conversion took 4 microseconds. > > Not significant if it's only done once. But it might be executed a > million times. Which adds up to 4 seconds. Still not worth spending hours (or even a few minutes) to optimize. -- Grant From pfeiffer at cs.nmsu.edu Sun Mar 25 11:57:12 2018 From: pfeiffer at cs.nmsu.edu (Joe Pfeiffer) Date: Sun, 25 Mar 2018 09:57:12 -0600 Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <1b605kury5.fsf@pfeifferfamily.net> Message-ID: <1b1sg8up0n.fsf@pfeifferfamily.net> bartc writes: > On 25/03/2018 15:53, Joe Pfeiffer wrote: >> ast writes: > >>> C = int( >>> "28871482380507712126714295971303939919776094592797" >>> "22700926516024197432303799152733116328983144639225" >>> "94197780311092934965557841894944174093380561511397" >>> "99994215424169339729054237110027510420801349667317" >>> "55152859226962916775325475044445856101949404200039" >>> "90443211677661994962953925045269871932907037356403" >>> "22737012784538991261203092448414947289768854060249" >>> "76768122077071687938121709811322297802059565867") > >> After following the thread for a while... you will, of course, simply >> have to do a string to int conversion no matter what approach you take >> to writing it. > > What, even with you write this: > > C = 12 > > ? I'm having a hard time parsing that sentence. If you're pointing out that when someone writes an int of reasonable length they don't mess with strings and explicit int conversions, of course you're right. This question is specficially about "very large" numbers. >> The number is a string of digits; it has to be converted >> to the internal representation. > > Which is usually done by a compiler, and it will only do it once not > each time the line is encountered in a running program. (Although a > compiler will also do the conversion when the line is never executed!) If you're using compiled python, yes (though I don't really know how the bytecode represents long numbers like this, or whether it would optimize away the explicit int conversion). Since it's a constant string, it would be a *really* easy optimization. >> Even if you write >> >> C = >> 2887148238050771212671429597130393991977609459279722700926516024197432303799152733116328983144639225941977803110929349655578418949441740933805615113979999421542416933972905423711002751042080134966731755152859226962916775325475044445856101949404200039904432116776619949629539250452698719329070373564032273701278453899126120309244841494728976885406024976768122077071687938121709811322297802059565867 >> >> the conversion happens. > > But not at runtime. If it's not in a loop, only once at runtime in any case. Does the interpreter save away constants as it goes, so it would only have to be done once in any case? From joepareti54 at gmail.com Sun Mar 25 12:15:09 2018 From: joepareti54 at gmail.com (joseph pareti) Date: Sun, 25 Mar 2018 18:15:09 +0200 Subject: issues when buidling python3.* on centos 7 Message-ID: I have a VM in the cloud running centos that comes with python 2.7 installed on it. For my purpose, I need python 3.5 (or 6?) However there is an issue with urlgrabber, and I believe this is due to inconsistencies with the 2 python versions, whcih I am not able to resolve. DETAILS INSTALL PYTHON 3 from scratch https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-local-programming-environment-on-centos-7 change /usr/bin/yum because yum does not support python 3 so force it to use python 2 in first line, else you get this: except KeyboardInterrupt, e: edits in /usr/libexec/urlgrabber-ext-down to remove syntax errors sudo yum -y install https://centos7.iuscommunity.org/ius-release.rpm [joepareti54 at xxx ~]$ sudo yum -y install python36u Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * epel: ftp.nluug.nl * ius: mirror.amsiohosting.net Traceback (most recent call last): File "/usr/libexec/urlgrabber-ext-down", line 22, in from urlgrabber.grabber import \ ModuleNotFoundError: No module named 'urlgrabber' Exiting on user cancel [joepareti54 at xxx ~]$ therefore wget http://mirror.centos.org/centos/7/os/x86_64/Packages/python-urlgrabber-3.10-8.el7.noarch.rpm sudo yum install python-urlgrabber-3.10-8.el7.noarch.rpm now rerun sudo yum -y install python36u basically same as here https://www.centos.org/forums/viewtopic.php?t=52325 the urlgrabber is not found The following may give a clue because of inconsistent python versions: [joepareti54 at xxx ~]$ python -V Python 3.5.2 :: Anaconda 4.3.0 (64-bit) [joepareti54 at xxx ~]$ rpm -ql python-urlgrabber | xargs -i dirname '{}' | sort | uniq | grep site-packages /usr/lib/python2.7/site-packages /usr/lib/python2.7/site-packages/urlgrabber [joepareti54 at xxx ~]$ so the install of python-urlgrabber-3.10-8.el7.noarch.rpm landed in the wrong place as suggested here https://forums.fedoraforum.org/showthread.php?95680-yum-broken-(-quot-No-module-named-urlgrabber-quot-) and the inconsistency explains why the grabber module is not found From bc at freeuk.com Sun Mar 25 12:30:33 2018 From: bc at freeuk.com (bartc) Date: Sun, 25 Mar 2018 17:30:33 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On 25/03/2018 16:47, Grant Edwards wrote: > On 2018-03-25, bartc wrote: >> On 25/03/2018 02:47, Steven D'Aprano wrote: >> >>> The Original Poster (OP) is concerned about saving, what, a tenth of a >>> microsecond in total? Hardly seems worth the effort, especially if you're >>> going to end up with something even slower. >> >> Using CPython on my machine, doing a string to int conversion that >> specific number took 200 times as long as doing a normal assignment. >> >> That conversion took 4 microseconds. >> >> Not significant if it's only done once. But it might be executed a >> million times. > > Which adds up to 4 seconds. > > Still not worth spending hours (or even a few minutes) to optimize. If this is a program that will only ever be used by one person, and that person will only ever run it once, and you know that bit is executed 1M time and not 50M, then you might be right. Remember that 4 seconds might be on top of dozens of other things that don't appear to be worth optimising. The chances are however that a program in development might be run hundreds or thousands of times. -- bartc From jugurtha.hadjar at gmail.com Sun Mar 25 13:17:30 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 18:17:30 +0100 Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: <1cd171f0-b4e8-1f60-96bb-2f53d82b3709@gmail.com> On 03/25/2018 04:31 PM, Ian Kelly wrote: > On Sun, Mar 25, 2018 at 9:01 AM, Jugurtha Hadjar > wrote: >> On 03/25/2018 03:25 PM, Terry Reedy wrote: >>> None.foo will raise AttributeError. >>> >> Right.. As I said, I tried to assume as little as possible about OP's code >> and namespace. Didn't want to include C1 in __init__ signature because I >> wasn't sure it was available in the namespace. >> >> It is easy to address, though: >> >> Example A: >> >> class C2(object): >> def __init__(self, parent=C2): >> self.parent = parent > This doesn't work as written (try it!) Sorry, I was writing on a my commute and made a mistake. I meant parent=C1. >> Furthermore, having a positional argument will not save us. We can still >> break the code if we do the following: >> >> class C2(object): >> def __init__(self, parent): >> self.parent = parent >> def foo(self): >> self.parent.foo() >> >> c1 = C1() >> c2 = C2(None) >> c2.foo() >> >> Making it positional didn't fix our wickedness. > The difference is that in this case the fault is on the caller for > passing a nonsense value. In the original the fault is on the class > author for providing a useless default and implying that the argument > is not required. The point about the default can be found in my previous answer: If it's my code and I know the object I want to use as a default lives in the module, I'll provide it as a default parameter, but we're talking about C1 and C2 in an abstract way and I chose to make the fewest assumptions possibles (maybe too few) OP's code, which may have made the keyword argument/positional argument/optional argument/None bubble up. My main point in my original reply: I much prefer restricting as little as possible and delaying instantiation as much as possible which has helped me a lot, especially with testing (for when I don't have access to a device represented by the class, or trying to test offline), making the library usage easier for myself and others, and avoiding a lot of problems down-range (for instance, hot swapping a hardware connection after class instantiation is easier when code it's written that way as opposed to have it as a required argument). There sure is scar tissue and it may not be to everyone's taste, but it has served me well. Again, I do provide defaults when it's code I write, but I can't guess what the original code does so I may have fell into muscle memory. -- ~ Jugurtha Hadjar, From none at gmail.com Sun Mar 25 13:18:23 2018 From: none at gmail.com (ast) Date: Sun, 25 Mar 2018 19:18:23 +0200 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: <5ab7d9f5$0$4812$426a74cc@news.free.fr> Le 25/03/2018 ? 03:47, Steven D'Aprano a ?crit?: > On Sun, 25 Mar 2018 00:05:56 +0100, Peter J. Holzer wrote: > > > The Original Poster (OP) is concerned about saving, what, a tenth of a > microsecond in total? Hardly seems worth the effort, especially if you're > going to end up with something even slower. > > I regret that many answers are malicious, as your. The question was how to enter a large number without going through a string, no matter why. This question is absolutely legitimate, contrary to some people think. It seems that it is not feasible unless writting it in a single line, which is awful. OK, go on with the string, and Wolfgang solution using triple quote is good. From jugurtha.hadjar at gmail.com Sun Mar 25 13:26:04 2018 From: jugurtha.hadjar at gmail.com (Jugurtha Hadjar) Date: Sun, 25 Mar 2018 18:26:04 +0100 Subject: Accessing parent objects In-Reply-To: References: Message-ID: <19842a4d-6aaa-329a-947b-31f095efc0b1@gmail.com> On 03/25/2018 12:07 PM, D'Arcy Cain wrote: > On 03/25/2018 05:10 AM, Jugurtha Hadjar wrote: >> ??????? print("I am {self.__class__.__name__} foo".format(self=self)) > Unrelated to the original issue but why not one of the following? > > print("I am {0.__class__.__name__} foo".format(self)) I prefer keyword arguments, but if I used it that way I'd do: print("I am {0} foo".format(self.__class__.__name__)) self.__class__.__name__ looks better (+ makes more sense to me) than 0.__class__.__name__. > print(f"I am {self.__class__.__name__} foo") > f-strings are only available in python 3.6 as far as I know and. -- ~ Jugurtha Hadjar, From rantingrickjohnson at gmail.com Sun Mar 25 13:33:49 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 25 Mar 2018 10:33:49 -0700 (PDT) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sunday, March 25, 2018 at 9:11:35 AM UTC-5, Steven D'Aprano wrote: > On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: [...] > I never said anything about not allowing it. But since > you've gone on the defence about parens-free function > calls, how is this for "consistency" in Ruby? > > [steve at ando ruby]$ ruby ws-example.rb > a + b => 7 > a+b => 7 > a+ b => 7 > a +b => 3 > > Here's the source code: > > # --- cut --- > def a(x=4) > x+2 > end > > b = 1 > print "a + b => ", (a + b), "\n" > print "a+b => ", (a+b), "\n" > print "a+ b => ", (a+ b), "\n" > print "a +b => ", (a +b), "\n" > # --- cut --- Short of demonstrating that you have knack for writing obfuscated code :-), i don't see the point here. One could easily induce the same level of obfuscation by redefining numeric operators in an unintuitive manner. But, if your point is that Ruby's "parens-free function calls" can be confusing in some highly contrived cases in which the programmer can't be bothered to write code in a manner that is intuitive, well then, all we have to do is add the parens, and poof!, we're back in the land of sanity again. Here, i took the liberty of un-obfuscating your code by doing the kind of back-breaking manual labor known only to the unfortunate inhabitants of Soviet-era gulags, and i did it by mererly adding the empty parens to each of your method calls. Here goes: UGGGGH! > print "a() + b => ", (a + b), "\n" ARRRRG! > print "a()+b => ", (a+b), "\n" OOOOOO! > print "a()+ b => ", (a+ b), "\n" EEEEEE! > print "a() +b => ", (a +b), "\n" (wipes-sweat-from-brow) Oh boy. That was tough. > No, its weird because in *both cases* there are no > arguments given, but in one case there are no arguments > passed, but in the other case, some unknown number of > invisible arguments are passed. > > Consider a bunch of Ruby function calls: > > f() # calls f with no arguments > f # calls f with no arguments > foo() # calls foo with no arguments > foo # calls foo with no arguments > bar() # calls bar with no arguments > bar # calls bar with no arguments > super() # calls the superclass method with no arguments > super # MAGIC HAPPENS! calls the superclass method with > some unknown number of arguments! The "unknown number of arguments" is equivalent to whatever is defined by the method. In this form, Ruby's "arg-less- super" is saving you the trouble of repeating code in a confined space (aka: method body). Because hey, misspellings can and do happen ya know. And if the intent of a method is merely to pass along the arguments _verbatim_ -- which is a common pattern utilized by OO programmers -- then why bother with all the superfluous typing and risk the spelling errors? Besides, Ruby does not abide by the Pythonic concept of "explicit is better than implicit"; and there is nothing wrong with that. Futhermore, since when did you assume that Ruby's super was a standard func/meth call? Because, it isn't. The only overlap in consistency between them, is that, as is the case with Ruby methods, you can call Ruby's super with or without parenthesis. But the similarities stop there. `super` (more generally) is a fundamental feature of most OO languages which provides a ubiquitous methodology for access to the superclass object. Now, how exactly that feature is implemented in each OO language depends greatly on the needs and opinions of the devs/users of said language. So of course Ruby's super will not behave _exactly_ like a Ruby function, nor should it! Neither should it be assumed to work exactly like Python's super, because now we're talking about a completely different language with completely different design philosophy. However, feel free to quote from any official source which argues (hopefully compellingly) that Ruby's super somehow fundamentally violates the nature of what is expected from an OO language. Finally, you forgot to show the third (explicit argument passing form) of Ruby's super: super(ag1, ar2, ..., argN) > If you want to argue that's a useful feature, okay, I'll > give you the benefit of the doubt. But useful or not, it's > still weird and surprising. And deeply, fundamentally > inconsistent. I'm not here to protect Ruby. But if Ruby's super is as "weird and surprising" and "deeply, fundamentally inconsistent" as you claim it to be, well then, you've failed to provide sufficient evidence here. We are entitled to our own opinions, Steven. But we are not entitled to our own facts. From rantingrickjohnson at gmail.com Sun Mar 25 14:34:16 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 25 Mar 2018 11:34:16 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: <291e5876-88de-8fee-d293-1d8e380982b7@gmail.com> Message-ID: On Sunday, March 25, 2018 at 10:02:20 AM UTC-5, Jugurtha Hadjar wrote: [...] > Furthermore, the only case I'd use a positional argument is > if I were 100% certain the code will not change, which I'm > not. And short of you possessing a crystal ball in good working order (mine's currently in the shop!), you must always assume that code will grow over time, and thus, write your code in a manner that is as easy to extend and maintain as possible. > Plus when the API will change (and it will), it will be > harder to deal with it that if it were a keyword argument. Huh? Was that a typo? Did you mean "_not_ a kwarg"??? > If you change the order you'd have to change the whole > codebase where the class/function is instantiated/called, If the args are positional, then yes. > whereas if you use a keyword argument, you could just catch > the old way call, issue a deprecation warning, then route > the call for the new way call. Whew! It _was_ a typo. ':-) > [...] > > Suppose I have a class like this: > > class Foo(object): > def __init__(self, name, height, address): > self.name = name > self.height = height > self.address = address > > Suppose I then want to change it to: > > class Foo(object): > def __init__(self, status, name, address, height): > self.status = status > self.name = name > self.height = height > self.address = address > > > This would totally screw up other parts of the code where > the class is used. I tend to use positional arguments when > it's *way* unlikely the code will ever change, and even > then I probably wouldn't. Positional arguments are not without their practical advantages though. For instance: you don't have to manually unpack the arguments into local vars, plus, you can let Python handle the underflow/overflow of arguments. Of course, the con is that positional arguments do not scale nicely; which leads to nasty backwards incompatible issues. It's a tough call, really. Because as programmers we are inclined to be naturally lazy (it's a virtue!), and thus, we'd prefer to offload as much work as we can on the processor. But if saving a few keystrokes now means hours of painful search&replace sessions later, hmm, probably best to choose the path with the best long-term outcome. From hjp-python at hjp.at Sun Mar 25 15:25:10 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 25 Mar 2018 21:25:10 +0200 Subject: Entering a very large number In-Reply-To: <5ab7d9f5$0$4812$426a74cc@news.free.fr> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <5ab7d9f5$0$4812$426a74cc@news.free.fr> Message-ID: <20180325192510.lwhge6uywuarkzot@hjp.at> On 2018-03-25 19:18:23 +0200, ast wrote: > Le 25/03/2018 ? 03:47, Steven D'Aprano a ?crit?: > > The Original Poster (OP) is concerned about saving, what, a tenth of a > > microsecond in total? Hardly seems worth the effort, especially if you're > > going to end up with something even slower. > > > > > > I regret that many answers are malicious, as your. I just looked up "malicious" in Merriam-Webster, just in case it has some meaning I wasn't previously aware of. Nope. It says: : having or showing a desire to cause harm to someone : given to, marked by, or arising from malice and malice is defined as: : desire to cause pain, injury, or distress to another : intent to commit an unlawful act or cause harm without legal justification or excuse Can you explain, why you think that Steven (or anybody else in this thread) wants to cause you harm, pain, injury or distress, or how his answer constitutes an unlawful act? > The question was how to enter a large number without > going through a string, no matter why. And the answer is: Write it all on one line. > This question is absolutely legitimate, contrary to > some people think. It certainly is. OTOH, the question why you would want to do this is also legitimate. So is the question why you think that the few microseconds it takes to parse the string are not acceptable. And finally: > It seems that it is not feasible unless writting it in > a single line, which is awful. > OK, go on with the string, and Wolfgang solution using > triple quote is good. If you think that your solution is not good enough, why do you think Wolfgang's solution is better? It still converts a string to an int and it does additional operations on the string. So it is slower and harder to understand than what you already had. (It also cannot be properly indented, but you may not care about that.) hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From hjp-python at hjp.at Sun Mar 25 15:50:27 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 25 Mar 2018 21:50:27 +0200 Subject: Accessing parent objects In-Reply-To: References: <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: <20180325195027.672qyouuaip7ux62@hjp.at> On 2018-03-25 14:49:44 +0000, Steven D'Aprano wrote: > The moniker "Python X-thousand" (Python 3000, 4000, 5000...) is usually > understood to mean a backwards-compatibility breaking version. Since > Python 4 will *not* be such a version, what you are calling 4000 is > better called "5000". If Python 4.0 will "merely be the release that comes after Python 3.9", as Nick Coghlan writes in https://opensource.com/life/14/9/why-python-4-wont-be-python-3, why will it be called 4.0 and not 3.10? Just to avoid two-digit minor numbers? hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rantingrickjohnson at gmail.com Sun Mar 25 16:30:14 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 25 Mar 2018 13:30:14 -0700 (PDT) Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: <3714df17-1a61-44ef-9c42-c46fdaeeb489@googlegroups.com> On Sunday, March 25, 2018 at 9:52:30 AM UTC-5, Steven D'Aprano wrote: > On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: [...] > But refusing to use super in modern, new-style classes that > don't have anything to do with tkinter is precisely the > sort of *foolish* consistency that Emerson was warning > about. Oh, obviously! Because in the late 1800's, Python's new style vs old style classes, and the blinkered state of Tkinter, was the subject of intense debate in philosophical circles. Yes-sir-eee! > Your position is equivalent to noting that tuples don't > have a sort method, and so "for the sake of consistency" > refusing to use list.sort() even when you *know* you have a > list. That's a BS equivalence and you know it. Tkinter's lack of new style classes was a design flaw, whereas, tuples are immutable and a sort method would violate the concept of immutability. > Or, and perhaps even more relevant, because *some* > functions don't support keyword arguments [...] for the > sake of consistency you must never use keyword arguments > even though your code would be improved by it For some definition of "more relevant", i suppose. > > Thus, a sporadic implementation of super() in my source > > code would violate the spirit of PEP8. > > That's the sort of dubious reading of PEP 8 that Guido was > warning against in his quote from Emerson. Just because > for technical reasons class Foo cannot use super, does not > mean that you should never use super in any other class for > the sake of "consistency". Consistency within a single > class is more important than a foolish consistency across > all classes. You cannot even handle basic reading and comprehension, can you? And i quote _again_ from PEP8: """ A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important. """ Using that paragraph as a guide, we can form the following hierarchal table: Consistency with every word of PEP8 (important) Consistency within a project (more important) Consistency within "scopes" (most important) My choice to abandon Python's super() in favor of consistency most directly aligns with the moderate #2 (Consistency within a project). You see Steven, not only is a single script considered a project; and not *ONLY* is a package considered a project; but yes, even the aggregate of all scripts composing an entire library (aka: ricklib!) is a project. Any questions? > Especially since by your own admission, you are *giving up > correctness* in order to buy "consistency": "sacrificing correctness" only in a purely ideological sense. Explicit inheritance (aka: hard-coding the superclass symbol) works just as well as implicit inheritance (aka: relying on the "resolution magic" of super). Of course, in the former case, a slight bit of onerous is placed on the programmer because he/she must know the exact spelling of the superclass symbol, whereas, OO's `super` is nothing but a lazy shortcut which looks up and returns the symbol for you. Thus, using super is similar to using a remote control to open your garage door as opposed to punching in a code. Either method will open the door just fine. The remote merely removes the need to remember a code. It's really not that complicated Steven... From hba008 at aol.com Sun Mar 25 16:31:24 2018 From: hba008 at aol.com (hba008 at aol.com) Date: Sun, 25 Mar 2018 16:31:24 -0400 Subject: Code for Using Keyboard to Play music notes "ABCDEFG" Message-ID: <1625edbc5ae-c86-17f4c@webjas-vaa145.srv.aolmail.net> Pyton Friends, Do you have any code that will play the notes "ABCDEFG" from my computer keyboard when a key is pressed ? For example if I press the "a" key the note "a" will sound out of my speaker. Thanks, BigB From hjp-python at hjp.at Sun Mar 25 17:29:07 2018 From: hjp-python at hjp.at (Peter J. Holzer) Date: Sun, 25 Mar 2018 23:29:07 +0200 Subject: [OT] multicore/cpu history In-Reply-To: <5A8AB7870200001B0001DF67@smtp1.astron.nl> References: <5A8AB7870200001B0001DF67@smtp1.astron.nl> Message-ID: <20180325212907.wu545d46ajru6waj@hjp.at> On 2018-02-19 12:39:51 +0100, Adriaan Renting wrote: > >>> On 17-2-2018 at 22:02, in message > , > Chris Angelico wrote: > > On Sun, Feb 18, 2018 at 5:05 AM, Steven D'Aprano > > wrote: > >> If you're talking about common desktop computers, I think you're > >> forgetting how recent multicore machines actually are. I'm having > >> difficulty finding when multicore machines first hit the market, > >> but it seems to have been well into the 21st century -- perhaps as > >> late as 2006 with the AMD Athelon 64 X2: > > > > No, I'm talking about big iron. Has Python been running on multi-CPU > > supercomputers earlier than that? > > > >> By the way, multiple CPU machines are different from CPUs with > >> multiple cores: > >> > >> http://smallbusiness.chron.com/multiple-cpu-vs-multicore-33195.html > > > > Yeah, it was always "multiple CPUs", not "multiple cores" when I was > > growing up. Yes, but the difference is only an implementation detail. Once chips became large enough that you could fit multiple cores on a single chip it became cheaper (and faster) to do this than put each core on a separate chip. Just like modern CPUs may contain a floating point unit, a memory controller, SATA and ethernet controllers, etc. when previously those would have been separate chips (or even separate cards). > > And it was only ever in reference to the expensive > > hardware that I could never even dream of working with. I was always > > on the single-CPU home-grade systems. > > > > Multicore became a thing with the Pentium 4 hyperthreading around ~2002 > for consumers, and > multi cpu was a thing much longer, even with "consumer grade" > hardware: > > I remember running 2 Mendocino 300 MHz Celerons on a Pentium II Xeon > motherboard to get a > multi-cpu machine for running multiple virtual machines for testing > purposes around 1998. I I bought my first multi-processor computer (a dual pentium-90) in 1995. AFAIK the Pentium was the first Intel processor intended for SMP (there were multiprocessor systems based on earlier intel processors but they used custom chipsets and were very much not consumer-grade). > This was not as Intel intended, but a quite cheap consumer grade > hardware solution. My dual pentium wasn't cheap, but the motherboard and the second processor weren't that expensive. The 21" monitor OTOH ... and did it have a tape drive? I don't remember, but I think it did. hp -- _ | Peter J. Holzer | we build much bigger, better disasters now |_|_) | | because we have much more sophisticated | | | hjp at hjp.at | management tools. __/ | http://www.hjp.at/ | -- Ross Anderson -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From rosuav at gmail.com Sun Mar 25 17:58:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 08:58:36 +1100 Subject: Accessing parent objects In-Reply-To: <3714df17-1a61-44ef-9c42-c46fdaeeb489@googlegroups.com> References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <3714df17-1a61-44ef-9c42-c46fdaeeb489@googlegroups.com> Message-ID: On Mon, Mar 26, 2018 at 7:30 AM, Rick Johnson wrote: >> Especially since by your own admission, you are *giving up >> correctness* in order to buy "consistency": > > "sacrificing correctness" only in a purely ideological > sense. > > Explicit inheritance (aka: hard-coding the superclass > symbol) works just as well as implicit inheritance (aka: > relying on the "resolution magic" of super). Of course, in > the former case, a slight bit of onerous is placed on the > programmer because he/she must know the exact spelling of > the superclass symbol, whereas, OO's `super` is nothing but > a lazy shortcut which looks up and returns the symbol for > you. Only, and let me stress that, ONLY, when your classes inherit from exactly *one* superclass, and you mandate that every class that inherits from any of your classes ALSO inherits from exactly one superclass. If *any* class ever inherits from one of your classes and also something else, hard-coded superclass names become straight-up wrong. Have you put code into your classes to prevent anyone from ever multiply inheriting? ChrisA From steve+comp.lang.python at pearwood.info Sun Mar 25 18:41:07 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 22:41:07 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <3714df17-1a61-44ef-9c42-c46fdaeeb489@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 13:30:14 -0700, Rick Johnson wrote: [...] >> Especially since by your own admission, you are *giving up correctness* >> in order to buy "consistency": > > "sacrificing correctness" only in a purely ideological sense. > > Explicit inheritance (aka: hard-coding the superclass symbol) works just > as well as implicit inheritance (aka: relying on the "resolution magic" > of super). Except when it doesn't, which is precisely why super was introduced in the first place. (A minor point: if you're going to quote me, quote me correctly: I said *giving up* not "sacrificing".) If your new-style classes don't use super, then they *literally* are incorrect, and are either buggy, or have a figurative landmine in your class, waiting to go off and introduce a bug if someone is unlucky enough to inherit from it under the right circumstances. Hint: google on "inheritance diamond problem", and maybe you will learn something. You might be lucky enough to use that class for a decade, and never be bitten by the flaw. If you never match those right circumstances (or perhaps that should be the wrong circumstances), then the flaw will never express itself and you may escape introducing an inheritance bug. But nevertheless, it will still be there, in your classes, ready to blow up in your face. > Of course, in the former case, a slight bit of onerous is > placed on the programmer because he/she must know the exact spelling of > the superclass symbol, whereas, OO's `super` is nothing but a lazy > shortcut which looks up and returns the symbol for you. No, *Ruby's super* is nothing but a lazy shortcut. Python's super does a lot more. -- Steve From darcy at VybeNetworks.com Sun Mar 25 18:52:53 2018 From: darcy at VybeNetworks.com (D'Arcy Cain) Date: Sun, 25 Mar 2018 17:52:53 -0500 Subject: String formatting In-Reply-To: <19842a4d-6aaa-329a-947b-31f095efc0b1@gmail.com> References: <19842a4d-6aaa-329a-947b-31f095efc0b1@gmail.com> Message-ID: Was "Accessing parent objects." On 03/25/2018 12:26 PM, Jugurtha Hadjar wrote: >> print("I am {0.__class__.__name__} foo".format(self)) > > I prefer keyword arguments, but if I used it that way I'd do: > > print("I am {0} foo".format(self.__class__.__name__)) These are contrived examples. In real code I might do something like this: fmt_class_name = "I am {0.__class__.__name__} foo".format ... for x class_list: print(fmt_class_name(x)) IOW when I define the format string I want it clear what it is supposed to print. The actual use of the format could happen in a completely different place. > self.__class__.__name__ looks better (+ makes more sense to me) than > 0.__class__.__name__. > >> print(f"I am {self.__class__.__name__} foo") > > f-strings are only available in python 3.6 as far as I know and. I'm not even sure if I am a big fan of f-strings but I do like to run current versions of Python so they are there for me. -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From steve+comp.lang.python at pearwood.info Sun Mar 25 18:52:59 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 22:52:59 +0000 (UTC) Subject: [OT] multicore/cpu history References: <5A8AB7870200001B0001DF67@smtp1.astron.nl> <20180325212907.wu545d46ajru6waj@hjp.at> Message-ID: On Sun, 25 Mar 2018 23:29:07 +0200, Peter J. Holzer wrote: [...] >> >> By the way, multiple CPU machines are different from CPUs with >> >> multiple cores: >> >> >> >> http://smallbusiness.chron.com/multiple-cpu-vs-multicore-33195.html >> > >> > Yeah, it was always "multiple CPUs", not "multiple cores" when I was >> > growing up. > > Yes, but the difference is only an implementation detail. Not really. With multiple CPUs, you have the option of running two distinct OSes in isolation, not merely virtual machines but actual distinct machines in the same box. And the CPUs don't necessarily need to be the same type, see for example the hybrid Apple Mac/Lisp Machine released in the late 1980s or early 90s. Of course, today, virtualisation is in many practical senses virtually as good (pun intended) as actual distinct machines with distinct CPUs, and the cost of adding multiple cores is typically cheaper than adding multiple CPUs, so there's little point unless you're working with serious supercomputer hardware where you need more cores than will fit on a single chip. -- Steve From steve+comp.lang.python at pearwood.info Sun Mar 25 18:53:34 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 22:53:34 +0000 (UTC) Subject: Accessing parent objects References: <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <20180325195027.672qyouuaip7ux62@hjp.at> Message-ID: On Sun, 25 Mar 2018 21:50:27 +0200, Peter J. Holzer wrote: > On 2018-03-25 14:49:44 +0000, Steven D'Aprano wrote: >> The moniker "Python X-thousand" (Python 3000, 4000, 5000...) is usually >> understood to mean a backwards-compatibility breaking version. Since >> Python 4 will *not* be such a version, what you are calling 4000 is >> better called "5000". > > If Python 4.0 will "merely be the release that comes after Python 3.9", > as Nick Coghlan writes in > https://opensource.com/life/14/9/why-python-4-wont-be-python-3, why will > it be called 4.0 and not 3.10? Just to avoid two-digit minor numbers? Yes. -- Steve From steve+comp.lang.python at pearwood.info Sun Mar 25 18:54:54 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 22:54:54 +0000 (UTC) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 10:33:49 -0700, Rick Johnson wrote: > On Sunday, March 25, 2018 at 9:11:35 AM UTC-5, Steven D'Aprano wrote: >> On Sun, 25 Mar 2018 04:49:21 -0700, Rick Johnson wrote: > [...] >> I never said anything about not allowing it. But since you've gone on >> the defence about parens-free function calls, how is this for >> "consistency" in Ruby? >> >> [steve at ando ruby]$ ruby ws-example.rb a + b => 7 >> a+b => 7 >> a+ b => 7 >> a +b => 3 >> >> Here's the source code: >> >> # --- cut --- >> def a(x=4) >> x+2 >> end >> >> b = 1 >> print "a + b => ", (a + b), "\n" >> print "a+b => ", (a+b), "\n" >> print "a+ b => ", (a+ b), "\n" >> print "a +b => ", (a +b), "\n" >> # --- cut --- > > > Short of demonstrating that you have knack for writing obfuscated code > :-), i don't see the point here. [supposed "fix" to the sample script snipped] You know Rick, every time I start to think that talking to you like an adult might result in a productive and intelligent conversation, you pull a stunt like this. Once I recover from laughing at your inability to read a simple eight line Ruby script, and then laughing even harder at your inability to tell the difference between a string in Ruby and an actual function call, I *might* come back and read the rest of your post. -- Steve From rosuav at gmail.com Sun Mar 25 19:03:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 10:03:43 +1100 Subject: [OT] multicore/cpu history In-Reply-To: References: <5A8AB7870200001B0001DF67@smtp1.astron.nl> <20180325212907.wu545d46ajru6waj@hjp.at> Message-ID: On Mon, Mar 26, 2018 at 9:52 AM, Steven D'Aprano wrote: > On Sun, 25 Mar 2018 23:29:07 +0200, Peter J. Holzer wrote: > > [...] >>> >> By the way, multiple CPU machines are different from CPUs with >>> >> multiple cores: >>> >> >>> >> http://smallbusiness.chron.com/multiple-cpu-vs-multicore-33195.html >>> > >>> > Yeah, it was always "multiple CPUs", not "multiple cores" when I was >>> > growing up. >> >> Yes, but the difference is only an implementation detail. > > Not really. With multiple CPUs, you have the option of running two > distinct OSes in isolation, not merely virtual machines but actual > distinct machines in the same box. And the CPUs don't necessarily need to > be the same type, see for example the hybrid Apple Mac/Lisp Machine > released in the late 1980s or early 90s. At what point does it change from being two CPUs to being one CPU and one auxiliary processing unit? Back in the 80s and early 90s, the auxiliary was most likely to be a floating-point unit; today, it'd be a graphics chip. But it could just as easily be a Lisp chip. ChrisA From Richard at Damon-Family.org Sun Mar 25 19:27:52 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 25 Mar 2018 19:27:52 -0400 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On 3/25/18 8:32 AM, bartc wrote: > On 25/03/2018 02:47, Steven D'Aprano wrote: >> On Sun, 25 Mar 2018 00:05:56 +0100, Peter J. Holzer wrote: >> >> [...] >>>> yes, good idea >>> >>> Not if you want to avoid that string to int conversion (as you stated). >>> >>> That is still there, but in addition you now split the string into a >>> list and then join the list into a different string. >> >> I'm glad I wasn't the only one who spotted that. >> >> There's something very curious about somebody worried about efficiency >> choosing a *less* efficient solution than what they started with. To >> quote W.A. Wulf: >> >> "More computing sins are committed in the name of efficiency (without >> necessarily achieving it) than for any other single reason ? including >> blind stupidity." >> >> As Donald Knuth observed: >> >> "We should forget about small efficiencies, say about 97% of the time: >> premature optimization is the root of all evil." >> >> The Original Poster (OP) is concerned about saving, what, a tenth of a >> microsecond in total? Hardly seems worth the effort, especially if you're >> going to end up with something even slower. > > Using CPython on my machine, doing a string to int conversion that > specific number took 200 times as long as doing a normal assignment. > > That conversion took 4 microseconds. > > Not significant if it's only done once. But it might be executed a > million times. > The other half of that thought is how does the 4 microseconds to create the constant compare to the operations USING that number. My guess is that for most things the usage will swamp the initialization, even if that is somewhat inefficient. One option, which in my opinion is more readable, is if you really want the efficency is to write the number on a single line, with a place guide comment above, something like # 00000000011111111112222222222333333333344444444445 # 12345678901234567890123456789012345678901234567890 foo = 13452356547242573457246245673472457723245634564564 This makes it very clear what the 47th digit of the number is (run the guide numbers in the direction that makes sense for you) Yes, you will have a very long line (several in fact with the guide comment), but that 'uglyness' is worth it if the time cost turns out to be significant. After all, correctness outweighs elegance (but maybe not clarity, but the long line isn't unclear, just a bit ugly), so if the time cost make the program violate the specs to be useful, you need a bit of ugly to get it to work. (The other option would be to initialize into a global variable, which has its own type of ugly)). From bc at freeuk.com Sun Mar 25 21:37:44 2018 From: bc at freeuk.com (bartc) Date: Mon, 26 Mar 2018 02:37:44 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On 26/03/2018 00:27, Richard Damon wrote: > On 3/25/18 8:32 AM, bartc wrote: >> Using CPython on my machine, doing a string to int conversion that >> specific number took 200 times as long as doing a normal assignment. >> >> That conversion took 4 microseconds. >> >> Not significant if it's only done once. But it might be executed a >> million times. >> > > The other half of that thought is how does the 4 microseconds to create > the constant compare to the operations USING that number. My guess is > that for most things the usage will swamp the initialization, even if > that is somewhat inefficient. Calling a function that sets up C using 'C = 288714...' on one line, and that then calculates D=C+C, takes 0.12 seconds to call 1000000 times. To do D=C*C, takes 2.2 seconds (I've subtracted the function call overhead of 0.25 seconds; there might not be any function call). If I instead initialise C using 'C = int("288712...")', then timings increase as follows: 0.12 => 3.7 seconds 2.2 => 5.9 seconds So the overhead /can/ be substantial, and /can/ be significant compared with doing bignum calculations. Of course, once initialised, C might be used a hundred times, then the overhead is less significant. But it is not small enough to just dismiss. -- bartc From rantingrickjohnson at gmail.com Sun Mar 25 22:16:12 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Sun, 25 Mar 2018 19:16:12 -0700 (PDT) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sunday, March 25, 2018 at 5:57:28 PM UTC-5, Steven D'Aprano wrote: > [supposed "fix" to the sample script snipped] > > You know Rick, every time I start to think that talking to > you like an adult might result in a productive and > intelligent conversation, you pull a stunt like this. Once > I recover from laughing at your inability to read a simple > eight line Ruby script, and then laughing even harder at > your inability to tell the difference between a string in > Ruby and an actual function call, I *might* come back and > read the rest of your post. Q1: Do my modifications throw an error??? If so, then provide this group with the exception message and you shall be redeemed. Q2: Did my "fix" fail to meet the highly generalized qualification of "un-obfuscating the code"??? If not, then explain to this group exactly how i failed. Steven, your highly competitive nature is apparent to any and all who frequent this group. And i'm sure i speak for everyone here when i say that we all appreciate the vast wealth of knowledge you bring to the table. But this time you have allowed your emotions to get the better of you, and now your friendly barbs have become purposeful misrepresentations. Listen, if i make a false statement or give bad advice on this list, then i _want_ someone to call me on it. But for you to cast accusations without providing any evidence to support those accusations, is beneath the dignity of this mailing list. I stand behind my "code fix" and i confidently claim that not only does it meet the requirements, it also runs without error. And if you disagree with my claim, then bring forth your evidence. From Richard at Damon-Family.org Sun Mar 25 22:35:06 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Sun, 25 Mar 2018 22:35:06 -0400 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: <38311d42-a989-5646-e681-83ba2aad2272@Damon-Family.org> On 3/25/18 9:37 PM, bartc wrote: > On 26/03/2018 00:27, Richard Damon wrote: >> On 3/25/18 8:32 AM, bartc wrote: > >>> Using CPython on my machine, doing a string to int conversion that >>> specific number took 200 times as long as doing a normal assignment. >>> >>> That conversion took 4 microseconds. >>> >>> Not significant if it's only done once. But it might be executed a >>> million times. >>> >> >> The other half of that thought is how does the 4 microseconds to >> create the constant compare to the operations USING that number. My >> guess is that for most things the usage will swamp the >> initialization, even if that is somewhat inefficient. > > Calling a function that sets up C using 'C = 288714...' on one line, > and that then calculates D=C+C, takes 0.12 seconds to call 1000000 times. > > To do D=C*C, takes 2.2 seconds (I've subtracted the function call > overhead of 0.25 seconds; there might not be any function call). > > If I instead initialise C using 'C = int("288712...")', then timings > increase as follows: > > 0.12? =>?? 3.7 seconds > 2.2?? =>?? 5.9 seconds > > So the overhead /can/ be substantial, and /can/ be significant > compared with doing bignum calculations. > > Of course, once initialised, C might be used a hundred times, then the > overhead is less significant. But it is not small enough to just dismiss. > And my point is that writing a program to just add or multiply once two FIXED big long numbers (since they are in the source code, that seems fixed), a million times seems? unlikely (and even then the cost isn't that bad, since that sounds like a run once program). And of course once the answer has been computed, it needs to be output somehow, which likely will add to the time significantly. The question would be do you have a REAL PRACTICAL case where this overhead actually makes a significant difference, or are we just talking a theoretical toy. It is a fact that it seems the language has no notation to express such a large number as a number in source code on mulitple lines (except as a string). I would think it would take something more than a toy program to persuade for something notation to do this directly (and there are alternatives that have been mentioned that aren't that bad for this sort of corner case). -- Richard Damon From drsalists at gmail.com Sun Mar 25 23:25:33 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Sun, 25 Mar 2018 20:25:33 -0700 Subject: Code for Using Keyboard to Play music notes "ABCDEFG" In-Reply-To: <1625edbc5ae-c86-17f4c@webjas-vaa145.srv.aolmail.net> References: <1625edbc5ae-c86-17f4c@webjas-vaa145.srv.aolmail.net> Message-ID: I imagine pygame could do this, if you give it: https://freesound.org/people/pinkyfinger/packs/4409/ On Sun, Mar 25, 2018 at 1:31 PM, Bernard via Python-list wrote: > > Pyton Friends, > Do you have any code that will play the notes "ABCDEFG" from my computer keyboard when a key is pressed ? For example if I press the "a" key the note "a" will sound out of my speaker. > Thanks, > BigB > -- > https://mail.python.org/mailman/listinfo/python-list From rustompmody at gmail.com Sun Mar 25 23:52:20 2018 From: rustompmody at gmail.com (Rustom Mody) Date: Sun, 25 Mar 2018 20:52:20 -0700 (PDT) Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <5ab7d9f5$0$4812$426a74cc@news.free.fr> <20180325192510.lwhge6uywuarkzot@hjp.at> Message-ID: <0e63c2c5-6913-4dea-bd8e-cbf3cf5e2340@googlegroups.com> On Monday, March 26, 2018 at 12:55:43 AM UTC+5:30, Peter J. Holzer wrote: > On 2018-03-25 19:18:23 +0200, ast wrote: > > Le 25/03/2018 ? 03:47, Steven D'Aprano a ?crit?: > > > The Original Poster (OP) is concerned about saving, what, a tenth of a > > > microsecond in total? Hardly seems worth the effort, especially if you're > > > going to end up with something even slower. > > > > > > > > > > I regret that many answers are malicious, as your. tl;dr: I plead guilty with my almost first post in this thread containing: ?In case there is a religious commitment to PEP 8 dicta? ? Apologies! > > I just looked up "malicious" in Merriam-Webster, just in case it has > some meaning I wasn't previously aware of. > > Nope. It says: > > : having or showing a desire to cause harm to someone > : given to, marked by, or arising from malice > > and malice is defined as: > > : desire to cause pain, injury, or distress to another > : intent to commit an unlawful act or cause harm without legal > justification or excuse > O come! The OP is apparently not an English speaker; just do s/malicious/gratuitously unkind/ or just s/malicious/rude/ My excuse if any is that I was using ?religious? in a precise way as ?belief-system? and the over-rigorous commitment to 72 (or whatever) length lines can be a belief?commitment that produces egregious results The deeper difficulty is that it is impossible to be human and not have some non-verifiable beliefs. And yet when these beliefs harden into ossified religions the results can be unimaginably pernicious In particular, the complement set: belief-system - classic-religion can contain the most pernicious examples: Just compare the number killed in the name of Islam or Christianity with those killed by communism and divide by the number of years these 'religions' held sway Some other examples: 1. Irreligion is a more pernicious belief-system than (classic) religion: https://youtu.be/6PbYoQw8M48 2. Science as a delusional belief-system: https://youtu.be/JKHUaNAxsTg 3. New-Atheism as a colonialist project: https://aquasanju.wordpress.com/2015/08/08/new-atheists-are-privilege-deniers-sushant-taing/ It is particularly striking that 2 above was banned! Shows how identical are the subconscious psychological drivers of medieval popes and today's scientists (so-called) From greg.ewing at canterbury.ac.nz Mon Mar 26 01:12:37 2018 From: greg.ewing at canterbury.ac.nz (Gregory Ewing) Date: Mon, 26 Mar 2018 18:12:37 +1300 Subject: Best practice for managing secrets (passwords, private keys) used by Python scripts running as daemons In-Reply-To: References: <1521823073.2871510.1313785456.2969373D@webmail.messagingengine.com> <20180324172456.t6qxwkwqmxutbcsb@hjp.at> Message-ID: Peter J. Holzer wrote: > (Historically, many unixes allowed all users to read the environment > variables of all processes. I don't know if this is still the case for > e.g. Solaris or AIX - or macOS) A quick test suggests it's still true in MacOSX 10.6: % ps aeuww USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 254 0.0 0.1 2436216 1668 s000 Ss 6:07pm 0:00.02 login -pf greg PATH=/usr/bin:/bin:/usr/sbin:/sbin TMPDIR=/var/folders/pt/ptWbIBidFOGihAoVFzVWZU+++TI/-Tmp-/ SHELL=/bin/tcsh HOME=/Users/greg USER=greg LOGNAME=greg DISPLAY=/tmp/launch-E3mIkg/org.x:0 SSH_AUTH_SOCK=/tmp/launch-Lq6SPh/Listeners Apple_PubSub_Socket_Render=/tmp/launch-k6nWyL/Render COMMAND_MODE=unix2003 __CF_USER_TEXT_ENCODING=0x1F5:0:0 TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=273.1 LANG=en_NZ.UTF-8 TERM=xterm-color ... -- Greg From dieter at handshake.de Mon Mar 26 02:02:25 2018 From: dieter at handshake.de (dieter) Date: Mon, 26 Mar 2018 08:02:25 +0200 Subject: Question regarding objects in __call__() methods References: Message-ID: <87h8p3tlvy.fsf@handshake.de> Arshpreet Singh writes: > ... > As debugging the code I got at line 10. I am sending a request to particular API and returning a request_object . further deep down it generates the "response_object" as from my requirements that should be JSON object but I am only getting Python-Object in hexa form, is there any way I can get to know how to use internet_object so I can get to know how to use that internet_object with response. To learn about an unknown object, you could use Python's "inspection" facilities. The easiest way is to use the builtin "help" function: "help(obj)", maybe "help(obj.__class__)". It tells you lots of useful things about "obj". F?rther inspection utilities: "dir", "vars" and the "inspect" module. Read the documentation to find out what they do. From arsh840 at gmail.com Mon Mar 26 02:09:07 2018 From: arsh840 at gmail.com (Arshpreet Singh) Date: Sun, 25 Mar 2018 23:09:07 -0700 (PDT) Subject: Question regarding objects in __call__() methods In-Reply-To: References: <87h8p3tlvy.fsf@handshake.de> Message-ID: <445515fc-c698-4ee9-a051-db943cadb2f0@googlegroups.com> On Monday, 26 March 2018 11:32:51 UTC+5:30, dieter wrote: > F?rther inspection utilities: "dir", "vars" and the "inspect" module. > Read the documentation to find out what they do. Thanks, Dieter, That is really helpful! From dieter at handshake.de Mon Mar 26 02:10:40 2018 From: dieter at handshake.de (dieter) Date: Mon, 26 Mar 2018 08:10:40 +0200 Subject: A question related to the PYTHONPATH References: Message-ID: <87d0zrtli7.fsf@handshake.de> adrien oyono writes: > I have recently read the documentation about how imports work on python, > and I was wondering why, when you execute a python file, the current > directory is not added by default to the PYTHONPATH ? Maybe, to avoid surprises? You can invoke a script from different positions in your file system. If PYTHONPATH would automatically get ".", the script's behaviour could depend on the position from where it is called. Prepending "." to "PYTHONPATH" could have dramatic effects: assume your script uses module "xxx" and your current working directory accidentally has a "xxx.py"; then this "xxx.py" would be used and it may behave much differently from another one. The problem is not so severe when "." is appended to "PYTHONPATH". Nevertheless, it likely is not a good idea to have script behaviour depending by default on the execution location. From antoon.pardon at vub.be Mon Mar 26 04:02:23 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Mon, 26 Mar 2018 10:02:23 +0200 Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> Message-ID: <1938907b-39d3-2764-078b-0bb1c39639ec@vub.be> On 25-03-18 00:54, Steven D'Aprano wrote: > > There's nothing wrong with super() in Python 2. You just have to > understand what you're doing. It's still the right solution for doing > inheritance the right way. > > ... > > The trick is to use new-style classes that inherit from object, and avoid > the old-style classes that don't: > > # Good > class Spam(object): > ... > > # Not so good > class Spam: > ... How good is that when almost all classes in the standard library are old style classes? -- Antoon From ben+python at benfinney.id.au Mon Mar 26 04:39:47 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Mar 2018 19:39:47 +1100 Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <1938907b-39d3-2764-078b-0bb1c39639ec@vub.be> Message-ID: <85po3rxmb0.fsf@benfinney.id.au> Antoon Pardon writes: > On 25-03-18 00:54, Steven D'Aprano wrote: > > The trick is to use new-style classes that inherit from object, and > > avoid the old-style classes that don't: > > > > # Good > > class Spam(object): > > ... > > > > # Not so good > > class Spam: > > ... > > How good is that when almost all classes in the standard library are > old style classes? That's a vague question. So, a vague answer: When that is the case, it's quite good. Is that satisfactory? Or would you care to elaborate on what the question is? -- \ ?Natural catastrophes are rare, but they come often enough. We | `\ need not force the hand of nature.? ?Carl Sagan, _Cosmos_, 1980 | _o__) | Ben Finney From antoon.pardon at vub.be Mon Mar 26 04:43:25 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Mon, 26 Mar 2018 10:43:25 +0200 Subject: Accessing parent objects In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> Message-ID: <5711145c-435e-0005-2c80-1e08709470c5@vub.be> On 25-03-18 06:29, Steven D'Aprano wrote: > Python 3 is now six point releases in (and very soon to have a seventh, > 3.7 being in beta as we speak). It is stable, feature-rich, and a joy to > work in. As well as a heap of great new features, there have been a > metric tonne of performance improvements, making Python 3 faster than 2.7 > for many tasks, e.g. But did they start up cleaning the standard library yet? I'll confess I'm only using 3.5 but when I go through the standard library I see a lot of classes still using the old style of calling the parant method, which makes it more difficult to use in a multiple inheritance scheme. Those that don't inheret, don't call super themselves, which makes them also more difficult to use in a multiple inheritance scheme. ? > For comparison, here's how Python's super works in 3.x: > > def method(self, spam, eggs, cheese): > result = super().method(spam, eggs, cheese) > > In other words, you must explicitly state the method you are calling, and > give the arguments you want to pass on. Of course you can manipulate or > remove those arguments as needed, or add new ones: One of the problems with this is that when you go through the mro you will ultimatly end up calling object.method(self, spam, eggs, cheese) which will thrown an exception like: AttributeError: 'super' object has no attribute 'method' So I find your example misleading. It is just one step, and you need a carefully designed hierarchy to make it work correctly and incorporating standard library classe into that hierarchy is not self-evident. -- Antoon From ben+python at benfinney.id.au Mon Mar 26 04:52:31 2018 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 26 Mar 2018 19:52:31 +1100 Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <5711145c-435e-0005-2c80-1e08709470c5@vub.be> Message-ID: <85lgefxlps.fsf@benfinney.id.au> Antoon Pardon writes: > But did they start up cleaning the standard library yet? I'll confess > I'm only using 3.5 but when I go through the standard library I see a > lot of classes still using the old style of calling the parant method, > which makes it more difficult to use in a multiple inheritance scheme. > Those that don't inheret, don't call super themselves, which makes > them also more difficult to use in a multiple inheritance scheme. Yes, there are large areas of the Python standard library that were written before Python 3 features, and some of those are thus less good than they could be. The solution is, as always: Someone needs to identify (in a bug report) what specifically should be improved; someone needs to come up with a specific implementation of that improvement (in a merge request, with unit tests); someone needs to maintain that code in the standard library on an ongoing basis. Feel free to take on any of that work; this is work that can be done by anyone in the Python community, we don't need to wait. -- \ ?? whoever claims any right that he is unwilling to accord to | `\ his fellow-men is dishonest and infamous.? ?Robert G. | _o__) Ingersoll, _The Liberty of Man, Woman and Child_, 1877 | Ben Finney From steve+comp.lang.python at pearwood.info Mon Mar 26 05:34:22 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 09:34:22 +0000 (UTC) Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: On Mon, 26 Mar 2018 02:37:44 +0100, bartc wrote: > Calling a function that sets up C using 'C = 288714...' on one line, and > that then calculates D=C+C, takes 0.12 seconds to call 1000000 times. > > To do D=C*C, takes 2.2 seconds (I've subtracted the function call > overhead of 0.25 seconds; there might not be any function call). Bart, this is a contrived example that really proves nothing. This is literally the sort of premature optimization that Knuth etc so frequently warn about. In *almost any* real program, you're not going to be calculating C+C over and over and over and over again, millions of times in a row, while doing *literally* nothing else. If your point is that, *hypothetically*, there could be some amazingly unlikely set of circumstances where that's exactly what I will need to do, then fine, I'll deal with those circumstances when it happens and not a moment before. And I'll probably deal with it by calculating C+C in Python, then hard- coding D= that number, reducing the runtime cost of the addition to zero. > If I instead initialise C using 'C = int("288712...")', then timings > increase as follows: Given that the original number given had 397 digits and has a bit length of 1318, I must admit to some curiosity as to how exactly you managed to cast it to a C int (32 bits on most platforms). It is too big for an int, a long (64 bits), a long-long (128 bits) or even a long-long-long-long-long-long-long-long-long-long-long-long-long- long-long-long (1024 bits), if such a thing even exists. So what exactly did you do? -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 26 06:01:22 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 10:01:22 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <1938907b-39d3-2764-078b-0bb1c39639ec@vub.be> Message-ID: On Mon, 26 Mar 2018 10:02:23 +0200, Antoon Pardon wrote: >> The trick is to use new-style classes that inherit from object, and >> avoid the old-style classes that don't: >> >> # Good >> class Spam(object): >> ... >> >> # Not so good >> class Spam: >> ... > > How good is that when almost all classes in the standard library are old > style classes? I'm glad you asked! I presume you're talking about Python 2, since the standard library of Python 3 is 100% "new-style". In Python 2, there are 76 new-style classes in the builtins, most of which can be subclassed: py> import __builtin__ as builtins py> sum(type(obj) is type for obj in builtins.__dict__.values()) 76 As a very quick-and-dirty test, I counted classes in the stdlib that subclassed object: [steve at ando python2.7]$ grep "class .*(object)" *.py | wc -l 55 [steve at ando python2.7]$ grep "class .*(object)" */*.py | wc -l 562 plus another 47 for list and int -- I didn't bother counting subclasses of dict, tuple, str, etc). So, in round figures, there are about 700 new style class in the Python 2.7 std lib. As a similarly quick-and-dirty test, I counted old-style classes: [steve at ando python2.7]$ grep "\Wclass .*[^\(\)]*:" *.py | grep -v -F "(" | wc -l 42 [steve at ando python2.7]$ grep "\Wclass .*[^\(\)]*:" */*.py | grep -v -F "(" | wc -l 365 I daresay that's an under-estimate, so let's double it to be safe. So we have (in round figures) about 700 new-style classes versus 800 old- style classes in Python 2.7. I'm no mathematician[1] but that hardly seems like "almost all" old-style classes to me. We can quibble about the exact numbers, and about which classes are the most commonly subclassed, and how many false positives there are from using grep, but however you look at it, it is clear that a significant portion of stdlib classes are new-style. [1] Actually, I am. -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 26 06:02:15 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 10:02:15 +0000 (UTC) Subject: [OT] multicore/cpu history References: <5A8AB7870200001B0001DF67@smtp1.astron.nl> <20180325212907.wu545d46ajru6waj@hjp.at> Message-ID: On Mon, 26 Mar 2018 10:03:43 +1100, Chris Angelico wrote: > At what point does it change from being two CPUs to being one CPU and > one auxiliary processing unit? When someone writes an OS that will run on the "auxiliary processing unit" alone, then it's probably time to start calling it a CPU :-) > Back in the 80s and early 90s, the > auxiliary was most likely to be a floating-point unit; today, it'd be a > graphics chip. But it could just as easily be a Lisp chip. Or a Forth chip. Hardware people can probably tell you what it is that CPUs do that FPUs and GPUs don't do. Or specialised Bitcoin mining chips. Whatever it is that they don't do, but CPUs do, is probably a good dividing line between "CPU" and "auxiliary chip". -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 26 06:23:54 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 10:23:54 +0000 (UTC) Subject: Accessing parent objects References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <5711145c-435e-0005-2c80-1e08709470c5@vub.be> Message-ID: On Mon, 26 Mar 2018 10:43:25 +0200, Antoon Pardon wrote: > But did they start up cleaning the standard library yet? I'll confess > I'm only using 3.5 but when I go through the standard library I see a > lot of classes still using the old style of calling the parant method, > which makes it more difficult to use in a multiple inheritance scheme. > Those that don't inheret, don't call super themselves, which makes them > also more difficult to use in a multiple inheritance scheme. Ah, that's a separate question! And it's a good observation. You are right that there is a lot of legacy code in the std lib that doesn't use super, and probably should. Fixing that is probably a good project for some kindly volunteer wanting to get into Python bug- fixing :-) But you can usually fix that in your own code with a simple adaptor class that delegates to the old-style class. See for example: https://code.activestate.com/recipes/577721 https://rhettinger.wordpress.com/2011/05/26/super-considered-super/ > One of the problems with this is that when you go through the mro you > will ultimatly end up calling object.method(self, spam, eggs, cheese) > which will thrown an exception like: AttributeError: 'super' object has > no attribute 'method' Why in the name of all the gods would you call super() on a method you aren't inheriting from a superclass??? I'm sorry if my example was inaccurate or misleading, but please be reasonable. Calling super in a method which you aren't overloading is not a reasonable thing to do. > So I find your example misleading. It is just one step, and you need a > carefully designed hierarchy to make it work correctly and incorporating > standard library classe into that hierarchy is not self-evident. You *always* need a carefully designed hierarchy. super or no super, single inheritance or delegation or multiple inheritance, you can't expect to just randomly throw classes together in a slap-dash manner and have them work. -- Steve From bc at freeuk.com Mon Mar 26 06:31:22 2018 From: bc at freeuk.com (bartc) Date: Mon, 26 Mar 2018 11:31:22 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: <_Z3uC.218815$TS3.193851@fx18.am4> On 26/03/2018 10:34, Steven D'Aprano wrote: > On Mon, 26 Mar 2018 02:37:44 +0100, bartc wrote: >> If I instead initialise C using 'C = int("288712...")', then timings >> increase as follows: > > Given that the original number given had 397 digits and has a bit length > of 1318, I must admit to some curiosity as to how exactly you managed to > cast it to a C int (32 bits on most platforms). > > It is too big for an int, a long (64 bits), a long-long (128 bits) or > even a long-long-long-long-long-long-long-long-long-long-long-long-long- > long-long-long (1024 bits), if such a thing even exists. > > > So what exactly did you do? I'm not sure why you think the language C came into it. I did this: def fn(): C = int( "28871482380507712126714295971303939919776094592797" "22700926516024197432303799152733116328983144639225" "94197780311092934965557841894944174093380561511397" "99994215424169339729054237110027510420801349667317" "55152859226962916775325475044445856101949404200039" "90443211677661994962953925045269871932907037356403" "22737012784538991261203092448414947289768854060249" "76768122077071687938121709811322297802059565867") # C = 2887148238050771212671429... [truncated for this post] D=C+C for i in range(1000000): fn() The purpose was to establish how such int("...") conversions compare in overheads with actual arithmetic with the resulting numbers. -- bartc From steve+comp.lang.python at pearwood.info Mon Mar 26 06:43:32 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 10:43:32 +0000 (UTC) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: On Sun, 25 Mar 2018 19:16:12 -0700, Rick Johnson wrote: > On Sunday, March 25, 2018 at 5:57:28 PM UTC-5, Steven D'Aprano wrote: > >> [supposed "fix" to the sample script snipped] >> >> You know Rick, every time I start to think that talking to you like an >> adult might result in a productive and intelligent conversation, you >> pull a stunt like this. Once I recover from laughing at your inability >> to read a simple eight line Ruby script, and then laughing even harder >> at your inability to tell the difference between a string in Ruby and >> an actual function call, I *might* come back and read the rest of your >> post. [...] > If not, then explain to this group exactly how i failed. Rick, you're supposedly familiar with Ruby. And yet, you didn't notice that your supposed "fix" didn't touch any executable code, all it did was modify the strings being printed. Because of this "fix", the printed strings no longer match the code being executed, but the strange, inconsistent behaviour still occurs. For the benefit of those who aren't Ruby experts, the code I used before (see below) defines a function called "a", then prints four ways of calling that function that differ only in whitespace. For example: print "a + b => ", (a + b), "\n" prints the string "a + b", then calls a and adds b using the same syntax as that shown in the string, then prints a newline. The kicker is that out of these four legal, parenthesis-free ways of calling function a, *three* of them interpret the expression as: call a with no arguments then add b using the binary plus operator but the last, differing only in whitespace, interprets it as: call a with a single argument, unary-plus b Here's the code again: # --- cut --- def a(x=4) x+2 end b = 1 print "a + b => ", (a + b), "\n" print "a+b => ", (a+b), "\n" print "a+ b => ", (a+ b), "\n" print "a +b => ", (a +b), "\n" # --- cut --- and your "fix" was to modify the *strings* without touching the actual function calls. Well done Rick. Of course we can solve the problem by always using parentheses when making function calls. But that misses the point that Ruby allows this inconsistency in the first place. -- Steve From bc at freeuk.com Mon Mar 26 06:45:33 2018 From: bc at freeuk.com (bartc) Date: Mon, 26 Mar 2018 11:45:33 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <38311d42-a989-5646-e681-83ba2aad2272@Damon-Family.org> Message-ID: On 26/03/2018 03:35, Richard Damon wrote: > On 3/25/18 9:37 PM, bartc wrote: >> So the overhead /can/ be substantial, and /can/ be significant >> compared with doing bignum calculations. >> >> Of course, once initialised, C might be used a hundred times, then the >> overhead is less significant. But it is not small enough to just dismiss. >> > And my point is that writing a program to just add or multiply once two > FIXED big long numbers (since they are in the source code, that seems > fixed), a million times seems? unlikely (and even then the cost isn't > that bad, since that sounds like a run once program). Similar overheads occur when you use string=>int even on small numbers: This code: C = int("12345") D = C+C # or C*C; about the same results takes 5 times as long (using my CPython 3.6.x on Windows) as: C = 12345 D = C+C Your arguments that this doesn't really matter would equally apply here. Yet you don't see Python code full of 'int("43")' instead of just '43' on the basis that the extra overhead is not significant, as the program might be run only once. A slightly worrying attitude in a language that has issues with performance, but no longer a surprising one. -- bartc From antoon.pardon at vub.be Mon Mar 26 07:04:21 2018 From: antoon.pardon at vub.be (Antoon Pardon) Date: Mon, 26 Mar 2018 13:04:21 +0200 Subject: Accessing parent objects In-Reply-To: <85lgefxlps.fsf@benfinney.id.au> References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <5711145c-435e-0005-2c80-1e08709470c5@vub.be> <85lgefxlps.fsf@benfinney.id.au> Message-ID: On 26-03-18 10:52, Ben Finney wrote: > Antoon Pardon writes: > >> But did they start up cleaning the standard library yet? I'll confess >> I'm only using 3.5 but when I go through the standard library I see a >> lot of classes still using the old style of calling the parant method, >> which makes it more difficult to use in a multiple inheritance scheme. >> Those that don't inheret, don't call super themselves, which makes >> them also more difficult to use in a multiple inheritance scheme. > Yes, there are large areas of the Python standard library that were > written before Python 3 features, and some of those are thus less good > than they could be. > > The solution is, as always: Someone needs to identify (in a bug report) > what specifically should be improved; someone needs to come up with a > specific implementation of that improvement (in a merge request, with > unit tests); someone needs to maintain that code in the standard > library on an ongoing basis. I would have thought there is already someome maintaining the standard library code. I would also have thought that a rather thorough review of the standard library code would have been part of the upgrade to python3. > Feel free to take on any of that work; this is work that can be done by > anyone in the Python community, we don't need to wait. > After having seen Raymond Hettingers talk: Super considered super!(*) My impression is that the developers wouldn't be interested in such a redesign. The idea I get away from it is that you can do what needs to be done, you just need to wrap the not cooperating standard library classes into ones that do the cooperation for you. (*) https://www.youtube.com/watch?v=EiOglTERPEo&spfreload=1 -- Antoon. From sjsumitj at gmail.com Mon Mar 26 07:17:26 2018 From: sjsumitj at gmail.com (Sum) Date: Mon, 26 Mar 2018 16:47:26 +0530 Subject: Getting "LazyImporter' object is not callable" error when trying to send email using python smtplib Message-ID: Hi, Getting "LazyImporter' object is not callable" error. I have enabled allow less secure app setting in sender gmail. Code : import smtplib from email import MIMEBase from email import MIMEText from email.mime.multipart import MIMEMultipart from email import Encoders import os def send_email(to, subject, text, filenames): try: gmail_user = 'xx at gmail.com' gmail_pwd = 'xxxx' msg = MIMEMultipart() msg['From'] = gmail_user msg['To'] = ", ".join(to) msg['Subject'] = subject msg.attach(MIMEText(text)) for file in filenames: part = MIMEBase('application', 'octet-stream') part.set_payload(open(file, 'rb').read()) Encoders.encode_base64(part) part.add_header('Content-Disposition', 'attachment; filename="%s"'% os.path.basename(file)) msg.attach(part) mailServer = smtplib.SMTP("smtp.gmail.com:465") mailServer.ehlo() mailServer.starttls() mailServer.ehlo() mailServer.login(gmail_user, gmail_pwd) mailServer.sendmail(gmail_user, to, msg.as_string()) mailServer.close() print('successfully sent the mail') except smtplib.SMTPException,error:: print str(error) if __name__ == '__main__': attachment_file = ['t1.txt','t2.csv'] to = "xxxxxx at gmail.com" TEXT = "Hello everyone" SUBJECT = "Testing sending using gmail" send_email(to, SUBJECT, TEXT, attachment_file) Getting below error : File "test_mail.py", line 64, in send_email(to, SUBJECT, TEXT, attachment_file) File "test_mail.py", line 24, in send_email msg.attach(MIMEText(text)) TypeError: 'LazyImporter' object is not callable From steve+comp.lang.python at pearwood.info Mon Mar 26 08:04:20 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 12:04:20 +0000 (UTC) Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> Message-ID: On Mon, 26 Mar 2018 11:31:22 +0100, bartc wrote: > On 26/03/2018 10:34, Steven D'Aprano wrote: >> So what exactly did you do? > > I'm not sure why you think the language C came into it. My misunderstanding. Sorry. -- Steve From Richard at Damon-Family.org Mon Mar 26 08:21:26 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 26 Mar 2018 08:21:26 -0400 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <38311d42-a989-5646-e681-83ba2aad2272@Damon-Family.org> Message-ID: <64bc1355-7179-1478-15ad-69176e3252ab@Damon-Family.org> On 3/26/18 6:45 AM, bartc wrote: > On 26/03/2018 03:35, Richard Damon wrote: >> On 3/25/18 9:37 PM, bartc wrote: > >>> So the overhead /can/ be substantial, and /can/ be significant >>> compared with doing bignum calculations. >>> >>> Of course, once initialised, C might be used a hundred times, then >>> the overhead is less significant. But it is not small enough to just >>> dismiss. >>> >> And my point is that writing a program to just add or multiply once >> two FIXED big long numbers (since they are in the source code, that >> seems fixed), a million times seems? unlikely (and even then the cost >> isn't that bad, since that sounds like a run once program). > > Similar overheads occur when you use string=>int even on small numbers: > > This code: > > ??? C = int("12345") > ??? D = C+C????? # or C*C; about the same results > > takes 5 times as long (using my CPython 3.6.x on Windows) as: > > ??? C = 12345 > ??? D = C+C > > Your arguments that this doesn't really matter would equally apply here. > > Yet you don't see Python code full of 'int("43")' instead of just '43' > on the basis that the extra overhead is not significant, as the > program might be run only once. > > A slightly worrying attitude in a language that has issues with > performance, but no longer a surprising one. > Strawman. The original argument was that there was sufficient clarity grounds to want to express the big number as a string. That base argument doesn't hold here, and in fact for small number the construct clearly makes the code less clear, so the construct fails there. If the numbers were formatted in a way that using a string, and doing some manipulations on it first, and then converting made sense, you might be able to make the argument, but then again it comes down to efficiency vs clarity, and for that, as long as efficiency hasn't suffered enough to start to become a correctness issue, clarity has an edge. -- Richard Damon From Richard at Damon-Family.org Mon Mar 26 08:30:21 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 26 Mar 2018 08:30:21 -0400 Subject: Entering a very large number In-Reply-To: <_Z3uC.218815$TS3.193851@fx18.am4> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> Message-ID: On 3/26/18 6:31 AM, bartc wrote: > On 26/03/2018 10:34, Steven D'Aprano wrote: >> On Mon, 26 Mar 2018 02:37:44 +0100, bartc wrote: > >>> If I instead initialise C using 'C = int("288712...")', then timings >>> increase as follows: >> >> Given that the original number given had 397 digits and has a bit length >> of 1318, I must admit to some curiosity as to how exactly you managed to >> cast it to a C int (32 bits on most platforms). >> >> It is too big for an int, a long (64 bits), a long-long (128 bits) or >> even a long-long-long-long-long-long-long-long-long-long-long-long-long- >> long-long-long (1024 bits), if such a thing even exists. >> >> >> So what exactly did you do? > > I'm not sure why you think the language C came into it. I did this: > > def fn(): > ??? C = int( > ??? "28871482380507712126714295971303939919776094592797" > ??? "22700926516024197432303799152733116328983144639225" > ??? "94197780311092934965557841894944174093380561511397" > ??? "99994215424169339729054237110027510420801349667317" > ??? "55152859226962916775325475044445856101949404200039" > ??? "90443211677661994962953925045269871932907037356403" > ??? "22737012784538991261203092448414947289768854060249" > ??? "76768122077071687938121709811322297802059565867") > > #?? C = 2887148238050771212671429... [truncated for this post] > > ??? D=C+C > > for i in range(1000000): > ??? fn() > > The purpose was to establish how such int("...") conversions compare > in overheads with actual arithmetic with the resulting numbers. > Of course if this was done in C with a version that had builtin bignum ints or an aggresive enough optimizer (or a Python that did a similar level of optimizations) this function would just test the speed of starting the program, as it actually does nothing and can be optimized away. Yes, something like this can beused to measure the base time to do something, but the real question should be is that time significant compared to the other things that the program is doing, Making a 200x improvement on code that takes 1% of the execution time saves you 0.995%, not normally worth it unless your program is currently running at 100.004% of the allowed (or acceptable) timing, if acceptable timing can even be defined that precisely. -- Richard Damon From darcy at VybeNetworks.com Mon Mar 26 08:31:06 2018 From: darcy at VybeNetworks.com (D'Arcy Cain) Date: Mon, 26 Mar 2018 07:31:06 -0500 Subject: Using object as a class Message-ID: <299fc94b-ed77-a5a7-7df0-617c39336b28@VybeNetworks.com> It's called a super class but it doesn't quite work like a normal class. >>> OBJ = object() >>> OBJ.x = 3 Traceback (most recent call last): File "", line 1, in AttributeError: 'object' object has no attribute 'x' I can fix this by creating a NULL class. >>> class NullObject(object): pass ... >>> OBJ = NullObject() >>> OBJ.x = 3 >>> OBJ.x 3 >>> Is this behaviour (object not quite like a class) documented anywhere? Does anyone know the rationale for this if any? In case anyone wants to know why I am doing this, sometimes I simply want an object to hold values that I can pass around. I don't need methods and I don't always know what variables I am going to need. And yes, I know that dict is the usual way to do this. -- D'Arcy J.M. Cain Vybe Networks Inc. http://www.VybeNetworks.com/ IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com From steve+comp.lang.python at pearwood.info Mon Mar 26 08:34:27 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 12:34:27 +0000 (UTC) Subject: Getting "LazyImporter' object is not callable" error when trying to send email using python smtplib References: Message-ID: On Mon, 26 Mar 2018 16:47:26 +0530, Sum wrote: > Hi, > > Getting "LazyImporter' object is not callable" error. I have enabled > allow less secure app setting in sender gmail. > > Code : The code you show is not the same as the code you are actually running. The error message you give says: File "test_mail.py", line 64, in send_email(to, SUBJECT, TEXT, attachment_file) File "test_mail.py", line 24, in send_email msg.attach(MIMEText(text)) TypeError: 'LazyImporter' object is not callable Notice the quoted line 24. Here it is again: msg.attach(MIMEText(text)) But line 24 in the code you give us is: msg.attach(part) which is different. Also, you have a line of code: except smtplib.SMTPException,error:: Notice the two colons at the end of the line? That's a syntax error. So the code you have shown us cannot possibly be the same as the code you are running. Please review your code, copy and paste the complete traceback, starting with the line "Traceback", and make sure that you have run the code you show us. It might help to read this page: http://sscce.org/ It's written for Java programmers, but the ideas apply equally to Python. Thank you, -- Steve From steve+comp.lang.python at pearwood.info Mon Mar 26 08:36:26 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 12:36:26 +0000 (UTC) Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <38311d42-a989-5646-e681-83ba2aad2272@Damon-Family.org> Message-ID: On Mon, 26 Mar 2018 11:45:33 +0100, bartc wrote: > Similar overheads occur when you use string=>int even on small numbers: > > This code: > > C = int("12345") > D = C+C # or C*C; about the same results > > takes 5 times as long (using my CPython 3.6.x on Windows) as: > > C = 12345 > D = C+C > > Your arguments that this doesn't really matter would equally apply here. Efficiency-wise, you are correct. But that's not why we (well, some of us...) write C = 12345 instead of C = int("12345") or C = (((int("1")*10 + int("2"))*10 + int("3"))*10 + int("4") )*10 + int(5) Even that second one would probably be an insignificant runtime cost for 99% of real applications. No, the reason why we write C = 12345 is because it is the most straight- forward, natural, idiomatic way to assign 12345, and therefore the least surprising and easiest to read and understand. The fact that it is also faster is a bonus. Contrariwise, we *do* write: D = Decimal("12.345") and similar idioms. When it comes to short integer values, there's no possible benefit to either readability or performance to write it as a string and convert. It is *harder* to read int("12345") than simply 12345. But when it comes to huge ints with hundreds or thousands of digits, that's not the case. In the absence of special syntax to support huge ints, the most readable way to include a HUGE literal int is as a string, and then perform our own processing: C = """123 456 789 012 345 678 901 234 567 ... 567 890 123 456 789 012 345 678 901 ... ...""" C = parse_and_convert(C) and so on. As the author, you get to decide how many extra work you are prepared to do in the conversion step, in order to buy you extra readability. For example, formatting the number in groups of three digits. Now, if we had a specialist language that was focused specifically on huge ints with thousands of digits, then it might be worth building in special syntax for it. Python is already half-way there: recent versions support using underscores in ints to make it easier to group digits. All we really need now is support for multi-line ints. But for a generalist language like Python, it's probably not too clever to try to support ever smaller niche requirements. Especially for an open source project, manpower is always at short supply. There are far more important priorities to attend to. -- Steve From bc at freeuk.com Mon Mar 26 08:46:55 2018 From: bc at freeuk.com (bartc) Date: Mon, 26 Mar 2018 13:46:55 +0100 Subject: Entering a very large number In-Reply-To: References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> Message-ID: <3Z5uC.351501$Jj7.332942@fx45.am4> On 26/03/2018 13:30, Richard Damon wrote: > On 3/26/18 6:31 AM, bartc wrote: >> The purpose was to establish how such int("...") conversions compare >> in overheads with actual arithmetic with the resulting numbers. >> > Of course if this was done in C with a version that had builtin bignum > ints or an aggresive enough optimizer (or a Python that did a similar > level of optimizations) this function would just test the speed of > starting the program, as it actually does nothing and can be optimized > away. Which is a nuisance. /You/ are trying to measure how long it takes to perform a task, the compiler is demonstrating how long it takes to /not/ perform it! So it can be very unhelpful. Hence my testing with CPython 3.6, rather than on something like PyPy which can give results that are meaningless. Because, for example, real code doesn't repeatedly execute the same pointless fragment millions of times. But a real context is too complicated to set up. Yes, something like this can beused to measure the base time to do > something, but the real question should be is that time significant > compared to the other things that the program is doing, Making a 200x > improvement on code that takes 1% of the execution time saves you > 0.995%, not normally worth it unless your program is currently running > at 100.004% of the allowed (or acceptable) timing, if acceptable timing > can even be defined that precisely. I'm usually concerned with optimisation in a more general sense than a specific program. Such a with a library function (where you don't know how it's going to be used); or with a particular byte-code in an interpreter (you don't know how often it will be encountered); or a generated code sequence in a compiler. But even 200x improvement on something that takes 1% of the time can be worthwhile if it is just one of dozens of such improvements. Sometimes these small, incremental changes in performance can add up. And even if it was just 1%, the aggregate savings across one million users of the program can be substantial, even if the individuals won't appreciate it. 1% extra battery life might be a handy five minutes for example. -- bartc From steve+comp.lang.python at pearwood.info Mon Mar 26 09:22:07 2018 From: steve+comp.lang.python at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 13:22:07 +0000 (UTC) Subject: Using object as a class References: <299fc94b-ed77-a5a7-7df0-617c39336b28@VybeNetworks.com> Message-ID: On Mon, 26 Mar 2018 07:31:06 -0500, D'Arcy Cain wrote: > Is this behaviour (object not quite like a class) documented anywhere? It's exactly like a class. It's an immutable class. You are making assumptions about what classes must be able to do. > Does anyone know the rationale for this if any? object is intended as the base class for all others. As such, it is absolutely dead simple, with no state and very few methods other than those required by everything. The usual way we use object instances is as identity-objects: objects with (almost) no behaviour other than their identity: SENTINEL = object() # later if obj is SENTINEL: ... A bit like None. The usual way to get what you want is a simple subclass of object, but even more convenient: py> from types import SimpleNamespace py> x = SimpleNamespace(x=1, y=2, z=3) py> x namespace(x=1, y=2, z=3) -- Steve From p.f.moore at gmail.com Mon Mar 26 09:24:12 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Mar 2018 14:24:12 +0100 Subject: Using object as a class In-Reply-To: <299fc94b-ed77-a5a7-7df0-617c39336b28@VybeNetworks.com> References: <299fc94b-ed77-a5a7-7df0-617c39336b28@VybeNetworks.com> Message-ID: In fact, object acts just like a user-defined class, with __slots__ set to empty: >>> class MyObj(object): ... __slots__ = () ... >>> o = MyObj() >>> o.x = 3 Traceback (most recent call last): File "", line 1, in AttributeError: 'MyObj' object has no attribute 'x' See https://docs.python.org/3.6/reference/datamodel.html#slots IIRC, most built in objects (of which object is one) behave as if they have __slots__ set (they don't actually, because they are written in C, but the effect is the same). Paul On 26 March 2018 at 13:31, D'Arcy Cain wrote: > It's called a super class but it doesn't quite work like a normal class. > >>>> OBJ = object() >>>> OBJ.x = 3 > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'object' object has no attribute 'x' > > I can fix this by creating a NULL class. > >>>> class NullObject(object): pass > ... >>>> OBJ = NullObject() >>>> OBJ.x = 3 >>>> OBJ.x > 3 >>>> > > Is this behaviour (object not quite like a class) documented anywhere? > Does anyone know the rationale for this if any? > > In case anyone wants to know why I am doing this, sometimes I simply > want an object to hold values that I can pass around. I don't need > methods and I don't always know what variables I am going to need. > > And yes, I know that dict is the usual way to do this. > > -- > D'Arcy J.M. Cain > Vybe Networks Inc. > http://www.VybeNetworks.com/ > IM:darcy at Vex.Net VoIP: sip:darcy at VybeNetworks.com > -- > https://mail.python.org/mailman/listinfo/python-list From pfeiffer at cs.nmsu.edu Mon Mar 26 10:14:53 2018 From: pfeiffer at cs.nmsu.edu (Joe Pfeiffer) Date: Mon, 26 Mar 2018 08:14:53 -0600 Subject: Entering a very large number References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> Message-ID: <1bwoxyudnm.fsf@pfeifferfamily.net> Steven D'Aprano writes: > On Mon, 26 Mar 2018 02:37:44 +0100, bartc wrote: > >> If I instead initialise C using 'C = int("288712...")', then timings >> increase as follows: > > Given that the original number given had 397 digits and has a bit length > of 1318, I must admit to some curiosity as to how exactly you managed to > cast it to a C int (32 bits on most platforms). > > It is too big for an int, a long (64 bits), a long-long (128 bits) or > even a long-long-long-long-long-long-long-long-long-long-long-long-long- > long-long-long (1024 bits), if such a thing even exists. > > So what exactly did you do? Curiously, the first time I read the original post I also somehow confused it with C code. But no, the variable is named C, it isn't written in the C language. From mittra at juno.com Mon Mar 26 10:55:59 2018 From: mittra at juno.com (qrious) Date: Mon, 26 Mar 2018 07:55:59 -0700 (PDT) Subject: Synthesize a new series Message-ID: <0f3cc65c-fdea-46e9-a66f-522b2650676c@googlegroups.com> I have a set of series of numbers. The set bears some common property about the series. For example, in one case, Series 1 can be the set of random number of odd numbers starting from 1 and Series N can be the set of random number of odd numbers starting from N. In another case, Series 1 can be the set of random number of odd numbers starting from 1, Series N can be the set of random number of (odd number)^N starting from 5 (arbitrary number), and so on. Each series can be of arbitrary length. Even though the set members are 'similar' in properties, the exact relationship is not known. I want to generate one additional member of the set that is 'similar' in properties with the existing one. Questions: 1. Will Machine Learning be useful for this? If so, which specific term I should do a search on. Sorry for the lack of knowledge here. 2. I want to minimize the amount of coding, with preference for 0 coding. Which off-the-shelf tool (preference: open source and free) will be useful for this? If programming is needed, I would like to use Python and associated libraries. From gheskett at shentel.net Mon Mar 26 11:30:54 2018 From: gheskett at shentel.net (Gene Heskett) Date: Mon, 26 Mar 2018 11:30:54 -0400 Subject: [OT] multicore/cpu history In-Reply-To: References: Message-ID: <201803261130.54340.gheskett@shentel.net> On Monday 26 March 2018 10:06:36 Dennis Lee Bieber wrote: > On Mon, 26 Mar 2018 10:02:15 +0000 (UTC), Steven D'Aprano > > declaimed the following: > >Hardware people can probably tell you what it is that CPUs do that > > FPUs and GPUs don't do. Or specialised Bitcoin mining chips. > > Whatever it is that they don't do, but CPUs do, is probably a good > > dividing line between "CPU" and "auxiliary chip". > > And then... to confuse matters... > > As I recall, the bootloader on the Raspberry Pi runs on the graphics > processor, and it sets up the memory image for Linux before passing > control to the ARM processor. Does that come with docs on how to change kernels in case the one you are running has to be rebooted 2-10 times to get the keyboard/mouse ducks all in a row so it doesn't randomly throw away events? > > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfraed at ix.netcom.com HTTP://wlfraed.home.netcom.com/ -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From rosuav at gmail.com Mon Mar 26 11:31:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Mar 2018 02:31:07 +1100 Subject: Entering a very large number In-Reply-To: <3Z5uC.351501$Jj7.332942@fx45.am4> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> <3Z5uC.351501$Jj7.332942@fx45.am4> Message-ID: On Mon, Mar 26, 2018 at 11:46 PM, bartc wrote: > On 26/03/2018 13:30, Richard Damon wrote: >> >> On 3/26/18 6:31 AM, bartc wrote: > > >>> The purpose was to establish how such int("...") conversions compare in >>> overheads with actual arithmetic with the resulting numbers. >>> >> Of course if this was done in C with a version that had builtin bignum >> ints or an aggresive enough optimizer (or a Python that did a similar level >> of optimizations) this function would just test the speed of starting the >> program, as it actually does nothing and can be optimized away. > > > Which is a nuisance. /You/ are trying to measure how long it takes to > perform a task, the compiler is demonstrating how long it takes to /not/ > perform it! So it can be very unhelpful. Yeah. It's so annoying that compilers work so hard to make your code fast, when all you want to do is measure exactly how slow it is. Compiler authors are stupid. ChrisA From ganesh1pal at gmail.com Mon Mar 26 11:37:22 2018 From: ganesh1pal at gmail.com (Ganesh Pal) Date: Mon, 26 Mar 2018 21:07:22 +0530 Subject: String Formatting with new .format() Message-ID: Hi Team, Just a quick suggestion, on string formatting with .format() which of the below is better , given both give the same result . >>> attempts = 1 >>> msg2 = "Hello" >>> print "Retry attempt:{0} for error:{1}".format(attempts,msg2) Retry attempt:1 for error:Hello OR >>> attempts = 1 >>> msg2 = "Hello" >>> print "Retry attempt:{0} for error:{0}".format(attempts,msg2) Retry attempt:1 for error:1 >>> PS : This is the silly question but I wanted to know if it really makes any difference I am on Python 2.7 and Linux Regards, Ganesh From rosuav at gmail.com Mon Mar 26 11:50:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Mar 2018 02:50:24 +1100 Subject: String Formatting with new .format() In-Reply-To: References: Message-ID: On Tue, Mar 27, 2018 at 2:37 AM, Ganesh Pal wrote: > Hi Team, > > Just a quick suggestion, on string formatting with .format() which of the > below is better , given both give the same result . > >>>> attempts = 1 >>>> msg2 = "Hello" >>>> print "Retry attempt:{0} for error:{1}".format(attempts,msg2) > Retry attempt:1 for error:Hello > > OR > >>>> attempts = 1 >>>> msg2 = "Hello" >>>> print "Retry attempt:{0} for error:{0}".format(attempts,msg2) > Retry attempt:1 for error:1 >>>> Given that both give the same result, I would recommend that purple be promoted as the most transcendental of integers. Or maybe they're not giving the same result. I'm a little confused here. ChrisA From sjsumitj at gmail.com Mon Mar 26 12:18:21 2018 From: sjsumitj at gmail.com (Sum) Date: Mon, 26 Mar 2018 21:48:21 +0530 Subject: Getting "LazyImporter' object is not callable" error when trying to send email using python smtplib In-Reply-To: References: Message-ID: Thanks Steve for your inputs. Now I am able to run the code successfully. # Made changes to import statements as below: from email.mime.base import MIMEBase from email.mime.text import MIMEText Apologies for the typo and indentation error in above mail. Regards, Sumit On Mon, Mar 26, 2018 at 6:04 PM, Steven D'Aprano < steve+comp.lang.python at pearwood.info> wrote: > On Mon, 26 Mar 2018 16:47:26 +0530, Sum wrote: > > > Hi, > > > > Getting "LazyImporter' object is not callable" error. I have enabled > > allow less secure app setting in sender gmail. > > > > Code : > > The code you show is not the same as the code you are actually running. > The error message you give says: > > File "test_mail.py", line 64, in > send_email(to, SUBJECT, TEXT, attachment_file) > File "test_mail.py", line 24, in send_email > msg.attach(MIMEText(text)) > TypeError: 'LazyImporter' object is not callable > > > Notice the quoted line 24. Here it is again: > > msg.attach(MIMEText(text)) > > But line 24 in the code you give us is: > > msg.attach(part) > > > which is different. Also, you have a line of code: > > except smtplib.SMTPException,error:: > > Notice the two colons at the end of the line? That's a syntax error. So > the code you have shown us cannot possibly be the same as the code you > are running. > > Please review your code, copy and paste the complete traceback, starting > with the line "Traceback", and make sure that you have run the code you > show us. > > It might help to read this page: > > http://sscce.org/ > > It's written for Java programmers, but the ideas apply equally to Python. > > > Thank you, > > > > -- > Steve > > -- > https://mail.python.org/mailman/listinfo/python-list > From rgaddi at highlandtechnology.invalid Mon Mar 26 12:27:00 2018 From: rgaddi at highlandtechnology.invalid (Rob Gaddi) Date: Mon, 26 Mar 2018 09:27:00 -0700 Subject: Pip Version Confusion In-Reply-To: References: <20180324154017.GA15995@mail.akwebsoft.com> Message-ID: On 03/24/2018 08:40 AM, Tim Johnson wrote: > I'm on Ubuntu 16.04. > > I'm getting the following message from pip: > > You are using pip version 8.1.1, however version 9.0.3 is available. > You should consider upgrading via the 'pip install --upgrade pip' command. > > # But then I get this : > tim at linus:~/Downloads$ which pip > /home/tim/.local/bin/pip > > # and this: > tim at linus:~/Downloads$ pip --version > pip 9.0.3 from /home/tim/.local/lib/python2.7/site-packages (python 2.7) > > Not sure how to resolve - please advise, > thanks > > -- > Tim > When pip 8.1.1 yells at you about version, are you installing something --user for yourself, or are you using sudo to install it systemwide. Because 'sudo which pip' is probably still pointed to the APT installed one. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. From nadir_musallam at hotmail.com Mon Mar 26 12:40:50 2018 From: nadir_musallam at hotmail.com (nadir musallam) Date: Mon, 26 Mar 2018 16:40:50 +0000 Subject: Fw: Fwd: Welcome to the "Python-list" mailing list In-Reply-To: <5AB7E414.1070606@stoneleaf.us> References: , <5AB7E414.1070606@stoneleaf.us> Message-ID: Dear All, I have tried installing Python3.6.4 on my computer as I am eager to begin a new career in data analytics. However I am running in to some problems when attempting to set up the software. I have downloaded Dev C++ software as per your recommendation earlier but still getting the same error when trying to launch the program. Your assistance would be very much appreciated. I hope this time I followed the proper instructions for the emailing list as it is not my intention to offend anyone. Below is the updated log file for your reference: [1D14:0FEC][2018-03-26T09:09:13]i001: Burn v3.10.3.3007, Windows v10.0 (Build 16299: Service Pack 0), path: C:\Users\DELL\AppData\Local\Temp\{38A48442-951B-485A-BF38-8B6263D4F87D}\.cr\python-3.6.4-amd64-webinstall.exe [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'ActionLikeInstalling' to value 'Installing' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'ActionLikeInstallation' to value 'Setup' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'ShortVersion' to value '3.6' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'ShortVersionNoDot' to value '36' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'WinVer' to value '3.6' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'WinVerNoDot' to value '36' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'InstallAllUsers' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'InstallLauncherAllUsers' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'TargetDir' to value '' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'DefaultAllUsersTargetDir' to value '[ProgramFiles64Folder]Python[WinVerNoDot]' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'TargetPlatform' to value 'x64' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'DefaultJustForMeTargetDir' to value '[LocalAppDataFolder]Programs\Python\Python[WinVerNoDot]' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'OptionalFeaturesRegistryKey' to value 'Software\Python\PythonCore\[WinVer]\InstalledFeatures' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'TargetDirRegistryKey' to value 'Software\Python\PythonCore\[WinVer]\InstallPath' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'DefaultCustomTargetDir' to value '' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'InstallAllUsersState' to value 'enabled' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'InstallLauncherAllUsersState' to value 'enabled' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'CustomInstallLauncherAllUsersState' to value '[InstallLauncherAllUsersState]' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'TargetDirState' to value 'enabled' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'CustomBrowseButtonState' to value 'enabled' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_core' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_exe' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_dev' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_lib' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_test' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_doc' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_tools' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_tcltk' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_pip' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_launcher' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'Include_launcherState' to value 'enabled' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_symbols' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Include_debug' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'LauncherOnly' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'DetectedLauncher' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'DetectedOldLauncher' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'AssociateFiles' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'Shortcuts' to value '1' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'PrependPath' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'CompileAll' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing numeric variable 'SimpleInstall' to value '0' [1D14:0FEC][2018-03-26T09:09:13]i000: Initializing string variable 'SimpleInstallDescription' to value '' [1D14:0FEC][2018-03-26T09:09:13]i009: Command Line: '-burn.clean.room=C:\Users\DELL\Desktop\Python\python-3.6.4-amd64-webinstall.exe -burn.filehandle.attached=616 -burn.filehandle.self=524' [1D14:0FEC][2018-03-26T09:09:13]i000: Setting string variable 'WixBundleOriginalSource' to value 'C:\Users\DELL\Desktop\Python\python-3.6.4-amd64-webinstall.exe' [1D14:0FEC][2018-03-26T09:09:13]i000: Setting string variable 'WixBundleOriginalSourceFolder' to value 'C:\Users\DELL\Desktop\Python\' [1D14:0FEC][2018-03-26T09:09:13]i000: Setting string variable 'WixBundleLog' to value 'C:\Users\DELL\AppData\Local\Temp\Python 3.6.4 (64-bit)_20180326090913.log' [1D14:0FEC][2018-03-26T09:09:13]i000: Setting string variable 'WixBundleName' to value 'Python 3.6.4 (64-bit)' [1D14:0FEC][2018-03-26T09:09:13]i000: Setting string variable 'WixBundleManufacturer' to value 'Python Software Foundation' [1D14:0FEC][2018-03-26T09:09:14]i000: Setting numeric variable 'CRTInstalled' to value 1 [1D14:144C][2018-03-26T09:09:14]i000: Did not find C:\Users\DELL\Desktop\Python\unattend.xml [1D14:144C][2018-03-26T09:09:14]i000: Setting string variable 'ActionLikeInstalling' to value 'Installing' [1D14:144C][2018-03-26T09:09:14]i000: Setting string variable 'ActionLikeInstallation' to value 'Setup' [1D14:144C][2018-03-26T09:09:14]i000: Setting version variable 'WixBundleFileVersion' to value '3.6.4150.0' [1D14:144C][2018-03-26T09:09:14]i000: Target OS is Windows 7 SP1 or later [1D14:0FEC][2018-03-26T09:09:14]i100: Detect begin, 58 packages [1D14:0FEC][2018-03-26T09:09:14]i107: Detected forward compatible bundle: {035e803e-bcc1-4b95-ab44-d33a027f963d}, type: Upgrade, scope: PerUser, version: 3.6.4150.0, enabled: No [1D14:0FEC][2018-03-26T09:09:14]i102: Detected related bundle: {035e803e-bcc1-4b95-ab44-d33a027f963d}, type: Upgrade, scope: PerUser, version: 3.6.4150.0, operation: None [1D14:0FEC][2018-03-26T09:09:14]e000: Related bundle {035e803e-bcc1-4b95-ab44-d33a027f963d} is preventing install [1D14:0FEC][2018-03-26T09:09:14]i000: Setting numeric variable 'InstallLauncherAllUsers' to value 1 [1D14:0FEC][2018-03-26T09:09:14]i000: Setting numeric variable 'Include_launcher' to value 1 [1D14:0FEC][2018-03-26T09:09:14]i000: Setting numeric variable 'DetectedLauncher' to value 1 [1D14:0FEC][2018-03-26T09:09:14]i000: Setting string variable 'Include_launcherState' to value 'disable' [1D14:0FEC][2018-03-26T09:09:14]i000: Setting string variable 'InstallLauncherAllUsersState' to value 'disable' [1D14:0FEC][2018-03-26T09:09:14]i000: Setting numeric variable 'AssociateFiles' to value 1 [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.0_x86, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.0_x64, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.1_x86, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.1_x64, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.2_x86, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.2_x64, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.3_x86, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: crt_14.0_v6.3_x64, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_AllUsers_pdb, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_JustForMe_pdb, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: core_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: dev_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: dev_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: dev_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: dev_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: exe_AllUsers, feature: DefaultFeature, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: exe_AllUsers, feature: Shortcuts, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_AllUsers_pdb, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: exe_JustForMe, feature: DefaultFeature, state: Local [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: exe_JustForMe, feature: Shortcuts, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_JustForMe_pdb, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: exe_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_AllUsers_pdb, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_JustForMe_pdb, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: lib_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_AllUsers_pdb, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_JustForMe_pdb, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: test_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: doc_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: doc_AllUsers, feature: DefaultFeature, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: doc_AllUsers, feature: Shortcuts, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: doc_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: doc_JustForMe, feature: DefaultFeature, state: Local [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: doc_JustForMe, feature: Shortcuts, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tools_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tools_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_AllUsers, feature: DefaultFeature, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_AllUsers, feature: AssociateFiles, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_AllUsers, feature: Shortcuts, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_AllUsers_pdb, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_AllUsers_pdb, feature: Symbols, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_AllUsers_d, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_AllUsers_d, feature: DebugBinaries, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_JustForMe, feature: DefaultFeature, state: Local [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_JustForMe, feature: AssociateFiles, state: Local [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_JustForMe, feature: Shortcuts, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_JustForMe_pdb, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_JustForMe_pdb, feature: Symbols, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: tcltk_JustForMe_d, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: tcltk_JustForMe_d, feature: DebugBinaries, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: launcher_AllUsers, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: launcher_AllUsers, feature: DefaultFeature, state: Local [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: launcher_AllUsers, feature: AssociateFiles, state: Local [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: launcher_JustForMe, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: launcher_JustForMe, feature: DefaultFeature, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i104: Detected package: launcher_JustForMe, feature: AssociateFiles, state: Absent [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: pip_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: pip_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: path_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: path_JustForMe, state: Present, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileall_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileallO_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileallOO_AllUsers, state: Absent, cached: None [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileall_JustForMe, state: Absent, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileallO_JustForMe, state: Absent, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i101: Detected package: compileallOO_JustForMe, state: Absent, cached: Complete [1D14:0FEC][2018-03-26T09:09:14]i000: Setting string variable 'TargetDir' to value 'C:\Users\DELL\AppData\Local\Programs\Python\Python36' [1D14:0FEC][2018-03-26T09:09:14]i199: Detect complete, result: 0x0 ________________________________ From: Ethan Furman Sent: Sunday, March 25, 2018 10:01 PM To: nadir musallam Cc: Tim Golden Subject: Re: Fwd: Welcome to the "Python-list" mailing list On 03/25/2018 10:18 AM, nadir musallam wrote: > I have already subscribed ( see below) to the emailing list and tried to follow the instructions in the rejection > message (without success) before sending that last email. Should I send an email again? Please advise. Yes, please send it again, to python-list at python.org. If there are still problems we'll figure it out then. -- ~Ethan~ From rantingrickjohnson at gmail.com Mon Mar 26 14:37:35 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Mon, 26 Mar 2018 11:37:35 -0700 (PDT) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> On Monday, March 26, 2018 at 5:46:03 AM UTC-5, Steven D'Aprano wrote: > Rick, you're supposedly familiar with Ruby. And yet, you > didn't notice that your supposed "fix" didn't touch any > executable code, all it did was modify the strings being > printed. Because the goal was to *UN-OBFUSCATE* the code, not to provide a solution for the problem _you_ created. > Because of this "fix", the printed strings no longer match > the code being executed, but the strange, inconsistent > behaviour still occurs. The supposed "inconsistent behavior" here has absolutely nothing to do with Ruby, no, it's all on _you_. _YOU_ are the one who created a non-sensical function with a single char name; and _YOU_ are the one who placed a call to that function in the middle of an expression, which, on first glance, looks to be a common numeric addition or string concatenation. There are no syntactical clues that `a` is a function. Thus, it is _YOU_ who is to blame for the supposed "unexpected output". Ruby followed the rules. But you didn't. Like a mad scientist you injected obfuscation into your source code and then blamed Ruby because your poor naming conventions and your clumsily formed expressions did not produce the "purdy little result" you expected. Congratulations, you have forsaken the most vital principle of good programming -> Readability Counts!. "Code is read more often than it is written" (ring a bell?). No self-respecting professional programmer would ever write in such a manner. Its the kind of crapola that will get you fired in 2 seconds. Yet, here you are -- THE GREAT AND POWERFUL D'APRANO -- thumping your chest over this horrendous code and acting as if we should be proud of you. Yip of the day: No one is proud of you. > Here's the code again: No thanks. I've seen more than i care to see of your little "examples". > Of course we can solve the problem by always using > parentheses when making function calls. I'm sorry, but _we_ are not required to solve the problems _you_ create. _You_ made that mess, and now _you_ can clean it up. > But that misses the point that Ruby allows this > inconsistency in the first place. Hmm, i see. And if the operator of an automobile purposesly drives off a cliff, who do you blame: driver or manufacturer? From drsalists at gmail.com Mon Mar 26 15:24:54 2018 From: drsalists at gmail.com (Dan Stromberg) Date: Mon, 26 Mar 2018 12:24:54 -0700 Subject: A question related to the PYTHONPATH In-Reply-To: <87d0zrtli7.fsf@handshake.de> References: <87d0zrtli7.fsf@handshake.de> Message-ID: On Sun, Mar 25, 2018 at 11:10 PM, dieter wrote: > adrien oyono writes: >> I have recently read the documentation about how imports work on python, >> and I was wondering why, when you execute a python file, the current >> directory is not added by default to the PYTHONPATH ? > > Maybe, to avoid surprises? > > You can invoke a script from different positions in your file system. > If PYTHONPATH would automatically get ".", the script's behaviour > could depend on the position from where it is called. IINM, this also has security implications. From ian.g.kelly at gmail.com Mon Mar 26 15:36:04 2018 From: ian.g.kelly at gmail.com (Ian Kelly) Date: Mon, 26 Mar 2018 13:36:04 -0600 Subject: A question related to the PYTHONPATH In-Reply-To: References: <87d0zrtli7.fsf@handshake.de> Message-ID: On Mon, Mar 26, 2018 at 1:24 PM, Dan Stromberg wrote: > On Sun, Mar 25, 2018 at 11:10 PM, dieter wrote: >> adrien oyono writes: >>> I have recently read the documentation about how imports work on python, >>> and I was wondering why, when you execute a python file, the current >>> directory is not added by default to the PYTHONPATH ? >> >> Maybe, to avoid surprises? >> >> You can invoke a script from different positions in your file system. >> If PYTHONPATH would automatically get ".", the script's behaviour >> could depend on the position from where it is called. > > IINM, this also has security implications. It's curious then that the -m invocation does add the current directory, since it's normally used for running standard library scripts like timeit or venv. From python at bladeshadow.org Mon Mar 26 15:43:05 2018 From: python at bladeshadow.org (Python) Date: Mon, 26 Mar 2018 14:43:05 -0500 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: <20180326194305.GX10273@bladeshadow.org> On Sun, Mar 25, 2018 at 10:33:49AM -0700, Rick Johnson wrote: > > [steve at ando ruby]$ ruby ws-example.rb > > a + b => 7 > > a+b => 7 > > a+ b => 7 > > a +b => 3 > > > > Here's the source code: > > > > # --- cut --- > > def a(x=4) > > x+2 > > end > > > > b = 1 > > print "a + b => ", (a + b), "\n" > > print "a+b => ", (a+b), "\n" > > print "a+ b => ", (a+ b), "\n" > > print "a +b => ", (a +b), "\n" > > # --- cut --- > > > Short of demonstrating that you have knack for writing > obfuscated code :-), i don't see the point here. I think the point is 7 != 3... > Here, i took the liberty of un-obfuscating your code [...] Not to me, you didn't. While I don't know ruby, I already got from context that "a" was a function call, including without the parens, albeit annoying to read. I've previously seen that in Perl, and never liked it there either, because you can't immediately tell by looking at it whether it is a variable or a function call. While I think that itself speaks to Stephen's consistency point, I don't think that's too controvercial. But what I don't get is how the last print statement prints 3. There seems to be some kind of magic happening related to the positioning of the + relative to the operand(s). That's neither consistent nor intuitive. Or, it's some other thing caused by a detail I overlooked, but if so your "unobfuscated" version did nothing to clarify what that detail is, even after scrutinizing it somewhat closely. In either case, I can't imagine how Ruby arrives at the answer 3. From python at bladeshadow.org Mon Mar 26 15:48:50 2018 From: python at bladeshadow.org (Python) Date: Mon, 26 Mar 2018 14:48:50 -0500 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> Message-ID: <20180326194850.GY10273@bladeshadow.org> On Mon, Mar 26, 2018 at 10:43:32AM +0000, Steven D'Aprano wrote: > The kicker is that out of these four legal, parenthesis-free ways of > calling function a, *three* of them interpret the expression as: > > call a with no arguments > then add b using the binary plus operator > > but the last, differing only in whitespace, interprets it as: > > call a with a single argument, unary-plus b Thanks for the explanation. That's fucking stupid. I usually prefer to refrain from cursing in posts but in this case I think that level of emphasis is required. And FWIW, this convinces me that Ruby is not a language I ever want to use. From rosuav at gmail.com Mon Mar 26 16:03:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Mar 2018 07:03:49 +1100 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> References: <394e297b-a613-423f-9c35-74b626e46280@googlegroups.com> <937e7690-6774-44e5-b846-b3d4f1ee4fe0@googlegroups.com> <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> Message-ID: On Tue, Mar 27, 2018 at 5:37 AM, Rick Johnson wrote: > The supposed "inconsistent behavior" here has absolutely > nothing to do with Ruby, no, it's all on _you_. _YOU_ are > the one who created a non-sensical function with a single > char name; and _YOU_ are the one who placed a call to that > function in the middle of an expression, which, on first > glance, looks to be a common numeric addition or string > concatenation. There are no syntactical clues that `a` is a > function. Thus, it is _YOU_ who is to blame for the supposed > "unexpected output". > > Ruby followed the rules. > > But you didn't. Yeah, it's entirely Steven's fault that he was expecting cricket but Ruby wanted to play Calvinball. But hey, Ruby was totally following the rules. ChrisA From python at bladeshadow.org Mon Mar 26 16:08:20 2018 From: python at bladeshadow.org (Python) Date: Mon, 26 Mar 2018 15:08:20 -0500 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> References: <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> Message-ID: <20180326200820.GZ10273@bladeshadow.org> On Mon, Mar 26, 2018 at 11:37:35AM -0700, Rick Johnson wrote: > > Because of this "fix", the printed strings no longer match > > the code being executed, but the strange, inconsistent > > behaviour still occurs. > > The supposed "inconsistent behavior" here has absolutely > nothing to do with Ruby, no, it's all on _you_. This is utter nonsense. Ruby's parser allowed the unintuitive/inconsistent behavior, so it has everything to do with Ruby. The fact that the language allows you to write such expressions means that SOMEONE WILL. Your own messages seem to promote Ruby's paren-less function calls as a good thing (which I vehemently disagree with, but I can accept that's a matter of preference). However, a language that allows (a + b) to evaluate to a different value from (a +b) (at the same execution point) is syntactically abysmal. > Ruby followed the rules. > But you didn't. Nonsense... Your language's syntax parser is what defines the rules. All of the expressions Stephen wrote did not yeild a syntax error, therefore he "followed the rules." > No self-respecting professional programmer would ever write > in such a manner. I think your expectation here is much too high. I've seen A LOT of Perl written by, for example, professional testers, of this ilk, and I've seen it cause bugs when a) they got the syntax very slightly wrong, or b) they got the syntax right but someone else to whom the intention wasn't clear "fixed" it. Humans are already good enough at making mistakes that they require no additional encouragement, such as what is provided by allowing such syntactical horrors. From auriocus at gmx.de Mon Mar 26 17:04:48 2018 From: auriocus at gmx.de (Christian Gollwitzer) Date: Mon, 26 Mar 2018 23:04:48 +0200 Subject: Entering a very large number In-Reply-To: <_Z3uC.218815$TS3.193851@fx18.am4> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> Message-ID: Am 26.03.18 um 12:31 schrieb bartc: > On 26/03/2018 10:34, Steven D'Aprano wrote: >> So what exactly did you do? > > I did this: > > def fn(): > ??? C = int( > ??? "28871482380507712126714295971303939919776094592797" > ??? "22700926516024197432303799152733116328983144639225" > ??? "94197780311092934965557841894944174093380561511397" > ??? "99994215424169339729054237110027510420801349667317" > ??? "55152859226962916775325475044445856101949404200039" > ??? "90443211677661994962953925045269871932907037356403" > ??? "22737012784538991261203092448414947289768854060249" > ??? "76768122077071687938121709811322297802059565867") > > #?? C = 2887148238050771212671429... [truncated for this post] > > ??? D=C+C > > for i in range(1000000): > ??? fn() > > The purpose was to establish how such int("...") conversions compare in > overheads with actual arithmetic with the resulting numbers. I still claim that it is not necessary to perform the conversion on each function call. For instance, the following change speeds it up by a factor of 7 on my machine: ======================================================= def fn(C = int( "28871482380507712126714295971303939919776094592797" "22700926516024197432303799152733116328983144639225" "94197780311092934965557841894944174093380561511397" "99994215424169339729054237110027510420801349667317" "55152859226962916775325475044445856101949404200039" "90443211677661994962953925045269871932907037356403" "22737012784538991261203092448414947289768854060249" "76768122077071687938121709811322297802059565867")): # C = 2887148238050771212671429... [truncated for this post] D=C+C return D for i in range(1000000): fn() #print(fn()) ======================================================= Out of pure laziness, I haven't compared it to the version with the long string. Maybe tha gurus here find yet another elegant way to assign the constant only once; my trial with "fn.C = foobar" was not successful. Christian From rantingrickjohnson at gmail.com Mon Mar 26 17:19:12 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Mon, 26 Mar 2018 14:19:12 -0700 (PDT) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <78ac92aa-a9e2-44d1-9675-05da3af7e790@googlegroups.com> <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> <20180326200820.GZ10273@bladeshadow.org> Message-ID: On Monday, March 26, 2018 at 3:09:38 PM UTC-5, Python wrote: > On Mon, Mar 26, 2018 at 11:37:35AM -0700, Rick Johnson wrote: [...] > > Ruby followed the rules. > > But you didn't. > > Nonsense... Your language's syntax parser is what defines > the rules. All of the expressions Stephen wrote did not > yeild a syntax error, therefore he "followed the rules." Hmm. If "syntax parser rules" could prevent poorly formatted code, then there'd be no need for style guides. > > No self-respecting professional programmer would ever > > write in such a manner. > > I think your expectation here is much too high. I've seen > A LOT of Perl written by, for example, professional > testers, of this ilk, and I've seen it cause bugs when a) > they got the syntax very slightly wrong, or b) they got the > syntax right but someone else to whom the intention wasn't > clear "fixed" it. Perl is a special case (and thus not a good retort) because the language itself was purposely intended to be cryptic. Thus, what Perl programmers consider to be "normal and perfectly acceptable code" would cause the rest of us to run away in horror. "Debug that mess?" o_O "NO WAY!" > Humans are already good enough at making mistakes that they > require no additional encouragement, such as what is > provided by allowing such syntactical horrors. Agreed. And that's why we must respect and follow the code styling wisdom which has been passed down by those who struggled before us. Sure, the behavior that Steven uncovered is odd, but it could be that Maz harbors a strong disliking for undisciplined pupils, and thus, he designed and placed this little trap in the hopes the pain it induced might encourage the petulant little meat-heads to follow some sensible styling rules. IOWs: It's not a bug, dude, it's a feature. PS: ("meat-head": it's what's for dinner) From gheskett at shentel.net Mon Mar 26 18:07:56 2018 From: gheskett at shentel.net (Gene Heskett) Date: Mon, 26 Mar 2018 18:07:56 -0400 Subject: [OT] multicore/cpu history In-Reply-To: References: <201803261130.54340.gheskett@shentel.net> Message-ID: <201803261807.56042.gheskett@shentel.net> On Monday 26 March 2018 12:12:46 Dennis Lee Bieber wrote: > On Mon, 26 Mar 2018 11:30:54 -0400, Gene Heskett > > > declaimed the following: > >On Monday 26 March 2018 10:06:36 Dennis Lee Bieber wrote: > > > > >> As I recall, the bootloader on the Raspberry Pi runs on the > >> graphics processor, and it sets up the memory image for Linux > >> before passing control to the ARM processor. > > > >Does that come with docs on how to change kernels in case the one you > > are running has to be rebooted 2-10 times to get the keyboard/mouse > > ducks all in a row so it doesn't randomly throw away events? > > If Broadcom hasn't changed things, the bootloader is a black-box blob > provided by Broadcom to the R-Pi foundation. The Linux kernel likely > falls under regular Linux image management (the bootloader doesn't > contain the Linux image, just runs on the graphics processor to load > Linux). I don't think R-Pi uses U-Boot (whereas the BeagleBone Black > is shoving everything into U-Boot -- making most of the BBB text books > out-of-date as the kernel no longer loads device tree overlays, > they've been pre-loaded by U-Boot) Which to me, isn't a lot of help. But, thats broadcom... Too bad I can't put a bounty on them. It seems to me that any outfit with more lawyers than engineers ought to fall over from top heavy and foot damage eventually, but they seem to be the exception to that rule. OTOH, its even harder to get any usable info out of Pine to facilitate using an rtai kernel on a product of theirs called a rock64. Which can build that kernel in under an hour. And it does use u-boot, or claims to. Here we have another credit card sized boy wonder computer thats probably 10x or more faster than a top of the line pi, with a usb3 port, and the info as to how to make it work seems locked in a safe behind non-disses they can't even admit to. Hell of a way to run a train. Thanks Dennis. > -- > Wulfraed Dennis Lee Bieber AF6VN > wlfraed at ix.netcom.com HTTP://wlfraed.home.netcom.com/ -- Cheers, Gene Heskett -- "There are four boxes to be used in defense of liberty: soap, ballot, jury, and ammo. Please use in that order." -Ed Howdershelt (Author) Genes Web page From sh at changeset.nyc Mon Mar 26 18:16:26 2018 From: sh at changeset.nyc (Sumana Harihareswara) Date: Mon, 26 Mar 2018 18:16:26 -0400 Subject: please test the new PyPI (now in beta) Message-ID: The new Python Package Index at https://pypi.org is now in beta. This means the site is robust, but we anticipate needing more user testing and changes before it is "production-ready" and can fully replace https://pypi.python.org . We hope to complete the transition before the end of April 2018. We're still working to ensure the new codebase and infrastructure are reliable. So please don't rely on it (yet) unless you can handle the occasional minor outage. But we want you to try the new PyPI, test it, and tell us if you have any problems. During the beta, we'll have IRC and Twitter livechats to hear from you: * Tuesday, March 27th, 16:00-17:00 UTC * Friday, March 30th, 14:00-15:00 UTC * Tuesday, April 3rd, 15:00-16:00 UTC * Friday, April 5th, 0:00-1:00 UTC More at https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html . For future PyPI news, please subscribe to the low-traffic PyPI announcement email list: https://mail.python.org/mm3/archives/list/pypi-announce at python.org/ Thank you. -- Sumana Harihareswara Warehouse project manager Changeset Consulting https://changeset.nyc From charles.wilt at gmail.com Mon Mar 26 18:56:37 2018 From: charles.wilt at gmail.com (Charles Wilt) Date: Mon, 26 Mar 2018 16:56:37 -0600 Subject: Fwd: Fwd: ftplib sending data out of order In-Reply-To: References: <87r2y0oepp.fsf@handshake.de> <22873.56950.619647.559064@localhost.localdomain> <22874.50195.673292.269593@localhost.localdomain> <22877.17910.246760.419726@localhost.localdomain> Message-ID: Just to close out this thread in case anybody finds it... We're reasonably sure the issue was cause by version 14.6 MP1 of Symantec Vontu; which is a Data Loss Prevention (DLP) product that "inspects" packets before they leave the PC. I believe the issue was reported to Symantec. In the meantime, we worked around the issue by extending the base ftplib.FTP class and adding a new function: #stortext combines block read from STORBINARY with ASCII transfer mode from STORLINEdef stortext(self, cmd, fp, blocksize=8192, callback=None, rest=None): self.voidcmd('TYPE A') conn = self.transfercmd(cmd, rest) while 1: buf = fp.read(blocksize) if not buf: break conn.sendall(buf) if callback: callback(buf) conn.close() return self.voidresp() We stopped having issues with scrambled code. As a bonus, the transfer run considerably faster than they did with storline. So we've continued to use our new function. On Mon, Jul 10, 2017 at 7:23 AM, Charles Wilt wrote: > I downloaded the 3.6.1 source and added some debug logging to > sock_sendall() and sock_send_impl() in socketmodule.c. > > As I understand it, sock_send_impl() is where the data is passed to the > OS. > > I can see that sock_send_impl() is getting the data in the correct order. > > > > > On Wed, Jul 5, 2017 at 2:03 PM, Dieter Maurer wrote: > >> Charles Wilt wrote at 2017-7-5 11:08 -0600: >> >On Mon, Jul 3, 2017 at 4:24 PM, Dieter Maurer >> wrote: >> > >> >> >> >> It is not the "readline" that changes depending on the callback. >> >> Keep in mind that "conn.sendall" can return before the bytes are >> actually >> >> sent. >> >> >> > >> >?Not according to this answer >> >https://stackoverflow.com/questions/34252273/what-is-the-di >> fference-between-socket-send-and-socket-sendall >> >> "sendall" internally calls "send" until all the bytes passed as argument >> are accepted by "send". Below "send", the things are packeted, transmitted >> and reassembled. >> >> If send returns, this does *not* mean that the data has been successfully >> transfered. It only means that a lower level has taken over responsibilty >> for the transfer. >> >> As a consequence, "sendall" may return before the data has been >> successfully transfered; it only means that the complete data was >> accepted by the lower (network) level. >> >> >> >And if sendall can return before the data is sent....wouldn't that make >> it >> >the wrong choice for an FTP client? >> >> Why? >> >> The data is transfered to the lower level and this has taken >> over responsibility for the transfer. "ftplib" is using a TCP socket. >> TCP promises to ensure a reliable transfer (by using retransmissions, if >> necessary and possible). Under normal conditions, the data will be >> transfered; >> thus, why wait until the transfer is complete. >> >> >> >> -- >> Dieter >> > > From python at bladeshadow.org Mon Mar 26 19:10:04 2018 From: python at bladeshadow.org (Python) Date: Mon, 26 Mar 2018 18:10:04 -0500 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> <20180326200820.GZ10273@bladeshadow.org> Message-ID: <20180326231004.GA10273@bladeshadow.org> On Mon, Mar 26, 2018 at 02:19:12PM -0700, Rick Johnson wrote: > On Monday, March 26, 2018 at 3:09:38 PM UTC-5, Python wrote: > > On Mon, Mar 26, 2018 at 11:37:35AM -0700, Rick Johnson wrote: > [...] > > > Ruby followed the rules. > > > But you didn't. > > > > Nonsense... Your language's syntax parser is what defines > > the rules. All of the expressions Stephen wrote did not > > yeild a syntax error, therefore he "followed the rules." > > Hmm. If "syntax parser rules" could prevent poorly formatted > code, then there'd be no need for style guides. It may be telling that my team has minimal style standards for C/C++, and none at all for Python... [We don't feel it requires any.] > > > No self-respecting professional programmer would ever > > > write in such a manner. I should also have said that I think this assumes a certain level of expertise with the language. We all start out as novices with every new language, and may spend a very long time there, depending on how much we use it... So, basically, I'm saying that's false. :) > > I think your expectation here is much too high. I've seen > > A LOT of Perl written by, for example, professional > > testers, of this ilk, and I've seen it cause bugs when a) > > they got the syntax very slightly wrong, or b) they got the > > syntax right but someone else to whom the intention wasn't > > clear "fixed" it. > > Perl is a special case (and thus not a good retort) because > the language itself was purposely intended to be cryptic. I think Larry would disagree with that assessment... > Thus, what Perl programmers consider to be "normal and > perfectly acceptable code" would cause the rest of us to run > away in horror. Regardless of what Larry says, I agree with that assessment. Perl has been relegated to "only use when there is no option" in my regieme, which basically means only when it is absolutely necessary to maintain some legacy garbage that's already Perl. > > Humans are already good enough at making mistakes that they > > require no additional encouragement, such as what is > > provided by allowing such syntactical horrors. > > Agreed. And that's why we must respect and follow the code > styling wisdom which has been passed down by those who > struggled before us. Sure, the behavior that Steven > uncovered is odd, but it could be that Maz harbors a strong > disliking for undisciplined pupils [...] > IOWs: It's not a bug, dude, it's a feature. I dunno, this smells a lot like BS. Ruby touts itself as being a simple language with elegant syntax. This thread is my only exposure to it to date, but what I've seen here is, frankly, the exact opposite of that. You should not need a map to distinguish function calls from variables, or operands from function arguments, let alone have the latter depend on the position of the operator token relative to any whitespace between it and the two tokens on either side of it. That's hard to read, complicated, inelegant, and just poor syntax. [My opinion, obviously.] That's not enough exposure to decide that the language isn't simple and elegant overall, but it certainly does induce a strong negative prejudice. From adrienoyono at gmail.com Mon Mar 26 19:52:46 2018 From: adrienoyono at gmail.com (oyono) Date: Mon, 26 Mar 2018 16:52:46 -0700 (PDT) Subject: A question related to the PYTHONPATH In-Reply-To: References: <87d0zrtli7.fsf@handshake.de> Message-ID: <9a12013e-7e3e-4ffb-ae04-faeba8e9bfe0@googlegroups.com> Le lundi 26 mars 2018 08:11:02 UTC+2, dieter a ?crit?: > adrien oyono writes: > > I have recently read the documentation about how imports work on python, > > and I was wondering why, when you execute a python file, the current > > directory is not added by default to the PYTHONPATH ? > > Maybe, to avoid surprises? > > You can invoke a script from different positions in your file system. > If PYTHONPATH would automatically get ".", the script's behaviour > could depend on the position from where it is called. > > Prepending "." to "PYTHONPATH" could have dramatic effects: > assume your script uses module "xxx" and your current working > directory accidentally has a "xxx.py"; then this "xxx.py" would > be used and it may behave much differently from another one. > > The problem is not so severe when "." is appended to "PYTHONPATH". > Nevertheless, it likely is not a good idea to have script behaviour > depending by default on the execution location. Thank you Dieter. I was thinking, maybe it could have been done this way to enforce not running module files that are supposed to be bundled into packages as "independant" python scripts...Therefore, running "python script.py" should be reserved to effectively independant python scripts that do not import from a user-created package or module. I think I need to learn a little bit more about the python cli. Many thanks again From ygguang1992 at gmail.com Mon Mar 26 20:13:50 2018 From: ygguang1992 at gmail.com (W Yg) Date: Mon, 26 Mar 2018 17:13:50 -0700 (PDT) Subject: String Formatting with new .format() In-Reply-To: References: Message-ID: ? 2018?3?26???? UTC+8??11:37:46?Ganesh Pal??? > Hi Team, > > Just a quick suggestion, on string formatting with .format() which of the > below is better , given both give the same result . > > >>> attempts = 1 > >>> msg2 = "Hello" > >>> print "Retry attempt:{0} for error:{1}".format(attempts,msg2) > Retry attempt:1 for error:Hello > > OR > > >>> attempts = 1 > >>> msg2 = "Hello" > >>> print "Retry attempt:{0} for error:{0}".format(attempts,msg2) > Retry attempt:1 for error:1 > >>> > > > PS : This is the silly question but I wanted to know if it really makes any > difference > > I am on Python 2.7 and Linux > > Regards, > Ganesh The method format of str can have extra arguments. I think it is more elegance than used like this way >>> print "Retry attempt:%s for error:%s". % (attempts,msg2) From jladasky at itu.edu Mon Mar 26 20:18:47 2018 From: jladasky at itu.edu (jladasky at itu.edu) Date: Mon, 26 Mar 2018 17:18:47 -0700 (PDT) Subject: Invoke an independent, non-blocking process from Python 3.6? Message-ID: <1545856e-ecc8-4502-b9ef-a86bc7aefea7@googlegroups.com> Hi folks, I've run into an odd situation. I have a custom USB peripheral device which generates real-time data. I monitor this device using a PyQt5 app that I wrote. Periodically I want to capture some of this data in files. Because of a transient OS bug which apparently involves a corner case in the Linux serial port driver (!), I cannot guarantee that I can transmit commands to the device to shut off data transmission while I'm saving a file. When my setup gets handicapped with just one-way communication, the PyQt5 file selector dialog box pauses for an unusually long time. My data gets saved to disk, but then my application segfaults. I have to restart the whole device. In time-critical data acquisition, this won't be good. My hypothesis is that the PyQt message queue is getting choked with data during the disk write operation, and/or when the QFileDialog box pops up. I can't prove this, but I am contemplating a work-around. I plan to separate my data analysis and annotation windows completely from the live data-acquisition application. I just modified my program so that the data streams to a file in /tmp from the acquisition program when I click a Record button. I can click a Stop button, and the /tmp file closes without crashing the live data application. I can repeat this as often as I want. So far, so good. My CPU has a dozen cores. I think that all I need to do now is to start a second Python interpreter which runs an analysis-only application. Then when the QFileDialog from the analysis app is on the screen, there's no chance that it would disrupt the message queue in the live-acquisition app. I have used multiprocessing before when I wrote some parallelized code. That program required significant communication between processes, and it's overkill for my purpose here. I don't need communication between the spawning (live data) program and the spawned program. In fact, to the extent that the live data program has to pay attention to anything besides the data stream, I think it could be bad. I have been investigating the subprocess module. I'm looking for something which behaves like subprocess.run("python3 my_program.py"), but which does not "Wait for command to complete, then return a CompletedProcess instance." Any suggestions are appreciated. Thanks! From ned at nedbatchelder.com Mon Mar 26 20:30:42 2018 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 26 Mar 2018 20:30:42 -0400 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: <20180326231004.GA10273@bladeshadow.org> References: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> <20180326200820.GZ10273@bladeshadow.org> <20180326231004.GA10273@bladeshadow.org> Message-ID: On 3/26/18 7:10 PM, Python wrote: >>> Humans are already good enough at making mistakes that they >>> require no additional encouragement, such as what is >>> provided by allowing such syntactical horrors. >> Agreed. And that's why we must respect and follow the code >> styling wisdom which has been passed down by those who >> struggled before us. Sure, the behavior that Steven >> uncovered is odd, but it could be that Maz harbors a strong >> disliking for undisciplined pupils > [...] >> IOWs: It's not a bug, dude, it's a feature. > I dunno, this smells a lot like BS. It's a losing game to engage Rick in debate: "Ranting Rick" isn't trying to enlighten, educate, or learn. He's trying to rile people up, and he is good at it. --Ned. From rosuav at gmail.com Mon Mar 26 20:34:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Mar 2018 11:34:17 +1100 Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: <20180326231004.GA10273@bladeshadow.org> References: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> <20180326200820.GZ10273@bladeshadow.org> <20180326231004.GA10273@bladeshadow.org> Message-ID: On Tue, Mar 27, 2018 at 10:10 AM, Python wrote: > Ruby touts itself as being a simple language with elegant syntax. > This thread is my only exposure to it to date, but what I've seen here > is, frankly, the exact opposite of that. You should not need a map to > distinguish function calls from variables, or operands from function > arguments, let alone have the latter depend on the position of the > operator token relative to any whitespace between it and the two > tokens on either side of it. That's hard to read, complicated, > inelegant, and just poor syntax. [My opinion, obviously.] That's not > enough exposure to decide that the language isn't simple and elegant > overall, but it certainly does induce a strong negative prejudice. My understanding of Ruby's design model is that it prefers "intuitive" to "easy to explain". That means there's a lot of magic to make things behave the way you would expect. The trouble is that you then end up with edge cases; the benefit is that you get an easy REPL. Let me compare three REPLs for you (I'm not a Ruby expert so I won't use it here): === JavaScript === rosuav at sikorsky:~$ node > function f() { ... let x = 1 ... let y = 2 ... return x + y ... } undefined > f() 3 > === Pike === rosuav at sikorsky:~$ pike Pike v8.1 release 11 running Hilfe v3.5 (Incremental Pike Frontend) Ok. > int f() { >> int x = 1; >> int y = 2; >> return x + y; >> } > f(); (1) Result: 3 > === Python === rosuav at sikorsky:~$ python3 Python 3.8.0a0 (heads/literal_eval-exception:ddcb2eb331, Feb 21 2018, 04:32:23) [GCC 6.3.0 20170516] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def f(): ... x = 1 ... y = 2 ... return x + y ... >>> f() 3 >>> ======= Python and Pike have fairly simple rules. In Python, a statement ends at the newline, and a function's body must be indented. That means I have to hit Tab at the beginning of each line until I'm done. (AIUI other Python front ends do things differently, but I don't have ipython handy to test.) Pike defines things by semicolons, so I have to toss a semicolon after each statement, even in the REPL (it's annoying to have to write "f();" not just "f()"), but the UI doesn't need indentation, and it just modifies the prompt to indicate nesting level (which is controlled by braces). JavaScript, on the other hand, has far more complex rules. The benefit is that, like with Python, semicolons aren't required, and like with Pike, indentation can be skipped when you're typing at the REPL. The downside is that... well, let's try breaking one of those lines. First, in Pike: rosuav at sikorsky:~$ pike Pike v8.1 release 11 running Hilfe v3.5 (Incremental Pike Frontend) Ok. > int f() { >> int x = 1; >> int y = 2; >> return >> x + y; >> } > > f() >> ; (1) Result: 3 > Exactly the same. The semicolons define it. A line break after "f()" requires the semi before the code runs. The line break after "return" doesn't change anything. rosuav at sikorsky:~$ node > function f() { ... let x = 1; ... let y = 2; ... return ... x + y; ... } undefined > f(); undefined > Strange... even though I have all the semicolons, it's still borked. Because JS has complicated rules about implicit semicolons. Yes, this is an edge case (the "return" statement takes an optional expression, so it's legal on its own - it'd be different if I broke the line before or after an equals sign, for instance, as the two lines wouldn't be legal on their own), but you get more edge cases when you create more magic. Ruby appears to have created a corner case here. It's an extremely weird one, and it's a consequence of the fact that (a) you can call a function just by naming it OR by naming it and giving a parameter, and (b) the plus operator has both unary and binary forms. But you can't change the plus operator without breaking a LOT of people's expectations (and even if you decide to disallow unary plus, the exact same problem will happen with unary minus, and you can't get rid of that one), and the ability for functions to accept optional arguments is an excellent feature. So the only way to solve this is to remove the ability for functions to be called without parentheses. Question: How do you get a reference to a Ruby function? Or are they not first-class objects? ChrisA From jcasale at activenetwerx.com Mon Mar 26 20:41:09 2018 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Tue, 27 Mar 2018 00:41:09 +0000 Subject: issues when buidling python3.* on centos 7 In-Reply-To: References: Message-ID: <3f71e946f26f4354ac91a804925a118d@activenetwerx.com> -----Original Message----- From: Python-list On Behalf Of joseph pareti Sent: Sunday, March 25, 2018 10:15 AM To: python-list at python.org Subject: issues when buidling python3.* on centos 7 > The following may give a clue because of inconsistent python versions: > > [joepareti54 at xxx ~]$ python -V > Python 3.5.2 :: Anaconda 4.3.0 (64-bit) You replaced your system Python with 3.5.2? Redhat (CentOS) wont work without its shipped Python 2.7.5 as the only `python`. Things like SCL's and IUS use alternative and safe ways to bring in additional versions without breaking the system version. I have never used your new version but it looks like it was not built (or installed) correctly for the platform. From rosuav at gmail.com Mon Mar 26 20:45:26 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 27 Mar 2018 11:45:26 +1100 Subject: Invoke an independent, non-blocking process from Python 3.6? In-Reply-To: <1545856e-ecc8-4502-b9ef-a86bc7aefea7@googlegroups.com> References: <1545856e-ecc8-4502-b9ef-a86bc7aefea7@googlegroups.com> Message-ID: On Tue, Mar 27, 2018 at 11:18 AM, wrote: > I have used multiprocessing before when I wrote some parallelized code. That program required significant communication between processes, and it's overkill for my purpose here. I don't need communication between the spawning (live data) program and the spawned program. In fact, to the extent that the live data program has to pay attention to anything besides the data stream, I think it could be bad. > > I have been investigating the subprocess module. I'm looking for something which behaves like subprocess.run("python3 my_program.py"), but which does not "Wait for command to complete, then return a CompletedProcess instance." > Yeah, that sounds like the right solution. So you want a "fire and forget" system. First, a caveat: you MAY have problems if the parent process ends before the child does. (They can be solved but I don't know whether naive use of subprocess will do that.) As far as I know, subprocess.run() will always wait for the process to complete. But you can use the Popen constructor. >>> subprocess.run(["/bin/bash", "-c", "sleep 2; echo bash"]); print("Python")bash CompletedProcess(args=['/bin/bash', '-c', 'sleep 2; echo bash'], returncode=0) Python >>> subprocess.Popen(["/bin/bash", "-c", "sleep 2; echo bash"]); print("Python") Python >>> bash You get back a subprocess.Popen object immediately. What I did here left its standard streams connected to my parent process's streams, so if the child process tries to print to stdout, it appears in the same console. If you set those to be pipes or anything, you have to worry about them filling up; otherwise, this should work for you. BTW, if you need to run something using the same Python interpreter that started you, "sys.executable" may help. ChrisA From rantingrickjohnson at gmail.com Mon Mar 26 21:42:22 2018 From: rantingrickjohnson at gmail.com (Rick Johnson) Date: Mon, 26 Mar 2018 18:42:22 -0700 (PDT) Subject: Ruby parens-free function calls [was Re: Accessing parent objects] In-Reply-To: References: <0c6b63da-fd8f-46fb-a91c-663f86251420@googlegroups.com> <20180326200820.GZ10273@bladeshadow.org> <20180326231004.GA10273@bladeshadow.org> Message-ID: On Monday, March 26, 2018 at 6:11:31 PM UTC-5, Python wrote: > On Mon, Mar 26, 2018 at 02:19:12PM -0700, Rick Johnson wrote: [...] > > Hmm. If "syntax parser rules" could prevent poorly > > formatted code, then there'd be no need for style guides. > > It may be telling that my team has minimal style standards > for C/C++, and none at all for Python... [We don't feel it > requires any.] Hmm, which would indicate to me you're part of a small, tight-nit group of like-minded folks. However, what works well for small groups begins to fall apart in larger social environment. > > > > No self-respecting professional programmer would ever > > > > write in such a manner. > > I should also have said that I think this assumes a certain > level of expertise with the language. We all start out as > novices with every new language, and may spend a very long > time there, depending on how much we use it... Hmm, very true. If only Steven had been more experienced with Ruby, then he would have known to avoid that pitfall. O:-) > > Perl is a special case (and thus not a good retort) > > because the language itself was purposely intended to be > > cryptic. > > I think Larry would disagree with that assessment... And that's an opinion he gets to have ;-) [...] > > IOWs: It's not a bug, dude, it's a feature. > > I dunno, this smells a lot like BS. > > Ruby touts itself as being a simple language with elegant > syntax. This thread is my only exposure to it to date, but > what I've seen here is, frankly, the exact opposite of > that. And do you normally judge the worthiness of an entire language based on one poorly styled non-idiomatic example created by somone who is (admittingly) not intimately familiar with the language? > You should not need a map to distinguish function > calls from variables, I agree that the Python mandated empty parens is a nicely intuitive indication of a func-call, and that's why i use the empty parens in all my Ruby code (even though Ruby does not force me to). Hmm, I'm not sure why Matz chose to make the syntax optional. But whatever his reason may have benn, i just hold down shift, and strike two keys, and -- whamo -- my coding style is consistent. > or operands from function arguments, Agreed. The syntax "foo(x, y, z)" nicely encapsulates the arguments whereas "foo x, y, z" does not. > let alone have the latter depend on the position of the > operator token relative to any whitespace between it and > the two tokens on either side of it. That's hard to read, > complicated, inelegant, and just poor syntax. [My opinion, > obviously.] That's not enough exposure to decide that the > language isn't simple and elegant overall, but it certainly > does induce a strong negative prejudice. You shouldn't dismiss Ruby over this one little issue. Need i remind you that Python has its fair share of warts? In fact, here is one involving Python's implicit self which occasionally bites even the seasoned pythonista: >>> class Foo(object): ... def bar(self, arg1, arg2, arg3): ... pass >>> foo = Foo() >>> foo.bar(1,2) Traceback (most recent call last): File "", line 1, in foo.bar(1,2) TypeError: bar() takes exactly 4 arguments (3 given) (A reinactment of programmer mental state follows) "Huh?" o_O "I gave _three_ arguments, you say?" o.O "Wuh-wuh-what?" O_O "_THREE_?" #_O "What the hell are your talking about Python?" ?_? "Are you blind?" (??????? ??? "Cause I'm _looking_ at the freaking code..." ?(???) "And i'm _looking_ at the freaking exception..." (???)? "but what you're telling me just don't add up!" \(`??) "YOU LIAR!!!" ?????(`??)?? ??? "Okay. Alright. Listen. I _swears_..." ?(????) "I only gave you *TWO* arguments -- just two!" (????)? "WOE IS ME!" ????? "I'm so terribly vexed!" (>_<) "What have i done to deserve this?" ? ?,_?`) From torriem at gmail.com Mon Mar 26 22:07:53 2018 From: torriem at gmail.com (Michael Torrie) Date: Mon, 26 Mar 2018 20:07:53 -0600 Subject: issues when buidling python3.* on centos 7 In-Reply-To: References: Message-ID: <72fc23f6-68dc-7607-829e-9f5c378bf178@gmail.com> On 03/25/2018 10:15 AM, joseph pareti wrote: > The following may give a clue because of inconsistent python versions: > > [joepareti54 at xxx ~]$ python -V > Python 3.5.2 :: Anaconda 4.3.0 (64-bit) What does 'which python' return? As Joseph said, hopefully you didn't overwrite /usr/bin/python with Python 3.5. If you did, you're hosed. You'll probably have to reinstall. Python 2.7 is required by just about everything on CentOS 7. If you want to use Python 3.4, install it via Software Collections Library. https://www.softwarecollections.org/en/ To use SCL, you'll need a working yum, though, so after you reinstall give it a try. From jladasky at itu.edu Mon Mar 26 23:32:06 2018 From: jladasky at itu.edu (jladasky at itu.edu) Date: Mon, 26 Mar 2018 20:32:06 -0700 (PDT) Subject: Invoke an independent, non-blocking process from Python 3.6? In-Reply-To: References: <1545856e-ecc8-4502-b9ef-a86bc7aefea7@googlegroups.com> Message-ID: <3b603a7f-6872-4a20-af96-df0f41099ee4@googlegroups.com> On Monday, March 26, 2018 at 5:45:40 PM UTC-7, Chris Angelico wrote: > On Tue, Mar 27, 2018 at 11:18 AM, wrote: > > I have used multiprocessing before when I wrote some parallelized code. That program required significant communication between processes, and it's overkill for my purpose here. I don't need communication between the spawning (live data) program and the spawned program. In fact, to the extent that the live data program has to pay attention to anything besides the data stream, I think it could be bad. > > > > I have been investigating the subprocess module. I'm looking for something which behaves like subprocess.run("python3 my_program.py"), but which does not "Wait for command to complete, then return a CompletedProcess instance." > > > As far as I know, subprocess.run() will always wait for the process to > complete. But you can use the Popen constructor. Thank you Chris, subprocess.Popen worked nicely for me. I had to set shell=True to make it work, but it did work. All parts of my program now operate independently and crash-free! From Richard at Damon-Family.org Mon Mar 26 23:49:07 2018 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 26 Mar 2018 23:49:07 -0400 Subject: Entering a very large number In-Reply-To: <3Z5uC.351501$Jj7.332942@fx45.am4> References: <5ab4f023$0$12260$426a74cc@news.free.fr> <5ab4fd3f$0$9289$426a34cc@news.free.fr> <20180324230556.klcvdfdzcotlztav@hjp.at> <_Z3uC.218815$TS3.193851@fx18.am4> <3Z5uC.351501$Jj7.332942@fx45.am4> Message-ID: On 3/26/18 8:46 AM, bartc wrote: > On 26/03/2018 13:30, Richard Damon wrote: >> On 3/26/18 6:31 AM, bartc wrote: > >>> The purpose was to establish how such int("...") conversions compare >>> in overheads with actual arithmetic with the resulting numbers. >>> >> Of course if this was done in C with a version that had builtin >> bignum ints or an aggresive enough optimizer (or a Python that did a >> similar level of optimizations) this function would just test the >> speed of starting the program, as it actually does nothing and can be >> optimized away. > > Which is a nuisance. /You/ are trying to measure how long it takes to > perform a task, the compiler is demonstrating how long it takes to > /not/ perform it! So it can be very unhelpful. > > Hence my testing with CPython 3.6, rather than on something like PyPy > which can give results that are meaningless. Because, for example, > real code doesn't repeatedly execute the same pointless fragment > millions of times. But a real context is too complicated to set up. The bigger issue is that these sort of micro-measurements aren't actually that good at measuring real quantitative performance costs. They can often give qualitative indications, but the way modern computers work, processing environment is extremely important in performance, so these sorts of isolated measure can often be misleading. The problem is that if you measure operation a, and then measure operation b, if you think that doing a then b in the loop that you will get a time of a+b, you will quite often be significantly wrong, as cache performance can drastically affect things. Thus you really need to do performance testing as part of a practical sized exercise, not a micro one, in order to get a real measurement. > > ?Yes, something like this can beused to measure the base time to do >> something, but the real question should be is that time significant >> compared to the other things that the program is doing, Making a 200x >> improvement on code that takes 1% of the execution time saves you >> 0.995%, not normally worth it unless your program is currently >> running at 100.004% of the allowed (or acceptable) timing, if >> acceptable timing can even be defined that precisely. > > I'm usually concerned with optimisation in a more general sense than a > specific program. > > Such a with a library function (where you don't know how it's going to > be used); or with a particular byte-code in an interpreter (you don't > know how often it will be encountered); or a generated code sequence > in a compiler. > > But even 200x improvement on something that takes 1% of the time can > be worthwhile if it is just one of dozens of such improvements. > Sometimes these small, incremental changes in performance can add up. > > And even if it was just 1%, the aggregate savings across one million > users of the program can be substantial, even if the individuals won't > appreciate it. 1% extra battery life might be a handy five minutes for > example. > Yes, but if you find where you are really spending your time, a similar effort may give significantly larger improvements. -- Richard Damon From dieter at handshake.de Tue Mar 27 01:42:28 2018 From: dieter at handshake.de (dieter) Date: Tue, 27 Mar 2018 07:42:28 +0200 Subject: A question related to the PYTHONPATH References: <87d0zrtli7.fsf@handshake.de> <9a12013e-7e3e-4ffb-ae04-faeba8e9bfe0@googlegroups.com> Message-ID: <87d0zqf517.fsf@handshake.de> oyono writes: > ... > I was thinking, maybe it could have been done this way to enforce not running module files that are supposed to be bundled into packages as "independant" python scripts...Therefore, running "python script.py" should be reserved to effectively independant python scripts that do not import from a user-created package or module. When you run "python