Mailman 3 November 2014 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

Extending traceback to (optionally) format and show locals.
by Robert Collins 18 Jan '15

18 Jan '15

For context please see http://bugs.python.org/issue22937 and http://bugs.python.org/issue22936. I have two questions I'm hoping to get answered through this thread: - does the change in question need a PEP? Antoine seemed to think it didn't, as long as it was opt-in (via the unittest CLI etc) - Is my implementation approach sound (for traceback, unittest I think I have covered :))? Implementation wise, I think its useful to work in the current traceback module layout - that is to alter extract_stack to (optionally) include rendered data about locals and then look for that and format it in format_list. I'm sure there is code out there that depends on the quadruple nature of extract_stack though, so I think we need to preserve that. Three strategies occured to me; one is to have parallel functions, one quadruple, one quintuple. A second one is to have the return value of extract_stack be a quintuple when a new keyword parameter include_locals is included. Lastly, and this is my preferred one, if we return a tuple subclass with an attribute containing a dict with the rendered data on the locals; this can be present but None, or even just absent when extract_stack was not asked to include locals. The last option is my preferred one because the other two both imply having a data structure which is likely to break existing code - and while you'd have to opt into having them it seems likely to require a systematic set of upgrades vs having an optional attribute that can be looked up. So - thoughts? -Rob -- Robert Collins <rbtcollins(a)hp.com> Distinguished Technologist HP Converged Cloud

4 7

Asynchronous IO ideas for Python
by Paul Colomiets 06 Dec '14

06 Dec '14

Hi, I've written an article about how I perceive the future of asynchronous I/O in Python. It's not something that should directly be incorporated into python now, but I believe it's useful for python-ideas list. https://medium.com/@paulcolomiets/the-future-of-asynchronous-io-in-python-c… And a place for comments at Hacker News: https://news.ycombinator.com/item?id=8662782 I hope being helpful with this writeup :) -- Paul

7 7

Fast context creation
by Stefano Borini 01 Dec '14

01 Dec '14

I have a situation where, no matter how the routine ends, I need to return the focus to a widget. It would be rather clever to do this with a context, but I would expect something like this to be a possible strategy with contextlib.context(enter=None, exit=lambda *args: my_widget.setFocus()): do what I need to do as far as I know, at the moment it's not possible. Am I right? I think it would be an easy and practical addition to the contextlib module to quickly register two routines for enter and exit.

8 10

improve compatibility
by Liam Marsh 29 Nov '14

29 Nov '14

hello, the problem is that even with extreme precaution, it is impossible to keep ALL modules compatible from a version to another. what I want to ask is this: -some "packs" which can, like py-compile, generate .pyc files, but using "old" versions of the default library, and of __builtins__. -One will go with every minor version and will be optionnal in the installation -any imported py file will be able to choose which version it wants with the "#! py recommended version X.X" or "#! py mandatory version X.X" commentaries at the begining of the file. thank you and have a nice day/evening/night.

3 2

Re: [Python-ideas] improve compatibility
by Liam Marsh 29 Nov '14

29 Nov '14

oops.. I sent an answer to andrew barnert, who have sent the first answer to my question, and not to the list... however , I do have an answer to my question. thank you! 2014-11-29 11:51 GMT+01:00 Andrew Barnert <abarnert(a)yahoo.com>: > On Nov 29, 2014, at 0:46, Liam Marsh <liam.marsh.home(a)gmail.com> wrote: > > hello, > > yes, the bug which gave me this idea is a py3.2 to py3.3 bug with Vpython, > a 3D graphs library. > > > And is the bug a stdlib bug, or a language bug? > > I also thought this will be the reconciliation between py2 and py3. (this > is why I thought the packs will include a version of the stdlibs) > > > The language differences between 2.x and 3.x are huge, and most of the > stdlib differences between 2.6 and 3.1 or 2.7 and 3.2 are related to those > language differences. Porting code to 3.x is primarily about fixing Unicode > stuff, or code that was already deprecated in 2.5 or 2.6 but wasn't broken > until 3.x. Having the 2.7 stdlib in 3.x would be a huge amount of work for > almost no benefit. > > in fact, how do the .pyc work? were them modified by the "language and > implementation changes"? how do them import other modules? > > > .pyc files are just compiled bytecode, with a version-specific header. > If you fixed the 2.7 stdlib to work in 3.x (which, again, would be a huge > amount of work), you could compile it with 3.4. > > But you're missing the fact that large chunks of the stdlib are written in > C, and compiler against the Python C API. And parts of the stdlib > (especially builtins and the sys module) are exposing types and functions > written in C that are part of the core implementation, so the 2.x version > of the sys module wouldn't be compatible with 2.x code anyway. > > Also, did you mean to write just to me instead of to the list? > > > thank you! > > > > 2014-11-29 6:07 GMT+01:00 Andrew Barnert <abarnert(a)yahoo.com>: > >> On Nov 28, 2014, at 8:29, Liam Marsh <liam.marsh.home(a)gmail.com> wrote: >> >> > hello, >> > the problem is that even with extreme precaution, it is impossible to >> keep ALL modules compatible from a version to another. >> >> Do you have a specific library or app that you've had a problem with? >> There were a handful of modules that had a problem with the 3.2 to 3.3 >> conversion, but every one I saw was caused by language and implementation >> changes, not stdlib changes. I don't think I've seen anything that works >> with 3.3 but not 3.4. I'm sure it's not impossible for such a thing to >> happen, but it would be helpful to have at least one real-life example. >> >> > what I want to ask is this: >> > -some "packs" which can, like py-compile, generate .pyc files, but >> using "old" versions of the default library, and of __builtins__. >> >> But how would this work? The same changes that broke a handful of >> third-party modules between 3.2 and 3.3 probably also mean that the 3.2 >> stdlib wouldn't work in 3.3 without minor changes. And as for builtins, >> most of those are exposing internals of the implementation, so trying to >> make the 3.2 builtins work with 3.3 would take a lot more work than just >> building the 3.2 code against 3.3. >> >> > -One will go with every minor version and will be optionnal in the >> installation >> > -any imported py file will be able to choose which version it wants >> with the >> > "#! py recommended version X.X" or >> > "#! py mandatory version X.X" commentaries at the begining of the >> file. >> > >> > thank you and have a nice day/evening/night. >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas(a)python.org >> > https://mail.python.org/mailman/listinfo/python-ideas >> > Code of Conduct: http://python.org/psf/codeofconduct/ >> > >

1 0

Re: [Python-ideas] Asynchronous IO ideas for Python]
by Trent Nelson 27 Nov '14

27 Nov '14

On Thu, Nov 27, 2014 at 09:59:45PM +0000, Charles-François Natali wrote: > 2014-11-27 1:18 GMT+00:00 Trent Nelson <trent(a)snakebite.org>: > > > > Everything else is just normal Python, nothing special -- it just conforms > > to the current constraints of PyParallel. Basically, the > > HttpServer.data_received() method will be invoked from parallel threads, not > > the main interpreter thread. > > So, still no garbage collection from the threads? Correct. Not having garbage collection has surprisingly not gotten in the way so far, so it's not even on the radar anymore. There are other means available for persisting objects past the lifetime of the parallel context, and you could always do an @async.call_from_main_thread if you want to have the main thread's memory allocator (and thus, GC) kick in. At one point, all these tests passed, just to give you an idea of some of the facilities that are available: https://bitbucket.org/tpn/pyparallel/src/89576c868a3f41747d138a473b090e0f2c… (I haven't removed any of those facilities, I just haven't spent any time on them since switching over to the async socket stuff, so I can't comment on their current state.) Trent. ----- End forwarded message -----

1 0

Re: [Python-ideas] Asynchronous IO ideas for Python
by Paul Colomiets 27 Nov '14

27 Nov '14

Hi Trent, On Wed, Nov 26, 2014 at 10:36 PM, Trent Nelson <trent(a)trent.me> wrote: > Have you seen this?: > > https://speakerdeck.com/trent/pyparallel-how-we-removed-the-gil-and-exploit… > > I spend the first 80-ish slides on async I/O. > > (That was a year ago. I've done 2-3 sprints on it since then and have gotten it to a point where I can back up the claims with hard numbers on load testing benchmarks, demonstrated in the most recent video: https://www.youtube.com/watch?v=4L4Ww3ROuro.) > > Thanks. Have never seen it before. Do you have some detailed description about how I/O subsystem works? From what I see, pyparallel relies on offloading I/O operations to the IOCP. And it seems that I/O kernel which I'm describing can replace IOCP on unix. And allowing the C code to do more high-level stuff (like reconnecting and message delimitation), would be of a benefit for pyparallel too, as it would allow less heap allocations, smaller number of heap snasphot, etc. Still PyParallel, is much bigger departure from current Python. As it doesn't allow arbitrary code to run in threads, so it's much longer perspective, than I/O kernel which will be just a library on PyPI. -- Paul

1 0