Mailman 3 July 2018 - Python-ideas

Re: [Python-ideas] Add the imath module
by Jeroen Demeyer 13 Jul '18

13 Jul '18

On 2018-07-13 10:43, Jacco van Dorp wrote: > If there's going to be failure possible primality tests, there should > also exist certain ones. No matter how slow it can be to do it. A certain primality test would be too specialized for the Python stdlib. If you really need that (which you typically don't), there are existing number theory packages.

2 1

Re: [Python-ideas] Add the imath module
by Jeroen Demeyer 12 Jul '18

12 Jul '18

If you want inspiration: https://github.com/sagemath/sage/blob/master/src/sage/arith/misc.py

1 0

anyone need a frozenset or bytearray literal?
by Gregory P. Smith 12 Jul '18

12 Jul '18

Completely obvious what it does, but it irritates my aesthetic sensibilities every time I see: frozenset({spam, eggs}) Why? Because I assume under the hood that creates a set of spam and eggs before calling frozenset to copy it into a new frozenset object before the original set is garbage collected. Wasteful. This is in fact what happens in CPython 3.7 today. I'd love to avoid this. I have no rational measured reason to believe it even matters (thus seeding this on python-ideas and not elsewhere), even though it would technically speed up frozenset creation. (a) detecting frozenset({}) as syntax to encode a frozenset in the python bytecode would be somewhat magical. it could break the person unfortunate enough to monkeypatch out the frozenset builtin (really? don't do that!). (b) abusing the concept of letter prefixes as we already have for strings on {} syntax would be possible but not at all obvious to a reader: f{} or c{} or r{} perhaps. but then someone would want a frozendict. (c) adding a .freeze() method to sets which would raise an exception if the set's refcount were > 1 and would mutate the type of the set object into a frozenset object in place. refcount assertions are bad, not all VMs need refcounts. The concept of a method that can mutate the type of the underlying object in place is... unpythonic. even though technically possible to implement within CPython. I'm -1 on all of my ideas above. But figured I'd toss them out there as food for thought for the concept. We lived for years without even having a set literal in the first place. So this isn't a big deal. frozenset is not the only base type that lacks a literals leading to loading values into these types involving creation of an intermediate throwaway object: bytearray. bytearray(b'short lived bytes object') I was going to suggest complex was in this boat as well, but upon investigation we appear to do constant folding (or amazing parsingon that so 0.3+0.6j does turn into a single LOAD_CONST instead of two consts and an addition. Nice! Not that I expect practical code to use complex numbers. -gps

8 11

Add new `Symbol` type
by Flavio Curella 10 Jul '18

10 Jul '18

More than once I've found myself wanting to create a 'sentinel' value. The most common use case is to differentiate between an argument that has not been provided, and an argument provided with the value `None`. This would be solvable by implementing something similar to what JavaScript calls [`Symbol`]( https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… ). This could be implemented as a 3rd-party library, but there won't be a way to have ['Global' Symbols]( https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Ob… ) Furthermore, without a common implementation in the std library, various Python libraries had to write their own implementations, which all differs in functionality and behavior. Is this something that the Python community is interested in? I'm willing to write the PEP

11 12

Re: [Python-ideas] Calling python from C completely statically
by Barry 10 Jul '18

10 Jul '18

> On 10 Jul 2018, at 19:12, Alberto Garcia <agarciaillera(a)gmail.com> wrote: > > Thank you Barry, > > but I see that the the way yo do it is dropping files to disk. I want to prevent any file to touch the disk except for the actual binary that has everything embedded in it. What file are you refering to? The zip is built onto the elf/exe. BTW this is all off topic for python ideas. You might want to take this offline. Barry > > Cheers > >> On Tue, Jul 10, 2018 at 10:25 AM Barry <barry(a)barrys-emacs.org> wrote: >> >> >>> On 9 Jul 2018, at 21:11, Alberto Garcia <agarciaillera(a)gmail.com> wrote: >>> >>> Hi, >>> >>> thank you for your response. I've downloaded the sources but I couldn't find any documentation. In addition I see that there is not a single C/C++ file. What I want to do is calling python from C. >>> Am I missing something? >> >> You need to grab the sources with an svn checkout. I guess you grabbed the binary kit. >> >> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Instal… >> >> Docs are in here. >> >> https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Instal… >> >> Barry >> >>> Cheers >>> >>> >>>> On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry(a)barrys-emacs.org> wrote: >>>> I think you might find Gordon McMillian's installer interesting to look at. It has a lot >>>> f the tech that I think you are looking for. >>>> >>>> Works up to python 2.7. I ended up taking over when Gordon stopped maintaining >>>> it and kept it going up to python 2.7. In principle the same ideas could be made to >>>> work in python 3 I believe. >>>> >>>> https://sourceforge.net/projects/meinc-installer/ >>>> >>>> The zip file is appended to the end of the .EXE or unix ELF fie. >>>> The boot strap knows how to import form the ZIP at the end of >>>> the binary. >>>> >>>> It also has a way to split out the .SO/.DLL files from the ZIP and >>>> allow them to be loaded. Single EXE mode. >>>> >>>> There are docs that explain how it works in the sources. >>>> >>>> Barry >>>> >>>> >>>> >>>>> On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera(a)gmail.com> wrote: >>>>> >>>>> Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory >>>>> >>>>> >>>>> >>>>>> On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera(a)gmail.com> wrote: >>>>>> Ohhhhhhhh I guess you mean this: >>>>>> https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Comm… >>>>>> >>>>>> Right? >>>>>> >>>>>>> On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera(a)gmail.com> wrote: >>>>>>> Thank you for your response, >>>>>>> >>>>>>> I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. >>>>>>> I guess that the zip file should have no compression at all right? >>>>>>> >>>>>>> When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that? >>>>>>> >>>>>>> And why changing sys.path again to the executable again? Which part of the executable? >>>>>>> >>>>>>> I'll put my efforts in this. >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>>> On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan(a)gmail.com> wrote: >>>>>>>> On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera(a)gmail.com> wrote: >>>>>>>> > Hey there, >>>>>>>> > >>>>>>>> > Yes, the part of having the pyd modules built in in library is already done. >>>>>>>> > I followed the instructions in the README. What I would like to know now is >>>>>>>> > how to embed the non frozen python (py) modules. Can you guys please point >>>>>>>> > me in the right direction. >>>>>>>> >>>>>>>> The gist is to: >>>>>>>> >>>>>>>> 1. take the entire Lib directory and put it in a zip archive >>>>>>>> 2. use the approach demonstrated in cx_freeze to point sys.path in >>>>>>>> your static executable at that zip archive >>>>>>>> 3. adjust your C code to point sys.path back at the executable itself, >>>>>>>> and then combine your executable and the zip archive into a single >>>>>>>> contiguous file (similar to what zipapp does with it's helper script >>>>>>>> and app archive) >>>>>>>> >>>>>>>> There are likely to still be rough edges when doing that, since this >>>>>>>> isn't a well tested configuration. When all else fails, find the part >>>>>>>> of the source code responsible for any error messages you're seeing, >>>>>>>> and try to work out if there's a setting you can tweak to avoid >>>>>>>> hitting that code path. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Nick. >>>>>>>> >>>>>>>> -- >>>>>>>> Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia >>>>>>> -- >>>>>>> Alberto García Illera >>>>>>> >>>>>>> GPG Public Key: https://goo.gl/twKUUv >>>>>> -- >>>>>> Alberto García Illera >>>>>> >>>>>> GPG Public Key: https://goo.gl/twKUUv >>>>> >>>>> >>>>> -- >>>>> Alberto García Illera >>>>> >>>>> GPG Public Key: https://goo.gl/twKUUv >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas(a)python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> >>> -- >>> Alberto García Illera >>> >>> GPG Public Key: https://goo.gl/twKUUv > > > -- > Alberto García Illera > > GPG Public Key: https://goo.gl/twKUUv

1 0

Calling python from C completely statically
by Alberto Garcia 10 Jul '18

10 Jul '18

Hi, I've been working for a while on having the entire python interpreter with all his modules statically linked in a binary that can be called with arbitrary argument which will be passed to the python interpreter. I've been able to statically compile Python in a single binary with no dependencies for Python2.7 and using using this code to call the interpreter with arbitrary code: #define Py_NO_ENABLE_SHARED #include <Python.h> int main(int argc, char** argv) { Py_NoSiteFlag = 1; Py_InitializeEx(0); PyRun_SimpleString(argv[1]); Py_Finalize(); } This program compiles and does not rely in any dependency but the "non frozen" (.py) modules are not loaded so when I do a simple `import random` I get: Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named random I've created a bug issue (https://bugs.python.org/issue34057) speaking about it and Nick Colghlan pointed: " cx_freeze is an illustrative example to look at in that regard, as it preconfigures the interpreter to be able to find the cx_freeze generated zip archive that has the program's Python modules in it: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Comm… The technique that cx_freeze doesn't use yet is to combine the statically linked Python binary and the generated zip archive into a single file (similar to what zipapp does), and adjust the sys.path definition inside the binary to refer back to the executable itself (since executable files can have arbitrary content appended, while zip files can have arbitrary content *pre*pended). " As I understand, cx_freeze creates a zip file with the dependencies for a specific python code and creates a zip with them. My question is, could I create a zip file with every standard module (plus some extra that I may install with pip) and using that zip file from C? If so, how can I do that? I'm happy to work on this if I get indications. BTW: I do not want to convert a python code snippet to exe or anything like that but calling python from C in an absolutely standalone version. Thank you -- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv

5 10

Add `rc` to distutils.version.StrictVersion
by Pål Grønås Drange 06 Jul '18

06 Jul '18

StrictVersion from distutils accepts version tags like 1.14.0 1.14.0a1 1.14.0b2 but not 1.14.0rc1 (nor 1.14.0c1). My suggestion: Add `rc` in the regexp and make it a `prerelease` (the latter comes for free by the current implementation). Most package maintainers have adopted the `rc` abbreviation for release candidate versioning, e.g. - numpy 1.14.0rc1 - scipy 1.1.0rc1 - plotly 3.0.0rc1 - pandas 0.23.0rc1 - matplotlib 2.2.0rc1 - dask 0.13.0rc1 - django 1.9rc1. All of these are available on PyPI. A natural way of sorting version numbers from pip is by simply using sorted(versions, key=distutils.version.StrictVersion), however, due to StrictVersion only accepting `a` and `b` as abbreviations, this does not work for the aforemention packages. The very obvious cons are: - touching 20 years old code [1] - StrictVersion is preserved "for anal retentives and software idealists", and I don't know if they agree. There might be more cons I fail to think of at this moment. [1] https://github.com/python/cpython/blob/master/Lib/distutils/version.py#L130 Pål Grønås Drange

2 1

Re: [Python-ideas] Add new `Symbol` type
by Flavio Curella 06 Jul '18

06 Jul '18

> What functionality does such a thing actually need? I think the requirements should be: * The resulting symbol behave exactly like None. IE: the symbol should not be an instance of object, but an instance of its own class * A symbol can optionally be globally unique. * Two symbols created by the same key must not be equal. IE: they have equal key, but different value * if we're trying to create global symbols with the same key, an exception is thrown This is mostly based on the Javascript spec.

3 2

Add a __cite__ method for scientific packages
by Andrei Kucharavy 04 Jul '18

04 Jul '18

Over the last 10 years, Python has slowly inched towards becoming the most popular scientific computing language, beating or seriously challenging Matlab, R, Mathematica and many specialized languages (S, SAS, ...) in numerous applications. A large part of this growth is driven by amazing community packages, such as numpy, scipy, scikits-learn, scikits-image, seaborn or pandas, just to name a few. Development of such packages represents a significant time investment by people working in academic environments. To be able to justify the investment of time into such package development and support, the developers usually associated them with a scientific article. The number of citations of those articles are considered as measures of the usefulness of articles and are required to justify the time spent on them. Unfortunately, as of now, a significant issue is that such packages are not cited despite being extensively used. Part of this is due to the difficulties with compiling the list of proper citations for each module (and, for libraries associated with multiple update publications, selecting the relevant citation). Part of this is due to users not realizing which of the modules they are using have associated publications and should be cited. To remediate to that situation, I suggest a __citation__ method associated to each package installation and import. Called from the __main__, __citation__() would scan __citation__ of all imported packages and return the list of all relevant top-level citations associated to the packages. As a scientific package developer working in academia, the problem is quite serious, and the solution seems relatively straightforward. What does Python core team think about addition and long-term maintenance of such a feature to the import and setup mechanisms? What do other users and scientific package developers think of such a mechanism for citations retrieval? Best, *Andrei Kucharavy*Post-Doc @ *Joel S. Bader* * Lab*Johns Hopkins University, Baltimore, USA.

18 28

Fwd: grouping / dict of lists
by Chris Barker 01 Jul '18

01 Jul '18

Ivan, Did you mean this to go to the list? I hope so, as I've cc-d it this time :-) On Sun, Jul 1, 2018 at 1:20 AM, Ivan Levkivskyi <levkivskyi(a)gmail.com> wrote: > On 1 July 2018 at 06:18, Chris Barker via Python-ideas < > python-ideas(a)python.org> wrote: > >> I'm really warming to the: >> >> Alternate: collections.Grouping >> >> version -- I really like this as a kind of custom mapping, rather than >> "just a function" (or alternate constructor) -- >> > I wanted the group to be represented as a set, not a list. I however > understand that list may be more common. Can we design an API that > would make this configurable? Something like: > > from collections import Grouping > > deps = Grouping(set) # list can be the default > deps.update(other_deps) # uses set.update or list.extend for every key > deps.add(trigger, target) # uses set.add or list.append > yeah, I thought about that too -- Michael was using set() in some of his examples. But the question is -- do we have a single switchable version or just too classes? > Probably allowing an arbitrary collection for values is to general/hard. > maybe not -- if we had the criteria that you pass in any collection you wanted, as long as it had either an .append() or .add()method, it would be pretty easy to do with duck typing magic. Sure -- a user could make a mess easily enough by passing in a weird custom class, but so what? Using something other than a set or list would be a "at your own risk" thing anyway. > Maybe we can just add a flag `unique=True` to the constructor, that will > cause using sets instead of lists for groups? > That's another, more robust, but less flexible option. Stay tuned for a prototype, if I can get it done fast.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 5