Mailman 3 January 2020 - Python-ideas

Recommend UTF-8 mode on Windows
by Inada Naoki Jan. 14, 2020

Jan. 14, 2020

Hi, all. I believe UTF-8 should be chosen by default for text encoding. * The default encoding for Python source file is UTF-8. * VS Code and even notepad uses UTF-8 by default. * Text files downloaded from the Internet is probably UTF-8. * UTF-8 is used when you use WSL regardless your system code page. * Windows 10 (1903) adds per-process option to change active code page to UTF-8 and call the system code page "legacy". [1] But it is difficult to change the default text encoding of Python … [View More]

5 11

Enhancing Zipapp
by Abdur-Rahmaan Janhangeer Jan. 13, 2020

Jan. 13, 2020

Note: draft simplified Abstract ====== This extracts aims at proposing enhancements to the generated zipapp executable Rationale ======= One area where there remains some difficulty in Python is packaging for end-user consumption. To that effect either the code is distributed in pure Python form with installers [1] or native executables are built for each target Os [2]. Currently by default, Python does not provide such utilities. This pro- posal aims at finalising a Python-specific archive … [View More]as the default VM exec- utable built on zipapp. In simple terms, it proposes to enhance zipapp from plain archive to app-level archive. Advantages of archives ================== Archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules [3] You can use archives for tasks such as lossless data compression, archiving, decompression, and archive unpacking. [4] Adding capabilities like digital signing is used to verify integrity and authenticity. Zip archives as apps ================ If we are to treat zip archives as app, here are some recommended features - [x] Main entry point A main entry point specifies which file to launch. Zipapp already solves this problem by either having a __main__.py [5] or specifying the entry point at the commandline ENTRYPOINT_MODULE:ENTRYPOINT_FUNCTION [6] - [ ] App info file An info file can have info such as author name, archiving date, company name etc. - [ ] Signing mechanism Mechanisms can be added to detect the integrity of the app. App hash can be used to check if the app has been modified and per-file hash can be used to detect what part has been modified. This can be further enhanced if needed. - [ ] Protecting meta data Metadata are not protected by basic signing. There existing ways to protect metadata and beyond [7] - [x] Pure-Python 3rd party package bundling In Python, as long as the 3rd party packages are pure-python packages, we can bundle and use them [6]. The user can maybe just include a requirements.txt - [ ] C-based 3rd party packages Zipapp by default was not meant to include packages at all. << The executable zip format is specifically designed for standalone use, without needing to be installed. They are in effect a multi-file version of a standalone Python script >> Though the previous point shows that this can be done. Now remains the issue of C-based packages. Distributing wheels might be the answer [8]. A zip archive is supposed to be standalone. A possible solution might be to include wheels and the wheels are installed in a site-packages folder. When running such an app, the interpreter will check first if the app-specific site-packages folder is empty, if not, install the wheels. This provides package- freezing ability. The only downside is longer first-run time. Only specifying packages to be installed is not an option as if you really want stand-alone apps, using the internet etc defeats the purpose. FAQ ==== Why not a package manager? --------------------------------------- The zipapp pep was introduced for a reason, for easing the running of archives. Maybe the package manager idea came from listening to people talking about packaging and pex and comparing it with package-managers like homebrew and concluded that pex and hence zipapp is not worth it and people would better off not complicate their lives with some zip utility. This proposal is not solving any problem at all -------------------------------------------------------------- This proposal aims at enhancing zipapp. Zipapp solved the problem. Zipapp had an aim. This proposal aims at helping zipapp better accompplish it's aim. References [1] https://pynsist.readthedocs.io/en/latest/ [2] https://www.pyinstaller.org [3] https://www.python.org/dev/peps/pep-0441/ [4] https://docs.oracle.com/javase/tutorial/deployment/jar/basicsindex.html [5] https://docs.python.org/3/library/zipapp.html [6] https://gist.github.com/lukassup/cf289fdd39124d5394513a169206631c [7] https://source.android.com/security/apksigning [8] https://pythonwheels.com Yours, Abdur-Rahmaan Janhangeer pythonmembers.club <http://www.pythonmembers.club/> | github <https://github.com/Abdur-rahmaanJ> Mauritius [View Less]

10 36

python -m quality of life improvements
by Soni L. Jan. 12, 2020

Jan. 12, 2020

currently python -m requires you to cwd to the desired package root. I'd like to suggest the ability to python -m relative/path/to/package/root/module.submodule and python -m /absolute/path/to/package/root/module.submodule thoughts?

7 28

Re: python -m anomaly Was: quality of life improvements
by Steve Barnes Jan. 11, 2020

Jan. 11, 2020

-----Original Message----- From: Steven D'Aprano <steve(a)pearwood.info> Sent: 11 January 2020 04:39 To: python-ideas(a)python.org Subject: [Python-ideas] Re: python -m quality of life improvements On Fri, Jan 10, 2020 at 11:53:10PM -0300, Soni L. wrote: > currently python -m requires you to cwd to the desired package root. > I'd like to suggest the ability to python -m > relative/path/to/package/root/module.submodule and python -m > /absolute/path/to/package/root/module.… [View More]submodule Oh, a further thought comes to mind... if your modules aren't on the PYTHONPATH, just drop the -m and you can execute any legal Python file, even if the name isn't a legal identifier. [steve@ando tmp]$ cp ~/python/spam/eggs.py /tmp/"some name".foo [steve@ando tmp]$ python3.5 "some name.foo" some name.foo So as I understand it, the functionality you want already exists, both for running scripts on the path using -m and scripts with arbitrary file names without -m. Have I missed some subtlety in your proposal? [Steve Barnes] I did notice one anomaly in the behaviour of python -m: taking your code structure above and using : > py -mzipfile -c eggs.zip spam\__init__.py spam\__main__.py spam\eggs.py to create eggs.zip I can run: > py eggs.zip Renaming eggs.zip to eggs.py I can run: >eggs.py E:\toolbuild\eggs.py\__main__.py Or >py eggs.py E:\toolbuild\eggs.py\__main__.py But get problems when I try: > py -meggs.py Traceback (most recent call last): File "C:\Python38_64\lib\runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "C:\Python38_64\lib\runpy.py", line 109, in _get_module_details __import__(pkg_name) ValueError: source code string cannot contain null bytes Gadget@GADGETLAPTOP E:\toolbuild > py -meggs Traceback (most recent call last): File "C:\Python38_64\lib\runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "C:\Python38_64\lib\runpy.py", line 153, in _get_module_details code = loader.get_code(mod_name) File "<frozen importlib._bootstrap_external>", line 916, in get_code File "<frozen importlib._bootstrap_external>", line 846, in source_to_code File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed ValueError: source code string cannot contain null bytes Is this expected behaviour or worth a ticket and some time investigating further? This was on Windows 10/64 with 3.8-64 but I get exactly the same with 3.7-64/32 & 2.7 and with python rather than py. Gadget/Steve [View Less]

2 1

.zfill() to add trailing zeros if parameter is negative
by Antonio Carlos Jorge Patricio Jan. 10, 2020

Jan. 10, 2020

H e l l o t o e v e r y o n e ! I d l i k e t o s u g g e s t . z f i l l ( ) t o h a v e a n o p t i o n a l p a r a m e t e r ( t h a t d e f a u l t s t o 0 ) s o t h e f i l l i n g c h a r a c t e r c o u l d b e c u s t o m i z e d . A n d s i n c e i t c u r r e n t l y d o e s n o t h i n g i f g i v e n a n e g a t i v e p a r a m e t e r , i t c o u l d m a y b e a d d t r a i l i n g z e r o e s ( o … [View More]

2 1

Add a PyObject_VaCallFunction to C API??
by hrfudan＠gmail.com Jan. 8, 2020

Jan. 8, 2020

The C API has a function PyObject_CallFunction( PyObject*, const char* fmt, ... ). It is a variadic function hence I couldn't pass a va_list to it to invoke the call. My question is, is it technically possible to provide a companion PyObject_VaCallFunction which takes a va_list, just like Py_VaBuildValue is to Py_BuildValue? Here's what I have found so far. I read the Objects/call.c about PyObject_CallFunction there, and actually found a function _PyObject_CallFunctionVa, which is close to … [View More]what I want, although it has two additional parameters. By taking a look at the implementation of PyObject_CallFunction, naively I was thinking a Va version of that could have been similarly implemented ( I'm ignoring the size_t issue ). I'd appreciate it if someone could enlighten me on this subject. Question rephrased: is there a technical reason why there is no PyObject_VaCallFunction in the API? If not, would it be possible to add it in? Motivation: I write scientific simulation code to be run on large clusters. The data generated can be huge. Although I know parallelization using C++, I don't know it using Python. A direct consequence is formidable data processing time with python, which is driving me nuts. I have two options, either parallelize python code or embed the python in C++. I'm favoring the latter, not just because I don't know about python parallelization, but more importantly because by embedding python in C++, I get to keep using the highly specialized C++ classes and some calculation routines, without having to duplicate essentially the same thing in Python, which is very time-saving. I could have just used the C API, but here is the all-time drawback of linking with Python libraries --- it messes up the linker runtime search path. Usually a Python installation has under its lib/ many common libraries such as libz.so, libhdf5.so ( or I probably should have said earlier that I primarily work on Linux ). They have different versions to, say, the libhdf5.so I'm using in the code. The upshot is that by linking my program with both the major library in which HDF5 links to one version, and the Python libraries in which HDF5 links to another, the runtime search path is always mixed up, which is not safe. So what I thought about doing is that I'm gonna create a C++ wrapper on Python, one that doesn't expose the raw Python to the client code at all. ( For example, there is no #include <Python.h> in any header of that library. Plus I could use C++ OOP to automate away keeping tracking of Py_INCREF/Py_DECREF. ) In fact it's very doable. Everything is straightforward, except variadic functions, such as PyObject_CallFunction, Py_BuildValue, and so on. Let me use PyObject_CallFunction to illustrate the problem. Let's say that I want to provide a wrapper to PyObject_CallFunction to the client, like this ( I'm taking the return type to void for simplicity ) void MyCallFunction(PyObject* obj, const char* fmt, ... ); I go ahead and put this line in the header "my_python_wrapper.h". Then in the "my_python_wrapper.c" file I would like to write the following implementation. void MyCallFunction(PyObject* obj, const char* fmt, ... ) { va_list args; va_start(args,fmt); PyObject(obj, fmt, args); // This function doesn't exist in the API ( yet? ) va_end(args); } This way, only the "my_python_wrapper.c" needs to link with Python Libraries. Any user of my_python_wrapper doesn't need to, which seems nice. ( In cmake lingo, I only need to target_link_libraries( my_python_wrapper PRIVATE ${PYTHON_LIBRARIES} ), instead of PUBLIC ) As one can see, the crux is that not all variadic functions in the API has a companion Va-ed version. So far I only found Py_VaBuildValue. I've worked out MyCallFunction() in my actual code in the same manner described above, but with Py_VaBuildValue. What I did was I send all variadic arguments to a MyBuildValue(PyObject*, const char*, ...), the in the .c file, MyBuildValue will generate a va_list and pass it onto Py_VaBuildValue, then I force the outcome to be a tuple and pass it to PyObject_CallObject and it works! Nontheless, this approach seems less straightforward to having a PyObject_VaCallFunction, so I'm guessing it may have performance penalty. I really appreciate it whoever takes their time to read this essay of mine! I apologize if I failed to use the idiomatic mark-up. Any comments, questions on this subject are welcome! [View Less]

10 12

Re: Python, Be Bold!
by James Lu Jan. 6, 2020

Jan. 6, 2020

I use macOS, and using Python is very confusing. - Apple's bundled Python 2.7. - Anaconda (Python scientific stack package manager) Python and conda. - Homebrew (3rd party package manager for macOS) Python and pip. I also believe that there is a PSF Python installer, but I am not sure. Python, Be Bold captures the spirit of it should not be a shame to have the interpreter/VM installed on end-users machines. It also facilitates the work of other python devs. You installed it because of one … [View More]

4 3

timezone-aware datetime.now
by Random832 Jan. 4, 2020

Jan. 4, 2020

Right now there is no simple way to get a timezone-aware time for the current time in the current time zone [without resorting to outside libraries like pytz], I would like to propose a simple extension to datetime.now to allow returning an aware datetime object using a plain single-offset tzinfo object: datetime.now('auto') shall return a datetime using a simple offset-based timezone object. [Incidentally, I also think that utcoffset() on naive timestamps should be able to return the offset … [View More]

1 1

Target-and-expression lists and if-try
by Andrew Barnert Jan. 3, 2020

Jan. 3, 2020

Every so often, someone suggests that Python should have a pattern matching case statement like Scala, C#, Swift, Haskell, etc. Or they just suggest that “Python needs algebraic data types” but end up concluding that the biggest part Python is missing is not the types themselves, but a practical way to do something useful with them, which basically means pattern matching. Anyway, with the addition of the walrus operator and @dataclass, Python is actually a lot closer than it used to be. I … [View More]believe with just two smallish syntax features, it would be possible to do almost anything you’d ever want in a library. The first is to extend unpacking assignment to target-or-expression lists. Like this: x, 0, z = vec Just like `x, y, z = vec`, this raises a TypeError if vec isn’t Iterable, and a ValueError if it has more or fewer than 3 elements, and otherwise, it binds x and z to the first and third elements. But it doesn’t bind anything to the second element; instead, it checks if that element is 0, and, if not, raises a ValueError. The second is an “if try” statement, which tries an expression and runs the body if that doesn’t raise (instead of if it’s truthy), but jumps to the next elif/else/statement (swallowing the exception) if it does. The actual value of the expression is discarded, but the walrus operator takes care of that: if try 0, y, z := vec: # do yz plane stuff elif try x, 0, z := vec: # do xz plane stuff elif try x, y, 0 := vec: # do xy plane stuff elif x, y, z := vec: # do slow/imprecise/whatever 3D stuff else: raise TypeError(f'{vec} is not a 3-vector!') Alternatively, this could just be a try expression that can be used anywhere: it’s truthy if evaluating doesn’t raise, falsey if it does. But I don’t think it’s needed anywhere but if/elif. Anyway, neither of these features seems very useful on its own. (It should be obvious how to rewrite that example as something just as readable that just unpacks and then checks the values with normal if.) But I think the two of them together will allow a pure-library implementation of pattern matching syntax that reads nicely and can do everything most other languages do—in particular, concisely and readably pattern match and deconstruct dataclasses (and other types with an opt-in protocol or registry, but dataclasses would work out of the box the way Scala case classes, Swift structs, etc. do). If people think either of these features is too horrible to contemplate no matter what the potential benefits, I won’t bother working through the details. But if people do want to see the details (including some good motivating examples—I’ll go through tutorials for those other languages to find some that don’t involve recursing into cons lists and other stuff you wouldn’t want to do in Python…), I’ll be happy to do so, and, if it works out, turn this into a more detailed proposal. And obviously, if people want to bikeshed the spelling before even seeing what it’s good for, well, this is Python-ideas. :) [View Less]

9 28

Re: Python, Be Bold!
by James Lu Jan. 3, 2020

Jan. 3, 2020

Ideally, we'd have a functional package manager: one that can delete binaries when disk space is running low, and recompiles from sources when the binary is needed again. It could store Python 2 as a series of diffs against Python 3. On Thu, Jan 2, 2020, 2:22 AM Abdur-Rahmaan Janhangeer <arj.python(a)gmail.com> wrote: > if not self-updating, at least the ability to update > > Yours, > > Abdur-Rahmaan Janhangeer > pythonmembers.club <http://www.pythonmembers.club/… [View More]

4 3