Mailman 3 March 2012 - Python-ideas

Re: [Python-ideas] PEP x: Static module/package inspection
by anatoly techtonik 29 Mar '12

29 Mar '12

(reposting this from Google Group once more as the previous post missed Mailing List, because I was not subscribed in Mailman) *Static module/package inspection* Abstract: - static: without execution (as opposed to dynamic) - module/package: .py or __init__.py file - inspection: get an overview of the contents *What should this do?* * * The proposal to add a mechanism to Python interpreter to get an outline of module/package contents without importing or executing module/package. The outline includes names of classes, functions, variables. It also should contain values for variables that could be provided without sophisticated calculations (e.g. a string, integer, but probably not expressions as it may lead to security leaks). *Why?* * * *user story PEPx.001:* As a Python package maintainer, I find it bothersome to repeatedly write bolierplate code (e.g. setup.py) to package my single file module. The reason I should write setup.py is to provide version and description info. This info is already available in my module source code. So I need to either copy/paste the info from the module manually, or to import (and hence execute) my module during packaging and installation, which I don't want either, because modules are often installed with root privileges. With this PEP, packing tool will be able to extract meta information from my module without executing it or without me manually copying version fields into some 'package configuration file'. *user story PEPx.002:* As a Python Application developer, I find it really complicated to provide plugin extension subsystem for my users. Users need a mechanism to switch between different versions of the plugin, and this mechanism is usually provided by external tool such as setuptools to manage and install multiple versions of plugins in local Python package repository. It is rather hard to create an alternative approach, because you are forced to maintain external meta-data about your plugin modules even in case it is already available inside the module. With this PEP, Python Application will be able to inspect meta-data embedded inside of plugins before choosing which version to load. This will also provide a standard mechanism for applications to check modules returned by packaging tools without executing them. This will greatly simplify writing and debugging custom plugins loaders on different platforms. *Feedback goal* At this stage I'd like to a community response to two separate questions: 1. If everybody feels this functionality will be useful for Python 2. If the solution is technically feasible

4 5

make __closure__ writable
by Yury Selivanov 29 Mar '12

29 Mar '12

Can we make the __closure__ attribute writeable? Since __code__ already is, and it is possible to tackle with the opcodes, having no way of rewriting __closure__ (except creating a completely new function) is annoying. I don't think it will somehow harm python, as those who want to break it can do it already in multiple ways, easier than playing with __closure__. - Yury

6 9

A more useful command-line wsgiref.simple_server?
by Masklinn 28 Mar '12

28 Mar '12

Currently, calling wsgiref.simple_server simply mounts the (bundled) demo app. I think that's a bit of a lost opportunity: as the community has mostly standardized on a *.wsgi/wsgi.py script providing an `application` name in its global namespace, it would be nice if wsgiref.simple_server could take such a file as parameter and mount the application provided: * This would allow testing that the script has no error without having to go through mounting it in e.g. mod_wsgi * It would make trivial/test applications (e.g. dynamic responders to local JS) simpler to bootstrap as there would be no need for the half-dozen lines of wsgiref.simple_server bootstrapping and "hard" dependency on wsgiref, import wsgiref.simple_server def application(environ, start_response): 'code' if __name__ == '__main__': httpd = make_server('', 8000, application) httpd.serve_forever() could become: def application(environ, start_response): 'code' Since wsgiref already supports `python -mwsgiref.simple_server`, the change would be pretty simple: * the first positional argument is the wsgi script if it is present it is `exec`'d, the `application` key is extracted from the locals and is mounted through make_server; if it is absent, then demo_app is mounted as before * the second positional argument is the host, defaulting to '' * the third positional argument is the port, defaulting to 8000 This way the current sanity test/"PHPInfo" demo app works as it did before, but it becomes possible to very easily serve a WSGI script with almost no overhead in the script itself. Thoughts?

3 5

Standard way to get caller name and easy call stack access
by anatoly techtonik 28 Mar '12

28 Mar '12

Seems like the only way to get caller name in Python is to use inspect module [1] It seems that this way is not recommended (I wonder why?) and there is no other. The stackoverflow question about how to get callers method namd [1] got 3k views over 1 year, so the recipe seems to be useful. I've developed a similar snippet [2] that also extracts a module and class name in addition to method. To me this snippet is very handy to insert in print statements to see who uses a certain function at run-time. The output from this function is: spyderlib.widgets.externalshell.introspection.NotificationThread.run Which no function of `inspect` and `traceback` modules can provide (correct me if I wrong, but I am pretty confident). The stuff they propose will look like: File "/mount/sdb5/projects/spyderlib/spyderlib/widgets/externalshell/introspection.py", line 183, in run write_packet(self.notify_socket, output) As a 'software engineer' I am more interested to get logical structure of call stack rather than physical location of code text pieces (which is important too, but secondary). Neither `traceback` nor `inspect` seems to be designed with this use case, and if you'll look closely - the traceback above misses class info completely. To get the name of the class from this traceback you need to know its location withing the lines of the file. So, I'd propose to include caller_name() function somewhere and name it somehow. But there is more to that. Maybe too late for Python 3, but still an idea - make call stack a normal Python feature (optional for stackless, but still a feature) and describe a standard for call stack names, to provide everybody a simple and safe way to see what's going on with execution stack (and an easy way to get caller names). Making it a feature means that there will be almost no overhead when looking up the name (id) of the caller callers. This will also be handy for logging. I wish I could shorten this proposal. The picture in my head is something like: __main__ spyderlib.widgets.externalshell.introspection.NotificationThread.run spyderlib.utils.bsdsocket.write_packet Every indented string is an unique id of function or other closure that makes up Python code. Would it rock? This will probably require describing all possible ways to run Python code to see where a call to caller_name() can appear to cover all possible situations. Is there already a list of these ways? 1. http://stackoverflow.com/questions/2654113/python-how-to-get-the-callers-me… 2. https://gist.github.com/2151727 -- anatoly t.

11 19

Re: [Python-ideas] My objections to implicit package directories
by Guido van Rossum 28 Mar '12

28 Mar '12

On Tue, Mar 27, 2012 at 1:20 PM, Phil Vandry <vandry(a)tzone.org> wrote: > On 2012-03-26 10:57 , Guido van Rossum wrote: >> >> On Mon, Mar 26, 2012 at 1:45 AM, Ronald Oussoren<ronaldoussoren(a)mac.com> >> wrote: >>> >>> Yes. On what platform are you? On unixy platforms filename extensions are >>> just a naming convention that can just as easily be used with directories. >> >> >> IIUC that's how almost all filesystems treat them. However desktop >> software often assigns specific meanings to them -- the user can >> configure these, but there's a large set of predefined bindings too, >> and many key applications also play this game (since there is, >> frankly, not much else to go by -- some important file types are not > > > On the Mac, at least, there is much more to go by: a 4-character file type > and a 4-character creator type associated with every file. The IANA > registration form for MIME types even lets registrants specify the mapping > between these 8 characters and a MIME type. > > However, these do seem to have fallen into disuse in recent versions of > MacOS. I never knew if this was an intentional downgrade or just a lack of > upkeep. Oh, I remember those from hacking MacOS decades ago... I suspect they have fallen by the wayside because only Mac-specific tools keep track of these as files are copied, moved, backed up, archived, restored, uploaded and downloaded, etc. -- --Guido van Rossum (python.org/~guido)

2 1

Add socketserver running property
by Giampaolo Rodolà 27 Mar '12

27 Mar '12

Hi all, I would like to attract some attention to: http://bugs.python.org/issue14375 Would this be acceptable? Regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

3 2

My objections to implicit package directories
by Nick Coghlan 26 Mar '12

26 Mar '12

It seems the consensus at the PyCon US sprints is that implicit package directories are a wonderful idea and we should have more of those. I still disagree (emphatically), but am prepared to go along with it so long as my documented objections are clearly and explicitly addressed in the new combined PEP, and the benefits ascribed to implicit package directories in the new PEP are more compelling than "other languages do it that way, so we should too". To save people having to trawl around various mailing list threads and reading through PEP 395, I'm providing those objections in a consolidated form here. If reading these objections in one place causes people to have second thoughts about the wisdom of implicit package directories, even better. 1. Implicit package directories go against the Zen of Python Getting this one out of the way first. As I see it, implicit package directories violate at least 4 of the design principles in the Zen: - Explicit is better than implicit (my calling them implicit package directories is a deliberate rhetorical ploy to harp on this point, although it's also an accurate name) - If the implementation is hard to explain, it's a bad idea (see the section about backwards compatibility challenges) - Readability counts (see the section about introducing ambiguity into filesystem layouts) - Errors should never pass silently (see the section about implicit relative imports from main) 2. Implicit package directories pose awkward backwards compatibility challenges It concerns me gravely that the consensus proposal MvL posted is *backwards incompatible with Python 3.2*, as it deliberately omits one of the PEP 402 features that provided that backwards compatibility. Specifically, under the consensus, a subdirectory "foo" of a directory on sys.path will shadow a "foo.py" or "foo/__init__.py" that appears later on sys.path. As Python 3.2 would have found that latter module/package correctly, this is an unacceptable breach of the backwards compatibility requirements. PEP 402 at least got this right by always executing the first "foo.py" or "foo/__init__.py" it found, even if another "foo" directory was found earlier in sys.path. We can't just wave that additional complexity away if an implicit package directory proposal is going to remain backwards compatible with current layouts (e.g. if an application's starting directory included a "json" subfolder containing json files rather than Python code, the consensus approach as posted by MvL would render the standard library's json module inaccessible) 3. Implicit package directories introduce ambiguity into filesystem layouts With the current Python package design, there is a clear 1:1 mapping between the filesystem layout and the module hierarchy. For example: parent/ # This directory goes on sys.path project/ # The "project" package __init__.py # Explicit package marker code.py # The "project.code" module tests/ # The "project.tests" package __init__.py # Explicit package marker test_code.py # The "projects.tests.test_code" module Any explicit package directory approach will preserve this 1:1 mapping. For example, under PEP 382: parent/ # This directory goes on sys.path project.pyp/ # The "project" package code.py # The "project.code" module tests.pyp/ # The "project.tests" package test_code.py # The "projects.tests.test_code" module With implicit package directories, you can no longer tell purely from the code structure which directory is meant to be added to sys.path, as there are at least two valid mappings to the Python module hierarchy: parent/ # This directory goes on sys.path project/ # The "project" package code.py # The "project.code" module tests/ # The "project.tests" package test_code.py # The "projects.tests.test_code" module parent/ project/ # This directory goes on sys.path code.py # The "code" module tests/ # The "tests" package test_code.py # The "tests.test_code" module What are implicit package directories buying us in exchange for this inevitable ambiguity? What can we do with them that can't be done with explicit package directories? And no, "Java does it that way" is not a valid argument. 4. Implicit package directories will permanently entrench current newbie-hostile behaviour in __main__ It's a fact of life that Python beginners learn that they can do a quick sanity check on modules they're writing by including an "if __name__ == '__main__':" section at the end and doing one of 3 things: - run "python mymodule.py" - hit F5 (or the relevant hot key) in their IDE - double click the module in their filesystem browser - start the Python REPL and do "import mymodule" However, there are some serious caveats to that as soon as you move the module inside a package: - if you use explicit relative imports, you can import it, but not run it directly using any of the above methods - if you rely on implicit relative imports, the above direct execution methods should work most of the time, but you won't be able to import it - if you use absolute imports for your own package, nothing will work (unless the parent directory for your package is already on sys.path) - if you only use absolute imports for *other* packages, everything should be fine The errors you get in these cases are *horrible*. The interpreter doesn't really know what is going on, so it gives the user bad error messages. In large part, the "Why are my imports broken?" section in PEP 395 exists because I sat down to try to document what does and doesn't work when you attempt to directly execute a module from inside a package directory. In building the list of what would work properly ("python -m" from the parent directory of the package) and what would sometimes break (everything else), I realised that instead of documenting the entire hairy mess, the 1:1 mapping from the filesystem layout to the Python module hierarchy meant we could *just fix it* to not do the wrong thing by default. If implicit package directories are blessed for inclusion in Python 3.3, that opportunity is lost forever - with the loss of the unambiguous 1:1 mapping from the filesystem layout to the module hierarchy, it's no longer possible for the interpreter to figure out the right thing to do without guessing. PJE proposed that newbies be instructed to add the following boilerplate to their modules if they want to use "if __name__ == '__main__':" for sanity checking: import pkgutil pkgutil.script_module(__name__, 'project.code.test_code') This completely defeats the purpose of having explicit relative imports in the language, as it embeds the absolute name of the module inside the module itself. If a package subtree is ever moved or renamed, you will have to manually fix every script_module() invocation in that subtree. Double-keying data like this is just plain bad design. The package structure should be recorded explicitly in exactly one place: the filesystem. PJE has other objections to the PEP 395 proposal, specifically relating to its behaviour on package layouts where the directories added to sys.path contain __init__.py files, such that the developer's intent is not accurately reflected in their filesystem layout. Such layouts are *broken*, and the misbehaviour under PEP 395 won't be any worse than the misbehaviour with the status quo (sys.path[0] is set incorrectly in either case, it will just be fixable under PEP 395 by removing the extraneous __init__.py files). A similar argument applies to cases where a parent package __init__ plays games with sys.path (although the PEP 395 algorithm could likely be refined to better handle that situation). Regardless, if implicit package directories are accepted into Python 3.3 in any form, I *will* be immediately marking PEP 395 as Rejected due to incompatibility with an accepted PEP. I'll then (eventually, once I'm less annoyed about the need to do so) write a new PEP to address a subset of the issues previously covered by PEP 395 that omits any proposals that rely on explicit package directories. Also, I consider it a requirement that any implicit packages PEP include an update to the tutorial to explain to beginners what will and won't work when they attempt to directly execute a module from inside a Python package. After all, such a PEP is closing off any possibility of ever fixing the problem: it should have to deal with the consequences. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

13 26

EBNF-module in stdlib
by Lukas Lueg 23 Mar '12

23 Mar '12

Hi. By all accounts, Python has many strong and flexible ways to deal with parsing of semi-structured data. Applications range from csv- and configuration-files over log- and report-data to all kinds of semi-structured formats. As far as I can see the standard library however either provides highly specialized modules (e.g. configparser, csv, json) or drives you off into very generic approaches (e.g. re) for you to implement the rest or use external modules. I started looking into RPython some time ago and found the included parser-module highly useful. It provides an (again extended) EBNF-parser with capable visitors to get parse-trees out of a simple BN-syntax. Is it worth the discussion to get our own EBNF-module into the standard library, so people who otherwise rely on pure regular expressions can build capable parsers with little effort? External modules (like simpleparser) and even internal modules like the ones mentioned above may even benefit from that. Regards Lukas

3 2

Add "default" keyword to itemgetter and attrgetter
by Miki Tebeka 23 Mar '12

23 Mar '12

Greetings, I apologize if this appears twice, Google groups gave me an error on the first one and I want to make sure this makes it through. Repeating http://bugs.python.org/issue14384 .... This way they will behave more like getattr and the dictionary get. If default is not specified, then if the item/attr not found, an execption will be raised, which is the current behavior. However if default is specified, then return it in case when item/attr not found - default value will be returned. I wanted this when trying to get configuration from a list of objects. I'd like to do get = attrgetter('foo', None) return get(args) or get(config) or get(env) In the case of multiple items/attrs then you return default in their place: attrgetter('x', 'y', default=7)(None) => (7, 7) In case of dotted attribute again it'll return default value of any of the attributes is not found: attrgetter('x.y', default=7)(None) => 7 BTW: This is inspired from Clojure's get-in (http://bit.ly/GGzqjh) function. Thanks, -- Miki

3 2

Save memory when forking with *really* immutable objects
by Larry Hastings 23 Mar '12

23 Mar '12

I'll admit in advance that this is in all likelyhood a terrible idea. What I'm curious about is why it wouldn't work, or if it wouldn't even help ;-) One problem for CPython is that it can't share data across processes very often. If you have an application server, and you fork a hundred processes to handle requests, your memory use will be "C * n * p" where C is a constant, n is the number of processes, and p is the average memory consumption of your app. I fear C is very nearly near 1.0. Most of Python's memory usage is on the heap, and Python uses its memory to store objects, and objects are reference counted, and reference counts change. So all the COW data pages get written to sooner or later. The obvious first step: add a magical reference count number that never changes, called Py_REF_ETERNAL. I added this to CPython trunk with a quick hack. It seemed to work; I threw in some asserts to test it, which passed, and it was passing the unit test suite. I discussed this with Martin who (as usual) made some excellent points. Martin suggests that this wouldn't help unless we could concentrate the Py_REF_ETERNAL objects in their own memory pools in the small block allocator. Otherwise we'd never get a page that didn't get written to sooner or later. Obviously all interned strings could get Py_REF_ETERNAL. A slightly more controversial idea: mark code objects (but only those that get unmarshaled, says Martin!) as Py_REF_ETERNAL too. Yes, you can unload code from sys.modules, but in practice if you ever import something you never throw it away for the life of the process. If we went this route we could probably mark most (all?) immutable objects that get unmarshaled with Py_REF_ETERNAL. Martin's statistics from writing the flexible string representation says that for a toy Django app, memory consumption is mostly strings, and most strings are short (< 16 or even < 8 bytes)... in other words, identifiers. So if you ran 100 toy Django instances it seems likely this would help! And no I haven't benchmarked it, /arry

9 8