Mailman 3 February 2010 - Python-Dev

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Glenn Linderman 07 Feb '10

07 Feb '10

On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: > When the Python executable is given a `-R` flag, or the environment > variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` > directory and write a `pyc` file to that directory with the hexlified > magic number as the base name. > After the discussion so far, my opinion is that if the source directory contains an appropriate python repositiory directory [1], and the version of Python implements PEP 3147, that there should be no need for -R or $PYTHONPYR to exist, but that such versions of Python would simply, and always look in the python repository directory for binaries. I've reached this conclusion for several reasons/benefits: 1) it makes the rules simpler for people finding the binaries 2) there is no "double lookup" to find a binary at run time 3) if the PEP changes to implement alternatives B or C in [1], then I hear a large consensus of people that like that behavior, to clean up the annoying clutter of .pyc files mixed with source. 4) There is no need to add or document the command line option or environment variable. [1] Alternative A... source-file-root.pyr, as in the PEP, Alt. B... source-file-dir/__pyr__ all versions/files in same lookaside directory, Alt. C... source-file-dir/__pyr_version__, each Python version with different bytecode would have some sort of version string or magic number that identifies it, and would look only in that directory for its .pyc/.pyo files. I prefer C for 4 reasons: 1) easier to blow away one version; 2) easier to see what that version has compiled; 3) most people use only one or two versions, so directory proliferation is limited; 4) even when there are 30 versions of Python, the subdirectories would contain the same order-of-magnitude count of files as the source directory for performance issues, if the file system has a knee in the performance curve as some do. -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking

5 4

Re: [Python-Dev] IO module improvements
by Antoine Pitrou 07 Feb '10

07 Feb '10

Le dimanche 07 février 2010 à 09:54 +0100, Pascal Chambon a écrit : > > Actually, TextIOWrapper is simply not thread-safe for most of its operations. I > > think we did the work for simple writing, though, since it's better for > > multi-threaded use of print(). > > Argh, I had the impression that all io streams were theoretically > thread-safe (although it's not documented so indeed). It needs > clarification maybe. It should first be discussed which classes need to be thread-safe. There is nothing about it in PEP 3116, and the first (pure Python) implementation of the io module had no locks anywhere. We later added locks to the Buffered classes because it seemed an obvious requirement for many use cases (for example object databases such as ZODB or Durus). > > You can, but be aware that _pyio is *really* slow... I'm not sure it would be a > > service to many users. > > > Hum... would a pure python module, augmented with cython declarations, > offer a speed similar to c modules ? Maybe I shall investigate that > way, because it would be great to have an implemntation which is both > safer and sufficiently quick... There's no obvious answer. I suspect that it won't be as fast as the current C implementation, because some things simply aren't possible or available in Python. But it could be "fast enough". You have to experiment. Regards Antoine.

1 0

Proposal for the getpass module
by Tarek Ziadé 06 Feb '10

06 Feb '10

Hello, I would like to propose a small change in the getpass module so it's able to get passwords from keyrings (like KWallet, Keychain, etc) The idea is to provide a getpass.cfg configuration file where people can provide the name of a function to use when getpass is called. Then third-party projects can implement this function. For example the Python Keyring library.[1] could be installed and configured to be used by people that wants getpass calls to be handled by this tool. That's a backward compatible change, and it avoids adding any new module in the stdlib. Plus, it offers a greatly improved getpass module with no risks for the stdlib stability : it becomes a reference implementation with an interface for third-party implementers. A prototype is here : http://bitbucket.org/tarek/getpass/ (work in progress but you can get the idea) [1] http://pypi.python.org/pypi/keyring Regards Tarek -- Tarek Ziadé | http://ziade.org

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw 06 Feb '10

06 Feb '10

On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote: >So what would you do for concurrent writers, then? The current >implementation relies on creat(O_EXCL) to be atomic, so a second >writer would just fail. This is but the only IO operation that is >guaranteed to be atomic (along with mkdir(2)), so reusing the current >approach doesn't work. I believe rename(2) is atomic also, at least on POSIX. I'm not sure if that helps us though. -Barry

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw 06 Feb '10

06 Feb '10

On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > >It would be nice if all the compilation files could be tucked >into one single zipfile per directory to reduce directory clutter. > >It has several benefits besides tidiness. It hides the implementation >details of when magic numbers get shifted. And it may allow faster >start-up times when the zipfile is in the disk cache. This is closer in spirit to the original (uncirculated) PEP which called for fat pyc files, but without the complicated implementation details. It's still an interesting approach to explore. Writer concurrency can be handled with dot-lock files, but that does incur some extra overhead, such as the remove() of the lock file. -Barry

1 0

Making loggerClass an attribute of the logger manager?
by Georg Brandl 06 Feb '10

06 Feb '10

Hi, recently I had a use case where I wanted to use logging in two completely separate parts of the same process. One of them needs to create instances a specific Logger subclass, while the other is fine with the default loggers. I got around the problem of the unique root node by using two Managers (and then using Manager.getLogger() instead of getLogger()), but I can only set the loggerClass globally. Making the loggerClass configurable per manager would solve the problem for me, and AFAICS since most applications don't use different managers anyway, there should not be any detrimental effects. What do you think? cheers, Georg

2 2

Re: [Python-Dev] Absolute imports in Python 2.x?
by Brett Cannon 06 Feb '10

06 Feb '10

On Wed, Feb 3, 2010 at 05:57, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > Mark Dickinson wrote: >> Agreed on all points. Would it be terrible to simply add all relevant >> tags the moment a PEP is accepted? E.g., if a PEP pronounces some >> particular behaviour deprecated in Python 3.3 and removed in Python >> 3.4, then corresponding release blockers for 3.3 and 3.4 could be >> opened as part of implementing the PEP. > > That strikes me as a really good idea, since the tracker already serves > as a global "to do" list for the project (heck, I've used it as a > semi-private to do list at times, when I've thought of things I need to > fix when I don't have the roundtuits free to actually fix them). Martin filled me in on how to add new versions, so when we need a new one people can let me know. -Brett > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org >

1 0

IO module improvements
by Pascal Chambon 06 Feb '10

06 Feb '10

Hello The new modular io system of python is awesome, but I'm running into some of its limits currently, while replacing the raw FileIO with a more advanced stream. So here are a few ideas and questions regarding the mechanisms of this IO system. Note that I'm speaking in python terms, but these ideas should also apply to the C implementation (with more programming hassle of course). - some streams have specific attributes (i.e mode, name...), but since they'll often been wrapped inside buffering or encoding streams, these attributes will not be available to the end user. So wouldn't it be great to implement some "transversal inheritance", simply by delegating to the underlying buffer/raw-stream, attribute retrievals which fail on the current stream ? A little __getattr__ should do it fine, shouldn't it ? By the way, I'm having trouble with the "name" attribute of raw files, which can be string or integer (confusing), ambiguous if containing a relative path, and which isn't able to handle the new case of my library, i.e opening a file from an existing file handle (which is ALSO an integer, like C file descriptors...) ; I propose we deprecate it for the benefit or more precise attributes, like "path" (absolute path) and "origin" (which can be "path", "fileno", "handle" and can be extended...). Methods too would deserve some auto-forwarding. If you want to bufferize a raw stream which also offers size(), times(), lock_file() and other methods, how can these be accessed from a top-level buffering/text stream ? So it would be interesting to have a system through which a stream can expose its additional features to top level streams, and at the same time tell these if they must flush() or not before calling these new methods (eg. asking the inode number of a file doesn't require flushing, but knowing its real size DOES require it.). - I feel thread-safety locking and stream stream status checking are currently overly complicated. All methods are filled with locking calls and CheckClosed() calls, which is both a performance loss (most io streams will have 3 such levels of locking, when 1 would suffice) and error-prone (some times ago I've seen in sources several functions in which checks and locks seemed lacking). Since we're anyway in a mood of imbricating streams, why not simply adding a "safety stream" on top of each stream chain returned by open() ? That layer could gracefully handle mutex locking, CheckClosed() calls, and even, maybe, the attribute/method forwarding I evocated above. I know a pure metaprogramming solution would maybe not suffice for performance-seekers, but static implementations should be doable as well. - some semantic decisions of the current system are somehow dangerous. For example, flushing errors occuring on close are swallowed. It seems to me that it's of the utmost importance that the user be warned if the bytes he wrote disappeared before reaching the kernel ; shouldn't we decidedly enforce a "don't hide errors" everywhere in the io module ?. Regards, Pascal

8 11

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw 06 Feb '10

06 Feb '10

On Feb 03, 2010, at 09:26 AM, Floris Bruynooghe wrote: >On Wed, Feb 03, 2010 at 06:14:44PM +1100, Ben Finney wrote: >> I don't understand the distinction you're making between those two >> options. Can you explain what you mean by each of “siblings” and >> “folder-per-folder”? > >sibilings: the original proposal, i.e.: > >foo.py >foo.pyr/ > MAGIC1.pyc > MAGIC1.pyo > ... >bar.py >bar.pyr/ > MAGIC1.pyc > MAGIC1.pyo > ... > >folder-per-folder: > >foo.py >bar.py >__pyr__/ > foo.MAGIC1.pyc > foo.MAGIC1.pyo > foo.MAGIC2.pyc > bar.MAGIC1.pyc > ... > >IIUC Correct. If necessary, I'll define those two terms in the PEP. -Barry

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Brett Cannon 06 Feb '10

06 Feb '10

On Wed, Feb 3, 2010 at 13:33, "Martin v. Löwis" <martin(a)v.loewis.de> wrote: > Guido van Rossum wrote: > > On Wed, Feb 3, 2010 at 12:47 PM, Nick Coghlan <ncoghlan(a)gmail.com> > wrote: > >> On the issue of __file__, I'd suggesting not being too hasty in > >> deprecating that in favour of __source__. While I can see a lot of value > >> in having it point to the source file more often with a different > >> attribute that points to the cached file, I don't see a lot of gain to > >> compensate for the pain of changing the name of __file__ itself. > > > > Can you clarify? In Python 3, __file__ always points to the source. > > Clearly that is the way of the future. For 99.99% of uses of __file__, > > if it suddenly never pointed to a .pyc file any more (even if one > > existed) that would be just fine. So what's this talk of switching to > > __source__? > > I originally proposed it, not knowing that Python 3 already changed the > meaning of __file__ for byte code files. > > What I really wanted to suggest is that it should be possible to tell > what gets really executed, plus what source file had been considered. > > So if __file__ is always the source file, a second attribute should tell > whether a byte code file got read (so that you can delete that in case > you doubt it's current, for example). > > What should be done for loaders? Right now we have get_filename() which is what __file__ is to be set to. For importlib there is source_path and bytecode_path, but both of those are specified to return None in the cases of source or bytecode are not available, respectively. The bare minimum, I think, is we need loaders to have mehod(s) that return the path to the source -- whether it exists or not, to set __file__ to -- and the path to bytecode if it exists -- to set __compiled__ or whatever attribute we come up with. That suggests to me either two new methods or one that returns a two-item tuple. We could possibly keep get_filename() and say that people need to compare its output to what source_path()-equivalent method returns, but that seems bad if the source location needs to be based on the bytecode location. My thinking is we deprecate get_filename() and introduce some new method that returns a two-item tuple (get_paths?). First item is where the source should be, and the second is where the bytecode is if it exists (else it's None). Putting both calculations into a single method seems better than a source_path()/bytecode_path() as the latter would quite possibly need source_path() to call bytecode_path() on its own to calculate where the source should be if it doesn't exist on top of the direct call to get_bytecode() for setting __compiled__ itself. -Brett > Regards, > Martin > > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org >

3 4