Mailman 3 February 2010 - Python-Dev

What to intern (e.g. func_code.co_filename)?
by Jake McGuire Feb. 14, 2010

Feb. 14, 2010

Has anyone come up with rules of thumb for what to intern and what the performance implications of interning are? I'm working on profiling App Engine again, and since they don't allow marshall I have to modify pstats to save the profile via pickle. While trying to get profiles under 1MB, I noticed that each function has its own copy of the filename in which it is defined, and sometimes these strings can be rather long. Creating a code object already interns a bunch of stuff; argument names, … [View More]

3 2

Summary of Python tracker Issues
by Python tracker Feb. 12, 2010

Feb. 12, 2010

ACTIVITY SUMMARY (02/05/10 - 02/12/10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2602 open (+33) / 17137 closed (+25) / 19739 total (+58) Open issues with patches: 1069 Average duration of open issues: 712 days. Median duration of open issues: 466 days. Open Issues Breakdown open 2567 (+33) pending 34 ( +0) Issues Created Or Reopened (59) … [View More]

1 0

unittest: shortDescription, _TextTestResult and other issues
by Michael Foord Feb. 12, 2010

Feb. 12, 2010

Hello all, I've been looking at outstanding unittest issues as part of my preparation for my PyCon talk. There are a couple of changes (minor) I'd like to make that I thought I ought to run past Python-Dev first. If I don't get any responses then I'll just do it, so you have been warned. :-) The great google merge into unittest happened at PyCon last year [1]. This included a change to TestCase.shortDescription() so that it would *always* include the test name, whereas previously it … [View More]would return the test docstring or None. The problem this change solved was that tests with a docstring would not have their name (test class and method name) reported during the test run. Unfortunately the change broke part of twisted test running. Reported as issue 7588: http://bugs.python.org/issue7588 It seems to me that the same effect (always reporting test name) can be achieved in _TextTestResult.getDescription(). I propose to revert the change to TestCase.shortDescription() (which has both a horrible name and a horrible implementation and should probably be renamed getDocstring so that what it does is obvious but never mind) and put the change into _TextTestResult. It annoys me that _TextTestResult is private, as you will almost certainly want to use it or subclass it when implementing custom test systems. I am going to rename it TextTestResult, alias the old name and document the old name as being deprecated. Another issue that I would like to address, but there are various possible approaches, is issue 7559: http://bugs.python.org/issue7559 Currently loadTestsFromName catches ImportError and rethrows as AttributeError. This is horrible (it obscures the original error) but there are backwards compatibility issues with fixing it. There are three possible approaches: 1) Leave it (the default) 2) Only throw an AttributeError if the import fails due to the name being invalid (the module not existing) otherwise allow the error through. (A minor but less serious change in behavior). 3) A new method that turns failures into pseudo-tests that fail with the original error when run. Possibly deprecating loadTestsFromName I favour option 3, but can't think of a good replacement name. :-) Comments welcomed. Despite deprecating (in the documentation - no actual deprecations warnings I believe) a lot of the duplicate ways of doing things (assert* favoured over fail* and assertEqual over assertEquals) we didn't include deprecating assert_ in favour of assertTrue. I would like to add that to the documentation. After 3.2 is out I would like to clean up the documentation, removing mention of the deprecated methods from the *main* documentation into a separate 'deprecated methods' section. They currently make the documentation very untidy. The unittest page should probably be split into several pages anyway and needs improving. Other outstanding minor issues: Allow dotted names for test discovery http://bugs.python.org/issue7780 - I intend to implement this as described in the last comment A 'check_order' optional argument (defaulting to True) for assertSequenceEqual http://bugs.python.org/issue7832 - needs patch The breaking of __unittest caused by splitting unittesst into a package needs fixing. The fix needs to work when Python is run without frames support (IronPython). http://bugs.python.org/issue7815 - needs patch Allow a __unittest (or similar) decorator for user implemented assert functions http://bugs.python.org/issue1705520 - needs patch Allow modules to define test_suite callable. http://bugs.python.org/issue7501 - I propose to close as rejected. Use load_tests instead. Display time taken by individual tests when in verbose mode. http://bugs.python.org/issue4080 - anyone any opinions? Allow automatic formatting of arguments in assert* failure messages. http://bugs.python.org/issue6966 - I propose to close as rejected removeTest() method on TestSuite http://bugs.python.org/issue1778410 - anyone any opinions? expect methods (delayed fail) http://bugs.python.org/issue3615 - any opinions? Personally I think that the TestCase API is big enough already All the best, Michael Foord [1] Mostly in revision 7-837. http://svn.python.org/view?view=rev&revision=70837 -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. [View Less]

6 13

Re: [Python-Dev] Python 2.6.5
by anatoly techtonik Feb. 11, 2010

Feb. 11, 2010

On Tue, Feb 2, 2010 at 8:08 PM, Barry Warsaw <barry(a)python.org> wrote: > I'm thinking about doing a Python 2.6.5 release soon. I've added the > following dates to the Python release schedule Google calendar: > > 2009-03-01 Python 2.6.5 rc 1 > 2009-03-15 Python 2.6.5 final > > This allows us to spend some time on 2.6.5 at Pycon if we want. Please let me > know if you have any concerns about those dates. I've noticed a couple of issues that 100% crash … [View More]

10 30

Re: [Python-Dev] Executing zipfiles and directories (was Re: PyCon Keynote)
by Glenn Linderman Feb. 11, 2010

Feb. 11, 2010

On approximately 1/27/2010 1:08 AM, came the following characters from the keyboard of Glenn Linderman: > Without reference to distutils, it seems the pieces are: > > 1) a way to decide what to include in the package > 2) code that knows where to put what is included, on one or more > platforms > 3) the process to create the ZIP file that includes 1 & 2, and call it > an appropriate name.py > > 3 looks easy, once 1 & 2 are figured out. distutils might … [View More]

1 0

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
by Collin Winter Feb. 10, 2010

Feb. 10, 2010

To follow up on some of the open issues: On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter <collinwinter(a)google.com> wrote: [snip] > Open Issues > =========== > > - *Code review policy for the ``py3k-jit`` branch.* How does the CPython > community want us to procede with respect to checkins on the ``py3k-jit`` > branch? Pre-commit reviews? Post-commit reviews? > > Unladen Swallow has enforced pre-commit reviews in our trunk, but we realize > &… [View More]nbsp;this may lead to long review/checkin cycles in a purely-volunteer > organization. We would like a non-Google-affiliated member of the CPython > development team to review our work for correctness and compatibility, but we > realize this may not be possible for every commit. The feedback we've gotten so far is that at most, only larger, more critical commits should be sent for review, while most commits can just go into the branch. Is that broadly agreeable to python-dev? > - *How to link LLVM.* Should we change LLVM to better support shared linking, > and then use shared linking to link the parts of it we need into CPython? The consensus has been that we should link shared against LLVM. Jeffrey Yasskin is now working on this in upstream LLVM. We are tracking this at http://code.google.com/p/unladen-swallow/issues/detail?id=130 and http://llvm.org/PR3201. > - *Prioritization of remaining issues.* We would like input from the CPython > development team on how to prioritize the remaining issues in the Unladen > Swallow codebase. Some issues like memory usage are obviously critical before > merger with ``py3k``, but others may fall into a "nice to have" category that > could be kept for resolution into a future CPython 3.x release. The big-ticket items here are what we expected: reducing memory usage and startup time. We also need to improve profiling options, both for oProfile and cProfile. > - *Create a C++ style guide.* Should PEP 7 be extended to include C++, or > should a separate C++ style PEP be created? Unladen Swallow maintains its own > style guide [#us-styleguide]_, which may serve as a starting point; the > Unladen Swallow style guide is based on both LLVM's [#llvm-styleguide]_ and > Google's [#google-styleguide]_ C++ style guides. Any thoughts on a CPython C++ style guide? My personal preference would be to extend PEP 7 to cover C++ by taking elements from http://code.google.com/p/unladen-swallow/wiki/StyleGuide and the LLVM and Google style guides (which is how we've been developing Unladen Swallow). If that's broadly agreeable, Jeffrey and I will work on a patch to PEP 7. Thanks, Collin Winter [View Less]

2 1

PyCon is coming! Tomorrow, Feb. 10th is the last day for pre-conference rates
by VanL Feb. 10, 2010

Feb. 10, 2010

PyCon is coming! Tomorrow (February 10th) is the last day for pre-conference rates. You can register for PyCon online at: <https://us.pycon.org/2010/register/> Register while it is still Feb. 10th somewhere in the world and rest easy in the knowledge that within 10 days you will enjoying the company of some of the finest Python hackers in the world. As an additional bonus, PyCon this year will be in Atlanta, making it an ideal location for those looking for a way to escape the late … [View More]

1 0

PEP 345 and PEP 386
by Guido van Rossum Feb. 9, 2010

Feb. 9, 2010

All, I've reviewed PEP 345 and PEP 386 and am satisfied that after some small improvements they will be accepted. Most of the discussion has already taken place. I have one comment on PEP 345: Why is author-email mandatory? I'm sure there are plenty of cases where either the author doesn't want their email address published, or their last know email address is no longer valid. (Tarek responded off-line that it isn't all that mandatory; I propose to say so in the PEP.) I am also looking at … [View More]

2 3

Request for review of issue 4037
by Steven D'Aprano Feb. 9, 2010

Feb. 9, 2010

Hello, I have submitted a patch and a test script for issue 4037 on the bug tracker, "doctest.py should include method descriptors when looking inside a class __dict__" http://bugs.python.org/issue4037 I would be grateful if somebody could review it please, and if suitable, commit it. Thank you. -- Steven D'Aprano

1 0

Re: [Python-Dev] Forking and Multithreading - enemy brothers
by Ben Walker Feb. 9, 2010

Feb. 9, 2010

Pascal Chambon writes: > I don't really get it there... it seems to me that multiprocessing only > requires picklability for the objects it needs to transfer, i.e those > given as arguments to the called function, and thsoe put into > multiprocessing queues/pipes. Global program data needn't be picklable - > on windows it gets wholly recreated by the child process, from python > bytecode. > > So if you're having pickle errors, it must be because the > "… [View More]object_from_module_xyz" itself is *not* picklable, maybe because it > contains references to unpicklable objets. In such case, properly > implementing pickle magic methods inside the object should do it, > shouldn't it ? I'm also a long time lurker (and in financial software, coincidentally). Pascal is correct here. We use a number of C++ libraries wrapped via Boost.Python to do various calculations. The typical function calls return wrapped C++ types. Boost.Python types are not, unfortunately, pickleable. For a number of technical reasons, and also unfortunately, we often have to load these libraries in their own process, but we want to hide this from our users. We accomplish this by pickling the instance data, but importing the types fresh when we unpickle, all implemented in the magic pickle methods. We would lose any information that was dynamically added to the type in the remote process, but we simply don't do that. We very often have many unpickleable objects imported somewhere when we spin off our processes using the multiprocess library, and this does not cause any problems. Jesse Noller <jnoller <at> gmail.com> writes: > We already have an implementation that spawns a > subprocess and then pushes the required state to the child. The > fundamental need for things to be pickleable *all the time* kinda > makes it annoying to work with. This requirement puts a fairly large additional strain on working with unwieldy, wrapped C++ libraries in a multiprocessing environment. I'm not very knowledgeable on the internals of the system, but would it be possible to have some kind of fallback system whereby if an object fails to pickle we instead send information about how to import it? This has all kinds of limitations - it only works for importable things (i.e. not instances), it can potentially lose information dynamically added to the object, etc., but I thought I would throw the idea out there so someone knowledgeable can decide if it has any merit. Ben [View Less]

3 2

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 9, 2010

Feb. 9, 2010

On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: >With a single cache directory, we could have an option to force writing >bytecode to a desired location. That might be useful on it's own for >creating runtime bytecode only installations for installers. One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide … [View More]

4 4

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
by Collin Winter Feb. 9, 2010

Feb. 9, 2010

Hi Craig, On Tue, Feb 2, 2010 at 4:42 PM, Craig Citro <craigcitro(a)gmail.com> wrote: >> Done. The diff is at >> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed >> Cython, Shedskin and a bunch of other alternatives to pure CPython. >> Some of that information is based on conversations I've had with the >> respective developers, and I'd appreciate corrections if I'm out of >> date. >> > > Well, it's a minor nit, but it … [View More]

1 0

__file__ is not always an absolute path
by Ezio Melotti Feb. 8, 2010

Feb. 8, 2010

In #7712 I was trying to change regrtest to always run the tests in a temporary CWD (e.g. /tmp/@test_1234_cwd/). The patches attached to the issue add a context manager that changes the CWD, and it works fine when I run ./python -m test.regrtest from trunk/. However, when I try from trunk/Lib/ it fails with ImportErrors (note that the latest patch by Florent Xicluna already tries to workaround the problem). The traceback points to "the_package = __import__(abstest, globals(), locals(), [])… [View More]

8 13

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 8, 2010

Feb. 8, 2010

On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: >It's also the case that having to run Python to manage my own filesystem >would very annoying. If a dev has a broken .pyc that prevents the >affected Python build from even starting how are they meant to use the >nonfunctioning interpreter to find and delete the offending file? How is >someone meant to find and delete the .pyc files if they prefer to use a >graphical file manager over (or in conjunction with) the command line?… [View More]

5 8

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: >I must admit I quite like the __pyr__ directory approach as well. Since >the interpreter knows the suffix it is looking for, names shouldn't >conflict. Using a single directory allows the name to be less cryptic, >too (e.g. __pycache__). Something else that occurs to me; the name of the directory (under folder-per-folder approach) probably ought to be the same as the name of the module attribute. There's probably no good reason … [View More]

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: >How about using an optionally relative cache dir setting to let >the user decide ? Why do we need that level of flexibility? -Barry

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >Can you clarify? In Python 3, __file__ always points to the source. >Clearly that is the way of the future. For 99.99% of uses of __file__, >if it suddenly never pointed to a .pyc file any more (even if one >existed) that would be just fine. So what's this talk of switching to >__source__? Upon further reflection, I agree. __file__ also points to the source in Python 2.7. Do we need an attribute to point to the compiled bytecode file? -Barry

5 6

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >It's an interesting challenge to write the file in such a way that >it's safe for a reader and writer to co-exist. Like Brett, I >considered an append-only scheme, but one needs to handle the case >where the bytecode for a particular magic number changes. At some >point you'd need to sweep garbage from the file. All solutions seem >unnecessarily complex, and unnecessary since in practice the case >should not come up. I … [View More]

2 2

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: >The pyc/pyo files are just an optimisation detail, and are essentially >temporary. Given that, if they were to live in a single directory, to me it >seems obvious that the default location for that should be in the system >temporary directory. I an immediately think of the following advantages: > >1. No one really complains too much about putting things in /tmp unless it >starts taking up too much space. In which case they … [View More]

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: >Martin v. Löwis schrieb: >> There is also the issue of race conditions with multiple simultaneous >> accesses. The original format for the PEP had race conditions for >> multiple simultaneous writers; ZIP will also have race conditions for >> concurrent readers/writers (as any new writer will have to overwrite >> the central directory, making the zip file temporarily unavailable - >> unless they copy … [View More]

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Glenn Linderman Feb. 7, 2010

Feb. 7, 2010

On approximately 1/30/2010 4:00 PM, came the following characters from the keyboard of Barry Warsaw: > When the Python executable is given a `-R` flag, or the environment > variable `$PYTHONPYR` is set, then Python will create a `foo.pyr` > directory and write a `pyc` file to that directory with the hexlified > magic number as the base name. > After the discussion so far, my opinion is that if the source directory contains an appropriate python repositiory directory [1], … [View More]

5 4

Re: [Python-Dev] IO module improvements
by Antoine Pitrou Feb. 7, 2010

Feb. 7, 2010

Le dimanche 07 février 2010 à 09:54 +0100, Pascal Chambon a écrit : > > Actually, TextIOWrapper is simply not thread-safe for most of its operations. I > > think we did the work for simple writing, though, since it's better for > > multi-threaded use of print(). > > Argh, I had the impression that all io streams were theoretically > thread-safe (although it's not documented so indeed). It needs > clarification maybe. It should first be discussed which classes … [View More]

1 0

Proposal for the getpass module
by Tarek Ziadé Feb. 7, 2010

Feb. 7, 2010

Hello, I would like to propose a small change in the getpass module so it's able to get passwords from keyrings (like KWallet, Keychain, etc) The idea is to provide a getpass.cfg configuration file where people can provide the name of a function to use when getpass is called. Then third-party projects can implement this function. For example the Python Keyring library.[1] could be installed and configured to be used by people that wants getpass calls to be handled by this tool. That's a … [View More]

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 6, 2010

Feb. 6, 2010

On Feb 01, 2010, at 11:28 PM, Martin v. Löwis wrote: >So what would you do for concurrent writers, then? The current >implementation relies on creat(O_EXCL) to be atomic, so a second >writer would just fail. This is but the only IO operation that is >guaranteed to be atomic (along with mkdir(2)), so reusing the current >approach doesn't work. I believe rename(2) is atomic also, at least on POSIX. I'm not sure if that helps us though. -Barry

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 6, 2010

Feb. 6, 2010

On Jan 31, 2010, at 11:04 AM, Raymond Hettinger wrote: >> It does this by >> allowing many different byte compilation files (.pyc files) to be >> co-located with the Python source file (.py file). > >It would be nice if all the compilation files could be tucked >into one single zipfile per directory to reduce directory clutter. > >It has several benefits besides tidiness. It hides the implementation >details of when magic numbers get shifted. And it may … [View More]

1 0