[Python-Dev] Python-Dev Digest, Vol 92, Issue 156

[Nick Efford]

On Fri, 25 Mar 2011, python-dev-request at python.org wrote:

> Send Python-Dev mailing list submissions to
>        python-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        http://mail.python.org/mailman/listinfo/python-dev
> or, via email, send a message with subject or body 'help' to
>        python-dev-request at python.org
>
> You can reach the person managing the list at
>        python-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
>
> Today's Topics:
>
>   1. Re: sprints and pushes (Nick Coghlan)
>   2. Re: Proposal for Python 3.3: dependence injection (Nick Coghlan)
>   3. Re: Proposal for Python 3.3: dependence injection (Xavier Morel)
>   4. Re: Proposal for Python 3.3: dependence injection (Nick Coghlan)
>   5. Re: CRLF line endings (Nick Coghlan)
>   6. Re: Unload a module written in C (Nick Coghlan)
>   7. Re: Unload a module written in C (Victor Stinner)
>   8. Re: [Python-checkins] cpython: Remove test_importable().
>      Couldn't see how to make this reliable across all (Nick Coghlan)
>   9. Re: Unload a module written in C (Nick Coghlan)
>  10. Python Core Mentorship program (Jesse Noller)
>  11. Re: CRLF line endings (Stefan Krah)
>  12. Re: Python Core Mentorship program (Jesse Noller)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 25 Mar 2011 21:00:25 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: "Stephen J. Turnbull" <stephen at xemacs.org>
> Cc: Tres Seaver <tseaver at palladion.com>, python-dev at python.org
> Subject: Re: [Python-Dev] sprints and pushes
> Message-ID:
>        <AANLkTik7qQZPs0SXyo4BpaHTX=KvoeVZfWjF827Z29G1 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 12:51 PM, Stephen J. Turnbull
> <stephen at xemacs.org> wrote:
>> Eg, much as I normally respect Barry's intuitions, his proposal (to
>> remove costly tests, without reference to the possibility of missing
>> something important) is IMHO absolutely the wrong criterion. ?I don't
>> really know about Python's test suite, but in XEmacs the time-
>> expensive tests are mostly the ones that involve timeouts and lots of
>> system calls because they interact with the OS -- and those are
>> precisely the finicky areas where a small change can occasion an
>> unexpected bug. ?For XEmacs, those bugs also are more likely than
>> average to be showstoppers for dependents, in the sense that until
>> they're fixed, the dependents can't be tested at all. ?So the cost of
>> *not* running those tests is relatively high, too.
>
> For Python, our most expensive, slow tests are generally process
> related or IO related (over time the threading related ones have been
> largely fixed to use Event based signalling rather than relying on
> timeouts, so they're significantly faster than they once were).
>
> These slow tests are *also* typically the most platform dependent
> tests, so a "green light" from a local test run is generally pretty
> inconclusive - you don't really find out whether you borked something
> until you get green lights on the buildbots for platforms other than
> those the patch was developed on.
>
> So I still see value in having a standard "smoke test" that runs
> through the platform independent stuff, to reduce the number of minor
> errors that needlessly cause the buildbots to go red.
>
> The idea would be to create a tiered test suite along the following lines:
>
> 1. The buildbots: run the entire (-uall) test suite across a variety
> of platforms. Performed for every commit pushed to
> hg.python.org/cpython.
> 2. Complete local test: run the entire (-uall) test suite on a local
> working copy. Recommended before first committing a fix or change to a
> branch.
> 3. Basic local test: run the test suite with no additional resources
> enabled on a local working copy. Current closest equivalent to a
> "smoke test"
> 4. Proposed "smoke test": quick test of platform independent code for
> use when merging heads after a push race
> 5. Specific tests: run specified tests for modules being worked on.
> Used during development to check fix validity and feature degree of
> completion.
>
> With the volume of platform dependent code we have in CPython and the
> standard library, the only way we're ever likely to achieve
> consistently green buildbots is to move to a "staging repo" model,
> where commits only get made to the central repo after they have
> already passed muster on the buildbots at least once. I think that's
> actually a good way for us to go in the long run, but I also think
> separating out a fast set of platform independent is a decent idea.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 25 Mar 2011 21:04:17 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: Simon Cross <hodgestar+pythondev at gmail.com>
> Cc: Jesus Cea <jcea at jcea.es>, Python DEV <python-dev at python.org>
> Subject: Re: [Python-Dev] Proposal for Python 3.3: dependence
>        injection
> Message-ID:
>        <AANLkTin23KPuqUbUXN+BC_OWnu2fGxieTWPQZ9qm42=+ at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 7:22 PM, Simon Cross
> <hodgestar+pythondev at gmail.com> wrote:
>> On Fri, Mar 25, 2011 at 1:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> As an example of the last point, perhaps rather than modifying all the
>>> *clients* of the socket module, it may make more sense to have tools
>>> in the socket module itself to temporarily customise the socket
>>> creation process in the current thread. The advantage of such an
>>> approach is that it would then work for 3rd party libraries that
>>> create sockets, without requiring any further modification.
>>
>> -2. ?That wouldn't allow one to use normal sockets in one 3rd party
>> library and special sockets in another 3rd party library.
>
> Uh, yes it would. One possible implementation is to use exactly the
> same techniques that are used to implement contexts in the decimal
> module.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 25 Mar 2011 12:07:50 +0100
> From: Xavier Morel <python-dev at masklinn.net>
> To: Simon Cross <hodgestar+pythondev at gmail.com>
> Cc: Jesus Cea <jcea at jcea.es>, Nick Coghlan <ncoghlan at gmail.com>,
>        Python DEV <python-dev at python.org>
> Subject: Re: [Python-Dev] Proposal for Python 3.3: dependence
>        injection
> Message-ID: <A780E8D4-C85C-48C7-A970-CA78763D8A31 at masklinn.net>
> Content-Type: text/plain; charset=us-ascii
>
> On 2011-03-25, at 10:22 , Simon Cross wrote:
>> On Fri, Mar 25, 2011 at 1:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> As an example of the last point, perhaps rather than modifying all the
>>> *clients* of the socket module, it may make more sense to have tools
>>> in the socket module itself to temporarily customise the socket
>>> creation process in the current thread. The advantage of such an
>>> approach is that it would then work for 3rd party libraries that
>>> create sockets, without requiring any further modification.
>>
>> -2.  That wouldn't allow one to use normal sockets in one 3rd party
>> library and special sockets in another 3rd party library.
>>
>> Schiavo
>> Simon
>
> Or even in "first-party" code (in the stdlib) to set different timeouts on different APIs (say, an xmlrpclib.ServerProxy and an IMAP4 client).
>
> For instance, currently as far as I can tell setting a socket timeout on an xmlrpclib.ServerProxy without setting a global timeout involves:
>
> * subclassing all of xmlrpclib.Serverproxy, xmlrpclib.Transport, httplib.HTTP and httplib.HTTPConnection
> * overloading __init__ on the ServerProxy subclass (and on Transport if the timeout is to be a parameter)
> * overloading make_connection on the Transport subclass (in order to use the HTTP subclass and propagate the timeout)
> * overloading _connection_class on the HTTP subclass
> * overloading connect on the HTTPConnection class
>
> This *could* be solved by wrapping a socket-related thread-local context manager around each call resulting in the creation of a socket, but these call sites may not even be under control.
>
> ------------------------------
>
> Message: 4
> Date: Fri, 25 Mar 2011 21:10:08 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: Glenn Linderman <v+python at g.nevcal.com>
> Cc: python-dev at python.org
> Subject: Re: [Python-Dev] Proposal for Python 3.3: dependence
>        injection
> Message-ID:
>        <AANLkTikgmses_8UH+o9WJsPXBSY+FA6PxJDBC39m3jf+ at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 12:22 PM, Glenn Linderman <v+python at g.nevcal.com> wrote:
>> On 3/24/2011 4:25 PM, Nick Coghlan wrote:
>>
>> As an example of the last point, perhaps rather than modifying all the
>> *clients* of the socket module, it may make more sense to have tools
>> in the socket module itself to temporarily customise the socket
>> creation process in the current thread. The advantage of such an
>> approach is that it would then work for 3rd party libraries that
>> create sockets, without requiring any further modification.
>>
>> Would be easier to implement that way, not requiring changes to every client
>> of the socket library, but in some circles that would be called "action at a
>> distance", and harder to understand.
>
> Oh, it is definitely action at a distance, and quite deliberately so.
>
> My model for the suggestion is the context objects in the decimal
> module. They offer a constrained way to affect the way the entire
> decimal module goes about its business, and through judicious use of
> thread local storage and context managers, allow this to be done
> without distorting the public API of the decimal objects themselves.
>
> There's no reason a solution along those lines wouldn't work for the
> socket module, or any other API that would similarly benefit from
> providing a mechanism for applications to indirectly control library
> behaviour without resorting to monkey-patching.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 5
> Date: Fri, 25 Mar 2011 21:12:19 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: Stefan Krah <stefan at bytereef.org>
> Cc: python-dev at python.org
> Subject: Re: [Python-Dev] CRLF line endings
> Message-ID:
>        <AANLkTinWp3vOGymzS2bnrp8S3aRgOXL26FXSDY2nhxtC at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 7:36 PM, Stefan Krah <stefan at bytereef.org> wrote:
>> Hi,
>>
>> A commit hook prevented pushing changes to the cdecimal repository:
>>
>> pushing to ssh://hg@hg.python.org/features/cdecimal
>> searching for changes
>> 8 changesets found
>> remote: adding changesets
>> remote: adding manifests
>> remote: adding file changes
>> remote: added 8 changesets with 117 changes to 117 files
>> remote: error: pretxnchangegroup.eol hook failed: Modules/cdecimal/tests/dnloop.patch in 80914c366edf should not have CRLF line endings
>> remote: transaction abort!
>> remote: rollback completed
>> remote: abort: Modules/cdecimal/tests/dnloop.patch in 80914c366edf should not have CRLF line endings
>>
>>
>> However, dnloop.patch is correct and must have CRLF line endings. How
>> can I disable the commit hook?
>
> Don't disable the commit hook, update .hgeol to flag that file as
> requiring CRLF line endings.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 6
> Date: Fri, 25 Mar 2011 21:14:59 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: Victor Stinner <victor.stinner at haypocalc.com>
> Cc: "\"Martin v." <martin at v.loewis.de>, python-dev at python.org
> Subject: Re: [Python-Dev] Unload a module written in C
> Message-ID:
>        <AANLkTimM-g3mgTiKNvj-koT6rdM0-MOqgk9a3jO6XJnP at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 8:14 PM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>> Le vendredi 25 mars 2011 ? 07:59 +0100, "Martin v. L?wis" a ?crit :
>>>> Is there a bug somewhere, or do I misunderstood something important?
>>>
>>> Module unloading is simply not implemented, and would be very difficult
>>> to implement.
>>
>> My problem is that if Python is embeded, my module will still be active
>> after Py_FinalizeEx(). For example, if it installed an handler for the
>> SIGSEGV signal: a segmentation fault will call the handler which will
>> try to get the interpreter state, but there is no more interpreter. I
>> don't know if it is a problem or not, but I would prefer to cleanup my
>> module on Py_FinalizeEx().
>
> And registering your cleanup function with atexit() isn't enough? Or
> does that remove the handler too early?
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 7
> Date: Fri, 25 Mar 2011 12:36:01 +0100
> From: Victor Stinner <victor.stinner at haypocalc.com>
> To: Nick Coghlan <ncoghlan at gmail.com>
> Cc: "\"Martin v." <martin at v.loewis.de>, python-dev at python.org
> Subject: Re: [Python-Dev] Unload a module written in C
> Message-ID: <1301052961.11097.3.camel at marge>
> Content-Type: text/plain; charset="UTF-8"
>
> Le vendredi 25 mars 2011 ? 21:14 +1000, Nick Coghlan a ?crit :
>> On Fri, Mar 25, 2011 at 8:14 PM, Victor Stinner
>> <victor.stinner at haypocalc.com> wrote:
>>> Le vendredi 25 mars 2011 ? 07:59 +0100, "Martin v. L?wis" a ?crit :
>>>>> Is there a bug somewhere, or do I misunderstood something important?
>>>>
>>>> Module unloading is simply not implemented, and would be very difficult
>>>> to implement.
>>>
>>> My problem is that if Python is embeded, my module will still be active
>>> after Py_FinalizeEx(). For example, if it installed an handler for the
>>> SIGSEGV signal: a segmentation fault will call the handler which will
>>> try to get the interpreter state, but there is no more interpreter. I
>>> don't know if it is a problem or not, but I would prefer to cleanup my
>>> module on Py_FinalizeEx().
>>
>> And registering your cleanup function with atexit() isn't enough? Or
>> does that remove the handler too early?
>
> atexit() is too late: when Python is embeded, Py_Finalize() may be
> called a long time before the program does really finish.
>
> Well, I never embeded Python in another program, but it looks like some
> people do initialize/finalize Python more than once:
>
> http://bugs.python.org/issue11321
> "9th import of module _pickle always crashes"
>
> In this issue, Python is initialized/finalized 20 times.
>
> Victor
>
>
>
> ------------------------------
>
> Message: 8
> Date: Fri, 25 Mar 2011 21:37:09 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: python-dev at python.org
> Cc: "raymond.hettinger" <python-checkins at python.org>
> Subject: Re: [Python-Dev] [Python-checkins] cpython: Remove
>        test_importable(). Couldn't see how to make this reliable across all
> Message-ID:
>        <AANLkTikXNhrsmAnPXCY1A+NdFLYGzGzdLoDHaWfPxi1d at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 3:51 AM, raymond.hettinger
> <python-checkins at python.org> wrote:
>> http://hg.python.org/cpython/rev/5adddc6be3c1
>> changeset: ? 68902:5adddc6be3c1
>> user: ? ? ? ?Raymond Hettinger <python at rcn.com>
>> date: ? ? ? ?Thu Mar 24 10:51:06 2011 -0700
>> summary:
>> ?Remove test_importable(). ?Couldn't see how to make this reliable across all platforms.
>
> For this particular use case, a temporary directory added to sys,path,
> then some subsequent cleanup of sys.modules, sys.path and
> sys.path_importer_cache would likely do the trick.
>
> I've actually never been a fan of using TESTFN in tests - the tempfile
> module always struck me as being significantly more useful. It also
> makes tidying up the files afterwards really easy - the temporary
> directory context manager will blow everything away for you.
>
> Then again, quite a few of the tests I've written involved convoluted
> module hierarchies, so I'm likely a little biased by my typical use
> cases :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 9
> Date: Fri, 25 Mar 2011 21:47:16 +1000
> From: Nick Coghlan <ncoghlan at gmail.com>
> To: Victor Stinner <victor.stinner at haypocalc.com>
> Cc: "\"Martin v." <martin at v.loewis.de>, python-dev at python.org
> Subject: Re: [Python-Dev] Unload a module written in C
> Message-ID:
>        <AANLkTik0ORWLnCn7WAi9miP97eGuVunNeEA0H8K3DmEX at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 9:36 PM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>>> And registering your cleanup function with atexit() isn't enough? Or
>>> does that remove the handler too early?
>>
>> atexit() is too late: when Python is embeded, Py_Finalize() may be
>> called a long time before the program does really finish.
>
> I'm talking about the Python "atexit" module - any callbacks
> registered there are invoked by Py_Finalize(), not by process
> termination.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia
>
>
> ------------------------------
>
> Message: 10
> Date: Fri, 25 Mar 2011 08:03:43 -0400
> From: Jesse Noller <jnoller at gmail.com>
> To: Python-Dev <python-dev at python.org>
> Subject: [Python-Dev] Python Core Mentorship program
> Message-ID:
>        <AANLkTin=6-tARdp2BM+XaKr8oGS8x27hJM=MhWqgDhSi at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello everyone:
>
> I wanted to take a moment to outline another idea which came out of
> PyCon 2011 this year from numerous sources - a Python Core Mentorship
> Program predicated on the idea that Python-Core, and Python as a whole
> would be served by further lowering the barrier to entry of
> contribution, and to provide a program to connect new programmers,
> students, women, and others to experienced Python-Core developers
> (Mentors).
>
> Brett's revamp of the Dev guide was part one of "secret plan to get
> more people involved in python-core" - this is another part, but I'm
> not sure of the numbering scheme.
>
> The mission of the Python Core Mentor Program is to provide an open
> and welcoming place to connect students, programmers - and anyone
> interested in contributing to the Python-Core development. This
> project is based on the idea that the best way to welcome new people
> into any project is a venue which connects them to mentors who can
> assist in guiding them through the contribution process, including
> discussions on lists such as python-dev, and python-ideas, the bug
> tracker, mercurial, code reviews, etc.
>
> Additionally, mentors will assist in something incredibly critical to
> maintain contributor interest: getting patches through the process and
> actually *committed*. We all know - not everyone who is mentor will
> have all the answers, so mentors also act as conduits to others who
> will have the answer.
>
> The project itself will (hopefully) be low in time-spent, and largely
> self-managing. We will start simple with a mailing list
> (core-mentorship at python.org) where mentors, and those who wish to be
> mentored or ask questions may do so. This mailing list will have a
> code of conduct which will help prevent flame wars, or other
> counterproductive discussions - a code of conduct also makes it clear
> to mentors what they''re agreeing to when they decide to participate.
>
> The new list will also have a closed, members-only archive. After
> consulting with other core developers, we believe it's easier to ask
> questions when you don't have to worry about Google picking up your
> words from a public archive.
>
> We want to make this list a resource for people to be able to get
> started, ask "silly" questions, and so on - our goal is to turn anyone
> who wishes to be into an active, sustainable committer to Python.
>
> Mentors will be asked to answer questions - but also assist people in
> need of help with discussions on the mailing lists and bug tracker
> (conversations on which could have become contentious or stressful)
> and generally to be advocates for the people being mentored. For
> example - if a person submits a patch to the tracker, the mentor list
> may help them through initial code reviews, or discussions with other
> core developers. The job is to act as an experienced proxy for them.
>
> The first step to this project is to ask for volunteer mentors -
> people who are willing to help answer questions on the list, and
> generally guide people as needed being as friendly and courteous and
> welcoming as possible.
>
> If you are interested in being a mentor - or have feedback about this
> plan in general, please feel free to reach out to me
> (jnoller at gmail.com) directly. My goal, once this is setup, is to have
> the project largely self-managing, with the PSF helping to market it
> to the community as a whole.
>
> Jesse
>
>
> ------------------------------
>
> Message: 11
> Date: Fri, 25 Mar 2011 13:59:32 +0100
> From: Stefan Krah <stefan at bytereef.org>
> To: python-dev at python.org
> Subject: Re: [Python-Dev] CRLF line endings
> Message-ID: <20110325125932.GA13028 at sleipnir.bytereef.org>
> Content-Type: text/plain; charset=us-ascii
>
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> However, dnloop.patch is correct and must have CRLF line endings. How
>>> can I disable the commit hook?
>>
>> Don't disable the commit hook, update .hgeol to flag that file as
>> requiring CRLF line endings.
>
> Thanks, that works well.
>
>
> Stefan Krah
>
>
>
>
> ------------------------------
>
> Message: 12
> Date: Fri, 25 Mar 2011 09:06:48 -0400
> From: Jesse Noller <jnoller at gmail.com>
> To: exarkun at twistedmatrix.com
> Cc: Python-Dev <python-dev at python.org>
> Subject: Re: [Python-Dev] Python Core Mentorship program
> Message-ID:
>        <AANLkTim9orMhAs3yZevWcCrZfwxTf3fLEw_h9SZBBZmK at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> On Fri, Mar 25, 2011 at 9:04 AM,  <exarkun at twistedmatrix.com> wrote:
>> On 12:03 pm, jnoller at gmail.com wrote:
>>>
>>> Hello everyone:
>>>
>>>
>>> The new list will also have a closed, members-only archive. After
>>> consulting with other core developers, we believe it's easier to ask
>>> questions when you don't have to worry about Google picking up your
>>> words from a public archive.
>>
>> Boggle.
>>
>> Jean-Paul
>>
>
> I assume that means your in, or you hate that idea?
>
>
> ------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
>
>
> End of Python-Dev Digest, Vol 92, Issue 156
> *******************************************
>

-- 
Dr Nick Efford, School of | E: N.D.Efford at leeds.ac.uk
Computing, University of  | T: +44 113 343 6809
Leeds, Leeds, LS2 9JT, UK | W: http://www.comp.leeds.ac.uk/nde/
--------------------------+-----------------------------------------
PGP fingerprint: 6ADF 16C2 4E2D 320B F537  8F3C 402D 1C78 A668 8492