From jyasskin at gmail.com  Sat Apr  4 20:30:20 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 4 Apr 2009 13:30:20 -0500
Subject: [stdlib-sig] [Python-Dev] core python tests
In-Reply-To: <43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
References: <49D3F8D0.8070805@wingware.com>
	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>
	<49D64748.70305@voidspace.org.uk>
	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>
	<49D64ECB.9040100@voidspace.org.uk> <20090404025534.GA12996@idyll.org>
	<49D6ED27.8030908@gmail.com> <loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
Message-ID: <5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>

On Sat, Apr 4, 2009 at 11:52 AM, Collin Winter <collinw at gmail.com> wrote:
> On Sat, Apr 4, 2009 at 7:33 AM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
>> Antoine Pitrou wrote:
>>>
>>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>>
>>>>
>>>> C. Titus Brown wrote:
>>>>
>>>>>
>>>>> I vote for a separate mailing list -- 'python-tests'? -- but I don't
>>>>> know exactly how splintered to make the conversation. ?It probably
>>>>> belongs at python.org but if you want me to host it, I can.
>>>>>
>>>>
>>>> If too many things get moved off to SIGs there won't be anything left
>>>> for python-dev to talk about ;)
>>>>
>>>
>>> There is already an stdlib-sig, which has been almost unused.
>>>
>>>
>>
>> stdlib-sig isn't *quite* right (the testing and benchmarking are as much
>> about core python as the stdlib) - although we could view the benchmarks and
>> tests themselves as part of the standard library...
>>
>> Either way we should get it underway. Collin and Jeffrey - happy to use
>> stdlib-sig?
>
> Works for me.

Me too.

bcc python-dev, -> stdlib-sig

First question: Do people want the unladen-swallow performance tests
in the CPython repository until the whole library gets moved out? If
so, where? Tools/performance? Lib/test/benchmarks?

Jeffrey

From solipsis at pitrou.net  Sat Apr  4 20:39:27 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 04 Apr 2009 20:39:27 +0200
Subject: [stdlib-sig] [Python-Dev] core python tests
In-Reply-To: <5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
References: <49D3F8D0.8070805@wingware.com>
	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>
	<49D64748.70305@voidspace.org.uk>
	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>
	<49D64ECB.9040100@voidspace.org.uk> <20090404025534.GA12996@idyll.org>
	<49D6ED27.8030908@gmail.com> <loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
Message-ID: <1238870367.6865.6.camel@fsol>

Le samedi 04 avril 2009 ? 13:30 -0500, Jeffrey Yasskin a ?crit :
> bcc python-dev, -> stdlib-sig
> 
> First question: Do people want the unladen-swallow performance tests
> in the CPython repository until the whole library gets moved out? If
> so, where? Tools/performance?

Tools/performance sounds good. But it shouldn't include third-party
software, otherwise I think a separate project in the sandbox would be
better.




From michael at voidspace.org.uk  Sat Apr  4 20:49:26 2009
From: michael at voidspace.org.uk (Michael Foord)
Date: Sat, 04 Apr 2009 19:49:26 +0100
Subject: [stdlib-sig] [Python-Dev] core python tests
In-Reply-To: <1238870367.6865.6.camel@fsol>
References: <49D3F8D0.8070805@wingware.com>	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>	<49D64748.70305@voidspace.org.uk>	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>	<49D64ECB.9040100@voidspace.org.uk>
	<20090404025534.GA12996@idyll.org>	<49D6ED27.8030908@gmail.com>
	<loom.20090404T110332-698@post.gmane.org>	<49D76FCD.8050303@voidspace.org.uk>	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
	<1238870367.6865.6.camel@fsol>
Message-ID: <49D7ABB6.7030608@voidspace.org.uk>

Antoine Pitrou wrote:
> Le samedi 04 avril 2009 ? 13:30 -0500, Jeffrey Yasskin a ?crit :
>   
>> bcc python-dev, -> stdlib-sig
>>
>> First question: Do people want the unladen-swallow performance tests
>> in the CPython repository until the whole library gets moved out? If
>> so, where? Tools/performance?
>>     
>
> Tools/performance sounds good. But it shouldn't include third-party
> software, otherwise I think a separate project in the sandbox would be
> better.
>   

If the code can't all be contributed to the PSF it probably has to go in 
a separate repo.

Michael

>
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From solipsis at pitrou.net  Sat Apr  4 20:58:08 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 04 Apr 2009 20:58:08 +0200
Subject: [stdlib-sig] [Python-Dev] core python tests
In-Reply-To: <49D7ABB6.7030608@voidspace.org.uk>
References: <49D3F8D0.8070805@wingware.com>
	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>
	<49D64748.70305@voidspace.org.uk>
	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>
	<49D64ECB.9040100@voidspace.org.uk> <20090404025534.GA12996@idyll.org>
	<49D6ED27.8030908@gmail.com> <loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
	<1238870367.6865.6.camel@fsol>  <49D7ABB6.7030608@voidspace.org.uk>
Message-ID: <1238871488.6865.8.camel@fsol>

Le samedi 04 avril 2009 ? 19:49 +0100, Michael Foord a ?crit :
> If the code can't all be contributed to the PSF it probably has to go in 
> a separate repo.

By third party software I meant stuff such as Django. We already have
external code in the repo which is probably not contributed to the
PSF ;) (e.g. zlib, sqlite). My point is only that it would be bad to
make the standard Python checkout much bigger only for performance
testing.



From solipsis at pitrou.net  Sun Apr  5 02:52:55 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 05 Apr 2009 02:52:55 +0200
Subject: [stdlib-sig] Benchmark suggestion
In-Reply-To: <1238871488.6865.8.camel@fsol>
References: <49D3F8D0.8070805@wingware.com>
	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>
	<49D64748.70305@voidspace.org.uk>
	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>
	<49D64ECB.9040100@voidspace.org.uk> <20090404025534.GA12996@idyll.org>
	<49D6ED27.8030908@gmail.com> <loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
	<1238870367.6865.6.camel@fsol>  <49D7ABB6.7030608@voidspace.org.uk>
	<1238871488.6865.8.camel@fsol>
Message-ID: <1238892775.6865.34.camel@fsol>

hi,

A possible benchmark addition could be json parsing/unparsing using the
pure Python parts of the simplejson library.

Other things which would be interesting to measure:
- a production-grade HTTP implementation (some parts of Twisted?
something else?)
- something involving Unicode

cheers

Antoine.



From python at rcn.com  Sun Apr  5 04:01:27 2009
From: python at rcn.com (Raymond Hettinger)
Date: Sat, 4 Apr 2009 19:01:27 -0700
Subject: [stdlib-sig] Benchmark suggestion
References: <49D3F8D0.8070805@wingware.com><43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com><49D64748.70305@voidspace.org.uk><43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com><49D64ECB.9040100@voidspace.org.uk>
	<20090404025534.GA12996@idyll.org><49D6ED27.8030908@gmail.com>
	<loom.20090404T110332-698@post.gmane.org><49D76FCD.8050303@voidspace.org.uk><43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com><5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com><1238870367.6865.6.camel@fsol>
	<49D7ABB6.7030608@voidspace.org.uk><1238871488.6865.8.camel@fsol>
	<1238892775.6865.34.camel@fsol>
Message-ID: <DAC2D614ABC142B5AB3DA7F0A35AEC95@RaymondLaptop1>

Instead of benchmarking the kind of things we prefer to do in C,
how about something like timing the sphinx build for the rst docs?
This is something that represents a real and typical use of Python.


----- Original Message ----- 
From: "Antoine Pitrou" <solipsis at pitrou.net>
To: "Michael Foord" <michael at voidspace.org.uk>
Cc: "Collin Winter" <collinw at gmail.com>; "Michael Foord" <fuzzyman at voidspace.org.uk>; "stdlib-sig" <stdlib-sig at python.org>
Sent: Saturday, April 04, 2009 5:52 PM
Subject: [stdlib-sig] Benchmark suggestion


> hi,
> 
> A possible benchmark addition could be json parsing/unparsing using the
> pure Python parts of the simplejson library.
> 
> Other things which would be interesting to measure:
> - a production-grade HTTP implementation (some parts of Twisted?
> something else?)
> - something involving Unicode
> 
> cheers
> 
> Antoine.
> 
> 
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig

From jyasskin at gmail.com  Sun Apr  5 06:53:09 2009
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Sat, 4 Apr 2009 23:53:09 -0500
Subject: [stdlib-sig] Benchmark suggestion
In-Reply-To: <DAC2D614ABC142B5AB3DA7F0A35AEC95@RaymondLaptop1>
References: <49D3F8D0.8070805@wingware.com>
	<loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
	<1238870367.6865.6.camel@fsol> <49D7ABB6.7030608@voidspace.org.uk>
	<1238871488.6865.8.camel@fsol> <1238892775.6865.34.camel@fsol>
	<DAC2D614ABC142B5AB3DA7F0A35AEC95@RaymondLaptop1>
Message-ID: <5d44f72f0904042153v3e1827e3mfc16a651eff41ef@mail.gmail.com>

On Sat, Apr 4, 2009 at 9:01 PM, Raymond Hettinger <python at rcn.com> wrote:
> Instead of benchmarking the kind of things we prefer to do in C,

CPython prefers to do some things in C because it can and doing so is
faster. The other implementations can't, and I think it's a fair
comparison to show the penalty that incurs. For that matter, it's
useful to show how much of a win CPython gets from the extra
complexity of the C extension module. If that win gets too small, it'd
be nice to delete the C extension. The implication of this is that the
benchmark should be able to run either the C or pure-Python
implementation, independently for the two pythons being compared. The
Unladen Swallow benchmarks currently aren't very good at that and
should be extended.

On the other hand, we already have the pickling tests, which can show
some of the C/non-C difference.

> how about something like timing the sphinx build for the rst docs?
> This is something that represents a real and typical use of Python.

Also a good benchmark.

> ----- Original Message ----- From: "Antoine Pitrou" <solipsis at pitrou.net>
> To: "Michael Foord" <michael at voidspace.org.uk>
> Cc: "Collin Winter" <collinw at gmail.com>; "Michael Foord"
> <fuzzyman at voidspace.org.uk>; "stdlib-sig" <stdlib-sig at python.org>
> Sent: Saturday, April 04, 2009 5:52 PM
> Subject: [stdlib-sig] Benchmark suggestion
>
>
>> hi,
>>
>> A possible benchmark addition could be json parsing/unparsing using the
>> pure Python parts of the simplejson library.
>>
>> Other things which would be interesting to measure:
>> - a production-grade HTTP implementation (some parts of Twisted?
>> something else?)
>> - something involving Unicode
>>
>> cheers
>>
>> Antoine.
>>
>>
>> _______________________________________________
>> stdlib-sig mailing list
>> stdlib-sig at python.org
>> http://mail.python.org/mailman/listinfo/stdlib-sig
>
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig
>



-- 
Namast?,
Jeffrey Yasskin
http://jeffrey.yasskin.info/

From solipsis at pitrou.net  Sun Apr  5 12:18:07 2009
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 05 Apr 2009 12:18:07 +0200
Subject: [stdlib-sig] Benchmark suggestion
In-Reply-To: <DAC2D614ABC142B5AB3DA7F0A35AEC95@RaymondLaptop1>
References: <49D3F8D0.8070805@wingware.com>
	<43aa6ff70904030919j725b375avfbe4c80c9f7bc464@mail.gmail.com>
	<49D64748.70305@voidspace.org.uk>
	<43aa6ff70904031035i62687614va480c93db09ade36@mail.gmail.com>
	<49D64ECB.9040100@voidspace.org.uk> <20090404025534.GA12996@idyll.org>
	<49D6ED27.8030908@gmail.com> <loom.20090404T110332-698@post.gmane.org>
	<49D76FCD.8050303@voidspace.org.uk>
	<43aa6ff70904040952g4aece85ajfceac04b7d857194@mail.gmail.com>
	<5d44f72f0904041130x5f805862t396787b8fbb5ce6f@mail.gmail.com>
	<1238870367.6865.6.camel@fsol>  <49D7ABB6.7030608@voidspace.org.uk>
	<1238871488.6865.8.camel@fsol> <1238892775.6865.34.camel@fsol>
	<DAC2D614ABC142B5AB3DA7F0A35AEC95@RaymondLaptop1>
Message-ID: <1238926687.6499.6.camel@fsol>

Le samedi 04 avril 2009 ? 19:01 -0700, Raymond Hettinger a ?crit :
> Instead of benchmarking the kind of things we prefer to do in C,
> how about something like timing the sphinx build for the rst docs?
> This is something that represents a real and typical use of Python.

Good idea indeed.
We might want to only build a subset of the docs however, otherwise it's
much too long :-)



From allison at parrot.org  Sun Apr 12 20:55:50 2009
From: allison at parrot.org (Allison Randal)
Date: Sun, 12 Apr 2009 11:55:50 -0700
Subject: [stdlib-sig] bootstrapping test suite
Message-ID: <49E23936.4060102@parrot.org>

For Pynie we're developing a bootstrapping test suite, which may be 
useful to other implementations as they migrate to 3.x. It uses Robert 
Collins' SubUnit, which decouples the testing framework from the tests. 
So the testing framework runs in CPython (or any reasonably complete 
Python 2.x implementation), while the tests run in Pynie. All that's 
required to pass the first test is an implementation of 'print' and 
string constants. The tests gradually build up in complexity through 
numeric constants, operators, control structures, functions, etc, and 
when we're done with them will walk all the way through to supporting 
the full Python 3.x syntax. When the bootstrap is complete, the test 
framework moves to Pynie too, integrated with the core CPython test suite.

It might eventually be useful to add to the py3k branch, though I 
wouldn't include it in release tarballs. Or, if another repo is started 
for the benchmark tests, the bootstrapping tests might be added there. 
We'll contribute it to the PSF, anyway, so it's available. It does 
depend on SubUnit, which isn't part of the standard library. I can strip 
it down to a minimal subset of SubUnit's features and include them in 
the test running script if that makes it more generally useful.

Right now the tests live in Lib/test/parrot in the Pynie repository, but 
will likely move to Lib/test/bootstrap (suggestions on Python-friendly 
naming welcome).

Allison

From collinw at gmail.com  Mon Apr 13 06:12:27 2009
From: collinw at gmail.com (Collin Winter)
Date: Sun, 12 Apr 2009 21:12:27 -0700
Subject: [stdlib-sig] bootstrapping test suite
In-Reply-To: <49E23936.4060102@parrot.org>
References: <49E23936.4060102@parrot.org>
Message-ID: <43aa6ff70904122112j1d897b37jba3899e97e859936@mail.gmail.com>

On Sun, Apr 12, 2009 at 11:55 AM, Allison Randal <allison at parrot.org> wrote:
> It might eventually be useful to add to the py3k branch, though I wouldn't
> include it in release tarballs. Or, if another repo is started for the
> benchmark tests, the bootstrapping tests might be added there. We'll
> contribute it to the PSF, anyway, so it's available. It does depend on
> SubUnit, which isn't part of the standard library. I can strip it down to a
> minimal subset of SubUnit's features and include them in the test running
> script if that makes it more generally useful.

I think including these in the common repository would be useful.
We've been doing something similar for our LLVM backend, and I can
think of at least one other infant Python implementation project that
would benefit from this kind of thing. I imagine every new Python
implementation goes through the phase of "well, we don't have enough
working to use unittest.py, but we still want tests"; I'd like to save
these projects the need to reinvent at least this particular wheel.

Collin Winter

From michael at voidspace.org.uk  Mon Apr 13 13:33:00 2009
From: michael at voidspace.org.uk (Michael Foord)
Date: Mon, 13 Apr 2009 12:33:00 +0100
Subject: [stdlib-sig] bootstrapping test suite
In-Reply-To: <49E23936.4060102@parrot.org>
References: <49E23936.4060102@parrot.org>
Message-ID: <49E322EC.6090206@voidspace.org.uk>

Allison Randal wrote:
> For Pynie we're developing a bootstrapping test suite, which may be 
> useful to other implementations as they migrate to 3.x. It uses Robert 
> Collins' SubUnit, which decouples the testing framework from the 
> tests. So the testing framework runs in CPython (or any reasonably 
> complete Python 2.x implementation), while the tests run in Pynie. All 
> that's required to pass the first test is an implementation of 'print' 
> and string constants. The tests gradually build up in complexity 
> through numeric constants, operators, control structures, functions, 
> etc, and when we're done with them will walk all the way through to 
> supporting the full Python 3.x syntax. When the bootstrap is complete, 
> the test framework moves to Pynie too, integrated with the core 
> CPython test suite.

Isn't Subunit GPL? That could be an issue for some projects.

Michael
>
> It might eventually be useful to add to the py3k branch, though I 
> wouldn't include it in release tarballs. Or, if another repo is 
> started for the benchmark tests, the bootstrapping tests might be 
> added there. We'll contribute it to the PSF, anyway, so it's 
> available. It does depend on SubUnit, which isn't part of the standard 
> library. I can strip it down to a minimal subset of SubUnit's features 
> and include them in the test running script if that makes it more 
> generally useful.
>
> Right now the tests live in Lib/test/parrot in the Pynie repository, 
> but will likely move to Lib/test/bootstrap (suggestions on 
> Python-friendly naming welcome).
>
> Allison
> _______________________________________________
> stdlib-sig mailing list
> stdlib-sig at python.org
> http://mail.python.org/mailman/listinfo/stdlib-sig


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog



From allison at parrot.org  Mon Apr 13 20:28:47 2009
From: allison at parrot.org (Allison Randal)
Date: Mon, 13 Apr 2009 11:28:47 -0700
Subject: [stdlib-sig] bootstrapping test suite
In-Reply-To: <49E322EC.6090206@voidspace.org.uk>
References: <49E23936.4060102@parrot.org> <49E322EC.6090206@voidspace.org.uk>
Message-ID: <49E3845F.10608@parrot.org>

Michael Foord wrote:
> 
> Isn't Subunit GPL? That could be an issue for some projects.

Aye, but the bootstrapping tests are just simple python 3 files that 
print "success" or "failure" to stdout, and would be licensed under 
whatever license the PSF chooses (presumably the PSF license).

Any test framework can run the tests without SubUnit. (The test 
framework and tests are completely decoupled.) I'm using SubUnit to link 
them into the CPython test suite.

Allison