Definitely is something on VS. If I run the created application from command line (inside the release folder) here is the output:<div><br></div><div><div>11/13/2008 10:08:26 AM</div><div>WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior to time of first</div>
<div> data point (Tue Jan 01 00:00:32 2008), current line: 9411</div><div> series: 3D</div><div>force==False, exiting..</div><div>0</div><div>11/13/2008 10:08:31 AM</div><div><br></div><div>Since the issue is not with IronPython itself, I will look for answers on other places. Thanks for all your help.</div>
<div><br></div><div>regards,</div><div><br></div><div>Wilfredo</div><br><div class="gmail_quote">On Thu, Nov 13, 2008 at 9:59 AM, Wilfredo Lugo <span dir="ltr"><<a href="mailto:wilfredo.lugo@gmail.com">wilfredo.lugo@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">Interesting thing. <div><br></div><div>When I run the program on release mode without the profile the performance issue remains:<div>
<div><br></div><div>11/13/2008 9:50:19 AM</div><div class="Ih2E3d"><div>WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior to time of first</div>
<div> data point (Tue Jan 01 00:00:32 2008), current line: 9411</div><div> series: 3D</div><div>force==False, exiting..</div><div>0</div></div><div>11/13/2008 9:53:07 AM</div><div><br></div><div>However, when I run the program using the VS profiler, this is what I get:</div>
<div><div><br></div><div>11/13/2008 9:55:31 AM</div><div class="Ih2E3d"><div>WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior to time of firs</div></div><div class="Ih2E3d"><div> data point (Tue Jan 01 00:00:32 2008), current line: 9411</div>
<div> series: 3D</div>
<div>force==False, exiting..</div><div>0</div></div><div>11/13/2008 9:55:35 AM</div><div><br></div><div>Inside the profiler, the code is executed as in ipy.exe (near 4 seconds). Why the execution using the profiler will run faster than in Release mode? Since the profiler execution is normal, I don't think there is not much information I could provide.</div>
<div><br></div><div>This is the latest code:</div><div><br></div><div><div> [STAThread]</div><div class="Ih2E3d"><div> static void Main(string[] args)</div><div> {</div></div><div class="Ih2E3d"><div>
ScriptEngine engine = Python.CreateEngine();</div>
<div> ScriptSource source = engine.CreateScriptSourceFromFile("interpolate.py");</div><div> PythonCompilerOptions options = (PythonCompilerOptions) engine.GetCompilerOptions();</div><div> options.Module |= ModuleOptions.Optimized;</div>
</div><div> options.Module |= ModuleOptions.ModuleBuiltins;</div><div class="Ih2E3d"><div> Console.WriteLine(DateTime.Now);</div><div> CompiledCode compiled = source.Compile(options);</div>
<div> compiled.Execute();</div>
<div> ScriptScope scope = compiled.DefaultScope;</div><div> Microsoft.Func<string, string, string, string, int, int> func = scope.GetVariable<Microsoft.Func<string, string, string, string, int, int>>("interpolate_start");</div>
<div> Console.WriteLine(func("3-day-data.txt", "3-day-output.txt", "2008-01-01 00:00:21","2008-01-03 23:59:08",60).ToString());</div><div> Console.WriteLine(DateTime.Now);</div>
<div> Console.ReadLine();</div><div> }</div></div></div><div><br></div></div><div><div></div><div class="Wj3C7c"><br><div class="gmail_quote">On Wed, Nov 12, 2008 at 11:16 PM, Dino Viehland <span dir="ltr"><<a href="mailto:dinov@microsoft.com" target="_blank">dinov@microsoft.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p><span style="font-size:11.0pt;color:#1F497D">I don't think either of those would cause such a large
difference. I/O should be the same. Two other differences I've noticed but
again I wouldn't expect them to matter. The first is a lack of [STAThread] attribute
on Main. The 2<sup>nd</sup> is you don't need the ScriptRuntime runtime =
Python.CreateRuntime(); call. CreateEngine will create a runtime and return
you the Python engine associated with it.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p><span style="font-size:11.0pt;color:#1F497D">I'll try and take a look at the differences between the code we
generate for the two scenarios tomorrow to see if I can see anything that is
different. If that doesn't I might need a repro or if you have a profiler that
would work well. Given the huge difference in the runtimes 10-20 random stack
traces during the running of the program might be enough to figure it out.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt">
<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a> [mailto:<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>] <b>On
Behalf Of </b>Wilfredo Lugo<br>
<b>Sent:</b> Wednesday, November 12, 2008 3:24 PM<div><div></div><div><br>
<b>To:</b> Discussion of IronPython<br>
<b>Subject:</b> Re: [IronPython] Performance Issue</div></div></span></p>
</div><div><div></div><div>
<p> </p>
<p>I set the ModuleBuiltins compiler option but the
behavior is the same. The only difference from the command line is that I
am calling a python function from the DLR, and on the command line I am just
calling the method from __main__. Do you think could be an issue?
The other thing is that the python script relies heavily on I/O, input
file is in the range of 18Mb and the data is stored on another file. Does
the DLR handles I/O differently than the command line?</p>
<div>
<p> </p>
</div>
<div>
<p>regards,</p>
</div>
<div>
<p> </p>
</div>
<div>
<p style="margin-bottom:12.0pt">wilfredo</p>
<div>
<p>On Wed, Nov 12, 2008 at 5:39 PM, Dino Viehland <<a href="mailto:dinov@microsoft.com" target="_blank">dinov@microsoft.com</a>> wrote:</p>
<div>
<div>
<p><span style="font-size:11.0pt;color:#1F497D">The only thing that immediately
pops out at me as being different is we also set ModuleOptions.ModuleBuiltins
in the command line case – but I wouldn't expect it to make such a large
difference. But try setting that option as well and let's see what
happens.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> <a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>
[mailto:<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>]
<b>On Behalf Of </b>Wilfredo Lugo<br>
<b>Sent:</b> Wednesday, November 12, 2008 12:16 PM</span></p>
<div>
<div>
<p><span style="font-size:10.0pt"><br>
<b>To:</b> Discussion of IronPython<br>
<b>Subject:</b> Re: [IronPython] Performance Issue</span></p>
</div>
</div>
</div>
<div>
<div>
<p> </p>
<p>Thanks. Performance improved, but still over two minutes. Here
is the latest code:</p>
<div>
<p> </p>
</div>
<div>
<div>
<p> static void Main(string[] args)</p>
</div>
<div>
<p> {</p>
</div>
<div>
<p> ScriptRuntime runtime =
Python.CreateRuntime();</p>
</div>
<div>
<p> ScriptEngine engine =
Python.CreateEngine();</p>
</div>
<div>
<p> ScriptSource source =
engine.CreateScriptSourceFromFile("interpolate.py");</p>
</div>
<div>
<p> PythonCompilerOptions options
= (PythonCompilerOptions) engine.GetCompilerOptions();</p>
</div>
<div>
<p> options.Module |=
ModuleOptions.Optimized;</p>
</div>
<div>
<p>
Console.WriteLine(DateTime.Now);</p>
</div>
<div>
<p> CompiledCode compiled =
source.Compile(options);</p>
</div>
<div>
<p> compiled.Execute();</p>
</div>
<div>
<p> ScriptScope scope =
compiled.DefaultScope;</p>
</div>
<div>
<p> Microsoft.Func<string,
string, string, string, int, int> func =
scope.GetVariable<Microsoft.Func<string, string, string, string, int,
int>>("interpolate_start");</p>
</div>
<div>
<p>
Console.WriteLine(func("3-day-data.txt",
"3-day-output.txt", "2008-01-01 00:00:21","2008-01-03
23:59:08",60).ToString());</p>
</div>
<div>
<p>
Console.WriteLine(DateTime.Now);</p>
</div>
<div>
<p> Console.ReadLine();</p>
</div>
<div>
<p> }</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>Output :</p>
</div>
<div>
<div>
<p>11/12/2008 4:10:10 PM</p>
</div>
<div>
<p>WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior to time of
first</p>
</div>
<div>
<p> data point (Tue Jan 01 00:00:32 2008), current line: 9411</p>
</div>
<div>
<p> series: 3D</p>
</div>
<div>
<p>force==False, exiting..</p>
</div>
<div>
<p>0</p>
</div>
<div>
<p>11/12/2008 4:12:45 PM</p>
</div>
</div>
<div>
<p> </p>
</div>
<div>
<p style="margin-bottom:12.0pt"> </p>
<div>
<p>On Wed, Nov 12, 2008 at 4:01 PM, Dino Viehland <<a href="mailto:dinov@microsoft.com" target="_blank">dinov@microsoft.com</a>>
wrote:</p>
<div>
<div>
<p><span style="font-size:11.0pt;color:#1F497D">Oh, sorry, I missed a
step… You'll need to call ScriptEngine.GetCompilerOptions(). Cast
that to a PythonCompilerOptions, and then do options.Module |=
ModuleOptions.Optimized; Finally do source.Compile(options) and then it
should give you the optimized code.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> <a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>
[mailto:<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>]
<b>On Behalf Of </b>Wilfredo Lugo<br>
<b>Sent:</b> Wednesday, November 12, 2008 11:45 AM</span></p>
<div>
<div>
<p><span style="font-size:10.0pt"><br>
<b>To:</b> Discussion of IronPython<br>
<b>Subject:</b> Re: [IronPython] Performance Issue</span></p>
</div>
</div>
</div>
<div>
<div>
<p> </p>
<p>Thanks!.</p>
<div>
<p> </p>
</div>
<div>
<p>I was able to get the default scope from CompiledCode, but I am still
getting the same performance. Here is the latest code : </p>
</div>
<div>
<div>
<p> static void Main(string[] args)</p>
</div>
<div>
<p> {</p>
</div>
<div>
<p> ScriptRuntime runtime =
Python.CreateRuntime();</p>
</div>
<div>
<p> ScriptEngine engine =
Python.CreateEngine();</p>
</div>
<div>
<p> ScriptSource source =
engine.CreateScriptSourceFromFile("interpolate.py");</p>
</div>
<div>
<p>
Console.WriteLine(DateTime.Now);</p>
</div>
<div>
<p> CompiledCode compiled =
source.Compile();</p>
</div>
<div>
<p> compiled.Execute();</p>
</div>
<div>
<p> ScriptScope scope =
compiled.DefaultScope;</p>
</div>
<div>
<p> Microsoft.Func<string,
string, string, string, int, int> func =
scope.GetVariable<Microsoft.Func<string, string, string, string, int,
int>>("interpolate_start");</p>
</div>
<div>
<p> Console.WriteLine(func("3-day-data.txt",
"3-day-output.txt", "2008-01-01 00:00:21","2008-01-03
23:59:08",60).ToString());</p>
</div>
<div>
<p>
Console.WriteLine(DateTime.Now);</p>
</div>
<div>
<p> Console.ReadLine();</p>
</div>
<div>
<p> }</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>This is the latest output :</p>
</div>
<div>
<div>
<p> 11/12/2008 3:13:01 PM</p>
</div>
<div>
<p> WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior
to time of first</p>
</div>
<div>
<p> data point (Tue Jan 01 00:00:32 2008), current line: 9411</p>
</div>
<div>
<p> series: 3D</p>
</div>
<div>
<p> force==False, exiting..</p>
</div>
<div>
<p> 0</p>
</div>
<div>
<p>11/12/2008 3:16:41 PM</p>
</div>
</div>
<div>
<p> </p>
</div>
<div>
<p>Is there could be a problem on how I am calling the interpolation function?
I mean from python.exe I do something like this:</p>
</div>
<div>
<div>
<p> </p>
</div>
<div>
<p>if __name__ == '__main__':</p>
</div>
<div>
<p> Interpolate('3-day-data.txt', '3-day-output-original.txt',
'2008-01-01 00:00:</p>
</div>
<div>
<p>21', '2008-01-03 23:59:08', intvl=60, maxgap = 5, stacked = True, stackedCol
= 2</p>
</div>
<div>
<p>, tformat = '%Y-%m-%d %H:%M:%S', debug = False).start()</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>To be able to call the script from the DLR I have something like this:</p>
</div>
<div>
<p> </p>
</div>
<div>
<div>
<p>def interpolate_start(inFile, outFile, startDate, endDate, intvl):</p>
</div>
<div>
<p> Interpolate(inFile, outFile,startDate,endDate,intvl, maxgap =
5, stacked = Tr</p>
</div>
<div>
<p>ue, stackedCol = 2, tformat = '%Y-%m-%d %H:%M:%S', debug = False).start()</p>
</div>
<div>
<p> return 0</p>
</div>
</div>
</div>
<div>
<p> </p>
</div>
<div>
<p>That's the only difference I found on how the script is called from python
and DLR.</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>Any comments?</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>regards,</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>wilfredo</p>
</div>
<p> </p>
<div>
<p>On Wed, Nov 12, 2008 at 2:56 PM, Dino Viehland <<a href="mailto:dinov@microsoft.com" target="_blank">dinov@microsoft.com</a>>
wrote:</p>
<div>
<div>
<p><span style="font-size:11.0pt;color:#1F497D">CompiledCode exposes the
default scope it executes in via the DefaultScope property. So hold onto
the result of source.Compile and after executing the code you can grab the
scope and use it to get your variable.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> <a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>
[mailto:<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>]
<b>On Behalf Of </b>Wilfredo Lugo<br>
<b>Sent:</b> Wednesday, November 12, 2008 10:25 AM<br>
<b>To:</b> Discussion of IronPython</span></p>
<div>
<div>
<p><span style="font-size:10.0pt"><br>
<b>Subject:</b> Re: [IronPython] Performance Issue</span></p>
</div>
</div>
</div>
<div>
<div>
<p> </p>
<p>Thanks.</p>
<div>
<p> </p>
</div>
<div>
<p>But then If I don't use my defined scope, how I could fill my function
delegate?</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>Right now I am using : </p>
</div>
<div>
<p> </p>
</div>
<div>
<p>Microsoft.Func<string, string, string, string, int, int> func =
scope.GetVariable<Microsoft.Func<string, string, string, string, int,
int>>("interpolate_start");</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>If I use engine.GetVariables<Func<....>() I still need to pass a
ScriptScope to it. How do I get the default scope (from ScriptEngine or
ScriptSource) to be able to get the delegate?</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>regards,</p>
</div>
<div>
<p> </p>
</div>
<div>
<p style="margin-bottom:12.0pt">wilfredo</p>
<div>
<p>On Wed, Nov 12, 2008 at 2:01 PM, Dino Viehland <<a href="mailto:dinov@microsoft.com" target="_blank">dinov@microsoft.com</a>>
wrote:</p>
<div>
<div>
<p><span style="font-size:11.0pt;color:#1F497D">Instead of doing:</span></p>
<div>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p> ScriptScope scope =
engine.CreateScope();</p>
<p>
Console.WriteLine(DateTime.Now);</p>
<p> source.Execute(scope);</p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
</div>
<p><span style="font-size:11.0pt;color:#1F497D">do:</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p><span style="font-size:11.0pt;color:#1F497D">
source.Compile().Execute()</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<p><span style="font-size:11.0pt;color:#1F497D">and your code should then run
in an optimized default scope for the code – just like it does at the command
line.</span></p>
<p><span style="font-size:11.0pt;color:#1F497D"> </span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p><b><span style="font-size:10.0pt">From:</span></b><span style="font-size:10.0pt"> <a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>
[mailto:<a href="mailto:users-bounces@lists.ironpython.com" target="_blank">users-bounces@lists.ironpython.com</a>]
<b>On Behalf Of </b>Wilfredo Lugo<br>
<b>Sent:</b> Wednesday, November 12, 2008 9:29 AM<br>
<b>To:</b> <a href="mailto:users@lists.ironpython.com" target="_blank">users@lists.ironpython.com</a><br>
<b>Subject:</b> Re: [IronPython] Performance Issue</span></p>
</div>
<div>
<div>
<p> </p>
<p> </p>
<div>
<p>I run the script directly from ipy.exe and it basically behaves pretty similar
to python.exe (it always took one second more, but I could live with that).
Here is the output: </p>
</div>
<div>
<div>
<p> </p>
</div>
<div>
<p>$ date +%M:%S;./ipy.exe interpolate.py;date +%M:%S</p>
</div>
<div>
<p>17:36</p>
</div>
<div>
<p>WARNING: desired starting time (Tue Jan 01 00:00:21 2008) prior to time of
first</p>
</div>
<div>
<p> data point (Tue Jan 01 00:00:32 2008), current line: 9411</p>
</div>
<div>
<p> series: 3D</p>
</div>
<div>
<p>force==False, exiting..</p>
</div>
<div>
<p>17:41</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>So, the problem is on the DLR side. Any clues?</p>
</div>
<div>
<p> </p>
</div>
<div>
<p>-------------------------------------------------------------------------</p>
</div>
<div>
<p>How does python.exe compare directly to ipy.exe? That is, try running
"date +%M:%S;ipy.exe interpolate.py;date +%M:%S" instead of using the
DLR hosting APIs.</p>
</div>
<div><pre>Thanks,</pre><pre> </pre><pre>Dave</pre><pre>From: <a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">users-bounces at lists.ironpython.com</a> [mailto:<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">users-bounces at lists.ironpython.com</a>] On Behalf Of Wilfredo Lugo</pre>
<pre>Sent: Wednesday, November 12, 2008 8:32 AM</pre><pre>To: <a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">users at lists.ironpython.com</a></pre><pre>Subject: [IronPython] Performance Issue</pre>
</div>
</div>
</div>
</div>
</div>
</div>
<p style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.ironpython.com" target="_blank">Users@lists.ironpython.com</a><br>
<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">http://lists.ironpython.com/listinfo.cgi/users-ironpython.com</a></p>
</div>
<p> </p>
</div>
</div>
</div>
</div>
</div>
<p style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.ironpython.com" target="_blank">Users@lists.ironpython.com</a><br>
<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">http://lists.ironpython.com/listinfo.cgi/users-ironpython.com</a></p>
</div>
<p> </p>
</div>
</div>
</div>
</div>
</div>
<p style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.ironpython.com" target="_blank">Users@lists.ironpython.com</a><br>
<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">http://lists.ironpython.com/listinfo.cgi/users-ironpython.com</a></p>
</div>
<p> </p>
</div>
</div>
</div>
</div>
</div>
</div>
<p style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.ironpython.com" target="_blank">Users@lists.ironpython.com</a><br>
<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">http://lists.ironpython.com/listinfo.cgi/users-ironpython.com</a></p>
</div>
<p> </p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.ironpython.com" target="_blank">Users@lists.ironpython.com</a><br>
<a href="http://lists.ironpython.com/listinfo.cgi/users-ironpython.com" target="_blank">http://lists.ironpython.com/listinfo.cgi/users-ironpython.com</a><br>
<br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>