[IronPython] change in standard library behavior for compiled .exe/.dll???

[Ken MacDonald]

Hi Dino,
Here's a portion of the the os$31 output from the .dll created with IP 2.6,
as dumped by ildasm (that's a cool tool!). The .dll we get with IP 2.7
(without explicitly adding in the std lib references on the compile command
line) does NOT contain os$#.anything....
Ken


.method public specialname static object
        os$31(class [IronPython]IronPython.Runtime.CodeContext
$globalContext,
              class [IronPython]IronPython.Runtime.FunctionCode
functionCode) cil managed
{
  .custom instance void
[Microsoft.Dynamic]Microsoft.Scripting.Runtime.CachedOptimizedCodeAttribute::.ctor(string[])
= ( 01 00 69 00 00 00 08 5F 5F 6E 61 6D 65 5F 5F 08   // ..i....__name__.

5F 5F 66 69 6C 65 5F 5F 07 5F 5F 64 6F 63 5F 5F   // __file__.__doc__

08 5F 5F 70 61 74 68 5F 5F 0C 5F 5F 62 75 69 6C   // .__path__.__buil

74 69 6E 73 5F 5F 0B 5F 5F 70 61 63 6B 61 67 65   // tins__.__package

5F 5F 03 73 79 73 05 65 72 72 6E 6F 06 5F 6E 61   // __.sys.errno._na

6D 65 73 07 5F 5F 61 6C 6C 5F 5F 11 5F 67 65 74   // mes.__all__._get

5F 65 78 70 6F 72 74 73 5F 6C 69 73 74 04 6E 61   // _exports_list.na

6D 65 07 6C 69 6E 65 73 65 70 05 5F 65 78 69 74   // me.linesep._exit

04 70 61 74 68 05 70 6F 73 69 78 02 6E 74 04 6C   // .path.posix.nt.l

69 6E 6B 03 6F 73 32 02 63 65 06 72 69 73 63 6F   // ink.os2.ce.risco

73 06 63 75 72 64 69 72 06 70 61 72 64 69 72 03   // s.curdir.pardir.

73 65 70 07 70 61 74 68 73 65 70 07 64 65 66 70   // sep.pathsep.defp

61 74 68 06 65 78 74 73 65 70 06 61 6C 74 73 65   // ath.extsep.altse

70 07 64 65 76 6E 75 6C 6C 08 53 45 45 4B 5F 53   // p.devnull.SEEK_S

45 54 08 53 45 45 4B 5F 43 55 52 08 53 45 45 4B   // ET.SEEK_CUR.SEEK

5F 45 4E 44 08 6D 61 6B 65 64 69 72 73 0A 72 65   // _END.makedirs.re

6D 6F 76 65 64 69 72 73 07 72 65 6E 61 6D 65 73   // movedirs.renames

04 77 61 6C 6B 07 65 6E 76 69 72 6F 6E 05 65 78   // .walk.environ.ex

65 63 6C 06 65 78 65 63 6C 65 06 65 78 65 63 6C   // ecl.execle.execl

70 07 65 78 65 63 6C 70 65 06 65 78 65 63 76 70   // p.execlpe.execvp

07 65 78 65 63 76 70 65 08 5F 65 78 65 63 76 70   // .execvpe._execvp

65 08 55 73 65 72 44 69 63 74 08 75 6E 73 65 74   // e.UserDict.unset

65 6E 76 08 5F 45 6E 76 69 72 6F 6E 06 67 65 74   // env._Environ.get

65 6E 76 07 5F 65 78 69 73 74 73 06 50 5F 57 41   // env._exists.P_WA

49 54 08 50 5F 4E 4F 57 41 49 54 09 50 5F 4E 4F   // IT.P_NOWAIT.P_NO

57 41 49 54 4F 09 5F 73 70 61 77 6E 76 65 66 06   // WAITO._spawnvef.

73 70 61 77 6E 76 07 73 70 61 77 6E 76 65 07 73   // spawnv.spawnve.s

70 61 77 6E 76 70 08 73 70 61 77 6E 76 70 65 06   // pawnvp.spawnvpe.

73 70 61 77 6E 6C 07 73 70 61 77 6E 6C 65 07 73   // spawnl.spawnle.s

70 61 77 6E 6C 70 08 73 70 61 77 6E 6C 70 65 06   // pawnlp.spawnlpe.

70 6F 70 65 6E 32 06 70 6F 70 65 6E 33 06 70 6F   // popen2.popen3.po

70 65 6E 34 09 5F 63 6F 70 79 5F 72 65 67 11 5F   // pen4._copy_reg._

6D 61 6B 65 5F 73 74 61 74 5F 72 65 73 75 6C 74   // make_stat_result

13 5F 70 69 63 6B 6C 65 5F 73 74 61 74 5F 72 65   // ._pickle_stat_re

73 75 6C 74 14 5F 6D 61 6B 65 5F 73 74 61 74 76   // sult._make_statv

66 73 5F 72 65 73 75 6C 74 16 5F 70 69 63 6B 6C   // fs_result._pickl

65 5F 73 74 61 74 76 66 73 5F 72 65 73 75 6C 74   // e_statvfs_result

07 75 72 61 6E 64 6F 6D 04 6C 69 73 74 0E 41 74   // .urandom.list.At

74 72 69 62 75 74 65 45 72 72 6F 72 03 64 69 72   // tributeError.dir

07 4F 53 45 72 72 6F 72 05 6D 6B 64 69 72 05 72   // .OSError.mkdir.r

6D 64 69 72 05 65 72 72 6F 72 06 72 65 6E 61 6D   // mdir.error.renam

65 07 6C 69 73 74 64 69 72 05 65 78 65 63 76 06   // e.listdir.execv.

65 78 65 63 76 65 06 70 75 74 65 6E 76 04 64 69   // execve.putenv.di

63 74 04 65 76 61 6C 04 54 72 75 65 09 4E 61 6D   // ct.eval.True.Nam

65 45 72 72 6F 72 05 46 61 6C 73 65 04 66 6F 72   // eError.False.for

6B 07 77 61 69 74 70 69 64 0A 57 49 46 53 54 4F   // k.waitpid.WIFSTO

50 50 45 44 0B 57 49 46 53 49 47 4E 41 4C 45 44   // PPED.WIFSIGNALED

08 57 54 45 52 4D 53 49 47 09 57 49 46 45 58 49   // .WTERMSIG.WIFEXI

54 45 44 0B 57 45 58 49 54 53 54 41 54 55 53 12   // TED.WEXITSTATUS.

44 65 70 72 65 63 61 74 69 6F 6E 57 61 72 6E 69   // DeprecationWarni

6E 67 0B 73 74 61 74 5F 72 65 73 75 6C 74 0E 73   // ng.stat_result.s

74 61 74 76 66 73 5F 72 65 73 75 6C 74 04 6F 70   // tatvfs_result.op

65 6E 08 4F 5F 52 44 4F 4E 4C 59 07 49 4F 45 72   // en.O_RDONLY.IOEr

72 6F 72 13 4E 6F 74 49 6D 70 6C 65 6D 65 6E 74   // ror.NotImplement

65 64 45 72 72 6F 72 03 6C 65 6E 04 72 65 61 64   // edError.len.read

05 63 6C 6F 73 65 0B 49 6D 70 6F 72 74 45 72 72   // .close.ImportErr

6F 72 00 00 )                                     // or..
  // Code size       27367 (0x6ae7)
  .maxstack  425
  .locals init (object[] V_0,
           class [IronPython]IronPython.Compiler.PythonGlobal[] V_1,
           class
[Microsoft.Scripting.Core]System.Runtime.CompilerServices.StrongBox`1<object[]>
V_2,
           object[] V_3,

On Fri, Oct 8, 2010 at 10:28 AM, Ken MacDonald <drken567 at gmail.com> wrote:

> Well, just was looking at the pyc.py again; the change you suggest is
> actually in the GenerateExe method, so should have no effect on what is
> contained in the .dll.
>
> The .dll is actually generated by this code:
>
>     print "Output:\n\t%s" % output
>     print "Target:\n\t%s" % target
>     print 'Platform:\n\t%s' % platform
>     print 'Machine:\n\t%s' % machine
>
>     print 'Compiling...'
>     clr.CompileModules(output + '.dll', mainModule = main_name, *files)
>
> so I'm getting the feeling that something changed in the CompileModules
> method between 2.6 and 2.7 such that it's no longer tracking down and
> including the std lib references.
>  Ken
>
>
> On Fri, Oct 8, 2010 at 10:08 AM, Ken MacDonald <drken567 at gmail.com> wrote:
>
>> HI Dino,
>> Tried your change, it came up with an error on the statement below:
>> TypeError: EmitCall() takes exactly 3 arguments (1 given)
>>
>> Not sure what the other args should be... it did seem to create a new
>> myapp.dll despite the error, but it was the same size (1.2 MB) as the new
>> version 2.7 dll, and was still missing std lib components. I'm not sure that
>> this is valid, considering the error in the EmitCall() noted above.
>>
>> I did do a primitive analysis of the app to find out which std lib
>> components it references; found roughly 25 of them, and as a test,
>> incorporated the largest 10 or so explicitly in the compile; the size of the
>> resulting dll was about 2.1 MB (much closer to the 2.6 dll version size, 2.9
>> MB), and was able to run it successfullyagainst a stripped-down std lib
>> where I had removed those 10 files. It seems fairly clear to me that the 2.6
>> version dll did include the components it required from the std lib.
>>
>> Starting to look really promising; not quite there yet!
>> Ken
>>
>>
>> On Thu, Oct 7, 2010 at 8:06 PM, Dino Viehland <dinov at microsoft.com>wrote:
>>
>>>  I think these changes came about due to this thread:
>>> http://www.mail-archive.com/users@lists.ironpython.com/msg08794.htmlwhere there was an issue w/ relative paths and starting an app.
>>>
>>>
>>>
>>> And you may have found the root of the problem but you didn’t quote the
>>> code change.  There’s these additional lines:
>>>
>>>
>>>
>>>    gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(System.IO.DirectoryInfo).GetMethod("get_FullName"), ())
>>>    gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(System.Environment).GetMethod("set_CurrentDirectory"), ())
>>>
>>>
>>>
>>> which might be causing the problem as we change the CWD before we really
>>> kick things off.  Does replacing the set_CurrentDirectory line in pyc.py
>>> with:
>>>
>>>
>>>
>>>    gen.EmitCall(OpCodes.Pop)
>>>
>>>
>>>
>>> possibly fix things for you (that’ll make that a NOP but should leave all
>>> the other changes in place)?
>>>
>>>
>>>
>>> *From:* users-bounces at lists.ironpython.com [mailto:
>>> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald
>>> *Sent:* Wednesday, October 06, 2010 1:26 PM
>>> *To:* Discussion of IronPython
>>>
>>> *Subject:* Re: [IronPython] change in standard library behavior for
>>> compiled .exe/.dll???
>>>
>>>
>>>
>>> As an FYI, the following change was made from the IP 2.6 version of
>>> pyc.py to the IP 2.7 version (newer version shown first):
>>>
>>> <     # get the ScriptCode assembly...
>>> <     gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ())
>>> <     gen.EmitCall(OpCodes.Callvirt,
>>> clr.GetClrType(Assembly).GetMethod("get_Location"), ())
>>> ---
>>> >     # get the ScriptCode assembly...
>>> >     gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(Assembly).GetMethod("GetEntryAssembly"), ());
>>> >     gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(Assembly).GetMethod("get_CodeBase"), ());
>>> >     gen.Emit(OpCodes.Newobj, clr.GetClrType(System.Uri).GetConstructor(
>>> (str, ) ));
>>> >     gen.EmitCall(OpCodes.Call,
>>> clr.GetClrType(System.Uri).GetMethod("get_LocalPath"), ());
>>>
>>> Don't know what these things do at this point, but wondering if the
>>> changes have to do with compiling the entire code tree into the application
>>> .DLL???
>>> Ken
>>>
>>> On Wed, Oct 6, 2010 at 4:04 PM, Ken MacDonald <drken567 at gmail.com>
>>> wrote:
>>>
>>> Hi Dino,
>>>
>>> On Wed, Oct 6, 2010 at 2:57 PM, Dino Viehland <dinov at microsoft.com>
>>> wrote:
>>>
>>> How are you distributing your app?  I’m assuming you’re going to have
>>> something like:
>>>
>>>
>>>
>>> MyApp\
>>>
>>> MyApp.exe
>>>
>>> MyApp.dll
>>>
>>> IronPython.dll
>>>
>>> IronPython.Modules.DLL
>>>
>>> …
>>>
>>>
>>>
>>> This is exactly how we have been distributing our app up to IP 2.6 with
>>> .NET 3.5. We also have another DLL with our resources in it; XAML, icons,
>>> image files, but no code per se. We have been distributing it this way to
>>> customer systems that do NOT have IronPython installed, or any sign of the
>>> std. library, or "os.py" specifically, and it's been working really, really
>>> well.
>>>
>>> We're trying to understand what changed moving to IP 2.7 and .NET 4.0
>>> that we should have to care about distributing the std. library now. The way
>>> it was before was quite simple and robust; deliver a small handful of files
>>> and it just worked, very easy to keep track of. Now instead of distributing
>>> 5 files, we suddenly need to distribute 500??? We've figured out that it
>>> certainly COULD work as described below, but it seems like a giant step
>>> backwards on several fronts, including the potential for folks to
>>> maliciously or accidentally tamper with the std lib. sources and affect the
>>> functioning of our app. So, how do we get back to the old/better
>>> functionality?
>>>
>>> On a *slightly *related note, our app imports some package directories
>>> in addition, e.g. "import ctypes". When python encounters a directory
>>> import, it looks for __init__.py in the directory, and derives the package
>>> import directions from there, as I understand it. However, I can't specify
>>> the "ctypes" directory as an argument to the pyc.py compile app; just causes
>>> it to croak. If I explicitly specified paths like "lib\ctypes\__init__.py"
>>> and the other files in the ctypes subdirectory, it seems like "import
>>> ctypes" would have no clue that the __init__.py that was compiled in had
>>> anything to do with the "ctypes" package, as the path names are presumably
>>> irrelevant to the compiler as long as they specify a python file. I'm
>>> considering mod'ing pyc.py to be able to incorporate a list of std lib
>>> modules to compile in: simple enough for the standalone files like "os.py",
>>> but the compile modules don't seem to be able to grok what to do with a
>>> package subdirectory.
>>> Ken
>>>
>>>
>>>
>>>
>>>
>>>
>>> You should be able to also distribute the standard library and just drop
>>> it into a Lib directory next to IronPython.dll:
>>>
>>> MyApp\
>>>
>>> MyApp.exe
>>>
>>> MyApp.dll
>>>
>>> IronPython.dll
>>>
>>> IronPython.Modules.DLL
>>>
>>> …
>>>
>>>                 Lib\
>>>
>>>                                 os.py
>>>
>>>
>>>
>>> That lib dir should be on sys.path at startup and so it should be
>>> available for importing.
>>>
>>>
>>>
>>> *From:* users-bounces at lists.ironpython.com [mailto:
>>> users-bounces at lists.ironpython.com] *On Behalf Of *Ken MacDonald
>>> *Sent:* Wednesday, October 06, 2010 11:42 AM
>>> *To:* Michael Foord
>>>
>>>
>>> *Cc:* Discussion of IronPython
>>> *Subject:* Re: [IronPython] change in standard library behavior for
>>> compiled .exe/.dll???
>>>
>>>
>>>
>>> Hi Michael,
>>>
>>>
>>> I started out on implementing this, but I am importing maybe a dozen of
>>> the std. library modules, which then import others, and so on. It appears
>>> that eventually, most of the std modules would have to be imported
>>> explicitly (perhaps 400 or so files) which might make for a somewhat
>>> cumbersome command line, incidentally also about 20K characters too long
>>> :-). I'm hoping to find a way to get this to work as well as it did under IP
>>> 2.5 / .NET 3.5.
>>>
>>> Noah: what kind of problems are YOU having with pyc.py under 4.0? Maybe
>>> one of us can suggest something if we have an understanding of what you're
>>> trying to do.
>>> Ken
>>>
>>> On Wed, Oct 6, 2010 at 7:08 AM, Michael Foord <michael at voidspace.org.uk>
>>> wrote:
>>>
>>> On 05/10/2010 22:27, Ken MacDonald wrote:
>>>
>>> I've been looking at the .exe's we built - using pyc.py - with IP
>>> 2.5/.Net 3.5 and IP2.7 / .NET 4.0. In the 2.7 .exe, it appears that the
>>> imports (like "os") are not being built into the .exe/.dll, but instead are
>>> required to be imported in source form, e.g. "os.py" must be somewhere on
>>> sys.path. In the IP 2.5 .exe's we had been building, they would run fine on
>>> machines without the IP standard library installed at all, in other words,
>>> with "os.py" not present on the machine at all. We did notice that the .exe
>>> in question went from being 2.9 MB in it's IP 2.5 incarnation, down to 1.2
>>> MB in the IP 2.7 version, and the newer version requires that the source
>>> code for the IP standard library be on the path. Is this a deliberate change
>>> in behavior? We never had to package the standard library source when we
>>> sent out .exe's to customers before
>>>
>>>
>>>
>>> Hmmm... I'm pretty sure I always had to explicitly compile and bundle the
>>> standard library with previous versions of Python. Odd. Anyway, the simple
>>> solution is to ensure that you add any standard library modules you use to
>>> the set you compile and ship.
>>>
>>>
>>>
>>> All the best,
>>>
>>> Michael Foord
>>>
>>>
>>>
>>> >"os" is not an assembly but a Python module from the standard library.
>>> You need to ensure >that the Python standard library (or the parts that you
>>> use and their dependencies) is on the >path.
>>>
>>>
>>> All the best,
>>>
>>> Michael Foord
>>>
>>>  and how do I ensure it gets found from my .exe - is there a specific
>>> env. variable, or the Windows %PATH% e.v., or something I haven't
>>> AddReference'd to????
>>> Thanks,
>>> Ken
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> Users mailing list
>>>
>>>  Users at lists.ironpython.com
>>>
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>>>
>>>
>>>  --
>>>
>>> http://www.voidspace.org.uk/blog
>>>
>>>
>>>
>>> READ CAREFULLY. By accepting and reading this email you agree,
>>>
>>> on behalf of your employer, to release me from all obligations
>>>
>>> and waivers arising from any and all NON-NEGOTIATED agreements,
>>>
>>> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
>>>
>>> confidentiality, non-disclosure, non-compete and acceptable use
>>>
>>> policies (”BOGUS AGREEMENTS”) that I have entered into with your
>>>
>>> employer, its partners, licensors, agents and assigns, in
>>>
>>> perpetuity, without prejudice to my ongoing rights and privileges.
>>>
>>> You further represent that you have the authority to release me
>>>
>>> from any BOGUS AGREEMENTS on behalf of your employer.
>>>
>>>
>>>
>>>
>>>
>>>  --
>>>
>>> http://www.voidspace.org.uk/blog
>>>
>>>
>>>
>>> READ CAREFULLY. By accepting and reading this email you agree,
>>>
>>> on behalf of your employer, to release me from all obligations
>>>
>>> and waivers arising from any and all NON-NEGOTIATED agreements,
>>>
>>> licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap,
>>>
>>> confidentiality, non-disclosure, non-compete and acceptable use
>>>
>>> policies (”BOGUS AGREEMENTS”) that I have entered into with your
>>>
>>> employer, its partners, licensors, agents and assigns, in
>>>
>>> perpetuity, without prejudice to my ongoing rights and privileges.
>>>
>>> You further represent that you have the authority to release me
>>>
>>> from any BOGUS AGREEMENTS on behalf of your employer.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.ironpython.com
>>> http://lists.ironpython.com/listinfo.cgi/users-ironpython.com
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ironpython-users/attachments/20101008/36e438b1/attachment.html>