Python as a Metro-style App
I just tried porting Python as a Metro (Windows 8) App, and failed. Metro Apps use a variant of the Windows API called WinRT that still allows to write native applications in C++, but restricts various APIs to a subset of the full Win32 functionality. For example, everything related to subprocess creation would not work; none of the byte-oriented file API seems to be present, and a number of file operation functions are absent as well (such as MoveFile). Regardless, porting Python ought to be feasible, except that it fails fundamentally with the preview release of Visual Studio. The problem is that compilation of C code is apparently not supported/tested in that preview release. When compiling a trivial C file in a Metro app, the compiler complains that a temporary file ending with "md" could not be found, most likely because the C compiler failed to generate it, whereas the C++ compiler would. I tried compiling the Python sources as C++, but that produced hundreds of compilation errors. Most of them are either about missing casts (e.g. from int to enum types, or from void * to other pointer types), or about the "static forward" declarations of type objects. For the latter, anonymous namespaces should be used. While it is feasible to replace static PyTypeObject foo; ... static PyTypeObject foo = { ... }; with Py_BEGIN_STATIC PyTypeObject foo; Py_END_STATIC ... Py_BEGIN_STATIC PyTypeObject foo = { ... }; Py_END_STATIC I'm not sure whether such a change would be accepted, in particular as Microsoft might fix the bug in the compiler until the final release of Windows 8. Regards, Martin
2012/1/7 "Martin v. Löwis" <martin@v.loewis.de>:
I just tried porting Python as a Metro (Windows 8) App, and failed.
Is this required for Python to run on Windows 8? Sorry if that's a dumb question. I'm not sure if "Metro App" is a special class of application. -- Regards, Benjamin
Zitat von Benjamin Peterson <benjamin@python.org>:
No. Existing applications ("desktop applications") will continue to work unmodified. Metro-style apps are primarily intended for smart phones and tablet PCs, and will be distributed through the Windows app store. The current VS prerelease supports both Intel and ARM processors for Apps. A related question is whether Python will compile unmodified with Visual Studio 11. Although I had some difficulties with that also so far, I expect that this will ultimately work (although not unmodified - the project files need to be updated, as will the packaging process). A then-related question is whether Python 3.3 should be compiled with Visual Studio 11. I'd still be in favor of that, provided Microsoft manages to release that soon enough. Regards, Martin
On Sat, Jan 7, 2012 at 16:07, <martin@v.loewis.de> wrote:
I'm guessing the change would have to be done before the first beta? It would have to be released awfully soon, and I haven't heard an estimated release date as of yet. I currently have the default branch mostly ported to VS 2010 save for a number of failed tests, FWIW.
On 7 January 2012 22:56, Eli Bendersky <eliben@gmail.com> wrote:
I would assume that Express should work, but the python.org distributed binaries will use the full version (IIUC, the official distribution uses some optimisations not present in Express - Profile Guided Optimisation, I believe). Paul.
On Sat, Jan 7, 2012 at 18:04, Paul Moore <p.f.moore@gmail.com> wrote:
The bigger issue is how Express doesn't (officially) support x64 builds, unless that's changing in VS11. Perhaps this is better for another topic, but is anyone using the PGO stuff? I know we have PGInstrument and PGUpdate build configurations but I've never seen them mentioned anywhere.
Zitat von Eli Bendersky <eliben@gmail.com>:
*Here*, I mean "Visual Studio 11, any edition". I don't think the edition matters for determining what version the project files have - any edition will be able to read the project files, Express or not. If you are specifically asking whether I would make the release of the express edition a prerequisite to releasing Python: no, I wouldn't. I would expect that Microsoft releases the express edition along with or soon after the commercial editions, and the commercial edition is sufficient for running the Python release process. Regards, Martin
On 1/7/2012 4:47 PM, Benjamin Peterson wrote:
No, normal 'desktop' programs will still run in desktop mode.
Sorry if that's a dumb question. I'm not sure if "Metro App" is a special class of application.
Yes. They are basically 'phone/touchpad' apps, and will be managed in the more or less the same way. They will probably only be available through MS storefront, after vetting by MS. Only Metro Apps will survive a system Refresh, along with user data. Traditional unvetted, direct-from-supplier, desktops apps will be wiped because they might be 'bad'. -- Terry Jan Reedy
On Sat, 07 Jan 2012 18:57:41 +0100 "Martin v. Löwis" <martin@v.loewis.de> wrote:
When you say MoveFile is absent, is MoveFileEx supported instead? Or is moving files just totally impossible? Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
I would hope they finally support compiling C code... Regards Antoine.
Antoine Pitrou:
When you say MoveFile is absent, is MoveFileEx supported instead?
WinRT strongly prefers asynchronous methods for all lengthy operations. The most likely call to use for moving files is StorageFile.MoveAsync. http://msdn.microsoft.com/en-us/library/windows/apps/br227219.aspx
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Asynchronous APIs will become much more important on all platforms in the future to ensure responsive user interfaces. Python should not be left behind. Neil
Antoine Pitrou:
How does it translate to C?
The simplest technique would be to use C++ code to bridge from C to the API. If you really wanted to you could explicitly call the function pointer in the COM vtable but doing COM in C is more effort than calling through C++.
They are more important now due to the use of phones and tablets together with distant file systems. Neil
Zitat von Antoine Pitrou <solipsis@pitrou.net>:
Not sure whether you are asking literally for *C*: please remember that my original report said that C is apparently not currently supported for Apps. In any case, for native C++ code, do StorageFile ^the_file = something(); the_file->MoveAsync(destinationFolder, "newfile.txt"); This may look like managed C++ to you, but it really compiles into native code. Regards, Martin
When you say MoveFile is absent, is MoveFileEx supported instead? Or is moving files just totally impossible?
I can't check the SDK headers right now, but according to the online documentation, MoveFileExW is indeed available. I'm not sure whether you are allowed to pass arbitrary file names in an App, though.
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
I'm not so sure. Even if the low-level Win32 API was not available, you might still be able to do useful things with the higher-level APIs, such as Windows.Storage (in case of file access). If you use, say, Windows.Storage.ApplicationData.RoamingSettings in your app, you should not actually worry what the file is named on disk (or whether there is a spinning disk in the system at all, which probably isn't). Regards, Martin
On Sat, Jan 7, 2012 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Win 8 is practically a new OS target - the nt module may need to be replaced with a metro module to handle it well. Accessing the WinRT APIs directly from Python will also require a set of Python projections for the API, which should be straightforward to generate from the WinRT metadata files. I know Dino Viehland did some work on that; not sure if he can elaborate or not though. Otherwise, IronPython would be the only option for writing Metro apps in Python - not that I'd be *horribly* upset at that :). IronPython is slowly growing Metro support, and it seems like most things will work, but the .NET framework shields it from a lot of the WinRT guts. - Jeff
We spent some time investigating Python/Win8 projections but we don't really have anything else to say right now, but it is certainly possible. I haven't been following this thread so maybe this was already discussed, but on the whole "new OS target" thing - if people want to write immersive apps in Python then there will need to be a new build of Python. One thing that might make that easier is the fact that the C runtime is still available to metro apps, even if the C runtime calls a banned API. So to the extent that Python is just a C program the "port" should be pretty easy and mostly involve disabling functionality that isn't available at all to metro apps. I have packaged up Python 2.7 in an appx and run the application verifier on it (this was a while ago, so things may have changed between now and then), the attached banned.txt includes the list of APIs which Python is using that aren't allowed for the curious. Also, people who write apps will need to distribute Python w/ their app, there's currently no sharing between apps. -----Original Message----- From: Jeff Hardy [mailto:jdhardy@gmail.com] Sent: Sunday, January 08, 2012 10:13 PM To: Antoine Pitrou Cc: python-dev@python.org; Dino Viehland Subject: Re: [Python-Dev] Python as a Metro-style App On Sat, Jan 7, 2012 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Win 8 is practically a new OS target - the nt module may need to be replaced with a metro module to handle it well. Accessing the WinRT APIs directly from Python will also require a set of Python projections for the API, which should be straightforward to generate from the WinRT metadata files. I know Dino Viehland did some work on that; not sure if he can elaborate or not though. Otherwise, IronPython would be the only option for writing Metro apps in Python - not that I'd be *horribly* upset at that :). IronPython is slowly growing Metro support, and it seems like most things will work, but the .NET framework shields it from a lot of the WinRT guts. - Jeff
Does that hold for all versions of the C runtime (i.e. is msvcr80.dll also exempt from the ban, or just the version that comes with VS 11)?
See the start of the thread: I tried to create a "WinRT Component DLL", and that failed, as VS would refuse to compile any C file in such a project. Not sure whether this is triggered by defining WINAPI_FAMILY=2, or any other compiler setting. I'd really love to use WINAPI_FAMILY=2, as compiler errors are much easier to fix than verifier errors. Regards, Martin
Martin wrote:
Does that hold for all versions of the C runtime (i.e. is msvcr80.dll also exempt from the ban, or just the version that comes with VS 11)?
Just the VS 11 CRT is allowed.
Let me see if I can try this. Hopefully I still have my VM w/ this all setup and I can see if I can get it building this way. I can always ping some people on the C++ team and ask them for help if I run into issues. I'll give it a shot tomorrow and get back to you.
Hi Dino, I reported that as a bug. If you need that for reference, see https://connect.microsoft.com/VisualStudio/feedback/details/717395/c1083-whe... Regards, Martin
Martin wrote:
I got the same errors as you - it seems like they're related to enabling the Immersive bit for the compile of the DLL. I'm not certain if that's necessary, when I did the run before to see if Python would pass the app store validation it didn't care that we didn't have the App Container bit set on the DLL (it did want NXCOMPAT and dynamic base set though). I was also able to just define WINAPI_FAMILY=2 in the .vcxproj file and I got the various expected errors when accessing banned APIs (it actually seems like a bunch were missing vs. what the validator reported, but maybe that's just an issue w/ the developer preview). Once I fixed those errors up I was able to get a DLL that successfully compiled. I'm going to ping some people on the windows team and see if the app container bit is or will be necessary for DLLs.
Dino wrote:
I heard back from the Windows team and they are going to require the app container bit to be set on all PE files (although they don't currently enforce it). I was able to compile a simple .c file and pass /link /appcontainer and that worked, so I'm going to try and figure out if there's some way to get the .vcxproj to build a working command line that includes that.
Am 09.01.2012 07:13, schrieb Jeff Hardy:
No, it's not. Everything continues to work just fine on Windows 8, as long as we keep developing desktop apps. Only if Metro Apps are the target things may need to be replaced (but only very few changes are necessary to the nt module to make it compile). Regards, Martin
Am 07.01.2012 18:57, schrieb "Martin v. Löwis":
Just wondering, do Metro apps define UNDER_CE or _WIN32_WCE? The point is that the old ANSI functions (CreateFileA etc) have been removed from the embedded MS Windows CE long ago, too, and MS Windows Mobile used to be a custom CE variant or at least strongly related. In any case, it could help using the existing (incomplete) CE port as base for Metro. Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.dominolaser.com ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Domino Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************
I have now completed building Python as a Metro DLL; the WinRT restrictions are fairly limited (code-wise, not so in impact). They are quite different from the CE restrictions. For example, CreateSemaphore is not available on WinRT, you have to use CreateSemaphoreExW (which is new in Windows Vista). No traces of the CE API can be seen in the restrictions, and the separation is done in a different manner (WINAPI_FAMILY==2). Regards, Martin
2012/1/7 "Martin v. Löwis" <martin@v.loewis.de>:
I just tried porting Python as a Metro (Windows 8) App, and failed.
Is this required for Python to run on Windows 8? Sorry if that's a dumb question. I'm not sure if "Metro App" is a special class of application. -- Regards, Benjamin
Zitat von Benjamin Peterson <benjamin@python.org>:
No. Existing applications ("desktop applications") will continue to work unmodified. Metro-style apps are primarily intended for smart phones and tablet PCs, and will be distributed through the Windows app store. The current VS prerelease supports both Intel and ARM processors for Apps. A related question is whether Python will compile unmodified with Visual Studio 11. Although I had some difficulties with that also so far, I expect that this will ultimately work (although not unmodified - the project files need to be updated, as will the packaging process). A then-related question is whether Python 3.3 should be compiled with Visual Studio 11. I'd still be in favor of that, provided Microsoft manages to release that soon enough. Regards, Martin
On Sat, Jan 7, 2012 at 16:07, <martin@v.loewis.de> wrote:
I'm guessing the change would have to be done before the first beta? It would have to be released awfully soon, and I haven't heard an estimated release date as of yet. I currently have the default branch mostly ported to VS 2010 save for a number of failed tests, FWIW.
On 7 January 2012 22:56, Eli Bendersky <eliben@gmail.com> wrote:
I would assume that Express should work, but the python.org distributed binaries will use the full version (IIUC, the official distribution uses some optimisations not present in Express - Profile Guided Optimisation, I believe). Paul.
On Sat, Jan 7, 2012 at 18:04, Paul Moore <p.f.moore@gmail.com> wrote:
The bigger issue is how Express doesn't (officially) support x64 builds, unless that's changing in VS11. Perhaps this is better for another topic, but is anyone using the PGO stuff? I know we have PGInstrument and PGUpdate build configurations but I've never seen them mentioned anywhere.
Zitat von Eli Bendersky <eliben@gmail.com>:
*Here*, I mean "Visual Studio 11, any edition". I don't think the edition matters for determining what version the project files have - any edition will be able to read the project files, Express or not. If you are specifically asking whether I would make the release of the express edition a prerequisite to releasing Python: no, I wouldn't. I would expect that Microsoft releases the express edition along with or soon after the commercial editions, and the commercial edition is sufficient for running the Python release process. Regards, Martin
On 1/7/2012 4:47 PM, Benjamin Peterson wrote:
No, normal 'desktop' programs will still run in desktop mode.
Sorry if that's a dumb question. I'm not sure if "Metro App" is a special class of application.
Yes. They are basically 'phone/touchpad' apps, and will be managed in the more or less the same way. They will probably only be available through MS storefront, after vetting by MS. Only Metro Apps will survive a system Refresh, along with user data. Traditional unvetted, direct-from-supplier, desktops apps will be wiped because they might be 'bad'. -- Terry Jan Reedy
On Sat, 07 Jan 2012 18:57:41 +0100 "Martin v. Löwis" <martin@v.loewis.de> wrote:
When you say MoveFile is absent, is MoveFileEx supported instead? Or is moving files just totally impossible? Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
I would hope they finally support compiling C code... Regards Antoine.
Antoine Pitrou:
When you say MoveFile is absent, is MoveFileEx supported instead?
WinRT strongly prefers asynchronous methods for all lengthy operations. The most likely call to use for moving files is StorageFile.MoveAsync. http://msdn.microsoft.com/en-us/library/windows/apps/br227219.aspx
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Asynchronous APIs will become much more important on all platforms in the future to ensure responsive user interfaces. Python should not be left behind. Neil
Antoine Pitrou:
How does it translate to C?
The simplest technique would be to use C++ code to bridge from C to the API. If you really wanted to you could explicitly call the function pointer in the COM vtable but doing COM in C is more effort than calling through C++.
They are more important now due to the use of phones and tablets together with distant file systems. Neil
Zitat von Antoine Pitrou <solipsis@pitrou.net>:
Not sure whether you are asking literally for *C*: please remember that my original report said that C is apparently not currently supported for Apps. In any case, for native C++ code, do StorageFile ^the_file = something(); the_file->MoveAsync(destinationFolder, "newfile.txt"); This may look like managed C++ to you, but it really compiles into native code. Regards, Martin
When you say MoveFile is absent, is MoveFileEx supported instead? Or is moving files just totally impossible?
I can't check the SDK headers right now, but according to the online documentation, MoveFileExW is indeed available. I'm not sure whether you are allowed to pass arbitrary file names in an App, though.
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
I'm not so sure. Even if the low-level Win32 API was not available, you might still be able to do useful things with the higher-level APIs, such as Windows.Storage (in case of file access). If you use, say, Windows.Storage.ApplicationData.RoamingSettings in your app, you should not actually worry what the file is named on disk (or whether there is a spinning disk in the system at all, which probably isn't). Regards, Martin
On Sat, Jan 7, 2012 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Win 8 is practically a new OS target - the nt module may need to be replaced with a metro module to handle it well. Accessing the WinRT APIs directly from Python will also require a set of Python projections for the API, which should be straightforward to generate from the WinRT metadata files. I know Dino Viehland did some work on that; not sure if he can elaborate or not though. Otherwise, IronPython would be the only option for writing Metro apps in Python - not that I'd be *horribly* upset at that :). IronPython is slowly growing Metro support, and it seems like most things will work, but the .NET framework shields it from a lot of the WinRT guts. - Jeff
We spent some time investigating Python/Win8 projections but we don't really have anything else to say right now, but it is certainly possible. I haven't been following this thread so maybe this was already discussed, but on the whole "new OS target" thing - if people want to write immersive apps in Python then there will need to be a new build of Python. One thing that might make that easier is the fact that the C runtime is still available to metro apps, even if the C runtime calls a banned API. So to the extent that Python is just a C program the "port" should be pretty easy and mostly involve disabling functionality that isn't available at all to metro apps. I have packaged up Python 2.7 in an appx and run the application verifier on it (this was a while ago, so things may have changed between now and then), the attached banned.txt includes the list of APIs which Python is using that aren't allowed for the curious. Also, people who write apps will need to distribute Python w/ their app, there's currently no sharing between apps. -----Original Message----- From: Jeff Hardy [mailto:jdhardy@gmail.com] Sent: Sunday, January 08, 2012 10:13 PM To: Antoine Pitrou Cc: python-dev@python.org; Dino Viehland Subject: Re: [Python-Dev] Python as a Metro-style App On Sat, Jan 7, 2012 at 2:57 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Depending on the extent of removed/disabled functionality, it might not be very interesting to have a Metro port at all.
Win 8 is practically a new OS target - the nt module may need to be replaced with a metro module to handle it well. Accessing the WinRT APIs directly from Python will also require a set of Python projections for the API, which should be straightforward to generate from the WinRT metadata files. I know Dino Viehland did some work on that; not sure if he can elaborate or not though. Otherwise, IronPython would be the only option for writing Metro apps in Python - not that I'd be *horribly* upset at that :). IronPython is slowly growing Metro support, and it seems like most things will work, but the .NET framework shields it from a lot of the WinRT guts. - Jeff
Does that hold for all versions of the C runtime (i.e. is msvcr80.dll also exempt from the ban, or just the version that comes with VS 11)?
See the start of the thread: I tried to create a "WinRT Component DLL", and that failed, as VS would refuse to compile any C file in such a project. Not sure whether this is triggered by defining WINAPI_FAMILY=2, or any other compiler setting. I'd really love to use WINAPI_FAMILY=2, as compiler errors are much easier to fix than verifier errors. Regards, Martin
Martin wrote:
Does that hold for all versions of the C runtime (i.e. is msvcr80.dll also exempt from the ban, or just the version that comes with VS 11)?
Just the VS 11 CRT is allowed.
Let me see if I can try this. Hopefully I still have my VM w/ this all setup and I can see if I can get it building this way. I can always ping some people on the C++ team and ask them for help if I run into issues. I'll give it a shot tomorrow and get back to you.
Hi Dino, I reported that as a bug. If you need that for reference, see https://connect.microsoft.com/VisualStudio/feedback/details/717395/c1083-whe... Regards, Martin
Martin wrote:
I got the same errors as you - it seems like they're related to enabling the Immersive bit for the compile of the DLL. I'm not certain if that's necessary, when I did the run before to see if Python would pass the app store validation it didn't care that we didn't have the App Container bit set on the DLL (it did want NXCOMPAT and dynamic base set though). I was also able to just define WINAPI_FAMILY=2 in the .vcxproj file and I got the various expected errors when accessing banned APIs (it actually seems like a bunch were missing vs. what the validator reported, but maybe that's just an issue w/ the developer preview). Once I fixed those errors up I was able to get a DLL that successfully compiled. I'm going to ping some people on the windows team and see if the app container bit is or will be necessary for DLLs.
Dino wrote:
I heard back from the Windows team and they are going to require the app container bit to be set on all PE files (although they don't currently enforce it). I was able to compile a simple .c file and pass /link /appcontainer and that worked, so I'm going to try and figure out if there's some way to get the .vcxproj to build a working command line that includes that.
Am 09.01.2012 07:13, schrieb Jeff Hardy:
No, it's not. Everything continues to work just fine on Windows 8, as long as we keep developing desktop apps. Only if Metro Apps are the target things may need to be replaced (but only very few changes are necessary to the nt module to make it compile). Regards, Martin
Am 07.01.2012 18:57, schrieb "Martin v. Löwis":
Just wondering, do Metro apps define UNDER_CE or _WIN32_WCE? The point is that the old ANSI functions (CreateFileA etc) have been removed from the embedded MS Windows CE long ago, too, and MS Windows Mobile used to be a custom CE variant or at least strongly related. In any case, it could help using the existing (incomplete) CE port as base for Metro. Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.dominolaser.com ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Domino Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************
I have now completed building Python as a Metro DLL; the WinRT restrictions are fairly limited (code-wise, not so in impact). They are quite different from the CE restrictions. For example, CreateSemaphore is not available on WinRT, you have to use CreateSemaphoreExW (which is new in Windows Vista). No traces of the CE API can be seen in the restrictions, and the separation is done in a different manner (WINAPI_FAMILY==2). Regards, Martin
participants (13)
-
"Martin v. Löwis" -
Antoine Pitrou -
Benjamin Peterson -
Brian Curtin -
Dino Viehland -
Eli Bendersky -
Jeff Hardy -
martin＠v.loewis.de
-
Neil Hodgson -
Paul Moore -
Terry Reedy -
Ulrich Eckhardt -
Xavier Morel