Parallel pyc construction
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time. Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
-----Original Message----- From: Paul F Dubois
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time.
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
I hadn't considered the option of not making .pyc files, though I've complained about .pyo files in the past with distutils, but now compilation is optional there. The .pyc and .pyo files certainly clutter a directory. If there is no significant performance improvement for loading and using .pyc files or the difference is only significant for large files or certain code constructs, maybe they shouldn't be automatically created. I guess this is another area for test cases. ka
Kevin Altis wrote:
I hadn't considered the option of not making .pyc files, though I've complained about .pyo files in the past with distutils, but now compilation is optional there. The .pyc and .pyo files certainly clutter a directory. If there is no significant performance improvement for loading and using .pyc files or the difference is only significant for large files or certain code constructs, maybe they shouldn't be automatically created. I guess this is another area for test cases.
There *is* a significant performance improvement (otherwise I doubt we'd have .pyc files in the first place ;-), but it only improves startup time. So it can make a big difference for short running processes, yet can be totally irrelevant for long running processes. Just
I forgot to say that the file clutter wouldn't be a problem if _pyc and _pyo sub-directories (or .pyc and .pyo to hide them in Unix) were automatically created and the files stuck in there, but I'm sure that would end up screwing up something else that relies on the .pyc and .pyo files being in the same directory as the .py or .pyw files. ka
-----Original Message----- From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On Behalf Of Kevin Altis Sent: Friday, January 10, 2003 9:55 AM To: python-dev@python.org Subject: RE: [Python-Dev] Parallel pyc construction
-----Original Message----- From: Paul F Dubois
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time.
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
I hadn't considered the option of not making .pyc files, though I've complained about .pyo files in the past with distutils, but now compilation is optional there. The .pyc and .pyo files certainly clutter a directory. If there is no significant performance improvement for loading and using .pyc files or the difference is only significant for large files or certain code constructs, maybe they shouldn't be automatically created. I guess this is another area for test cases.
ka
I hadn't considered the option of not making .pyc files, though I've complained about .pyo files in the past with distutils, but now compilation is optional there. The .pyc and .pyo files certainly clutter a directory. If there is no significant performance improvement for loading and using .pyc files or the difference is only significant for large files or certain code constructs, maybe they shouldn't be automatically created. I guess this is another area for test cases.
Oh, in most cases .pyc/.pyo files *do* give significant speedup; the parser + bytecode compiler are really slow. It's just that Paul's machine is so fast and his program runs so long that he doesn't care. But many people do. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, Jan 10, 2003, Paul F Dubois wrote:
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
Why not make the .py directory read-only? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ
Why not make the .py directory read-only?
Excellent suggestion for a work-around. --Guido van Rossum (home page: http://www.python.org/~guido/)
Thanks to everyone who has tried to help me with this problem. I will try to make a command line option for this. The .py files in question belong to the user and I don't have any control over where they are; and I don't know about them ahead of time so I cannot precompile them. The user wrote the files so all I know is that someone is trying to import something. Each of the hundreds or thousands of Pythons reads the same Python program. However, since we are using a parallel processor and the problems will run for minutes if not months, the cost of any imports does not matter. It is interesting that the other set of people who care about this are doing little embedded stuff, sort of the exact opposite end of the computing spectrum. Paul
-----Original Message----- From: guido@odiug.zope.com [mailto:guido@odiug.zope.com] On Behalf Of Guido van Rossum Sent: Friday, January 10, 2003 10:37 AM To: Aahz Cc: Paul F Dubois; python-dev@python.org Subject: Re: [Python-Dev] Parallel pyc construction
Why not make the .py directory read-only?
Excellent suggestion for a work-around.
--Guido van Rossum (home page: http://www.python.org/~guido/)
On Sun, Jan 12, 2003 at 10:10:25AM -0800, Paul F Dubois wrote:
Thanks to everyone who has tried to help me with this problem. I will try to make a command line option for this.
I have already uploaded a patch on the SF bug/feature request. Feel free to use it. Anybody want to review it? http://python.org/sf/602345 There is a patch for 2.3 as well as 2.2. Neal
On Fri, Jan 10, 2003 at 09:43:19AM -0800, Paul F Dubois wrote:
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time.
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
A while ago I fixed a problem when there were more than 64k items used to create a list. The fix went into 2.2.2 I believe. For 2.3 some sizes were increased from 2 to 4 bytes so the problem shouldn't occur. Here's the bug: http://python.org/sf/561858 There is a bug (aka feature request) assigned to me: http://python.org/sf/602345 option for not writing .py[co] files I haven't done anything with it yet. Feel free to submit a patch. What version of python had this problem? Can you make a test case? Neal
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time.
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
I don't think we have such an option, but it's a good idea. If you submit a patch, we'll add it. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido van Rossum]
On a 384 processor job we have once again encountered that old question of corrupted .pyc files, sometimes resulting in an error, sometimes in a silent wrong behavior later. I know this was allegedly fixed previously but it looks like it doesn't really work. We lost a couple of weeks work this time.
Didn't we talk about an option to not make pyc files? I can't seem to find it. (We totally don't care about the cost of imports. The documentation mentions "ihooks" but not the module itself. I know that importing has been an area of create turmoil so I don't really know where to look.) I couldn't even find the list of command-line options for Python itself except a mention of -c in the tutorial. Any pointers would be appreciated.
I don't think we have such an option, but it's a good idea. If you submit a patch, we'll add it.
What about PEP 301 and an import hook? Couldn't a custom import hook be written up that didn't output a .py file? I would think it could be as simple as finding the file, opening it, and then compiling it as a module and inserting it directly into ``sys.modules``. Wouldn't that circumvent any .py(c|o) writing? Of course this assumes Paul is using 2.3, but even if he isn't couldn't a solution be used like that, helping to prevent needing to write a patch for Python (unless you want this in the 2.2 branch)? -Brett
What about PEP 301 and an import hook? Couldn't a custom import hook be written up that didn't output a .py file? I would think it could be as simple as finding the file, opening it, and then compiling it as a module and inserting it directly into ``sys.modules``. Wouldn't that circumvent any .py(c|o) writing?
I think using an import hook to prevent writing .pyc files is way too much work. You can't use the built-in code because that *does* write the .pyc files, so you have to redo all the work that the standard hook does. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
What about PEP 301 and an import hook? Couldn't a custom import hook be written up that didn't output a .py file? I would think it could be as simple as finding the file, opening it, and then compiling it as a module and inserting it directly into ``sys.modules``. Wouldn't that circumvent any .py(c|o) writing?
I think using an import hook to prevent writing .pyc files is way too much work. You can't use the built-in code because that *does* write the .pyc files, so you have to redo all the work that the standard hook does.
But it's another fine use case for a to-be-written PEP-302-compliant ihooks.py replacement. I'll make a mental note. Just
participants (8)
-
Aahz -
Brett Cannon -
Guido van Rossum -
Just van Rossum -
Kevin Altis -
Neal Norwitz -
Paul F Dubois -
Tim Peters