Compiling the source without stat
Hi. I just started to port latest python 2.7.2 to another platform (don't think you're eager to know... well it's CASIO ClassPad). And I faced a "minor" problem... It hasn't got stat or fstat or anything. (It supports a very limited set of c std lib). As pyport.c suggested, i defined both DONT_HAVE_STAT and DONT_HAVE_FSTAT, but problems only began. It failed to compile most of import.c, particularly because it fell into the wrong `#if !defined(PYOS_Something)' blocks. Sometimes it just fell into an #else part which assumed stat are available. So although HAVE_STAT is meant to control file operations, most of the source code aren't implement to use it. You see how "minor" the problem was? So now I need advice from developers. Is there a fix for it? What a question... definitely no replacement to stat. It's 99% definite that I can't compile further without touching the source code. I have to #define my own PYOS_whatever and handle files in my own way. In that case where should my special file handling cases go? I saw some marshal.c code which seemed it wants to abstract away platform's file handling from source code; but from what I understood it can't be made to use alternate file handling methods. If there is anything I should do (maybe show you my handmade pyconfig.h?) tell me. [My first post in a mailing list... Should I say] Best Regards, Hossein [in here?]
Hossein wrote:
Hi. I just started to port latest python 2.7.2 to another platform (don't think you're eager to know... well it's CASIO ClassPad). And I faced a "minor" problem... It hasn't got stat or fstat or anything. (It supports a very limited set of c std lib). As pyport.c suggested, i defined both DONT_HAVE_STAT and DONT_HAVE_FSTAT, but problems only began. It failed to compile most of import.c, particularly because it fell into the wrong `#if !defined(PYOS_Something)' blocks. Sometimes it just fell into an #else part which assumed stat are available. So although HAVE_STAT is meant to control file operations, most of the source code aren't implement to use it. You see how "minor" the problem was? So now I need advice from developers. Is there a fix for it? What a question... definitely no replacement to stat. It's 99% definite that I can't compile further without touching the source code. I have to #define my own PYOS_whatever and handle files in my own way. In that case where should my special file handling cases go? I saw some marshal.c code which seemed it wants to abstract away platform's file handling from source code; but from what I understood it can't be made to use alternate file handling methods. If there is anything I should do (maybe show you my handmade pyconfig.h?) tell me.
See http://bugs.python.org/issue12082. Currently neither Python 2.x nor 3.x can be compiled without stat() or fstat(). Python 2.7 almost compiles, but Python 3 depends heavily on them. The problem boils down to the fact that you cannot really check whether a filesystem entry is a directory without calling stat() or fstat(). My personal opinion is that support for DONT_HAVE_STAT and DONT_HAVE_FSTAT defines should be dropped because they don't work, and would only be useful in a very limited set of cases.
[My first post in a mailing list... Should I say] Best Regards, n> Hossein [in here?]
Yeah, why not? :) Regards, Petri
It's 99% definite that I can't compile further without touching the source code. I have to #define my own PYOS_whatever and handle files in my own way. In that case where should my special file handling cases go?
It's difficult to say how to proceed. On one hand, I don't see an overwhelming need to support systems without stat, and am tempted to say that you are on your own. On the other hand, it appears that people keep asking for it, from time to time. So if it was possible to support such systems without making the code too convoluted, it may be worth supporting it. One thing seems clear: without stat(), we cannot possibly support .pyc files, at least not in __pycache__. So one consequence of a lacking stat should be that all the code dealing with caching of byte code files needs to be disabled. Supporting .pyc as modules might still be possible. It's questionable how to deal with path searching in the absence of stat. Testing for the presence of a file is possible in principle by trying to open the file, and closing it when it was found to be present. So in the places where we only check for the presence of a file, an alternative implementation could be provided. In any case, it needs someone to champion such a project, preferably in an ongoing manner (i.e. several years). So if you are interested, you should - volunteer to maintain stat-less systems for some time - create a port of Python 3 that works stat-less - come back to python-dev for review to determine whether it's worth to support such systems. Alternatively, you can just make your own fork of Python, which you may or may not publish. Regards, Martin
On 12/14/2011 2:26 PM, Petri Lehtinen wrote:
The problem boils down to the fact that you cannot really check whether a filesystem entry is a directory without calling stat() or fstat().
My personal opinion is that support for DONT_HAVE_STAT and DONT_HAVE_FSTAT defines should be dropped because they don't work, and would only be useful in a very limited set of cases.
At present, it seems to be an attractive nuisance, tempting people like Hossein to try something that does not work. -- Terry Jan Reedy
On 12/14/2011 05:04 PM, Hossein wrote:
If there is anything I should do
You can determine what the code that calls stat() is trying to do, and implement that with other primitives that your platform provides. For example, you can determine whether a file exists by trying to open it in read-only mode and checking the error. You can find whether a filesystem path names a directory by trying to chdir into it and checking the error. You can find the size of a regular file by opening it and seeking to the end. These substitutions would not be acceptable for a desktop system, but may be perfectly adequate for an embedded one that doesn't provide stat() in the first place. Either way, I expect that you will need to modify the sources. Finally, are you 100% sure that your platform doesn't provide another API similar to stat()?
It does have its own file handling functions: Opening, getting the size, enumerating directories, etc. It has its own limitations too. No dates supported, folders only one level deep, maximum 99 files inside each folder, etc. There is not a function called stat. But I am considering faking it, will explain in another reply.
I wanted to say something in the bug page petri showed ( http://bugs.python.org/issue12082 ) however I though about first discussing it here. If faking a stat struct and a function to fill it solves the problem, and checking for existing files and folders is the only thing that python needs to be compiled (i'm talking about 2.7) then it's possible to fail-check it by just trying to open the file. If you don't want to change the stat mechanism, you can create a new #define which can let user point it to his own faked stat function and struct. I'm currently trying to fake stat to see what happens next, but I guess I will have more problems with file handling later. By the way, some people with the same problem there said they "used" python by setting the Py_DontWriteBytecodeFlag flag, but here my problem is that i can't compile it. Dunno what they really did.
Le jeudi 15 décembre 2011 15:29:23 vous avez écrit :
If faking a stat struct and a function to fill it solves the problem, and checking for existing files and folders is the only thing that python needs to be compiled (i'm talking about 2.7) then it's possible to fail-check it by just trying to open the file.
It's better to only work on Python 3.3. I consider "support platform without stat" as a new feature, and new features are only accepted in Python 3.3. Victor
Am 15.12.2011 12:59, schrieb Hossein:
I wanted to say something in the bug page petri showed ( http://bugs.python.org/issue12082 ) however I though about first discussing it here. If faking a stat struct and a function to fill it solves the problem, and checking for existing files and folders is the only thing that python needs to be compiled (i'm talking about 2.7) then it's possible to fail-check it by just trying to open the file.
That's not true. It also looks at the file modification time. Regards, Martin
participants (7)
-
"Martin v. Löwis" -
Hossein -
Hossein Azadmanesh -
Hrvoje Niksic -
Petri Lehtinen -
Terry Reedy -
Victor Stinner