Link errors while translating with VS2010 and 64bit
Amaury, It seems that there is a general issue with the compiler/link . I did the translations without the _hashlib and ssl and after a while I got the following errors. [platform:ERROR] Creating library c:\users\flatline\appdata\local\temp\usessi on-default-57\testing_1\libpypy-c.lib and object c:\users\flatline\appdata\local\temp\usession-default-57\testing_1\libpypy-c.exp [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr [platform:ERROR] implement_64.obj : error LNK2019: unresolved external symbol _inflateInit2_ referenced in function _pypy_g_ccall_inflateInit2___z_streamPtr_Sig ned_arrayPtr [platform:ERROR] implement_64.obj : error LNK2019: unresolved external symbol _inflate referenced in function _pypy_g_ccall_inflate__z_streamPtr_Signed [platform:ERROR] implement_67.obj : error LNK2019: unresolved external symbol _deflateEnd referenced in function _pypy_g_ccall_deflateEnd__z_streamPtr [platform:ERROR] implement_69.obj : error LNK2019: unresolved external symbol _deflate referenced in function _pypy_g_ccall_deflate__z_streamPtr_Signed [platform:ERROR] implement_70.obj : error LNK2019: unresolved external symbol _adler32 referenced in function _pypy_g_ccall_adler32__Unsigned_arrayPtr_Unsigned [platform:ERROR] implement_71.obj : error LNK2019: unresolved external symbol _deflateInit2_ referenced in function _pypy_g_ccall_deflateInit2___z_streamPtr_Sig ned_Signed_S [platform:ERROR] implement_71.obj : error LNK2019: unresolved external symbol _crc32 referenced in function _pypy_g_ccall_crc32__Unsigned_arrayPtr_Unsigned [platform:ERROR] c:\users\flatline\appdata\local\temp\usession-default-57\testing_1\libpypy-c.dll : fatal error LNK1120: 8 unresolved externals [Timer] Timings: [Timer] annotate --- 1447.8 s [Timer] rtype_lltype --- 768.2 s [Timer] backendopt_lltype --- 415.9 s [Timer] stackcheckinsertion_lltype --- 60.1 s [Timer] database_c --- 498.3 s [Timer] source_c --- 1190.5 s [Timer] compile_c --- 216.4 s [Timer] =========================================== [Timer] Total: --- 4597.3 s [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "translate.py", line 290, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 812, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\tool\taskengine.py", line 116, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 288, in _do [translation:ERROR] res = func() [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\driver.py", line 575, in task_compile_c [translation:ERROR] cbuilder.compile(**kwds) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 527, in compile [translation:ERROR] self.executable_name = compiler.build(shared=shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 95, in build [translation:ERROR] return self._build(shared=shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\c\genc.py", line 90, in _build [translation:ERROR] standalone=not shared) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 52, in compile [translation:ERROR] return self._finish_linking(ofiles, eci, outputfilename, standalone) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 194, in _finish_linking [translation:ERROR] return self._link(cc_link, ofiles, largs, standalone, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 176, in _link [translation:ERROR] self._execute_c_compiler(self.link, args, exe_name) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\__init__.py", line 118, in _execute_c_compiler [translation:ERROR] self._handle_error(returncode, stderr, stdout, outname) [translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\translator\platform\windows.py", line 199, in _handle_error [translation:ERROR] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: <CompilationError err=' Creating library c:\\users\\flatline\\appdata\\local\\temp\\usession-default-57\\testing_1\\lib pypy-c.lib and obje...e\\appdata\\local\\temp\\usession-default-57\\testing_1\\libpypy-c.dll : fatal error LNK1120: 8 unresolved externals\n'> Thanks, Tasos On Mon, Jan 31, 2011 at 11:48 PM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
Amaury,
Isn't there a way to fix the SSL issue? I was looking checking for name decoration issues or call-conv issues but there seems none.
Is there any guess why it fails from the translation procedure but works when I execute the commands in cmd prompt ?
(I also tried disabling the env processing that is happening, leaving POpen to get the default one but nothing :( )
Thanks, Tasos
On Mon, Jan 31, 2011 at 11:44 PM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
2011/1/31 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Hello Amaury,
Yes. It's the same compiler error :
[translation:ERROR] File "c:\Users\flatline\development\pypy\src\pypy\module\_hashlib\interp_hashlib.py",
Ah, indeed the _hashlib module should be disabled when openssl cannot be processed. I'll try to fix it, in the meantime I suggest you to skip this module:
python translate.py targetpypystandalone --withoutmod-_hashlib
-- Amaury Forgeot d'Arc
Hi, 2011/2/1 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Amaury,
It seems that there is a general issue with the compiler/link .
I did the translations without the _hashlib and ssl and after a while I got the following errors. [...] [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr
We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*) Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors. -- Amaury Forgeot d'Arc
On Tue, Feb 1, 2011 at 10:11 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Hi,
2011/2/1 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Amaury,
It seems that there is a general issue with the compiler/link .
I did the translations without the _hashlib and ssl and after a while I got the following errors. [...] [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr
We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*)
Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors.
Surprisingly enough, I have
-- Amaury Forgeot d'Arc _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Amaury, I'll try and I'll let you know. Thanks, Tasos On 1 Φεβ 2011, at 10:11, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Hi,
2011/2/1 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Amaury,
It seems that there is a general issue with the compiler/link .
I did the translations without the _hashlib and ssl and after a while I got the following errors. [...] [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr
We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*)
Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors.
-- Amaury Forgeot d'Arc
Amaury, You were right, there are some issues with type sizes. test_genc and test_newgc are failing. I'll try to see what I can do to fix them. Are there any directions I should move towards ? Thanks, Tasos On Tue, Feb 1, 2011 at 10:20 AM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
Amaury,
I'll try and I'll let you know.
Thanks, Tasos
On 1 Φεβ 2011, at 10:11, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Hi,
2011/2/1 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Amaury,
It seems that there is a general issue with the compiler/link .
I did the translations without the _hashlib and ssl and after a while I got the following errors. [...] [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr
We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*)
Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors.
-- Amaury Forgeot d'Arc
Hi, A naive guess would be the different sizes between 64bit windows, and 64 bit linux? (long int is 32bit on windows and 64bit on linux)
64-bit data models Data model short (integer) int long (integer) long long pointers/size_t Sample operating systems LLP64/ IL32P64 16 32 32 64 64 Microsoft Windows (X64/IA-64) LP64/ I32LP64 16 32 64 64 64 Most Unix and Unix-like systems, e.g. Solaris, Linux, and Mac OS X ILP64 16 64 64 64 64 HAL Computer Systems port of Solaris to SPARC64 SILP64 64 64 64 64 64 Unicos On Tue, Feb 1, 2011 at 5:10 PM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
Amaury,
You were right, there are some issues with type sizes. test_genc and test_newgc are failing.
I'll try to see what I can do to fix them. Are there any directions I should move towards ?
Thanks, Tasos
On Tue, Feb 1, 2011 at 10:20 AM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
Amaury,
I'll try and I'll let you know.
Thanks, Tasos
On 1 Φεβ 2011, at 10:11, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
Hi,
2011/2/1 Tasos Vogiatzoglou <tvoglou@gmail.com>:
Amaury,
It seems that there is a general issue with the compiler/link .
I did the translations without the _hashlib and ssl and after a while I got the following errors. [...] [platform:ERROR] implement_52.obj : error LNK2019: unresolved external symbol _inflateEnd referenced in function _pypy_g_ccall_inflateEnd__z_streamPtr
We have never tried the win64 platform, and I don't have access to a Windows 64bit machine. But I suspect that even if you removed all external dependencies, the result would not work; pypy's compilation tools implicitly assume that sizeof(long)==sizeof(void*)
Before running a translation, could you run the tests in pypy/translator/c ? in the pypy directory, run: python test_all.py translator/c I'd like at least the files "test_genc" and "test_newgc" to pass without errors.
-- Amaury Forgeot d'Arc
_______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
2011/2/1 René Dudfield <renesd@gmail.com>:
Hi,
A naive guess would be the different sizes between 64bit windows, and 64 bit linux? (long int is 32bit on windows and 64bit on linux)
From http://en.wikipedia.org/wiki/64-bit
64-bit data models Data model short (integer) int long (integer) long long pointers/size_t Sample operating systems LLP64/ IL32P64 16 32 32 64 64 Microsoft Windows (X64/IA-64) LP64/ I32LP64 16 32 64 64 64 Most Unix and Unix-like systems, e.g. Solaris, Linux, and Mac OS X ILP64 16 64 64 64 64 HAL Computer Systems port of Solaris to SPARC64 SILP64 64 64 64 64 64 Unicos
From what Amaury said, you need to identify places where long is used and it is assumed that it has 64bits. A first guess, in general, is that _every_ use of long assumes it to be 64bit on 64bits systems.
What about using intptr_t (or uintptr_t) from stdint.h? IOW, you'd simply replace long by this type everywhere, including in functions which generate C code through some library. I don't know the details, could somebody help here? IIRC this header was introduced in C99, and for sure this type is optional - i.e., it must be declared if it is available at all. Of course, there might be systems where this type is not available, so you need to provide a fallback (in a configure-like fashion). If this header is available on (modern) Windows environments (the only system where long is 32 bits), using "long" as fallback is safe. Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
Hello Paolo, Dunno if I am in the correct direction but till now I found two things 1) test_genc was throwing an error on fixup_c_type (complaining about conflicting field type between HLOCAL and pointer to char array) so I changed it to an opaque pointer (and the HANDLE type) 2) test_genc passed the tests, so the next step was to change the env to use the 64bit libs changing from vsvars32.bat to vcvarsall.bat with amd64 param in translator/platform/windows.py # _get_msvc_env(vsver): After that change I had lot's of linking issues like [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_Py_BuildValue referenced in function malloc_counters [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_PyMethod_New referenced in function gencfunc_descr_get [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get Any idea ? Thanks, Tasos On Wed, Feb 2, 2011 at 1:35 AM, Paolo Giarrusso <pgiarrusso@mathematik.uni-marburg.de> wrote:
2011/2/1 René Dudfield <renesd@gmail.com>:
Hi,
A naive guess would be the different sizes between 64bit windows, and 64 bit linux? (long int is 32bit on windows and 64bit on linux)
From http://en.wikipedia.org/wiki/64-bit
64-bit data models Data model short (integer) int long (integer) long long pointers/size_t Sample operating systems LLP64/ IL32P64 16 32 32 64 64 Microsoft Windows (X64/IA-64) LP64/ I32LP64 16 32 64 64 64 Most Unix and Unix-like systems, e.g. Solaris, Linux, and Mac OS X ILP64 16 64 64 64 64 HAL Computer Systems port of Solaris to SPARC64 SILP64 64 64 64 64 64 Unicos
From what Amaury said, you need to identify places where long is used and it is assumed that it has 64bits. A first guess, in general, is that _every_ use of long assumes it to be 64bit on 64bits systems.
What about using intptr_t (or uintptr_t) from stdint.h? IOW, you'd simply replace long by this type everywhere, including in functions which generate C code through some library. I don't know the details, could somebody help here? IIRC this header was introduced in C99, and for sure this type is optional - i.e., it must be declared if it is available at all.
Of course, there might be systems where this type is not available, so you need to provide a fallback (in a configure-like fashion). If this header is available on (modern) Windows environments (the only system where long is 32 bits), using "long" as fallback is safe.
Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/ _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
2011/2/2 Tasos Vogiatzoglou <tvoglou@gmail.com>:
After that change I had lot's of linking issues like
[platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_Py_BuildValue referenced in function malloc_counters [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_PyMethod_New referenced in function gencfunc_descr_get [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get
Any idea ?
You are certainly building an extension module for the host Python (the one which runs the tests). You should check that the link command finds the correct Python27.lib, corresponding to your installation of Python. btw, are you really running a 64bit Python? -- Amaury Forgeot d'Arc
Amaury, You are right. My mistake. It was picking the 2.6 32bit python. Now is working and tests are passing Thanks, Tasos On Wed, Feb 2, 2011 at 2:06 AM, Amaury Forgeot d'Arc <amauryfa@gmail.com> wrote:
2011/2/2 Tasos Vogiatzoglou <tvoglou@gmail.com>:
After that change I had lot's of linking issues like
[platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_Py_BuildValue referenced in function malloc_counters [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp_PyMethod_New referenced in function gencfunc_descr_get [platform:ERROR] testing_25.obj : error LNK2019: unresolved external symbol __imp__Py_NoneStruct referenced in function gencfunc_descr_get
Any idea ?
You are certainly building an extension module for the host Python (the one which runs the tests). You should check that the link command finds the correct Python27.lib, corresponding to your installation of Python. btw, are you really running a 64bit Python?
-- Amaury Forgeot d'Arc
Hi Tasos, On Wed, Feb 2, 2011 at 1:15 AM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
You are right. My mistake. It was picking the 2.6 32bit python. Now is working and tests are passing
A few extra notes. What is needed to make this work is three-fold: * make sure that in the C code, "intptr_t" is generated instead of "long" for lltype.Signed, and similarly "uintptr_t" instead of "unsigned long" for lltype.Unsigned. This involves mainly fixing translator/c/primitive.py to declare types using "intptr_t" instead of "long", and probably fixing a number of other uses of "long" in the .py files in that directory. * make sure that in the hand-written headers translator/c/src/*.h, we use "intptr_t" instead of "long" whenever it's needed (mostly everywhere, I suppose, but not absolutely everywhere, e.g. not in calls to external C functions that are declared to take or return a long). The changes above should fix the generation of code. Now the third part is the longest, probably. You need to distinguish the two types used in RPython, which are lltype.Signed and rffi.LONG, and which are equal so far. Of course rffi.LONG should remain equivalent to the C long. This probably needs looking around in all modules and in pypy/rlib/, to make sure that we use the correct one, either lltype.Signed or rffi.LONG. Fortunately, errors resulting from this confusion will probably just give translation-time errors, so we can fix the places one after the other. But you need to carefully check the declaration of external functions that are implemented in translator/c/src/*.h, like pypy/module/signal/interp_signal.py (this one happens to use mostly rffi.INT, so it should work right now, with the exception of LONG_STRUCT that uses a lltype.Signed). A bientôt, Armin.
Armin, thanks for the directions. I'll start working on them and I'll report back. Regards, Tasos On 3 Φεβ 2011, at 14:30, Armin Rigo <arigo@tunes.org> wrote:
Hi Tasos,
On Wed, Feb 2, 2011 at 1:15 AM, Tasos Vogiatzoglou <tvoglou@gmail.com> wrote:
You are right. My mistake. It was picking the 2.6 32bit python. Now is working and tests are passing
A few extra notes. What is needed to make this work is three-fold:
* make sure that in the C code, "intptr_t" is generated instead of "long" for lltype.Signed, and similarly "uintptr_t" instead of "unsigned long" for lltype.Unsigned. This involves mainly fixing translator/c/primitive.py to declare types using "intptr_t" instead of "long", and probably fixing a number of other uses of "long" in the .py files in that directory.
* make sure that in the hand-written headers translator/c/src/*.h, we use "intptr_t" instead of "long" whenever it's needed (mostly everywhere, I suppose, but not absolutely everywhere, e.g. not in calls to external C functions that are declared to take or return a long).
The changes above should fix the generation of code. Now the third part is the longest, probably. You need to distinguish the two types used in RPython, which are lltype.Signed and rffi.LONG, and which are equal so far. Of course rffi.LONG should remain equivalent to the C long. This probably needs looking around in all modules and in pypy/rlib/, to make sure that we use the correct one, either lltype.Signed or rffi.LONG. Fortunately, errors resulting from this confusion will probably just give translation-time errors, so we can fix the places one after the other. But you need to carefully check the declaration of external functions that are implemented in translator/c/src/*.h, like pypy/module/signal/interp_signal.py (this one happens to use mostly rffi.INT, so it should work right now, with the exception of LONG_STRUCT that uses a lltype.Signed).
A bientôt,
Armin.
participants (6)
-
Amaury Forgeot d'Arc -
Armin Rigo -
Maciej Fijalkowski -
Paolo Giarrusso -
René Dudfield -
Tasos Vogiatzoglou