On Mon, May 16, 2011 at 5:08 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

Hi Dan,

...

I see there has been some recent activity on Windows issues in the last few days, but I didn't see anything exactly related to this issue, so I opened ticket: https://github.com/stefanv/scikits.image/issues/22 (I only mention this because I don't know if anyone besides Stefan gets notified when an issue is created). The traceback and other details are in the issue on github.

I tried manually loading FreeImage.dll using ctypes in an ipython session and had the same issue, checking FreeImage's header file shows that the function should be there though. Anyone have any ideas on what else I can try?

I suspect it's not just FreeImage_GetPitch that isn't visible, but all FreeImage_ API calls. (You could check this easily enough though...) I think GetPitch is just the first one that gets looked up.

My guess is that the library on Win7 has different name mangling than is expected. I think that in general the library gets loaded as a "cdll" instead of a "windll" on windows, which appears to work with my limited testing on a XP 32-bit virtual machine. Perhaps try loading the library as a windll via ctypes and then see if any of the symbols are visible.

Or you could dump the dll's symbol table and see what's there... (MinGW has some tools for this -- pexports I think? I bet there are MS tools for the same as part of VC++ but I have no idea.)

Sorry to not be more direct help, Zach

Hi Zach, Loading the library with WinDLL worked (unfortunately in the freeimage plugin code it looks like that's handled by numpy.ctypeslib.load_library which apparently doesn't support anything other than cdll). I read up on ctypes and saw that you can also call functions by ordinal. So loading FreeImage.dll using the cdll calling convention and accessing via ordinal worked fine as well. So I'm not sure what your preferred fix would be, I guess just changing the load code to handle cdll versus windll is more clear than mapping functions to ordinals. -Dan

Hi, An additional difficulty is that on some versions of windows with some versions of the FreeImage library, the cdll calling convention is the one to use (I seem to recall, from my limited testing). So I suppose the fix will have to be to load the library one way and test if the functions can be found, and if not, load it the other way. Ugh. Plus the problem with numpy.ctypeslib.load_library. Getting this fixed plus the 64-bit crash bug I tracked down earlier (https://github.com/zachrahan/scikits.image/tree/FreeImage64 ) will hopefully make FreeImage a viable option for image IO for more folks. Zach On May 17, 2011, at 8:07 AM, Stéfan van der Walt wrote:

Hi Dan

On Tue, May 17, 2011 at 5:58 AM, Dan Farmer <dfarmernv@gmail.com> wrote:

...

So I'm not sure what your preferred fix would be, I guess just changing the load code to handle cdll versus windll is more clear than mapping functions to ordinals.

It sounds like we should submit a patch to numpy, but then also use an updated version in scikits.image in the meanwhile. It'd be great if you could have a further look.

Thanks! Stéfan

Cool -- thanks, Dan. Probably best to just branch off of Stefan's master, and then send him a pull request for your new branch. My FreeImage64 branch didn't touch any of the library-loading machinery at all (you could look at its diff), so any changes you make would merge cleanly with mine. Zach On May 17, 2011, at 12:08 PM, Dan Farmer wrote:

Ok, I'll see if I can get the windll thing coded up. In terms of "git-fu" should I base those changes on your branch you listed or wait for that to be merged? I don't want to create a merging mess by having branches of branches, but maybe I'm underestimating git.

-Dan

On Tue, May 17, 2011 at 5:25 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Hi,

An additional difficulty is that on some versions of windows with some versions of the FreeImage library, the cdll calling convention is the one to use (I seem to recall, from my limited testing). So I suppose the fix will have to be to load the library one way and test if the functions can be found, and if not, load it the other way. Ugh.

Plus the problem with numpy.ctypeslib.load_library.

Getting this fixed plus the 64-bit crash bug I tracked down earlier (https://github.com/zachrahan/scikits.image/tree/FreeImage64 ) will hopefully make FreeImage a viable option for image IO for more folks.

Zach

On May 17, 2011, at 8:07 AM, Stéfan van der Walt wrote:

...

Hi Dan

On Tue, May 17, 2011 at 5:58 AM, Dan Farmer <dfarmernv@gmail.com> wrote:

...

So I'm not sure what your preferred fix would be, I guess just changing the load code to handle cdll versus windll is more clear than mapping functions to ordinals.

It sounds like we should submit a patch to numpy, but then also use an updated version in scikits.image in the meanwhile. It'd be great if you could have a further look.

Thanks! Stéfan

So I made the change/minor fork for ctypeslib.load_library and it worked great for me. I had a few thoughts though. 1. The idea about pushing back to Numpy sounds great, but I wonder how to test it well enough? I imagine this is the type of thing that is used a lot and could break a lot of code if the behavior changes (e.g., people have already worked around it in some other manner, etc.) -- this one is minor from the scikits perspective, I just haven't really thought about how to change something in such a mature library before. 2. The "auto-detect" idea that Zach mentioned sounds good (try to import and call a function and if it fails try to load using the other calling convention and try again), but again thinking about Stefan's upstream comment that requires that load_library has some knowledge about the library being loaded (e.g., it needs a candidate test function to try to call). Anyway just thinking out loud, I'll push the branch in the morning (late here) and we can see if some other windows users can try it out to make sure it works on other systems. Thanks, Dan 2011/5/17 Stéfan van der Walt <stefan@sun.ac.za>:

On Tue, May 17, 2011 at 6:44 PM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Probably best to just branch off of Stefan's master, and then send him a pull request for your new branch. My FreeImage64 branch didn't touch any of the library-loading machinery at all (you could look at its diff), so any changes you make would merge cleanly with mine.

Zach's changes are now all merged into my branch.  Thanks, Zach!

Cheers Stéfan

I tested it on Linux and found that I had an indentation error. After a series of embarrassingly stupid commits (sorry :( ) it now works for me on Win7 x64 and Ubuntu 10.04 32bit. Thanks, Dan On Fri, May 20, 2011 at 12:10 AM, Dan Farmer <dfarmernv@gmail.com> wrote:

Sorry for the delay, I pushed this branch

https://github.com/dfarmer/scikits.image/compare/master...dfarmer-windll

I've tested it on my system (Win7 x64 EPD 7), but obviously it needs regression testing in linux (working on my vm, turns out Ubuntu 10.04 doesn't have any of the requirements for scikits.image at the correct versions) and should probably be tested on Windows XP. So if anyone has a chance to test it that'd be cool, otherwise I'll at least try to run it on ubuntu in the next day or two.

Thanks, Dan

On Thu, May 19, 2011 at 12:07 AM, Dan Farmer <dfarmernv@gmail.com> wrote:

...

So I made the change/minor fork for ctypeslib.load_library and it worked great for me. I had a few thoughts though.

1. The idea about pushing back to Numpy sounds great, but I wonder how to test it well enough? I imagine this is the type of thing that is used a lot and could break a lot of code if the behavior changes (e.g., people have already worked around it in some other manner, etc.) -- this one is minor from the scikits perspective, I just haven't really thought about how to change something in such a mature library before.

2. The "auto-detect" idea that Zach mentioned sounds good (try to import and call a function and if it fails try to load using the other calling convention and try again), but again thinking about Stefan's upstream comment that requires that load_library has some knowledge about the library being loaded (e.g., it needs a candidate test function to try to call).

Anyway just thinking out loud, I'll push the branch in the morning (late here) and we can see if some other windows users can try it out to make sure it works on other systems.

Thanks, Dan

2011/5/17 Stéfan van der Walt <stefan@sun.ac.za>:

...

On Tue, May 17, 2011 at 6:44 PM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Probably best to just branch off of Stefan's master, and then send him a pull request for your new branch. My FreeImage64 branch didn't touch any of the library-loading machinery at all (you could look at its diff), so any changes you make would merge cleanly with mine.

Zach's changes are now all merged into my branch.  Thanks, Zach!

Cheers Stéfan

On Sun, May 22, 2011 at 3:19 PM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

I've been meaning to test this on my XP image, but I know that the previous version (which opened the library as cdll) worked properly on XP. So I wonder if there aren't different versions of the FreeImage dll that were compiled differently? This may complicate matters!

Strange that it worked at all then. Does anybody know the difference between cdll and windll? Regards Stéfan

Zach's essentially got it. It has to do with the different x86 calling conventions that are available. Most compilers use cdecl, but Microsoft's compilers sometimes use something called stdcall (which, looking at the symbols from FreeImage.dll, appears to be the case there). So ctypes.cdll loads a dll using cdecl convention and ctypes.windll loads a dll using stdcall. The unfortunate thing is that you're not guaranteed to always get stdcall dlls on windows and cdecl everywhere else. MSVC++ can compile cdecl exe's also, it's just a flag. Since this particular code is for the freeimage bindings though my hope is that if it passes the "download the latest freeimage binary, unzip and run" test then maybe we can call it good? -Dan On Mon, May 23, 2011 at 5:12 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Strange that it worked at all then.  Does anybody know the difference between cdll and windll?

It's to do with how the symbols are declared in the library, and how the arguments are pushed to the stack (or something similar). Not all dlls on windows seem to be compiled with the windll calling convention, though I don't know why or in what circumstances you see that.

Anyway, I'll test further when I get a chance (sorry, very busy here).

Dan, your patch works fine on XP. I was in error in thinking that some other version of FreeImage.dll was compiled as cdecl. (Long story...) Zach On May 23, 2011, at 11:33 AM, Zachary Pincus wrote:

Dan, if you could check that the relevant binaries (x64 and i386) of the latest version of FreeImage available online are stdcall, then that fix should be good.

Otherwise (or perhaps in any case) we could add a "sniff_function" parameter to load_library, which would be the name of a symbol to look for in the library, and if not present, try to open the library with the other convention. Something like:

if sys.platform == 'win32': windll = ctypes.windll(libpath) if sniff_function is not None and not hasattr(dll, sniff_function): cdll = ctypes.cdll(libpath) if hasattr(cdll, sniff_function): return cdll else: raise AttributeError("Could not find function %s in %s"%(sniff_function, libpath)) return windll

On May 23, 2011, at 11:16 AM, Dan Farmer wrote:

...

Zach's essentially got it. It has to do with the different x86 calling conventions that are available. Most compilers use cdecl, but Microsoft's compilers sometimes use something called stdcall (which, looking at the symbols from FreeImage.dll, appears to be the case there). So ctypes.cdll loads a dll using cdecl convention and ctypes.windll loads a dll using stdcall.

The unfortunate thing is that you're not guaranteed to always get stdcall dlls on windows and cdecl everywhere else. MSVC++ can compile cdecl exe's also, it's just a flag. Since this particular code is for the freeimage bindings though my hope is that if it passes the "download the latest freeimage binary, unzip and run" test then maybe we can call it good?

-Dan

On Mon, May 23, 2011 at 5:12 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

...

Strange that it worked at all then. Does anybody know the difference between cdll and windll?

It's to do with how the symbols are declared in the library, and how the arguments are pushed to the stack (or something similar). Not all dlls on windows seem to be compiled with the windll calling convention, though I don't know why or in what circumstances you see that.

Anyway, I'll test further when I get a chance (sorry, very busy here).

On Tue, May 24, 2011 at 2:21 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Dan, your patch works fine on XP. I was in error in thinking that some other version of FreeImage.dll was compiled as cdecl. (Long story...)

Does the patch work everywhere, then?

I have no reason to believe it wouldn't, but I haven't tested exhaustively. (Packing for a trip right now...) Zach

On Thu, May 26, 2011 at 5:33 AM, James Turner <jturner@gemini.edu> wrote:

Just wondering whether you're planning on doing a sprint at SciPy this year?

I think that's a great idea, if we can get enough developers together. By that stage, we should have the new backends in place, and there are plenty of interesting side projects to integrate as well. Will you be there? Stéfan

I think that's a great idea, if we can get enough developers together. By that stage, we should have the new backends in place, and there are plenty of interesting side projects to integrate as well. Will you be there?

Yes, I've signed up for the sprints without a clear agenda and am just finalizing my flights for my ~4 week trip. I might take off slightly early on 16th, as it's my birthday and a bit depressing to spend it all in the basement, but depends who's around and doing what :-). Hopefully I can help out a little bit (eg. never did the PyFITS save function from last year). Or maybe if I get a chance to think about it beforehand I could do something slightly more ambitious this time. I'll at least try to remind myself how git works :-). Cheers, James.

There'll be plenty to do!

It might be good to add some interpolation functions, but I'm not sure how realistic it is for 2 days' work (by me). There's some Netlib stuff that could probably "just" be wrapped, if you're interested, but it's ForTran and I don't know whether (doubt) you'd want to require that for compilation? Also, it might be too much along the lines of what I need and not what you want for the scikit, but it's just something to think about... Cheers, James.

On Thu, May 26, 2011 at 7:34 PM, James Turner <jturner@gemini.edu> wrote:

...

It might be good to add some interpolation functions, but I'm not sure how realistic it is for 2 days' work (by me). There's some Netlib stuff that could probably "just" be wrapped, if you're interested, but it's ForTran and I don't know whether (doubt) you'd want to require that for compilation? Also, it might be too much along the lines of what I need and not what you want for the scikit, but it's just something to think about...

Fast interpolation algorithms using PyOpenCL may be worth looking at. I heard today of an interesting interpolation method that does not go exactly through data-points, but is extremely efficient to implement--will look into that some more. Either way, being able to size up and down large images efficiently would certainly be useful.

Is it just going "close enough" to the data points or does it do something

2011/5/26 Stéfan van der Walt <stefan@sun.ac.za> like least-squares fitting to noisy data? If the latter, do you have a reference for this method? Thanks, Ralf

Scipy's ndimage does B-spline interpolation well, although I've never been able to decipher exactly what happens below the hood :)

Cheers Stéfan

2011/5/31 Stéfan van der Walt <stefan@sun.ac.za>

On Fri, May 27, 2011 at 6:22 PM, Ralf Gommers <ralf.gommers@googlemail.com> wrote:

...

...

Fast interpolation algorithms using PyOpenCL may be worth looking at. I heard today of an interesting interpolation method that does not go exactly through data-points, but is extremely efficient to implement--will look into that some more. Either way, being able to size up and down large images efficiently would certainly be useful.

Is it just going "close enough" to the data points or does it do something like least-squares fitting to noisy data? If the latter, do you have a reference for this method?

http://dip.sun.ac.za/~herbst/research/publications/modeling.pdf

Have a look at the new paragraph starting on p. 337.

This stuff is really interesting :)

It is.

The technique is called subdivision surfaces, which I also didn't hear of before but which seems to be common in computer graphics. A good summary of related techniques and their applications is given in Chapter 2 of Krishnamurthy, "FITTING SMOOTH SURFACES TO DENSE POLYGON MESHES"; http://graphics.stanford.edu/papers/surfacefitting/surf_fit.pdf Cheers, Ralf

Sorry, this thread sort of went sideways :P I just wanted to follow up and say that I've tested it on all of my machines (Win 7 and Ubuntu) and Zach tried it on his Win XP system -- I think that's all the testing we can come up with. I believe that this should be good to go. I'd still invite anyone else who uses the freeimage plugin to try it out if you have time, but I think in theory this is good to go. -Dan On Tue, May 24, 2011 at 4:31 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

On Tue, May 24, 2011 at 2:21 AM, Zachary Pincus <zachary.pincus@yale.edu> wrote:

...

Dan, your patch works fine on XP. I was in error in thinking that some other version of FreeImage.dll was compiled as cdecl. (Long story...)

Does the patch work everywhere, then?

I have no reason to believe it wouldn't, but I haven't tested exhaustively. (Packing for a trip right now...)

Zach

No big deal! I just figured I'd try to close out the last question before I forgot about it. On Mon, May 30, 2011 at 2:30 PM, James Turner <jturner@gemini.edu> wrote:

...

Sorry, this thread sort of went sideways :P

Oh, sorry I didn't change your title...