> The steps should be for a given image (ex s-aqua):Basically correct.
> *calculate saturation and brightness from gray scale s-aqua. (s1,b1)
> *choose hue (a1)
> Do the same for each fluo
> Use the 5x3 hsb values to calculate a matrix to do the convertion 5 channels->3 RGB
>
> Is it correct?
You could also just calculate brightness from the gray images, and leave the saturation constant.
Also, my suggestion was to use the "screen" operator to combine each image. The 5x3 matrix multiplication effectively makes 5 RGB images (one from each fluor), and then sums them together. In general, I find that "screen" works better than "sum" for making visually-appealing overlays of fluorescence channels.
For [0,1]-valued float images, the formula for screen-combining two images a and b is: a+b-ab. (http://en.wikipedia.org/wiki/Blend_modes#Screen )
Anyway, I'd try both sum and screen to see which looks best.
Zach