Best multispectral to RGB conversion
Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html?showComment=1332...>. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here<https://github.com/jeanpat/MFISH> Best Regard Jean-Patrick Pommier
Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here
More details on the input images would be helpful: are they the result of performing spectral unmixing, so each image represents the abundance of a particular fluorophore? Or are the images the raw results? The best approach would probably differ between the two options. If the images are unmixed, then you can calculate the "true" spectrum at each pixel using the images and the emission spectra from the fluorophores. Then using these spectra, you could calculate the effective color in a number of ways, probably the simplest being a weighted average of the "red", "green", and "blue" regions of the spectra. You'd want to look at the psychophysics literature to figure out how best to do this, but I'm sure there are a ton of references online for converting a spectrum into an RGB color. If the images are mixed (that is, raw), it's even easier: figure out the "apparent color" of the emission filter for each image (see below for sample code), and then colorize each image by the apparent color, and screen the images together. (Code for screening a set of images together is below, too -- 'multi_screen()') Or, returning to the unmixed case, you could forget using the "natural" colors, and instead choose a set of colors maximally dissimilar from one another in some color space as the color for each fluorophore. This might make visualization simpler. You'd choose the colors (not sure how best), and then colorize the images and screen them together. Zach def screen(src, dst, max=255): return max - (((max - src)*(max - dst))/max) def multi_screen(images, max=255): dst = images[0] for image in images[1:]: dst = screen(image, dst, max) return dst def wavelength_to_rgb(l): # http://www.physics.sfasu.edu/astro/color/spectra.html # So-called "Bruton's Algorithm" assert (350 <= l <= 780) if l < 440: R = (440-l)/(440-350.) G = 0 B = 1 elif l < 490: R = 0 G = (l-440)/(490-440.) B = 1 elif l < 510: R = 0 G = 1 B = (510-l)/(510-490.) elif l < 580: R = (l-510)/(580-510.) G = 1 B = 0 elif l < 645: R = 1 G = (645-l)/(645-580.) B = 0 else: R = 1 G = 0 B = 0 if l > 700: intensity = 0.3 + 0.7 * (780-l)/(780-700.) elif l < 420: intensity = 0.3 + 0.7 * (l-350)/(420-350.) else: intensity = 1 return intensity*numpy.array([R,G,B], dtype=float)
Each image is a raw image of a given fluorochrome. A given fluorochrome targets a subset of chromosomes, more informations can be found in: *Automatic Segmentation and Classification of Multiplex-Fluorescence In-Situ Hybridization Chromosome Images by Hyo Hun Choi, B.S., M.S. * Choi gives an example of CCD integration times: * An example of integration times is [DAPI, Aqua, Green, Gold, Red, Far Red] = [0.14, 6, 0.76, 6, 2.96, 1.4] seconds* He also gives an example of spectral mixing in the raw images: <https://lh3.googleusercontent.com/-dw-YEQxPyR4/T2r66g-u0EI/AAAAAAAAAvo/NLfMv...> With the vysis data set (example:https://github.com/jeanpat/MFISH/tree/master/MFISH_TIFF/V13__40), a given fluorochrome corresponds to several chromosomes, in the spectrumAqua image the chromosomes 3,10,11,13,15,20,21,22,X,Y are visibles. A: S. Aqua D DAPI F: Far Red G: S. Green R: S. Red Y: S. Gold To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves? Best regards Jean-Pat Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit :
Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html?showComment=1332...>. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here<https://github.com/jeanpat/MFISH>
Best Regard
Jean-Patrick Pommier
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit :
Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html?showComment=1332...>. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here<https://github.com/jeanpat/MFISH>
Best Regard
Jean-Patrick Pommier
<https://lh5.googleusercontent.com/-N3cAlGUsoSU/T2r-_A2iTtI/AAAAAAAAAv0/5RFfB...> Sorry, the spectral mixing table was not included (see before, I failed to insert an image where I want ....) Le jeudi 22 mars 2012 11:26:38 UTC+1, jip a écrit :
Each image is a raw image of a given fluorochrome.
A given fluorochrome targets a subset of chromosomes, more informations can be found in:
*Automatic Segmentation and Classification of Multiplex-Fluorescence In-Situ Hybridization Chromosome Images by Hyo Hun Choi, B.S., M.S. * Choi gives an example of CCD integration times: * An example of integration times is [DAPI, Aqua, Green, Gold, Red, Far Red] = [0.14, 6, 0.76, 6, 2.96, 1.4] seconds*
He also gives an example of spectral mixing in the raw images:
<https://lh3.googleusercontent.com/-dw-YEQxPyR4/T2r66g-u0EI/AAAAAAAAAvo/NLfMv...> With the vysis data set (example: https://github.com/jeanpat/MFISH/tree/master/MFISH_TIFF/V13__40), a given fluorochrome corresponds to several chromosomes, in the spectrumAqua image the chromosomes 3,10,11,13,15,20,21,22,X,Y are visibles.
A: S. Aqua D DAPI F: Far Red G: S. Green R: S. Red Y: S. Gold
To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves?
Best regards Jean-Pat
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit :
Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html?showComment=1332...>. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here<https://github.com/jeanpat/MFISH>
Best Regard
Jean-Patrick Pommier
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit :
Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html?showComment=1332...>. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here<https://github.com/jeanpat/MFISH>
Best Regard
Jean-Patrick Pommier
Jean-Pat,
To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves?
Wait, are you trying to perform spectral unmixing (5 channels -> 25 channels, one per chromosome), or false coloring (5 channels -> 3 channel RGB)? The first email made it sound like the latter, but this one seems more on the former... If you want to do unmixing, there's a large literature about the MFISH case specifically, and I think there are some pretty well-known methods at this point. Presumably it would help to know the integration times and spectral characteristics of the instrument / fluorophores, especially given the table you appended to the second email about the spectral mixing issues. False-coloring is an easier subject. You'd just need to choose "target" RGB colors for each of the 5 raw images, colorize each image by that color, and screen them together. Most of the art is in choosing good "target" colors. You could choose based on the original colors ("aqua" etc.), or choose equidistant points on the hue-circle (with equivalent saturation, and brightness determined by the original grayscale pixel intensity). Zach On Mar 22, 2012, at 6:26 AM, jip wrote:
Each image is a raw image of a given fluorochrome.
A given fluorochrome targets a subset of chromosomes, more informations can be found in:
Automatic Segmentation and Classification of Multiplex-Fluorescence In-Situ Hybridization Chromosome Images by Hyo Hun Choi, B.S., M.S. Choi gives an example of CCD integration times:
An example of integration times is [DAPI, Aqua, Green, Gold, Red, Far Red] = [0.14, 6, 0.76, 6, 2.96, 1.4] seconds
He also gives an example of spectral mixing in the raw images:
With the vysis data set (example:https://github.com/jeanpat/MFISH/tree/master/MFISH_TIFF/V13__40), a given fluorochrome corresponds to several chromosomes, in the spectrumAqua image the chromosomes 3,10,11,13,15,20,21,22,X,Y are visibles.
A: S. Aqua D DAPI F: Far Red G: S. Green R: S. Red Y: S. Gold
To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves?
Best regards Jean-Pat
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit : Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here
Best Regard
Jean-Patrick Pommier
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit : Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here
Best Regard
Jean-Patrick Pommier
Le jeudi 22 mars 2012 16:43:29 UTC+1, Zachary Pincus a écrit :
Jean-Pat,
To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves?
Wait, are you trying to perform spectral unmixing (5 channels -> 25 channels, one per chromosome), or false coloring (5 channels -> 3 channel RGB)? The first email made it sound like the latter, but this one seems more on the former...
Sorry, you're right, my first question was about the 5 ->3 channels issue.
If you want to do unmixing, there's a large literature about the MFISH case specifically, and I think there are some pretty well-known methods at this point. Presumably it would help to know the integration times and spectral characteristics of the instrument / fluorophores, especially given the table you appended to the second email about the spectral mixing issues.
False-coloring is an easier subject. You'd just need to choose "target" RGB colors for each of the 5 raw images, colorize each image by that color, and screen them together. Most of the art is in choosing good "target" colors. You could choose based on the original colors ("aqua" etc.),
I met that problem when I chose the matrix coef<http://dip4fish.blogspot.fr/2011/05/making-mfish-image.html>(with Stefan's code<http://scikits-image.googlegroups.com/attach/47734e89812cf070/mfish2.py?gda=50mjJEYAAAAqA5IhiUlAD9seu9KfIPGeo0O4UeZ8bNXTrc8MzC849tXYX1TPrmUiszAk75wvtTtx40jamwa1UURqDcgHarKEE-Ea7GxYMt0t6nY0uV5FIQ&part=4>). The matrix was doing the job with: 'R': [0, 0, 0.05, 0.15, 0.8], 'V': [0.1, 0.8, 0.05, 0.05, 0], 'B': [0.8, 0.15, 0.05, 0, 0]}
or choose equidistant points on the hue-circle (with equivalent saturation, and brightness determined by the original grayscale pixel intensity).
The steps should be for a given image (ex s-aqua): *calculate saturation and brightness from gray scale s-aqua. (s1,b1) *choose hue (a1) Do the same for each fluo Use the 5x3 hsb values to calculate a matrix to do the convertion 5 channels->3 RGB Is it correct? Jean-Pat Zach
On Mar 22, 2012, at 6:26 AM, jip wrote:
Each image is a raw image of a given fluorochrome.
A given fluorochrome targets a subset of chromosomes, more informations can be found in:
Automatic Segmentation and Classification of Multiplex-Fluorescence In-Situ Hybridization Chromosome Images by Hyo Hun Choi, B.S., M.S. Choi gives an example of CCD integration times:
An example of integration times is [DAPI, Aqua, Green, Gold, Red, Far Red] = [0.14, 6, 0.76, 6, 2.96, 1.4] seconds
He also gives an example of spectral mixing in the raw images:
With the vysis data set (example: https://github.com/jeanpat/MFISH/tree/master/MFISH_TIFF/V13__40<https://github.com/jeanpat/MFISH/tree/master/MFISH_TIFF/V13__40>), a given fluorochrome corresponds to several chromosomes, in the spectrumAqua image the chromosomes 3,10,11,13,15,20,21,22,X,Y are visibles.
A: S. Aqua D DAPI F: Far Red G: S. Green R: S. Red Y: S. Gold
To perform "perfect" spectral unmixing, I suppose it's necessary to know the integration time, the characteristic of the microscope bandpass filters, the emission spectrum of each fluorochrome; but would it be possible to make some kind of "estimated" unmixing from only the images themselves?
Best regards Jean-Pat
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit : Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here
Best Regard
Jean-Patrick Pommier
Le mercredi 21 mars 2012 12:10:16 UTC+1, jip a écrit : Dear all, Some one asked me how I choose the matrix coef to convert a five multispectral image to a three (RGB) multispectral image. I feel that there is a best way to do that by maximising some distance in a color space. How would you do that? Examples of multispectral images are available here
Best Regard
Jean-Patrick Pommier
The steps should be for a given image (ex s-aqua): *calculate saturation and brightness from gray scale s-aqua. (s1,b1) *choose hue (a1) Do the same for each fluo Use the 5x3 hsb values to calculate a matrix to do the convertion 5 channels->3 RGB
Is it correct?
Basically correct. You could also just calculate brightness from the gray images, and leave the saturation constant. Also, my suggestion was to use the "screen" operator to combine each image. The 5x3 matrix multiplication effectively makes 5 RGB images (one from each fluor), and then sums them together. In general, I find that "screen" works better than "sum" for making visually-appealing overlays of fluorescence channels. For [0,1]-valued float images, the formula for screen-combining two images a and b is: a+b-ab. (http://en.wikipedia.org/wiki/Blend_modes#Screen ) Anyway, I'd try both sum and screen to see which looks best. Zach
Thank you Jean-Pat 2012/3/22 Zachary Pincus <zachary.pincus@yale.edu>
The steps should be for a given image (ex s-aqua): *calculate saturation and brightness from gray scale s-aqua. (s1,b1) *choose hue (a1) Do the same for each fluo Use the 5x3 hsb values to calculate a matrix to do the convertion 5 channels->3 RGB
Is it correct?
Basically correct.
You could also just calculate brightness from the gray images, and leave the saturation constant.
Also, my suggestion was to use the "screen" operator to combine each image. The 5x3 matrix multiplication effectively makes 5 RGB images (one from each fluor), and then sums them together. In general, I find that "screen" works better than "sum" for making visually-appealing overlays of fluorescence channels.
For [0,1]-valued float images, the formula for screen-combining two images a and b is: a+b-ab. (http://en.wikipedia.org/wiki/Blend_modes#Screen )
Anyway, I'd try both sum and screen to see which looks best.
Zach
participants (3)
-
Jean-Patrick Pommier -
jip
-
Zachary Pincus