Ludos,
First let me clarify: the factor in front of the continuous forms is 1/sqrt(2pi), not sqrt(2/pi) like I said previously (I was confused).
if i do
iDST(DST(f(r))
on discrete data, i get the original data back (expected result). this would indicate that
1) either the sqrt(2/pi) norm shouldn't be used for discrete data (why?)
I don't think the factors should be added on at all. The mere transform and invtransform functions should handle all factors. This is related to what I want to type later on.
2) the sqrt(2/pi) factor is taken care of somewhere inside the DST function (how?)
Yes. It should be. Like I said before, the convention chosen doesn't really matter as long as the you do the FT and then the IFT, because the factors from the two operations must multiple together to give 1/(2pi) in the end (this is built into the answer. The problem with convention only comes when you want to compare numbers of transformed data (still in fourier space) with someone else. Then you need to know if your conventions are the same.
this is the thing i'd really love to know - is the answer 1 or answer 2 correct?
I think the actual answer lies in the fact that the fourier transform is actually derived from the fourier series. The fourier series is a sum of some coefficients times an exponential. There is an equation for calculating these coefficients and it is something like 1/(2pi) times an integral, and thats where the 1/(2pi) comes in (actually it comes in when the formula for the coefficients is derived). See the book Functions of a complex variable by Carrier, Krook, and Pearson for the details of this derivation. I think that if you see this, all your concerns about the factors will be taken care of.
-Frank