Hello scipy users, If one is using 1ofC encoding, what does PCA applied to 1ofC result in? 1ofc is nice if you're trying to get the posterior probability distribution as output (sigmoid single output unit feedforward network), however would this still hold true after applying PCA to reduce input dimensionality? Does this even make "sense" for 1ofc? Furthermore, is there any solid literature reviewing encoding schemes for nnets etc? Thanks, Conor
Conor Robinson wrote:
If one is using 1ofC encoding, what does PCA applied to 1ofC result in? 1ofc is nice if you're trying to get the posterior probability distribution as output (sigmoid single output unit feedforward network), however would this still hold true after applying PCA to reduce input dimensionality? Does this even make "sense" for 1ofc? Furthermore, is there any solid literature reviewing encoding schemes for nnets etc?
From what I've been told, it doesn't make sense to apply PCA to categorical variables, although it is a common practice.
If you want the posterior probability distribution at the outputs, then I believe you want to use softmax. The neural nets FAQ http://www.faqs.org/faqs/ai-faq/neural-nets/ has some practical advice about encoding, with citations to the literature (part 2, "How should categories be encoded?"), as well as some recommendations on the literature in general (part 4). If you haven't read Bishop (1995), I highly recommend it. Aarre -- Aarre Laakso http://www.laakshmi.com/aarre/
Thanks Aarre, I have both of those resources, they are good, but I'm really looking for a study comparing encoding schema. I've been combing the University California libraries, however most studies use a "hand wave" gesture or don't mention how they encode whatsoever. I've developed a basic technique for compressing 1ofC, but I would like to see what others have done. In bishop around p. 230 he notes you can use cross entropy with sigmoid for probabilities. Conor On 8/28/06, Aarre Laakso <aarre@pair.com> wrote:
Conor Robinson wrote:
If one is using 1ofC encoding, what does PCA applied to 1ofC result in? 1ofc is nice if you're trying to get the posterior probability distribution as output (sigmoid single output unit feedforward network), however would this still hold true after applying PCA to reduce input dimensionality? Does this even make "sense" for 1ofc? Furthermore, is there any solid literature reviewing encoding schemes for nnets etc?
From what I've been told, it doesn't make sense to apply PCA to categorical variables, although it is a common practice.
If you want the posterior probability distribution at the outputs, then I believe you want to use softmax.
The neural nets FAQ http://www.faqs.org/faqs/ai-faq/neural-nets/ has some practical advice about encoding, with citations to the literature (part 2, "How should categories be encoded?"), as well as some recommendations on the literature in general (part 4). If you haven't read Bishop (1995), I highly recommend it.
Aarre
-- Aarre Laakso http://www.laakshmi.com/aarre/ _______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
What is 1ofC? Googling for it didn't turn up anything useful. --bb On 8/30/06, Conor Robinson <conor.robinson@gmail.com> wrote:
Thanks Aarre,
I have both of those resources, they are good, but I'm really looking for a study comparing encoding schema. I've been combing the University California libraries, however most studies use a "hand wave" gesture or don't mention how they encode whatsoever. I've developed a basic technique for compressing 1ofC, but I would like to see what others have done. In bishop around p. 230 he notes you can use cross entropy with sigmoid for probabilities.
Conor
One 1ofC is a way of encoding categories for a neural network or other model For example: attribute: color red 100 green 010 blue 001 You can run into problems, however if you have an attribute with lets say 400 different categories, aka the curse of dimensionality. On one had you would like your net to consider all categories (given there are ample examples of each in your training set). RBF functions are especially bad at this (look into the biological premise of the RBF), but fast to train. Feed forward nets are better, but slower to train. In a typical sample you may not have examples of all 400 categories and you would like to compress your input so the net is looking at statically relevant information. Im looking into a GA process for compressing 1ofC, and in general papers or a plethora of information on encoding and represention. In my opinion the represention problem is the cornerstone of any model in most cases. On 8/30/06, Bill Baxter <wbaxter@gmail.com> wrote:
What is 1ofC? Googling for it didn't turn up anything useful.
--bb
On 8/30/06, Conor Robinson <conor.robinson@gmail.com> wrote:
Thanks Aarre,
I have both of those resources, they are good, but I'm really looking for a study comparing encoding schema. I've been combing the University California libraries, however most studies use a "hand wave" gesture or don't mention how they encode whatsoever. I've developed a basic technique for compressing 1ofC, but I would like to see what others have done. In bishop around p. 230 he notes you can use cross entropy with sigmoid for probabilities.
Conor
_______________________________________________ SciPy-user mailing list SciPy-user@scipy.org http://projects.scipy.org/mailman/listinfo/scipy-user
participants (3)
-
Aarre Laakso -
Bill Baxter -
Conor Robinson