ruminations on Component Analysis

Most of what I have done in the past year has centered around projections. The crazy part about all of the stuff I have tried doing so far is that I have been doing it without exactly knowing what I was doing. I never had any formal training on PCA, LDA or kernel methods (unless youtube/ wikipedia qualifies as formal). So, now that I am taking MLSP and Pattern Recognition, there are lots of eureka moments that I go through each class. Still, some of the ‘world experts’ that teach our classes (Fernando de la torre, for instance) are at such a high level of understanding that it is hard to keep up with their perspectives on the matter. Fernando, for instance has come up with a master equation that explains PCA, LDA, Kernel PCA, Spectral Clustering and even K-Means as variations of the same theme. I didn’t quite get what the equation means but the general idea seemed to be minimization of reconstruction error, with certain tweaking parameters that define each projection.

The problem with this high level treatment is that we just hear a couple of sentences about each of these topics in class and that is it. The task of picking up the necessary tool and implementing it is left to the student. Spectral clustering and Independent Component Analyses have struck my fancy out of the things that have been mentioned in Fernando’s two classes. And obviously, kernel methods and SVM. Robust PCA was a very interesting idea too.  There is so much to learn, it is crazy!

For the next few days I will try to read up on particular topics each day and blog about it. I read up a bit on ICA today and implemented a few things. I will write about it tomorrow when the hour is more reasonable, but I was surprised at how much better than PCA it was. Actually, the pre-processing step for ICA does something similar (if not exactly) to PCA. With this implementation, I also realized the benefits of having a standard dataset. Since I have the dataset of 15 appliances that I have already done PCA, LDA on, I can immediately know how ICA performs on it- and the results are encouraging. I am thinking of adding it to the paper that  we plan on submitting for the AEI journal.

One thought on “ruminations on Component Analysis

  1. I look forward to your future blog posts on this theme. It may be useful for you to also document the assumptions that each of these methods is making abut the nature of the signals/data. For instance, ICA has some normality (Guassian) assumptions that render it useless in some applications.

    The idea of expanding on some of these for the paper is also good.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>