This is, of course, a matter of opinion. I do believe that there should be extra punishment for hate crimes because they are more likely to damage the entire basis of our society.
Other crimes are committed on a much more individual basis. A revenge crime, for example, is done specifically because of some problem between the two individuals involved. These crimes happen because of factors related to the indviduals involved. The idea that individuals could hate one another does not erode the very foundations of our society.
By contrast, hate crimes are motivated by ideas that do eat at the basis of our society. Our society is predicated on the idea that all people are equal. When you commit a crime that is done specifically because you do not believe that another group of people is equal to you, you are attacking the very basis of our society in a very serious way.
Therefore, hate crimes are more dangerous to our society as a whole than crimes that are motivated by more individual factors.