How do I know what the appropriate literary criticism aproach must apply in a certain literary genre, or literary work?
This is a great question. I agree with the previous poster and want to add my own thoughts on the topic. (I teach literature and literary theory courses, and I'm always glad to talk about the two subjects!)
For me, that most important thing is that the particular approach in literary criticism "fit" the particular literary work that you are looking at. "Fit" does not mean, of course, that you can only apply feminist theory to works written by women or marxist theory to works by working-class writers. What this does mean is that the concerns of the text (broadly defined) match up with the concerns of the theory in a way that will produce an interesting and insightful discussion of both. Sometimes the best "fit" is an entirely unexpected pairing of a particular theoretical approach and a particular text.
For example, the short poem "Restaurant" by Maxine Hong Kingston gives us insight into what goes on in the basement kitchen of a restaurant. The workers are very busy, are made to feel small, and have no share in what they produce: the final lines of the poem have the kitchen workers, as they head out to the street, glance through the glass windows of the ground-floor restaurant to watch diners eating by candelight. It makes complete sense to apply a marxist approach to this poem. Other approaches might work, too, but this poem calls out for a marxist reading.
To give another example, the long short story "The Dead" by James Joyce focuses almost exclusively on the inner workings of the mind of one character, Gabriel Conroy, as he celebrates the holiday with family and friends. With the emphasis on his thoughts, reactions, obsessions, anxieties, and so on, this short story begs for a psychoanalytic reading.
Sometimes it's worthwhile to apply to two or more very different theories to one text in order to see which theory works best in the end; each result can be judged by whether that particular theory yields an insightful (and, ideally, fresh) perspective.
Theoretical approaches are often not mutually exclusive and can sometimes be combined in interesting ways.
That is a good question. The most important thing to remember with literary criticism is that there are no rules. This basically means that you can use any literary theory to any text. Some of the most interesting studies are when this is done. For example, Thomas Kuhn's theory of scientific change has been applied to anthropology, theology and history with great profit. I will add a link to the bottom to show you one example. Also you should keep in mind by doing this, each theory forces you to ask a different set of questions. The more you ask, the better your insights will be. So, think outside of the box.
There is no rule that a work can only be viewed through one theory of literary criticism. Multiple theories can be applied with different passages for greater explication. Literary theories should not button hole a work but bring out its greatest artistic merit.