If we want to analyse a literary text using literary theories, how can we know the theory that should be adapted to that work?it is said that theories are like keys: each one fits its door
The premise behind literary theories is that a particular individual or group of individuals develop or adhere to set principles in a given literary theory, like Russian Formalism. Many theories also are founded in ideological precepts and have embedded assumptions to prove, such as Marxist theory or Feminist theory. Since (1) many of us are not strong advocates of ideological precepts (e.g., Marxism) and (2) have no assumptions to prove (e.g., binary prevalence of deconstruction), we can enjoy selecting from the pot-pouri of literary theories. We'd make our choices based on (a) the characteristics of the novel, play, etc or on (b) what we perceive as significant in the novel etc.
For instance, Pride and Prejudice lends itself to analysis through Feminist theory because of all the women and their varying roles, while Master and Commander: The Far Side of the World (also a great movie) lends itself to Deconstructionism because a good number of the characters appear in marginalized capacities that are prime for deconstruction relating to the binary of master and servant, which reveals the socio-political realities of colonial English life.
I am not sure that I agree with the wording of the question. It implies that a given literary work can be analysed correctly only by one literary theory. Instead I feel that a single literary work can be analysed by a lot of different literary strategies. It is not the literary theory that should be adapted to the work, but the work that should be analysed through the lenses of the literary theory we choose to use. But the choice of which school of criticism to follow in a literary analysis is ours. It cannot be dictated by the literary text that we are analysing and depends what elements of the text we are particularly interested in. For example, texts produced in the former colonies of European countries invite analysis informed by postcolonial theories. Yet, these theories can be fruitfully applied to texts produced by white Europeans about colonies, thus highlighting stereotyping and racist representations.
A book that I have found particularly useful to explain how different reading strategies can be applied to novels is Peter Messent's New Readings of the American Novel (1998). While, as the title suggests, Messent's primary focus and examples are American novels, his study provides methodological insights that can be helpful to scholars and students of other literatures.