As your question implies, there are a number of ways to approach the critical analysis of a novel, a task that is made even more complex by the several "schools" of literary analysis--that is, critics who advocate approaches to critical analysis through history, sociology, politics, economics, gender-related concerns, as well as critical theories that argue critical analysis--formalism, deconstructionism--requires only the text under discussion, nothing external.
From a practical standpoint, however, critical analysis of novels begins with an understanding of what is important in a particular novel--characterization, setting, plot, conflict (among others)--and the nature of the novel under discussion: historical novel, novel of manners, detective novel, psychological novel, regional novel, epistolary novel, to name only a few permutations.
Perhaps the most important step in critical analysis is to identify the novel's organizing theme--whether that theme is carried by characterization, plot, setting, conflict between individuals and/or society. In many cases, of course, a novel's organizing principal may be multifaceted, dependant upon several aspects of human experience and leading us along several tracks at once that will require a sophisticated integration of critical theory.
If we define novels as fictional representations of life and human experience, then we can approach a particular novel's structure in an Aristotelian way--for example, does the novel have a discernible beginning, middle, end? Are the characters believable (can you recognize them?), and is the conflict universal or idiosyncratic (that is, unique to one or more characters)?
Sometimes--and this sounds simplistic (and is)--the best analysis comes out of a simple question--Is this a good story? And then answer that question by close attention to textual elements. On the other hand, if the novel seems incomprehensible, based on your own experience, then you need to understand what aspects of the novel make it incomprehensible--the structure, characters, conflict, use of language, lack of organizing principle, individual experience to which no one can relate.