Looking at the text, it is clear beginning from Act I that Hamlet makes a conscious decision to feign madness, which is to act as though he has lost all reason and rationality. What is unspoken in the text, though a large part of cultural Renaissance life, is the conflict between the cultural view and the Protestant view of revenge killing.
Hamlet is clearly a dedicated Protestant because he has been away at university in Witenberg. Wittenberg houses the university at which Protestant reformer Martin Luther was educated and at which he later taught as a professor. The late King Hamlet was clearly not a Protestant--or at least not to the extent to which Hamlet is--since his dying mission is to coerce his son into playing his culturally required (but religiously denied) role of committing revenge killing to avenge the assassination by Claudius.
Fortinbras illustrates perfectly the culturally required reaction of a son whose father has been murdered. This reaction is even more profoundly required of a Prince whose King/Father has been assassinated. Hamlet is placed in a harrowing position: It's not a mere act he has to choose to perform or not; it is a spiritual action that has the potential, in his mind, to damn him to hell everlasting.
For this reason, Hamlet must be sure to an irreproachable degree that he understands the Ghost's identity correctly and the guilt of Claudius correctly [add to this the personal and spiritual horror Hamlet feels in seeing his mother marry in opposition to Catholic and Protestant religious precepts that disallow marriage between a man and woman formerly related by marriage (in other words, the brother- or sister-in-law)]. Even then he must have the courage to abandon his spiritual beliefs to honor his cultural duty as imposed upon him by the Ghost, who, in Hamlet's eyes, is a harbinger of evil and horror, for no matter which way Hamlet turns, he is doomed: He is doomed to dishonor King, Father, and country or he is doomed to dishonor and disobey the precepts and commands of his religion and God.
If there is any true madness, the madness comes form this: Hamlet is caught between the proverbial rock and hard place. His life is dammed and doomed no matter what he does. He eventually quits trying to choose and simply acts according to the cultural example Fortinbras sets before him. All die as a result of Hamlet's reaction to Fortinbras' example. This seems to condemn the cultural requirement for revenge even though Fortinbras carries it off with such aplomb and with such honor.
Even though cultural requirements--represented in Fortinbras (thus the significant similarities and striking differences between Fortinbras and Hamlet)--win out in the end and seem to reign supreme as correct norms of action, there remains an underlying question: While all appearances indicate that revenge killing is the correct path to take, needing no contemplation prior to action, is this really true? Is it really true that revenge killing is right whereas religious beliefs, which give God the power of revenge and humans the duty of mercy, are wrong?
If there is true madness in Hamlet, then surely it stems from trying to know and understand the answer to this question, especially since it is no less than God and King/Father who are pressing against him from two opposing sides; who are creating the rock and the hard place that he is caught between.