Clearly, Hamlet is not mad. However, he is under a great deal of stress. No doubt, Hamlet is shaken by the visitation of his father's ghost. This is not a normal, everyday occurrence. Still, Hamlet has his reasoning abilities. The proof of this is in the fact that Hamlet is hesitant about avenging his father's death based only on the message from his father's ghost. Hamlet does not immediately act on his father's message. He weighs out the message. He seeks proof that his Uncle Claudius did in fact kill his father.
Hamlet is very clever. He uses good sense. He does not make a rash decision. He utilizes the help of some traveling actors. He has the traveling actors reenact his father's murder. Then Hamlet carefully watches his Uncle Claudius' reaction. He is carefully checking for signs of guilt. As the actors reenact his father's death, Hamlet carefully observes his Uncle Claudius and learns that Claudius indeed shows signs of guilt. Claudius is flustered and cannot remain seated during the reeenactment. He cries out for light and flees the room.
Claudius' actions convince Hamlet that Claudius is indeed guilty. Still, Hamlet waits for the right time to avenge his father's death. When Claudius is praying, confessing his guilt, Hamlet decides not to avenge his father's death for fear that Claudius will die and find redemption from his sincere confession in prayer.
Hamlet appears to be very sensible. He seems to be clearly evaluating the situation. He shows no signs of madness. He seems to know exactly what he is doing.
The only thing Hamlet does wrong is that he waits too long to kill Claudius. Claudius plans to have Hamlet murdered before Hamlet can kill him. On Hamlet's death bed, he gets his revenge and forces Claudius to drink of the poison meant for Hamlet. Then Hamlet dies.