The quote means that there is an internal component within each person's psyche that prevents complete contentment when transgressions have been committed. Such a quote can be read in multiple directions. On one hand, it can be seen as a statement that there is a moral universal structure, and when an individual violates this particular configuration, their sense of guilt lacks totality and security because of the transgression committed. Another way the statement can be read is that there is no moral structure within the world. The lack of security felt is because this is confirmed when one violates it. There becomes a greater sense of fear and insecurity in the realization that there is no moral structure that is supposed to guide one's moral actions. Dostoevsky's quote can be examined in multiple contexts, confirming the complex nature of the philosophy in his work.
"A guilty conscience never feels secure" by Publius Syrus is a maxim that is true of many characters in literature. One such example is Fyodor Dostovesky's novel, Crime and Punishment in which a young student named Raskolnikov murders an old woman money lender who lives above him to prove his theory that extraordinary men are above the law since in their brillance they "think new thoughts" and, thus, contribute to society. He determines to prove his theory by murdering a cynical old pawnbroker and her sister. However, he does not consider his conscience in his equation.
After Raskolnikov kills the old woman, he reads the newspaper the next day--nothing is in it about a murder on his street. Still, he is watchful, attentive to any prolonged glance at himself or any policeman passing by. Days pass without incident, and this pattern continues until one day, he can stand it no longer. Raskolniknov's comes into contact with his conscience, and he is made wretched by his guilt. He begins to believe that the police officer in charge of the murder watches him. Raskolnikov becomes paranoic, imagining that the officer suspects him. And, the more Raskolnikov intellectualizes, the more imprisoned he becomes. When the officer does notice that Raskolnikov is acting strangely, the sergeant then follows him and eventually arrests him.
Constant worry that he would be found out is actually what has caught Raslkolnikov because he begins to act oddly from his imaginings that the sergeant suspects him.
In Shakespeare's Hamlet, for instance, the same is true. Claudius, who has murdered Hamlet's father, the king, assumes the throne and marries Hamlet's mother. But, because Hamlet begins to act strangely, Claudius worries that he will be found out as one of the murderers. He devises various plans to rid himself of Hamlet, whom he suspects knows much about him. His final plan fails, however, and he himself is killed along with his victims.