First, it is important to understand that this is a quotation from a novel, narrated in the first person, rather than a direct expression of Dostoevsky's own view. However, the author often addressed the problem to which he alludes here: that of how to exercise free will in a world of scientific and mathematical certainties. It therefore clearly fits into his philosophy, though this philosophy is not the programmatic worldview of an academic thinker, but a moral and artistic exploration of ideas.
The problem for Dostoevsky is that a purely rational view of life seems to exclude the possibility of freedom. You are not free to believe that 2 + 2 = 5. To exercise freedom in this sense is merely to be mistaken or mad. If there is a single choice one can make that is more rational than any other, just as there is a correct solution for any mathematical problem, then one can always predict exactly how a rational man will act. Therefore, the rational man has no freedom, and the only escape from mental slavery is into irrationality.