A science fiction novel presents a story based in the context of the future and/or the development of science and technology and its effects on the society in the story. Science itself can be said to be almost a character in the novel. It affects the characters, their goals, and the outcome of how those goals are met.
Dystopian fiction, however, is more about the present. While the action of the story may take place in the future, it reflects in a major way the circumstances of the time in which it is written.
In Brave New World, for example, science is used to create a willing citizenry to further the goals of the powers that be. By implying that the main goal of science is to improve the life of the individual, it in fact makes that life one of easy control. It is the whip in the hand of the master.
A story that centers on a dystopian environment is one that is opposite to a utopian environment, or a perfect or ideal society.
The dystopian environment is a society that lacks harmony, is distorted politically, culturally and socially. A good example of a dystopian Science Fiction novel is "The Time Machine" by H.G. Well, where he tells the story of a time traveller who is in search of a perfect society, he goes far into the future and finds a perfect dystopia where everything he imagined the future should be is not there, but society has reverted through evolution to a primitive existence. He is vastly disappointed at what he finds in the future, really shocked.
Science Fiction novels are often set in the future and involve fantasy, advanced technology. In the world of Science Fiction, anything is possible, time travel, space travel into other galaxies, all different kinds of aliens, even aliens and humans getting along as part of the same federation of planets.
Where dystopian novels are predictors of doom,stories about the dark future that awaits if we don't change our ways in the present, Science Fiction can be fun, interesting and have a happy ending.
Some great Science Fiction writers, H.G. Wells, Isac Asimov, George Orwell.
Dystopian just means the opposite of Utopia and Utopia means perfect society. Therefore, dystopian novels depict an utterly imperfect and flawed society. 1984 is a superb example of this. Dystopian novels can also be science fiction novels if they refer to a future society or utilize futuristic technology. Science fiction novels don't always have to be dystopian. For instance To the Moon and Back is not about society at all--its just about getting to the Moon.