The short answer to this is no, Victor in Shelley's Frankenstein is not an anti-hero.
Anti-heroes usually get the little things wrong, but the big things right. They may not be polite and fit into society and may not think like society does (they are nonconformists), but they possess a higher sense of justice than others. When it matters and counts, anti-heroes do the right thing.
An anti-hero may also be a rogue, but will be a charming rogue--at least to those he wants to charm.
Victor is good at the little things. He is polite and respectful and studious--he probably would have made a great husband (which doesn't matter in this novel, because his wife is killed). But he gets the big things wrong, the things that really matter. He fails to take responsibility for his creation. He fails to nurture his "son" as a father should. He creates life then neglects it.
The creation of life might serve as an instance of "bucking" the system in his day, and make him an anti-hero, except that he doesn't take care of what he's created and he causes tragedy. An anti-hero is ethical when it counts. Victor is not.