The Scarlet Letter can be classified as a historical romance under the sub-genre of Colonial American romance primarily because of its setting. Literary works of this genre are often placed during the time period covering the early settlements and the time frame period before the American Revolution, obviously because after the Revolution the former colonies ceased to be under that status.
This being said, let us focus on the "romantic" elements of The Scarlet Letter. The novel belongs to the genre of Romanticism because its main themes include the treatment of the daily realities of decision making, choices, consequences, pain, joy, and life itself. Nowhere, except with an exception of magical realism (the asteroid scene and Pearl's elfish behavior), do we see "miracles" ocurring to any of the main characters that would lift the punishments that they have brought upon themselves. Hester continues to pay for her infidelity and Dimmesdale brings himself to his last wit as a result of his own sins. Chillingworth achieves an empty victory that leads nowhere and Pearl, once elfish and mysterious, ends up normally married and with child in the end.
Hence, the reality of the situations and the setting in which these situations exist are what make of The Scarlet Letter a historical romance.